You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Linyuxin <li...@huawei.com> on 2017/02/21 03:44:32 UTC
[SparkSQL] pre-check syntex before running spark job?
Hi All,
Is there any tool/api to check the sql syntax without running spark job actually?
Like the siddhiQL on storm here:
SiddhiManagerService. validateExecutionPlan
https://github.com/wso2/siddhi/blob/master/modules/siddhi-core/src/main/java/org/wso2/siddhi/core/SiddhiManagerService.java
it can validate the syntax before running the sql on storm
this is very useful for exposing sql string as a DSL of the platform.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
答复: [SparkSQL] pre-check syntex before running spark job?
Posted by Linyuxin <li...@huawei.com>.
Hi Gurdit Singh
Thanks. It is very helpful.
发件人: Gurdit Singh [mailto:Gurdit.Singh@bitwiseglobal.com]
发送时间: 2017年2月22日 13:31
收件人: Linyuxin <li...@huawei.com>; Irving Duran <ir...@gmail.com>; Yong Zhang <ja...@hotmail.com>
抄送: Jacek Laskowski <ja...@japila.pl>; user <us...@spark.apache.org>
主题: RE: [SparkSQL] pre-check syntex before running spark job?
Hi, you can use spark sql Antlr grammer for pre check you syntax.
https://github.com/apache/spark/blob/acf71c63cdde8dced8d108260cdd35e1cc992248/sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4
From: Linyuxin [mailto:linyuxin@huawei.com]
Sent: Wednesday, February 22, 2017 7:34 AM
To: Irving Duran <ir...@gmail.com>>; Yong Zhang <ja...@hotmail.com>>
Cc: Jacek Laskowski <ja...@japila.pl>>; user <us...@spark.apache.org>>
Subject: 答复: [SparkSQL] pre-check syntex before running spark job?
Actually，I want a standalone jar as I can check the syntax without spark execution environment
发件人: Irving Duran [mailto:irving.duran@gmail.com]
发送时间: 2017年2月21日 23:29
收件人: Yong Zhang <ja...@hotmail.com>>
抄送: Jacek Laskowski <ja...@japila.pl>>; Linyuxin <li...@huawei.com>>; user <us...@spark.apache.org>>
主题: Re: [SparkSQL] pre-check syntex before running spark job?
You can also run it on REPL and test to see if you are getting the expected result.
Thank You,
Irving Duran
On Tue, Feb 21, 2017 at 8:01 AM, Yong Zhang <ja...@hotmail.com>> wrote:
You can always use explain method to validate your DF or SQL, before any action.
Yong
________________________________
From: Jacek Laskowski <ja...@japila.pl>>
Sent: Tuesday, February 21, 2017 4:34 AM
To: Linyuxin
Cc: user
Subject: Re: [SparkSQL] pre-check syntex before running spark job?
Hi,
Never heard about such a tool before. You could use Antlr to parse SQLs (just as Spark SQL does while parsing queries). I think it's a one-hour project.
Jacek
On 21 Feb 2017 4:44 a.m., "Linyuxin" <li...@huawei.com>> wrote:
Hi All,
Is there any tool/api to check the sql syntax without running spark job actually?
Like the siddhiQL on storm here:
SiddhiManagerService. validateExecutionPlan
https://github.com/wso2/siddhi/blob/master/modules/siddhi-core/src/main/java/org/wso2/siddhi/core/SiddhiManagerService.java
it can validate the syntax before running the sql on storm
this is very useful for exposing sql string as a DSL of the platform.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org<ma...@spark.apache.org>
RE: [SparkSQL] pre-check syntex before running spark job?
Posted by Gurdit Singh <Gu...@bitwiseglobal.com>.
Hi, you can use spark sql Antlr grammer for pre check you syntax.
https://github.com/apache/spark/blob/acf71c63cdde8dced8d108260cdd35e1cc992248/sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4
From: Linyuxin [mailto:linyuxin@huawei.com]
Sent: Wednesday, February 22, 2017 7:34 AM
To: Irving Duran <ir...@gmail.com>; Yong Zhang <ja...@hotmail.com>
Cc: Jacek Laskowski <ja...@japila.pl>; user <us...@spark.apache.org>
Subject: 答复: [SparkSQL] pre-check syntex before running spark job?
Actually，I want a standalone jar as I can check the syntax without spark execution environment
发件人: Irving Duran [mailto:irving.duran@gmail.com]
发送时间: 2017年2月21日 23:29
收件人: Yong Zhang <ja...@hotmail.com>>
抄送: Jacek Laskowski <ja...@japila.pl>>; Linyuxin <li...@huawei.com>>; user <us...@spark.apache.org>>
主题: Re: [SparkSQL] pre-check syntex before running spark job?
You can also run it on REPL and test to see if you are getting the expected result.
Thank You,
Irving Duran
On Tue, Feb 21, 2017 at 8:01 AM, Yong Zhang <ja...@hotmail.com>> wrote:
You can always use explain method to validate your DF or SQL, before any action.
Yong
________________________________
From: Jacek Laskowski <ja...@japila.pl>>
Sent: Tuesday, February 21, 2017 4:34 AM
To: Linyuxin
Cc: user
Subject: Re: [SparkSQL] pre-check syntex before running spark job?
Hi,
Never heard about such a tool before. You could use Antlr to parse SQLs (just as Spark SQL does while parsing queries). I think it's a one-hour project.
Jacek
On 21 Feb 2017 4:44 a.m., "Linyuxin" <li...@huawei.com>> wrote:
Hi All,
Is there any tool/api to check the sql syntax without running spark job actually?
Like the siddhiQL on storm here:
SiddhiManagerService. validateExecutionPlan
https://github.com/wso2/siddhi/blob/master/modules/siddhi-core/src/main/java/org/wso2/siddhi/core/SiddhiManagerService.java
it can validate the syntax before running the sql on storm
this is very useful for exposing sql string as a DSL of the platform.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org<ma...@spark.apache.org>
答复: [SparkSQL] pre-check syntex before running spark job?
Posted by Linyuxin <li...@huawei.com>.
Actually，I want a standalone jar as I can check the syntax without spark execution environment
发件人: Irving Duran [mailto:irving.duran@gmail.com]
发送时间: 2017年2月21日 23:29
收件人: Yong Zhang <ja...@hotmail.com>
抄送: Jacek Laskowski <ja...@japila.pl>; Linyuxin <li...@huawei.com>; user <us...@spark.apache.org>
主题: Re: [SparkSQL] pre-check syntex before running spark job?
You can also run it on REPL and test to see if you are getting the expected result.
Thank You,
Irving Duran
On Tue, Feb 21, 2017 at 8:01 AM, Yong Zhang <ja...@hotmail.com>> wrote:
You can always use explain method to validate your DF or SQL, before any action.
Yong
________________________________
From: Jacek Laskowski <ja...@japila.pl>>
Sent: Tuesday, February 21, 2017 4:34 AM
To: Linyuxin
Cc: user
Subject: Re: [SparkSQL] pre-check syntex before running spark job?
Hi,
Never heard about such a tool before. You could use Antlr to parse SQLs (just as Spark SQL does while parsing queries). I think it's a one-hour project.
Jacek
On 21 Feb 2017 4:44 a.m., "Linyuxin" <li...@huawei.com>> wrote:
Hi All,
Is there any tool/api to check the sql syntax without running spark job actually?
Like the siddhiQL on storm here:
SiddhiManagerService. validateExecutionPlan
https://github.com/wso2/siddhi/blob/master/modules/siddhi-core/src/main/java/org/wso2/siddhi/core/SiddhiManagerService.java
it can validate the syntax before running the sql on storm
this is very useful for exposing sql string as a DSL of the platform.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org<ma...@spark.apache.org>
Re: [SparkSQL] pre-check syntex before running spark job?
Posted by Irving Duran <ir...@gmail.com>.
You can also run it on REPL and test to see if you are getting the expected
result.
Thank You,
Irving Duran
On Tue, Feb 21, 2017 at 8:01 AM, Yong Zhang <ja...@hotmail.com> wrote:
> You can always use explain method to validate your DF or SQL, before any
> action.
>
>
> Yong
>
>
> ------------------------------
> *From:* Jacek Laskowski <ja...@japila.pl>
> *Sent:* Tuesday, February 21, 2017 4:34 AM
> *To:* Linyuxin
> *Cc:* user
> *Subject:* Re: [SparkSQL] pre-check syntex before running spark job?
>
> Hi,
>
> Never heard about such a tool before. You could use Antlr to parse SQLs
> (just as Spark SQL does while parsing queries). I think it's a one-hour
> project.
>
> Jacek
>
> On 21 Feb 2017 4:44 a.m., "Linyuxin" <li...@huawei.com> wrote:
>
> Hi All,
> Is there any tool/api to check the sql syntax without running spark job
> actually?
>
> Like the siddhiQL on storm here:
> SiddhiManagerService. validateExecutionPlan
> https://github.com/wso2/siddhi/blob/master/modules/siddhi-
> core/src/main/java/org/wso2/siddhi/core/SiddhiManagerService.java
> it can validate the syntax before running the sql on storm
>
> this is very useful for exposing sql string as a DSL of the platform.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>
>
Re: [SparkSQL] pre-check syntex before running spark job?
Posted by Yong Zhang <ja...@hotmail.com>.
You can always use explain method to validate your DF or SQL, before any action.
Yong
________________________________
From: Jacek Laskowski <ja...@japila.pl>
Sent: Tuesday, February 21, 2017 4:34 AM
To: Linyuxin
Cc: user
Subject: Re: [SparkSQL] pre-check syntex before running spark job?
Hi,
Never heard about such a tool before. You could use Antlr to parse SQLs (just as Spark SQL does while parsing queries). I think it's a one-hour project.
Jacek
On 21 Feb 2017 4:44 a.m., "Linyuxin" <li...@huawei.com>> wrote:
Hi All,
Is there any tool/api to check the sql syntax without running spark job actually?
Like the siddhiQL on storm here:
SiddhiManagerService. validateExecutionPlan
https://github.com/wso2/siddhi/blob/master/modules/siddhi-core/src/main/java/org/wso2/siddhi/core/SiddhiManagerService.java
it can validate the syntax before running the sql on storm
this is very useful for exposing sql string as a DSL of the platform.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org<ma...@spark.apache.org>
Re: [SparkSQL] pre-check syntex before running spark job?
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi,
Never heard about such a tool before. You could use Antlr to parse SQLs
(just as Spark SQL does while parsing queries). I think it's a one-hour
project.
Jacek
On 21 Feb 2017 4:44 a.m., "Linyuxin" <li...@huawei.com> wrote:
Hi All,
Is there any tool/api to check the sql syntax without running spark job
actually?
Like the siddhiQL on storm here:
SiddhiManagerService. validateExecutionPlan
https://github.com/wso2/siddhi/blob/master/modules/
siddhi-core/src/main/java/org/wso2/siddhi/core/SiddhiManagerService.java
it can validate the syntax before running the sql on storm
this is very useful for exposing sql string as a DSL of the platform.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org