You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by LantaoJin <gi...@git.apache.org> on 2018/03/22 06:58:16 UTC
[GitHub] spark pull request #20876: [SPARK-23653][SQL] Capture sql statements user in...
GitHub user LantaoJin opened a pull request:
https://github.com/apache/spark/pull/20876
[SPARK-23653][SQL] Capture sql statements user input and show them in…
… SQL UI
## What changes were proposed in this pull request?
[SPARK-4871](https://issues.apache.org/jira/browse/SPARK-4871) had already added the sql statement in job description for using spark-sql. But it has some problems:
1. long sql statement cannot be displayed in description column.
2. variables like `${var}`, `${env:var}` in sql cannot be resolved.
spark-sql --hiveconf a=avalue --hivevar b=bvalue
`spark-sql> select '${a}', '${b}';`
3. sql statement submitted in spark-shell or spark-submit cannot be covered.
In eBay, most spark sql applications like ETL, reporting using spark-submit to schedule their jobs with a few sql files. The sql statement in those applications cannot be saw in current spark UI.
More detail a scenario is team A developed a framework to submit application with sql sentences in a file
> spark-submit --master yarn-cluster --class com.ebay.SQLFramework -s biz.sql
In the biz.sql, there are many sql sentences like
> create or replace temporary view view_a select xx from table ${old_db}.table_a where dt=${check_date};
> insert overwrite table ${new_db}.table_a select xx from view_a join ${new_db}.table_b;
> ...
Team B (Platform) need to capture the really sql sentences which are executed in whole cluster, as the sql files from Team A contains many variables. A better way is recording the really sql sentence in EventLog.
## How was this patch tested?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/LantaoJin/spark SPARK-23653-NEW
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20876.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20876
----
commit b147b27d6241f1654e7770a5518f66a61d8430e1
Author: LantaoJin <ji...@...>
Date: 2018-03-22T06:47:57Z
[SPARK-23653][SQL] Capture sql statements user input and show them in SQL UI
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20876: [SPARK-23653][SQL] Capture sql statements user input and...
Posted by LantaoJin <gi...@git.apache.org>.
Github user LantaoJin commented on the issue:
https://github.com/apache/spark/pull/20876
In https://github.com/apache/spark/pull/20803, the implementation is to bind sql text to DF. That's not good and will introduce many unexpected issues. I open this PR with new implementation. @jerryshao @cloud-fan and @dongjoon-hyun, sorry for that need you move to here to review.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20876: [SPARK-23653][SQL] Capture sql statements user input and...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20876
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20876: [SPARK-23653][SQL] Capture sql statements user input and...
Posted by LantaoJin <gi...@git.apache.org>.
Github user LantaoJin commented on the issue:
https://github.com/apache/spark/pull/20876
Could anyone else attend to review this? Or should it be closed?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20876: [SPARK-23653][SQL] Capture sql statements user in...
Posted by LantaoJin <gi...@git.apache.org>.
Github user LantaoJin closed the pull request at:
https://github.com/apache/spark/pull/20876
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20876: [SPARK-23653][SQL] Capture sql statements user input and...
Posted by LantaoJin <gi...@git.apache.org>.
Github user LantaoJin commented on the issue:
https://github.com/apache/spark/pull/20876
Hi, @jerryshao @cloud-fan, may I have some update?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20876: [SPARK-23653][SQL] Capture sql statements user input and...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20876
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20876: [SPARK-23653][SQL] Capture sql statements user input and...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20876
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org