You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by GitBox <gi...@apache.org> on 2019/12/28 10:07:28 UTC
[GitHub] [zeppelin] zjffdu opened a new pull request #3579: [ZEPPELIN-4522].
Support multiple sql statements for SparkSqlInterpreter
zjffdu opened a new pull request #3579: [ZEPPELIN-4522]. Support multiple sql statements for SparkSqlInterpreter
URL: https://github.com/apache/zeppelin/pull/3579
### What is this PR for?
Use the SqlSplitter in `zeppelin-interpreter` to split sql and execute in SparkSqlInterpreter. Nothing changes for the previous single sql statement paragraph. But just multiple result will be displayed for multiple sql statements.
### What type of PR is it?
[Feature]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4522
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] asfgit closed pull request #3579: [ZEPPELIN-4522].
Support multiple sql statements for SparkSqlInterpreter
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #3579: [ZEPPELIN-4522]. Support multiple sql statements for SparkSqlInterpreter
URL: https://github.com/apache/zeppelin/pull/3579
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] zjffdu commented on a change in pull request #3579:
[ZEPPELIN-4522]. Support multiple sql statements for SparkSqlInterpreter
Posted by GitBox <gi...@apache.org>.
zjffdu commented on a change in pull request #3579: [ZEPPELIN-4522]. Support multiple sql statements for SparkSqlInterpreter
URL: https://github.com/apache/zeppelin/pull/3579#discussion_r361796512
##########
File path: spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkSqlInterpreter.java
##########
@@ -82,26 +85,35 @@ public InterpreterResult internalInterpret(String st, InterpreterContext context
sparkInterpreter.getZeppelinContext().setInterpreterContext(context);
SQLContext sqlc = sparkInterpreter.getSQLContext();
SparkContext sc = sqlc.sparkContext();
- sc.setLocalProperty("spark.scheduler.pool", context.getLocalProperties().get("pool"));
- sc.setJobGroup(Utils.buildJobGroupId(context), Utils.buildJobDesc(context), false);
-
- try {
- Method method = sqlc.getClass().getMethod("sql", String.class);
- int maxResult = Integer.parseInt(context.getLocalProperties().getOrDefault("limit",
- "" + sparkInterpreter.getZeppelinContext().getMaxResult()));
- String msg = sparkInterpreter.getZeppelinContext().showData(
- method.invoke(sqlc, st), maxResult);
- sc.clearJobGroup();
- return new InterpreterResult(Code.SUCCESS, msg);
- } catch (Exception e) {
- if (Boolean.parseBoolean(getProperty("zeppelin.spark.sql.stacktrace"))) {
- return new InterpreterResult(Code.ERROR, ExceptionUtils.getStackTrace(e));
+
+ StringBuilder builder = new StringBuilder();
+ List<String> sqls = sqlSplitter.splitSql(st);
+ for (String sql : sqls) {
+ sc.setLocalProperty("spark.scheduler.pool", context.getLocalProperties().get("pool"));
+ sc.setJobGroup(Utils.buildJobGroupId(context), Utils.buildJobDesc(context), false);
+
+ try {
+ Method method = sqlc.getClass().getMethod("sql", String.class);
+ int maxResult = Integer.parseInt(context.getLocalProperties().getOrDefault("limit",
+ "" + sparkInterpreter.getZeppelinContext().getMaxResult()));
+ String result = sparkInterpreter.getZeppelinContext().showData(
+ method.invoke(sqlc, sql), maxResult);
+ sc.clearJobGroup();
+ builder.append(result);
+ } catch (Exception e) {
+ if (Boolean.parseBoolean(getProperty("zeppelin.spark.sql.stacktrace"))) {
Review comment:
Make sense, fixed it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] zjffdu commented on a change in pull request #3579:
[ZEPPELIN-4522]. Support multiple sql statements for SparkSqlInterpreter
Posted by GitBox <gi...@apache.org>.
zjffdu commented on a change in pull request #3579: [ZEPPELIN-4522]. Support multiple sql statements for SparkSqlInterpreter
URL: https://github.com/apache/zeppelin/pull/3579#discussion_r361856115
##########
File path: spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkSqlInterpreter.java
##########
@@ -82,26 +85,37 @@ public InterpreterResult internalInterpret(String st, InterpreterContext context
sparkInterpreter.getZeppelinContext().setInterpreterContext(context);
SQLContext sqlc = sparkInterpreter.getSQLContext();
SparkContext sc = sqlc.sparkContext();
+
+ StringBuilder builder = new StringBuilder();
+ List<String> sqls = sqlSplitter.splitSql(st);
+ int maxResult = Integer.parseInt(context.getLocalProperties().getOrDefault("limit",
+ "" + sparkInterpreter.getZeppelinContext().getMaxResult()));
+
sc.setLocalProperty("spark.scheduler.pool", context.getLocalProperties().get("pool"));
sc.setJobGroup(Utils.buildJobGroupId(context), Utils.buildJobDesc(context), false);
-
+ String curSql = null;
try {
- Method method = sqlc.getClass().getMethod("sql", String.class);
- int maxResult = Integer.parseInt(context.getLocalProperties().getOrDefault("limit",
- "" + sparkInterpreter.getZeppelinContext().getMaxResult()));
- String msg = sparkInterpreter.getZeppelinContext().showData(
- method.invoke(sqlc, st), maxResult);
- sc.clearJobGroup();
- return new InterpreterResult(Code.SUCCESS, msg);
+ for (String sql : sqls) {
Review comment:
Empty will be returned if there's no sql, I just added one test case which only 2 sql comments in the paragraph text. Thanks for the review @alexott
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] zjffdu commented on a change in pull request #3579:
[ZEPPELIN-4522]. Support multiple sql statements for SparkSqlInterpreter
Posted by GitBox <gi...@apache.org>.
zjffdu commented on a change in pull request #3579: [ZEPPELIN-4522]. Support multiple sql statements for SparkSqlInterpreter
URL: https://github.com/apache/zeppelin/pull/3579#discussion_r361796491
##########
File path: spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkSqlInterpreter.java
##########
@@ -82,26 +85,35 @@ public InterpreterResult internalInterpret(String st, InterpreterContext context
sparkInterpreter.getZeppelinContext().setInterpreterContext(context);
SQLContext sqlc = sparkInterpreter.getSQLContext();
SparkContext sc = sqlc.sparkContext();
- sc.setLocalProperty("spark.scheduler.pool", context.getLocalProperties().get("pool"));
- sc.setJobGroup(Utils.buildJobGroupId(context), Utils.buildJobDesc(context), false);
-
- try {
- Method method = sqlc.getClass().getMethod("sql", String.class);
- int maxResult = Integer.parseInt(context.getLocalProperties().getOrDefault("limit",
- "" + sparkInterpreter.getZeppelinContext().getMaxResult()));
- String msg = sparkInterpreter.getZeppelinContext().showData(
- method.invoke(sqlc, st), maxResult);
- sc.clearJobGroup();
- return new InterpreterResult(Code.SUCCESS, msg);
- } catch (Exception e) {
- if (Boolean.parseBoolean(getProperty("zeppelin.spark.sql.stacktrace"))) {
- return new InterpreterResult(Code.ERROR, ExceptionUtils.getStackTrace(e));
+
+ StringBuilder builder = new StringBuilder();
+ List<String> sqls = sqlSplitter.splitSql(st);
+ for (String sql : sqls) {
+ sc.setLocalProperty("spark.scheduler.pool", context.getLocalProperties().get("pool"));
+ sc.setJobGroup(Utils.buildJobGroupId(context), Utils.buildJobDesc(context), false);
+
+ try {
+ Method method = sqlc.getClass().getMethod("sql", String.class);
+ int maxResult = Integer.parseInt(context.getLocalProperties().getOrDefault("limit",
+ "" + sparkInterpreter.getZeppelinContext().getMaxResult()));
+ String result = sparkInterpreter.getZeppelinContext().showData(
+ method.invoke(sqlc, sql), maxResult);
+ sc.clearJobGroup();
Review comment:
Make sense, and I think we didn't reset the local property(job group is just a local property) in other places, created ZEPPELIN-4523 to track it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] zjffdu commented on a change in pull request #3579:
[ZEPPELIN-4522]. Support multiple sql statements for SparkSqlInterpreter
Posted by GitBox <gi...@apache.org>.
zjffdu commented on a change in pull request #3579: [ZEPPELIN-4522]. Support multiple sql statements for SparkSqlInterpreter
URL: https://github.com/apache/zeppelin/pull/3579#discussion_r361796465
##########
File path: spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkSqlInterpreter.java
##########
@@ -82,26 +85,35 @@ public InterpreterResult internalInterpret(String st, InterpreterContext context
sparkInterpreter.getZeppelinContext().setInterpreterContext(context);
SQLContext sqlc = sparkInterpreter.getSQLContext();
SparkContext sc = sqlc.sparkContext();
- sc.setLocalProperty("spark.scheduler.pool", context.getLocalProperties().get("pool"));
- sc.setJobGroup(Utils.buildJobGroupId(context), Utils.buildJobDesc(context), false);
-
- try {
- Method method = sqlc.getClass().getMethod("sql", String.class);
- int maxResult = Integer.parseInt(context.getLocalProperties().getOrDefault("limit",
- "" + sparkInterpreter.getZeppelinContext().getMaxResult()));
- String msg = sparkInterpreter.getZeppelinContext().showData(
- method.invoke(sqlc, st), maxResult);
- sc.clearJobGroup();
- return new InterpreterResult(Code.SUCCESS, msg);
- } catch (Exception e) {
- if (Boolean.parseBoolean(getProperty("zeppelin.spark.sql.stacktrace"))) {
- return new InterpreterResult(Code.ERROR, ExceptionUtils.getStackTrace(e));
+
+ StringBuilder builder = new StringBuilder();
+ List<String> sqls = sqlSplitter.splitSql(st);
+ for (String sql : sqls) {
+ sc.setLocalProperty("spark.scheduler.pool", context.getLocalProperties().get("pool"));
+ sc.setJobGroup(Utils.buildJobGroupId(context), Utils.buildJobDesc(context), false);
+
+ try {
+ Method method = sqlc.getClass().getMethod("sql", String.class);
Review comment:
Make sense, and I think we didn't reset the spark local properties in other places. Created ZEPPELIN-4523 for it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] alexott commented on a change in pull request #3579:
[ZEPPELIN-4522]. Support multiple sql statements for SparkSqlInterpreter
Posted by GitBox <gi...@apache.org>.
alexott commented on a change in pull request #3579: [ZEPPELIN-4522]. Support multiple sql statements for SparkSqlInterpreter
URL: https://github.com/apache/zeppelin/pull/3579#discussion_r361794139
##########
File path: spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkSqlInterpreter.java
##########
@@ -82,26 +85,35 @@ public InterpreterResult internalInterpret(String st, InterpreterContext context
sparkInterpreter.getZeppelinContext().setInterpreterContext(context);
SQLContext sqlc = sparkInterpreter.getSQLContext();
SparkContext sc = sqlc.sparkContext();
- sc.setLocalProperty("spark.scheduler.pool", context.getLocalProperties().get("pool"));
- sc.setJobGroup(Utils.buildJobGroupId(context), Utils.buildJobDesc(context), false);
-
- try {
- Method method = sqlc.getClass().getMethod("sql", String.class);
- int maxResult = Integer.parseInt(context.getLocalProperties().getOrDefault("limit",
- "" + sparkInterpreter.getZeppelinContext().getMaxResult()));
- String msg = sparkInterpreter.getZeppelinContext().showData(
- method.invoke(sqlc, st), maxResult);
- sc.clearJobGroup();
- return new InterpreterResult(Code.SUCCESS, msg);
- } catch (Exception e) {
- if (Boolean.parseBoolean(getProperty("zeppelin.spark.sql.stacktrace"))) {
- return new InterpreterResult(Code.ERROR, ExceptionUtils.getStackTrace(e));
+
+ StringBuilder builder = new StringBuilder();
+ List<String> sqls = sqlSplitter.splitSql(st);
+ for (String sql : sqls) {
+ sc.setLocalProperty("spark.scheduler.pool", context.getLocalProperties().get("pool"));
+ sc.setJobGroup(Utils.buildJobGroupId(context), Utils.buildJobDesc(context), false);
+
+ try {
+ Method method = sqlc.getClass().getMethod("sql", String.class);
+ int maxResult = Integer.parseInt(context.getLocalProperties().getOrDefault("limit",
+ "" + sparkInterpreter.getZeppelinContext().getMaxResult()));
+ String result = sparkInterpreter.getZeppelinContext().showData(
+ method.invoke(sqlc, sql), maxResult);
+ sc.clearJobGroup();
+ builder.append(result);
+ } catch (Exception e) {
+ if (Boolean.parseBoolean(getProperty("zeppelin.spark.sql.stacktrace"))) {
Review comment:
Maybe output the SQL clause that caused error as part of the error, in addition to exception? I don't remember, if Spark SQL always shows the original SQL, or not.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] alexott commented on a change in pull request #3579:
[ZEPPELIN-4522]. Support multiple sql statements for SparkSqlInterpreter
Posted by GitBox <gi...@apache.org>.
alexott commented on a change in pull request #3579: [ZEPPELIN-4522]. Support multiple sql statements for SparkSqlInterpreter
URL: https://github.com/apache/zeppelin/pull/3579#discussion_r361794016
##########
File path: spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkSqlInterpreter.java
##########
@@ -82,26 +85,35 @@ public InterpreterResult internalInterpret(String st, InterpreterContext context
sparkInterpreter.getZeppelinContext().setInterpreterContext(context);
SQLContext sqlc = sparkInterpreter.getSQLContext();
SparkContext sc = sqlc.sparkContext();
- sc.setLocalProperty("spark.scheduler.pool", context.getLocalProperties().get("pool"));
- sc.setJobGroup(Utils.buildJobGroupId(context), Utils.buildJobDesc(context), false);
-
- try {
- Method method = sqlc.getClass().getMethod("sql", String.class);
- int maxResult = Integer.parseInt(context.getLocalProperties().getOrDefault("limit",
- "" + sparkInterpreter.getZeppelinContext().getMaxResult()));
- String msg = sparkInterpreter.getZeppelinContext().showData(
- method.invoke(sqlc, st), maxResult);
- sc.clearJobGroup();
- return new InterpreterResult(Code.SUCCESS, msg);
- } catch (Exception e) {
- if (Boolean.parseBoolean(getProperty("zeppelin.spark.sql.stacktrace"))) {
- return new InterpreterResult(Code.ERROR, ExceptionUtils.getStackTrace(e));
+
+ StringBuilder builder = new StringBuilder();
+ List<String> sqls = sqlSplitter.splitSql(st);
+ for (String sql : sqls) {
+ sc.setLocalProperty("spark.scheduler.pool", context.getLocalProperties().get("pool"));
+ sc.setJobGroup(Utils.buildJobGroupId(context), Utils.buildJobDesc(context), false);
Review comment:
Does it makes sense to move this outside of the loop? Or at least move some things like, `Utils.buildJobGroupId(context)`, `Utils.buildJobDesc(context)`, ...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] alexott commented on a change in pull request #3579:
[ZEPPELIN-4522]. Support multiple sql statements for SparkSqlInterpreter
Posted by GitBox <gi...@apache.org>.
alexott commented on a change in pull request #3579: [ZEPPELIN-4522]. Support multiple sql statements for SparkSqlInterpreter
URL: https://github.com/apache/zeppelin/pull/3579#discussion_r361794076
##########
File path: spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkSqlInterpreter.java
##########
@@ -82,26 +85,35 @@ public InterpreterResult internalInterpret(String st, InterpreterContext context
sparkInterpreter.getZeppelinContext().setInterpreterContext(context);
SQLContext sqlc = sparkInterpreter.getSQLContext();
SparkContext sc = sqlc.sparkContext();
- sc.setLocalProperty("spark.scheduler.pool", context.getLocalProperties().get("pool"));
- sc.setJobGroup(Utils.buildJobGroupId(context), Utils.buildJobDesc(context), false);
-
- try {
- Method method = sqlc.getClass().getMethod("sql", String.class);
- int maxResult = Integer.parseInt(context.getLocalProperties().getOrDefault("limit",
- "" + sparkInterpreter.getZeppelinContext().getMaxResult()));
- String msg = sparkInterpreter.getZeppelinContext().showData(
- method.invoke(sqlc, st), maxResult);
- sc.clearJobGroup();
- return new InterpreterResult(Code.SUCCESS, msg);
- } catch (Exception e) {
- if (Boolean.parseBoolean(getProperty("zeppelin.spark.sql.stacktrace"))) {
- return new InterpreterResult(Code.ERROR, ExceptionUtils.getStackTrace(e));
+
+ StringBuilder builder = new StringBuilder();
+ List<String> sqls = sqlSplitter.splitSql(st);
+ for (String sql : sqls) {
+ sc.setLocalProperty("spark.scheduler.pool", context.getLocalProperties().get("pool"));
+ sc.setJobGroup(Utils.buildJobGroupId(context), Utils.buildJobDesc(context), false);
+
+ try {
+ Method method = sqlc.getClass().getMethod("sql", String.class);
Review comment:
Maybe this as well?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] alexott commented on a change in pull request #3579:
[ZEPPELIN-4522]. Support multiple sql statements for SparkSqlInterpreter
Posted by GitBox <gi...@apache.org>.
alexott commented on a change in pull request #3579: [ZEPPELIN-4522]. Support multiple sql statements for SparkSqlInterpreter
URL: https://github.com/apache/zeppelin/pull/3579#discussion_r361794087
##########
File path: spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkSqlInterpreter.java
##########
@@ -82,26 +85,35 @@ public InterpreterResult internalInterpret(String st, InterpreterContext context
sparkInterpreter.getZeppelinContext().setInterpreterContext(context);
SQLContext sqlc = sparkInterpreter.getSQLContext();
SparkContext sc = sqlc.sparkContext();
- sc.setLocalProperty("spark.scheduler.pool", context.getLocalProperties().get("pool"));
- sc.setJobGroup(Utils.buildJobGroupId(context), Utils.buildJobDesc(context), false);
-
- try {
- Method method = sqlc.getClass().getMethod("sql", String.class);
- int maxResult = Integer.parseInt(context.getLocalProperties().getOrDefault("limit",
- "" + sparkInterpreter.getZeppelinContext().getMaxResult()));
- String msg = sparkInterpreter.getZeppelinContext().showData(
- method.invoke(sqlc, st), maxResult);
- sc.clearJobGroup();
- return new InterpreterResult(Code.SUCCESS, msg);
- } catch (Exception e) {
- if (Boolean.parseBoolean(getProperty("zeppelin.spark.sql.stacktrace"))) {
- return new InterpreterResult(Code.ERROR, ExceptionUtils.getStackTrace(e));
+
+ StringBuilder builder = new StringBuilder();
+ List<String> sqls = sqlSplitter.splitSql(st);
+ for (String sql : sqls) {
+ sc.setLocalProperty("spark.scheduler.pool", context.getLocalProperties().get("pool"));
+ sc.setJobGroup(Utils.buildJobGroupId(context), Utils.buildJobDesc(context), false);
+
+ try {
+ Method method = sqlc.getClass().getMethod("sql", String.class);
+ int maxResult = Integer.parseInt(context.getLocalProperties().getOrDefault("limit",
+ "" + sparkInterpreter.getZeppelinContext().getMaxResult()));
+ String result = sparkInterpreter.getZeppelinContext().showData(
+ method.invoke(sqlc, sql), maxResult);
+ sc.clearJobGroup();
Review comment:
And move this to `finally`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] zjffdu commented on a change in pull request #3579:
[ZEPPELIN-4522]. Support multiple sql statements for SparkSqlInterpreter
Posted by GitBox <gi...@apache.org>.
zjffdu commented on a change in pull request #3579: [ZEPPELIN-4522]. Support multiple sql statements for SparkSqlInterpreter
URL: https://github.com/apache/zeppelin/pull/3579#discussion_r361796422
##########
File path: spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkSqlInterpreter.java
##########
@@ -82,26 +85,35 @@ public InterpreterResult internalInterpret(String st, InterpreterContext context
sparkInterpreter.getZeppelinContext().setInterpreterContext(context);
SQLContext sqlc = sparkInterpreter.getSQLContext();
SparkContext sc = sqlc.sparkContext();
- sc.setLocalProperty("spark.scheduler.pool", context.getLocalProperties().get("pool"));
- sc.setJobGroup(Utils.buildJobGroupId(context), Utils.buildJobDesc(context), false);
-
- try {
- Method method = sqlc.getClass().getMethod("sql", String.class);
- int maxResult = Integer.parseInt(context.getLocalProperties().getOrDefault("limit",
- "" + sparkInterpreter.getZeppelinContext().getMaxResult()));
- String msg = sparkInterpreter.getZeppelinContext().showData(
- method.invoke(sqlc, st), maxResult);
- sc.clearJobGroup();
- return new InterpreterResult(Code.SUCCESS, msg);
- } catch (Exception e) {
- if (Boolean.parseBoolean(getProperty("zeppelin.spark.sql.stacktrace"))) {
- return new InterpreterResult(Code.ERROR, ExceptionUtils.getStackTrace(e));
+
+ StringBuilder builder = new StringBuilder();
+ List<String> sqls = sqlSplitter.splitSql(st);
+ for (String sql : sqls) {
+ sc.setLocalProperty("spark.scheduler.pool", context.getLocalProperties().get("pool"));
+ sc.setJobGroup(Utils.buildJobGroupId(context), Utils.buildJobDesc(context), false);
Review comment:
Make sense
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] alexott commented on a change in pull request #3579:
[ZEPPELIN-4522]. Support multiple sql statements for SparkSqlInterpreter
Posted by GitBox <gi...@apache.org>.
alexott commented on a change in pull request #3579: [ZEPPELIN-4522]. Support multiple sql statements for SparkSqlInterpreter
URL: https://github.com/apache/zeppelin/pull/3579#discussion_r361839978
##########
File path: spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkSqlInterpreter.java
##########
@@ -82,26 +85,37 @@ public InterpreterResult internalInterpret(String st, InterpreterContext context
sparkInterpreter.getZeppelinContext().setInterpreterContext(context);
SQLContext sqlc = sparkInterpreter.getSQLContext();
SparkContext sc = sqlc.sparkContext();
+
+ StringBuilder builder = new StringBuilder();
+ List<String> sqls = sqlSplitter.splitSql(st);
+ int maxResult = Integer.parseInt(context.getLocalProperties().getOrDefault("limit",
+ "" + sparkInterpreter.getZeppelinContext().getMaxResult()));
+
sc.setLocalProperty("spark.scheduler.pool", context.getLocalProperties().get("pool"));
sc.setJobGroup(Utils.buildJobGroupId(context), Utils.buildJobDesc(context), false);
-
+ String curSql = null;
try {
- Method method = sqlc.getClass().getMethod("sql", String.class);
- int maxResult = Integer.parseInt(context.getLocalProperties().getOrDefault("limit",
- "" + sparkInterpreter.getZeppelinContext().getMaxResult()));
- String msg = sparkInterpreter.getZeppelinContext().showData(
- method.invoke(sqlc, st), maxResult);
- sc.clearJobGroup();
- return new InterpreterResult(Code.SUCCESS, msg);
+ for (String sql : sqls) {
Review comment:
what will be in the `builder` if we'll have empty list of SQLs?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] zjffdu commented on a change in pull request #3579:
[ZEPPELIN-4522]. Support multiple sql statements for SparkSqlInterpreter
Posted by GitBox <gi...@apache.org>.
zjffdu commented on a change in pull request #3579: [ZEPPELIN-4522]. Support multiple sql statements for SparkSqlInterpreter
URL: https://github.com/apache/zeppelin/pull/3579#discussion_r361796465
##########
File path: spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkSqlInterpreter.java
##########
@@ -82,26 +85,35 @@ public InterpreterResult internalInterpret(String st, InterpreterContext context
sparkInterpreter.getZeppelinContext().setInterpreterContext(context);
SQLContext sqlc = sparkInterpreter.getSQLContext();
SparkContext sc = sqlc.sparkContext();
- sc.setLocalProperty("spark.scheduler.pool", context.getLocalProperties().get("pool"));
- sc.setJobGroup(Utils.buildJobGroupId(context), Utils.buildJobDesc(context), false);
-
- try {
- Method method = sqlc.getClass().getMethod("sql", String.class);
- int maxResult = Integer.parseInt(context.getLocalProperties().getOrDefault("limit",
- "" + sparkInterpreter.getZeppelinContext().getMaxResult()));
- String msg = sparkInterpreter.getZeppelinContext().showData(
- method.invoke(sqlc, st), maxResult);
- sc.clearJobGroup();
- return new InterpreterResult(Code.SUCCESS, msg);
- } catch (Exception e) {
- if (Boolean.parseBoolean(getProperty("zeppelin.spark.sql.stacktrace"))) {
- return new InterpreterResult(Code.ERROR, ExceptionUtils.getStackTrace(e));
+
+ StringBuilder builder = new StringBuilder();
+ List<String> sqls = sqlSplitter.splitSql(st);
+ for (String sql : sqls) {
+ sc.setLocalProperty("spark.scheduler.pool", context.getLocalProperties().get("pool"));
+ sc.setJobGroup(Utils.buildJobGroupId(context), Utils.buildJobDesc(context), false);
+
+ try {
+ Method method = sqlc.getClass().getMethod("sql", String.class);
Review comment:
Make sense, and I think we didn't reset the spark local properties in other places. Created ZEPPELIN-4523 for it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services