You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by del680202 <gi...@git.apache.org> on 2017/01/30 04:51:45 UTC
[GitHub] zeppelin pull request #1960: [ZEPPELIN-2030] Select statement doesn't work i...
GitHub user del680202 opened a pull request:
https://github.com/apache/zeppelin/pull/1960
[ZEPPELIN-2030] Select statement doesn't work in LivySparkSQLInterpreter
### What is this PR for?
I try to execute sql query by ``%livy.sql``, I got a error as below
```
ERROR [2017-01-30 11:15:57,060] ({pool-2-thread-6} LivySparkSQLInterpreter.java[interpret]:143) - Exception in LivySparkSQLInterpreter while interpret
java.lang.StringIndexOutOfBoundsException: String index out of range: 28
at java.lang.String.substring(String.java:1963)
at org.apache.zeppelin.livy.LivySparkSQLInterpreter.parseSQLOutput(LivySparkSQLInterpreter.java:177)
at org.apache.zeppelin.livy.LivySparkSQLInterpreter.interpret(LivySparkSQLInterpreter.java:128)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:489)
at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
```
I seem ``parseSQLOutput`` method try to filter out ``only showing top 1 row`` but it doesn't.
Therefore, I fix this issue by modifying condition
### What type of PR is it?
Bug Fix
### Todos
* [x] - none
### What is the Jira issue?
[ZEPPELIN-2030](https://issues.apache.org/jira/browse/ZEPPELIN-2030)
### How should this be tested?
try to execute ``%livy.sql SELECT * FROM bank`` on tutorial
### Screenshots (if appropriate)
Before
After
### Questions:
* Does the licenses files need update?
* Is there breaking changes for older versions?
* Does this needs documentation?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/del680202/zeppelin bug-2030
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/1960.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1960
----
commit 8f499c49ecb4bca1077674de12f44c9b67e4e66e
Author: Chin Tzulin <jp...@w022341412910m.local>
Date: 2017-01-30T04:40:21Z
[ZEPPELIN-2030] Select statement doesn't work in LivySparkSQLInterpreter
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1960: [ZEPPELIN-2030] Select statement doesn't work in LivyS...
Posted by del680202 <gi...@git.apache.org>.
Github user del680202 commented on the issue:
https://github.com/apache/zeppelin/pull/1960
I test the same query by ``%jdbc``, I got the same result with ``%spark.sql``
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1960: [ZEPPELIN-2030] Select statement doesn't work in LivyS...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1960
@del680202 Can you paste the full stacktrace ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1960: [ZEPPELIN-2030] Select statement doesn't work in LivyS...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1960
lgtm
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1960: [ZEPPELIN-2030] Select statement doesn't work in LivyS...
Posted by minahlee <gi...@git.apache.org>.
Github user minahlee commented on the issue:
https://github.com/apache/zeppelin/pull/1960
Thank you for quick response! Let me merge this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1960: [ZEPPELIN-2030] Select statement doesn't work in LivyS...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1960
@del680202 Can you try this on `%spark.sql` as well ? I guess `Output exceeds...` may also cause issues in displaying table format data in `%spark.sql`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #1960: [ZEPPELIN-2030] Select statement doesn't work i...
Posted by del680202 <gi...@git.apache.org>.
GitHub user del680202 reopened a pull request:
https://github.com/apache/zeppelin/pull/1960
[ZEPPELIN-2030] Select statement doesn't work in LivySparkSQLInterpreter
### What is this PR for?
I try to execute sql query by ``%livy.sql``, I got a error as below
```
ERROR [2017-01-30 11:15:57,060] ({pool-2-thread-6} LivySparkSQLInterpreter.java[interpret]:143) - Exception in LivySparkSQLInterpreter while interpret
java.lang.StringIndexOutOfBoundsException: String index out of range: 28
at java.lang.String.substring(String.java:1963)
at org.apache.zeppelin.livy.LivySparkSQLInterpreter.parseSQLOutput(LivySparkSQLInterpreter.java:177)
at org.apache.zeppelin.livy.LivySparkSQLInterpreter.interpret(LivySparkSQLInterpreter.java:128)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:489)
at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
```
I seem ``parseSQLOutput`` method try to filter out ``only showing top 1 row`` but it doesn't.
Therefore, I fix this issue by modifying condition
### What type of PR is it?
Bug Fix
### Todos
* [x] - none
### What is the Jira issue?
[ZEPPELIN-2030](https://issues.apache.org/jira/browse/ZEPPELIN-2030)
### How should this be tested?
try to execute ``%livy.sql SELECT * FROM bank`` on tutorial
### Screenshots (if appropriate)
Before
After
### Questions:
* Does the licenses files need update?
* Is there breaking changes for older versions?
* Does this needs documentation?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/del680202/zeppelin bug-2030
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/1960.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1960
----
commit 8f499c49ecb4bca1077674de12f44c9b67e4e66e
Author: Chin Tzulin <jp...@w022341412910m.local>
Date: 2017-01-30T04:40:21Z
[ZEPPELIN-2030] Select statement doesn't work in LivySparkSQLInterpreter
commit d6811006efe4d36f9163a2d9d3a703d241d948c5
Author: Chin Tzulin <jp...@w022341412910m.local>
Date: 2017-01-31T06:26:20Z
Add test case into LivySQLInterpreterTest.java
commit 86ebc82ea9b5b70d7d9eaefc538d38aeaeae5029
Author: Chin Tzulin <jp...@w022341412910m.local>
Date: 2017-01-31T08:34:38Z
Fix test problem
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1960: [ZEPPELIN-2030] Select statement doesn't work in LivyS...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1960
Thanks @del680202 Could you add unit test in `LivySQLInterpreterTest.java`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1960: [ZEPPELIN-2030] Select statement doesn't work in LivyS...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1960
hmm, right. `LivySparkSQLInterpreter` delegate interpret to `LivySparkInterpreter`. So it is possible to get `Output exceeds...` in `LivySparkSQLInterpreter`. Thanks @del680202
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1960: [ZEPPELIN-2030] Select statement doesn't work in LivyS...
Posted by del680202 <gi...@git.apache.org>.
Github user del680202 commented on the issue:
https://github.com/apache/zeppelin/pull/1960
Oh.. When I test ``%livy.sql`` by my hive table, ``TEXT`` result ``Output exceeds 102400. Truncated.`` causes a failure.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1960: [ZEPPELIN-2030] Select statement doesn't work in LivyS...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1960
@del680202 Thanks for checking.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1960: [ZEPPELIN-2030] Select statement doesn't work in LivyS...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1960
I am not sure whether that would help, because `Output exceeds ...` is generated by zeppelin, not by livy or spark. https://github.com/apache/zeppelin/blob/master/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/InterpreterOutput.java#L189
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #1960: [ZEPPELIN-2030] Select statement doesn't work i...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/zeppelin/pull/1960
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #1960: [ZEPPELIN-2030] Select statement doesn't work i...
Posted by del680202 <gi...@git.apache.org>.
Github user del680202 closed the pull request at:
https://github.com/apache/zeppelin/pull/1960
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1960: [ZEPPELIN-2030] Select statement doesn't work in LivyS...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1960
I suspect it is caused by other issues rather than `Output exceeds...`. Because message `Output exceeds...` should be generated after the method `parseSQLOutput`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1960: [ZEPPELIN-2030] Select statement doesn't work in LivyS...
Posted by del680202 <gi...@git.apache.org>.
Github user del680202 commented on the issue:
https://github.com/apache/zeppelin/pull/1960
OK, I will add test cases for this issue
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1960: [ZEPPELIN-2030] Select statement doesn't work in LivyS...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/zeppelin/pull/1960
@zjffdu
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1960: [ZEPPELIN-2030] Select statement doesn't work in LivyS...
Posted by del680202 <gi...@git.apache.org>.
Github user del680202 commented on the issue:
https://github.com/apache/zeppelin/pull/1960
Actually, I inject ``LOGGER.info(line)`` into ``parseSQLOutput`` for tracing input.
Then I got string at least line in result
``INFO [2017-01-31 16:24:19,584] ({pool-2-thread-5} LivySparkSQLInterpreter.java[parseSQLOutput]:175) - Output exceeds 102400. Truncated.``
Anyway, I try to replace condition by matching row format ``line.matches("^\\|.*\\|$")``.
It works well with me now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1960: [ZEPPELIN-2030] Select statement doesn't work in LivyS...
Posted by minahlee <gi...@git.apache.org>.
Github user minahlee commented on the issue:
https://github.com/apache/zeppelin/pull/1960
@zjffdu @del680202 Is this ready? :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1960: [ZEPPELIN-2030] Select statement doesn't work in LivyS...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/zeppelin/pull/1960
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1960: [ZEPPELIN-2030] Select statement doesn't work in LivyS...
Posted by del680202 <gi...@git.apache.org>.
Github user del680202 commented on the issue:
https://github.com/apache/zeppelin/pull/1960
@zjffdu
I had try to query sample table by ``%spark.sql``.
Alert message is not red but it works.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1960: [ZEPPELIN-2030] Select statement doesn't work in LivyS...
Posted by del680202 <gi...@git.apache.org>.
Github user del680202 commented on the issue:
https://github.com/apache/zeppelin/pull/1960
@zjffdu
I got error message as below
```
ERROR [2017-01-31 16:34:34,431] ({pool-2-thread-7} LivySparkSQLInterpreter.java[interpret]:143) - Exception in LivySparkSQLInterpreter while interpret
java.lang.StringIndexOutOfBoundsException: String index out of range: 34
at java.lang.String.substring(String.java:1963)
at org.apache.zeppelin.livy.LivySparkSQLInterpreter.parseSQLOutput(LivySparkSQLInterpreter.java:177)
at org.apache.zeppelin.livy.LivySparkSQLInterpreter.interpret(LivySparkSQLInterpreter.java:128)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:495)
at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
```
Maybe I can replace condition by matching row format ``line.matches("^\\|.*\\|$")`` instead of skipping magic string such as ``only showing..`` or ``Output exceeds...``?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1960: [ZEPPELIN-2030] Select statement doesn't work in LivyS...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/zeppelin/pull/1960
this is odd, we should have test that catches this condition - could you check if we could add to livy interpreter tests for this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1960: [ZEPPELIN-2030] Select statement doesn't work in LivyS...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1960
Yes, the PR is ready. CI failure is not relevant. Please help include it in 0.7
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---