You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by jongyoul <gi...@git.apache.org> on 2016/06/03 18:47:59 UTC
[GitHub] incubator-zeppelin pull request #957: ZEPPELIN-900 Show interpreter logs in ...
GitHub user jongyoul opened a pull request:
https://github.com/apache/incubator-zeppelin/pull/957
ZEPPELIN-900 Show interpreter logs in front side's result area while processing
### What is this PR for?
Showing logs while running paragraph. This is POC level now.
### What type of PR is it?
[Improvement]
### Todos
* [x] - PoC
* [ ] - TBD
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-900
### How should this be tested?
Just build and run long running command
```
%sh
sleep 10 && echo "success"
```
```
sc.version
sc.parallelize(1 to 100000).map(p => p/100 -> p).reduceByKey(_+_).collect().foreach(println)
Thread.sleep(10000)
```
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jongyoul/incubator-zeppelin ZEPPELIN-900
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-zeppelin/pull/957.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #957
----
commit 38f01eaf5cdf608f1aadec1b319459735ae62952
Author: Jongyoul Lee <jo...@gmail.com>
Date: 2016-06-03T17:52:50Z
Changed getProgress method to return log and progress
POC level
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin issue #957: ZEPPELIN-900 Show interpreter logs in front s...
Posted by jongyoul <gi...@git.apache.org>.
Github user jongyoul commented on the issue:
https://github.com/apache/incubator-zeppelin/pull/957
@bzz Thanks for the review. Quick answer for last question. InterpreterResult include new logs only after `getProgress()` called before. See [here](https://github.com/apache/incubator-zeppelin/pull/957/files#diff-f9d22f7302ae356454cdcc637942856fR145). Thus logs actually doesn't grow O(N) if getProgress is called periodically. But I'm also finding a better way to reduce memory usage.
Another problem with this implementation, it cannot covers multi-tenancy at all because that are thread-unsafe and unaware users from interpreter context.
And I also think of finding a better name of Remote*.
@felixcheung I'm very glad of you to review this PR, and give some idea. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin issue #957: ZEPPELIN-900 Show interpreter logs in front s...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/incubator-zeppelin/pull/957
WIP?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin issue #957: ZEPPELIN-900 Show interpreter logs in front s...
Posted by bzz <gi...@git.apache.org>.
Github user bzz commented on the issue:
https://github.com/apache/incubator-zeppelin/pull/957
@felixcheung It looks like it's in PoC stage now.
AFAIK the [gist of this proposal](https://github.com/apache/incubator-zeppelin/pull/957/files#diff-f9d22f7302ae356454cdcc637942856fR140) is to make interpreter's Log4J in-memory writeAppender available to the frontend, which sounds awesome to me.
@jongyoul some quick feedback
- usually adding screenshots helps to get people more excited and provide better feedback and review changes faster
- AFAIK interpreter logs can grow at least O(N) where N is the number of job executions. Will that work, given that interpreter logs (parts?) are going to be transmitted to potentially remote machine?
Alos, could you please help me to understand, will `InterpreterResult` return *full interpreter logs* from every time it is asked for (i.e by `getProgress()`) and send them?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---