You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by astroshim <gi...@git.apache.org> on 2015/12/21 03:44:18 UTC
[GitHub] incubator-zeppelin pull request: Release resource after cron job.
GitHub user astroshim opened a pull request:
https://github.com/apache/incubator-zeppelin/pull/557
Release resource after cron job.
### What is this PR for?
Release resource after cron schedule job.
### What type of PR is it?
Improvement
### Todos
* [ ] - add check-box for release resource to the zeppelin-web.
* [ ] - add release resource(interpreter restart) function to notebook.
### Is there a relevant Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-524.
### How should this be tested?
please refer to the screenshots.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/astroshim/incubator-zeppelin ZEPPELIN-524
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-zeppelin/pull/557.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #557
----
commit ef510c7afda8639f593c2c1bea58086fd7e88a2a
Author: astroshim <hs...@nflabs.com>
Date: 2015-12-20T19:01:10Z
release resource after cron job.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: Auto-restart interpreters on cron...
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/557#issuecomment-167169459
I think interpreter restart should abort pending / running jobs.
If it raise `Scheduler already terminated` need to be fixed. @HeartSaVioR do you mind creating an issue for it? I think can take care of it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: Release resource after cron job.
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/557#issuecomment-166184258
Thanks @astroshim. This is really useful feature.
Do you mind adding an unittest?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: Auto-restart interpreters on cron...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-zeppelin/pull/557
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: Auto-restart interpreters on cron...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/557#issuecomment-167163100
Actually I was trying to restart the interpreter first for each job run via REST API, but gave up because it throws ```Scheduler already terminated``` when restarting interpreter and job run is occurred nearly same time.
```
ERROR [2015-12-22 18:05:00,001] ({DefaultQuartzScheduler_Worker-7} QuartzScheduler.java[schedulerError]:2425) - Job (note.2B83NUA15 threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: java.lang.RuntimeException: Scheduler already terminated]
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: java.lang.RuntimeException: Scheduler already terminated
at org.apache.zeppelin.scheduler.RemoteScheduler.submit(RemoteScheduler.java:140)
at org.apache.zeppelin.notebook.Note.runAll(Note.java:326)
at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:391)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
```
After this, I couldn't handle interpreter via UI. Restart doesn't work so I should kill interpreter via ```kill -9```.
Furthermore, if my understanding is right, interpreter restart aborts current jobs which are running or pending.
So it would be better to restart interpreter manually.
If we really want to restart interpreter automatically, we should care about below scenarios.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: Release resource after cron job.
Posted by bzz <gi...@git.apache.org>.
Github user bzz commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/557#issuecomment-166214952
Wow, I did not realize that by "release resources" you mean full interpreter restart!
This will kill i.e context, associated with the interpreter and all the cached data in cluster.
May be we could call it something like "auto-restart interpreter on cron execution", then? Both in PR title and in Zeppelin UI.
Or explore some ways to actually release resources without full interpreter restart (which seems much harder).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: Auto-restart interpreters on cron...
Posted by astroshim <gi...@git.apache.org>.
Github user astroshim commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/557#issuecomment-167461320
@Leemoonsoo Thank you for making clear.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: Auto-restart interpreters on cron...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/557#issuecomment-167172536
@Leemoonsoo
It occurred while I was experimenting several things together so I'm not 100% sure that we can't handle interpreter via UI.
I'll try to reproduce and report it when it is reproducible.
Btw, if we think it's valid to abort pending / running jobs, it may be better to add aborted jobs to pending queue automatically so that new interpreter process can run these, since we may couldn't recognize some jobs are aborted because of automatic interpreter restart.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: Auto-restart interpreters on cron...
Posted by astroshim <gi...@git.apache.org>.
Github user astroshim commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/557#issuecomment-167243619
Hello. @felixcheung
You're right.. The interpreter process is shared now but I think interpreter process will be separated as user option.
what do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: Auto-restart interpreters on cron...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/557#issuecomment-167152568
Would it be possible the interpreter restart happens in the middle of another running notebook if more one are scheduled to around the same time?
Interpreter are shared right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: Auto-restart interpreters on cron...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/557#issuecomment-167280810
@Leemoonsoo
I took a look at previous logs, and its behavior was somewhat strange.
I'll file an issue regarding strange behavior with logs. (Would posting to mailing list be more appreciated?)
Btw, I can't reproduce issue so my case shouldn't block this PR. But I agree with @felixcheung and another approach @astroshim stated is better than current approach.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: Release resource after cron job.
Posted by astroshim <gi...@git.apache.org>.
Github user astroshim commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/557#issuecomment-166837719
Dear @bzz
I totally agree with you. I'll update the text and testcase.
Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: Auto-restart interpreters on cron...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/557#issuecomment-167275152
@astroshim right, I think separated interpreter process for cron job or similar would be a good approach for this issues. I know that there are places with a dozen of users sharing Zeppelin & interpreter and restarting interpreter seem dangerous.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: Release resource after cron job.
Posted by bzz <gi...@git.apache.org>.
Github user bzz commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/557#issuecomment-166569734
@astroshim I see, you are right, it's hard to say what else can do the job.
I think if we just update the PR and the text in Zeppelin UI to something like "Auto-restart interpreters on cron execution" + add some tests, if possible - that would be great.
What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: Auto-restart interpreters on cron...
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/557#issuecomment-167616036
Tested and LGTM.
Merge if there're no more discussions
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: Release resource after cron job.
Posted by astroshim <gi...@git.apache.org>.
Github user astroshim commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/557#issuecomment-166184648
Thank @Leemoonsoo your feedback.
I'm going to implement the unittest.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: Release resource after cron job.
Posted by astroshim <gi...@git.apache.org>.
Github user astroshim commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/557#issuecomment-166234115
@bzz Thanks your feedback.
Yes, "release resource" means restarting all binded interpreters on the note.
I was thinking restarting the interpreters is the obvious way to release resources.
If you have any better ways to release resource without the interpreters restart, please let me know.
Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---