You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by ruseel <gi...@git.apache.org> on 2017/09/01 09:53:15 UTC
[GitHub] zeppelin pull request #2558: SparkInterpreter with
GitHub user ruseel opened a pull request:
https://github.com/apache/zeppelin/pull/2558
SparkInterpreter with
### What is this PR for?
executing these two pagragraph in spark interperter
----
sc.setLocalProperty("a", "1")
----
sc.getLocalProerty("a")
----
evaluated to unexpected result "null". fix that confusion.
---
sc.setLocalProperty(...) should be more deterministic.
and honor design of SparkContext. SparkContext.setLocalProperty(...) is using ThreadLocal. So let me say that design of SparkContext has assumtion single-thread user.
But before this commit, SparkInterpreter's ExecutorService is created with Executor.newSchedulerService(100). So User might perceive situation like sc.setLocalProperty(...) is not working.
fix by using singleThread for Schduler.
### What type of PR is it?
Bug Fix
### Todos
### What is the Jira issue?
### How should this be tested?
make note with two paragraph
----
sc.setLocalProperty("a", "1")
----
sc.getLocalProerty("a")
----
run paragraph separatly
### Screenshots (if appropriate)
### Questions:
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ruseel/zeppelin spark-sc-in-one-thread
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/2558.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2558
----
commit 1564afd5ff5316f31fe9dda9a788dd5216fc22b6
Author: stephen <st...@vcnc.co.kr>
Date: 2017-09-01T08:42:46Z
use self-created thread for schduler loop
Scheduler of zeppelin has run-forever loop. Before this commit
schduler is running on executor created from ExecutorFactory.
So there is a coupling that "ExecutorFactory should create thread-pool
grater than 1". Seems to be a bad practice.
commit 980a1754fa5d2632c204686619d6e40874a961dd
Author: stephen <st...@vcnc.co.kr>
Date: 2017-09-01T08:51:05Z
sc.setLocalProperty(...) should be more deterministic
and honor design of SparkContext. SparkContext.setLocalProperty(...)
is using ThreadLocal. So let me say that design of SparkContext has
assumtion single-thread user.
But before this commit, SparkInterpreter's ExecutorService is created
with Executor.newSchedulerService(100). So User might perceive situation
like sc.setLocalProperty(...) is not working.
fix by using singleThread for Schduler.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2558: sc.setLocalProperty(...) should be more deterministic
Posted by 1ambda <gi...@git.apache.org>.
Github user 1ambda commented on the issue:
https://github.com/apache/zeppelin/pull/2558
Hi, @ruseel! Thanks for the contribution.
Could u setup travis CI your forked zeppelin repositiory?
Please setup by swich on 'zeppelin' repository at https://travis-ci.org/profile and travis-ci.
And then make sure 'Build branch updates' option is enabled in the settings https://travis-ci.org/ruseel/zeppelin/settings.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2558: sc.setLocalProperty(...) should be more deterministic
Posted by swkimme <gi...@git.apache.org>.
Github user swkimme commented on the issue:
https://github.com/apache/zeppelin/pull/2558
@Leemoonsoo Let me introduce you a new guy on Between, @ruseel (Stephen)
We found a bug while looking into Spark's scheduler, would u review this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #2558: sc.setLocalProperty(...) should be more determi...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/zeppelin/pull/2558
---
[GitHub] zeppelin issue #2558: sc.setLocalProperty(...) should be more deterministic
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/zeppelin/pull/2558
hi - where are we on this?
---
[GitHub] zeppelin issue #2558: sc.setLocalProperty(...) should be more deterministic
Posted by 1ambda <gi...@git.apache.org>.
Github user 1ambda commented on the issue:
https://github.com/apache/zeppelin/pull/2558
@ruseel Thanks for the update. Could you resolve zjffdu's last comment?
---
[GitHub] zeppelin issue #2558: sc.setLocalProperty(...) should be more deterministic
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2558
What about `PySparkInterpreter` and `SparkRInterpreter` ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2558: sc.setLocalProperty(...) should be more deterministic
Posted by ruseel <gi...@git.apache.org>.
Github user ruseel commented on the issue:
https://github.com/apache/zeppelin/pull/2558
@1ambda setup for forked repository `ruseel/zeppelin` is completed.
---