You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by zjffdu <gi...@git.apache.org> on 2018/06/28 07:14:39 UTC
[GitHub] zeppelin pull request #3044: ZEPPELIN-3563. Add pool to paragraph property t...
GitHub user zjffdu opened a pull request:
https://github.com/apache/zeppelin/pull/3044
ZEPPELIN-3563. Add pool to paragraph property that use spark interpreter
### What is this PR for?
Allow user to specify the pool when running spark sql in concurrent approach.
### What type of PR is it?
[Feature]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3563
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-3563
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/3044.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3044
----
commit 541c34ce754be59452a902872ee154a3c04940c9
Author: Jeff Zhang <zj...@...>
Date: 2018-06-27T07:52:12Z
ZEPPELIN-3563. Add pool to paragraph property that use spark interpreter
----
---
[GitHub] zeppelin pull request #3044: ZEPPELIN-3563. Add pool to paragraph property t...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/zeppelin/pull/3044
---
[GitHub] zeppelin issue #3044: ZEPPELIN-3563. Add pool to paragraph property that use...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/3044
We could do that but it depends on the `farischeduler.xml` that user specified. If user didn't specify the user name as pool name in `fairscheduler.xml`, then it doesn't make sense to do that in zeppelin side.
---
[GitHub] zeppelin issue #3044: ZEPPELIN-3563. Add pool to paragraph property that use...
Posted by Tagar <gi...@git.apache.org>.
Github user Tagar commented on the issue:
https://github.com/apache/zeppelin/pull/3044
I meant Zeppelin would set `spark.scheduler.pool` to username of the user that was used to authenticate into Zeppelin.
---
[GitHub] zeppelin issue #3044: ZEPPELIN-3563. Add pool to paragraph property that use...
Posted by Tagar <gi...@git.apache.org>.
Github user Tagar commented on the issue:
https://github.com/apache/zeppelin/pull/3044
Would it be possible to set `spark.scheduler.pool` to authenticated user name as well,
as described in https://issues.apache.org/jira/browse/ZEPPELIN-3334
I think it's a much better default than setting it manually in each paragraph.
(still a great feature of having ability to set pool at a paragraph level too)
---
[GitHub] zeppelin issue #3044: ZEPPELIN-3563. Add pool to paragraph property that use...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/3044
@Tagar One approach I can think of is to support variable substitution.
e.g.
```
%spark(pool=${name})
...
```
---
[GitHub] zeppelin issue #3044: ZEPPELIN-3563. Add pool to paragraph property that use...
Posted by Tagar <gi...@git.apache.org>.
Github user Tagar commented on the issue:
https://github.com/apache/zeppelin/pull/3044
> We could do that but it depends on the farischeduler.xml that user specified. If user didn't specify the user name as pool name in fairscheduler.xml, then it doesn't make sense to do that in zeppelin side.
Yep, I understand that. They have to specify a pool name in `farischeduler.xml` either way - if it's given manually through paragraph property as proposed in this PR, or if it's taken from authenticated username as proposed in https://issues.apache.org/jira/browse/ZEPPELIN-3334 .
Thanks.
---
[GitHub] zeppelin issue #3044: ZEPPELIN-3563. Add pool to paragraph property that use...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/3044
@Tagar What do you mean authenticated user name ? Do you mean impersonation in one spark app ? I don't believe it is possible in spark.
---
[GitHub] zeppelin issue #3044: ZEPPELIN-3563. Add pool to paragraph property that use...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/3044
Will merge it if no more comments.
---
[GitHub] zeppelin issue #3044: ZEPPELIN-3563. Add pool to paragraph property that use...
Posted by Tagar <gi...@git.apache.org>.
Github user Tagar commented on the issue:
https://github.com/apache/zeppelin/pull/3044
Thanks @zjffdu - that looks good to me. What about {username} instead of {name} ?
---