You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by anandsubbu <gi...@git.apache.org> on 2018/01/05 06:24:26 UTC
[GitHub] metron pull request #888: METRON-1389: Zeppelin notebook import does not wor...
GitHub user anandsubbu opened a pull request:
https://github.com/apache/metron/pull/888
METRON-1389: Zeppelin notebook import does not work with Ambari 2.6
## Contributor Comments
In recent versions of Ambari, [Shiro Authentication](https://zeppelin.apache.org/docs/0.6.0/security/shiroauthentication.html) is enabled on the Zeppelin server, which requires a user/password authentication in order to interact with Zeppelin REST API.
This pull request adds support for using the Shiro auth credentials and thus being able to import Zeppelin notebooks. Backward compatibility with non-authenticated Zeppelin has been maintained as well so that the full dev platform (which runs on an older Ambari version) is not broken.
**Steps to test the PR**
* In Ambari UI, go to Services -> Metron -> 'Service Actions' dropdown -> 'Zeppelin Notebook Import'
* Validate that the Metron notebooks are imported by launching the Zeppelin UI (Service -> Zeppelin Notebook -> 'Quick Links' dropdown -> Zeppelin UI)
**Testing Done**
* Ran the changes against Ambari ver 2.6.0.0 (which uses Shiro authentication) and verified that notebooks are imported properly
* Ran the changes against Ambari ver 2.4.2.0 (which does not uses Shiro authentication) and verified that notebooks are imported properly
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [ ] Have you written or updated unit tests and or integration tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/anandsubbu/incubator-metron METRON-1389
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/888.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #888
----
commit e767022e6c87922f7a0c3ca143b7482548a0908c
Author: Anand Subramanian <as...@...>
Date: 2018-01-05T06:10:04Z
Add Shiro authentication support
----
---
[GitHub] metron pull request #888: METRON-1389: Zeppelin notebook import does not wor...
Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/888#discussion_r161955419
--- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py ---
@@ -162,9 +163,21 @@ def zeppelin_notebook_import(self, env):
--- End diff --
In Ambari, the Shiro INI content appears as a template like here:
<img width="1274" alt="zeppelin-config-shiro-ini" src="https://user-images.githubusercontent.com/20395490/35026106-0a038a28-fb6f-11e7-93e3-371e5fa353d0.png">
And this can be fetched using Ambari API as follows:
http://<ambari-node>:8080/api/v1/clusters/cl1/configurations?type=zeppelin-shiro-ini&tag=version1
Which I am fetching in the `status_params.py` as
```
zeppelin_shiro_ini_content = config['configurations']['zeppelin-shiro-ini']['shiro_ini_content']
```
I found it easy to parse this from the content. Could you please clarify how I can use `dict` to get this info?
---
[GitHub] metron pull request #888: METRON-1389: Zeppelin notebook import does not wor...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/888#discussion_r161746734
--- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py ---
@@ -162,9 +163,21 @@ def zeppelin_notebook_import(self, env):
--- End diff --
Do we have to reg ext this out? Can't we just pull it out of the dict?
---
[GitHub] metron issue #888: METRON-1389: Zeppelin notebook import does not work with ...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/888
+1 by inspection
---
[GitHub] metron pull request #888: METRON-1389: Zeppelin notebook import does not wor...
Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/888#discussion_r161735380
--- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py ---
@@ -150,12 +151,36 @@ def zeppelin_notebook_import(self, env):
env.set_params(params)
Logger.info(ambari_format('Searching for Zeppelin Notebooks in {metron_config_zeppelin_path}'))
+
+ # With Ambari 2.5+, Zeppelin server is enabled to work with Shiro authentication, which requires user/password
+ # for authentication (see https://zeppelin.apache.org/docs/0.6.0/security/shiroauthentication.html for details).
+ ses = requests.session()
+
+ # Check if authentication is enabled on the Zeppelin server
+ try:
+ conn = ses.get(ambari_format('http://{zeppelin_server_url}/api/login'))
+
+ # Establish connection if authentication is enabled
+ try:
+ # The following credentials are created at install time by Ambari at /etc/zeppelin/conf/shiro.ini
+ # when Shiro auth is enabled on the Zeppelin server
+ zeppelin_payload = {'userName': 'admin', 'password' : 'admin'}
--- End diff --
Hi @cestella and @ottobackwards - I found a way to get the Shiro INI content from Ambari and parse the admin credentials. Please have a look at my latest commit.
---
[GitHub] metron issue #888: METRON-1389: Zeppelin notebook import does not work with ...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/888
+1 by inspection. Nice work @anandsubbu .
We'll need to figure out this intermittent test failure (impacting all PRs, not just yours) before we merge.
---
[GitHub] metron pull request #888: METRON-1389: Zeppelin notebook import does not wor...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/888#discussion_r161240092
--- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py ---
@@ -150,12 +151,36 @@ def zeppelin_notebook_import(self, env):
env.set_params(params)
Logger.info(ambari_format('Searching for Zeppelin Notebooks in {metron_config_zeppelin_path}'))
+
+ # With Ambari 2.5+, Zeppelin server is enabled to work with Shiro authentication, which requires user/password
+ # for authentication (see https://zeppelin.apache.org/docs/0.6.0/security/shiroauthentication.html for details).
+ ses = requests.session()
+
+ # Check if authentication is enabled on the Zeppelin server
+ try:
+ conn = ses.get(ambari_format('http://{zeppelin_server_url}/api/login'))
+
+ # Establish connection if authentication is enabled
+ try:
+ # The following credentials are created at install time by Ambari at /etc/zeppelin/conf/shiro.ini
+ # when Shiro auth is enabled on the Zeppelin server
+ zeppelin_payload = {'userName': 'admin', 'password' : 'admin'}
--- End diff --
Why wouldn't we expose this through the ambari configuration?
---
[GitHub] metron pull request #888: METRON-1389: Zeppelin notebook import does not wor...
Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu closed the pull request at:
https://github.com/apache/metron/pull/888
---
[GitHub] metron pull request #888: METRON-1389: Zeppelin notebook import does not wor...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/metron/pull/888
---
[GitHub] metron pull request #888: METRON-1389: Zeppelin notebook import does not wor...
Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/888#discussion_r162261463
--- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py ---
@@ -162,9 +163,21 @@ def zeppelin_notebook_import(self, env):
--- End diff --
Cool, thanks.
---
[GitHub] metron pull request #888: METRON-1389: Zeppelin notebook import does not wor...
Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/888#discussion_r163022818
--- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py ---
@@ -150,12 +152,48 @@ def zeppelin_notebook_import(self, env):
env.set_params(params)
Logger.info(ambari_format('Searching for Zeppelin Notebooks in {metron_config_zeppelin_path}'))
+
+ # With Ambari 2.5+, Zeppelin server is enabled to work with Shiro authentication, which requires user/password
+ # for authentication (see https://zeppelin.apache.org/docs/0.6.0/security/shiroauthentication.html for details).
+ ses = requests.session()
--- End diff --
Sure thing @nickwallen . Please see latest commit where I have refactored this into a separate method.
Further, I tested the latest changes on a full-dev environment (w/Ambari 2.4.2) as well as a 12-node cluster (w/Ambari 2.6.0) to confirm the notebooks are imported fine.
---
[GitHub] metron pull request #888: METRON-1389: Zeppelin notebook import does not wor...
Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/888#discussion_r160591890
--- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py ---
@@ -150,12 +151,36 @@ def zeppelin_notebook_import(self, env):
env.set_params(params)
Logger.info(ambari_format('Searching for Zeppelin Notebooks in {metron_config_zeppelin_path}'))
+
+ # With Ambari 2.5+, Zeppelin server is enabled to work with Shiro authentication, which requires user/password
+ # for authentication (see https://zeppelin.apache.org/docs/0.6.0/security/shiroauthentication.html for details).
+ ses = requests.session()
+
+ # Check if authentication is enabled on the Zeppelin server
+ try:
+ conn = ses.get(ambari_format('http://{zeppelin_server_url}/api/login'))
+
+ # Establish connection if authentication is enabled
+ try:
+ # The following credentials are created at install time by Ambari at /etc/zeppelin/conf/shiro.ini
+ # when Shiro auth is enabled on the Zeppelin server
+ zeppelin_payload = {'userName': 'admin', 'password' : 'admin'}
--- End diff --
Hi @cestella , thanks for taking a look. The Zeppelin auth credentials are not related to ambari credentials. They are as defined in `/etc/zeppelin/conf/shiro.ini` on the Zeppelin host, which is set up by default during stack deploy time.
Here's a snippet:
```
[root@hcpca-1 ~]# cat /etc/zeppelin/conf/shiro.ini
[users]
# List of users with their password allowed to access Zeppelin.
# To use a different strategy (LDAP / Database / ...) check the shiro doc at http://shiro.apache.org/configuration.html#Configuration-INISections
admin = admin, admin
<snip>
```
When I created the pull request, I considered the option of reading `shiro.ini` file and parsing for the credentials. I did not pursue that further since I wanted to address the immediate need of ability to import notebook on an out-of-the-box Metron cluster. I understand that a thorough way would be to handle the [different INI settings](http://shiro.apache.org/configuration.html#Configuration-INISections) provided by shiro, and could potentially be the scope of another pull request.
Please let me know if it makes sense.
---
[GitHub] metron pull request #888: METRON-1389: Zeppelin notebook import does not wor...
Posted by anandsubbu <gi...@git.apache.org>.
GitHub user anandsubbu reopened a pull request:
https://github.com/apache/metron/pull/888
METRON-1389: Zeppelin notebook import does not work with Ambari 2.6
## Contributor Comments
In recent versions of Ambari, [Shiro Authentication](https://zeppelin.apache.org/docs/0.6.0/security/shiroauthentication.html) is enabled on the Zeppelin server, which requires a user/password authentication in order to interact with Zeppelin REST API.
This pull request adds support for using the Shiro auth credentials and thus being able to import Zeppelin notebooks. Backward compatibility with non-authenticated Zeppelin has been maintained as well so that the full dev platform (which runs on an older Ambari version) is not broken.
**Steps to test the PR**
* In Ambari UI, go to Services -> Metron -> 'Service Actions' dropdown -> 'Zeppelin Notebook Import'
* Validate that the Metron notebooks are imported by launching the Zeppelin UI (Service -> Zeppelin Notebook -> 'Quick Links' dropdown -> Zeppelin UI)
**Testing Done**
* Ran the changes against Ambari ver 2.6.0.0 (which uses Shiro authentication) and verified that notebooks are imported properly
* Ran the changes against Ambari ver 2.4.2.0 (which does not uses Shiro authentication) and verified that notebooks are imported properly
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [ ] Have you written or updated unit tests and or integration tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/anandsubbu/incubator-metron METRON-1389
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/888.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #888
----
commit e767022e6c87922f7a0c3ca143b7482548a0908c
Author: Anand Subramanian <as...@...>
Date: 2018-01-05T06:10:04Z
Add Shiro authentication support
commit b8780df0a0f1231a18597302302a3bd247b7dfd6
Author: Anand Subramanian <as...@...>
Date: 2018-01-16T11:52:02Z
Get admin user creds from Zeppelin shiro config in Ambari
commit 3a799da66ad1c9568f5fcb85cdc8ea8de8c9872a
Author: Anand Subramanian <as...@...>
Date: 2018-01-22T09:16:19Z
Feedback from Nick - refactor auth check into a separate method
----
---
[GitHub] metron pull request #888: METRON-1389: Zeppelin notebook import does not wor...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/888#discussion_r162669706
--- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py ---
@@ -150,12 +152,48 @@ def zeppelin_notebook_import(self, env):
env.set_params(params)
Logger.info(ambari_format('Searching for Zeppelin Notebooks in {metron_config_zeppelin_path}'))
+
+ # With Ambari 2.5+, Zeppelin server is enabled to work with Shiro authentication, which requires user/password
+ # for authentication (see https://zeppelin.apache.org/docs/0.6.0/security/shiroauthentication.html for details).
+ ses = requests.session()
--- End diff --
Would it make sense to factor out the authentication logic into a separate function named something like `zeppelin_authenticate`? That is a fairly complicated bit in and out of itself.
---
[GitHub] metron pull request #888: METRON-1389: Zeppelin notebook import does not wor...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/888#discussion_r162029895
--- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py ---
@@ -162,9 +163,21 @@ def zeppelin_notebook_import(self, env):
--- End diff --
I'm sorry, I did not see that you where getting an ini, that it was a 'blob' in ambari. You are entirely correct.
---
[GitHub] metron issue #888: METRON-1389: Zeppelin notebook import does not work with ...
Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on the issue:
https://github.com/apache/metron/pull/888
@ottobackwards @cestella and @nickwallen - any other feedback on this PR?
The travis failures seems to be in an unrelated piece of code.
---
[GitHub] metron pull request #888: METRON-1389: Zeppelin notebook import does not wor...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/888#discussion_r160228422
--- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py ---
@@ -150,12 +151,36 @@ def zeppelin_notebook_import(self, env):
env.set_params(params)
Logger.info(ambari_format('Searching for Zeppelin Notebooks in {metron_config_zeppelin_path}'))
+
+ # With Ambari 2.5+, Zeppelin server is enabled to work with Shiro authentication, which requires user/password
+ # for authentication (see https://zeppelin.apache.org/docs/0.6.0/security/shiroauthentication.html for details).
+ ses = requests.session()
+
+ # Check if authentication is enabled on the Zeppelin server
+ try:
+ conn = ses.get(ambari_format('http://{zeppelin_server_url}/api/login'))
+
+ # Establish connection if authentication is enabled
+ try:
+ # The following credentials are created at install time by Ambari at /etc/zeppelin/conf/shiro.ini
+ # when Shiro auth is enabled on the Zeppelin server
+ zeppelin_payload = {'userName': 'admin', 'password' : 'admin'}
--- End diff --
Is this always going to be admin/admin or would it be the user/pass for the ambari instance?
---
[GitHub] metron pull request #888: METRON-1389: Zeppelin notebook import does not wor...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/888#discussion_r161239480
--- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py ---
@@ -150,12 +151,36 @@ def zeppelin_notebook_import(self, env):
env.set_params(params)
Logger.info(ambari_format('Searching for Zeppelin Notebooks in {metron_config_zeppelin_path}'))
+
+ # With Ambari 2.5+, Zeppelin server is enabled to work with Shiro authentication, which requires user/password
+ # for authentication (see https://zeppelin.apache.org/docs/0.6.0/security/shiroauthentication.html for details).
+ ses = requests.session()
+
+ # Check if authentication is enabled on the Zeppelin server
+ try:
+ conn = ses.get(ambari_format('http://{zeppelin_server_url}/api/login'))
+
+ # Establish connection if authentication is enabled
+ try:
+ # The following credentials are created at install time by Ambari at /etc/zeppelin/conf/shiro.ini
+ # when Shiro auth is enabled on the Zeppelin server
+ zeppelin_payload = {'userName': 'admin', 'password' : 'admin'}
--- End diff --
so, I wonder if we should add a configuration parameter for zeppelin user/pass. If people change the default user/pass of zeppelin, this won't work, I think. The other option is whether or not there is an existing config in ambari where this is held (maybe not though). Beyond that, we could always parse the shiro.ini (even as simple as a little python snippet like:
```
with open('/etc/zeppelin/conf/shiro.ini', 'r') as f:
seen_users = False
for line in f:
if '[users]' in line:
seen_users = True
if seen_users and 'admin = ' in line:
tokens = line.strip().split(',')
username = tokens[0].split(' ')[-1].strip()
password = tokens[1].strip()
print "username: {}, password: {}".format(username, password)
break
```
I am just a bit concerned at having this go in as it is given this assumption, but I could probably be talked into it by another committer.
---