You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by GitBox <gi...@apache.org> on 2018/12/19 20:00:43 UTC
[GitHub] merrimanr opened a new pull request #1275: METRON-1878: Add Metron
as a Knox service
merrimanr opened a new pull request #1275: METRON-1878: Add Metron as a Knox service
URL: https://github.com/apache/metron/pull/1275
## Contributor Comments
This PR adds REST and the Alerts UI as services in Knox.
### Changed Included
- Fixed minor path issues in various UI related code. Files include:
- app.component.html - path should be relative
- authentication.service.ts - logout should match other endpoints
- alert-search.directive.ts - path should be relative
- index.html - a relative base href means Knox does not have to rewrite links
- Created Metron Knox topologies and service definition files for REST and Alerts UI. The service definition files are relatively simple and should look similar to other services.
- Created a script to deploy Metron files to Knox
- Added a Spring "knox" profile that configures Swagger with the right endpoint paths and enables the Knox SSO ServletFilter.
- Added service definition files to rpm spec
- Added "apiRoot" and "loginPath" properties to the Alerts UI to format REST request urls and redirect to the right login page after logout.
- Added documentation and unit tests
### Testing
This feature can be tested in full dev with some manual configuration. After spinning up full dev:
1. Add Knox as a Service in Ambari (Admin > Stacks and Versions > Add Service next to Knox)
2. Follow the instructions [here](https://github.com/apache/metron/tree/master/metron-deployment/development#knox-demo-ldap) to enable the Knox LDAP in full dev. You should be able to authentication with REST using guest/guest-password after this is done.
3. Install the Metron topology and service definition files by running the script at `/usr/metron/0.7.0/bin/install_metron_knox.sh`. This script copies the `metron.xml` and `metronsso.xml` topology files to `/usr/hdp/current/knox-server/conf/topologies` and the service definition files to `/usr/hdp/current/knox-server/data/services/metron-rest/0.6.1` and `/usr/hdp/current/knox-server/data/services/metron-alerts/0.7.0`
4. Update the REST API path for the Alerts UI. Currently expressjs proxies REST requests from the Alerts UI but this will now be done by Knox instead. The `apiRoot` setting in `app-config.json` is used to construct the url for REST requests. Change this setting in `/usr/metron/0.7.0/web/alerts-ui/assets/app-config.json` from:
```
{
"apiRoot": "/api/v1"
}
```
to:
```
{
"apiRoot": "/gateway/metron/metron-rest/api/v1"
}
```
5. Update the login path for the Alerts UI. The Alerts UI normally has a dedicated login page but this is replaced with a Knox login page when Knox is enabled. Knox will intercept the request, redirect to the login page, then redirect back to the original requested url. Set this property to the home page of the Alerts UI. Change the `loginPath` setting in `/usr/metron/0.7.0/web/alerts-ui/assets/app-config.json` from:
```
{
"loginPath": "/login"
}
```
to:
```
{
"loginPath": "/gateway/metron/metron-alerts/"
}
```
Any changes to `app-config.json` require a page refresh.
6. Add the `knox` Spring active profile in Ambari. Navigate to `Ambari > Metron > Configs > REST`.
Change the `Active Spring profiles` setting from `dev` to `dev,knox`. This sets the Swagger context path to Knox and adds the Knox SSO servlet filter.
7. Update the Spring options in Ambari. Get the Knox public key with the following command on full dev:
```
openssl s_client -connect node1:8443 < /dev/null | openssl x509 | grep -v 'CERTIFICATE' | paste -sd "" -
```
Copy the value generated from that command. Navigate to `Ambari > Metron > Configs > REST` and change the `Metron Spring options` to:
```
--knox.root=/gateway/metron/metron-rest --knox.sso.pubkey=<key generated from openssl command>
```
The `knox.root` setting sets the Knox root path. The Swagger interface is served by the REST application and provides links for the various endpoints. Swagger must be configured with the new Knox root path for REST. The `knox.sso.pubkey` setting sets the Knox public key that will be used to verify Knox SSO tokens.
8. Restart REST with Ambari and you should now be able to access both Swagger and the Alerts UI through Knox with SSO enabled. The REST and Alerts UI urls are now:
- Swagger - https://node1:8443/gateway/metron/metron-rest/swagger-ui.html
- Alerts UI - https://node1:8443/gateway/metron/metron-alerts/alerts-list
Everything should function normally including login, logout and session timeout. To test the SSO part:
1. Request the global config with this url in a browser:
```
https://node1:8443/gateway/metron/metron-rest/api/v1/global/config
```
You should be redirected to the Knox SSO url with the `originalUrl` param properly set:
```
https://node1:8443/gateway/knoxsso/knoxauth/login.html?originalUrl=https://node1:8443/gateway/metron/metron-rest/api/v1/global/config
```
2. After authenticating with LDAP credentials, you should be successfully redirected and see the global config in the browser.
3. Navigate to the Alerts UI:
```
https://node1:8443/gateway/metron/metron-alerts/alerts-list
```
You should go directly to the Alerts UI without being prompted for credentials.
4. To verify roles are working correctly, authenticate as the `admin` user with `admin/admin-password`. The `https://node1:8443/gateway/metron/metron-rest/api/v1/user/whoami/roles` endpoint should return `ROLE_USER` and `ROLE_ADMIN` just as it currently does.
5. Click the logout link in the Alerts UI and verify you are redirected to the Knox login page. The same thing should happen if your session times out. The session timeout can be changed in `metronsso.xml` with the `` property. To change from the default of 30 seconds to 5 minutes, change the property from this:
```
<param>
<name>knoxsso.token.ttl</name>
<value>30000</value>
</param>
```
to this:
```
<param>
<name>knoxsso.token.ttl</name>
<value>300000</value>
</param>
```
### Outstanding Items
#### Installing Metron in Knox
Installing a service in Knox involves 2 steps:
1. Deploy the service definition files mentioned earlier by copying them to Knox directories on the machine where Knox is installed
2. Either update an existing Knox topology file or create a new topology file. This file configures a collection of services. In our case this includes authentication and urls to the services exposed.
What is the best way to install these files in Knox? The approach in this PR only works if Metron is installed on the same machine as the Knox gateway.
I also opted to create a dedicated Metron topology file. I think this will give us more control and allow us to expose properties in a more user-friendly way. Knox has a default topology that you can configure in Ambari but it's exposed as is: a big xml file. Does anyone think we should use the default topology file instead?
#### Adding Knox to the stack
This process requires that we add Knox separately through Ambari. Do we want to make Knox a dependency for Metron similar to Kafka, Storm, etc?
Does it make sense to require that Knox and Metron are colocated? Doing this simplifies installation and configuration and I would vote we make this a requirement unless there is a good reason not to. Does anyone think there are cases where users would need to install them on different machines? If we do that, how do we install the service definition files?
#### Ambari automation
Currently setting up Metron with Knox is a manual process. I believe all of this can easily be automated with Ambari. How far do we want to go in this initial pass? Do we want to just document it for now? If we did want to automate this setup with Ambari, we would need to:
- Put the app-config.json under Ambari control and expose the `apiRoot` setting
- Add the `knox.root` setting to the REST config file (`rest_application.yml`) and expose it through an Ambari setting
- Add an Ambari setting similar to the `LDAP enabled` setting that automatically adds the `knox` spring profile to the list of active profiles
- Possibly put the metron.xml topology file under Ambari control
- Add a call to `install_metron_knox.sh` in the appropriate Metron Mpack script
#### Quick Links
At this point I do not know how to make Ambari quick links dynamic. You can bind the port to a property but I don't know if you can do that for the scheme, host, and path. This means that after you follow these instructions for setting up Metron with Knox, the quick links will still point to the urls without Knox. Does anyone know how to change quick links to point to Knox automatically? There has been discussion of enabling Knox by default so in that case we can just set the quick links to the Knox urls.
Hopefully a working example will provide more context as we plan out this feature.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
```
mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to verify your changes?
- [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services