You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by merrimanr <gi...@git.apache.org> on 2018/11/19 22:25:10 UTC
[GitHub] metron pull request #1275: METRON-1878: Add Metron as a Knox service
GitHub user merrimanr opened a pull request:
https://github.com/apache/metron/pull/1275
METRON-1878: Add Metron as a Knox service
## Contributor Comments
This PR adds REST and the Alerts UI as services in Knox. Currently this is incomplete and meant as a starting point to give reviewers an idea of the changes involved. The outstanding items are detailed below.
### Changed Included
- Fixed minor path issues in various UI related code. Files include:
- app.component.html - path should be relative
- authentication.service.ts - logout should match other endpoints
- alert-search.directive.ts - path should be relative
- index.html - a relative base href means Knox does not have to rewrite links
- Created Metron Knox topology and service definition files for REST and Alerts UI. The service definition files are relatively simple and should look similar to other services.
- Created a script to deploy Metron files to Knox
- Added a Spring "knox" profile that configures Swagger with the right endpoint paths
- Added service definition files to rpm spec
### Testing
This feature can be tested in full dev with some manual configuration. After spinning up full dev:
1. Add Knox as a Service in Ambari (Admin > Stacks and Versions > Add Service next to Knox)
2. Follow the instructions [here](https://github.com/apache/metron/tree/master/metron-deployment/development#knox-demo-ldap) to enable the Knox LDAP in full dev. You should be able to authentication with REST using guest/guest-password after this is done.
3. Install the Metron topology and service definition files by running the script at `/usr/metron/0.6.1/bin/install_metron_knox.sh`. This script copies the metron.xml topology file to `/usr/hdp/current/knox-server/conf/topologies` and the service definition files to `/usr/hdp/current/knox-server/data/services/metron-rest/0.6.1` and `/usr/hdp/current/knox-server/data/services/metron-alerts/0.6.1`
4. Update the REST API path for the Alerts UI. Currently expressjs proxies REST requests from the Alerts UI but this will now be done by Knox instead. The `apiRoot` setting in `app-config.json` is used to construct the url for REST requests. Change this setting in `/usr/metron/0.6.1/web/alerts-ui/assets/app-config.json` from:
```
{
"apiRoot": "/api/v1"
}
```
to:
```
{
"apiRoot": "/gateway/metron/metron-rest/api/v1"
}
```
5. Update the REST API path for Swagger. The Swagger interface is served by the REST application and provides links for the various endpoints. Swagger must be configured with the new Knox root path for REST. Navigate to `Ambari > Metron > Configs > REST`. Change the `Active Spring profiles` setting from `dev` to `dev,knox` and change the `Metron Spring options` to `--knox.root=/gateway/metron/metron-rest`. This configures Swagger to prepend the endpoint paths with the `knox.root` Spring property.
6. Restart REST with Ambari and you should now be able to access both Swagger and the Alerts UI through Knox:
- Swagger - https://node1:8443/gateway/metron/metron-rest/swagger-ui.html
- Alerts UI - https://node1:8443/gateway/metron/metron-alerts/alerts-list
Everything should function normally including login and logout.
### Outstanding Items
#### Installing Metron in Knox
Installing a service in Knox involves 2 steps:
1. Deploy the service definition files mentioned earlier by copying them to Knox directories on the machine where Knox is installed
2. Either update an existing Knox topology file or create a new topology file. This file configures a collection of services. In our case this includes authentication and urls to the services exposed.
What is the best way to install these files in Knox? The approach in this PR only works if Metron is installed on the same machine as the Knox gateway.
I also opted to create a dedicated Metron topology file. I think this will give us more control and allow us to expose properties in a more user-friendly way. Knox has a default topology that you can configure in Ambari but it's exposed as is: a big xml file. Does anyone think we should use the default topology file instead?
#### Adding Knox to the stack
This process requires that we add Knox separately through Ambari. Do we want to make Knox a dependency for Metron similar to Kafka, Storm, etc?
Does it make sense to require that Knox and Metron are colocated? Doing this simplifies installation and configuration and I would vote we make this a requirement unless there is a good reason not to. Does anyone think there are cases where users would need to install them on different machines? If we do that, how do we install the service definition files?
#### Ambari automation
Currently setting up Metron with Knox is a manual process. I believe all of this can easily be automated with Ambari. How far do we want to go in this initial pass? Do we want to just document it for now? If we did want to automate this setup with Ambari, we would need to:
- Put the app-config.json under Ambari control and expose the `apiRoot` setting
- Add the `knox.root` setting to the REST config file (`rest_application.yml`) and expose it through an Ambari setting
- Add an Ambari setting similar to the `LDAP enabled` setting that automatically adds the `knox` spring profile to the list of active profiles
- Possibly put the metron.xml topology file under Ambari control
- Add a call to `install_metron_knox.sh` in the appropriate Metron Mpack script
#### Quick Links
At this point I do not know how to make Ambari quick links dynamic. You can bind the port to a property but I don't know if you can do that for the scheme, host, and path. This means that after you follow these instructions for setting up Metron with Knox, the quick links will still point to the urls without Knox. Does anyone know how to change quick links to point to Knox automatically? There has been discussion of enabling Knox by default so in that case we can just set the quick links to the Knox urls.
Hopefully a working example will provide more context as we plan out this feature.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
```
mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to verify your changes?
- [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/merrimanr/incubator-metron knox-mpack
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1275.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1275
----
commit c0345e2faba1f10f495a4c3c3c5ec8b5137e2d1c
Author: merrimanr <me...@...>
Date: 2018-11-16T20:58:46Z
initial commit
commit 6cf7303af1d5cb06fea0d70a52f553c82878de3f
Author: merrimanr <me...@...>
Date: 2018-11-19T21:05:24Z
missing license headers and rpm spec updates
----
---
[GitHub] metron issue #1275: METRON-1878: Add Metron as a Knox service
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/1275
I will be looking into this further, but I see a lot of the following in the management UI logs. This repeated for more than 80k lines in a matter of minutes:
```
18/12/06 20:13:23 INFO zookeeper.ClientCnxn: Opening socket connection to server node1/192.168.66.121:2181. Will not attempt to authenticate using SASL (unknown error)
18/12/06 20:13:23 WARN zookeeper.ClientCnxn: Session 0x1678019c6bb0099 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
18/12/06 20:13:23 ERROR controller.RestExceptionHandler: Encountered error: Unable to get column metadata
org.apache.metron.rest.RestException: Unable to get column metadata
at org.apache.metron.rest.service.impl.SearchServiceImpl.search(SearchServiceImpl.java:95)
at org.apache.metron.rest.controller.SearchController.search(SearchController.java:54)
at sun.reflect.GeneratedMethodAccessor239.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:877)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:783)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:877)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:851)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:215)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.metron.indexing.dao.search.InvalidSearchException: Unable to get column metadata
at org.apache.metron.elasticsearch.dao.ElasticsearchSearchDao.buildSearchRequest(ElasticsearchSearchDao.java:155)
at org.apache.metron.elasticsearch.dao.ElasticsearchSearchDao.search(ElasticsearchSearchDao.java:127)
at org.apache.metron.elasticsearch.dao.ElasticsearchDao.search(ElasticsearchDao.java:196)
at org.apache.metron.elasticsearch.dao.ElasticsearchMetaAlertSearchDao.search(ElasticsearchMetaAlertSearchDao.java:81)
at org.apache.metron.elasticsearch.dao.ElasticsearchMetaAlertDao.search(ElasticsearchMetaAlertDao.java:209)
at org.apache.metron.rest.service.impl.SearchServiceImpl.search(SearchServiceImpl.java:92)
... 87 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:171)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:145)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:192)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
... 1 more
18/12/06 20:13:24 INFO zookeeper.ClientCnxn: Opening socket connection to server node1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
18/12/06 20:13:24 WARN zookeeper.ClientCnxn: Session 0x1678019c6bb0099 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
18/12/06 20:13:24 INFO zookeeper.ClientCnxn: Opening socket connection to server node1/192.168.66.121:2181. Will not attempt to authenticate using SASL (unknown error)
18/12/06 20:13:24 WARN zookeeper.ClientCnxn: Session 0x1678019c6bb009b for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
18/12/06 20:13:24 INFO zookeeper.ClientCnxn: Opening socket connection to server node1/192.168.66.121:2181. Will not attempt to authenticate using SASL (unknown error)
18/12/06 20:13:24 WARN zookeeper.ClientCnxn: Session 0x1678019c6bb009a for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
18/12/06 20:13:24 INFO zookeeper.ClientCnxn: Opening socket connection to server node1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
18/12/06 20:13:24 WARN zookeeper.ClientCnxn: Session 0x1678019c6bb009a for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
```
---
[GitHub] metron issue #1275: METRON-1878: Add Metron as a Knox service
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/1275
@merrimanr Where does this leave the management UI if I enable Knox?
---
[GitHub] metron issue #1275: METRON-1878: Add Metron as a Knox service
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/1275
Re: quick links; you might be out of luck, based on https://issues.apache.org/jira/browse/AMBARI-21325. There might be ways around it (or require manual updating), but beyond that I don't know.
---
[GitHub] metron issue #1275: METRON-1878: Add Metron as a Knox service
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/1275
EDIT - that's the rest logs, not management UI logs. When I shutdown the management UI, the exceptions stop.
---
[GitHub] metron issue #1275: METRON-1878: Add Metron as a Knox service
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/1275
Thoughts on the outstanding items
**Installing Metron in Knox**
- re: "The approach in this PR only works if Metron is installed on the same machine as the Knox gateway." I know of Hadoop installs using multiple Knox instances with a load balancer in front of them. I would expect this to be a reasonable expectation for people with sufficiently large Metron installs (or many tenants or whatever). I expect that to influence the decision here.
- re: topology file. Can you give a bit more detail on the pros/cons? Is there any standard that other implementors are using or recommendations in the docs?
**Adding Knox to the stack**
- As noted above, multiple Knox instances with load balancer is a potential configuration users will want. In that case, colocation is not a given.
- I don't think Knox as a required service is a given. I wish Ambari let you have optional services and gate things more directly. We might be able to mimic that functionality by hiding the relevant configs unless Knox is enabled. It might still require the user to hit a toggle, but it might be possible to check against a Knox service config directly for that. I've never tried it (and it might be nice to do in general for some of our other configs, e.g. Solr and ES).
** Ambari Automation **
- I'm inclined to consider Ambari necessary for effort as a whole to be really done. I'd prefer it to be done now just to have it done and coherent, but I'm fine with it being a follow-on PR (given the usual assumptions that everything continues to work as expected and is documented, otherwise I'd be inclined to use feature branch). It means docs are throwaway once that goes in, but I'm personally fine with it being manually documented then automated later.
** Quick Links**
- As mentioned above, I think we're out of luck here.
---
[GitHub] metron issue #1275: METRON-1878: Add Metron as a Knox service
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/1275
I'm not sure if we can have Ambari do that in a general way. Right now we do have the Metron client get distributed throughout for the purpose of setting up Storm security. We could also do the same thing for whatever we need to distribute for Knox. It means a Metron Client needs to coexist with any Knox servers, but that seems like a reasonable restriction (even if I'd rather not have to deal with it). I think that would also address the custom topology question.
If we're okay with putting more load on the Metron Client, I'd rather use the custom topology and just do it that way.
For quicklinks, I'm okay with having the manual update process documented as part of the Knox setup, as long as it's noted its not a Metron issue, it's an Ambari/Knox interaction issue (and maybe just link to the Jira)
---
[GitHub] metron issue #1275: METRON-1878: Add Metron as a Knox service
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/metron/pull/1275
Thanks for the feedback @justinleet.
- Colocating: You bring up a good point about multiple Knox instances. The primary challenge here is that enabling Metron on Knox requires copying files to local directories on each Knox Gateway. If Knox exposed this through a REST API or could read files from HDFS this would not be an issue but I'm not aware that either of those options are supported. This is not an issue with other services (HDFS, HBase, etc) because those services have been added to the Knox project and are installed along with Knox. We could go that route but there are some major drawbacks (see the "Knox SSO feature branch review and features" discussion thread). It's not clear to me how we solve this in an automated way. Is this (copying Metron files to Knox Gateway hosts) possible with Ambari?
- Custom topology vs Knox default: There are several advantages to maintaining our own topology file:
- We have complete control over how we expose settings in Ambari. The Knox default topology is exposed in Ambari as raw xml.
- We have control over the url value where the Knox default url is `/gateway/default/<service name>`. There may be a way to change this but I don't know how.
- The topology file will be smaller and easier to understand because it only contains Metron services.
The main disadvantage I see is related to the issue described above: we need a way to copy this
file to Knox Gateway hosts and update it.
- Ambari automation: I do believe everything still works as before in this PR. I think some decisions we make here will influence that work so a follow on seems appropriate to me.
- Quick Links: I agree, looks like we're out of luck here. I think we'll either need to make Knox the default and make the Metron quick links match or live with quick links being updated manually.
---