You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by merrimanr <gi...@git.apache.org> on 2017/04/18 21:52:15 UTC
[GitHub] incubator-metron pull request #535: METRON-859: Use REST application with Ke...
GitHub user merrimanr opened a pull request:
https://github.com/apache/incubator-metron/pull/535
METRON-859: Use REST application with Kerberos
## Contributor Comments
This PR enables the REST application to function in a Kerberized cluster. Testing instructions are as follows:
1. Follow the instructions in /incubator-metron/metron-deployment/vagrant/Kerberos-setup.md to Kerberize Metron in full dev.
2. Follow the instructions in the "Quick Dev" section of /incubator-metron/metron-interface/metron-rest/README.md to deploy and start the REST application "locally on the Quick Dev host" (full dev host in this case). Make sure you use the start command for a Kerberos enabled cluster (the one with the -Djava.security.auth.login.config parameter).
3. Navigate to http://node1:8082/swagger-ui.html and exercise the various endpoints. The endpoints should function normally and there should be no security-related errors returned.
The REST application uses keytabs to authenticate the various services. A summary of where these come from for each service:
- Kafka and Zookeeper principals/keytabs come from the client_jaas.conf file created during Kerberos setup
- HDFS and Storm principals/keytabs come from the Spring kerberos.principal and kerberos.keytab properties (should there be separate properties for each service?)
Unit tests were added for new Kerberos related code but integration tests still assume a non-Kerberized testing environment. I believe adjusting our integration testing framework to support Kerberos is out of scope for this PR and I'm not sure it's even possible.
This was also tested for regressions against a non-Kerberized cluster (start command is differently obviously).
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron (Incubating).
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to verify your changes?
- [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
bin/generate-md.sh
mvn site:site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommened that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/merrimanr/incubator-metron METRON-859
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/535.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #535
----
commit 17e3b273bda19d5365e177c1bc8d00fbb226e378
Author: merrimanr <me...@gmail.com>
Date: 2017-04-18T15:24:08Z
updated rpm build ansible task
commit 525b384dd9703acd6003ae07c879e1bee5813c9a
Author: merrimanr <me...@gmail.com>
Date: 2017-04-18T17:27:56Z
Added Kerberos support to the REST application
commit 03d4e48f8a59dc05758bfb7d917db4f4c9ec594e
Author: merrimanr <me...@gmail.com>
Date: 2017-04-18T21:16:41Z
Merge remote-tracking branch 'mirror/master' into METRON-859
commit 20144bcee36caa55237ea3d61f266d9673b7cccb
Author: merrimanr <me...@gmail.com>
Date: 2017-04-18T21:35:10Z
Changed principle to principal
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #535: METRON-859: Use REST application with Ke...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr closed the pull request at:
https://github.com/apache/incubator-metron/pull/535
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #535: METRON-859: Use REST application with Ke...
Posted by merrimanr <gi...@git.apache.org>.
GitHub user merrimanr reopened a pull request:
https://github.com/apache/incubator-metron/pull/535
METRON-859: Use REST application with Kerberos
## Contributor Comments
This PR enables the REST application to function in a Kerberized cluster. Testing instructions are as follows:
1. Follow the instructions in /incubator-metron/metron-deployment/vagrant/Kerberos-setup.md to Kerberize Metron in full dev.
2. Follow the instructions in the "Quick Dev" section of /incubator-metron/metron-interface/metron-rest/README.md to deploy and start the REST application "locally on the Quick Dev host" (full dev host in this case). Make sure you use the start command for a Kerberos enabled cluster (the one with the -Djava.security.auth.login.config parameter).
3. Navigate to http://node1:8082/swagger-ui.html and exercise the various endpoints. The endpoints should function normally and there should be no security-related errors returned.
The REST application uses keytabs to authenticate the various services. A summary of where these come from for each service:
- Kafka and Zookeeper principals/keytabs come from the client_jaas.conf file created during Kerberos setup
- HDFS and Storm principals/keytabs come from the Spring kerberos.principal and kerberos.keytab properties (should there be separate properties for each service?)
Unit tests were added for new Kerberos related code but integration tests still assume a non-Kerberized testing environment. I believe adjusting our integration testing framework to support Kerberos is out of scope for this PR and I'm not sure it's even possible.
This was also tested for regressions against a non-Kerberized cluster (start command is differently obviously).
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron (Incubating).
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to verify your changes?
- [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
bin/generate-md.sh
mvn site:site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommened that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/merrimanr/incubator-metron METRON-859
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/535.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #535
----
commit 17e3b273bda19d5365e177c1bc8d00fbb226e378
Author: merrimanr <me...@gmail.com>
Date: 2017-04-18T15:24:08Z
updated rpm build ansible task
commit 525b384dd9703acd6003ae07c879e1bee5813c9a
Author: merrimanr <me...@gmail.com>
Date: 2017-04-18T17:27:56Z
Added Kerberos support to the REST application
commit 03d4e48f8a59dc05758bfb7d917db4f4c9ec594e
Author: merrimanr <me...@gmail.com>
Date: 2017-04-18T21:16:41Z
Merge remote-tracking branch 'mirror/master' into METRON-859
commit 20144bcee36caa55237ea3d61f266d9673b7cccb
Author: merrimanr <me...@gmail.com>
Date: 2017-04-18T21:35:10Z
Changed principle to principal
commit 7973639803d72a074af27db3ece6efafc3c1d8e9
Author: merrimanr <me...@gmail.com>
Date: 2017-04-18T22:02:04Z
Remove hard-coded version
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #535: METRON-859: Use REST application with Ke...
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/535#discussion_r112539709
--- Diff: metron-interface/metron-rest/src/test/resources/README.vm ---
@@ -128,11 +128,22 @@ The metron-rest application will be available at http://localhost:8080/swagger-u
To run the application locally on the Quick Dev host, package the application and scp the archive to node1:
```
mvn clean package
-scp ./target/metron-rest-$METRON_VERSION-archive.tar.gz root@node1:~/
+scp ./target/metron-rest-$METRON_VERSION-archive.tar.gz root@node1:$METRON_HOME
```
-Login to node1 and unarchive the metron-rest application. Start the application on a different port to avoid conflicting with Ambari:
+Login to node1 and unarchive the metron-rest application:
```
-java -jar ./lib/metron-rest-$METRON_VERSION.jar --spring.profiles.active=vagrant,dev --server.port=8082
+ssh root@node1
+cd $METRON_HOME && tar xf ./metron-rest-$METRON_VERSION-archive.tar.gz
+```
+Start the application on a different port to avoid conflicting with Ambari:
+```
+java -jar $METRON_HOME/lib/metron-rest-$METRON_VERSION.jar --spring.profiles.active=vagrant,dev --server.port=8082
+```
+In a cluster with Kerberos enabled the start command must be run as the metron user, include references to the jaas and krb5.confg files, and kerberos support enabled:
+```
+su metron
+cd $METRON_HOME
--- End diff --
This needs to be the metron user's home. Otherwise a permission error occurs (because root owns $METRON_HOME)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #535: METRON-859: Use REST application with Kerberos
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:
https://github.com/apache/incubator-metron/pull/535
My +1 stands, thanks for the contribution!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #535: METRON-859: Use REST application with Kerberos
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/incubator-metron/pull/535
The REST API is documented with annotations. This is standard practice and powers the in context documentation in the Swagger UI. It was requested that this documentation also be included in the README so README.vm (a velocity template) is provided to generate the API README content from the annotations.
The catch is that you have to keep the READMEs in sync outside of the API section. Or, without it, you would have to keep the README in sync with the annotations. In practice I'm not sure which one is more worse, it's annoying either way. My personal preference would be to reference Swagger for REST documentation (and not duplicate it) but I understand why we included it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #535: METRON-859: Use REST application with Kerberos
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/incubator-metron/pull/535
It would take some work but I think it could be done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #535: METRON-859: Use REST application with Ke...
Posted by merrimanr <gi...@git.apache.org>.
GitHub user merrimanr reopened a pull request:
https://github.com/apache/incubator-metron/pull/535
METRON-859: Use REST application with Kerberos
## Contributor Comments
This PR enables the REST application to function in a Kerberized cluster. Testing instructions are as follows:
1. Follow the instructions in /incubator-metron/metron-deployment/vagrant/Kerberos-setup.md to Kerberize Metron in full dev.
2. Follow the instructions in the "Quick Dev" section of /incubator-metron/metron-interface/metron-rest/README.md to deploy and start the REST application "locally on the Quick Dev host" (full dev host in this case). Make sure you use the start command for a Kerberos enabled cluster (the one with the -Djava.security.auth.login.config parameter).
3. Navigate to http://node1:8082/swagger-ui.html and exercise the various endpoints. The endpoints should function normally and there should be no security-related errors returned.
The REST application uses keytabs to authenticate the various services. A summary of where these come from for each service:
- Kafka and Zookeeper principals/keytabs come from the client_jaas.conf file created during Kerberos setup
- HDFS and Storm principals/keytabs come from the Spring kerberos.principal and kerberos.keytab properties (should there be separate properties for each service?)
Unit tests were added for new Kerberos related code but integration tests still assume a non-Kerberized testing environment. I believe adjusting our integration testing framework to support Kerberos is out of scope for this PR and I'm not sure it's even possible.
This was also tested for regressions against a non-Kerberized cluster (start command is differently obviously).
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron (Incubating).
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to verify your changes?
- [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
bin/generate-md.sh
mvn site:site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommened that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/merrimanr/incubator-metron METRON-859
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/535.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #535
----
commit 17e3b273bda19d5365e177c1bc8d00fbb226e378
Author: merrimanr <me...@gmail.com>
Date: 2017-04-18T15:24:08Z
updated rpm build ansible task
commit 525b384dd9703acd6003ae07c879e1bee5813c9a
Author: merrimanr <me...@gmail.com>
Date: 2017-04-18T17:27:56Z
Added Kerberos support to the REST application
commit 03d4e48f8a59dc05758bfb7d917db4f4c9ec594e
Author: merrimanr <me...@gmail.com>
Date: 2017-04-18T21:16:41Z
Merge remote-tracking branch 'mirror/master' into METRON-859
commit 20144bcee36caa55237ea3d61f266d9673b7cccb
Author: merrimanr <me...@gmail.com>
Date: 2017-04-18T21:35:10Z
Changed principle to principal
commit 7973639803d72a074af27db3ece6efafc3c1d8e9
Author: merrimanr <me...@gmail.com>
Date: 2017-04-18T22:02:04Z
Remove hard-coded version
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #535: METRON-859: Use REST application with Kerberos
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/incubator-metron/pull/535
@merrimanr Just curious, what's the difference between a vm file and an md file?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #535: METRON-859: Use REST application with Ke...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr closed the pull request at:
https://github.com/apache/incubator-metron/pull/535
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #535: METRON-859: Use REST application with Kerberos
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/incubator-metron/pull/535
Agreed this isn't Kerberos related. I just wasn't clear what the file was and why it is in test. Any reason the utility can't go in src/main along with the readme? Separate Jira, of course, if it's even reasonable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #535: METRON-859: Use REST application with Kerberos
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on the issue:
https://github.com/apache/incubator-metron/pull/535
@merrimanr , does the README.vm generate an html documentation file in target? If so, send me the install location, and I'll make a patch to the site-book tools to also add it to the site-book nav menu.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #535: METRON-859: Use REST application with Kerberos
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:
https://github.com/apache/incubator-metron/pull/535
Pending the one doc change I noted!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #535: METRON-859: Use REST application with Kerberos
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on the issue:
https://github.com/apache/incubator-metron/pull/535
@merrimanr but is the update of README.md part of the component build? Or can it be made so? That way if a developer makes changes/additions in the annotations, but misses re-committing the README, git will remind them after the build.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #535: METRON-859: Use REST application with Kerberos
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/incubator-metron/pull/535
Docs have been updated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #535: METRON-859: Use REST application with Kerberos
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on the issue:
https://github.com/apache/incubator-metron/pull/535
@merrimanr , okay, thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #535: METRON-859: Use REST application with Kerberos
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/incubator-metron/pull/535
To be honest, I don't really care that much. Where would you like it to be? There is a runner class in test/java/org/apache/metron/rest/utils/ReadMeUtils.java that applies the template so that would have to go with it. Or we could just get rid of it and make people duplicate documentation in annotations and READMEs. Or we could just add a reference in the README to Swagger and be done with it.
It should be a separate Jira either way because it's not related to Kerberos.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #535: METRON-859: Use REST application with Kerberos
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:
https://github.com/apache/incubator-metron/pull/535
I was able to spin this up in full dev without issues. There might be some conflicts with METRON-799, but I would expect that to primarily be on the README.
+1 on what's here right now, and we'll sort of 799 however it needs to be sorted out.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #535: METRON-859: Use REST application with Ke...
Posted by merrimanr <gi...@git.apache.org>.
GitHub user merrimanr reopened a pull request:
https://github.com/apache/incubator-metron/pull/535
METRON-859: Use REST application with Kerberos
## Contributor Comments
This PR enables the REST application to function in a Kerberized cluster. Testing instructions are as follows:
1. Follow the instructions in /incubator-metron/metron-deployment/vagrant/Kerberos-setup.md to Kerberize Metron in full dev.
2. Follow the instructions in the "Quick Dev" section of /incubator-metron/metron-interface/metron-rest/README.md to deploy and start the REST application "locally on the Quick Dev host" (full dev host in this case). Make sure you use the start command for a Kerberos enabled cluster (the one with the -Djava.security.auth.login.config parameter).
3. Navigate to http://node1:8082/swagger-ui.html and exercise the various endpoints. The endpoints should function normally and there should be no security-related errors returned.
The REST application uses keytabs to authenticate the various services. A summary of where these come from for each service:
- Kafka and Zookeeper principals/keytabs come from the client_jaas.conf file created during Kerberos setup
- HDFS and Storm principals/keytabs come from the Spring kerberos.principal and kerberos.keytab properties (should there be separate properties for each service?)
Unit tests were added for new Kerberos related code but integration tests still assume a non-Kerberized testing environment. I believe adjusting our integration testing framework to support Kerberos is out of scope for this PR and I'm not sure it's even possible.
This was also tested for regressions against a non-Kerberized cluster (start command is differently obviously).
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron (Incubating).
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to verify your changes?
- [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
bin/generate-md.sh
mvn site:site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommened that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/merrimanr/incubator-metron METRON-859
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/535.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #535
----
commit 17e3b273bda19d5365e177c1bc8d00fbb226e378
Author: merrimanr <me...@gmail.com>
Date: 2017-04-18T15:24:08Z
updated rpm build ansible task
commit 525b384dd9703acd6003ae07c879e1bee5813c9a
Author: merrimanr <me...@gmail.com>
Date: 2017-04-18T17:27:56Z
Added Kerberos support to the REST application
commit 03d4e48f8a59dc05758bfb7d917db4f4c9ec594e
Author: merrimanr <me...@gmail.com>
Date: 2017-04-18T21:16:41Z
Merge remote-tracking branch 'mirror/master' into METRON-859
commit 20144bcee36caa55237ea3d61f266d9673b7cccb
Author: merrimanr <me...@gmail.com>
Date: 2017-04-18T21:35:10Z
Changed principle to principal
commit 7973639803d72a074af27db3ece6efafc3c1d8e9
Author: merrimanr <me...@gmail.com>
Date: 2017-04-18T22:02:04Z
Remove hard-coded version
commit d8fe16fedc9d87a89d0579e81f89b4bdad2d5e3f
Author: merrimanr <me...@gmail.com>
Date: 2017-04-20T16:32:27Z
Merge remote-tracking branch 'mirror/master' into METRON-859
# Conflicts:
# metron-deployment/vagrant/Kerberos-setup.md
commit 82c2efb174195e4ad3d115504110c3b24072ef99
Author: merrimanr <me...@gmail.com>
Date: 2017-04-20T17:22:36Z
testing without spring kerberos dependency
commit 04487bca41b5cfa356e07209fba03767621e5af1
Author: merrimanr <me...@gmail.com>
Date: 2017-04-20T17:38:36Z
testing for corrupt jars
commit b20f655500c381b5ea247f6d33966d1ad58e8b37
Author: merrimanr <me...@gmail.com>
Date: 2017-04-20T17:50:24Z
removing corrupted jar
commit 1faba5c787be5b33899538abe4e2674f4378d99f
Author: merrimanr <me...@gmail.com>
Date: 2017-04-20T17:55:53Z
reverted travis config
commit 9d7113e4ce1e9f32d78b269640d9ecd8b542e1fb
Author: merrimanr <me...@gmail.com>
Date: 2017-04-21T13:08:18Z
Merge remote-tracking branch 'mirror/master' into METRON-859
# Conflicts:
# metron-deployment/vagrant/Kerberos-setup.md
commit 6e6043fc70215a1a919116271635d89090c067ff
Author: merrimanr <me...@gmail.com>
Date: 2017-04-21T15:23:05Z
Updated REST README to resolve conflicts with METRON-799
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #535: METRON-859: Use REST application with Ke...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/535
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #535: METRON-859: Use REST application with Kerberos
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/incubator-metron/pull/535
I think src/main is fine. @mattf-horton there are no new files generated, this simply updates the metron-rest/README.md file. In theory you don't have to use it and it's not part of any automated workflow.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #535: METRON-859: Use REST application with Kerberos
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/incubator-metron/pull/535
Should that be in the test src directory? `metron-interface/metron-rest/src/test/resources/README.vm`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #535: METRON-859: Use REST application with Ke...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr closed the pull request at:
https://github.com/apache/incubator-metron/pull/535
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #535: METRON-859: Use REST application with Ke...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr closed the pull request at:
https://github.com/apache/incubator-metron/pull/535
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #535: METRON-859: Use REST application with Ke...
Posted by merrimanr <gi...@git.apache.org>.
GitHub user merrimanr reopened a pull request:
https://github.com/apache/incubator-metron/pull/535
METRON-859: Use REST application with Kerberos
## Contributor Comments
This PR enables the REST application to function in a Kerberized cluster. Testing instructions are as follows:
1. Follow the instructions in /incubator-metron/metron-deployment/vagrant/Kerberos-setup.md to Kerberize Metron in full dev.
2. Follow the instructions in the "Quick Dev" section of /incubator-metron/metron-interface/metron-rest/README.md to deploy and start the REST application "locally on the Quick Dev host" (full dev host in this case). Make sure you use the start command for a Kerberos enabled cluster (the one with the -Djava.security.auth.login.config parameter).
3. Navigate to http://node1:8082/swagger-ui.html and exercise the various endpoints. The endpoints should function normally and there should be no security-related errors returned.
The REST application uses keytabs to authenticate the various services. A summary of where these come from for each service:
- Kafka and Zookeeper principals/keytabs come from the client_jaas.conf file created during Kerberos setup
- HDFS and Storm principals/keytabs come from the Spring kerberos.principal and kerberos.keytab properties (should there be separate properties for each service?)
Unit tests were added for new Kerberos related code but integration tests still assume a non-Kerberized testing environment. I believe adjusting our integration testing framework to support Kerberos is out of scope for this PR and I'm not sure it's even possible.
This was also tested for regressions against a non-Kerberized cluster (start command is differently obviously).
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron (Incubating).
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to verify your changes?
- [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
bin/generate-md.sh
mvn site:site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommened that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/merrimanr/incubator-metron METRON-859
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/535.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #535
----
commit 17e3b273bda19d5365e177c1bc8d00fbb226e378
Author: merrimanr <me...@gmail.com>
Date: 2017-04-18T15:24:08Z
updated rpm build ansible task
commit 525b384dd9703acd6003ae07c879e1bee5813c9a
Author: merrimanr <me...@gmail.com>
Date: 2017-04-18T17:27:56Z
Added Kerberos support to the REST application
commit 03d4e48f8a59dc05758bfb7d917db4f4c9ec594e
Author: merrimanr <me...@gmail.com>
Date: 2017-04-18T21:16:41Z
Merge remote-tracking branch 'mirror/master' into METRON-859
commit 20144bcee36caa55237ea3d61f266d9673b7cccb
Author: merrimanr <me...@gmail.com>
Date: 2017-04-18T21:35:10Z
Changed principle to principal
commit 7973639803d72a074af27db3ece6efafc3c1d8e9
Author: merrimanr <me...@gmail.com>
Date: 2017-04-18T22:02:04Z
Remove hard-coded version
commit d8fe16fedc9d87a89d0579e81f89b4bdad2d5e3f
Author: merrimanr <me...@gmail.com>
Date: 2017-04-20T16:32:27Z
Merge remote-tracking branch 'mirror/master' into METRON-859
# Conflicts:
# metron-deployment/vagrant/Kerberos-setup.md
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---