You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@metron.apache.org by nickwallen <gi...@git.apache.org> on 2017/04/11 21:31:07 UTC

[GitHub] incubator-metron pull request #524: METRON-836 Use Pycapa with Kerberos

GitHub user nickwallen opened a pull request:

    https://github.com/apache/incubator-metron/pull/524

    METRON-836 Use Pycapa with Kerberos

    This PR makes the necessary changes for Pycapa to work in a Kerberized environment.  
    * The previous Kafka client library used by Pycapa did not support Kerberos.  Switched to another that supports Kerberos and is based on librdkafka, which is the same library used by Fastcapa and the Bro Plugin.
    * Added docs on how to install and setup Pycapa for Kerberos.
    * Added lots of additional docs around usage.
    
    I tested this change on plain-vanilla Quick Dev, a kerberized-Quick Dev and also a bare-metal cluster that is using Kerberos.
    
    ## Pull Request Checklist
    
    - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). 
    - [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
    - [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
    - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
    - [x] Have you included steps or a guide to how the change may be verified and tested manually?
    - [x] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via:
    - [x] Have you written or updated unit tests and or integration tests to verify your changes?
    - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? 
    - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
    - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
    
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/nickwallen/incubator-metron METRON-836

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-metron/pull/524.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #524
    
----
commit f481ed3787c79a897ae3eca1bbde4451abfceec3
Author: Nick Allen <ni...@nickallen.org>
Date:   2017-04-10T15:10:02Z

    METRON-836 Use Pycapa with Kerberos

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #524: METRON-836 Use Pycapa with Kerberos

Posted by nickwallen <gi...@git.apache.org>.

Github user nickwallen commented on the issue:

    https://github.com/apache/incubator-metron/pull/524
  
    Yes, good feedback.  I had updated the README to mention Py 2.7, but I now just realized that I never pushed that out.  Will get that pushed out and updated.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #524: METRON-836 Use Pycapa with Kerberos

Posted by nickwallen <gi...@git.apache.org>.

Github user nickwallen commented on the issue:

    https://github.com/apache/incubator-metron/pull/524
  
    I updated the validation instructions to setup a Kerberized environment and then validate Pycapa against that environment.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #524: METRON-836 Use Pycapa with Kerberos

Posted by merrimanr <gi...@git.apache.org>.

Github user merrimanr commented on the issue:

https://github.com/apache/incubator-metron/pull/524

I was able to get this to work by performing one extra step. The PR description did not include the Kafka ACL command to permission the topic, only the "pycapa" group. After I did this, I was able to verify data with the pycapa --consumer command. The topic ACL command is included in the Pycapa README so I think simply adding the ACL command for the group is all that's needed.

I did notice that running the pycapa --producer command causes a Kafka service ticket to be cached. Since we're passing in the metron principal and keytab I would expect that ticket to be in the cache instead. While it's unexpected it did not cause any errors.

One last minor documentation issue: this PR description includes steps to install Python 2.7 but nowhere is that mentioned in the README. I know it's only needed because full dev is centos 6 but it might be helpful to at least call out Python 2.7 as a prereq. Maybe it's assumed the user knows that but it tripped me up when I was initially trying to spin this up and that wasn't included in the PR description.

The only issue above I feel is necessary to address would adding the extra Kafka ACL command for the group to the README. Pending that change, +1.

---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #524: METRON-836 Use Pycapa with Kerberos

Posted by JonZeolla <gi...@git.apache.org>.

Github user JonZeolla commented on the issue:

    https://github.com/apache/incubator-metron/pull/524
  
    Sorry, I started reviewing this but it dropped off when @merrimanr mentioned on the dev list that he was also reviewing it.  If there's no progress in the next day or two, I may be able to take another look.  I did follow your initial instructions and it looked good to me, I just haven't followed your latest instructions.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #524: METRON-836 Use Pycapa with Kerberos

Posted by nickwallen <gi...@git.apache.org>.

Github user nickwallen commented on the issue:

    https://github.com/apache/incubator-metron/pull/524
  
    Based on feedback, I have added a series of specific steps to validate this change.  That should help simplify validation.  I should have included that on the first pass.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #524: METRON-836 Use Pycapa with Kerberos

Posted by nickwallen <gi...@git.apache.org>.

Github user nickwallen commented on the issue:

    https://github.com/apache/incubator-metron/pull/524
  
    Calling all reviewers.  Would love to get this closed out and off my plate.  Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---