You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by MohanDV <gi...@git.apache.org> on 2018/07/31 09:55:45 UTC
[GitHub] metron pull request #1136: METRON-1701: Update steps to run the pycapa on ke...
GitHub user MohanDV opened a pull request:
https://github.com/apache/metron/pull/1136
METRON-1701: Update steps to run the pycapa on kerberized cluster.
## Contributor Comments
Updated the steps to run Pycapa with Kerberos
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [X] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [X] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [X] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
```
mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh
```
- [ ] Have you written or updated unit tests and or integration tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [X] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/MohanDV/metron METRON-1701
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1136.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1136
----
commit 8f44e2732159bcf615f5883a9c48e7776954ac98
Author: Mohan Venkateshaiah <mo...@...>
Date: 2018-07-31T09:50:31Z
Update steps to run the pycapa on kerberized cluster.
----
---
[GitHub] metron pull request #1136: METRON-1701: Update steps to run the pycapa on ke...
Posted by MohanDV <gi...@git.apache.org>.
Github user MohanDV commented on a diff in the pull request:
https://github.com/apache/metron/pull/1136#discussion_r206863799
--- Diff: metron-sensors/pycapa/README.md ---
@@ -239,25 +239,47 @@ The probe can be used in a Kerberized environment. Follow these additional step
* The keytab used is located at `/etc/security/keytabs/metron.headless.keytab`
* The service principal is `metron@EXAMPLE.COM`
+ If it is not, ensure that you have `libsasl` or `libsasl2` installed. On CentOS, this can be installed with the following command.
+ ```
+ yum install -y cyrus-sasl cyrus-sasl-devel cyrus-sasl-gssapi
+ ```
+
+ 1. Use the non-binary wheel to install confluent-kafka-python.
+
+ The Python client README (https://github.com/confluentinc/confluent-kafka-python) has an important note for Kerberos case:
+ ```
+ **NOTE:** The pre-built Linux wheels do NOT contain SASL Kerberos support.
+ If you need SASL Kerberos support you must install librdkafka and
+ its dependencies using the above repositories and then build
+ confluent-kafka from source.
+ ```
--- End diff --
done
---
[GitHub] metron pull request #1136: METRON-1701: Update steps to run the pycapa on ke...
Posted by MohanDV <gi...@git.apache.org>.
Github user MohanDV commented on a diff in the pull request:
https://github.com/apache/metron/pull/1136#discussion_r206863111
--- Diff: metron-sensors/pycapa/README.md ---
@@ -239,25 +239,47 @@ The probe can be used in a Kerberized environment. Follow these additional step
* The keytab used is located at `/etc/security/keytabs/metron.headless.keytab`
* The service principal is `metron@EXAMPLE.COM`
+ If it is not, ensure that you have `libsasl` or `libsasl2` installed. On CentOS, this can be installed with the following command.
+ ```
+ yum install -y cyrus-sasl cyrus-sasl-devel cyrus-sasl-gssapi
+ ```
+
+ 1. Use the non-binary wheel to install confluent-kafka-python.
--- End diff --
Moved these instruction to the top as description .
---
[GitHub] metron issue #1136: METRON-1701: Update steps to run the pycapa on kerberize...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/1136
+1 This looks great. Thanks for making these updates. Very useful!
---
[GitHub] metron pull request #1136: METRON-1701: Update steps to run the pycapa on ke...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1136#discussion_r206503184
--- Diff: metron-sensors/pycapa/README.md ---
@@ -239,25 +239,47 @@ The probe can be used in a Kerberized environment. Follow these additional step
* The keytab used is located at `/etc/security/keytabs/metron.headless.keytab`
* The service principal is `metron@EXAMPLE.COM`
+ If it is not, ensure that you have `libsasl` or `libsasl2` installed. On CentOS, this can be installed with the following command.
+ ```
+ yum install -y cyrus-sasl cyrus-sasl-devel cyrus-sasl-gssapi
+ ```
+
+ 1. Use the non-binary wheel to install confluent-kafka-python.
+
+ The Python client README (https://github.com/confluentinc/confluent-kafka-python) has an important note for Kerberos case:
+ ```
+ **NOTE:** The pre-built Linux wheels do NOT contain SASL Kerberos support.
+ If you need SASL Kerberos support you must install librdkafka and
+ its dependencies using the above repositories and then build
+ confluent-kafka from source.
+ ```
+
+ so you will need to use the non-binary wheel to install confluent-kafka-python and build/install librdkafka separately
+
1. Build Librdkafka with SASL support (` --enable-sasl`) and install at your chosen $PREFIX.
```
- wget https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz -O - | tar -xz
- cd librdkafka-0.9.4/
+ wget https://github.com/edenhill/librdkafka/archive/v0.11.5.tar.gz -O - | tar -xz
+ cd librdkafka-0.11.5/
./configure --prefix=$PREFIX --enable-sasl
make
make install
```
1. Validate Librdkafka does indeed support SASL. Run the following command and ensure that `sasl` is returned as a built-in feature.
```
- $ examples/rdkafka_example -X builtin.features
- builtin.features = gzip,snappy,ssl,sasl,regex
- ```
-
- If it is not, ensure that you have `libsasl` or `libsasl2` installed. On CentOS, this can be installed with the following command.
- ```
- yum install -y cyrus-sasl cyrus-sasl-devel cyrus-sasl-gssapi
+ $ examples/rdkafka_example -X builtin.features
+ builtin.features = gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins
```
+1. The source install of confluent-kafka:
+
+ ```
+ // If you have already installed, remove the binary wheel python client first, repeat until it says no longer installed
--- End diff --
Can you extract the comment from the code block? Seems like it just just be part of the text.
---
[GitHub] metron pull request #1136: METRON-1701: Update steps to run the pycapa on ke...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1136#discussion_r206503684
--- Diff: metron-sensors/pycapa/README.md ---
@@ -54,8 +54,8 @@ General notes on the installation of Pycapa.
```
export PREFIX=/usr
- wget https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz -O - | tar -xz
- cd librdkafka-0.9.4/
+ wget https://github.com/edenhill/librdkafka/archive/v0.11.5.tar.gz -O - | tar -xz
+ cd librdkafka-0.11.5/
--- End diff --
Did you test both the kerberos and non-kerberos deployment of Pycapa with Librdkafka 0.11.5 ? The README is now advising this version, so I just want to make sure it works in all cases.
---
[GitHub] metron pull request #1136: METRON-1701: Update steps to run the pycapa on ke...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1136#discussion_r206502701
--- Diff: metron-sensors/pycapa/README.md ---
@@ -239,25 +239,47 @@ The probe can be used in a Kerberized environment. Follow these additional step
* The keytab used is located at `/etc/security/keytabs/metron.headless.keytab`
* The service principal is `metron@EXAMPLE.COM`
+ If it is not, ensure that you have `libsasl` or `libsasl2` installed. On CentOS, this can be installed with the following command.
+ ```
+ yum install -y cyrus-sasl cyrus-sasl-devel cyrus-sasl-gssapi
+ ```
+
+ 1. Use the non-binary wheel to install confluent-kafka-python.
--- End diff --
You say "you will need to use the non-binary wheel", but in this step there are no instructions on how to do that.
Should everything here just be added as additional commentary to what is currently step 4 ("The source install of confluent-kafka?") I believe that is the step that actually has the instructions on how to do that.
---
[GitHub] metron pull request #1136: METRON-1701: Update steps to run the pycapa on ke...
Posted by MohanDV <gi...@git.apache.org>.
Github user MohanDV commented on a diff in the pull request:
https://github.com/apache/metron/pull/1136#discussion_r206864051
--- Diff: metron-sensors/pycapa/README.md ---
@@ -54,8 +54,8 @@ General notes on the installation of Pycapa.
```
export PREFIX=/usr
- wget https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz -O - | tar -xz
- cd librdkafka-0.9.4/
+ wget https://github.com/edenhill/librdkafka/archive/v0.11.5.tar.gz -O - | tar -xz
+ cd librdkafka-0.11.5/
--- End diff --
Yes it works for unsecured deployment as well.
---
[GitHub] metron pull request #1136: METRON-1701: Update steps to run the pycapa on ke...
Posted by MohanDV <gi...@git.apache.org>.
Github user MohanDV commented on a diff in the pull request:
https://github.com/apache/metron/pull/1136#discussion_r206862766
--- Diff: metron-sensors/pycapa/README.md ---
@@ -239,25 +239,47 @@ The probe can be used in a Kerberized environment. Follow these additional step
* The keytab used is located at `/etc/security/keytabs/metron.headless.keytab`
* The service principal is `metron@EXAMPLE.COM`
+ If it is not, ensure that you have `libsasl` or `libsasl2` installed. On CentOS, this can be installed with the following command.
--- End diff --
done
---
[GitHub] metron pull request #1136: METRON-1701: Update steps to run the pycapa on ke...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1136#discussion_r206501313
--- Diff: metron-sensors/pycapa/README.md ---
@@ -239,25 +239,47 @@ The probe can be used in a Kerberized environment. Follow these additional step
* The keytab used is located at `/etc/security/keytabs/metron.headless.keytab`
* The service principal is `metron@EXAMPLE.COM`
+ If it is not, ensure that you have `libsasl` or `libsasl2` installed. On CentOS, this can be installed with the following command.
--- End diff --
Can we start numbering the list from here? Just add 1. before this line.
---
[GitHub] metron pull request #1136: METRON-1701: Update steps to run the pycapa on ke...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/metron/pull/1136
---
[GitHub] metron pull request #1136: METRON-1701: Update steps to run the pycapa on ke...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1136#discussion_r206502941
--- Diff: metron-sensors/pycapa/README.md ---
@@ -239,25 +239,47 @@ The probe can be used in a Kerberized environment. Follow these additional step
* The keytab used is located at `/etc/security/keytabs/metron.headless.keytab`
* The service principal is `metron@EXAMPLE.COM`
+ If it is not, ensure that you have `libsasl` or `libsasl2` installed. On CentOS, this can be installed with the following command.
+ ```
+ yum install -y cyrus-sasl cyrus-sasl-devel cyrus-sasl-gssapi
+ ```
+
+ 1. Use the non-binary wheel to install confluent-kafka-python.
+
+ The Python client README (https://github.com/confluentinc/confluent-kafka-python) has an important note for Kerberos case:
+ ```
+ **NOTE:** The pre-built Linux wheels do NOT contain SASL Kerberos support.
+ If you need SASL Kerberos support you must install librdkafka and
+ its dependencies using the above repositories and then build
+ confluent-kafka from source.
+ ```
--- End diff --
Its probably sufficient to just say "The pre-built Linux wheels do NOT contain SASL Kerberos support." and include a link to the README. No need to copy verbatim the whole paragraph. IMHO.
---