You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by mmiklavc <gi...@git.apache.org> on 2017/04/03 16:01:43 UTC

[GitHub] incubator-metron pull request #507: METRON-819: Document kafka console produ...

GitHub user mmiklavc opened a pull request:

    https://github.com/apache/incubator-metron/pull/507

    METRON-819: Document kafka console producer parameter for sensors with kerberos

    Addresses https://issues.apache.org/jira/browse/METRON-819
    
    Adds a note about adding the security protocol property to sensors leveraging the Kafka console producer.
    
    ## Pull Request Checklist
    
    Thank you for submitting a contribution to Apache Metron (Incubating).  
    Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.  
    Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.  
    
    
    In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
    
    ### For all changes:
    - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). 
    - [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
    - [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
    
    ### For documentation related changes:
    - [X] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
    
      ```
      cd site-book
      bin/generate-md.sh
      mvn site:site
      ```
    
    #### Note:
    Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
    It is also recommened that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/mmiklavc/incubator-metron METRON-819

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-metron/pull/507.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #507
    
----
commit fb911d942d795e03cb171ec2963e962490355512
Author: Michael Miklavcic <mi...@gmail.com>
Date:   2017-04-03T15:58:21Z

    METRON-819: Document kafka console producer parameter for sensors with kerberos

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #507: METRON-819: Document kafka console producer par...

Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:

    https://github.com/apache/incubator-metron/pull/507
  
    Don't merge this - need to move this doc change to the new manual kerberos doc in metron-deployment dir. A fix for a bad merge is pending.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #507: METRON-819: Document kafka console producer par...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/incubator-metron/pull/507
  
    I think I am confusing steps (12) and (13) from your instructions or something.  But something else weird is going on.  I'm just not sure what.  
    
    It seems like the ACLs were set and then at some point they got unset somehow.  I'm going to start over and walk through it all again.  Maybe I made a mistake.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #507: METRON-819: Document kafka console producer par...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/incubator-metron/pull/507
  
    > @mmiklavc  Can you try listing and applying acls with the root user instead of metron?
    
    Ok, sure.
    
    1. As root, I can see the ACLs.  But oddly there are none set.
    ```
    [root@node1 ~]# kafka-acls.sh --list --topic yaf --authorizer-properties zookeeper.connect=node1:2181
    Current ACLs for resource `Topic:yaf`:
    
    ```
    
    2. Then I set the ACLs again.  So this looks pretty good.  Now it looks like they were set.
    ```
    [root@node1 ~]# kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:metron --group yaf_parser;
    Adding ACLs for resource `Group:yaf_parser`:
     	User:metron has Allow permission for operations: All from hosts: *
    
    Current ACLs for resource `Group:yaf_parser`:
     	User:metron has Allow permission for operations: All from hosts: *
    
    ```
    
    3.   But then if I immediately check them again and they do not exist.  Weird.
    ```
    [root@node1 ~]# kafka-acls.sh --list --topic yaf --authorizer-properties zookeeper.connect=node1:2181 --authorizer kafka.security.auth.SimpleAclAuthorizer
    Current ACLs for resource `Topic:yaf`:
    
    ```



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #507: METRON-819: Document kafka console producer par...

Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:

    https://github.com/apache/incubator-metron/pull/507
  
    @nickwallen Thanks for testing the Kerberos instructions out!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #507: METRON-819: Document kafka console producer par...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/incubator-metron/pull/507
  
    >  I'm also curious about the difference in setup that required the jaas file here versus the original setup that allowed you to use the console consumer successfully per the comments above.
    
    In the original test the JAAS stuff was configured as a by-product of something else I was working on.  So I have never gotten this to work without configuring JAAS.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #507: METRON-819: Document kafka console producer par...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/incubator-metron/pull/507
  
    > Anything going on in the kafka broker logs in /var/log/...?
    
    @cestella Nothing interesting that I can find in the logs, unfortunately.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #507: METRON-819: Document kafka console producer par...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/incubator-metron/pull/507
  
    > can you elaborate on this fix/config a bit? I think we should definitely add this detail to the doc. It looks like you've created a yaf user and prin
    
    In the text that I pasted, I was switching between two different use cases.  So I accidentally pasted the JAAS setup for my 'yaf' principal, versus the 'bro' principal.  So ignore that.
    
    But I cannot think of anything beyond the two steps I defined above.  I just create a JAAS file telling it which keytab and principal to use, then update the JVM's security config so that it knows where to find the JAAS file, then it works for me.
    
    This is just the one way I have gotten it to work.  There may be a better/simpler way that I am not aware of.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #507: METRON-819: Document kafka console producer par...

Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:

    https://github.com/apache/incubator-metron/pull/507
  
    Merge with master again


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #507: METRON-819: Document kafka console producer par...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/incubator-metron/pull/507
  
    The issues that I am having currently are with Quick Dev.  But I have actually been able to do this on a separate cluster in a slightly different way.  On the other cluster, I did not use the `--group` option when setting the ACL.  If I did set the group, then I had to ensure that the group matched what was used by the `kafka-console-producer`.
    
    So as a test, I granted access without the `--group`.
    
    1. Grant access.  Look ma, no group.
    ```
    [root@node1 ~]# kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:metron --topic yaf
    Adding ACLs for resource `Topic:yaf`:
     	User:metron has Allow permission for operations: All from hosts: *
    
    Current ACLs for resource `Topic:yaf`:
     	User:metron has Allow permission for operations: All from hosts: *
    ```
    
    2. Validate the ACL.  Looks good this time.
    ```
    [root@node1 ~]# kafka-acls.sh --list --topic yaf --authorizer-properties zookeeper.connect=node1:2181 --authorizer kafka.security.auth.SimpleAclAuthorizer
    Current ACLs for resource `Topic:yaf`:
     	User:metron has Allow permission for operations: All from hosts: *
    
    ```
    
    3. And now I can send data successfully.
    ```
    [root@node1 ~]# echo "foo" | kafka-console-producer.sh --broker-list node1:6667 --topic yaf --security-protocol SASL_PLAINTEXT
    [2017-04-07 17:05:28,830] WARN The TGT cannot be renewed beyond the next expiry date: Sat Apr 08 16:11:26 UTC 2017.This process will not be able to authenticate new SASL connections after that time (for example, it will not be able to authenticate a new connection with a Kafka Broker).  Ask your system administrator to either increase the 'renew until' time by doing : 'modprinc -maxrenewlife null ' within kadmin, or instead, to generate a keytab for null. Because the TGT's expiry cannot be further extended by refreshing, exiting refresh thread now. (org.apache.kafka.common.security.kerberos.KerberosLogin)
    ```
    



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Re: [GitHub] incubator-metron issue #507: METRON-819: Document kafka console producer par...

Posted by Michael Miklavcic <mi...@gmail.com>.
Can you try listing and applying acls with the root user instead of metron?

On Fri, Apr 7, 2017 at 10:29 AM, nickwallen <gi...@git.apache.org> wrote:

> Github user nickwallen commented on the issue:
>
>     https://github.com/apache/incubator-metron/pull/507
>
>     I went through your instructions and all seemed well with the world.
> But then I tried to use the `kafka-console-producer` to actually write data
> to Kafka and it fails  Any ideas what the problem might be?
>
>     ```
>     [metron@node1 ~]$ kinit -kt /etc/security/keytabs/metron.headless.keytab
> metron@EXAMPLE.COM
>     [metron@node1 ~]$ echo "foo" | kafka-console-producer.sh
> --broker-list node1:6667 --topic yaf --security-protocol SASL_PLAINTEXT
>     [2017-04-07 16:29:00,639] WARN The TGT cannot be renewed beyond the
> next expiry date: Sat Apr 08 16:28:58 UTC 2017.This process will not be
> able to authenticate new SASL connections after that time (for example, it
> will not be able to authenticate a new connection with a Kafka Broker).
> Ask your system administrator to either increase the 'renew until' time by
> doing : 'modprinc -maxrenewlife null ' within kadmin, or instead, to
> generate a keytab for null. Because the TGT's expiry cannot be further
> extended by refreshing, exiting refresh thread now.
> (org.apache.kafka.common.security.kerberos.KerberosLogin)
>     [2017-04-07 16:29:00,897] WARN Error while fetching metadata with
> correlation id 0 : {yaf=TOPIC_AUTHORIZATION_FAILED}
> (org.apache.kafka.clients.NetworkClient)
>     [2017-04-07 16:29:00,897] ERROR Error when sending message to topic
> yaf with key: null, value: 3 bytes with error: (org.apache.kafka.clients.
> producer.internals.ErrorLoggingCallback)
>     org.apache.kafka.common.errors.TopicAuthorizationException: Not
> authorized to access topics: [yaf]
>     ```
>
>     I then tried to go back and check the Kafka ACLs and am now getting an
> error.  I was able to set the ACLs, but now I cannot see them.
>
>     ```
>     [metron@node1 ~]$ kinit -kt /etc/security/keytabs/metron.headless.keytab
> metron@EXAMPLE.COM
>     [metron@node1 ~]$ kafka-acls.sh --list --topic yaf
> --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181
>     [2017-04-07 16:24:47,794] WARN Could not login: the client is being
> asked for a password, but the Zookeeper client code does not currently
> support obtaining a password from the user. Make sure that the client is
> configured to use a ticket cache (using the JAAS configuration setting
> 'useTicketCache=true)' and restart the client. If you still get this
> message after that, the TGT in the ticket cache has expired and must be
> manually refreshed. To do so, first determine if you are using a password
> or a keytab. If the former, run kinit in a Unix shell in the environment of
> the user who is running this Zookeeper client using the command 'kinit
> <princ>' (where <princ> is the name of the client's Kerberos principal). If
> the latter, do 'kinit -k -t <keytab> <princ>' (where <princ> is the name of
> the Kerberos principal, and <keytab> is the location of the keytab file).
> After manually refreshing your cache, restart this client. If you continue
> to see this message after manually refreshing yo
>  ur cache, ensure that your KDC host's clock is in sync with this host's
> clock. (org.apache.zookeeper.client.ZooKeeperSaslClient)
>     [2017-04-07 16:24:47,796] WARN SASL configuration failed:
> javax.security.auth.login.LoginException: No password provided Will
> continue connection to Zookeeper server without SASL authentication, if
> Zookeeper server allows it. (org.apache.zookeeper.ClientCnxn)
>     Error while executing ACL command: Authentication failure
>     org.I0Itec.zkclient.exception.ZkAuthFailedException: Authentication
> failure
>         at org.I0Itec.zkclient.ZkClient.waitForKeeperState(ZkClient.
> java:946)
>
>     ```
>
>
> ---
> If your project is set up for it, you can reply to this email and have your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, please
> contact infrastructure at infrastructure@apache.org or file a JIRA ticket
> with INFRA.
> ---
>

[GitHub] incubator-metron issue #507: METRON-819: Document kafka console producer par...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/incubator-metron/pull/507
  
    I went through your instructions and all seemed well with the world.  But then I tried to use the `kafka-console-producer` to actually write data to Kafka and it fails  Any ideas what the problem might be?
    
    ```
    [metron@node1 ~]$ kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
    [metron@node1 ~]$ echo "foo" | kafka-console-producer.sh --broker-list node1:6667 --topic yaf --security-protocol SASL_PLAINTEXT
    [2017-04-07 16:29:00,639] WARN The TGT cannot be renewed beyond the next expiry date: Sat Apr 08 16:28:58 UTC 2017.This process will not be able to authenticate new SASL connections after that time (for example, it will not be able to authenticate a new connection with a Kafka Broker).  Ask your system administrator to either increase the 'renew until' time by doing : 'modprinc -maxrenewlife null ' within kadmin, or instead, to generate a keytab for null. Because the TGT's expiry cannot be further extended by refreshing, exiting refresh thread now. (org.apache.kafka.common.security.kerberos.KerberosLogin)
    [2017-04-07 16:29:00,897] WARN Error while fetching metadata with correlation id 0 : {yaf=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)
    [2017-04-07 16:29:00,897] ERROR Error when sending message to topic yaf with key: null, value: 3 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
    org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [yaf]
    ```
    
    I then tried to go back and check the Kafka ACLs and am now getting an error.  I was able to set the ACLs, but now I cannot see them.
    
    ```
    [metron@node1 ~]$ kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
    [metron@node1 ~]$ kafka-acls.sh --list --topic yaf --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181
    [2017-04-07 16:24:47,794] WARN Could not login: the client is being asked for a password, but the Zookeeper client code does not currently support obtaining a password from the user. Make sure that the client is configured to use a ticket cache (using the JAAS configuration setting 'useTicketCache=true)' and restart the client. If you still get this message after that, the TGT in the ticket cache has expired and must be manually refreshed. To do so, first determine if you are using a password or a keytab. If the former, run kinit in a Unix shell in the environment of the user who is running this Zookeeper client using the command 'kinit <princ>' (where <princ> is the name of the client's Kerberos principal). If the latter, do 'kinit -k -t <keytab> <princ>' (where <princ> is the name of the Kerberos principal, and <keytab> is the location of the keytab file). After manually refreshing your cache, restart this client. If you continue to see this message after manually refreshing yo
 ur cache, ensure that your KDC host's clock is in sync with this host's clock. (org.apache.zookeeper.client.ZooKeeperSaslClient)
    [2017-04-07 16:24:47,796] WARN SASL configuration failed: javax.security.auth.login.LoginException: No password provided Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. (org.apache.zookeeper.ClientCnxn)
    Error while executing ACL command: Authentication failure
    org.I0Itec.zkclient.exception.ZkAuthFailedException: Authentication failure
    	at org.I0Itec.zkclient.ZkClient.waitForKeeperState(ZkClient.java:946)
    
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #507: METRON-819: Document kafka console producer par...

Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on the issue:

    https://github.com/apache/incubator-metron/pull/507
  
    @nickwallen , this is a stab in the dark, but FWIT:  Is Ambari in the mix, and is it an Ambari-managed parameter?  If it's an Ambari-managed parameter, it will set back any changes you make while ambari-agent is running.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #507: METRON-819: Document kafka console producer par...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/incubator-metron/pull/507
  
    +1 Rockin!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #507: METRON-819: Document kafka console producer par...

Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on the issue:

    https://github.com/apache/incubator-metron/pull/507
  
    I got a very similar issue, which is why I've been holding off with #510 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #507: METRON-819: Document kafka console producer par...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/incubator-metron/pull/507
  
    I also remember (after fighting with this for a while this morning) that if you don't have a JAAS config file defined then it won't work.  Should this be part of the docs in this PR??
    
    Without it, you just get this error.
    ```
    [root@y136 ~]# kafka-console-consumer.sh  --zookeeper y113:2181 --topic bro --security-protocol SASL_PLAINTEXT
    [2017-04-28 16:47:20,596] WARN Could not login: the client is being asked for a password, but the Zookeeper client code does not currently support obtaining a password from the user. Make sure that the client is configured to use a ticket cache (using the JAAS configuration setting 'useTicketCache=true)' and restart the client. If you still get this message after that, the TGT in the ticket cache has expired and must be manually refreshed. To do so, first determine if you are using a password or a keytab. If the former, run kinit in a Unix shell in the environment of the user who is running this Zookeeper client using the command 'kinit <princ>' (where <princ> is the name of the client's Kerberos principal). If the latter, do 'kinit -k -t <keytab> <princ>' (where <princ> is the name of the Kerberos principal, and <keytab> is the location of the keytab file). After manually refreshing your cache, restart this client. If you continue to see this message after manually refreshing yo
 ur cache, ensure that your KDC host's clock is in sync with this host's clock. (org.apache.zookeeper.client.ZooKeeperSaslClient)
    [2017-04-28 16:47:20,599] WARN SASL configuration failed: javax.security.auth.login.LoginException: No password provided Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. (org.apache.zookeeper.ClientCnxn)
    No brokers found in ZK.
    ```
    After doing the following, then it works for me.
    
    1. Define `~/.java.login.config` 
    
    ```
    [root@y137 ~]# cat ~/.java.login.config
    KafkaClient {
      com.sun.security.auth.module.Krb5LoginModule required
      useTicketCache=false
      useKeyTab=true
      principal="yaf/y137...@EXAMPLE.COM"
      keyTab="/etc/security/keytabs/yaf.service.keytab"
      renewTicket=true
      debug=true
      serviceName="kafka"
      storeKey=true;
    };
    ```
    
    2. Tell the JVM where to find your JAAS file.
    
    ```
    [root@y137 ~]# cat /usr/lib/jvm/java-1.8.0-openjdk/jre/lib/security/java.security | grep login
    # Class to instantiate as the javax.security.auth.login.Configuration
    login.configuration.provider=sun.security.provider.ConfigFile
    # Default login configuration file
    #login.config.url.1=file:${user.home}/.java.login.config
    login.config.url.1=file:${user.home}/.java.login.config
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #507: METRON-819: Document kafka console producer par...

Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:

    https://github.com/apache/incubator-metron/pull/507
  
    @nickwallen can you elaborate on this fix/config a bit? I think we should definitely add this detail to the doc. It looks like you've created a yaf user and principal here. Any additional setup or configuration required? I'm also curious about the difference in setup that required the jaas file here versus the original setup that allowed you to use the console consumer successfully per the comments above.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #507: METRON-819: Document kafka console produ...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/507#discussion_r111164016
  
    --- Diff: metron-deployment/vagrant/Kerberos-setup.md ---
    @@ -221,6 +221,10 @@ curl -XGET "${ZOOKEEPER}:9200/yaf*/_count"
     
     25. You should have data flowing from the parsers all the way through to the indexes. This completes the Kerberization instructions
     
    +### Sensors
    +
    +For sensors that leverage the Kafka console producer to pipe data into Metron, e.g. Snort and Yaf, you will need to modify the corresponding sensor shell script to append the SASL security protocol property. `--security-protocol SASL_PLAINTEXT`
    +
    --- End diff --
    
    Should we call out the need to `kinit` beforehand?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #507: METRON-819: Document kafka console producer par...

Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on the issue:

    https://github.com/apache/incubator-metron/pull/507
  
    > Is Ambari in the mix
    
    Urm, sorry.  That was old behavior, they changed it so Ambari doesn't do that any more.  It prevented too many useful config tweaks.  Now, ambari-agent only resets config state to match the ambari database, at startup time and when configs in Ambari are changed.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #507: METRON-819: Document kafka console producer par...

Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:

    https://github.com/apache/incubator-metron/pull/507
  
    I didn't have any problems using the Kafka shell commands without a JAAS file. That was on single node Vagrant, but I'm not sure there should be much of a difference in this case. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #507: METRON-819: Document kafka console producer par...

Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:

    https://github.com/apache/incubator-metron/pull/507
  
    @nickwallen  Anything going on in the kafka broker logs in `/var/log/...`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #507: METRON-819: Document kafka console producer par...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/incubator-metron/pull/507
  
    FYI I was able to get this working.  Mike's docs are 100% correct, there were just a few minor steps that tripped me up (like using relative paths instead of absolute paths.)  I updated those just to help others avoid the same stupid user mistakes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #507: METRON-819: Document kafka console produ...

Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/507#discussion_r112297415
  
    --- Diff: metron-deployment/vagrant/Kerberos-setup.md ---
    @@ -221,6 +221,10 @@ curl -XGET "${ZOOKEEPER}:9200/yaf*/_count"
     
     25. You should have data flowing from the parsers all the way through to the indexes. This completes the Kerberization instructions
     
    +### Sensors
    +
    +For sensors that leverage the Kafka console producer to pipe data into Metron, e.g. Snort and Yaf, you will need to modify the corresponding sensor shell script to append the SASL security protocol property. `--security-protocol SASL_PLAINTEXT`
    +
    --- End diff --
    
    Added the kinit


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---