You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by merrimanr <gi...@git.apache.org> on 2016/10/25 15:57:34 UTC

[GitHub] incubator-metron pull request #323: Snort logs are now sent to Kafka via kaf...

GitHub user merrimanr opened a pull request:

    https://github.com/apache/incubator-metron/pull/323

    Snort logs are now sent to Kafka via kafka-console-producer

    This PR attempts to solve the issue of snort data not flowing after an extended amount of time by removing Flume as a dependency and simplifying the process.  Flume is replaced by a combination of the "tail" command and kafka-console-producer, making it much easier to troubleshoot.
    
    The implementation was inspired by the yaf sensor.  The one significant change is in the "metron-deployment/roles/snort/templates/snort-producer" init script.  The yaf init script stops the child processes by name with "killall" which doesn't work in this case because "snort-producer" is not in the first "tail" command.  Instead the process group id is retrieved in line 64 and passed to the "kill" command in line 67.
    
    This has been tested in quick-dev, including managing the sensor through Monit.  I searched for other Flume dependencies but could not find any, therefore Flume was removed completely.    

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/merrimanr/incubator-metron METRON-514

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-metron/pull/323.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #323
    
----
commit 5515562f3a8e9e9f292c614c7542acdb91c59e58
Author: rmerriman <rm...@hortonworks.com>
Date:   2016-10-25T15:41:01Z

    Snort logs are now sent to Kafka via kafka-console-producer

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #323: Snort logs are now sent to Kafka via kaf...

Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/323#discussion_r84939411
  
    --- Diff: metron-deployment/roles/snort/tasks/producer.yml ---
    @@ -15,5 +15,18 @@
     #  limitations under the License.
     #
     ---
    -hdp_repo_def: "http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.4.2.0/hdp.repo"
    -java_home: /usr/jdk64/jdk1.8.0_60
    +- name: Create Snort producer home directory
    +  file:
    +    path: "{{ snort_producer_home }}"
    +    state: directory
    +    mode: 0755
    +
    +- name: Install Snort producer start script
    +  template: src=start-snort-producer.sh dest={{ snort_producer_home }}/start-snort-producer.sh mode=0755
    +
    +- name: Install init.d service script
    +  template: src=snort-producer dest=/etc/init.d/snort-producer mode=0755
    +
    +- name: Register the service with systemd
    +  shell: systemctl enable snort-producer
    +  when: ansible_distribution == "CentOS" and ansible_distribution_major_version == "7"
    --- End diff --
    
    I don't think you need this. The way it is currently written, it will enable service start on boot for Centos 7 only. If you want to enable on bootup, you can use the Ansible service module with enabled set to yes. Ansible will (should) take care of figuring out which os you're running an do the right thing.
    Otherwise, you can just remove this block.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #323: METRON-514: Snort kafka topic fails to get any ...

Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:

    https://github.com/apache/incubator-metron/pull/323
  
    @merrimanr - I was hoping to merge this in today, but it's conflicted. Think you'd have time to take a peek?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #323: METRON-514: Snort kafka topic fails to g...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/323#discussion_r85134447
  
    --- Diff: metron-deployment/roles/snort/tasks/producer.yml ---
    @@ -15,5 +15,18 @@
     #  limitations under the License.
     #
     ---
    -hdp_repo_def: "http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.4.2.0/hdp.repo"
    -java_home: /usr/jdk64/jdk1.8.0_60
    +- name: Create Snort producer home directory
    +  file:
    +    path: "{{ snort_producer_home }}"
    +    state: directory
    +    mode: 0755
    +
    +- name: Install Snort producer start script
    +  template: src=start-snort-producer.sh dest={{ snort_producer_home }}/start-snort-producer.sh mode=0755
    +
    +- name: Install init.d service script
    +  template: src=snort-producer dest=/etc/init.d/snort-producer mode=0755
    +
    +- name: Register the service with systemd
    +  shell: systemctl enable snort-producer
    +  when: ansible_distribution == "CentOS" and ansible_distribution_major_version == "7"
    --- End diff --
    
    Okay removed those lines from producer.yml and yaf.yml


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #323: METRON-514: Snort kafka topic fails to g...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/323#discussion_r85796965
  
    --- Diff: metron-deployment/roles/yaf/tasks/yaf.yml ---
    @@ -48,7 +48,3 @@
     
     - name: Install init.d service script
       template: src=yaf dest=/etc/init.d/yaf mode=0755
    -
    --- End diff --
    
    Sorry, late to the party. Had to expand the 'thingie' to see that prior conversation.   
    
    If I remember correctly, the original reason that was added was because if you just deploy and then run 'service yaf start' on CentOS 7, Systemd would fail the service immediately.  Doing the `enable` made Systemd aware of the older SysV script, so that when we start it later in the deployment, it works.  
    
    But this was probably not the best way to do it, as this has the effect of starting the service on boot, which is not what we want.  
    
    I'm good with taking this out (as you have it), since your testing has been successful without it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #323: METRON-514: Snort kafka topic fails to get any ...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:

    https://github.com/apache/incubator-metron/pull/323
  
    https://issues.apache.org/jira/browse/METRON-514
    
    On Tue, Oct 25, 2016 at 9:57 PM, ottobackwards <no...@github.com>
    wrote:
    
    > Is there a jira for this?
    >
    > \u2014
    > You are receiving this because you authored the thread.
    > Reply to this email directly, view it on GitHub
    > <https://github.com/apache/incubator-metron/pull/323#issuecomment-256236889>,
    > or mute the thread
    > <https://github.com/notifications/unsubscribe-auth/AATCePIs_4uJ2Fj9SRMOqqyeYNqqX2tKks5q3sGSgaJpZM4KgKn4>
    > .
    >



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #323: Snort logs are now sent to Kafka via kafka-cons...

Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:

    https://github.com/apache/incubator-metron/pull/323
  
    Is there a jira for this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #323: METRON-514: Snort kafka topic fails to g...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/323#discussion_r85773748
  
    --- Diff: metron-deployment/roles/yaf/tasks/yaf.yml ---
    @@ -48,7 +48,3 @@
     
     - name: Install init.d service script
       template: src=yaf dest=/etc/init.d/yaf mode=0755
    -
    --- End diff --
    
    Did you intend to remove these lines related to YAF and CentOS 7?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #323: METRON-514: Snort kafka topic fails to get any ...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:

    https://github.com/apache/incubator-metron/pull/323
  
    @dlyle65535 conflict is resolved


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #323: Snort logs are now sent to Kafka via kaf...

Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/323#discussion_r84944487
  
    --- Diff: metron-deployment/roles/snort/tasks/producer.yml ---
    @@ -15,5 +15,18 @@
     #  limitations under the License.
     #
     ---
    -hdp_repo_def: "http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.4.2.0/hdp.repo"
    -java_home: /usr/jdk64/jdk1.8.0_60
    +- name: Create Snort producer home directory
    +  file:
    +    path: "{{ snort_producer_home }}"
    +    state: directory
    +    mode: 0755
    +
    +- name: Install Snort producer start script
    +  template: src=start-snort-producer.sh dest={{ snort_producer_home }}/start-snort-producer.sh mode=0755
    +
    +- name: Install init.d service script
    +  template: src=snort-producer dest=/etc/init.d/snort-producer mode=0755
    +
    +- name: Register the service with systemd
    +  shell: systemctl enable snort-producer
    +  when: ansible_distribution == "CentOS" and ansible_distribution_major_version == "7"
    --- End diff --
    
    Probably. I'm not sure if the intention is for the services to start on boot up or not. I suspect not.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #323: METRON-514: Snort kafka topic fails to g...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/incubator-metron/pull/323


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #323: METRON-514: Snort kafka topic fails to get any ...

Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:

    https://github.com/apache/incubator-metron/pull/323
  
    Thanks @merrimanr, based on your testing I'm +1 by inspection (travis pending)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #323: METRON-514: Snort kafka topic fails to g...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/323#discussion_r85785670
  
    --- Diff: metron-deployment/roles/yaf/tasks/yaf.yml ---
    @@ -48,7 +48,3 @@
     
     - name: Install init.d service script
       template: src=yaf dest=/etc/init.d/yaf mode=0755
    -
    --- End diff --
    
    I removed it based on a comment in this PR from @dlyle65535 .  Happy to put it back in if it makes sense.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---