You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by JonZeolla <gi...@git.apache.org> on 2018/11/28 16:01:18 UTC
[GitHub] metron-bro-plugin-kafka pull request #20: METRON-1910: bro plugin segfaults ...
GitHub user JonZeolla opened a pull request:
https://github.com/apache/metron-bro-plugin-kafka/pull/20
METRON-1910: bro plugin segfaults on src/KafkaWriter.cc:72
## Contributor Comments
Bro was segfaulting on the KafkaWriter.cc destructor because it was attempting to destroy things that didn't exist.
### Testing
Spin up apache/metron full-dev with `vagrant --ansible-skip-tags="sensor-stubs,solr" up` and then run bro against some pcap, such as:
```
vagrant ssh
sudo su -
export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin
service pcap-replay restart
broctl stop || broctl stop # Handle bro crashes on shutdown
service snort-producer stop
Service snortd stop
Service yaf stop
yum -y install wireshark
sed -i 's/redef Kafka::logs_to_send = .*/redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG, FTP::LOG, Files::LOG, Known::CERTS_LOG, SMTP::LOG, SSL::LOG, Weird::LOG, Notice::LOG, DHCP::LOG, SSH::LOG, Software::LOG, RADIUS::LOG, X509::LOG, Known::DEVICES_LOG, RFB::LOG, Stats::LOG, CaptureLoss::LOG, SIP::LOG);/' /usr/local/bro/share/bro/site/local.bro
echo "redef Kafka::logs_to_exclude = set(Conn::LOG, DHCP::LOG);" >> /usr/local/bro/share/bro/site/local.bro
echo "redef Known::cert_tracking = ALL_HOSTS;" >> /usr/local/bro/share/bro/site/local.bro
echo "redef Software::asset_tracking = ALL_HOSTS;" >> /usr/local/bro/share/bro/site/local.bro
sed -i '86 a @load policy/protocols/dhcp/known-devices-and-hostnames.bro' /usr/local/bro/share/bro/site/local.bro
screen
kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro
# New screen window
for folder in nitroba example-traffic ssh ftp radius rfb; do mkdir -p ~/brotmp/${folder}; done
wget https://www.bro.org/static/traces/exercise-traffic.pcap -O ~/brotmp/example-traffic/exercise-traffic.pcap
wget http://downloads.digitalcorpora.org/corpora/network-packet-dumps/2008-nitroba/nitroba.pcap -O ~/brotmp/nitroba/nitroba.pcap
wget https://www.bro.org/static/traces/ssh.pcap -O ~/brotmp/ssh/ssh.pcap
wget https://github.com/markofu/pcaps/blob/master/PracticalPacketAnalysis/ppa-capture-files/ftp.pcap?raw=true -O ~/brotmp/ftp/ftp.pcap
wget https://github.com/EmpowerSecurityAcademy/wireshark/blob/master/radius_localhost.pcapng?raw=true -O ~/brotmp/radius/radius_localhost.pcapng
wget https://github.com/kholia/my-pcaps/blob/master/VNC/07-vnc-openwall-3.7.pcap?raw=true -O ~/brotmp/rfb/rfb.pcap
cd ~/brotmp/example-traffic
bro -r exercise-traffic.pcap /usr/local/bro/share/bro/site/local.bro -C
cd ~/brotmp/nitroba
bro -r nitroba.pcap /usr/local/bro/share/bro/site/local.bro -C
cd ~/brotmp/ssh
bro -r ssh.pcap /usr/local/bro/share/bro/site/local.bro -C
cd ~/brotmp/ftp
bro -r ftp.pcap /usr/local/bro/share/bro/site/local.bro -C
cd ~/brotmp/radius
editcap -F libpcap radius_localhost.pcapng radius_localhost.pcap
bro -r radius_localhost.pcap /usr/local/bro/share/bro/site/local.bro -C
cd ~/brotmp/rfb
bro -r rfb.pcap /usr/local/bro/share/bro/site/local.bro -C
```
Notice that after each pcap is done processing it doesn't segfault.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron's Bro kafka writer plugin.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [X] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [X] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [X] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [X] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [X] Have you included steps or a guide to how the change may be verified and tested manually?
- [X] Have you ensured that the full suite of tests and checks have been executed via:
```
bro-pkg test $GITHUB_USERNAME/metron-bro-plugin-kafka --version $BRANCH
```
- [ ] Have you written or updated unit tests and or integration tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building and running locally with Apache Metron's [Vagrant full-dev environment](https://github.com/apache/metron/tree/master/metron-deployment/development/centos6) or the equivalent?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JonZeolla/metron-bro-plugin-kafka METRON-1910
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron-bro-plugin-kafka/pull/20.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20
----
commit a6663ff8f100686199c4e49c138fc2befc3bffe9
Author: Jon Zeolla <ze...@...>
Date: 2018-11-28T15:10:48Z
METRON-1910: bro plugin segfaults on src/KafkaWriter.cc:72
----
---
[GitHub] metron-bro-plugin-kafka issue #20: METRON-1910: bro plugin segfaults on src/...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron-bro-plugin-kafka/pull/20
No problem @JonZeolla. I can help track it down too when I get some free time.
---
[GitHub] metron-bro-plugin-kafka issue #20: METRON-1910: bro plugin segfaults on src/...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on the issue:
https://github.com/apache/metron-bro-plugin-kafka/pull/20
I wish I had more time to work on this, but it may be a while before I could revisit. What I found was that if any of the `delete`s are duplicated it segfaults on exit. From some commit messages it seems that in bro, the destructor is called from the main thread, but `DoFinish` is called from a child thread, so we should keep this cleanup in `DoFinish`.
---
[GitHub] metron-bro-plugin-kafka issue #20: METRON-1910: bro plugin segfaults on src/...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron-bro-plugin-kafka/pull/20
Can you provide some commentary on the root cause and your solution?
---