You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/11/20 02:05:00 UTC
[jira] [Commented] (METRON-1088) Upgrade bro to 2.5.2
[ https://issues.apache.org/jira/browse/METRON-1088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16258722#comment-16258722 ]
ASF GitHub Bot commented on METRON-1088:
----------------------------------------
GitHub user JonZeolla opened a pull request:
https://github.com/apache/metron/pull/844
METRON-1088: Upgrade bro to 2.5.2
## Contributor Comments
This upgrades bro to 2.5.2, and natively provides support for many of the fields from a default bro 2.5.2 install. Specifically, this attempts to provide initial support the default-on fields of the following logs:
- [Conn](https://www.bro.org/sphinx/scripts/base/protocols/conn/main.bro.html#type-Conn::Info)
- [DPD](https://www.bro.org/sphinx-git/scripts/base/frameworks/dpd/main.bro.html#type-DPD::Info)
- [FTP](https://www.bro.org/sphinx/scripts/base/protocols/ftp/info.bro.html#type-FTP::Info)
- [Files](https://www.bro.org/sphinx/scripts/base/frameworks/files/main.bro.html#type-Files::Info)
- [CertsInfo](https://www.bro.org/sphinx/scripts/policy/protocols/ssl/known-certs.bro.html#type-Known::CertsInfo)
- [SMTP](https://www.bro.org/sphinx/scripts/base/protocols/smtp/main.bro.html#type-SMTP::Info)
- [SSL](https://www.bro.org/sphinx/scripts/base/protocols/ssl/main.bro.html#type-SSL::Info)
- [Weird](https://www.bro.org/sphinx/scripts/base/frameworks/notice/weird.bro.html#type-Weird::Info)
- [Notice](https://www.bro.org/sphinx/scripts/base/frameworks/notice/main.bro.html#type-Notice::Info)
- [DHCP](https://www.bro.org/sphinx/scripts/base/protocols/dhcp/main.bro.html#type-DHCP::Info)
- [SSH](https://www.bro.org/sphinx/scripts/base/protocols/ssh/main.bro.html#type-SSH::Info)
- [Software](https://www.bro.org/sphinx/scripts/base/frameworks/software/main.bro.html#type-Software::Info)
- [Radius](https://www.bro.org/sphinx/scripts/base/protocols/radius/main.bro.html#type-RADIUS::Info)
- [X509](https://www.bro.org/sphinx/scripts/base/files/x509/main.bro.html#type-X509::Info)
- [DevicesInfo](https://www.bro.org/sphinx/scripts/policy/misc/known-devices.bro.html#type-Known::DevicesInfo)
- [RFB](https://www.bro.org/sphinx-git/scripts/base/protocols/rfb/main.bro.html#type-RFB::Info)
- [Stats](https://www.bro.org/sphinx/scripts/policy/misc/stats.bro.html#type-Stats::Info)
- [CaptureLoss](https://www.bro.org/sphinx/scripts/policy/misc/capture-loss.bro.html#type-CaptureLoss::Info)
- [SIP](https://www.bro.org/sphinx/scripts/base/protocols/sip/main.bro.html#type-SIP::Info)
Still outstanding:
- [ ] Unit/integration tests.
# Testing
1. Create a working directory and pull in this PR (selfishly using my branch of `checkout-pr` from [metron-commit-stuff](https://github.com/jonzeolla/metron-commit-stuff/tree/support-bro-plugin) to test some recent updates)
```
# mkdir ~/tmp
# git clone https://github.com/jonzeolla/metron-commit-stuff ~/tmp/metron-commit-stuff
# cd ~/tmp/metron-commit-stuff
# git checkout support-bro-plugin
# cd
# ~/tmp/metron-commit-stuff/checkout-pr 844
Please select a repository:
1) metron
2) metron-bro-plugin-kafka
Selection [metron]: metron
<snip>
# cd metron-pr844
# sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" metron-deployment/vagrant/full-dev-platform/Vagrantfile
# cd metron-deployment/vagrant/full-dev-platform/
# vagrant up
```
1. Set up the environment in full-dev.
```
vagrant ssh
sudo su -
export PATH=$PATH:/usr/local/bro/bin
service monit stop && service sensor-stubs stop bro && broctl stop
yum -y install jq wireshark screen
```
1. Configure kafka in local.bro so all of the currently supported bro logs are being sent.
```
sed -i 's/redef Kafka::logs_to_send = .*/redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG, FTP::LOG, Files::LOG, Known::CERTS_LOG, SMTP::LOG, SSL::LOG, Weird::LOG, Notice::LOG, DHCP::LOG, SSH::LOG, Software::LOG, RADIUS::LOG, X509::LOG, Known::DEVICES_LOG, RFB::LOG, Stats::LOG, CaptureLoss::LOG, SIP::LOG);/' /usr/local/bro/share/bro/site/local.bro
echo "redef Kafka::debug = \"all\";" >> /usr/local/bro/share/bro/site/local.bro
echo "redef Known::cert_tracking = ALL_HOSTS;" >> /usr/local/bro/share/bro/site/local.bro
echo "redef Software::asset_tracking = ALL_HOSTS;" >> /usr/local/bro/share/bro/site/local.bro
sed -i '86 a @load policy/protocols/dhcp/known-devices-and-hostnames.bro' /usr/local/bro/share/bro/site/local.bro
```
1. Monitor the bro kafka topic
```
export PATH=$PATH:/usr/hdp/current/kafka-broker/bin
screen
kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro
```
1. Monitor the storm logs.
```
# Ctrl+A c to make a new screen window
# Use Ctrl+A n to cycle through screen sessions
# In your new screen window, look at the storm logs (The "failed to parse" errors for ip_src_addr and ip_dst_addr are expected, and should be addressed as a part of METRON-939)
tail -f /var/log/storm/workers-artifacts/indexing-*/*/worker.log | grep -i "org.elasticsearch.index.mapper.MapperParsingException: failed to parse"
# You may want to evaluate worker.log for other errors, but the prior command is helpful to cut through some of the failed indexing of IPv6 addresses
```
1. Run bro against some public pcaps.
```
# In the first of your three screen sessions
# These are kept separate so that the flat file log output won't stomp the prior ones, for ingest validation
for folder in nitroba example-traffic ssh ftp radius rfb; do mkdir -p ~/brotmp/${folder}; done
wget https://www.bro.org/static/traces/exercise-traffic.pcap -O ~/brotmp/example-traffic/exercise-traffic.pcap
wget http://downloads.digitalcorpora.org/corpora/network-packet-dumps/2008-nitroba/nitroba.pcap -O ~/brotmp/nitroba/nitroba.pcap
wget https://www.bro.org/static/traces/ssh.pcap -O ~/brotmp/ssh/ssh.pcap
wget https://github.com/markofu/pcaps/blob/master/PracticalPacketAnalysis/ppa-capture-files/ftp.pcap?raw=true -O ~/brotmp/ftp/ftp.pcap
wget https://github.com/EmpowerSecurityAcademy/wireshark/blob/master/radius_localhost.pcapng?raw=true -O ~/brotmp/radius/radius_localhost.pcapng
wget https://github.com/kholia/my-pcaps/blob/master/VNC/07-vnc-openwall-3.7.pcap?raw=true -O ~/brotmp/rfb/rfb.pcap
cd ~/brotmp/example-traffic
bro -r exercise-traffic.pcap /usr/local/bro/share/bro/site/local.bro -C
cd ~/brotmp/nitroba
bro -r nitroba.pcap /usr/local/bro/share/bro/site/local.bro -C
cd ~/brotmp/ssh
bro -r ssh.pcap /usr/local/bro/share/bro/site/local.bro -C
cd ~/brotmp/ftp
bro -r ftp.pcap /usr/local/bro/share/bro/site/local.bro -C
cd ~/brotmp/radius
editcap -F libpcap radius_localhost.pcapng radius_localhost.pcap
bro -r radius_localhost.pcap /usr/local/bro/share/bro/site/local.bro -C
cd ~/brotmp/rfb
bro -r rfb.pcap /usr/local/bro/share/bro/site/local.bro -C
```
1. Validate that terminals 2 and 3 don't have any errors that you don't expect.
1. Verify proper indexing in ES and availability in kibana.
```
# Check around and make sure things look okay
declare -a exists notexists; for protocol in http dns conn dpd ftp files known_certs smtp ssl weird notice dhcp ssh software radius x509 known_devices stats capture_loss sip rfb; do if [[ $(curl -s -XGET "node1:9200/bro*/_search?q=protocol:${protocol}" | jq '.hits.hits') == '[]' ]]; then notexists+=" ${protocol}"; else exists+=" ${protocol}"; fi; done; if [ ${#notexists[@]} -ne 0 ]; then echo -e "\n\n\033[0mThe following do exist in ES: ${exists[@]}\n\033[0;31mThe following do NOT exist in ES: ${notexists[@]}\033[0m"; else echo 'All of the log types are in ES! Success!'; fi; unset exists notexists
# Check Kibana. For example: http://node1:5000/app/kibana#/visualize/create?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-15y,mode:quick,to:now))&_a=(filters:!(),linked:!f,query:(query_string:(analyze_wildcard:!t,query:'*')),uiState:(),vis:(aggs:!((id:'3',params:(field:protocol,order:desc,orderBy:'2',size:25),schema:segment,type:terms),(id:'2',params:(),schema:metric,type:count)),listeners:(),params:(addLegend:!t,addTimeMarker:!f,addTooltip:!t,defaultYExtents:!f,mode:stacked,scale:linear,setYExtents:!f,shareYAxis:!t,times:!(),yAxis:()),title:'',type:histogram))&indexPattern=bro*&type=histogram
# OPTIONAL testing
# Run `/usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head` and look around
curl -XGET node1:9200/_cat/indices # First column should be all green
curl -XGET "node1:9200/bro*/_count" # Check the count of entries in the bro index, you can re-run bro against specific PCAPs and watch this increase, etc.
```
1. Make sure the right bro version was installed.
```
if [[ "$(bro -v 2>&1)" == "bro version 2.5.2" ]]; then echo 'Success!'; else echo 'ERROR: Not running bro 2.5.2'; fi
```
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you to follow these guidelines and ask you to double check the following:
### For all changes:
- [X] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [X] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [X] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [X] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [X] Have you included steps or a guide to how the change may be verified and tested manually?
- [X] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [X] Have you written or updated unit tests and or integration tests to verify your changes?
- [X] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [X] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JonZeolla/metron METRON-1088
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/844.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #844
----
commit 36829ab14b34c9059a3e05adc89bf593ea9a5b90
Author: Jon Zeolla <ze...@gmail.com>
Date: 2017-11-16T17:26:23Z
METRON-1088: Upgrade bro to the latest release (step 1 of 2)
commit 04bcce2274d7165e3e64ce3b013ece407d3593f9
Author: Jon Zeolla <ze...@gmail.com>
Date: 2017-11-18T02:02:51Z
Merge branch 'master' of https://github.com/apache/metron into METRON-1088
commit ad6db1b9449c16a7f09debaa7e1d44b8106c749d
Author: Jon Zeolla <ze...@gmail.com>
Date: 2017-11-19T06:18:58Z
METRON-1088: Upgrade bro to the latest release (step 2 of 2)
----
> Upgrade bro to 2.5.2
> --------------------
>
> Key: METRON-1088
> URL: https://issues.apache.org/jira/browse/METRON-1088
> Project: Metron
> Issue Type: Improvement
> Reporter: Jon Zeolla
> Assignee: Jon Zeolla
>
> Upgrade bro from 2.4.2 to the latest release.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)