You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by basvdl <gi...@git.apache.org> on 2017/04/14 19:10:41 UTC
[GitHub] incubator-metron pull request #531: Metron 854 create dhcp dump parser
GitHub user basvdl opened a pull request:
https://github.com/apache/incubator-metron/pull/531
Metron 854 create dhcp dump parser
I've written a DHCPDump parser for Metron. I would like to submit my current version and receive feedback to get it finalized.
The code includes basic Unit and Integration test and covers most of the fields within the DHCPDump log events.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/basvdl/incubator-metron METRON-854_Create_DHCPDump_Parser
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/531.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #531
----
commit a37def86b3cccfc0d4c958e6fe3ef26627dabec5
Author: Bas van de Lustgraaf <ba...@lustgraaf.nl>
Date: 2017-04-14T17:43:50Z
METRON-854 Create DHCPDump Parser Initial commit contains Parser and Test
commit 96208e0c6de831f659c0a1b7f7ede6558c3cc984
Author: Bas van de Lustgraaf <ba...@lustgraaf.nl>
Date: 2017-04-14T18:01:34Z
METRON-854 Create DHCPDump Parser Add Integration test
commit d100ba73e37845b1474ab0b0905379a765f9c4cf
Author: Bas van de Lustgraaf <ba...@lustgraaf.nl>
Date: 2017-04-14T18:20:32Z
METRON-854 Create DHCPDump Parser Fixed ArrayToString conversion error during build
commit 6e23510bd44638b8f446c7ccdb78f7aefa254d0a
Author: Bas van de Lustgraaf <ba...@lustgraaf.nl>
Date: 2017-04-14T18:52:18Z
METRON-854 Create DHCPDump Parser Fixed test class by replacing @BeforeClass
commit e0f0b90472f9b19fdb1bb81fa025feb800870698
Author: Bas van de Lustgraaf <ba...@lustgraaf.nl>
Date: 2017-04-14T19:02:22Z
METRON-854 Create DHCPDump Parser Added guid key to example parsed data to pass integration test
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #531: METRON-854 create dhcp dump parser
Posted by basvdl <gi...@git.apache.org>.
Github user basvdl commented on the issue:
https://github.com/apache/incubator-metron/pull/531
I agree that using the original format is the preferred. Are we able to ship and parse the original multi line format and put the separate lines back together before or during the Metron parsing stage?
To give you an idea of the output format of the original DHCPDump available on Ubuntu (https://launchpad.net/ubuntu/+source/dhcpdump/1.8-2).
[dhcpdump.log.txt](https://github.com/apache/incubator-metron/files/955083/dhcpdump.log.txt)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #531: METRON-854 create dhcp dump parser
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/531
@basvdl I should have first said, thanks for all your hard work!
You probably have already thought through many of these options, so please educate me on their pros/cons. I am going to lay out all of the possibilities that I can think of just so that we don't leave any stone unturned.
(1) Alter the Source of Telemetry - This is the option that you chose. This has the downside that we now have a forked, external dependency that we are relying on you to maintain going forward. That's a lot of work for you!
(2) Use an Alternative Source of Telemetry - Is there an alternative to `DHCPDump` out there that we could use instead? I think this would be the simplest approach assuming one exists.
(3) Reunite lines at the parser - Each line would come in as a separate Kafka message. We cannot rely on ordering of those messages, so we would need some kind of unique identifier contained within each message to then reunite them all into a single DHCPDump record. This would be very different from our current architecture and is not currently supported.
(4) Transport Mechanism - How do you envision getting the DHCPDump records into Kafka? aka the transport mechanism? No matter what, we need some kind of transport mechanism. You could potentially create a solution that takes the separate record lines, stitches them into a single line, then pushes them to Kafka as a complete DHCPDump record.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #531: METRON-854 create dhcp dump parser
Posted by simonellistonball <gi...@git.apache.org>.
Github user simonellistonball commented on the issue:
https://github.com/apache/incubator-metron/pull/531
The Bro parsers is actually pretty generic, and will take whatever json bro dumps out. From a quick inspection you should just need to configure the bro instance to send out dhcp, and in theory it should just work. (I've not tested that though)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #531: METRON-854 create dhcp dump parser
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on the issue:
https://github.com/apache/incubator-metron/pull/531
I would love to see Metron have a solution for both approaches - ingesting DHCP server logs, as well as DHCP observations based on network traffic. Like @ottobackwards mentioned, not everyone can get the right infrastructure/viewpoint on their network to run something like Bro and get the DHCP traffic to their sensors to be processed.
I have definitely sent more than just DNS and HTTP from Bro to Metron and it has been properly ingested, but to date I haven't done DHCP. Like @simonellistonball and @nickwallen mentioned, both the parser and the kafka plugin are setup to handle new bro logs quite well, and a while back I worked on updating Metron's support for more Bro sources via [METRON-508](https://github.com/JonZeolla/incubator-metron/commit/736cc39525f9f08f6e781faea2610e893327e74c). I just never had a chance to test it, so I haven't yet opened a PR.
Once #545 and #547 get merged into master, and I'm able to finish [METRON-813](https://issues.apache.org/jira/browse/METRON-813), I would be happy to work on anything related to Bro and DHCP logs at scale, including finishing up METRON-508. I have two hardware bro environments and my larger one currently sees about 7 million DHCP observations/day and sends ~30,000 messages per second into Metron.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #531: METRON-854 create dhcp dump parser
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/531
> As an alternative method for getting DHCP data out of pcap, you might consider the existing Bro sensor, which essentially does what dhcpdump does...
The current Bro parser only supports HTTP and DNS records coming out of Bro. Bro does a lot more indeed, but we don't have parsers for all of that yet.
I think enhancing the existing Bro parser to support DHCP records would be an excellent approach. The Bro plugin that we have can be configured to land any of the record types in Kafka, so that would not be a problem.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #531: METRON-854 create dhcp dump parser
Posted by simonellistonball <gi...@git.apache.org>.
Github user simonellistonball commented on the issue:
https://github.com/apache/incubator-metron/pull/531
As an alternative method for getting DHCP data out of pcap, you might consider the existing Bro sensor, which essentially does what dhcpdump does, but for a wider range of protocols, in a more sophisticated way. We also already have a built in parser.
That said it would great to have this parser too for people not looking for the full range of bro.
The multi-line aspect may not be an issue. The boundary for Metron is the Kafka message, not really the line, so if you can split the log into multi-line chunks prior to kafka, potentially with something like NiFi based on a delimiter. The way to do this is to use nifi to insert a true delimiter (not end of line) and then use the SplitContent to send individual log entries via kafka. It's a little heavy, but solves the multi-line problem as long as you're not going to crazy levels of throughput e.g. hundreds of thousands of EPS.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #531: Metron 854 create dhcp dump parser
Posted by basvdl <gi...@git.apache.org>.
Github user basvdl commented on the issue:
https://github.com/apache/incubator-metron/pull/531
Please note that the original DHCPDump format is multi line (http://www.mavetju.org/unix/dhcpdump-man.php). Since this is inconvenient to collect, ship and parse we have modified the DHCPDump to generate single line output. The compatible version of DHCPDump for this parser, is available through my github (https://github.com/basvdl/dhcpdump).
This note is also enclosed as comment in the code itself.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #531: METRON-854 create dhcp dump parser
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/531
We also have a `JSONMapParser` that was contributed after the original Bro parser. The data coming out of the Bro plugin can be configured to be JSON. That's how we typically use it anyways. Can we just use the `JSONMapParser` to parse Bro records?
I cannot think of any reason it wouldn't work off the top of my head, but I am not sure. Then you wouldn't even have to write any code at all.
And also with PR #545 we can now send different Bro record types to their own Kafka topics. I have included an example in the README on how you can do that. I only mention that if that becomes a problem for the `JSONMapParser` and what you are trying to achieve.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #531: Metron 854 create dhcp dump parser
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/531
Can you edit the title to start with METRON-854? If it doesn't the scripts won't work with jira
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #531: METRON-854 create dhcp dump parser
Posted by basvdl <gi...@git.apache.org>.
Github user basvdl commented on the issue:
https://github.com/apache/incubator-metron/pull/531
Thnx for the heads-up. METRON-777 is a great improvement!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #531: METRON-854 create dhcp dump parser
Posted by basvdl <gi...@git.apache.org>.
Github user basvdl commented on the issue:
https://github.com/apache/incubator-metron/pull/531
@nickwallen, these are indeed the options we have discussed...
> I am going to lay out all of the possibilities that I can think of just so that we don't leave any stone unturned.
(1) Alter the Source of Telemetry - ...
(2) Use an Alternative Source of Telemetry - ...
(3) Reunite lines at the parser - ...
(4) Transport Mechanism - ...
1. Alter the Source of Telemetry - I agree with you that this is the least preferred method.
2. Use an Alternative Source of Telemetry - The alternative I've looked into was `tcpdump`, but this is less detailed.
3. Reunite lines at the parser - This will not give you a reliable solution, mainly due to the reason you have given: 'We cannot rely on ordering of the messages'
4. Transport Mechanism - In our case we are shipping the log using (Mi)NiFi. We could look into a custom NiFi processor.
Another option that just came as a brainwave, maybe we can develop a kind of yaf / yafscii solution. Where you pipe the output of DHCPDump into the stdin of a `DHCPDumpToSingleLine` which will stitch the lines together and output single line events to disk.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #531: METRON-854 create dhcp dump parser
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/531
> Since this is inconvenient to collect, ship and parse we have modified the DHCPDump to generate single line output. The compatible version of DHCPDump for this parser, is available through my github (https://github.com/basvdl/dhcpdump).
Is there no way to handle the original, multi-line format? I think we are limiting the usefulness of this by relying on a modified version of the original source program.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #531: Metron 854 create dhcp dump parser
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/531
Please keep an eye on METRON-777. If that hits first, then I'll help you re-do your parser as a parser extension
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #531: METRON-854 create dhcp dump parser
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/531
unless of course someone can't use bro for some reason
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---