You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by "Andre (JIRA)" <ji...@apache.org> on 2016/11/02 01:52:59 UTC
[jira] [Commented] (METRON-157) Create CEF Parser
[ https://issues.apache.org/jira/browse/METRON-157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15627347#comment-15627347 ]
Andre commented on METRON-157:
------------------------------
[~kylerichardson] no worries.
I thought it would be helpful to other projects so I crafted it outside NiFi. It works but if you find any bug (CEF is such a lovely standard...) let me know and I will try o fix.
Also note ParCEFone is now available on maven central[1] and the NIFI processor has also been merged to master (should be out of the box on 1.1.x).
[1] - https://mvnrepository.com/artifact/com.fluenda/ParCEFone/1.2.0
> Create CEF Parser
> -----------------
>
> Key: METRON-157
> URL: https://issues.apache.org/jira/browse/METRON-157
> Project: Metron
> Issue Type: New Feature
> Reporter: Domenic Puzio
> Priority: Minor
> Labels: platform
> Fix For: 0.2.2BETA
>
>
> Create a parser for CEF (Common Event Format). CEF is a very common formatting for security data sources; it is used by FireEye, Adallom, Imperva WAF, CyberArk, and others. The parser should be flexible enough to work for any of these data sources. CEF uses shorthand field names, so field names should be changed to human-readable and Metron-friendly equivalents. CEF custom labels (cs1Label, flexString1Label, etc.) should be converted appropriately.
> Below are sample messages and their expected parsed output.
> Adallom CEF
> 2016-04-01T09:29:11.356-0400 CEF:0|Adallom|Adallom|1.0|56fe779ee4b0459f4e9a484a|ALERT_CABINET_EVENT_MATCH_AUDIT|0|msg=Activity policy 'User download/view file' was triggered by 'scolbert@gmail.com' suser=wanderson@rock.com start=1459517280810 end=1459517280810 audits=["AVPR-4oIPeFmuZ3CKKrg","AVPR-wx80cd9PUpAu2aj","AVPR-6XGPeFmuZ3CKKvx","AVPSALn_qE4Kgs_8_yK9","AVPSASW3gw_f3aEvgEmi"] services=["APPID_SXC"] users=["lvader@hotmail.com"] cs6=https://abcd-remote.console.arc.com/#/alerts/56fe779ee4b0459f4e9a484a cs6Label=consoleUrl
> ...
> {"source.type":"adallom","device_version":"1.0","severity":"0","device_product":"Adallom","services":"[\"APPID_SXC\"]","src_username":"wanderson@rock.com","message":"Activity policy 'User download\/view file' was triggered by 'scolbert@gmail.com'","users":"[\"lvader@hotmail.com\"]","consoleUrl":"https:\/\/abcd-remote.console.arc.com\/#\/alerts\/56fe779ee4b0459f4e9a484a","event_class_id":"56fe779ee4b0459f4e9a484a","original_string":"2016-04-01T09:29:11.356-0400 CEF:0|Adallom|Adallom|1.0|56fe779ee4b0459f4e9a484a|ALERT_CABINET_EVENT_MATCH_AUDIT|0|msg=Activity policy 'User download\/view file' was triggered by 'scolbert@gmail.com' suser=wanderson@rock.com start=1459517280810 end=1459517280810 audits=[\"AVPR-4oIPeFmuZ3CKKrg\",\"AVPR-wx80cd9PUpAu2aj\",\"AVPR-6XGPeFmuZ3CKKvx\",\"AVPSALn_qE4Kgs_8_yK9\",\"AVPSASW3gw_f3aEvgEmi\"] services=[\"APPID_SXC\"] users=[\"lvader@hotmail.com\"] cs6=https:\/\/abcd-remote.console.arc.com\/#\/alerts\/56fe779ee4b0459f4e9a484a cs6Label=consoleUrl","header":"2016-04-01T09:29:11.356-0400 CEF:0","event_name":"ALERT_CABINET_EVENT_MATCH_AUDIT","startTime":"1459517280810","device_vendor":"Adallom","endTime":"1459517280810","audits":"[\"AVPR-4oIPeFmuZ3CKKrg\",\"AVPR-wx80cd9PUpAu2aj\",\"AVPR-6XGPeFmuZ3CKKvx\",\"AVPSALn_qE4Kgs_8_yK9\",\"AVPSASW3gw_f3aEvgEmi\"]","timestamp":1459502951000}
> CyberArk CEF
> Mar 21 14:05:02 HHHPVATN1 CEF:0|Cyber-Ark|Vault|7.20.0091|295|Retrieve password|5|act=Retrieve password suser=spilgrim fname=Root\ABC phobos3 - COMP dvc=120.99.70.3 shost=10.44.134.78 dhost= duser= externalId= app= reason= cs1Label="Affected User Name" cs1= cs2Label="Safe Name" cs2=Security Vulnerability Mgmt cs3Label="Device Type" cs3= cs4Label="Database" cs4= cs5Label="Other info" cs5=101.198.70.93 cn1Label="Request Id" cn1= cn2Label="Ticket Id" cn2=Needed to verify config files being pulled msg=Needed to verify config files being pulled
> ...
> {"timestamp":1458569102000,"source.type":"cyberark","device_version":"7.20.0091","device_product":"Vault","fileName":"Root\\ABC phobos3 - COMP","src_username":"spilgrim","\"Other info\"":"101.198.70.93","\"Ticket Id\"":"Needed to verify config files being pulled ","deviceAddress":"120.99.70.3","severity":"5","deviceAction":"Retrieve password","message":"Needed to verify config files being pulled","event_class_id":"295","original_string":"Mar 21 14:05:02 HHHPVATN1 CEF:0|Cyber-Ark|Vault|7.20.0091|295|Retrieve password|5|act=Retrieve password suser=spilgrim fname=Root\\ABC phobos3 - COMP dvc=120.99.70.3 shost=10.44.134.78 dhost= duser= externalId= app= reason= cs1Label=\"Affected User Name\" cs1= cs2Label=\"Safe Name\" cs2=Security Vulnerability Mgmt cs3Label=\"Device Type\" cs3= cs4Label=\"Database\" cs4= cs5Label=\"Other info\" cs5=101.198.70.93 cn1Label=\"Request Id\" cn1= cn2Label=\"Ticket Id\" cn2=Needed to verify config files being pulled msg=Needed to verify config files being pulled","\"Safe Name\"":"Security Vulnerability Mgmt","header":"Mar 21 14:05:02 HHHPVATN1 CEF:0","event_name":"Retrieve password","device_vendor":"Cyber-Ark","src_hostname":"10.44.134.78"}
> WAF CEF
> <14>CEF:0|Imperva Inc.|SecureSphere|10.0.0.4_16|ABC - Secure Login.vm Page Rate Limit UK - Source IP||High|act=alert dst=17.43.200.42 dpt=88 duser=${Alert.username} src=10.31.45.69 spt=34435 proto=TCP rt=31 March 2016 13:04:55 cat=Alert cs1= cs1Label=Policy cs2=ABC-Secure cs2Label=ServerGroup cs3=servers_svc cs3Label=ServiceName cs4=server_app cs4Label=ApplicationName cs5=QA cs5Label=Description
> ...
> {"source.type":"waf","device_version":"10.0.0.4_16","severity":"High","device_product":"SecureSphere","ServerGroup":"ABC-Secure","ApplicationName":"server_app","Description":"QA","deviceAction":"alert","ip_dst_port":"88","dst_username":"${Alert.username}","priority":"14","deviceEventCategory":"Alert","protocol":"TCP","ip_dst_addr":"17.43.200.42","ip_src_port":"34435","event_class_id":"ABC - Secure Login.vm Page Rate Limit UK - Source IP","ServiceName":"servers_svc","original_string":"<14>CEF:0|Imperva Inc.|SecureSphere|10.0.0.4_16|ABC - Secure Login.vm Page Rate Limit UK - Source IP||High|act=alert dst=17.43.200.42 dpt=88 duser=${Alert.username} src=10.31.45.69 spt=34435 proto=TCP rt=31 March 2016 13:04:55 cat=Alert cs1= cs1Label=Policy cs2=ABC-Secure cs2Label=ServerGroup cs3=servers_svc cs3Label=ServiceName cs4=server_app cs4Label=ApplicationName cs5=QA cs5Label=Description","header":"<14>CEF:0","device_vendor":"Imperva Inc.","ip_src_addr":"10.31.45.69","timestamp":1459429495000}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)