You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by "Neha Sinha (JIRA)" <ji...@apache.org> on 2016/09/21 04:48:20 UTC
[jira] [Commented] (METRON-440) DSL parse exception seen for Bro
Topology
[ https://issues.apache.org/jira/browse/METRON-440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15508735#comment-15508735 ]
Neha Sinha commented on METRON-440:
-----------------------------------
On further investigation, I found that the function DOMAIN_REMOVE_SUBDOMAINS fails to handle empty strings.I got the same exception as above for the below parser config :-
====================================================================
PARSER Config: bro
{
"parserClassName":"org.apache.metron.parsers.bro.BasicBroParser",
"sensorTopic":"bro",
"parserConfig": {},
"fieldTransformations" : [
{
"transformation" : "STELLAR"
,"output" : ["full_hostname", "domain_without_subdomains","is_alert", "new_field" ]
,"config" : {
"domain_without_subdomains" : "DOMAIN_REMOVE_SUBDOMAINS('')",
"is_alert" :"true",
"new_field" : "SPLIT(ip_dst_addr,'.')"
}
}
]
}
====================================================================
> DSL parse exception seen for Bro Topology
> -----------------------------------------
>
> Key: METRON-440
> URL: https://issues.apache.org/jira/browse/METRON-440
> Project: Metron
> Issue Type: Bug
> Affects Versions: 0.2.2BETA
> Reporter: Neha Sinha
>
> I updated the bro parser to the following in my environment and uploaded to zookeeper.
> Post that i am seeing dsl parse exception messages for Bro topology.
> Bro Parser
> =========================================
> PARSER Config: bro
> {
> "parserClassName":"org.apache.metron.parsers.bro.BasicBroParser",
> "sensorTopic":"bro",
> "parserConfig": {},
> "fieldTransformations" : [
> {
> "transformation" : "STELLAR"
> ,"output" : [ "full_hostname", "domain_without_subdomains", "is_alert" ]
> ,"config" : {
> "full_hostname" : "URL_TO_HOST(url)"
> ,"domain_without_subdomains" : "DOMAIN_REMOVE_SUBDOMAINS(full_hostname)"
> ,"is_alert" :"true"
> }
> }
> ]
> }
> ==================================================
> Bro logs
> ===================================================
> 2016-08-23 10:54:45.108 b.s.d.executor [ERROR]
> org.apache.metron.common.dsl.ParseException: Unable to pop an empty stack
> at org.apache.metron.common.stellar.StellarCompiler.popStack(StellarCompiler.java:397) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.StellarCompiler.exitTransformationFunc(StellarCompiler.java:250) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.StellarParser$TransformationFuncContext.exitRule(StellarParser.java:1634) ~[stormjar.jar:?]
> at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422) ~[stormjar.jar:?]
> at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.StellarParser.transformation(StellarParser.java:158) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:57) ~[stormjar.jar:?]
> at org.apache.metron.common.field.transformation.StellarTransformation.map(StellarTransformation.java:46) ~[stormjar.jar:?]
> at org.apache.metron.common.configuration.FieldTransformer.transform(FieldTransformer.java:111) ~[stormjar.jar:?]
> at org.apache.metron.common.configuration.FieldTransformer.transformAndUpdate(FieldTransformer.java:123) ~[stormjar.jar:?]
> at org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:116) [stormjar.jar:?]
> at backtype.storm.daemon.executor$fn__5492$tuple_action_fn__5494.invoke(executor.clj:684) [storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
> at backtype.storm.daemon.executor$mk_task_receiver$fn__5415.invoke(executor.clj:431) [storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
> at backtype.storm.disruptor$clojure_handler$reify__4991.onEvent(disruptor.clj:58) [storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
> at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) [storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
> at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) [storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
> at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) [storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
> at backtype.storm.daemon.executor$fn__5492$fn__5505$fn__5556.invoke(executor.clj:813) [storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
> at backtype.storm.util$async_loop$fn__644.invoke(util.clj:479) [storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]
> ===================================================
> Zookeeper Dump
> ===================================================
> [root@metron-test1-3 parsers]# /usr/metron/0.2.0BETA/bin/zk_load_configs.sh -z metron-test1-3.openstacklocal:2181 -m DUMP -i /usr/metron/0.2.0BETA/config/zookeeper/
> log4j:WARN No appenders could be found for logger (org.apache.curator.framework.imps.CuratorFrameworkImpl).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
> GLOBAL Config: global
> {
> "es.clustername": "metron",
> "es.ip": "metron-test1-10.openstacklocal",
> "es.port": "9300",
> "es.date.format": "yyyy.MM.dd.HH"
> }
> PARSER Config: bluecoat
> {
> "parserClassName":"org.apache.metron.parsers.bluecoat.BasicBluecoatParser",
> "sensorTopic":"bluecoat",
> "parserConfig": {}
> }
> PARSER Config: websphere
> {
> "parserClassName":"org.apache.metron.parsers.websphere.GrokWebSphereParser",
> "sensorTopic":"websphere",
> "parserConfig":
> {
> "grokPath":"/patterns/websphere",
> "patternLabel":"WEBSPHERE",
> "timestampField":"timestamp_string",
> "dateFormat":"yyyy MMM dd HH:mm:ss"
> }
> }
> PARSER Config: squid
> {
> "parserClassName": "org.apache.metron.parsers.GrokParser",
> "sensorTopic": "squid",
> "parserConfig": {
> "grokPath": "/patterns/squid",
> "patternLabel": "SQUID_DELIMITED",
> "timestampField": "timestamp"
> },
> "fieldTransformations" : [
> {
> "transformation" : "STELLAR"
> ,"output" : [ "full_hostname", "domain_without_subdomains" ]
> ,"config" : {
> "full_hostname" : "URL_TO_HOST(url)"
> ,"domain_without_subdomains" : "DOMAIN_REMOVE_SUBDOMAINS(full_hostname)"
> }
> }
> ]
> }
> PARSER Config: bro
> {
> "parserClassName":"org.apache.metron.parsers.bro.BasicBroParser",
> "sensorTopic":"bro",
> "parserConfig": {},
> "fieldTransformations" : [
> {
> "transformation" : "STELLAR"
> ,"output" : [ "full_hostname", "domain_without_subdomains", "is_alert" ]
> ,"config" : {
> "full_hostname" : "URL_TO_HOST(url)"
> ,"domain_without_subdomains" : "DOMAIN_REMOVE_SUBDOMAINS(full_hostname)"
> ,"is_alert" :"true"
> }
> }
> ]
> }
> PARSER Config: snort
> {
> "parserClassName":"org.apache.metron.parsers.snort.BasicSnortParser",
> "sensorTopic":"snort",
> "parserConfig": {}
> }
> PARSER Config: yaf
> {
> "parserClassName":"org.apache.metron.parsers.GrokParser",
> "sensorTopic":"yaf",
> "fieldTransformations" : [
> {
> "input" : "protocol"
> ,"transformation": "IP_PROTOCOL"
> }
> ],
> "parserConfig":
> {
> "grokPath":"/patterns/yaf",
> "patternLabel":"YAF_DELIMITED",
> "timestampField":"start_time",
> "timeFields": ["start_time", "end_time"],
> "dateFormat":"yyyy-MM-dd HH:mm:ss.S"
> }
> }
> ENRICHMENT Config: websphere
> {
> "index": "websphere",
> "batchSize": 5,
> "enrichment": {
> "fieldMap": {
> "geo": [
> "ip_src_addr"
> ],
> "host": [
> "ip_src_addr"
> ]
> },
> "fieldToTypeMap": {
> "ip_src_addr": [
> "playful_classification"
> ]
> }
> }
> }
> ENRICHMENT Config: bro
> {
> "index": "bro",
> "batchSize": 5,
> "enrichment" : {
> "fieldMap": {
> "geo": ["ip_dst_addr", "ip_src_addr"],
> "host": ["host"]
> }
> },
> "threatIntel": {
> "fieldMap": {
> "hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"]
> },
> "fieldToTypeMap": {
> "ip_src_addr" : ["malicious_ip"],
> "ip_dst_addr" : ["malicious_ip"]
> }
> }
> }
> ENRICHMENT Config: snort
> {
> "index": "snort",
> "batchSize": 1,
> "enrichment" : {
> "fieldMap":
> {
> "geo": ["ip_dst_addr", "ip_src_addr"],
> "host": ["host"]
> }
> },
> "threatIntel" : {
> "fieldMap":
> {
> "hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"]
> },
> "fieldToTypeMap":
> {
> "ip_src_addr" : ["malicious_ip"],
> "ip_dst_addr" : ["malicious_ip"]
> },
> "triageConfig" : {
> "riskLevelRules" : {
> "not(IN_SUBNET(ip_dst_addr, '192.168.0.0/24'))" : 10
> },
> "aggregator" : "MAX"
> }
> }
> }
> ENRICHMENT Config: yaf
> {
> "index": "yaf",
> "batchSize": 5,
> "enrichment" : {
> "fieldMap":
> {
> "geo": ["ip_dst_addr", "ip_src_addr"],
> "host": ["host"]
> }
> },
> "threatIntel": {
> "fieldMap":
> {
> "hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"]
> },
> "fieldToTypeMap":
> {
> "ip_src_addr" : ["malicious_ip"],
> "ip_dst_addr" : ["malicious_ip"]
> }
> }
> }
> ===================================================
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)