[jira] [Created] (METRON-371) Errors seen in enrichment bolts for squid logs

["Anand Subramanian (JIRA)" <ji...@apache.org>]

Anand Subramanian created METRON-371:
----------------------------------------

             Summary: Errors seen in enrichment bolts for squid logs
                 Key: METRON-371
                 URL: https://issues.apache.org/jira/browse/METRON-371
             Project: Metron
          Issue Type: Improvement
    Affects Versions: 0.3.0BETA
         Environment: 12 node setup created on openstack running build as of Aug 8th. See git log snippet below:

{code}
[root@metron-test-13 metron-deployment]# git log
commit b9282b438422d56fac23301dc854a39ae7d83a83
Author: mmiklavc <mi...@gmail.com>
Date:   Mon Aug 8 15:25:20 2016 -0400

    METRON-356 Modify Storm topology.classpath via configuration (mmiklavc via cestella) closes apache/incubator-metron#204
<snip>
{code}
            Reporter: Anand Subramanian
            Priority: Minor


When I ran a test for the squid proxy sensor, I could see the following errors being thrown in the enrichment kafkaspout log file. 

{code}
2016-08-11 09:07:26.629 o.a.m.e.b.EnrichmentSplitterBolt [ERROR] Unable to retrieve a sensor enrichment config of squid
2016-08-11 09:07:26.630 o.a.m.e.b.EnrichmentJoinBolt [ERROR] Unable to retrieve a sensor enrichment config of squid
2016-08-11 09:07:26.631 o.a.m.e.b.EnrichmentSplitterBolt [ERROR] Unable to retrieve sensor config: squid
2016-08-11 09:07:26.631 o.a.m.e.b.ThreatIntelJoinBolt [ERROR] Unable to retrieve sensor config: squid
{code}

*Testing Steps*
1) Ensure squid topology is up.
2) Inject the following message to the kafka-producer to ingest 

{code}
"1461576382.642    161 127.0.0.1 TCP_MISS/200 103701 GET http://www.abc.com/ - DIRECT/199.27.79.73 text/html"
{code}

3) Wait for the enrichment and index to be generated. 
4) Review the enrichment kafkaspout log file and the error can be seen. 

After discussing with [~dlyle], this error is apparently due to the missing enrichments for squid (see attached zkconfig.txt). If the squid enrichment were added manually, then the error messages are not seen. 

Also that for some of the sensors (squid, in this case), it might be normal to not enrich some types of data.

Now, this message showing up as ERROR is not representative of the above statement where we do not want to enrich some fields, on purpose. WARNNG or INFO might be a more appropriate way to log these messages. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)