[GitHub] incubator-metron issue #449: METRON-701 Triage Metrics Produced by the Profi...

[cestella <gi...@git.apache.org>]

Github user cestella commented on the issue:

    https://github.com/apache/incubator-metron/pull/449
  
    So, I think this is an interesting approach.  My issues with it are:
    * You seem to be sending every profile into kafka, not just the configured ones.
    * You seem to be assuming that one value only is being sent into the telemetry and it's the value that you store in HBase
    
    I'd recommend, rather, that you make the `result` field more complex by making it a map where the key is the source (e.g. "hbase" or "kafka").  This allows you to separate the storage structure by storage medium.  You may, for instance, want to *STORE* a stats object in Hbase, but only send along the mean and standard deviation.  Also, I'd recommend allowing `result`to be either a string (which would presume only hbase is supported) or a Map, which would explicitly specify the structure for just the sources you want to write to.
    
    Here's a worked example config for maximum clarity (!):
    ```
    {
      "profiles": [
        {
          "profile": "test",
          "foreach": "'global'",
          "onlyif": "source.type == 'squid'",
          "init":    { "stats": "STATS_INIT()" },
          "update":  { "stats": "STATS_ADD(stats, LENGTH(url))" },
          "result":  {
             "hbase" : "stats",
             "kafka" : "{ 'mean' : STATS_MEAN(stats), 'stddev' : STATS_SD(stats) }"
           }
          
        }
      ]
    }
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---