[jira] [Commented] (METRON-2284) Metron Profiler for Spark doesn't work as expected

["Michael Miklavcic (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/METRON-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16954000#comment-16954000 ] 

Michael Miklavcic commented on METRON-2284:
-------------------------------------------

Can we have you try the following exercise from the REPL using your profile?

https://github.com/apache/metron/blob/master/metron-analytics/metron-profiler-repl/README.md#getting-started


{code:java}
[Stellar]>>> val := SET_INIT()
[]
[Stellar]>>> devicehostname := 'windows9.something.com'
windows9.something.com
[Stellar]>>> val := SET_ADD(val, IS_EMPTY(devicehostname))
[false]
[Stellar]>>> conf := SHELL_EDIT()
# add the following profile contents in the vi editor that comes up:
{
 "profiles":[
   {
     "profile":"batchtest5",
     "onlyif":"devicehostname == 'windows9.something.com'",
     "foreach":"devicehostname",
     "init":{
       "val":"SET_INIT()"
     },
     "update":{
       "val":"SET_ADD(val, IS_EMPTY(devicehostname))"
     },
    "result":{
       "profile":"val"
    }
   }
 ],
 "timestampField":"timestamp"
}
[Stellar]>>> profiler := PROFILER_INIT(conf)
Profiler{1 profile(s), 0 messages(s), 0 route(s)}
[Stellar]>>> msg := SHELL_EDIT()
# add this record
{"devicehostname": "windows9.something.com", "timestamp": 1567241981000}
[Stellar]>>> PROFILER_APPLY(msg, profiler)
Profiler{1 profile(s), 1 messages(s), 1 route(s)}
[Stellar]>>> values := PROFILER_FLUSH(profiler)
[{period={duration=900000, period=1741379, start=1567241100000, end=1567242000000}, profile=batchtest5, groups=[], value=[false], entity=windows9.something.com}]
{code}


I'm seeing "value" set to false, as expected, at least from the REPL. Let's see if we can verify that part of the functionality matches up as expected and go from there.

> Metron Profiler for Spark doesn't work as expected
> --------------------------------------------------
>
>                 Key: METRON-2284
>                 URL: https://issues.apache.org/jira/browse/METRON-2284
>             Project: Metron
>          Issue Type: Bug
>    Affects Versions: 0.7.1
>            Reporter: Maxim Dashenko
>            Priority: Major
>
> Used command:
> {code}
> /usr/hdp/current/spark2-client/bin/spark-submit --class org.apache.metron.profiler.spark.cli.BatchProfilerCLI --properties-file /usr/hcp/current/metron/config/batch-profiler.properties ~/metron-profiler-spark-0.7.1.1.9.1.0-6.jar --config /usr/hcp/current/metron/config/batch-profiler.properties --profiles ~/profiler.json
> {code}
>  cat /usr/hcp/current/metron/config/batch-profiler.properties
> {code}
> profiler.batch.input.path=/tmp/test_data.logs
> profiler.batch.input.format=json
> profiler.period.duration=15
> profiler.period.duration.units=MINUTES
> {code}
>  
> cat ~/profiler.json
> {code}
> {
>  "profiles":[
>    {
>      "profile":"batchtest5",
>      "onlyif":"source.type == 'testsource' and devicehostname == 'windows9.something.com'",
>      "foreach":"devicehostname",
>      "init":{
>        "val":"SET_INIT()"
>      },
>      "update":{
>        "val":"SET_ADD(val, IS_EMPTY(devicehostname))"
>      },
>     "result":{
>        "profile":"val"
>     }
>    }
>  ],
>  "timestampField":"timestamp"
> }
> {code}
>  cat test_data.logs
> {code}
> {"devicehostname": "windows9.something.com", "timestamp": 1567241981000, "source.type": "testsource"}
> {code}
> Stellar statement
> {code}
> PROFILE_GET('batchtest5', 'windows9.something.com', PROFILE_FIXED(100, 'DAYS'))
> {code}
> Returns:
> {code}
> [[true]]
> {code}
> Expected result:
> {code}
> [[false]]
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)