You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by "Nick Allen (JIRA)" <ji...@apache.org> on 2017/08/22 18:11:00 UTC
[jira] [Updated] (METRON-1120) Profile's 'groupBy' Expression Has
No Reference to Time
[ https://issues.apache.org/jira/browse/METRON-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nick Allen updated METRON-1120:
-------------------------------
Fix Version/s: 0.4.1
> Profile's 'groupBy' Expression Has No Reference to Time
> -------------------------------------------------------
>
> Key: METRON-1120
> URL: https://issues.apache.org/jira/browse/METRON-1120
> Project: Metron
> Issue Type: Bug
> Reporter: Nick Allen
> Assignee: Nick Allen
> Fix For: 0.4.1
>
>
> It is often the case that patterns and behaviors will differ based on calendar effects like day of week. For example, activity on a weekday can be very different from a weekend. The Profiler's "Group By" functionality is one way to account for calendar effects.
> This profile definition operates over any incoming telemetry that has an `ip_src_addr` and a `timestamp` field. It produces a profile that segments the data by day of week. It does by using a 'groupBy' expression to extract the day of week from the telemetry's `timestamp` field.
> {code}
> {
> "profiles": [
> {
> "profile": "calender-effects",
> "onlyif": "exists(ip_src_addr) and exists(timestamp)",
> "foreach": "ip_src_addr",
> "init": { "count": 0 },
> "update": { "count": "count + 1" },
> "result": "count",
> "groupBy": ["DAY_OF_WEEK(TO_EPOCH_TIMESTAMP(timestamp, 'yyyy-MM-dd HH:mm:ss', 'GMT'))"]
> }
> ]
> }
> {code}
> When retrieving profile data using the Profiler Client API, I only want to retrieve data from the same day of week to account for any calendar effects. The following example retrieves profile data only for Thursdays over the past 60 days.
> {code}
> >>> thursday := 5
> >>> PROFILE_GET("calendar-effects", "10.0.0.1", PROFILE_FIXED(60, "DAYS"), [thursday])
> {code}
> h3. The Problem
> The `groupBy` expression only has access to the Profile's `result` value. It does not have any way to reference the current tick time in the Profiler. Here is an example showing the problem.
> Define the profile and a message.
> {code}
> [Stellar]>>> conf
> {
> "profiles": [
> {
> "profile": "calender-effects",
> "onlyif": "exists(ip_src_addr) and exists(timestamp)",
> "foreach": "ip_src_addr",
> "init": { "count": "0" },
> "update": { "count": "count + 1" },
> "result": "count",
> "groupBy": ["DAY_OF_WEEK(TO_EPOCH_TIMESTAMP(timestamp, 'yyyy-MM-dd HH:mm:ss', 'GMT'))"]
> }
> ]
> }
> [Stellar]>>> msg
> {
> "ip_src_addr": "10.0.0.1",
> "protocol": "HTTPS",
> "length": "10",
> "bytes_in": 234,
> "timestamp": "2017-08-17 09:00:00"
> }
> {code}
> Initialize the Profiler and apply the message a few times.
> {code}
> [Stellar]>>> p := PROFILER_INIT(conf)
> [Stellar]>>> PROFILER_APPLY(msg, p)
> org.apache.metron.profiler.StandAloneProfiler@9472c85
> [Stellar]>>> PROFILER_APPLY(msg, p)
> org.apache.metron.profiler.StandAloneProfiler@9472c85
> [Stellar]>>> PROFILER_APPLY(msg, p)
> org.apache.metron.profiler.StandAloneProfiler@9472c85
> {code}
> Flush the profile, which will trigger execution of the `groupBy` expression.
> {code}
> [Stellar]>>> PROFILER_FLUSH(p)
> [!] Bad 'groupBy' expression: Unexpected type: expected=Object, actual=null, expression=DAY_OF_WEEK(TO_EPOCH_TIMESTAMP(timestamp, 'yyyy-MM-dd HH:mm:ss', 'GMT')), profile=calender-effects, entity=10.0.0.1
> org.apache.metron.stellar.dsl.ParseException: Bad 'groupBy' expression: Unexpected type: expected=Object, actual=null, expression=DAY_OF_WEEK(TO_EPOCH_TIMESTAMP(timestamp, 'yyyy-MM-dd HH:mm:ss', 'GMT')), profile=calender-effects, entity=10.0.0.1
> at org.apache.metron.profiler.DefaultProfileBuilder.execute(DefaultProfileBuilder.java:257)
> at org.apache.metron.profiler.DefaultProfileBuilder.flush(DefaultProfileBuilder.java:159)
> at org.apache.metron.profiler.DefaultMessageDistributor.lambda$flush$0(DefaultMessageDistributor.java:101)
> at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
> at org.apache.metron.profiler.DefaultMessageDistributor.flush(DefaultMessageDistributor.java:99)
> at org.apache.metron.profiler.StandAloneProfiler.flush(StandAloneProfiler.java:82)
> at org.apache.metron.profiler.client.stellar.ProfilerFunctions$ProfilerFlush.apply(ProfilerFunctions.java:191)
> at org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:556)
> at org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:160)
> at org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:152)
> at org.apache.metron.stellar.common.shell.StellarExecutor.execute(StellarExecutor.java:287)
> at org.apache.metron.stellar.common.shell.StellarShell.handleStellar(StellarShell.java:270)
> at org.apache.metron.stellar.common.shell.StellarShell.execute(StellarShell.java:409)
> at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalArgumentException: Unexpected type: expected=Object, actual=null, expression=DAY_OF_WEEK(TO_EPOCH_TIMESTAMP(timestamp, 'yyyy-MM-dd HH:mm:ss', 'GMT'))
> at org.apache.metron.stellar.common.DefaultStellarStatefulExecutor.execute(DefaultStellarStatefulExecutor.java:128)
> at org.apache.metron.profiler.DefaultProfileBuilder.lambda$execute$3(DefaultProfileBuilder.java:253)
> at java.util.ArrayList.forEach(ArrayList.java:1249)
> at org.apache.metron.profiler.DefaultProfileBuilder.execute(DefaultProfileBuilder.java:253)
> ... 16 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)