You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by nickwallen <gi...@git.apache.org> on 2016/09/10 02:32:50 UTC
[GitHub] incubator-metron pull request #246: METRON-411 Support Greater Range of Prof...
GitHub user nickwallen opened a pull request:
https://github.com/apache/incubator-metron/pull/246
METRON-411 Support Greater Range of Profile Periods
Depends on existing pull requests #236 #237 and #242 . The diff might look a little wonky until those other PRs are merged.
### Problem
The current Profiler's deterministic row key design limits the range of profile period durations available to a user. The original goal was to support 15 minute periods which it does, along with 5, 20, 30 and 60 minute periods. Different types of profiles and input data will require a range of periods and the Profiler should be flexible in this regard.
Currently, a profile period must start on each and every hour. While 1, 2, 3, 4, 6, or 240 periods per hour is acceptable, 5 periods per hour is not. The number of periods per hour must be either a multiple or divisor of 60. Also, a period cannot be any longer than an hour. In some cases 2, 4, or 8 hour periods might be useful.
### Solution
This change uses a much simpler method of calculating row keys and allows the user to set the specific duration of each profile period. Based on this duration, each period starting at the epoch, is assigned a monotonically increasing value which is used as part of the row key. Periods can span multiple hours and there is no arbitrary durations that are unacceptable, like 5 periods per hour.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/incubator-metron METRON-411
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/246.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #246
----
commit 444bfb41fc872194c1fbbf9820248f85c5066c51
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-08-23T17:03:15Z
METRON-389 Create Java API to Read Profile Data During Model Scoring
commit 3bb55c63f70d1493df0dbd70fd548ec8b6ce3030
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-08-30T01:23:03Z
METRON-389 Missed removal of old tests during merge
commit 71b50c293ed896d948bc63cbc40d370bdc2b003a
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-08-30T01:31:55Z
METRON-389 Change to Context was lost in merge
commit b9e017c1101fb3a67435d3462234870bf0ed9717
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-08-30T13:23:56Z
METRON-389 Was missing changes from METRON-377
commit 1ddeb24087729692ec502a4108bf04b655c235cb
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-08-30T13:27:36Z
METRON-389 Removed unnecessary commented-out code
commit 4af9b196bc4012162bc5cc3758a09c50321feccd
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-08-30T13:46:10Z
METRON-389 Made sure the docs are consistent; periodsPerHour can be a multiple or divisor of 60.
commit ee4c5145536c4e97360381cd8b8a7bf6bd65db1b
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-08-30T14:01:20Z
METRON-389 Was missing updated test from METRON-377
commit 155a08715939ad99892ec18c884f6124495c9434
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-08-30T16:42:04Z
METRON-399 Stellar Date Functions Should Default to Current Time
commit ed4700074e57e0cea701530befabd658d28be64e
Author: cstella <ce...@gmail.com>
Date: 2016-08-26T17:21:13Z
METRON-396: Make Stellar function resolution happen via an annotation and classpath search
commit 7b471c94ac82863f02ed081999855072349167dc
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-09-02T18:09:36Z
METRON-391 Refactor so HBaseProfileClient does not refer to static fields which causes unnecessary dependency
commit 1135c4261c7a8e5e5b74a7b88b33e982e4eb2ebf
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-09-02T18:10:56Z
METRON-391 Create Stellar Function to Read Profile Data for Model Scoring
commit b19b9dcffed7a608d092bb289a8867ae9810977a
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-09-02T19:51:26Z
METRON-404 Stellar Compiler Hides Function Initialization Errors
commit f75ce385cf707756913478c4b7cfb47c4e3753ea
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-09-06T17:04:03Z
METRON-391 Improved examples
commit 75f996757dd9b9730a1dd563d28ae3d97aec57a6
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-09-06T19:37:21Z
METRON-391 Rename of ProfileWriter
commit 40893167a00fda6ceb8b3ca665d85ffbaaf1f83a
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-09-06T20:03:52Z
Merge 'apache/master' into METRON-391
commit 008f6f23b9b1b8b997a4ce1d9f1e92302a975ce5
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-09-06T20:19:47Z
Still need to call initialize which was lost in the merge
commit 34382290cd8aeb5eb7ba3abab42319ddf0f5644f
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-09-07T19:03:28Z
Merge remote-tracking branch 'apache/master' into METRON-391
commit 97599d69e703829ea823625b0a79a408c3e861ce
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-09-07T19:30:02Z
METRON-391 Removed unnecessary code from test
commit 38a3619b5f4a84219b03ed99316d7e423f2acbcf
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-09-07T19:47:37Z
METRON-391 Renamed to reset() and added docs
commit 9a7d67f66ef1c57aa516ee0281019864f2b3a5b9
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-09-08T12:21:01Z
METRON-391 Fixed README from bad merge
commit fa14f2b0747e6c802673e909a7b90d48726c18ab
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-09-09T13:39:44Z
METRON-391 Improved docs
commit d677c6a539e7a113e12a78272a1ce282d03284dc
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-09-10T02:10:03Z
METRON-411 Support Greater Range of Profile Periods
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #246: METRON-411 Support Greater Range of Profile Per...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/246
There may be some dependencies we need to change to make the Stellar client functions visible out-of-the-box in the Enrichment topology. This PR doesn't touch the Profiler client side of things though. That seems unrelated in my mind. Might make sense to work that separately.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #246: METRON-411 Support Greater Range of Profile Per...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/246
Created METRON-420 for that. Will take a run at it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #246: METRON-411 Support Greater Range of Profile Per...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/246
Could you please walk us through a test plan of how this feature should be exercised in quickdev?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #246: METRON-411 Support Greater Range of Prof...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/246
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #246: METRON-411 Support Greater Range of Profile Per...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/246
Reconciled with master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #246: METRON-411 Support Greater Range of Profile Per...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/246
That sounds fair. +1 as long as in that follow-on PR gets created and prioritized.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #246: METRON-411 Support Greater Range of Profile Per...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/246
Create a profile using the profiler just as you would before. Instead of defining the number of 'periods per hour' just define the duration of each period '20 minutes' or '4 hours'. The behavior of the profiler should be exactly the same.
All of the unit and integration tests have been updated to reflect this change and should pass. The integration test itself uses a '5 second' profile period.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #246: METRON-411 Support Greater Range of Profile Per...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/246
I'm specifically interested in making sure that you can call the stellar functions around the profiler from within the enrichment topology on vagrant, actually. I may be missing it, but I don't see a dependency from the enrichment project to the profiler client.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #246: METRON-411 Support Greater Range of Profile Per...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/246
Rewrote history to match master and #242 on which this is dependent.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #246: METRON-411 Support Greater Range of Profile Per...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/246
Cool, looks great. Much cleaner. :)
+1 if you tested in vagrant already. If not, then I'll hold the +1 til you let me know.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---