You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by nickwallen <gi...@git.apache.org> on 2016/09/06 17:20:09 UTC
[GitHub] incubator-metron pull request #242: METRON-391 Create Stellar Function to Re...
GitHub user nickwallen opened a pull request:
https://github.com/apache/incubator-metron/pull/242
METRON-391 Create Stellar Function to Read Profile Data for Model Scoring
### [METRON-391](https://issues.apache.org/jira/browse/METRON-391)
A Java API was created as part of #236 to allow Profile data to be read during model scoring. A Stellar function(s) was created that uses this API to read the profile data. This would be used in conjunction with other Stellar functions like MODEL_APPLY.
#### Dependencies
This PR depends on #236, #237, #233, and #240 .
#### Examples
Retrieve all values for 'entity1' from 'profile1' over the past 4 hours.
```
PROFILE_GET('profile1', 'entity1', 4, 'HOURS')
```
Retrieve all values for 'entity1' from 'profile1' over the past 2 days.
```
PROFILE_GET('profile1', 'entity1', 2, 'DAYS')
```
Retrieve all values for 'entity1' from 'profile1' that occurred on 'weekdays' over the past month.
```
PROFILE_GET('profile1', 'entity1', 1, 'MONTHS', 'weekdays')
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/incubator-metron METRON-391
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/242.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #242
----
commit 444bfb41fc872194c1fbbf9820248f85c5066c51
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-08-23T17:03:15Z
METRON-389 Create Java API to Read Profile Data During Model Scoring
commit 3bb55c63f70d1493df0dbd70fd548ec8b6ce3030
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-08-30T01:23:03Z
METRON-389 Missed removal of old tests during merge
commit 71b50c293ed896d948bc63cbc40d370bdc2b003a
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-08-30T01:31:55Z
METRON-389 Change to Context was lost in merge
commit b9e017c1101fb3a67435d3462234870bf0ed9717
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-08-30T13:23:56Z
METRON-389 Was missing changes from METRON-377
commit 1ddeb24087729692ec502a4108bf04b655c235cb
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-08-30T13:27:36Z
METRON-389 Removed unnecessary commented-out code
commit 4af9b196bc4012162bc5cc3758a09c50321feccd
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-08-30T13:46:10Z
METRON-389 Made sure the docs are consistent; periodsPerHour can be a multiple or divisor of 60.
commit ee4c5145536c4e97360381cd8b8a7bf6bd65db1b
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-08-30T14:01:20Z
METRON-389 Was missing updated test from METRON-377
commit 155a08715939ad99892ec18c884f6124495c9434
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-08-30T16:42:04Z
METRON-399 Stellar Date Functions Should Default to Current Time
commit ed4700074e57e0cea701530befabd658d28be64e
Author: cstella <ce...@gmail.com>
Date: 2016-08-26T17:21:13Z
METRON-396: Make Stellar function resolution happen via an annotation and classpath search
commit 7b471c94ac82863f02ed081999855072349167dc
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-09-02T18:09:36Z
METRON-391 Refactor so HBaseProfileClient does not refer to static fields which causes unnecessary dependency
commit 1135c4261c7a8e5e5b74a7b88b33e982e4eb2ebf
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-09-02T18:10:56Z
METRON-391 Create Stellar Function to Read Profile Data for Model Scoring
commit b19b9dcffed7a608d092bb289a8867ae9810977a
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-09-02T19:51:26Z
METRON-404 Stellar Compiler Hides Function Initialization Errors
commit f75ce385cf707756913478c4b7cfb47c4e3753ea
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-09-06T17:04:03Z
METRON-391 Improved examples
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #242: METRON-391 Create Stellar Function to Re...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/242#discussion_r78423347
--- Diff: metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/GetProfile.java ---
@@ -0,0 +1,179 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.client.stellar;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.hadoop.hbase.client.HTableInterface;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.utils.ConversionUtils;
+import org.apache.metron.profiler.client.HBaseProfilerClient;
+import org.apache.metron.profiler.client.ProfilerClient;
+import org.apache.metron.profiler.hbase.ColumnBuilder;
+import org.apache.metron.profiler.hbase.RowKeyBuilder;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static java.lang.String.format;
+import static org.apache.metron.common.dsl.Context.Capabilities.PROFILER_COLUMN_BUILDER;
+import static org.apache.metron.common.dsl.Context.Capabilities.PROFILER_HBASE_TABLE;
+import static org.apache.metron.common.dsl.Context.Capabilities.PROFILER_ROW_KEY_BUILDER;
+
+/**
+ * A Stellar function that can retrieve data contained within a Profile.
+ *
+ * PROFILE_GET
+ *
+ * Retrieve all values for 'entity1' from 'profile1' over the past 4 hours.
+ *
+ * <code>PROFILE_GET('profile1', 'entity1', 4, 'HOURS')</code>
+ *
+ * Retrieve all values for 'entity1' from 'profile1' over the past 2 days.
+ *
+ * <code>PROFILE_GET('profile1', 'entity1', 2, 'DAYS')</code>
+ *
+ * Retrieve all values for 'entity1' from 'profile1' that occurred on 'weekdays' over the past month.
+ *
+ * <code>PROFILE_GET('profile1', 'entity1', 1, 'MONTHS', 'weekdays')</code>
+ *
+ */
+@Stellar(
+ namespace="PROFILE",
+ name="GET",
+ description="Retrieves a series of values from a stored profile.",
+ params={
+ "profile - The name of the profile.",
+ "entity - The name of the entity.",
+ "durationAgo - How long ago should values be retrieved from?",
+ "units - The units of 'durationAgo'.",
+ "groups - Optional - The groups used to sort the profile."
+ },
+ returns="The profile measurements."
+)
+public class GetProfile implements StellarFunction {
+
+ private ProfilerClient client;
+
+ /**
+ * Initialization.
+ */
+ @Override
+ public void initialize(Context context) {
+
+ Context.Capabilities[] required = { PROFILER_HBASE_TABLE, PROFILER_ROW_KEY_BUILDER, PROFILER_COLUMN_BUILDER };
--- End diff --
Not sure what you mean. Can you point me to some code?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #242: METRON-391 Create Stellar Function to Read Prof...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/242
What I mean is, add some docs describing when it is appropriate to use a Capability versus the global configuration. Having multiple ways of configuring things is a little icky. I'm sure there is a good reason why, but I have no idea because there are no comments/docs in the code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #242: METRON-391 Create Stellar Function to Read Prof...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/242
No, the new capabilities are not defined in those topologies. I wasn't sure how we wanted to handle that.
It seems a little icky to define these capabilities in our existing topologies. When we continue to add new Stellar functions, are we also going to always add the necessary Capabilities in the topology initialization? This gets messy quick.
Or If a user adds a Stellar function in their own JAR, how are they going to use it if it requires additional Capabilities in the Context which is hard-coded in Metron?
I feel like we need to externalize this somehow. The current mechanism might not scale too well. What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #242: METRON-391 Create Stellar Function to Read Prof...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/242
@nickwallen Yeah, that's why I don't want to treat these as capabilities, but rather pull them from the global configs. The function should configure itself from the global config, IMO. That way the only capability that we need for config is `GLOBAL_CONFIG` and maybe `SENSOR_CONFIG`. I added and used the `GLOBAL_CONFIG` capability in my PR for the HBase enrichment functions [here](https://github.com/apache/incubator-metron/pull/234)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #242: METRON-391 Create Stellar Function to Re...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/242#discussion_r78433698
--- Diff: metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/GetProfile.java ---
@@ -0,0 +1,179 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.client.stellar;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.hadoop.hbase.client.HTableInterface;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.utils.ConversionUtils;
+import org.apache.metron.profiler.client.HBaseProfilerClient;
+import org.apache.metron.profiler.client.ProfilerClient;
+import org.apache.metron.profiler.hbase.ColumnBuilder;
+import org.apache.metron.profiler.hbase.RowKeyBuilder;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static java.lang.String.format;
+import static org.apache.metron.common.dsl.Context.Capabilities.PROFILER_COLUMN_BUILDER;
+import static org.apache.metron.common.dsl.Context.Capabilities.PROFILER_HBASE_TABLE;
+import static org.apache.metron.common.dsl.Context.Capabilities.PROFILER_ROW_KEY_BUILDER;
+
+/**
+ * A Stellar function that can retrieve data contained within a Profile.
+ *
+ * PROFILE_GET
+ *
+ * Retrieve all values for 'entity1' from 'profile1' over the past 4 hours.
+ *
+ * <code>PROFILE_GET('profile1', 'entity1', 4, 'HOURS')</code>
+ *
+ * Retrieve all values for 'entity1' from 'profile1' over the past 2 days.
+ *
+ * <code>PROFILE_GET('profile1', 'entity1', 2, 'DAYS')</code>
+ *
+ * Retrieve all values for 'entity1' from 'profile1' that occurred on 'weekdays' over the past month.
+ *
+ * <code>PROFILE_GET('profile1', 'entity1', 1, 'MONTHS', 'weekdays')</code>
+ *
+ */
+@Stellar(
+ namespace="PROFILE",
+ name="GET",
+ description="Retrieves a series of values from a stored profile.",
+ params={
+ "profile - The name of the profile.",
+ "entity - The name of the entity.",
+ "durationAgo - How long ago should values be retrieved from?",
+ "units - The units of 'durationAgo'.",
+ "groups - Optional - The groups used to sort the profile."
+ },
+ returns="The profile measurements."
+)
+public class GetProfile implements StellarFunction {
+
+ private ProfilerClient client;
+
+ /**
+ * Initialization.
+ */
+ @Override
+ public void initialize(Context context) {
+
+ Context.Capabilities[] required = { PROFILER_HBASE_TABLE, PROFILER_ROW_KEY_BUILDER, PROFILER_COLUMN_BUILDER };
+ validateContext(context, required);
+
+ HTableInterface table = (HTableInterface) context.getCapability(PROFILER_HBASE_TABLE).get();
+ RowKeyBuilder rowKeyBuilder = (RowKeyBuilder) context.getCapability(PROFILER_ROW_KEY_BUILDER).get();
+ ColumnBuilder columnBuilder = (ColumnBuilder) context.getCapability(PROFILER_COLUMN_BUILDER).get();
+
+ client = new HBaseProfilerClient(table, rowKeyBuilder, columnBuilder);
+ }
+
+ /**
+ * Is the function initialized?
+ */
+ @Override
+ public boolean isInitialized() {
+ return client != null;
+ }
+
+ /**
+ * Apply the function.
+ * @param args The function arguments.
+ * @param context
+ */
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+
+ String profile = getArg(0, String.class, args);
+ String entity = getArg(1, String.class, args);
+ long durationAgo = getArg(2, Long.class, args);
+ String unitsName = getArg(3, String.class, args);
+ TimeUnit units = TimeUnit.valueOf(unitsName);
+ List<Object> groups = getGroupsArg(4, args);
+
+ return client.fetch(profile, entity, durationAgo, units, Integer.class, groups);
--- End diff --
Yes, exactly. Specifying the type of numbers you expect to get back from the profile.
Could accept a string argument, "integer", "double", etc, and then manually convert to the correct class; Integer.class, Double.class. But I really don't want to do it that way.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #242: METRON-391 Create Stellar Function to Re...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/242#discussion_r78433046
--- Diff: metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/GetProfile.java ---
@@ -0,0 +1,179 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.client.stellar;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.hadoop.hbase.client.HTableInterface;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.utils.ConversionUtils;
+import org.apache.metron.profiler.client.HBaseProfilerClient;
+import org.apache.metron.profiler.client.ProfilerClient;
+import org.apache.metron.profiler.hbase.ColumnBuilder;
+import org.apache.metron.profiler.hbase.RowKeyBuilder;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static java.lang.String.format;
+import static org.apache.metron.common.dsl.Context.Capabilities.PROFILER_COLUMN_BUILDER;
+import static org.apache.metron.common.dsl.Context.Capabilities.PROFILER_HBASE_TABLE;
+import static org.apache.metron.common.dsl.Context.Capabilities.PROFILER_ROW_KEY_BUILDER;
+
+/**
+ * A Stellar function that can retrieve data contained within a Profile.
+ *
+ * PROFILE_GET
+ *
+ * Retrieve all values for 'entity1' from 'profile1' over the past 4 hours.
+ *
+ * <code>PROFILE_GET('profile1', 'entity1', 4, 'HOURS')</code>
+ *
+ * Retrieve all values for 'entity1' from 'profile1' over the past 2 days.
+ *
+ * <code>PROFILE_GET('profile1', 'entity1', 2, 'DAYS')</code>
+ *
+ * Retrieve all values for 'entity1' from 'profile1' that occurred on 'weekdays' over the past month.
+ *
+ * <code>PROFILE_GET('profile1', 'entity1', 1, 'MONTHS', 'weekdays')</code>
+ *
+ */
+@Stellar(
+ namespace="PROFILE",
+ name="GET",
+ description="Retrieves a series of values from a stored profile.",
+ params={
+ "profile - The name of the profile.",
+ "entity - The name of the entity.",
+ "durationAgo - How long ago should values be retrieved from?",
+ "units - The units of 'durationAgo'.",
+ "groups - Optional - The groups used to sort the profile."
+ },
+ returns="The profile measurements."
+)
+public class GetProfile implements StellarFunction {
+
+ private ProfilerClient client;
+
+ /**
+ * Initialization.
+ */
+ @Override
+ public void initialize(Context context) {
+
+ Context.Capabilities[] required = { PROFILER_HBASE_TABLE, PROFILER_ROW_KEY_BUILDER, PROFILER_COLUMN_BUILDER };
+ validateContext(context, required);
+
+ HTableInterface table = (HTableInterface) context.getCapability(PROFILER_HBASE_TABLE).get();
+ RowKeyBuilder rowKeyBuilder = (RowKeyBuilder) context.getCapability(PROFILER_ROW_KEY_BUILDER).get();
+ ColumnBuilder columnBuilder = (ColumnBuilder) context.getCapability(PROFILER_COLUMN_BUILDER).get();
+
+ client = new HBaseProfilerClient(table, rowKeyBuilder, columnBuilder);
+ }
+
+ /**
+ * Is the function initialized?
+ */
+ @Override
+ public boolean isInitialized() {
+ return client != null;
+ }
+
+ /**
+ * Apply the function.
+ * @param args The function arguments.
+ * @param context
+ */
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+
+ String profile = getArg(0, String.class, args);
+ String entity = getArg(1, String.class, args);
+ long durationAgo = getArg(2, Long.class, args);
+ String unitsName = getArg(3, String.class, args);
+ TimeUnit units = TimeUnit.valueOf(unitsName);
+ List<Object> groups = getGroupsArg(4, args);
+
+ return client.fetch(profile, entity, durationAgo, units, Integer.class, groups);
--- End diff --
What is hard coded as an integer? The argument index (e.g. `getArg(i, ...)`)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #242: METRON-391 Create Stellar Function to Re...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/242#discussion_r78433288
--- Diff: metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/GetProfile.java ---
@@ -0,0 +1,179 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.client.stellar;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.hadoop.hbase.client.HTableInterface;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.utils.ConversionUtils;
+import org.apache.metron.profiler.client.HBaseProfilerClient;
+import org.apache.metron.profiler.client.ProfilerClient;
+import org.apache.metron.profiler.hbase.ColumnBuilder;
+import org.apache.metron.profiler.hbase.RowKeyBuilder;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static java.lang.String.format;
+import static org.apache.metron.common.dsl.Context.Capabilities.PROFILER_COLUMN_BUILDER;
+import static org.apache.metron.common.dsl.Context.Capabilities.PROFILER_HBASE_TABLE;
+import static org.apache.metron.common.dsl.Context.Capabilities.PROFILER_ROW_KEY_BUILDER;
+
+/**
+ * A Stellar function that can retrieve data contained within a Profile.
+ *
+ * PROFILE_GET
+ *
+ * Retrieve all values for 'entity1' from 'profile1' over the past 4 hours.
+ *
+ * <code>PROFILE_GET('profile1', 'entity1', 4, 'HOURS')</code>
+ *
+ * Retrieve all values for 'entity1' from 'profile1' over the past 2 days.
+ *
+ * <code>PROFILE_GET('profile1', 'entity1', 2, 'DAYS')</code>
+ *
+ * Retrieve all values for 'entity1' from 'profile1' that occurred on 'weekdays' over the past month.
+ *
+ * <code>PROFILE_GET('profile1', 'entity1', 1, 'MONTHS', 'weekdays')</code>
+ *
+ */
+@Stellar(
+ namespace="PROFILE",
+ name="GET",
+ description="Retrieves a series of values from a stored profile.",
+ params={
+ "profile - The name of the profile.",
+ "entity - The name of the entity.",
+ "durationAgo - How long ago should values be retrieved from?",
+ "units - The units of 'durationAgo'.",
+ "groups - Optional - The groups used to sort the profile."
+ },
+ returns="The profile measurements."
+)
+public class GetProfile implements StellarFunction {
+
+ private ProfilerClient client;
+
+ /**
+ * Initialization.
+ */
+ @Override
+ public void initialize(Context context) {
+
+ Context.Capabilities[] required = { PROFILER_HBASE_TABLE, PROFILER_ROW_KEY_BUILDER, PROFILER_COLUMN_BUILDER };
+ validateContext(context, required);
+
+ HTableInterface table = (HTableInterface) context.getCapability(PROFILER_HBASE_TABLE).get();
+ RowKeyBuilder rowKeyBuilder = (RowKeyBuilder) context.getCapability(PROFILER_ROW_KEY_BUILDER).get();
+ ColumnBuilder columnBuilder = (ColumnBuilder) context.getCapability(PROFILER_COLUMN_BUILDER).get();
+
+ client = new HBaseProfilerClient(table, rowKeyBuilder, columnBuilder);
+ }
+
+ /**
+ * Is the function initialized?
+ */
+ @Override
+ public boolean isInitialized() {
+ return client != null;
+ }
+
+ /**
+ * Apply the function.
+ * @param args The function arguments.
+ * @param context
+ */
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+
+ String profile = getArg(0, String.class, args);
+ String entity = getArg(1, String.class, args);
+ long durationAgo = getArg(2, Long.class, args);
+ String unitsName = getArg(3, String.class, args);
+ TimeUnit units = TimeUnit.valueOf(unitsName);
+ List<Object> groups = getGroupsArg(4, args);
+
+ return client.fetch(profile, entity, durationAgo, units, Integer.class, groups);
--- End diff --
nvm, you mean the return for the client.fetch, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #242: METRON-391 Create Stellar Function to Re...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/242#discussion_r78432614
--- Diff: metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/GetProfile.java ---
@@ -0,0 +1,179 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.client.stellar;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.hadoop.hbase.client.HTableInterface;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.utils.ConversionUtils;
+import org.apache.metron.profiler.client.HBaseProfilerClient;
+import org.apache.metron.profiler.client.ProfilerClient;
+import org.apache.metron.profiler.hbase.ColumnBuilder;
+import org.apache.metron.profiler.hbase.RowKeyBuilder;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static java.lang.String.format;
+import static org.apache.metron.common.dsl.Context.Capabilities.PROFILER_COLUMN_BUILDER;
+import static org.apache.metron.common.dsl.Context.Capabilities.PROFILER_HBASE_TABLE;
+import static org.apache.metron.common.dsl.Context.Capabilities.PROFILER_ROW_KEY_BUILDER;
+
+/**
+ * A Stellar function that can retrieve data contained within a Profile.
+ *
+ * PROFILE_GET
+ *
+ * Retrieve all values for 'entity1' from 'profile1' over the past 4 hours.
+ *
+ * <code>PROFILE_GET('profile1', 'entity1', 4, 'HOURS')</code>
+ *
+ * Retrieve all values for 'entity1' from 'profile1' over the past 2 days.
+ *
+ * <code>PROFILE_GET('profile1', 'entity1', 2, 'DAYS')</code>
+ *
+ * Retrieve all values for 'entity1' from 'profile1' that occurred on 'weekdays' over the past month.
+ *
+ * <code>PROFILE_GET('profile1', 'entity1', 1, 'MONTHS', 'weekdays')</code>
+ *
+ */
+@Stellar(
+ namespace="PROFILE",
+ name="GET",
+ description="Retrieves a series of values from a stored profile.",
+ params={
+ "profile - The name of the profile.",
+ "entity - The name of the entity.",
+ "durationAgo - How long ago should values be retrieved from?",
+ "units - The units of 'durationAgo'.",
+ "groups - Optional - The groups used to sort the profile."
+ },
+ returns="The profile measurements."
+)
+public class GetProfile implements StellarFunction {
+
+ private ProfilerClient client;
+
+ /**
+ * Initialization.
+ */
+ @Override
+ public void initialize(Context context) {
+
+ Context.Capabilities[] required = { PROFILER_HBASE_TABLE, PROFILER_ROW_KEY_BUILDER, PROFILER_COLUMN_BUILDER };
+ validateContext(context, required);
+
+ HTableInterface table = (HTableInterface) context.getCapability(PROFILER_HBASE_TABLE).get();
+ RowKeyBuilder rowKeyBuilder = (RowKeyBuilder) context.getCapability(PROFILER_ROW_KEY_BUILDER).get();
+ ColumnBuilder columnBuilder = (ColumnBuilder) context.getCapability(PROFILER_COLUMN_BUILDER).get();
+
+ client = new HBaseProfilerClient(table, rowKeyBuilder, columnBuilder);
+ }
+
+ /**
+ * Is the function initialized?
+ */
+ @Override
+ public boolean isInitialized() {
+ return client != null;
+ }
+
+ /**
+ * Apply the function.
+ * @param args The function arguments.
+ * @param context
+ */
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+
+ String profile = getArg(0, String.class, args);
+ String entity = getArg(1, String.class, args);
+ long durationAgo = getArg(2, Long.class, args);
+ String unitsName = getArg(3, String.class, args);
+ TimeUnit units = TimeUnit.valueOf(unitsName);
+ List<Object> groups = getGroupsArg(4, args);
+
+ return client.fetch(profile, entity, durationAgo, units, Integer.class, groups);
--- End diff --
I did not mean to leave this hard-coded as an integer. Trying to think of a good way to specify this easily from Stellar. I am not sure how yet. Hmm.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #242: METRON-391 Create Stellar Function to Read Prof...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/242
Rewrite the commit history to match master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #242: METRON-391 Create Stellar Function to Re...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/242#discussion_r78424348
--- Diff: metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/GetProfile.java ---
@@ -0,0 +1,179 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.client.stellar;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.hadoop.hbase.client.HTableInterface;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.utils.ConversionUtils;
+import org.apache.metron.profiler.client.HBaseProfilerClient;
+import org.apache.metron.profiler.client.ProfilerClient;
+import org.apache.metron.profiler.hbase.ColumnBuilder;
+import org.apache.metron.profiler.hbase.RowKeyBuilder;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static java.lang.String.format;
+import static org.apache.metron.common.dsl.Context.Capabilities.PROFILER_COLUMN_BUILDER;
+import static org.apache.metron.common.dsl.Context.Capabilities.PROFILER_HBASE_TABLE;
+import static org.apache.metron.common.dsl.Context.Capabilities.PROFILER_ROW_KEY_BUILDER;
+
+/**
+ * A Stellar function that can retrieve data contained within a Profile.
+ *
+ * PROFILE_GET
+ *
+ * Retrieve all values for 'entity1' from 'profile1' over the past 4 hours.
+ *
+ * <code>PROFILE_GET('profile1', 'entity1', 4, 'HOURS')</code>
+ *
+ * Retrieve all values for 'entity1' from 'profile1' over the past 2 days.
+ *
+ * <code>PROFILE_GET('profile1', 'entity1', 2, 'DAYS')</code>
+ *
+ * Retrieve all values for 'entity1' from 'profile1' that occurred on 'weekdays' over the past month.
+ *
+ * <code>PROFILE_GET('profile1', 'entity1', 1, 'MONTHS', 'weekdays')</code>
+ *
+ */
+@Stellar(
+ namespace="PROFILE",
+ name="GET",
+ description="Retrieves a series of values from a stored profile.",
+ params={
+ "profile - The name of the profile.",
+ "entity - The name of the entity.",
+ "durationAgo - How long ago should values be retrieved from?",
+ "units - The units of 'durationAgo'.",
+ "groups - Optional - The groups used to sort the profile."
+ },
+ returns="The profile measurements."
+)
+public class GetProfile implements StellarFunction {
+
+ private ProfilerClient client;
+
+ /**
+ * Initialization.
+ */
+ @Override
+ public void initialize(Context context) {
+
+ Context.Capabilities[] required = { PROFILER_HBASE_TABLE, PROFILER_ROW_KEY_BUILDER, PROFILER_COLUMN_BUILDER };
--- End diff --
also, I will note that it tees us up to add support for reinitialization when configs change. That way the functions can adjust their configs at runtime based on a zookeeper update.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #242: METRON-391 Create Stellar Function to Re...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/242#discussion_r78421803
--- Diff: metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/GetProfile.java ---
@@ -0,0 +1,179 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.client.stellar;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.hadoop.hbase.client.HTableInterface;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.utils.ConversionUtils;
+import org.apache.metron.profiler.client.HBaseProfilerClient;
+import org.apache.metron.profiler.client.ProfilerClient;
+import org.apache.metron.profiler.hbase.ColumnBuilder;
+import org.apache.metron.profiler.hbase.RowKeyBuilder;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static java.lang.String.format;
+import static org.apache.metron.common.dsl.Context.Capabilities.PROFILER_COLUMN_BUILDER;
+import static org.apache.metron.common.dsl.Context.Capabilities.PROFILER_HBASE_TABLE;
+import static org.apache.metron.common.dsl.Context.Capabilities.PROFILER_ROW_KEY_BUILDER;
+
+/**
+ * A Stellar function that can retrieve data contained within a Profile.
+ *
+ * PROFILE_GET
+ *
+ * Retrieve all values for 'entity1' from 'profile1' over the past 4 hours.
+ *
+ * <code>PROFILE_GET('profile1', 'entity1', 4, 'HOURS')</code>
+ *
+ * Retrieve all values for 'entity1' from 'profile1' over the past 2 days.
+ *
+ * <code>PROFILE_GET('profile1', 'entity1', 2, 'DAYS')</code>
+ *
+ * Retrieve all values for 'entity1' from 'profile1' that occurred on 'weekdays' over the past month.
+ *
+ * <code>PROFILE_GET('profile1', 'entity1', 1, 'MONTHS', 'weekdays')</code>
+ *
+ */
+@Stellar(
+ namespace="PROFILE",
+ name="GET",
+ description="Retrieves a series of values from a stored profile.",
+ params={
+ "profile - The name of the profile.",
+ "entity - The name of the entity.",
+ "durationAgo - How long ago should values be retrieved from?",
+ "units - The units of 'durationAgo'.",
+ "groups - Optional - The groups used to sort the profile."
+ },
+ returns="The profile measurements."
+)
+public class GetProfile implements StellarFunction {
+
+ private ProfilerClient client;
+
+ /**
+ * Initialization.
+ */
+ @Override
+ public void initialize(Context context) {
+
+ Context.Capabilities[] required = { PROFILER_HBASE_TABLE, PROFILER_ROW_KEY_BUILDER, PROFILER_COLUMN_BUILDER };
--- End diff --
If these capabilities are not specified, please pull them from the global config as backup.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #242: METRON-391 Create Stellar Function to Re...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/242
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #242: METRON-391 Create Stellar Function to Re...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/242#discussion_r78434213
--- Diff: metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/GetProfile.java ---
@@ -0,0 +1,179 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.client.stellar;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.hadoop.hbase.client.HTableInterface;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.utils.ConversionUtils;
+import org.apache.metron.profiler.client.HBaseProfilerClient;
+import org.apache.metron.profiler.client.ProfilerClient;
+import org.apache.metron.profiler.hbase.ColumnBuilder;
+import org.apache.metron.profiler.hbase.RowKeyBuilder;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static java.lang.String.format;
+import static org.apache.metron.common.dsl.Context.Capabilities.PROFILER_COLUMN_BUILDER;
+import static org.apache.metron.common.dsl.Context.Capabilities.PROFILER_HBASE_TABLE;
+import static org.apache.metron.common.dsl.Context.Capabilities.PROFILER_ROW_KEY_BUILDER;
+
+/**
+ * A Stellar function that can retrieve data contained within a Profile.
+ *
+ * PROFILE_GET
+ *
+ * Retrieve all values for 'entity1' from 'profile1' over the past 4 hours.
+ *
+ * <code>PROFILE_GET('profile1', 'entity1', 4, 'HOURS')</code>
+ *
+ * Retrieve all values for 'entity1' from 'profile1' over the past 2 days.
+ *
+ * <code>PROFILE_GET('profile1', 'entity1', 2, 'DAYS')</code>
+ *
+ * Retrieve all values for 'entity1' from 'profile1' that occurred on 'weekdays' over the past month.
+ *
+ * <code>PROFILE_GET('profile1', 'entity1', 1, 'MONTHS', 'weekdays')</code>
+ *
+ */
+@Stellar(
+ namespace="PROFILE",
+ name="GET",
+ description="Retrieves a series of values from a stored profile.",
+ params={
+ "profile - The name of the profile.",
+ "entity - The name of the entity.",
+ "durationAgo - How long ago should values be retrieved from?",
+ "units - The units of 'durationAgo'.",
+ "groups - Optional - The groups used to sort the profile."
+ },
+ returns="The profile measurements."
+)
+public class GetProfile implements StellarFunction {
+
+ private ProfilerClient client;
+
+ /**
+ * Initialization.
+ */
+ @Override
+ public void initialize(Context context) {
+
+ Context.Capabilities[] required = { PROFILER_HBASE_TABLE, PROFILER_ROW_KEY_BUILDER, PROFILER_COLUMN_BUILDER };
+ validateContext(context, required);
+
+ HTableInterface table = (HTableInterface) context.getCapability(PROFILER_HBASE_TABLE).get();
+ RowKeyBuilder rowKeyBuilder = (RowKeyBuilder) context.getCapability(PROFILER_ROW_KEY_BUILDER).get();
+ ColumnBuilder columnBuilder = (ColumnBuilder) context.getCapability(PROFILER_COLUMN_BUILDER).get();
+
+ client = new HBaseProfilerClient(table, rowKeyBuilder, columnBuilder);
+ }
+
+ /**
+ * Is the function initialized?
+ */
+ @Override
+ public boolean isInitialized() {
+ return client != null;
+ }
+
+ /**
+ * Apply the function.
+ * @param args The function arguments.
+ * @param context
+ */
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+
+ String profile = getArg(0, String.class, args);
+ String entity = getArg(1, String.class, args);
+ long durationAgo = getArg(2, Long.class, args);
+ String unitsName = getArg(3, String.class, args);
+ TimeUnit units = TimeUnit.valueOf(unitsName);
+ List<Object> groups = getGroupsArg(4, args);
+
+ return client.fetch(profile, entity, durationAgo, units, Integer.class, groups);
--- End diff --
Now that I think about it, I'm not sure why you're assuming that the result is always a Number? You could pass back a bloom filter from the profiler just as easily or a biased sample or whatever, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #242: METRON-391 Create Stellar Function to Read Prof...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/242
It appears that these functions require a number of capabilities (profiler hbase table, etc) that are not setup by default in the parser or enrichment topology. Have you tested calling this function from a field transformation in the parser or as part of a triage rule?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #242: METRON-391 Create Stellar Function to Re...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/242#discussion_r78424147
--- Diff: metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/GetProfile.java ---
@@ -0,0 +1,179 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.client.stellar;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.hadoop.hbase.client.HTableInterface;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.utils.ConversionUtils;
+import org.apache.metron.profiler.client.HBaseProfilerClient;
+import org.apache.metron.profiler.client.ProfilerClient;
+import org.apache.metron.profiler.hbase.ColumnBuilder;
+import org.apache.metron.profiler.hbase.RowKeyBuilder;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static java.lang.String.format;
+import static org.apache.metron.common.dsl.Context.Capabilities.PROFILER_COLUMN_BUILDER;
+import static org.apache.metron.common.dsl.Context.Capabilities.PROFILER_HBASE_TABLE;
+import static org.apache.metron.common.dsl.Context.Capabilities.PROFILER_ROW_KEY_BUILDER;
+
+/**
+ * A Stellar function that can retrieve data contained within a Profile.
+ *
+ * PROFILE_GET
+ *
+ * Retrieve all values for 'entity1' from 'profile1' over the past 4 hours.
+ *
+ * <code>PROFILE_GET('profile1', 'entity1', 4, 'HOURS')</code>
+ *
+ * Retrieve all values for 'entity1' from 'profile1' over the past 2 days.
+ *
+ * <code>PROFILE_GET('profile1', 'entity1', 2, 'DAYS')</code>
+ *
+ * Retrieve all values for 'entity1' from 'profile1' that occurred on 'weekdays' over the past month.
+ *
+ * <code>PROFILE_GET('profile1', 'entity1', 1, 'MONTHS', 'weekdays')</code>
+ *
+ */
+@Stellar(
+ namespace="PROFILE",
+ name="GET",
+ description="Retrieves a series of values from a stored profile.",
+ params={
+ "profile - The name of the profile.",
+ "entity - The name of the entity.",
+ "durationAgo - How long ago should values be retrieved from?",
+ "units - The units of 'durationAgo'.",
+ "groups - Optional - The groups used to sort the profile."
+ },
+ returns="The profile measurements."
+)
+public class GetProfile implements StellarFunction {
+
+ private ProfilerClient client;
+
+ /**
+ * Initialization.
+ */
+ @Override
+ public void initialize(Context context) {
+
+ Context.Capabilities[] required = { PROFILER_HBASE_TABLE, PROFILER_ROW_KEY_BUILDER, PROFILER_COLUMN_BUILDER };
--- End diff --
Sure, You notice [here](https://github.com/cestella/incubator-metron/blob/cd6348279744ba521d99f1dbe71030fafad6cd1a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/stellar/SimpleHBaseEnrichmentFunctions.java#L91) that I'm doing something similar except instead of insisting that the capability be there prior to the function, I look for the configuration information (i.e. the table provider and such) in the global config. This way, we don't have to add new capabilities to every single topology (parser, enrichment and now profiler) for every stellar function. Instead, they can pull their configs from the global config.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #242: METRON-391 Create Stellar Function to Re...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/242#discussion_r78437044
--- Diff: metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/GetProfile.java ---
@@ -0,0 +1,179 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.client.stellar;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.hadoop.hbase.client.HTableInterface;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.utils.ConversionUtils;
+import org.apache.metron.profiler.client.HBaseProfilerClient;
+import org.apache.metron.profiler.client.ProfilerClient;
+import org.apache.metron.profiler.hbase.ColumnBuilder;
+import org.apache.metron.profiler.hbase.RowKeyBuilder;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static java.lang.String.format;
+import static org.apache.metron.common.dsl.Context.Capabilities.PROFILER_COLUMN_BUILDER;
+import static org.apache.metron.common.dsl.Context.Capabilities.PROFILER_HBASE_TABLE;
+import static org.apache.metron.common.dsl.Context.Capabilities.PROFILER_ROW_KEY_BUILDER;
+
+/**
+ * A Stellar function that can retrieve data contained within a Profile.
+ *
+ * PROFILE_GET
+ *
+ * Retrieve all values for 'entity1' from 'profile1' over the past 4 hours.
+ *
+ * <code>PROFILE_GET('profile1', 'entity1', 4, 'HOURS')</code>
+ *
+ * Retrieve all values for 'entity1' from 'profile1' over the past 2 days.
+ *
+ * <code>PROFILE_GET('profile1', 'entity1', 2, 'DAYS')</code>
+ *
+ * Retrieve all values for 'entity1' from 'profile1' that occurred on 'weekdays' over the past month.
+ *
+ * <code>PROFILE_GET('profile1', 'entity1', 1, 'MONTHS', 'weekdays')</code>
+ *
+ */
+@Stellar(
+ namespace="PROFILE",
+ name="GET",
+ description="Retrieves a series of values from a stored profile.",
+ params={
+ "profile - The name of the profile.",
+ "entity - The name of the entity.",
+ "durationAgo - How long ago should values be retrieved from?",
+ "units - The units of 'durationAgo'.",
+ "groups - Optional - The groups used to sort the profile."
+ },
+ returns="The profile measurements."
+)
+public class GetProfile implements StellarFunction {
+
+ private ProfilerClient client;
+
+ /**
+ * Initialization.
+ */
+ @Override
+ public void initialize(Context context) {
+
+ Context.Capabilities[] required = { PROFILER_HBASE_TABLE, PROFILER_ROW_KEY_BUILDER, PROFILER_COLUMN_BUILDER };
+ validateContext(context, required);
+
+ HTableInterface table = (HTableInterface) context.getCapability(PROFILER_HBASE_TABLE).get();
+ RowKeyBuilder rowKeyBuilder = (RowKeyBuilder) context.getCapability(PROFILER_ROW_KEY_BUILDER).get();
+ ColumnBuilder columnBuilder = (ColumnBuilder) context.getCapability(PROFILER_COLUMN_BUILDER).get();
+
+ client = new HBaseProfilerClient(table, rowKeyBuilder, columnBuilder);
+ }
+
+ /**
+ * Is the function initialized?
+ */
+ @Override
+ public boolean isInitialized() {
+ return client != null;
+ }
+
+ /**
+ * Apply the function.
+ * @param args The function arguments.
+ * @param context
+ */
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+
+ String profile = getArg(0, String.class, args);
+ String entity = getArg(1, String.class, args);
+ long durationAgo = getArg(2, Long.class, args);
+ String unitsName = getArg(3, String.class, args);
+ TimeUnit units = TimeUnit.valueOf(unitsName);
+ List<Object> groups = getGroupsArg(4, args);
+
+ return client.fetch(profile, entity, durationAgo, units, Integer.class, groups);
--- End diff --
Yes, I think that would be a good enhancement. But right now it assumes Numeric. Mainly because of serialization. Opened METRON-415 to track the enhancement of supporting a variety of types as profile values.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #242: METRON-391 Create Stellar Function to Read Prof...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/242
Agreed, however in this case and in every case I can think of as a stellar function writer, you should use the configs from zookeeper to configure your function.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #242: METRON-391 Create Stellar Function to Read Prof...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/242
The Stellar function is now configured via global properties instead of Capabilities.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #242: METRON-391 Create Stellar Function to Read Prof...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/242
Ok. We probably need to document the distinction between a Capability and a Global config. It was not clear to me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #242: METRON-391 Create Stellar Function to Read Prof...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/242
+1, looks good. I'll tackle METRON-415 to expand this to non-numeric data.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---