You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by cestella <gi...@git.apache.org> on 2017/04/07 04:04:19 UTC
[GitHub] incubator-metron pull request #517: METRON-831: Add lambda expressions and r...
GitHub user cestella opened a pull request:
https://github.com/apache/incubator-metron/pull/517
METRON-831: Add lambda expressions and rudimentary functional programming primitives to Stellar
## Contributor Comments
To enable use-cases where the user may want to do simple iteration, we can expose simple functional primitives such as:
* `MAP`
* `REDUCE`
* `FILTER`
To fully support this, we will need lambda expressions as arguments to these functions.
I also added a simple `LIST_ADD` to enable better testing of the functions and because it was an easy-to-fill gap.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron (Incubating).
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [ ] Have you included steps or a guide to how the change may be verified and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to verify your changes?
- [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
bin/generate-md.sh
mvn site:site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommened that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron METRON-831
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/517.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #517
----
commit 73d085d103b5489a97ff770d7ee0bf02c327234a
Author: cstella <ce...@gmail.com>
Date: 2017-04-07T01:16:52Z
METRON-831: Add lambda functions and rudimentary functional primitives
commit 42fcb0ad2d75f5b032844794081f71123426c68d
Author: cstella <ce...@gmail.com>
Date: 2017-04-07T03:58:18Z
Added docs, changed around the syntax a slight bit to allow named args.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #517: METRON-831: Add lambda expressions and rudiment...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/517
+1 by review. The 'assumes one argument' part, actually the 'assumes' part isn't my favorite thing, but we can look at that when we have interactive stellar writing in the ui ( and can validate before deploy ) better sorted out.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #517: METRON-831: Add lambda expressions and rudiment...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/517
The higher order functions so only work on lists at the moment. I figured we could add that at contribution a bit later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #517: METRON-831: Add lambda expressions and rudiment...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/517
@jjmeyer0 Thanks so much for the review. I fixed the support for nulls and added unit tests to test it. I also adjusted the readme appropriately.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #517: METRON-831: Add lambda expressions and r...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/517
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #517: METRON-831: Add lambda expressions and r...
Posted by jjmeyer0 <gi...@git.apache.org>.
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/517#discussion_r110520408
--- Diff: metron-platform/metron-common/README.md ---
@@ -466,6 +512,14 @@ The `!=` operator is the negation of the above.
* IANA Number
* Returns: The protocol name associated with the IANA number.
+### `REDUCE`
+ * Description: Reduces a list by a binary lambda expression. That is, the expression takes two arguments. Usage example: `REDUCE( [ 1, 2, 3 ] , (x, y) -> x + y)` would sum the input list, yielding `6`.
+ * Input:
+ * list - List of arguments.
+ * binary_operation - The lambda expression function to apply to reduce the list. It is assumed that this takes two arguments, the first being the running total and the second being an item from the list.
+ * initial_value - The initial value to use.
--- End diff --
This example is missing the `initial_value`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #517: METRON-831: Add lambda expressions and rudiment...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/517
Well, if @JonZeolla then I can't argue. ;)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #517: METRON-831: Add lambda expressions and r...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/517#discussion_r110523410
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/LambdaExpression.java ---
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.common.stellar;
+
+import org.apache.metron.common.dsl.Token;
+import org.apache.metron.common.dsl.VariableResolver;
+
+import java.util.*;
--- End diff --
agreed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #517: METRON-831: Add lambda expressions and r...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/517#discussion_r110522955
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/FunctionalFunctions.java ---
@@ -0,0 +1,121 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.common.dsl.functions;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.metron.common.dsl.BaseStellarFunction;
+import org.apache.metron.common.dsl.Stellar;
+import org.apache.metron.common.stellar.LambdaExpression;
+
+import java.util.ArrayList;
+import java.util.List;
+
+public class FunctionalFunctions {
+ @Stellar(name="MAP"
+ , description="Applies lambda expression to a list of arguments. e.g. MAP( [ 'foo', 'bar' ] , ( x ) -> TO_UPPER(x) ) would yield [ 'FOO', 'BAR' ]"
+ , params = {
+ "list - List of arguments."
+ ,"transform_expression - The lambda expression to apply. This expression is assumed to take one argument."
+ }
+ , returns = "The input list transformed item-wise by the lambda expression."
+ )
+ public static class Map extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> args) {
+ List<Object> input = (List<Object>) args.get(0);
+ LambdaExpression expression = (LambdaExpression)args.get(1);
+ if(input == null || expression == null) {
+ return input;
+ }
+ List<Object> ret = new ArrayList<>();
+ for(Object o : input) {
+ if(o != null) {
--- End diff --
You're totally right, I was deferring to `ImmutableList`'s assumptions, which may not be (probably not, in fact) valid here. I'll adjust accordingly and support nulls.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #517: METRON-831: Add lambda expressions and rudiment...
Posted by jjmeyer0 <gi...@git.apache.org>.
Github user jjmeyer0 commented on the issue:
https://github.com/apache/incubator-metron/pull/517
+1 by review. I also spun it up in full dev and tested some basic stellar statements using the new functions. Everything looks good. I think this is a great PR and will be very useful to people.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #517: METRON-831: Add lambda expressions and r...
Posted by jjmeyer0 <gi...@git.apache.org>.
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/517#discussion_r110519705
--- Diff: metron-platform/metron-common/README.md ---
@@ -73,6 +73,27 @@ Below is how the `==` operator is expected to work:
The `!=` operator is the negation of the above.
+## Stellar Language Lambda Expressions
+
+Stellar provides the capability to pass lambda expressions to functions
+which wish to support that layer of indirection. The syntax is:
+* `(named_variables) -> stellar_expression` : Lambda expression with named variables
+ * For instance, the lambda expression which calls `TO_UPPER` on a named argument `x` could be be expressed as `(x) -> TO_UPPER(x)`.
+* `var -> stellar_expression` : Lambda expression with a single named variable, `var`
+ * For instance, the lambda expression which calls `TO_UPPER` on a named argument `x` could be expressed as `x -> TO_UPPER(x)`. Note, this is more succinct but equivalent to the example directly above.
+* `() -> stellar_expression` : Lambda expression with no named variables.
+ * If no named variables are needed, you may omit the named variable section. For instance, the lambda expression which returns a constant `false` would be `() -> false`
+
+where
+* `named_variables` is a comma separated list of variables to use in the Stellar expression
+* `stellar_expression` is an arbitrary stellar expression
+
+
+In the core language functions, we support basic functional programming primitives such as
+* `MAP` - Applies a lambda expression over a list of input. For instance `MAP([ 'foo', 'bar'], (x) -> TO_UPPER(x) )` returns `[ 'FOO', 'BAR' ]`
+* `FILTER` - Filters a list by a predicate in the form of a lambda expression. For instance `FILTER([ 'foo', 'bar'], (x ) -> x == 'foo' )` returns `[ 'foo' ]`
+* `REDUCE` - Applies a function over a list of input. For instance `MAP([ 'foo', 'bar'], (x) -> TO_UPPER(x) )` returns `[ 'FOO', 'BAR' ]`
--- End diff --
I believe you may have forgotten to update your copy/paste for the `REDUCE` documentation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #517: METRON-831: Add lambda expressions and rudiment...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/517
@ottobackwards Thanks for the +1 and yeah, I definitely hear the concern. I wanted a rudimentary set of functional primitives, but there should be a follow-up to correct some issues with them around capability, such as supporting more complex data structures than lists (i.e. supporting `MAP({ 'key' : 'value', 'key2' : 'value2' }, (k,v) -> JOIN([k, v], ':'))` should be `['key:value', 'key2:value2'])`). In that case, it wouldn't assume a single argument.
Oh, also, we do actually validate before a zookeeper push all stellar statements. It will fail if you botch the stellar. I'm looking forward to a proper UI around stellar, though, for sure. :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #517: METRON-831: Add lambda expressions and rudiment...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/517
Do the higher order functions only work on lists right now? For example, can I reduce a map? If not, we can tackle later, but just want to understand the scope.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #517: METRON-831: Add lambda expressions and r...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/517#discussion_r110522921
--- Diff: metron-platform/metron-common/README.md ---
@@ -466,6 +512,14 @@ The `!=` operator is the negation of the above.
* IANA Number
* Returns: The protocol name associated with the IANA number.
+### `REDUCE`
+ * Description: Reduces a list by a binary lambda expression. That is, the expression takes two arguments. Usage example: `REDUCE( [ 1, 2, 3 ] , (x, y) -> x + y)` would sum the input list, yielding `6`.
+ * Input:
+ * list - List of arguments.
+ * binary_operation - The lambda expression function to apply to reduce the list. It is assumed that this takes two arguments, the first being the running total and the second being an item from the list.
+ * initial_value - The initial value to use.
--- End diff --
Yep, agreed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #517: METRON-831: Add lambda expressions and rudiment...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on the issue:
https://github.com/apache/incubator-metron/pull/517
+1 to sticking with Java(ish) syntax, because I can recall that we've also modeled against Java in other stellar functions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #517: METRON-831: Add lambda expressions and r...
Posted by jjmeyer0 <gi...@git.apache.org>.
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/517#discussion_r110520191
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/LambdaExpression.java ---
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.common.stellar;
+
+import org.apache.metron.common.dsl.Token;
+import org.apache.metron.common.dsl.VariableResolver;
+
+import java.util.*;
--- End diff --
Nit, explode star import
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #517: METRON-831: Add lambda expressions and rudiment...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/517
Really great contribution. This is going to be so useful.
I agree that it might be better to mimic the syntax from an existing language. Personally, I think any of the 3 options defined would work well, but based on our roots in Java and the ubiquity of Java, I'd stick with Java's approach.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #517: METRON-831: Add lambda expressions and r...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/517#discussion_r110523398
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/FunctionalFunctions.java ---
@@ -0,0 +1,121 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.common.dsl.functions;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.metron.common.dsl.BaseStellarFunction;
+import org.apache.metron.common.dsl.Stellar;
+import org.apache.metron.common.stellar.LambdaExpression;
+
+import java.util.ArrayList;
+import java.util.List;
+
+public class FunctionalFunctions {
+ @Stellar(name="MAP"
+ , description="Applies lambda expression to a list of arguments. e.g. MAP( [ 'foo', 'bar' ] , ( x ) -> TO_UPPER(x) ) would yield [ 'FOO', 'BAR' ]"
+ , params = {
+ "list - List of arguments."
+ ,"transform_expression - The lambda expression to apply. This expression is assumed to take one argument."
+ }
+ , returns = "The input list transformed item-wise by the lambda expression."
+ )
+ public static class Map extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> args) {
+ List<Object> input = (List<Object>) args.get(0);
+ LambdaExpression expression = (LambdaExpression)args.get(1);
+ if(input == null || expression == null) {
+ return input;
+ }
+ List<Object> ret = new ArrayList<>();
+ for(Object o : input) {
+ if(o != null) {
+ ret.add(expression.apply(ImmutableList.of(o)));
+ }
+ }
+ return ret;
+ }
+ }
+
+ @Stellar(name="FILTER"
+ , description="Applies a filter in the form of a lambda expression to a list. e.g. FILTER( [ 'foo', 'bar' ] , (x) -> x == 'foo') would yield [ 'foo']"
+ , params = {
+ "list - List of arguments."
+ ,"predicate - The lambda expression to apply. This expression is assumed to take one argument and return a boolean."
+ }
+ , returns = "The input list filtered by the predicate."
+ )
+ public static class Filter extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> args) {
+ List<Object> input = (List<Object>) args.get(0);
+ LambdaExpression expression = (LambdaExpression) args.get(1);
+ if(input == null || expression == null) {
+ return input;
+ }
+ List<Object> ret = new ArrayList<>();
+ for(Object o : input) {
+ if(o == null) {
--- End diff --
yes, your example should work and return `[null]`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #517: METRON-831: Add lambda expressions and rudiment...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/517
I spun this up in full-dev and did the following to test:
* Download the Alexa top 1m data set
```
wget http://s3.amazonaws.com/alexa-static/top-1m.csv.zip
unzip top-1m.csv.zip
```
* Stage import file
```
head -n 10000 top-1m.csv > top-10k.csv
head -n 10 top-1m.csv > top-10.csv
hadoop fs -put top-10k.csv /tmp
```
* Create an extractor.json for the CSV data by editing `extractor.json` and pasting in these contents:
```
{
"config" : {
"zk_quorum" : "node1:2181",
"columns" : {
"rank" : 0,
"domain" : 1
},
"value_transform" : {
"domain" : "DOMAIN_REMOVE_TLD(domain)",
"port" : "es.port"
},
"value_filter" : "LENGTH(domain) > 0",
"indicator_column" : "domain",
"indicator_transform" : {
"indicator" : "DOMAIN_REMOVE_TLD(indicator)"
},
"indicator_filter" : "LENGTH(indicator) > 0",
"type" : "top_domains",
"separator" : ","
},
"extractor" : "CSV"
}
```
* Import enriched data
`echo "truncate 'enrichment'" | hbase shell && $METRON_HOME/bin/flatfile_loader.sh -i ./top-10k.csv -t enrichment -c t -e ./extractor.json -p 5 -b 128 && echo "count 'enrichment'" | hbase shell`
* Open up the stellar shell via `$METRON_HOME/bin/stellar -z node1` and execute the following:
```
MAP(['google', 'pdf2doc', 'yahoo'], indicator -> MAP_GET('domain', ENRICHMENT_GET('top_domains', indicator, 'enrichment', 't')) )
```
You should see `[google, pdf2doc, yahoo]` returned.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #517: METRON-831: Add lambda expressions and r...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/517#discussion_r110523411
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/FunctionalFunctions.java ---
@@ -0,0 +1,121 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.common.dsl.functions;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.metron.common.dsl.BaseStellarFunction;
+import org.apache.metron.common.dsl.Stellar;
+import org.apache.metron.common.stellar.LambdaExpression;
+
+import java.util.ArrayList;
+import java.util.List;
+
+public class FunctionalFunctions {
+ @Stellar(name="MAP"
+ , description="Applies lambda expression to a list of arguments. e.g. MAP( [ 'foo', 'bar' ] , ( x ) -> TO_UPPER(x) ) would yield [ 'FOO', 'BAR' ]"
+ , params = {
+ "list - List of arguments."
+ ,"transform_expression - The lambda expression to apply. This expression is assumed to take one argument."
+ }
+ , returns = "The input list transformed item-wise by the lambda expression."
+ )
+ public static class Map extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> args) {
+ List<Object> input = (List<Object>) args.get(0);
+ LambdaExpression expression = (LambdaExpression)args.get(1);
+ if(input == null || expression == null) {
+ return input;
+ }
+ List<Object> ret = new ArrayList<>();
+ for(Object o : input) {
+ if(o != null) {
+ ret.add(expression.apply(ImmutableList.of(o)));
+ }
+ }
+ return ret;
+ }
+ }
+
+ @Stellar(name="FILTER"
+ , description="Applies a filter in the form of a lambda expression to a list. e.g. FILTER( [ 'foo', 'bar' ] , (x) -> x == 'foo') would yield [ 'foo']"
+ , params = {
+ "list - List of arguments."
+ ,"predicate - The lambda expression to apply. This expression is assumed to take one argument and return a boolean."
+ }
+ , returns = "The input list filtered by the predicate."
+ )
+ public static class Filter extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> args) {
+ List<Object> input = (List<Object>) args.get(0);
+ LambdaExpression expression = (LambdaExpression) args.get(1);
+ if(input == null || expression == null) {
+ return input;
+ }
+ List<Object> ret = new ArrayList<>();
+ for(Object o : input) {
+ if(o == null) {
+ continue;
+ }
+ Object result = expression.apply(ImmutableList.of(o));
+ if(result != null && result instanceof Boolean && (Boolean)result) {
+ ret.add(o);
+ }
+ }
+ return ret;
+ }
+ }
+
+ @Stellar(name="REDUCE"
+ , description="Reduces a list by a binary lambda expression. That is, the expression takes two arguments. Usage example: REDUCE( [ 1, 2, 3 ] , (x, y) -> x + y) would sum the input list, yielding 6."
+ , params = {
+ "list - List of arguments."
+ ,"binary_operation - The lambda expression function to apply to reduce the list. It is assumed that this takes two arguments, the first being the running total and the second being an item from the list."
+ ,"initial_value - The initial value to use."
+ }
+ , returns = "The reduction of the list."
+ )
+ public static class Reduce extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> args) {
+ List<Object> input = (List<Object>) args.get(0);
+ if(input == null || input.size() < 3) {
+ return null;
+ }
+ LambdaExpression expression = (LambdaExpression) args.get(1);
+
+ Object runningResult = args.get(2);
+ if(expression == null || runningResult == null) {
+ return null;
+ }
+ for(int i = 0;i < input.size();++i) {
+ Object rhs = input.get(i);
+ if(rhs == null) {
--- End diff --
yep, we should support nulls
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #517: METRON-831: Add lambda expressions and rudiment...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/517
So, the syntax chosen here for lambda expressions is `&(var_0, var_1, ..., var_n : expr)`. This seemed like the simplest option to me, but it's possible that we should mimic existing systems such as:
* Pythonish: `lambda var_0, var1, ..., var_n : expr`
* Javaish: `(var_0, var_1, ..., var_n) -> expr `
* Scalaish: `(var_0, var_1, .., var_n) => expr`
Is there any feedback on the syntax? Would you like to see something different?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #517: METRON-831: Add lambda expressions and r...
Posted by jjmeyer0 <gi...@git.apache.org>.
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/517#discussion_r110520125
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/FunctionalFunctions.java ---
@@ -0,0 +1,121 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.common.dsl.functions;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.metron.common.dsl.BaseStellarFunction;
+import org.apache.metron.common.dsl.Stellar;
+import org.apache.metron.common.stellar.LambdaExpression;
+
+import java.util.ArrayList;
+import java.util.List;
+
+public class FunctionalFunctions {
+ @Stellar(name="MAP"
+ , description="Applies lambda expression to a list of arguments. e.g. MAP( [ 'foo', 'bar' ] , ( x ) -> TO_UPPER(x) ) would yield [ 'FOO', 'BAR' ]"
+ , params = {
+ "list - List of arguments."
+ ,"transform_expression - The lambda expression to apply. This expression is assumed to take one argument."
+ }
+ , returns = "The input list transformed item-wise by the lambda expression."
+ )
+ public static class Map extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> args) {
+ List<Object> input = (List<Object>) args.get(0);
+ LambdaExpression expression = (LambdaExpression)args.get(1);
+ if(input == null || expression == null) {
+ return input;
+ }
+ List<Object> ret = new ArrayList<>();
+ for(Object o : input) {
+ if(o != null) {
+ ret.add(expression.apply(ImmutableList.of(o)));
+ }
+ }
+ return ret;
+ }
+ }
+
+ @Stellar(name="FILTER"
+ , description="Applies a filter in the form of a lambda expression to a list. e.g. FILTER( [ 'foo', 'bar' ] , (x) -> x == 'foo') would yield [ 'foo']"
+ , params = {
+ "list - List of arguments."
+ ,"predicate - The lambda expression to apply. This expression is assumed to take one argument and return a boolean."
+ }
+ , returns = "The input list filtered by the predicate."
+ )
+ public static class Filter extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> args) {
+ List<Object> input = (List<Object>) args.get(0);
+ LambdaExpression expression = (LambdaExpression) args.get(1);
+ if(input == null || expression == null) {
+ return input;
+ }
+ List<Object> ret = new ArrayList<>();
+ for(Object o : input) {
+ if(o == null) {
--- End diff --
This will cause the following Stellar expression to return an empty list:
`FILTER([ 'foo', 'bar', null ], (item) -> item == null )`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #517: METRON-831: Add lambda expressions and r...
Posted by jjmeyer0 <gi...@git.apache.org>.
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/517#discussion_r110520123
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/FunctionalFunctions.java ---
@@ -0,0 +1,121 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.common.dsl.functions;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.metron.common.dsl.BaseStellarFunction;
+import org.apache.metron.common.dsl.Stellar;
+import org.apache.metron.common.stellar.LambdaExpression;
+
+import java.util.ArrayList;
+import java.util.List;
+
+public class FunctionalFunctions {
+ @Stellar(name="MAP"
+ , description="Applies lambda expression to a list of arguments. e.g. MAP( [ 'foo', 'bar' ] , ( x ) -> TO_UPPER(x) ) would yield [ 'FOO', 'BAR' ]"
+ , params = {
+ "list - List of arguments."
+ ,"transform_expression - The lambda expression to apply. This expression is assumed to take one argument."
+ }
+ , returns = "The input list transformed item-wise by the lambda expression."
+ )
+ public static class Map extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> args) {
+ List<Object> input = (List<Object>) args.get(0);
+ LambdaExpression expression = (LambdaExpression)args.get(1);
+ if(input == null || expression == null) {
+ return input;
+ }
+ List<Object> ret = new ArrayList<>();
+ for(Object o : input) {
+ if(o != null) {
--- End diff --
What happens if null means something to the user? They wouldn't be able to apply null to their expression. Would this ever be a use case?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #517: METRON-831: Add lambda expressions and r...
Posted by jjmeyer0 <gi...@git.apache.org>.
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/517#discussion_r110520142
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/FunctionalFunctions.java ---
@@ -0,0 +1,121 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.common.dsl.functions;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.metron.common.dsl.BaseStellarFunction;
+import org.apache.metron.common.dsl.Stellar;
+import org.apache.metron.common.stellar.LambdaExpression;
+
+import java.util.ArrayList;
+import java.util.List;
+
+public class FunctionalFunctions {
+ @Stellar(name="MAP"
+ , description="Applies lambda expression to a list of arguments. e.g. MAP( [ 'foo', 'bar' ] , ( x ) -> TO_UPPER(x) ) would yield [ 'FOO', 'BAR' ]"
+ , params = {
+ "list - List of arguments."
+ ,"transform_expression - The lambda expression to apply. This expression is assumed to take one argument."
+ }
+ , returns = "The input list transformed item-wise by the lambda expression."
+ )
+ public static class Map extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> args) {
+ List<Object> input = (List<Object>) args.get(0);
+ LambdaExpression expression = (LambdaExpression)args.get(1);
+ if(input == null || expression == null) {
+ return input;
+ }
+ List<Object> ret = new ArrayList<>();
+ for(Object o : input) {
+ if(o != null) {
+ ret.add(expression.apply(ImmutableList.of(o)));
+ }
+ }
+ return ret;
+ }
+ }
+
+ @Stellar(name="FILTER"
+ , description="Applies a filter in the form of a lambda expression to a list. e.g. FILTER( [ 'foo', 'bar' ] , (x) -> x == 'foo') would yield [ 'foo']"
+ , params = {
+ "list - List of arguments."
+ ,"predicate - The lambda expression to apply. This expression is assumed to take one argument and return a boolean."
+ }
+ , returns = "The input list filtered by the predicate."
+ )
+ public static class Filter extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> args) {
+ List<Object> input = (List<Object>) args.get(0);
+ LambdaExpression expression = (LambdaExpression) args.get(1);
+ if(input == null || expression == null) {
+ return input;
+ }
+ List<Object> ret = new ArrayList<>();
+ for(Object o : input) {
+ if(o == null) {
+ continue;
+ }
+ Object result = expression.apply(ImmutableList.of(o));
+ if(result != null && result instanceof Boolean && (Boolean)result) {
+ ret.add(o);
+ }
+ }
+ return ret;
+ }
+ }
+
+ @Stellar(name="REDUCE"
+ , description="Reduces a list by a binary lambda expression. That is, the expression takes two arguments. Usage example: REDUCE( [ 1, 2, 3 ] , (x, y) -> x + y) would sum the input list, yielding 6."
+ , params = {
+ "list - List of arguments."
+ ,"binary_operation - The lambda expression function to apply to reduce the list. It is assumed that this takes two arguments, the first being the running total and the second being an item from the list."
+ ,"initial_value - The initial value to use."
+ }
+ , returns = "The reduction of the list."
+ )
+ public static class Reduce extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> args) {
+ List<Object> input = (List<Object>) args.get(0);
+ if(input == null || input.size() < 3) {
+ return null;
+ }
+ LambdaExpression expression = (LambdaExpression) args.get(1);
+
+ Object runningResult = args.get(2);
+ if(expression == null || runningResult == null) {
+ return null;
+ }
+ for(int i = 0;i < input.size();++i) {
+ Object rhs = input.get(i);
+ if(rhs == null) {
--- End diff --
Same question as the other functions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #517: METRON-831: Add lambda expressions and rudiment...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/517
Ok, I migrated to a more java-ish syntax. I like it better, it's more succinct too, I think. The following are examples:
* `MAP( [ 1, 2, 3 ], (x) -> x*2)` would yield `[2, 4, 6]`
* `MAP( [ 1, 2, 3], x -> x*2)` would yield `[2, 4, 6]`
* `REDUCE( [1, 2, 3], (sum, x) -> sum + x)` would yield `6`
* `FILTER( [1, 2, 3], () -> true)` would yield `[1, 2, 3]`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #517: METRON-831: Add lambda expressions and r...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/517#discussion_r110522907
--- Diff: metron-platform/metron-common/README.md ---
@@ -73,6 +73,27 @@ Below is how the `==` operator is expected to work:
The `!=` operator is the negation of the above.
+## Stellar Language Lambda Expressions
+
+Stellar provides the capability to pass lambda expressions to functions
+which wish to support that layer of indirection. The syntax is:
+* `(named_variables) -> stellar_expression` : Lambda expression with named variables
+ * For instance, the lambda expression which calls `TO_UPPER` on a named argument `x` could be be expressed as `(x) -> TO_UPPER(x)`.
+* `var -> stellar_expression` : Lambda expression with a single named variable, `var`
+ * For instance, the lambda expression which calls `TO_UPPER` on a named argument `x` could be expressed as `x -> TO_UPPER(x)`. Note, this is more succinct but equivalent to the example directly above.
+* `() -> stellar_expression` : Lambda expression with no named variables.
+ * If no named variables are needed, you may omit the named variable section. For instance, the lambda expression which returns a constant `false` would be `() -> false`
+
+where
+* `named_variables` is a comma separated list of variables to use in the Stellar expression
+* `stellar_expression` is an arbitrary stellar expression
+
+
+In the core language functions, we support basic functional programming primitives such as
+* `MAP` - Applies a lambda expression over a list of input. For instance `MAP([ 'foo', 'bar'], (x) -> TO_UPPER(x) )` returns `[ 'FOO', 'BAR' ]`
+* `FILTER` - Filters a list by a predicate in the form of a lambda expression. For instance `FILTER([ 'foo', 'bar'], (x ) -> x == 'foo' )` returns `[ 'foo' ]`
+* `REDUCE` - Applies a function over a list of input. For instance `MAP([ 'foo', 'bar'], (x) -> TO_UPPER(x) )` returns `[ 'FOO', 'BAR' ]`
--- End diff --
Yep, agreed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---