You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by anandsubbu <gi...@git.apache.org> on 2017/04/06 19:43:56 UTC
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
GitHub user anandsubbu opened a pull request:
https://github.com/apache/incubator-metron/pull/516
METRON-830 Adding StringFunctions to Stellar - chop, prependifmissing, appendifmissing and countmatches
## Contributor Comments
Added stellar functions and unit tests for the following:
chop -> Remove the last character from a String.
prependifmissing -> Prepends the prefix to the start of the string if the string does not already start with any of the prefixes.
appendifmissing -> Appends the suffix to the end of the string if the string does not already end with any of the suffixes.
countmatches -> Counts how many times the substring appears in the larger string.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron (Incubating).
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to verify your changes?
- [NA] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
bin/generate-md.sh
mvn site:site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommened that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/anandsubbu/incubator-metron METRON-830
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/516.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #516
----
commit 62d7544e420f0cd067c272c6bacaf53be545b2c9
Author: Anand Subramanian <as...@hortonworks.com>
Date: 2017-04-06T19:39:05Z
Added string functions and unit tests
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r110294012
--- Diff: metron-platform/metron-common/src/test/java/org/apache/metron/common/dsl/functions/StringFunctionsTest.java ---
@@ -207,4 +207,78 @@ public void testFormatWithNoArguments() throws Exception {
public void testFormatWithMissingArguments() throws Exception {
run("FORMAT('missing arg: %d')", Collections.emptyMap());
}
+
+
+ /**
+ * CHOP StringFunction
+ * @throws Exception
+ */
+ @Test
+ public void testChop() throws Exception {
+ Assert.assertEquals("ab", run("CHOP('abc')", new HashedMap()));
+ Assert.assertEquals(null, run("CHOP(null)", new HashedMap()));
+ Assert.assertEquals("abc", run("CHOP(msg)", ImmutableMap.of("msg", "abc\r\n")));
+ Assert.assertEquals("", run("CHOP(msg)", ImmutableMap.of("msg", "\n")));
+ }
+
+ @Test(expected = ParseException.class)
+ public void testChopWithMissingArguments() throws Exception {
+ run("CHOP()", Collections.emptyMap());
+ }
+
+ /**
+ * PREPENDIFMISSING StringFunction
+ * @throws Exception
+ */
+ @Test
+ public void testPrependIfMissing() throws Exception {
+ Assert.assertEquals("xyzabc", run("PREPENDIFMISSING('abc', 'xyz')", new HashedMap()));
+ Assert.assertEquals("xyzXYZabc", run("PREPENDIFMISSING('XYZabc', 'xyz', 'mno')", new HashedMap()));
+ Assert.assertEquals(null, run("PREPENDIFMISSING(null, null, null)", new HashedMap()));
+ Assert.assertEquals("xyz", run("PREPENDIFMISSING('', 'xyz', null)", new HashedMap()));
+ }
+
+ @Test(expected = ParseException.class)
+ public void testPrependIfMissingWithIncorrectArgs() throws Exception {
--- End diff --
I may be wrong, but I think JUnit expected errors do not auto-continue inside an errored test case. So, you're only testing the first sub-case.
To fix, you either need 3 test cases, or you need to do try/catch/finally contexts around each `run()`, where the catch eats the exception but sets a marker flag, and the finally asserts the marker flag.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r110581823
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java ---
@@ -343,4 +343,89 @@ public Object apply(List<Object> args) {
return String.format(format, formatArgs);
}
}
+
+ @Stellar( name="CHOP"
+ , description = "Remove the last character from a String"
+ , params = { "the String to chop last character from, may be null"}
+ , returns = "String without last character, null if null String input"
+ )
+ public static class chop extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> strings) {
+
+ if(strings.size() == 0) {
+ throw new IllegalArgumentException("[CHOP] missing argument: string to be chopped");
+ }
+
+ String chop = StringUtils.chop((String) strings.get(0));
+ return chop;
+ }
+ }
+
+ @Stellar( name = "PREPENDIFMISSING"
+ , description = "Prepends the prefix to the start of the string if the string does not already start with any of the prefixes"
+ , params = { "str - The string.", "prefix - The prefix to prepend to the start of the string", "prefixes - Additional prefixes that are valid" }
+ , returns = "A new String if prefix was prepended, the same string otherwise."
+ )
+ public static class prependifmissing extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> strings) {
+
+ String prefixed;
+ switch (strings.size()) {
+ case 2: prefixed = org.apache.commons.lang3.StringUtils.prependIfMissing((String) strings.get(0), (String) strings.get(1));
+ break;
+ case 3: prefixed = org.apache.commons.lang3.StringUtils.prependIfMissing((String) strings.get(0), (String) strings.get(1), (String) strings.get(2));
+ break;
+ default: throw new IllegalArgumentException("[PREPENDIFMISSING] incorrect arguments. Usage: PREPENDIFMISSING <String> <prefix> [<prefix>...]");
--- End diff --
Now included the arguments provided by user as well, apart from Usage info.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r110672618
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java ---
@@ -343,4 +343,89 @@ public Object apply(List<Object> args) {
return String.format(format, formatArgs);
}
}
+
+ @Stellar( name="CHOP"
--- End diff --
Okay, thanks. I stand corrected about the URI/URL.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r110964390
--- Diff: metron-platform/metron-common/README.md ---
@@ -167,6 +172,14 @@ The `!=` operator is the negation of the above.
| [ `WEEK_OF_YEAR`](#week_of_year) |
| [ `YEAR`](#year) |
+### `APPEND_IF_MISSING`
+ * Description: Appends the suffix to the end of the string if the string does not already end with any of the suffixes.
+ * Input:
+ * string - The string to be appended.
+ * suffix - The string suffix to append to the end of the string.
+ * suffixes - Optional - Additional string suffixes that are valid terminators.
--- End diff --
Please state the format of "suffixes". Is it a string, a regex, an array of strings, or what? If it is a single string, how does it represent plural suffixes?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r111098953
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java ---
@@ -343,4 +343,89 @@ public Object apply(List<Object> args) {
return String.format(format, formatArgs);
}
}
+
+ @Stellar( name="CHOP"
+ , description = "Remove the last character from a String"
+ , params = { "the String to chop last character from, may be null"}
+ , returns = "String without last character, null if null String input"
+ )
+ public static class chop extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> strings) {
+
+ if(strings.size() == 0) {
+ throw new IllegalArgumentException("[CHOP] missing argument: string to be chopped");
--- End diff --
Hi @mattf-horton , with the above change where we check for `strings.get(0) == null || strings.get(0).toString().length() `, the following test results in a failure:
`Assert.assertEquals("abc", run("CHOP(msg)", ImmutableMap.of("msg", "abc\r\n")));`
Pardon my limited understanding, but this is what I inferred:
- The `if` condition for null and 0 length check occurs twice.
- In the case of the `msg` test case where we check for `\n\r`, the initial contents of the `strings` is null and the variable substitution for `msg` occurs subsequently. So, this would end up failing for the 1st check.
Due to this, I will modify the checks to only verify for `strings == null` and `string.size() == 0`. Please let me know if there is a better way this can be done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r111076022
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java ---
@@ -343,4 +343,89 @@ public Object apply(List<Object> args) {
return String.format(format, formatArgs);
}
}
+
+ @Stellar( name="CHOP"
+ , description = "Remove the last character from a String"
+ , params = { "the String to chop last character from, may be null"}
+ , returns = "String without last character, null if null String input"
+ )
+ public static class chop extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> strings) {
+
+ if(strings.size() == 0) {
+ throw new IllegalArgumentException("[CHOP] missing argument: string to be chopped");
--- End diff --
Thanks, made this change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r110273359
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java ---
@@ -343,4 +343,89 @@ public Object apply(List<Object> args) {
return String.format(format, formatArgs);
}
}
+
+ @Stellar( name="CHOP"
--- End diff --
What is the expected use of `chop` in Metron? Would `chomp` be more useful? Would just like to know your thoughts.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r111010577
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java ---
@@ -343,4 +343,89 @@ public Object apply(List<Object> args) {
return String.format(format, formatArgs);
}
}
+
+ @Stellar( name="CHOP"
+ , description = "Remove the last character from a String"
+ , params = { "the String to chop last character from, may be null"}
+ , returns = "String without last character, null if null String input"
+ )
+ public static class chop extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> strings) {
+
+ if(strings.size() == 0) {
+ throw new IllegalArgumentException("[CHOP] missing argument: string to be chopped");
--- End diff --
I think you need
```
if (strings == null || strings.size() == 0 || strings.get(0) == null || strings.get(0).length == 0) { ...
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #516: METRON-830 Adding StringFunctions to Stellar - ...
Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on the issue:
https://github.com/apache/incubator-metron/pull/516
Thanks a lot @cestella for your inputs and help. The latest commit addresses feedback from @mattf-horton and @cestella.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r110629309
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java ---
@@ -343,4 +343,89 @@ public Object apply(List<Object> args) {
return String.format(format, formatArgs);
}
}
+
+ @Stellar( name="CHOP"
--- End diff --
If we are going to do URL/URI work, we should create functions off of the URI / URL classes and use their specific parsing for uri named component parts
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r110431257
--- Diff: metron-platform/metron-common/src/test/java/org/apache/metron/common/dsl/functions/StringFunctionsTest.java ---
@@ -207,4 +207,78 @@ public void testFormatWithNoArguments() throws Exception {
public void testFormatWithMissingArguments() throws Exception {
run("FORMAT('missing arg: %d')", Collections.emptyMap());
}
+
+
+ /**
+ * CHOP StringFunction
+ * @throws Exception
+ */
+ @Test
+ public void testChop() throws Exception {
+ Assert.assertEquals("ab", run("CHOP('abc')", new HashedMap()));
+ Assert.assertEquals(null, run("CHOP(null)", new HashedMap()));
+ Assert.assertEquals("abc", run("CHOP(msg)", ImmutableMap.of("msg", "abc\r\n")));
--- End diff --
Because it's coming in through a variable `msg`. The idea being that it could be existent in a message flowing through the topologies. Stellar can't represent `\n` and `\r`, but they can get in there honestly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r110582728
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java ---
@@ -343,4 +343,89 @@ public Object apply(List<Object> args) {
return String.format(format, formatArgs);
}
}
+
+ @Stellar( name="CHOP"
+ , description = "Remove the last character from a String"
+ , params = { "the String to chop last character from, may be null"}
+ , returns = "String without last character, null if null String input"
+ )
+ public static class chop extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> strings) {
+
+ if(strings.size() == 0) {
+ throw new IllegalArgumentException("[CHOP] missing argument: string to be chopped");
+ }
+
+ String chop = StringUtils.chop((String) strings.get(0));
+ return chop;
+ }
+ }
+
+ @Stellar( name = "PREPENDIFMISSING"
+ , description = "Prepends the prefix to the start of the string if the string does not already start with any of the prefixes"
+ , params = { "str - The string.", "prefix - The prefix to prepend to the start of the string", "prefixes - Additional prefixes that are valid" }
+ , returns = "A new String if prefix was prepended, the same string otherwise."
+ )
+ public static class prependifmissing extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> strings) {
+
+ String prefixed;
+ switch (strings.size()) {
+ case 2: prefixed = org.apache.commons.lang3.StringUtils.prependIfMissing((String) strings.get(0), (String) strings.get(1));
--- End diff --
Changed the import statement to use lang3.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r110293542
--- Diff: metron-platform/metron-common/src/test/java/org/apache/metron/common/dsl/functions/StringFunctionsTest.java ---
@@ -207,4 +207,78 @@ public void testFormatWithNoArguments() throws Exception {
public void testFormatWithMissingArguments() throws Exception {
run("FORMAT('missing arg: %d')", Collections.emptyMap());
}
+
+
+ /**
+ * CHOP StringFunction
+ * @throws Exception
+ */
+ @Test
+ public void testChop() throws Exception {
+ Assert.assertEquals("ab", run("CHOP('abc')", new HashedMap()));
+ Assert.assertEquals(null, run("CHOP(null)", new HashedMap()));
+ Assert.assertEquals("abc", run("CHOP(msg)", ImmutableMap.of("msg", "abc\r\n")));
+ Assert.assertEquals("", run("CHOP(msg)", ImmutableMap.of("msg", "\n")));
+ }
+
+ @Test(expected = ParseException.class)
+ public void testChopWithMissingArguments() throws Exception {
+ run("CHOP()", Collections.emptyMap());
+ }
+
+ /**
+ * PREPENDIFMISSING StringFunction
+ * @throws Exception
+ */
+ @Test
+ public void testPrependIfMissing() throws Exception {
+ Assert.assertEquals("xyzabc", run("PREPENDIFMISSING('abc', 'xyz')", new HashedMap()));
+ Assert.assertEquals("xyzXYZabc", run("PREPENDIFMISSING('XYZabc', 'xyz', 'mno')", new HashedMap()));
--- End diff --
Please add a case that exercises non-null alt prefix, eg:
```
Assert.assertEquals("mnoXYZabc", run("PREPENDIFMISSING('mnoXYZabc', 'xyz', 'mno')", new HashedMap()));
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #516: METRON-830 Adding StringFunctions to Stellar - ...
Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on the issue:
https://github.com/apache/incubator-metron/pull/516
@mattf-horton - if you feel the latest commit addresses your feedback, can you please help with merging this pull request? I already have your +1 and @ottobackwards 's as well. Thank you!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r110286424
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java ---
@@ -343,4 +343,89 @@ public Object apply(List<Object> args) {
return String.format(format, formatArgs);
}
}
+
+ @Stellar( name="CHOP"
+ , description = "Remove the last character from a String"
+ , params = { "the String to chop last character from, may be null"}
+ , returns = "String without last character, null if null String input"
+ )
+ public static class chop extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> strings) {
+
+ if(strings.size() == 0) {
+ throw new IllegalArgumentException("[CHOP] missing argument: string to be chopped");
+ }
+
+ String chop = StringUtils.chop((String) strings.get(0));
+ return chop;
+ }
+ }
+
+ @Stellar( name = "PREPENDIFMISSING"
+ , description = "Prepends the prefix to the start of the string if the string does not already start with any of the prefixes"
+ , params = { "str - The string.", "prefix - The prefix to prepend to the start of the string", "prefixes - Additional prefixes that are valid" }
--- End diff --
With all multi-param Stellar functions, please format the inputs one per line following the pattern of, eg, [here](https://github.com/apache/incubator-metron/blob/master/metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/DataStructureFunctions.java#L57-L59)
Also, if the third argument is optional, please note that parenthetically. And what is the type of `prefixes`? String with a regex? List of Strings? Clarifying this will also make clear that `prefixes` is *not* a varargs sequence. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r111012694
--- Diff: metron-platform/metron-common/README.md ---
@@ -167,6 +172,14 @@ The `!=` operator is the negation of the above.
| [ `WEEK_OF_YEAR`](#week_of_year) |
| [ `YEAR`](#year) |
+### `APPEND_IF_MISSING`
+ * Description: Appends the suffix to the end of the string if the string does not already end with any of the suffixes.
+ * Input:
+ * string - The string to be appended.
+ * suffix - The string suffix to append to the end of the string.
+ * suffixes - Optional - Additional string suffixes that are valid terminators.
--- End diff --
Ah, got it. The documentation you point at for apache commons PrependIfMission method actually uses signature:
```
CharSequence... suffixes
```
so it is a varargs usage. You can do the same with Stellar, or we have found it better to allow a list of strings instead (which you would explode into the calling varargs in the java code). Using a list instead of varargs makes it much easier to extend the function later if we choose.
It's up to you whether to do this now. We are trying to close a RC, so there is some time pressure. Maybe you should just open a jira to enhance this later with a list arg.
The current implementation of your call to PrependIfMissing is such that the "additionalsuffix" description you suggest would be correct.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r110581688
--- Diff: metron-platform/metron-common/src/test/java/org/apache/metron/common/dsl/functions/StringFunctionsTest.java ---
@@ -207,4 +207,78 @@ public void testFormatWithNoArguments() throws Exception {
public void testFormatWithMissingArguments() throws Exception {
run("FORMAT('missing arg: %d')", Collections.emptyMap());
}
+
+
+ /**
+ * CHOP StringFunction
+ * @throws Exception
+ */
+ @Test
+ public void testChop() throws Exception {
+ Assert.assertEquals("ab", run("CHOP('abc')", new HashedMap()));
+ Assert.assertEquals(null, run("CHOP(null)", new HashedMap()));
+ Assert.assertEquals("abc", run("CHOP(msg)", ImmutableMap.of("msg", "abc\r\n")));
+ Assert.assertEquals("", run("CHOP(msg)", ImmutableMap.of("msg", "\n")));
+ }
+
+ @Test(expected = ParseException.class)
+ public void testChopWithMissingArguments() throws Exception {
+ run("CHOP()", Collections.emptyMap());
+ }
+
+ /**
+ * PREPENDIFMISSING StringFunction
+ * @throws Exception
+ */
+ @Test
+ public void testPrependIfMissing() throws Exception {
+ Assert.assertEquals("xyzabc", run("PREPENDIFMISSING('abc', 'xyz')", new HashedMap()));
+ Assert.assertEquals("xyzXYZabc", run("PREPENDIFMISSING('XYZabc', 'xyz', 'mno')", new HashedMap()));
--- End diff --
Done, added.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r110995412
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java ---
@@ -343,4 +343,89 @@ public Object apply(List<Object> args) {
return String.format(format, formatArgs);
}
}
+
+ @Stellar( name="CHOP"
+ , description = "Remove the last character from a String"
+ , params = { "the String to chop last character from, may be null"}
+ , returns = "String without last character, null if null String input"
+ )
+ public static class chop extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> strings) {
+
+ if(strings.size() == 0) {
+ throw new IllegalArgumentException("[CHOP] missing argument: string to be chopped");
--- End diff --
Do you think adding a check for null as follows, would be the right way?
```
if(strings == null || strings.size() == 0) {
throw new IllegalArgumentException("[CHOP] missing argument: string to be chopped");
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r110582764
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java ---
@@ -343,4 +343,89 @@ public Object apply(List<Object> args) {
return String.format(format, formatArgs);
}
}
+
+ @Stellar( name="CHOP"
+ , description = "Remove the last character from a String"
+ , params = { "the String to chop last character from, may be null"}
+ , returns = "String without last character, null if null String input"
+ )
+ public static class chop extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> strings) {
+
+ if(strings.size() == 0) {
+ throw new IllegalArgumentException("[CHOP] missing argument: string to be chopped");
+ }
+
+ String chop = StringUtils.chop((String) strings.get(0));
+ return chop;
+ }
+ }
+
+ @Stellar( name = "PREPENDIFMISSING"
--- End diff --
Sure, changed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r110992116
--- Diff: metron-platform/metron-common/README.md ---
@@ -167,6 +172,14 @@ The `!=` operator is the negation of the above.
| [ `WEEK_OF_YEAR`](#week_of_year) |
| [ `YEAR`](#year) |
+### `APPEND_IF_MISSING`
+ * Description: Appends the suffix to the end of the string if the string does not already end with any of the suffixes.
+ * Input:
+ * string - The string to be appended.
+ * suffix - The string suffix to append to the end of the string.
+ * suffixes - Optional - Additional string suffixes that are valid terminators.
--- End diff --
Hi @mattf-horton , I was trying to following the same wording as the [apache commons documentation](https://commons.apache.org/proper/commons-lang/apidocs/org/apache/commons/lang3/StringUtils.html) for PrependIfMissing, which uses the plural form.
Let me know if it would be better if I changed the text as follows:
`* suffix - The string suffix to append to the end of the string.`
`* additionalsuffix - Optional - An additional string suffix that is a valid terminator.`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #516: METRON-830 Adding StringFunctions to Stellar - ...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/516
+1, thanks for the contribution
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #516: METRON-830 Adding StringFunctions to Stellar - ...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/516
This looks good to me, anand. Thanks so much for the contribution. I'll spin it up and try the functions out in a REPL. @mattf-horton @ottobackwards anything other outstanding concerns?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r110293191
--- Diff: metron-platform/metron-common/src/test/java/org/apache/metron/common/dsl/functions/StringFunctionsTest.java ---
@@ -207,4 +207,78 @@ public void testFormatWithNoArguments() throws Exception {
public void testFormatWithMissingArguments() throws Exception {
run("FORMAT('missing arg: %d')", Collections.emptyMap());
}
+
+
+ /**
+ * CHOP StringFunction
+ * @throws Exception
+ */
+ @Test
+ public void testChop() throws Exception {
+ Assert.assertEquals("ab", run("CHOP('abc')", new HashedMap()));
+ Assert.assertEquals(null, run("CHOP(null)", new HashedMap()));
+ Assert.assertEquals("abc", run("CHOP(msg)", ImmutableMap.of("msg", "abc\r\n")));
--- End diff --
Why does this work, when the string "abc\r\n" is never processed by any of the shell or java string input processors that might convert it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r110584819
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java ---
@@ -343,4 +343,89 @@ public Object apply(List<Object> args) {
return String.format(format, formatArgs);
}
}
+
+ @Stellar( name="CHOP"
--- End diff --
IMHO the `chop` function would be helpful when fine-tuning string extraction operations through stellar (E.g. when extracting specific variations of hostnames or URLs).
Aside of that, I went ahead and added the `chomp` string function as well since I agree that it would be of use.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r110629490
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java ---
@@ -343,4 +343,89 @@ public Object apply(List<Object> args) {
return String.format(format, formatArgs);
}
}
+
+ @Stellar( name="CHOP"
--- End diff --
That is not to say the CHOP cannot be useful however
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r110292164
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java ---
@@ -343,4 +343,89 @@ public Object apply(List<Object> args) {
return String.format(format, formatArgs);
}
}
+
+ @Stellar( name="CHOP"
+ , description = "Remove the last character from a String"
+ , params = { "the String to chop last character from, may be null"}
+ , returns = "String without last character, null if null String input"
+ )
+ public static class chop extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> strings) {
+
+ if(strings.size() == 0) {
+ throw new IllegalArgumentException("[CHOP] missing argument: string to be chopped");
+ }
+
+ String chop = StringUtils.chop((String) strings.get(0));
+ return chop;
+ }
+ }
+
+ @Stellar( name = "PREPENDIFMISSING"
+ , description = "Prepends the prefix to the start of the string if the string does not already start with any of the prefixes"
+ , params = { "str - The string.", "prefix - The prefix to prepend to the start of the string", "prefixes - Additional prefixes that are valid" }
+ , returns = "A new String if prefix was prepended, the same string otherwise."
+ )
+ public static class prependifmissing extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> strings) {
+
+ String prefixed;
+ switch (strings.size()) {
+ case 2: prefixed = org.apache.commons.lang3.StringUtils.prependIfMissing((String) strings.get(0), (String) strings.get(1));
+ break;
+ case 3: prefixed = org.apache.commons.lang3.StringUtils.prependIfMissing((String) strings.get(0), (String) strings.get(1), (String) strings.get(2));
+ break;
+ default: throw new IllegalArgumentException("[PREPENDIFMISSING] incorrect arguments. Usage: PREPENDIFMISSING <String> <prefix> [<prefix>...]");
+ }
+ return prefixed;
+ }
+ }
+
+ @Stellar( name = "APPENDIFMISSING"
+ , description = "Appends the suffix to the end of the string if the string does not already end with any of the suffixes"
+ , params = { "str - The string.", "suffix - The suffix to append to the end of the string", "suffixes - Additional suffixes that are valid terminators" }
+ , returns = "A new String if suffix was appended, the same string otherwise."
+ )
+ public static class appendifmissing extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> strings) {
+
+ String suffixed;
+ switch (strings.size()) {
+ case 2:
+ suffixed = org.apache.commons.lang3.StringUtils.appendIfMissing((String) strings.get(0), (String) strings.get(1));
+ break;
+ case 3:
+ suffixed = org.apache.commons.lang3.StringUtils.appendIfMissing((String) strings.get(0), (String) strings.get(1), (String) strings.get(2));
+ break;
+ default:
+ throw new IllegalArgumentException("[APPENDIFMISSING] incorrect arguments. Usage: APPENDIFMISSING <String> <prefix> [<prefix>...]");
+ }
+ return suffixed;
+ }
+ }
+
+ @Stellar( name = "COUNTMATCHES"
+ , description = "Counts how many times the substring appears in the larger string"
+ , params = { "str - the CharSequence to check, may be null", "sub - the substring to count, may be null"}
+ , returns = "the number of occurrences, 0 if either CharSequence is null"
--- End diff --
"the number of non-overlapping occurrences..."
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #516: METRON-830 Adding StringFunctions to Stellar - ...
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on the issue:
https://github.com/apache/incubator-metron/pull/516
+1, thanks Anand!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r110291091
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java ---
@@ -343,4 +343,89 @@ public Object apply(List<Object> args) {
return String.format(format, formatArgs);
}
}
+
+ @Stellar( name="CHOP"
+ , description = "Remove the last character from a String"
+ , params = { "the String to chop last character from, may be null"}
+ , returns = "String without last character, null if null String input"
+ )
+ public static class chop extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> strings) {
+
+ if(strings.size() == 0) {
+ throw new IllegalArgumentException("[CHOP] missing argument: string to be chopped");
+ }
+
+ String chop = StringUtils.chop((String) strings.get(0));
+ return chop;
+ }
+ }
+
+ @Stellar( name = "PREPENDIFMISSING"
+ , description = "Prepends the prefix to the start of the string if the string does not already start with any of the prefixes"
+ , params = { "str - The string.", "prefix - The prefix to prepend to the start of the string", "prefixes - Additional prefixes that are valid" }
+ , returns = "A new String if prefix was prepended, the same string otherwise."
+ )
+ public static class prependifmissing extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> strings) {
+
+ String prefixed;
+ switch (strings.size()) {
+ case 2: prefixed = org.apache.commons.lang3.StringUtils.prependIfMissing((String) strings.get(0), (String) strings.get(1));
+ break;
+ case 3: prefixed = org.apache.commons.lang3.StringUtils.prependIfMissing((String) strings.get(0), (String) strings.get(1), (String) strings.get(2));
+ break;
+ default: throw new IllegalArgumentException("[PREPENDIFMISSING] incorrect arguments. Usage: PREPENDIFMISSING <String> <prefix> [<prefix>...]");
--- End diff --
@cestella , isn't there boilerplate in the Stellar executor that automagically dumps the @Stellar declaration on any error? So one doesn't have to implement `Usage:` in the operator?
@anandsubbu , but you can still provide useful contextual info, like what arguments were actually provided, or in this case, at least how many (incorrect number) of arguments were provided.
The issues raised in PREPENDIFMISSING also apply to APPENDIFMISSING, of course.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r110965525
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java ---
@@ -343,4 +343,89 @@ public Object apply(List<Object> args) {
return String.format(format, formatArgs);
}
}
+
+ @Stellar( name="CHOP"
+ , description = "Remove the last character from a String"
+ , params = { "the String to chop last character from, may be null"}
+ , returns = "String without last character, null if null String input"
+ )
+ public static class chop extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> strings) {
+
+ if(strings.size() == 0) {
+ throw new IllegalArgumentException("[CHOP] missing argument: string to be chopped");
--- End diff --
Any thoughts about why treat nulls and empty strings differently?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r110964577
--- Diff: metron-platform/metron-common/README.md ---
@@ -434,6 +466,14 @@ The `!=` operator is the negation of the above.
* dateTime - The datetime as a long representing the milliseconds since unix epoch
* Returns: The current month (0-based).
+### `PREPEND_IF_MISSING`
+ * Description: Prepends the prefix to the start of the string if the string does not already start with any of the prefixes.
+ * Input:
+ * string - The string to be prepended.
+ * prefix - The string prefix to prepend to the start of the string.
+ * prefixes - Optional - Additional string prefixes that are valid.
--- End diff --
ditto
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r111075935
--- Diff: metron-platform/metron-common/README.md ---
@@ -167,6 +172,14 @@ The `!=` operator is the negation of the above.
| [ `WEEK_OF_YEAR`](#week_of_year) |
| [ `YEAR`](#year) |
+### `APPEND_IF_MISSING`
+ * Description: Appends the suffix to the end of the string if the string does not already end with any of the suffixes.
+ * Input:
+ * string - The string to be appended.
+ * suffix - The string suffix to append to the end of the string.
+ * suffixes - Optional - Additional string suffixes that are valid terminators.
--- End diff --
Okay, I will retain additionalsuffix for now. Created METRON-847 to track the enhancement.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r110581713
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java ---
@@ -343,4 +343,89 @@ public Object apply(List<Object> args) {
return String.format(format, formatArgs);
}
}
+
+ @Stellar( name="CHOP"
+ , description = "Remove the last character from a String"
+ , params = { "the String to chop last character from, may be null"}
+ , returns = "String without last character, null if null String input"
+ )
+ public static class chop extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> strings) {
+
+ if(strings.size() == 0) {
+ throw new IllegalArgumentException("[CHOP] missing argument: string to be chopped");
+ }
+
+ String chop = StringUtils.chop((String) strings.get(0));
+ return chop;
+ }
+ }
+
+ @Stellar( name = "PREPENDIFMISSING"
+ , description = "Prepends the prefix to the start of the string if the string does not already start with any of the prefixes"
+ , params = { "str - The string.", "prefix - The prefix to prepend to the start of the string", "prefixes - Additional prefixes that are valid" }
+ , returns = "A new String if prefix was prepended, the same string otherwise."
+ )
+ public static class prependifmissing extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> strings) {
+
+ String prefixed;
+ switch (strings.size()) {
+ case 2: prefixed = org.apache.commons.lang3.StringUtils.prependIfMissing((String) strings.get(0), (String) strings.get(1));
+ break;
+ case 3: prefixed = org.apache.commons.lang3.StringUtils.prependIfMissing((String) strings.get(0), (String) strings.get(1), (String) strings.get(2));
+ break;
+ default: throw new IllegalArgumentException("[PREPENDIFMISSING] incorrect arguments. Usage: PREPENDIFMISSING <String> <prefix> [<prefix>...]");
+ }
+ return prefixed;
+ }
+ }
+
+ @Stellar( name = "APPENDIFMISSING"
+ , description = "Appends the suffix to the end of the string if the string does not already end with any of the suffixes"
+ , params = { "str - The string.", "suffix - The suffix to append to the end of the string", "suffixes - Additional suffixes that are valid terminators" }
+ , returns = "A new String if suffix was appended, the same string otherwise."
+ )
+ public static class appendifmissing extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> strings) {
+
+ String suffixed;
+ switch (strings.size()) {
+ case 2:
+ suffixed = org.apache.commons.lang3.StringUtils.appendIfMissing((String) strings.get(0), (String) strings.get(1));
+ break;
+ case 3:
+ suffixed = org.apache.commons.lang3.StringUtils.appendIfMissing((String) strings.get(0), (String) strings.get(1), (String) strings.get(2));
+ break;
+ default:
+ throw new IllegalArgumentException("[APPENDIFMISSING] incorrect arguments. Usage: APPENDIFMISSING <String> <prefix> [<prefix>...]");
+ }
+ return suffixed;
+ }
+ }
+
+ @Stellar( name = "COUNTMATCHES"
+ , description = "Counts how many times the substring appears in the larger string"
+ , params = { "str - the CharSequence to check, may be null", "sub - the substring to count, may be null"}
+ , returns = "the number of occurrences, 0 if either CharSequence is null"
--- End diff --
Corrected.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/516
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r110290675
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java ---
@@ -343,4 +343,89 @@ public Object apply(List<Object> args) {
return String.format(format, formatArgs);
}
}
+
+ @Stellar( name="CHOP"
+ , description = "Remove the last character from a String"
+ , params = { "the String to chop last character from, may be null"}
+ , returns = "String without last character, null if null String input"
+ )
+ public static class chop extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> strings) {
+
+ if(strings.size() == 0) {
+ throw new IllegalArgumentException("[CHOP] missing argument: string to be chopped");
+ }
+
+ String chop = StringUtils.chop((String) strings.get(0));
+ return chop;
+ }
+ }
+
+ @Stellar( name = "PREPENDIFMISSING"
+ , description = "Prepends the prefix to the start of the string if the string does not already start with any of the prefixes"
+ , params = { "str - The string.", "prefix - The prefix to prepend to the start of the string", "prefixes - Additional prefixes that are valid" }
+ , returns = "A new String if prefix was prepended, the same string otherwise."
+ )
+ public static class prependifmissing extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> strings) {
+
+ String prefixed;
+ switch (strings.size()) {
+ case 2: prefixed = org.apache.commons.lang3.StringUtils.prependIfMissing((String) strings.get(0), (String) strings.get(1));
--- End diff --
It's not a bad thing to give the FQ class name for StringUtils -- I initially thought line 361 referred to metron-common/src/main/java/org/apache/metron/common/utils/StringUtils.java, and this makes clear it doesn't. But why use commons.lang.StringUtils in line 361 and commons.lang3.StringUtils here? Shouldn't they all just use lang3?
And you should *either* import the library and refer to it by short name, or not import it and refer to it by FQ class name, I think.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r110291263
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java ---
@@ -343,4 +343,89 @@ public Object apply(List<Object> args) {
return String.format(format, formatArgs);
}
}
+
+ @Stellar( name="CHOP"
+ , description = "Remove the last character from a String"
+ , params = { "the String to chop last character from, may be null"}
+ , returns = "String without last character, null if null String input"
+ )
+ public static class chop extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> strings) {
+
+ if(strings.size() == 0) {
+ throw new IllegalArgumentException("[CHOP] missing argument: string to be chopped");
+ }
+
+ String chop = StringUtils.chop((String) strings.get(0));
+ return chop;
+ }
+ }
+
+ @Stellar( name = "PREPENDIFMISSING"
--- End diff --
`PREPEND_IF_MISSING` would be more consistent with usual usage.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r110270759
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java ---
@@ -343,4 +343,89 @@ public Object apply(List<Object> args) {
return String.format(format, formatArgs);
}
}
+
+ @Stellar( name="CHOP"
+ , description = "Remove the last character from a String"
+ , params = { "the String to chop last character from, may be null"}
+ , returns = "String without last character, null if null String input"
+ )
+ public static class chop extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> strings) {
+
+ if(strings.size() == 0) {
+ throw new IllegalArgumentException("[CHOP] missing argument: string to be chopped");
--- End diff --
Seems like if an empty string gives IAE, then so should a null argument. But I could also support returning the empty string for an empty string, and null for null.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r110582745
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java ---
@@ -343,4 +343,89 @@ public Object apply(List<Object> args) {
return String.format(format, formatArgs);
}
}
+
+ @Stellar( name="CHOP"
+ , description = "Remove the last character from a String"
+ , params = { "the String to chop last character from, may be null"}
+ , returns = "String without last character, null if null String input"
+ )
+ public static class chop extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> strings) {
+
+ if(strings.size() == 0) {
+ throw new IllegalArgumentException("[CHOP] missing argument: string to be chopped");
+ }
+
+ String chop = StringUtils.chop((String) strings.get(0));
+ return chop;
+ }
+ }
+
+ @Stellar( name = "PREPENDIFMISSING"
+ , description = "Prepends the prefix to the start of the string if the string does not already start with any of the prefixes"
+ , params = { "str - The string.", "prefix - The prefix to prepend to the start of the string", "prefixes - Additional prefixes that are valid" }
--- End diff --
Sure, done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r110581665
--- Diff: metron-platform/metron-common/src/test/java/org/apache/metron/common/dsl/functions/StringFunctionsTest.java ---
@@ -207,4 +207,78 @@ public void testFormatWithNoArguments() throws Exception {
public void testFormatWithMissingArguments() throws Exception {
run("FORMAT('missing arg: %d')", Collections.emptyMap());
}
+
+
+ /**
+ * CHOP StringFunction
+ * @throws Exception
+ */
+ @Test
+ public void testChop() throws Exception {
+ Assert.assertEquals("ab", run("CHOP('abc')", new HashedMap()));
+ Assert.assertEquals(null, run("CHOP(null)", new HashedMap()));
+ Assert.assertEquals("abc", run("CHOP(msg)", ImmutableMap.of("msg", "abc\r\n")));
+ Assert.assertEquals("", run("CHOP(msg)", ImmutableMap.of("msg", "\n")));
+ }
+
+ @Test(expected = ParseException.class)
+ public void testChopWithMissingArguments() throws Exception {
+ run("CHOP()", Collections.emptyMap());
+ }
+
+ /**
+ * PREPENDIFMISSING StringFunction
+ * @throws Exception
+ */
+ @Test
+ public void testPrependIfMissing() throws Exception {
+ Assert.assertEquals("xyzabc", run("PREPENDIFMISSING('abc', 'xyz')", new HashedMap()));
+ Assert.assertEquals("xyzXYZabc", run("PREPENDIFMISSING('XYZabc', 'xyz', 'mno')", new HashedMap()));
+ Assert.assertEquals(null, run("PREPENDIFMISSING(null, null, null)", new HashedMap()));
+ Assert.assertEquals("xyz", run("PREPENDIFMISSING('', 'xyz', null)", new HashedMap()));
+ }
+
+ @Test(expected = ParseException.class)
+ public void testPrependIfMissingWithIncorrectArgs() throws Exception {
--- End diff --
Yup, you're right. Added try/catch blocks now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r110292921
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java ---
@@ -343,4 +343,89 @@ public Object apply(List<Object> args) {
return String.format(format, formatArgs);
}
}
+
+ @Stellar( name="CHOP"
+ , description = "Remove the last character from a String"
+ , params = { "the String to chop last character from, may be null"}
+ , returns = "String without last character, null if null String input"
+ )
+ public static class chop extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> strings) {
+
+ if(strings.size() == 0) {
+ throw new IllegalArgumentException("[CHOP] missing argument: string to be chopped");
+ }
+
+ String chop = StringUtils.chop((String) strings.get(0));
+ return chop;
+ }
+ }
+
+ @Stellar( name = "PREPENDIFMISSING"
+ , description = "Prepends the prefix to the start of the string if the string does not already start with any of the prefixes"
+ , params = { "str - The string.", "prefix - The prefix to prepend to the start of the string", "prefixes - Additional prefixes that are valid" }
+ , returns = "A new String if prefix was prepended, the same string otherwise."
+ )
+ public static class prependifmissing extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> strings) {
+
--- End diff --
Invalid Cast exception?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #516: METRON-830 Adding StringFunctions to Stellar - ...
Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on the issue:
https://github.com/apache/incubator-metron/pull/516
Hi @mattf-horton and @ottobackwards , thank you for the review. Please see review comments addressed as a part of my latest commit, and also responses inline.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r110294160
--- Diff: metron-platform/metron-common/src/test/java/org/apache/metron/common/dsl/functions/StringFunctionsTest.java ---
@@ -207,4 +207,78 @@ public void testFormatWithNoArguments() throws Exception {
public void testFormatWithMissingArguments() throws Exception {
run("FORMAT('missing arg: %d')", Collections.emptyMap());
}
+
+
+ /**
+ * CHOP StringFunction
+ * @throws Exception
+ */
+ @Test
+ public void testChop() throws Exception {
+ Assert.assertEquals("ab", run("CHOP('abc')", new HashedMap()));
+ Assert.assertEquals(null, run("CHOP(null)", new HashedMap()));
+ Assert.assertEquals("abc", run("CHOP(msg)", ImmutableMap.of("msg", "abc\r\n")));
+ Assert.assertEquals("", run("CHOP(msg)", ImmutableMap.of("msg", "\n")));
+ }
+
+ @Test(expected = ParseException.class)
+ public void testChopWithMissingArguments() throws Exception {
+ run("CHOP()", Collections.emptyMap());
+ }
+
+ /**
+ * PREPENDIFMISSING StringFunction
+ * @throws Exception
+ */
+ @Test
+ public void testPrependIfMissing() throws Exception {
+ Assert.assertEquals("xyzabc", run("PREPENDIFMISSING('abc', 'xyz')", new HashedMap()));
+ Assert.assertEquals("xyzXYZabc", run("PREPENDIFMISSING('XYZabc', 'xyz', 'mno')", new HashedMap()));
+ Assert.assertEquals(null, run("PREPENDIFMISSING(null, null, null)", new HashedMap()));
+ Assert.assertEquals("xyz", run("PREPENDIFMISSING('', 'xyz', null)", new HashedMap()));
+ }
+
+ @Test(expected = ParseException.class)
+ public void testPrependIfMissingWithIncorrectArgs() throws Exception {
+ run("PREPENDIFMISSING()", Collections.emptyMap());
+ run("PREPENDIFMISSING('abc')", Collections.emptyMap());
+ run("PREPENDIFMISSING('abc', 'def', 'ghi', 'jkl')", Collections.emptyMap());
+ }
+
+ /**
+ * APPENDIFMISSING StringFunction
--- End diff --
Same issues here of course, and in COUNTMATCHES.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #516: METRON-830 Adding StringFunctions to Stellar - ...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/516
Thanks for the contribution!
Quick review, first thing please follow the style guide for naming classes. You can see from the other stellar function classes.
It would be good to have some tests that aren't with string or null, to verify the behavior.
What happens if I pass in a number literal?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/516#discussion_r110965812
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java ---
@@ -343,4 +343,120 @@ public Object apply(List<Object> args) {
return String.format(format, formatArgs);
}
}
+
+ @Stellar( name="CHOMP"
+ , description = "Removes one newline from end of a String if it's there, otherwise leave it alone. A newline is \"\\n\", \"\\r\", or \"\\r\\n\""
+ , params = { "the String to chomp a newline from, may be null"}
+ , returns = "String without newline, null if null String input"
+ )
+ public static class Chomp extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> strings) {
+
+ if(strings.size() == 0) {
+ throw new IllegalArgumentException("[CHOMP] missing argument: string to chomp a newline from");
+ }
+
+ String chomp = StringUtils.chomp((String) strings.get(0));
+ return chomp;
+ }
+ }
+ @Stellar( name="CHOP"
+ , description = "Remove the last character from a String"
+ , params = { "the String to chop last character from, may be null"}
+ , returns = "String without last character, null if null String input"
+ )
+ public static class Chop extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> strings) {
+
+ if(strings.size() == 0) {
+ throw new IllegalArgumentException("[CHOP] missing argument: string to be chopped");
+ }
+
+ String chop = StringUtils.chop((String) strings.get(0));
+ return chop;
+ }
+ }
+
+ @Stellar( name = "PREPEND_IF_MISSING"
+ , description = "Prepends the prefix to the start of the string if the string does not already start with any of the prefixes"
+ , params = {
+ "str - The string."
+ , "prefix - The string prefix to prepend to the start of the string"
+ , "prefixes - Optional - Additional string prefixes that are valid"
+ }
+ , returns = "A new String if prefix was prepended, the same string otherwise."
+ )
+ public static class PrependIfMissing extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> strings) {
+
+ String prefixed;
+ switch (strings.size()) {
+ case 2: prefixed = StringUtils.prependIfMissing((String) strings.get(0), (String) strings.get(1));
+ break;
+ case 3: prefixed = StringUtils.prependIfMissing((String) strings.get(0), (String) strings.get(1), (String) strings.get(2));
+ break;
+ default: throw new IllegalArgumentException("[PREPEND_IF_MISSING] incorrect arguments: " + strings.toString() + "\nUsage: PREPEND_IF_MISSING <String> <prefix> [<prefix>...]");
+ }
+ return prefixed;
+ }
+ }
+
+ @Stellar( name = "APPEND_IF_MISSING"
+ , description = "Appends the suffix to the end of the string if the string does not already end with any of the suffixes"
+ , params = {
+ "str - The string."
+ , "suffix - The string suffix to append to the end of the string"
+ , "suffixes - Optional - Additional string suffixes that are valid terminators"
--- End diff --
Same doc concern as in the .md file: Is "suffixes" a string, a regex, an array of strings, or what? If it is a single string, how does it represent plural suffixes?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---