You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@metron.apache.org by ma...@apache.org on 2017/04/27 22:13:36 UTC
[01/26] incubator-metron git commit: METRON-830 Adding
StringFunctions to Stellar - chop, prependifmissing,
appendifmissing and countmatches (anandsubbu via mattf-horton) closes
apache/incubator-metron#516 [Forced Update!]
Repository: incubator-metron
Updated Branches:
refs/heads/Metron_0.4.0 3e16292d9 -> 68bd6c520 (forced update)
METRON-830 Adding StringFunctions to Stellar - chop, prependifmissing, appendifmissing and countmatches (anandsubbu via mattf-horton) closes apache/incubator-metron#516
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/e6992d56
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/e6992d56
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/e6992d56
Branch: refs/heads/Metron_0.4.0
Commit: e6992d56816c37f8f63274a5a1ce33d37d69dc68
Parents: d30d8b9
Author: anandsubbu <as...@hortonworks.com>
Authored: Sun Apr 16 23:13:38 2017 -0700
Committer: mattf <ma...@apache.org>
Committed: Sun Apr 16 23:13:38 2017 -0700
----------------------------------------------------------------------
metron-platform/metron-common/README.md | 44 ++-
.../common/dsl/functions/StringFunctions.java | 132 ++++++++-
.../dsl/functions/StringFunctionsTest.java | 267 +++++++++++++++++--
3 files changed, 417 insertions(+), 26 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/e6992d56/metron-platform/metron-common/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/README.md b/metron-platform/metron-common/README.md
index 8b910f8..de3a1b7 100644
--- a/metron-platform/metron-common/README.md
+++ b/metron-platform/metron-common/README.md
@@ -99,11 +99,15 @@ In the core language functions, we support basic functional programming primitiv
| |
| ---------- |
| [ `ABS`](../../metron-analytics/metron-statistics#abs) |
+| [ `APPEND_IF_MISSING`](#append_if_missing) |
| [ `BIN`](../../metron-analytics/metron-statistics#bin) |
| [ `BLOOM_ADD`](#bloom_add) |
| [ `BLOOM_EXISTS`](#bloom_exists) |
| [ `BLOOM_INIT`](#bloom_init) |
| [ `BLOOM_MERGE`](#bloom_merge) |
+| [ `CHOP`](#chop) |
+| [ `CHOMP`](#chomp) |
+| [ `COUNT_MATCHES`](#count_matches) |
| [ `DAY_OF_MONTH`](#day_of_month) |
| [ `DAY_OF_WEEK`](#day_of_week) |
| [ `DAY_OF_YEAR`](#day_of_year) |
@@ -145,9 +149,10 @@ In the core language functions, we support basic functional programming primitiv
| [ `MAP`](#map) |
| [ `MAP_EXISTS`](#map_exists) |
| [ `MONTH`](#month) |
+| [ `PREPEND_IF_MISSING`](#prepend_if_missing) |
| [ `PROFILE_GET`](#profile_get) |
-| [ `PROFILE_FIXED`](#profile_fixed) |
-| [ `PROFILE_WINDOW`](#profile_window) |
+| [ `PROFILE_FIXED`](#profile_fixed) |
+| [ `PROFILE_WINDOW`](#profile_window) |
| [ `PROTOCOL_TO_NAME`](#protocol_to_name) |
| [ `REDUCE`](#reduce) |
| [ `REGEXP_MATCH`](#regexp_match) |
@@ -192,6 +197,14 @@ In the core language functions, we support basic functional programming primitiv
| [ `WEEK_OF_YEAR`](#week_of_year) |
| [ `YEAR`](#year) |
+### `APPEND_IF_MISSING`
+ * Description: Appends the suffix to the end of the string if the string does not already end with any of the suffixes.
+ * Input:
+ * string - The string to be appended.
+ * suffix - The string suffix to append to the end of the string.
+ * additionalsuffix - Optional - Additional string suffix that is a valid terminator.
+ * Returns: A new String if prefix was prepended, the same string otherwise.
+
### `BLOOM_ADD`
* Description: Adds an element to the bloom filter passed in
* Input:
@@ -219,6 +232,25 @@ In the core language functions, we support basic functional programming primitiv
* bloomfilters - A list of bloom filters to merge
* Returns: Bloom Filter or null if the list is empty
+### `CHOP`
+ * Description: Remove the last character from a String
+ * Input:
+ * string - the String to chop last character from, may be null
+ * Returns: String without last character, null if null String input
+
+### `CHOMP`
+ * Description: Removes one newline from end of a String if it's there, otherwise leave it alone. A newline is "\n", "\r", or "\r\n"
+ * Input:
+ * string - the String to chomp a newline from, may be null
+ * Returns: String without newline, null if null String input
+
+### `COUNT_MATCHES`
+ * Description: Counts how many times the substring appears in the larger string.
+ * Input:
+ * string - the CharSequence to check, may be null.
+ * substring/character - the substring or character to count, may be null.
+ * Returns: the number of non-overlapping occurrences, 0 if either CharSequence is null.
+
### `DAY_OF_MONTH`
* Description: The numbered day within the month. The first day within the month has a value of 1.
* Input:
@@ -480,6 +512,14 @@ In the core language functions, we support basic functional programming primitiv
* dateTime - The datetime as a long representing the milliseconds since unix epoch
* Returns: The current month (0-based).
+### `PREPEND_IF_MISSING`
+ * Description: Prepends the prefix to the start of the string if the string does not already start with any of the prefixes.
+ * Input:
+ * string - The string to be prepended.
+ * prefix - The string prefix to prepend to the start of the string.
+ * additionalprefix - Optional - Additional string prefix that is valid.
+ * Returns: A new String if prefix was prepended, the same string otherwise.
+
### `PROFILE_GET`
* Description: Retrieves a series of values from a stored profile.
* Input:
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/e6992d56/metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java
index 1086da3..1ae790d 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java
@@ -21,7 +21,7 @@ package org.apache.metron.common.dsl.functions;
import com.google.common.base.Joiner;
import com.google.common.base.Splitter;
import com.google.common.collect.Iterables;
-import org.apache.commons.lang.StringUtils;
+import org.apache.commons.lang3.StringUtils;
import org.apache.metron.common.dsl.BaseStellarFunction;
import org.apache.metron.common.dsl.ParseException;
import org.apache.metron.common.dsl.Stellar;
@@ -343,4 +343,134 @@ public class StringFunctions {
return String.format(format, formatArgs);
}
}
+
+ @Stellar( name="CHOMP"
+ , description = "Removes one newline from end of a String if it's there, otherwise leave it alone. A newline is \"\\n\", \"\\r\", or \"\\r\\n\""
+ , params = { "the String to chomp a newline from, may be null"}
+ , returns = "String without newline, null if null String input"
+ )
+ public static class Chomp extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> strings) {
+
+ if(strings == null || strings.size() == 0 ) {
+ throw new IllegalArgumentException("[CHOMP] missing argument: string to be chopped");
+ }
+ String var = strings.get(0) == null?null: (String) strings.get(0);
+ if(var == null) {
+ return null;
+ }
+ else if(var.length() == 0) {
+ return var;
+ }
+ else {
+ return StringUtils.chomp(var);
+ }
+ }
+ }
+ @Stellar( name="CHOP"
+ , description = "Remove the last character from a String"
+ , params = { "the String to chop last character from, may be null"}
+ , returns = "String without last character, null if null String input"
+ )
+ public static class Chop extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> strings) {
+
+ if(strings == null || strings.size() == 0 ) {
+ throw new IllegalArgumentException("[CHOP] missing argument: string to be chopped");
+ }
+ String var = strings.get(0) == null?null: (String) strings.get(0);
+ if(var == null) {
+ return null;
+ }
+ else if(var.length() == 0) {
+ return var;
+ }
+ else {
+ return StringUtils.chop(var);
+ }
+ }
+ }
+
+ @Stellar( name = "PREPEND_IF_MISSING"
+ , description = "Prepends the prefix to the start of the string if the string does not already start with any of the prefixes"
+ , params = {
+ "str - The string."
+ , "prefix - The string prefix to prepend to the start of the string"
+ , "additionalprefix - Optional - Additional string prefix that is valid"
+ }
+ , returns = "A new String if prefix was prepended, the same string otherwise."
+ )
+ public static class PrependIfMissing extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> strings) {
+
+ String prefixed;
+ switch (strings.size()) {
+ case 2: prefixed = StringUtils.prependIfMissing((String) strings.get(0), (String) strings.get(1));
+ break;
+ case 3: prefixed = StringUtils.prependIfMissing((String) strings.get(0), (String) strings.get(1), (String) strings.get(2));
+ break;
+ default: throw new IllegalArgumentException("[PREPEND_IF_MISSING] incorrect arguments: " + strings.toString() + "\nUsage: PREPEND_IF_MISSING <String> <prefix> [<prefix>...]");
+ }
+ return prefixed;
+ }
+ }
+
+ @Stellar( name = "APPEND_IF_MISSING"
+ , description = "Appends the suffix to the end of the string if the string does not already end with any of the suffixes"
+ , params = {
+ "str - The string."
+ , "suffix - The string suffix to append to the end of the string"
+ , "additionalsuffix - Optional - Additional string suffix that is a valid terminator"
+ }
+ , returns = "A new String if suffix was appended, the same string otherwise."
+ )
+ public static class AppendIfMissing extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> strings) {
+
+ String suffixed;
+ switch (strings.size()) {
+ case 2:
+ suffixed = StringUtils.appendIfMissing((String) strings.get(0), (String) strings.get(1));
+ break;
+ case 3:
+ suffixed = StringUtils.appendIfMissing((String) strings.get(0), (String) strings.get(1), (String) strings.get(2));
+ break;
+ default:
+ throw new IllegalArgumentException("[APPEND_IF_MISSING] incorrect arguments. Usage: APPEND_IF_MISSING <String> <prefix> [<prefix>...]");
+ }
+ return suffixed;
+ }
+ }
+
+ @Stellar( name = "COUNT_MATCHES"
+ , description = "Counts how many times the substring appears in the larger string"
+ , params = {
+ "str - the CharSequence to check, may be null"
+ , "sub - the substring to count, may be null"
+ }
+ , returns = "the number of non-overlapping occurrences, 0 if either CharSequence is null"
+ )
+ public static class CountMatches extends BaseStellarFunction {
+
+ @Override
+ public Object apply(List<Object> strings) {
+
+ if(strings.size() != 2) {
+ throw new IllegalArgumentException("[COUNT_MATCHES] incorrect arguments. Usage: COUNT_MATCHES <String> <substring>");
+ }
+
+ int matchcount;
+ matchcount = StringUtils.countMatches((String) strings.get(0), (String) strings.get(1));
+ return matchcount;
+ }
+ }
+
}
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/e6992d56/metron-platform/metron-common/src/test/java/org/apache/metron/common/dsl/functions/StringFunctionsTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/test/java/org/apache/metron/common/dsl/functions/StringFunctionsTest.java b/metron-platform/metron-common/src/test/java/org/apache/metron/common/dsl/functions/StringFunctionsTest.java
index d8854f8..8899535 100644
--- a/metron-platform/metron-common/src/test/java/org/apache/metron/common/dsl/functions/StringFunctionsTest.java
+++ b/metron-platform/metron-common/src/test/java/org/apache/metron/common/dsl/functions/StringFunctionsTest.java
@@ -67,40 +67,40 @@ public class StringFunctionsTest {
}
@Test
- public void testLeftRightFills() throws Exception{
+ public void testLeftRightFills() throws Exception {
final Map<String, Object> variableMap = new HashMap<String, Object>() {{
put("foo", null);
put("bar", null);
- put("notInt","oh my");
+ put("notInt", "oh my");
}};
//LEFT
- Object left = run("FILL_LEFT('123','X', 10)",new HashedMap());
+ Object left = run("FILL_LEFT('123','X', 10)", new HashedMap());
Assert.assertNotNull(left);
- Assert.assertEquals(10,((String)left).length());
- Assert.assertEquals("XXXXXXX123",(String)left);
+ Assert.assertEquals(10, ((String) left).length());
+ Assert.assertEquals("XXXXXXX123", (String) left);
//RIGHT
Object right = run("FILL_RIGHT('123','X', 10)", new HashedMap());
Assert.assertNotNull(right);
- Assert.assertEquals(10,((String)right).length());
- Assert.assertEquals("123XXXXXXX",(String)right);
+ Assert.assertEquals(10, ((String) right).length());
+ Assert.assertEquals("123XXXXXXX", (String) right);
//INPUT ALREADY LENGTH
Object same = run("FILL_RIGHT('123','X', 3)", new HashedMap());
- Assert.assertEquals(3,((String)same).length());
- Assert.assertEquals("123",(String)same);
+ Assert.assertEquals(3, ((String) same).length());
+ Assert.assertEquals("123", (String) same);
//INPUT BIGGER THAN LENGTH
Object tooBig = run("FILL_RIGHT('1234567890','X', 3)", new HashedMap());
- Assert.assertEquals(10,((String)tooBig).length());
- Assert.assertEquals("1234567890",(String)tooBig);
+ Assert.assertEquals(10, ((String) tooBig).length());
+ Assert.assertEquals("1234567890", (String) tooBig);
//NULL VARIABLES
boolean thrown = false;
- try{
+ try {
run("FILL_RIGHT('123',foo,bar)", variableMap);
- }catch(ParseException pe) {
+ } catch (ParseException pe) {
thrown = true;
Assert.assertTrue(pe.getMessage().contains("are both required"));
}
@@ -108,9 +108,9 @@ public class StringFunctionsTest {
thrown = false;
// NULL LENGTH
- try{
+ try {
run("FILL_RIGHT('123','X',bar)", variableMap);
- }catch(ParseException pe) {
+ } catch (ParseException pe) {
thrown = true;
Assert.assertTrue(pe.getMessage().contains("are both required"));
}
@@ -118,9 +118,9 @@ public class StringFunctionsTest {
thrown = false;
// NULL FILL
- try{
+ try {
run("FILL_RIGHT('123',foo, 7)", variableMap);
- }catch(ParseException pe) {
+ } catch (ParseException pe) {
thrown = true;
Assert.assertTrue(pe.getMessage().contains("are both required"));
}
@@ -130,7 +130,7 @@ public class StringFunctionsTest {
// NON INTEGER LENGTH
try {
run("FILL_RIGHT('123','X', 'z' )", new HashedMap());
- }catch(ParseException pe){
+ } catch (ParseException pe) {
thrown = true;
Assert.assertTrue(pe.getMessage().contains("not a valid Integer"));
}
@@ -140,7 +140,7 @@ public class StringFunctionsTest {
// EMPTY STRING PAD
try {
Object returnValue = run("FILL_RIGHT('123','', 10 )", new HashedMap());
- }catch(ParseException pe) {
+ } catch (ParseException pe) {
thrown = true;
Assert.assertTrue(pe.getMessage().contains("cannot be an empty"));
}
@@ -150,7 +150,7 @@ public class StringFunctionsTest {
//MISSING LENGTH PARAMETER
try {
run("FILL_RIGHT('123',foo)", variableMap);
- }catch(ParseException pe){
+ } catch (ParseException pe) {
thrown = true;
Assert.assertTrue(pe.getMessage().contains("expects three"));
}
@@ -160,8 +160,8 @@ public class StringFunctionsTest {
@Test
public void shannonEntropyTest() throws Exception {
//test empty string
- Assert.assertEquals(0.0, (Double)run("STRING_ENTROPY('')", new HashMap<>()), 0.0);
- Assert.assertEquals(0.0, (Double)run("STRING_ENTROPY(foo)", ImmutableMap.of("foo", "")), 0.0);
+ Assert.assertEquals(0.0, (Double) run("STRING_ENTROPY('')", new HashMap<>()), 0.0);
+ Assert.assertEquals(0.0, (Double) run("STRING_ENTROPY(foo)", ImmutableMap.of("foo", "")), 0.0);
/*
Now consider the string aaaaaaaaaabbbbbccccc or 10 a's followed by 5 b's and 5 c's.
@@ -173,7 +173,7 @@ public class StringFunctionsTest {
-p(a)*log_2(p(a)) - p(b)*log_2(p(b)) - p(c)*log_2(p(c)) =
-0.5*-1 - 0.25*-2 - 0.25*-2 = 1.5
*/
- Assert.assertEquals(1.5, (Double)run("STRING_ENTROPY(foo)", ImmutableMap.of("foo", "aaaaaaaaaabbbbbccccc")), 0.0);
+ Assert.assertEquals(1.5, (Double) run("STRING_ENTROPY(foo)", ImmutableMap.of("foo", "aaaaaaaaaabbbbbccccc")), 0.0);
}
@Test
@@ -207,4 +207,225 @@ public class StringFunctionsTest {
public void testFormatWithMissingArguments() throws Exception {
run("FORMAT('missing arg: %d')", Collections.emptyMap());
}
+
+
+ /**
+ * CHOMP StringFunction
+ *
+ * @throws Exception
+ */
+ @Test
+ public void testChomp() throws Exception {
+ Assert.assertEquals("abc", run("CHOMP('abc')", new HashedMap()));
+ Assert.assertEquals("abc", run("CHOMP(msg)", ImmutableMap.of("msg", "abc\r\n")));
+ Assert.assertEquals("", run("CHOMP(msg)", ImmutableMap.of("msg", "\n")));
+ Assert.assertEquals("", run("CHOMP('')", new HashedMap()));
+ Assert.assertEquals(null, run("CHOMP(msg)", new HashedMap()));
+ Assert.assertEquals(null, run("CHOMP(null)", new HashedMap()));
+
+ // No input
+ boolean thrown = false;
+ try {
+ run("CHOMP()", Collections.emptyMap());
+ } catch (ParseException pe) {
+ thrown = true;
+ Assert.assertTrue(pe.getMessage().contains("missing argument"));
+ }
+ Assert.assertTrue(thrown);
+ thrown = false;
+
+ // Integer input
+ try {
+ run("CHOMP(123)", Collections.emptyMap());
+ } catch (ParseException pe) {
+ thrown = true;
+ Assert.assertTrue(pe.getMessage().contains("cannot be cast"));
+ }
+ Assert.assertTrue(thrown);
+
+ }
+
+ /**
+ * CHOP StringFunction
+ *
+ * @throws Exception
+ */
+ @Test
+ public void testChop() throws Exception {
+ Assert.assertEquals("ab", run("CHOP('abc')", new HashedMap()));
+ Assert.assertEquals(null, run("CHOP(null)", new HashedMap()));
+ Assert.assertEquals(null, run("CHOP(msg)", new HashedMap()));
+ Assert.assertEquals("abc", run("CHOP(msg)", ImmutableMap.of("msg", "abc\r\n")));
+ Assert.assertEquals("", run("CHOP(msg)", ImmutableMap.of("msg", "")));
+ Assert.assertEquals("", run("CHOP(msg)", ImmutableMap.of("msg", "\n")));
+ Assert.assertEquals("", run("CHOP('')", new HashedMap()));
+
+ // No input
+ boolean thrown = false;
+ try {
+ run("CHOP()", Collections.emptyMap());
+ } catch (ParseException pe) {
+ thrown = true;
+ Assert.assertTrue(pe.getMessage().contains("missing argument"));
+ }
+ Assert.assertTrue(thrown);
+ thrown = false;
+
+ // Integer input
+ try {
+ run("CHOP(123)", Collections.emptyMap());
+ } catch (ParseException pe) {
+ thrown = true;
+ Assert.assertTrue(pe.getMessage().contains("cannot be cast"));
+ }
+ Assert.assertTrue(thrown);
+
+ }
+
+ /**
+ * PREPEND_IF_MISSING StringFunction
+ */
+ @Test
+ public void testPrependIfMissing() throws Exception {
+ Assert.assertEquals("xyzabc", run("PREPEND_IF_MISSING('abc', 'xyz')", new HashedMap()));
+ Assert.assertEquals("xyzXYZabc", run("PREPEND_IF_MISSING('XYZabc', 'xyz', 'mno')", new HashedMap()));
+ Assert.assertEquals("mnoXYZabc", run("PREPEND_IF_MISSING('mnoXYZabc', 'xyz', 'mno')", new HashedMap()));
+ Assert.assertEquals(null, run("PREPEND_IF_MISSING(null, null, null)", new HashedMap()));
+ Assert.assertEquals("xyz", run("PREPEND_IF_MISSING('', 'xyz', null)", new HashedMap()));
+
+ // No input
+ boolean thrown = false;
+ try {
+ run("PREPEND_IF_MISSING()", Collections.emptyMap());
+ } catch (ParseException pe) {
+ thrown = true;
+ Assert.assertTrue(pe.getMessage().contains("incorrect arguments"));
+ }
+ Assert.assertTrue(thrown);
+ thrown = false;
+
+ // Incorrect number of arguments - 1
+ try {
+ run("PREPEND_IF_MISSING('abc')", Collections.emptyMap());
+ } catch (ParseException pe) {
+ thrown = true;
+ Assert.assertTrue(pe.getMessage().contains("incorrect arguments"));
+ }
+ Assert.assertTrue(thrown);
+ thrown = false;
+
+ // Incorrect number of arguments - 2
+ try {
+ run("PREPEND_IF_MISSING('abc', 'def', 'ghi', 'jkl')", Collections.emptyMap());
+ } catch (ParseException pe) {
+ thrown = true;
+ Assert.assertTrue(pe.getMessage().contains("incorrect arguments"));
+ }
+ Assert.assertTrue(thrown);
+ thrown = false;
+
+ // Integer input
+ try {
+ run("PREPEND_IF_MISSING(123, 'abc')", Collections.emptyMap());
+ } catch (ParseException pe) {
+ thrown = true;
+ Assert.assertTrue(pe.getMessage().contains("cannot be cast"));
+ }
+ Assert.assertTrue(thrown);
+
+ }
+
+ /**
+ * APPEND_IF_MISSING StringFunction
+ */
+ @Test
+ public void testAppendIfMissing() throws Exception {
+ Assert.assertEquals("apachemetron", run("APPEND_IF_MISSING('apache', 'metron')", new HashedMap()));
+ Assert.assertEquals("abcXYZxyz", run("APPEND_IF_MISSING('abcXYZ', 'xyz', 'mno')", new HashedMap()));
+ Assert.assertEquals(null, run("APPEND_IF_MISSING(null, null, null)", new HashedMap()));
+ Assert.assertEquals("xyz", run("APPEND_IF_MISSING('', 'xyz', null)", new HashedMap()));
+
+ // No input
+ boolean thrown = false;
+ try {
+ run("APPEND_IF_MISSING()", Collections.emptyMap());
+ } catch (ParseException pe) {
+ thrown = true;
+ Assert.assertTrue(pe.getMessage().contains("incorrect arguments"));
+ }
+ Assert.assertTrue(thrown);
+ thrown = false;
+
+ // Incorrect number of arguments - 1
+ try {
+ run("APPEND_IF_MISSING('abc')", Collections.emptyMap());
+ } catch (ParseException pe) {
+ thrown = true;
+ Assert.assertTrue(pe.getMessage().contains("incorrect arguments"));
+ }
+ Assert.assertTrue(thrown);
+ thrown = false;
+
+ // Incorrect number of arguments - 2
+ try {
+ run("APPEND_IF_MISSING('abc', 'def', 'ghi', 'jkl')", Collections.emptyMap());
+ } catch (ParseException pe) {
+ thrown = true;
+ Assert.assertTrue(pe.getMessage().contains("incorrect arguments"));
+ }
+ Assert.assertTrue(thrown);
+ thrown = false;
+
+ // Integer input
+ try {
+ run("APPEND_IF_MISSING(123, 'abc')", Collections.emptyMap());
+ } catch (ParseException pe) {
+ thrown = true;
+ Assert.assertTrue(pe.getMessage().contains("cannot be cast"));
+ }
+ Assert.assertTrue(thrown);
+
+ }
+
+ /**
+ * COUNT_MATCHES StringFunction
+ */
+ @Test
+ public void testCountMatches() throws Exception {
+ Assert.assertEquals(0, (int) run("COUNT_MATCHES(null, '*')", new HashedMap()));
+ Assert.assertEquals(2, (int) run("COUNT_MATCHES('apachemetron', 'e')", new HashedMap()));
+ Assert.assertEquals(2, (int) run("COUNT_MATCHES('anand', 'an')", new HashedMap()));
+ Assert.assertEquals(0, (int) run("COUNT_MATCHES('abcd', null)", new HashedMap()));
+
+ // No input
+ boolean thrown = false;
+ try {
+ run("COUNT_MATCHES()", Collections.emptyMap());
+ } catch (ParseException pe) {
+ thrown = true;
+ Assert.assertTrue(pe.getMessage().contains("incorrect arguments"));
+ }
+ Assert.assertTrue(thrown);
+ thrown = false;
+
+ // Incorrect number of arguments - 1
+ try {
+ run("COUNT_MATCHES('abc')", Collections.emptyMap());
+ } catch (ParseException pe) {
+ thrown = true;
+ Assert.assertTrue(pe.getMessage().contains("incorrect arguments"));
+ }
+ Assert.assertTrue(thrown);
+ thrown = false;
+
+ // Integer input
+ try {
+ run("COUNT_MATCHES(123, 456)", Collections.emptyMap());
+ } catch (ParseException pe) {
+ thrown = true;
+ Assert.assertTrue(pe.getMessage().contains("cannot be cast"));
+ }
+ Assert.assertTrue(thrown);
+
+ }
}
[11/26] incubator-metron git commit: The MPack should function in a
kerberized cluster (justinleet) closes apache/incubator-metron#518
Posted by ma...@apache.org.
The MPack should function in a kerberized cluster (justinleet) closes apache/incubator-metron#518
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/41b5b105
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/41b5b105
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/41b5b105
Branch: refs/heads/Metron_0.4.0
Commit: 41b5b1050cf29e5dcb5d0c36b5a9dbd1cafa745e
Parents: 55062fb
Author: justinjleet <ju...@gmail.com>
Authored: Mon Apr 17 15:16:49 2017 -0400
Committer: leet <le...@apache.org>
Committed: Thu Apr 20 16:03:29 2017 -0400
----------------------------------------------------------------------
metron-deployment/Kerberos-manual-setup.md | 217 +++++++++++++++++++
metron-deployment/README.md | 15 ++
.../common-services/METRON/CURRENT/metainfo.xml | 4 +-
.../package/scripts/enrichment_commands.py | 11 +
.../CURRENT/package/scripts/metron_client.py | 2 +-
.../CURRENT/package/scripts/metron_service.py | 16 --
.../CURRENT/package/scripts/parser_commands.py | 9 +-
.../package/templates/client_jaas.conf.j2 | 2 +-
.../CURRENT/package/templates/storm.config.j2 | 2 +-
.../ambari-storm-site-properties.png | Bin 0 -> 49605 bytes
.../readme-images/ambari-storm-site.png | Bin 0 -> 134251 bytes
.../readme-images/custom-storm-site-final.png | Bin 0 -> 128117 bytes
.../enable-kerberos-configure-kerberos.png | Bin 0 -> 136196 bytes
.../readme-images/enable-kerberos-started.png | Bin 0 -> 149382 bytes
.../readme-images/enable-kerberos.png | Bin 0 -> 16497 bytes
metron-deployment/vagrant/Kerberos-setup.md | 200 ++---------------
.../ambari-storm-site-properties.png | Bin 49605 -> 0 bytes
.../vagrant/readme-images/ambari-storm-site.png | Bin 134251 -> 0 bytes
.../readme-images/custom-storm-site-final.png | Bin 128117 -> 0 bytes
.../enable-kerberos-configure-kerberos.png | Bin 136196 -> 0 bytes
.../readme-images/enable-kerberos-started.png | Bin 149382 -> 0 bytes
.../vagrant/readme-images/enable-kerberos.png | Bin 16497 -> 0 bytes
22 files changed, 269 insertions(+), 209 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/Kerberos-manual-setup.md
----------------------------------------------------------------------
diff --git a/metron-deployment/Kerberos-manual-setup.md b/metron-deployment/Kerberos-manual-setup.md
new file mode 100644
index 0000000..4eaa725
--- /dev/null
+++ b/metron-deployment/Kerberos-manual-setup.md
@@ -0,0 +1,217 @@
+# Setting Up Kerberos outside of an Ambari Management Pack
+The Ambari Management pack will manage Kerberization when used.
+**Note:** These are instructions for Kerberizing Metron Storm topologies from Kafka to Kafka. This does not cover the sensor connections or MAAS.
+General Kerberization notes can be found in the metron-deployment [README.md](README.md)
+
+## Setup the KDC
+See [Setup the KDC](vagrant/Kerberos-setup.md)
+
+4. Setup the admin and metron user principals. You'll kinit as the metron user when running topologies. Make sure to remember the passwords.
+ ```
+ kadmin.local -q "addprinc admin/admin"
+ kadmin.local -q "addprinc metron"
+ ```
+
+## Kerberize Metron
+
+1. Stop all topologies - we will restart them again once Kerberos has been enabled.
+ ```
+ for topology in bro snort enrichment indexing; do storm kill $topology; done
+ ```
+
+2. Create the metron user HDFS home directory
+ ```
+ sudo -u hdfs hdfs dfs -mkdir /user/metron && \
+ sudo -u hdfs hdfs dfs -chown metron:hdfs /user/metron && \
+ sudo -u hdfs hdfs dfs -chmod 770 /user/metron
+ ```
+
+3. In [Ambari](http://node1:8080), setup Storm to run with Kerberos and run worker jobs as the submitting user:
+
+ a. Add the following properties to custom storm-site:
+ ```
+ topology.auto-credentials=['org.apache.storm.security.auth.kerberos.AutoTGT']
+ nimbus.credential.renewers.classes=['org.apache.storm.security.auth.kerberos.AutoTGT']
+ supervisor.run.worker.as.user=true
+ ```
+
+ b. In the Storm config section in Ambari, choose \u201cAdd Property\u201d under custom storm-site:
+
+ 
+
+ c. In the dialog window, choose the \u201cbulk property add mode\u201d toggle button and add the below values:
+
+ 
+
+4. Kerberize the cluster via Ambari. More detailed documentation can be found [here](http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html).
+
+ a. For this exercise, choose existing MIT KDC (this is what we setup and installed in the previous steps.)
+
+ 
+
+ 
+
+ b. Setup Kerberos configuration. Realm is EXAMPLE.COM. The admin principal will end up as admin/admin@EXAMPLE.COM when testing the KDC. Use the password you entered during the step for adding the admin principal.
+
+ 
+
+ c. Click through to \u201cStart and Test Services.\u201d Let the cluster spin up, but don't worry about starting up Metron via Ambari - we're going to run the parsers manually against the rest of the Hadoop cluster Kerberized. The wizard will fail at starting Metron, but this is OK. Click \u201ccontinue.\u201d When you\u2019re finished, the custom storm-site should look similar to the following:
+
+ 
+
+5. Setup Metron keytab
+ ```
+ kadmin.local -q "ktadd -k metron.headless.keytab metron@EXAMPLE.COM" && \
+ cp metron.headless.keytab /etc/security/keytabs && \
+ chown metron:hadoop /etc/security/keytabs/metron.headless.keytab && \
+ chmod 440 /etc/security/keytabs/metron.headless.keytab
+ ```
+
+6. Kinit with the metron user
+ ```
+ kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+ ```
+
+7. First create any additional Kafka topics you will need. We need to create the topics before adding the required ACLs. The current full dev installation will deploy bro, snort, enrichments, and indexing only. e.g.
+ ```
+ ${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper ${ZOOKEEPER}:2181 --create --topic yaf --partitions 1 --replication-factor 1
+ ```
+
+8. Setup Kafka ACLs for the topics
+ ```
+ export KERB_USER=metron
+ for topic in bro enrichments indexing snort; do
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --topic ${topic}
+ done
+ ```
+
+9. Setup Kafka ACLs for the consumer groups
+ ```
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group bro_parser
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group snort_parser
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group yaf_parser
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group enrichments
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group indexing
+ ```
+
+10. Add metron user to the Kafka cluster ACL
+ ```
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --cluster kafka-cluster
+ ```
+
+11. We also need to grant permissions to the HBase tables. Kinit as the hbase user and add ACLs for metron.
+ ```
+ kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-metron_cluster@EXAMPLE.COM
+ echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
+ echo "grant 'metron', 'RW', 'enrichment'" | hbase shell
+ ```
+
+12. Create a \u201c.storm\u201d directory in the metron user\u2019s home directory and switch to that directory.
+ ```
+ su metron
+ mkdir ~/.storm
+ cd ~/.storm
+ ```
+
+13. Create a custom client jaas file. This should look identical to the Storm client jaas file located in /etc/storm/conf/client_jaas.conf except for the addition of a Client stanza. The Client stanza is used for Zookeeper. All quotes and semicolons are necessary.
+ ```
+ cat << EOF > client_jaas.conf
+ StormClient {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useTicketCache=true
+ renewTicket=true
+ serviceName="nimbus";
+ };
+ Client {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ keyTab="/etc/security/keytabs/metron.headless.keytab"
+ storeKey=true
+ useTicketCache=false
+ serviceName="zookeeper"
+ principal="metron@EXAMPLE.COM";
+ };
+ KafkaClient {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ keyTab="/etc/security/keytabs/metron.headless.keytab"
+ storeKey=true
+ useTicketCache=false
+ serviceName="kafka"
+ principal="metron@EXAMPLE.COM";
+ };
+ EOF
+ ```
+
+14. Create a storm.yaml with jaas file info. Set the array of nimbus hosts accordingly.
+ ```
+ cat << EOF > storm.yaml
+ nimbus.seeds : ['node1']
+ java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
+ storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
+ EOF
+ ```
+
+15. Create an auxiliary storm configuration json file in the metron user\u2019s home directory. Note the login config option in the file points to our custom client_jaas.conf.
+ ```
+ cat << EOF > ~/storm-config.json
+ {
+ "topology.worker.childopts" : "-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
+ }
+ EOF
+ ```
+
+16. Setup enrichment and indexing.
+
+ a. Modify enrichment.properties as root located at `${METRON_HOME}/config/enrichment.properties`
+ ```
+ if [[ $EUID -ne 0 ]]; then
+ echo -e "\nERROR:\tYou must be root to run these commands. You may need to type exit."
+ else
+ sed -i 's/kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/' ${METRON_HOME}/config/enrichment.properties
+ sed -i 's/topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/' ${METRON_HOME}/config/enrichment.properties
+ fi
+ ```
+
+ b. Modify elasticsearch.properties as root located at `${METRON_HOME}/config/elasticsearch.properties`
+ ```
+ if [[ $EUID -ne 0 ]]; then
+ echo -e "\nERROR:\tYou must be root to run these commands. You may need to type exit."
+ else
+ sed -i 's/kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/' ${METRON_HOME}/config/elasticsearch.properties
+ sed -i 's/topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/' ${METRON_HOME}/config/elasticsearch.properties
+ fi
+ ```
+
+17. Distribute the custom jaas file and the keytab to each supervisor node, in the same locations as above. This ensures that the worker nodes can authenticate. For a one node cluster, nothing needs to be done.
+
+18. Kinit with the metron user again
+ ```
+ su metron
+ cd
+ kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+ ```
+
+19. Restart the parser topologies. Be sure to pass in the new parameter, \u201c-ksp\u201d or \u201c--kafka_security_protocol.\u201d Run this from the metron home directory.
+ ```
+ for parser in bro snort; do
+ ${METRON_HOME}/bin/start_parser_topology.sh -z ${ZOOKEEPER}:2181 -s ${parser} -ksp SASL_PLAINTEXT -e storm-config.json
+ done
+ ```
+
+20. Now restart the enrichment and indexing topologies.
+ ```
+ ${METRON_HOME}/bin/start_enrichment_topology.sh
+ ${METRON_HOME}/bin/start_elasticsearch_topology.sh
+ ```
+
+Metron should be ready to receieve data.
+
+## Push Data
+See [Push Data](vagrant/Kerberos-setup.md)
+
+### Other useful commands
+See [Other useful commands](vagrant/Kerberos-setup.md)
+
+#### References
+* [https://github.com/apache/storm/blob/master/SECURITY.md](https://github.com/apache/storm/blob/master/SECURITY.md)
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/README.md b/metron-deployment/README.md
index 66aa5f4..6f1653b 100644
--- a/metron-deployment/README.md
+++ b/metron-deployment/README.md
@@ -146,6 +146,21 @@ Build the Ambari Mpack to get the dashboard updated appropriately.
Once the MPack is installed, run the Kibana service's action "Load Template" to install dashboards. This will completely overwrite the .kibana in Elasticsearch, so use with caution.
+## Kerberos
+The MPack can allow Metron to be installed and then Kerberized, or installed on top of an already Kerberized cluster. This is done through Ambari's standard Kerberization setup.
+
+### Caveats
+* For nodes using a Metron client and a local repo, the repo must exist on all nodes (e.g via createrepo). This repo can be empty; only the main Metron services need the RPMs.
+* A Metron client must be installed on each supervisor node in a secured cluster. This is to ensure that the Metron keytab and client_jaas.conf get distributed in order to allow reading and writing from Kafka.
+ * When Metron is already installed on the cluster, this should be done before Kerberizing.
+ * When addding Metron to an already Kerberized cluster, ensure that all supervisor nodes receive a Metron client.
+* Storm (and Metron) must be restarted after Metron is installed on an already Kerberized cluster. Several Storm configs get updated, and Metron will be unable to write to Kafka without a restart.
+ * Kerberizing a cluster with an existing Metron already has restarts of all services during Kerberization, so it's unneeded.
+
+Instructions for setup on Full Dev can be found at [Kerberos-setup.md](vagrant/Kerberos-setup.md). These instructions can also be used for setting up KDC and testing.
+
+### Kerberos Without an MPack
+Using the MPack is preferred, but instructions for Kerberizing manually can be found at [Kerberos-manual-setup.md](Kerberos-manual-setup.md)
## TODO
- Support Ubuntu deployments
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml
index 25237f5..a4e82e2 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml
@@ -177,9 +177,9 @@
<component>
<name>METRON_CLIENT</name>
- <displayName>METRON Client</displayName>
+ <displayName>Metron Client</displayName>
<category>CLIENT</category>
- <cardinality>1+</cardinality>
+ <cardinality>0+</cardinality>
<versionAdvertised>true</versionAdvertised>
<commandScript>
<script>scripts/metron_client.py</script>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py
index 4fda07b..136eade 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py
@@ -21,6 +21,7 @@ from resource_management.core.logger import Logger
from resource_management.core.resources.system import Execute, File
import metron_service
+from metron_security import kinit
# Wrap major operations and functionality in this class
class EnrichmentCommands:
@@ -184,6 +185,11 @@ class EnrichmentCommands:
def create_hbase_tables(self):
Logger.info("Creating HBase Tables")
+ if self.__params.security_enabled:
+ kinit(self.__params.kinit_path_local,
+ self.__params.hbase_keytab_path,
+ self.__params.hbase_principal_name,
+ execute_user=self.__params.hbase_user)
cmd = "echo \"create '{0}','{1}'\" | hbase shell -n"
add_enrichment_cmd = cmd.format(self.__params.enrichment_table, self.__params.enrichment_cf)
Execute(add_enrichment_cmd,
@@ -208,6 +214,11 @@ class EnrichmentCommands:
def set_hbase_acls(self):
Logger.info("Setting HBase ACLs")
+ if self.__params.security_enabled:
+ kinit(self.__params.kinit_path_local,
+ self.__params.hbase_keytab_path,
+ self.__params.hbase_principal_name,
+ execute_user=self.__params.hbase_user)
cmd = "echo \"grant '{0}', 'RW', '{1}'\" | hbase shell -n"
add_enrichment_acl_cmd = cmd.format(self.__params.metron_user, self.__params.enrichment_table)
Execute(add_enrichment_acl_cmd,
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_client.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_client.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_client.py
index 65831c9..32fe0e9 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_client.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_client.py
@@ -46,4 +46,4 @@ class MetronClient(Script):
raise ClientComponentHasNoStatus()
if __name__ == "__main__":
- MetronClient().execute()
\ No newline at end of file
+ MetronClient().execute()
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py
index f2cb88d..be1654f 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py
@@ -68,22 +68,6 @@ def get_running_topologies(params):
Logger.info("Topologies: " + str(topologiesDict))
return topologiesDict
- try:
- stormjson = json.loads(stdout)
- except ValueError, e:
- Logger.info('Stdout: ' + str(stdout))
- Logger.info('Stderr: ' + str(stderr))
- Logger.exception(str(e))
- return {}
-
- topologiesDict = {}
-
- for topology in stormjson['topologies']:
- topologiesDict[topology['name']] = topology['status']
-
- Logger.info("Topologies: " + str(topologiesDict))
- return topologiesDict
-
def load_global_config(params):
Logger.info('Create Metron Local Config Directory')
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/parser_commands.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/parser_commands.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/parser_commands.py
index 965502a..e4cae47 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/parser_commands.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/parser_commands.py
@@ -113,11 +113,10 @@ class ParserCommands:
if self.__params.security_enabled:
# Append the extra configs needed for secured cluster.
start_cmd_template = start_cmd_template + ' -e ~' + self.__params.metron_user + '/.storm/storm.config'
- if self.__params.security_enabled:
- metron_security.kinit(self.__params.kinit_path_local,
- self.__params.metron_keytab_path,
- self.__params.metron_principal_name,
- execute_user=self.__params.metron_user)
+ metron_security.kinit(self.__params.kinit_path_local,
+ self.__params.metron_keytab_path,
+ self.__params.metron_principal_name,
+ execute_user=self.__params.metron_user)
for parser in self.get_parser_list():
Logger.info('Starting ' + parser)
Execute(start_cmd_template.format(self.__params.metron_home,
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/client_jaas.conf.j2
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/client_jaas.conf.j2 b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/client_jaas.conf.j2
index ff2498c..c0a047e 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/client_jaas.conf.j2
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/client_jaas.conf.j2
@@ -41,4 +41,4 @@ KafkaClient {
useTicketCache=false
serviceName="kafka"
principal="{{metron_principal_name}}";
-};
\ No newline at end of file
+};
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/storm.config.j2
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/storm.config.j2 b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/storm.config.j2
index caf1221..85fdf5c 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/storm.config.j2
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/storm.config.j2
@@ -19,4 +19,4 @@
{
"topology.worker.childopts" : "-Djava.security.auth.login.config={{metron_home}}/client_jaas.conf",
"topology.auto-credentials" : {{topology_auto_credentials_double_quotes}}
-}
\ No newline at end of file
+}
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/readme-images/ambari-storm-site-properties.png
----------------------------------------------------------------------
diff --git a/metron-deployment/readme-images/ambari-storm-site-properties.png b/metron-deployment/readme-images/ambari-storm-site-properties.png
new file mode 100755
index 0000000..e0050a2
Binary files /dev/null and b/metron-deployment/readme-images/ambari-storm-site-properties.png differ
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/readme-images/ambari-storm-site.png
----------------------------------------------------------------------
diff --git a/metron-deployment/readme-images/ambari-storm-site.png b/metron-deployment/readme-images/ambari-storm-site.png
new file mode 100755
index 0000000..5ff2d24
Binary files /dev/null and b/metron-deployment/readme-images/ambari-storm-site.png differ
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/readme-images/custom-storm-site-final.png
----------------------------------------------------------------------
diff --git a/metron-deployment/readme-images/custom-storm-site-final.png b/metron-deployment/readme-images/custom-storm-site-final.png
new file mode 100755
index 0000000..9b383d5
Binary files /dev/null and b/metron-deployment/readme-images/custom-storm-site-final.png differ
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/readme-images/enable-kerberos-configure-kerberos.png
----------------------------------------------------------------------
diff --git a/metron-deployment/readme-images/enable-kerberos-configure-kerberos.png b/metron-deployment/readme-images/enable-kerberos-configure-kerberos.png
new file mode 100755
index 0000000..212c64b
Binary files /dev/null and b/metron-deployment/readme-images/enable-kerberos-configure-kerberos.png differ
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/readme-images/enable-kerberos-started.png
----------------------------------------------------------------------
diff --git a/metron-deployment/readme-images/enable-kerberos-started.png b/metron-deployment/readme-images/enable-kerberos-started.png
new file mode 100755
index 0000000..96adb51
Binary files /dev/null and b/metron-deployment/readme-images/enable-kerberos-started.png differ
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/readme-images/enable-kerberos.png
----------------------------------------------------------------------
diff --git a/metron-deployment/readme-images/enable-kerberos.png b/metron-deployment/readme-images/enable-kerberos.png
new file mode 100755
index 0000000..bb46923
Binary files /dev/null and b/metron-deployment/readme-images/enable-kerberos.png differ
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/vagrant/Kerberos-setup.md
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/Kerberos-setup.md b/metron-deployment/vagrant/Kerberos-setup.md
index 27a56b0..f02cc5f 100644
--- a/metron-deployment/vagrant/Kerberos-setup.md
+++ b/metron-deployment/vagrant/Kerberos-setup.md
@@ -1,5 +1,8 @@
# Setting Up Kerberos in Vagrant Full Dev
**Note:** These are instructions for Kerberizing Metron Storm topologies from Kafka to Kafka. This does not cover the sensor connections or MAAS.
+General Kerberization notes can be found in the metron-deployment [README.md](../README.md)
+
+## Setup the KDC
1. Build full dev and ssh into the machine
```
@@ -19,12 +22,7 @@
export METRON_HOME="/usr/metron/${METRON_VERSION}"
```
-3. Stop all topologies - we will restart them again once Kerberos has been enabled.
- ```
- for topology in bro snort enrichment indexing; do storm kill $topology; done
- ```
-
-4. Setup Kerberos
+3. Setup Kerberos
```
# Note: if you copy/paste this full set of commands, the kdb5_util command will not run as expected, so run the commands individually to ensure they all execute
# set 'node1' to the correct host for your kdc
@@ -39,210 +37,46 @@
chkconfig kadmin on
```
-5. Setup the admin and metron user principals. You'll kinit as the metron user when running topologies. Make sure to remember the passwords.
+4. Setup the admin user principal. You'll kinit as the metron user when running topologies. Make sure to remember the password.
```
kadmin.local -q "addprinc admin/admin"
- kadmin.local -q "addprinc metron"
- ```
-
-6. Create the metron user HDFS home directory
- ```
- sudo -u hdfs hdfs dfs -mkdir /user/metron && \
- sudo -u hdfs hdfs dfs -chown metron:hdfs /user/metron && \
- sudo -u hdfs hdfs dfs -chmod 770 /user/metron
- ```
-
-7. In [Ambari](http://node1:8080), setup Storm to run with Kerberos and run worker jobs as the submitting user:
-
- a. Add the following properties to custom storm-site:
```
- topology.auto-credentials=['org.apache.storm.security.auth.kerberos.AutoTGT']
- nimbus.credential.renewers.classes=['org.apache.storm.security.auth.kerberos.AutoTGT']
- supervisor.run.worker.as.user=true
- ```
-
- b. In the Storm config section in Ambari, choose \u201cAdd Property\u201d under custom storm-site:
- 
-
- c. In the dialog window, choose the \u201cbulk property add mode\u201d toggle button and add the below values:
-
- 
-
-8. Kerberize the cluster via Ambari. More detailed documentation can be found [here](http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html).
+## Ambari Setup
+1. Kerberize the cluster via Ambari. More detailed documentation can be found [here](http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html).
a. For this exercise, choose existing MIT KDC (this is what we setup and installed in the previous steps.)
- 
+ 
- 
+ 
b. Setup Kerberos configuration. Realm is EXAMPLE.COM. The admin principal will end up as admin/admin@EXAMPLE.COM when testing the KDC. Use the password you entered during the step for adding the admin principal.
- 
-
- c. Click through to \u201cStart and Test Services.\u201d Let the cluster spin up, but don't worry about starting up Metron via Ambari - we're going to run the parsers manually against the rest of the Hadoop cluster Kerberized. The wizard will fail at starting Metron, but this is OK. Click \u201ccontinue.\u201d When you\u2019re finished, the custom storm-site should look similar to the following:
+ 
- 
+ c. Click through to \u201cStart and Test Services.\u201d Let the cluster spin up.
-9. Setup Metron keytab
- ```
- kadmin.local -q "ktadd -k metron.headless.keytab metron@EXAMPLE.COM" && \
- cp metron.headless.keytab /etc/security/keytabs && \
- chown metron:hadoop /etc/security/keytabs/metron.headless.keytab && \
- chmod 440 /etc/security/keytabs/metron.headless.keytab
- ```
-
-10. Kinit with the metron user
+## Push Data
+1. Kinit with the metron user
```
kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
```
-11. First create any additional Kafka topics you will need. We need to create the topics before adding the required ACLs. The current full dev installation will deploy bro, snort, enrichments, and indexing only. e.g.
- ```
- ${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper ${ZOOKEEPER}:2181 --create --topic yaf --partitions 1 --replication-factor 1
- ```
-
-12. Setup Kafka ACLs for the topics
- ```
- export KERB_USER=metron
- for topic in bro enrichments indexing snort; do
- ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --topic ${topic}
- done
- ```
-
-13. Setup Kafka ACLs for the consumer groups
- ```
- ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group bro_parser
- ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group snort_parser
- ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group yaf_parser
- ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group enrichments
- ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group indexing
- ```
-
-14. Add metron user to the Kafka cluster ACL
- ```
- ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --cluster kafka-cluster
- ```
-
-15. We also need to grant permissions to the HBase tables. Kinit as the hbase user and add ACLs for metron.
- ```
- kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-metron_cluster@EXAMPLE.COM
- echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
- echo "grant 'metron', 'RW', 'enrichment'" | hbase shell
- ```
-
-16. Create a \u201c.storm\u201d directory in the metron user\u2019s home directory and switch to that directory.
- ```
- su metron
- mkdir ~/.storm
- cd ~/.storm
- ```
-
-17. Create a custom client jaas file. This should look identical to the Storm client jaas file located in /etc/storm/conf/client_jaas.conf except for the addition of a Client stanza. The Client stanza is used for Zookeeper. All quotes and semicolons are necessary.
- ```
- cat << EOF > client_jaas.conf
- StormClient {
- com.sun.security.auth.module.Krb5LoginModule required
- useTicketCache=true
- renewTicket=true
- serviceName="nimbus";
- };
- Client {
- com.sun.security.auth.module.Krb5LoginModule required
- useKeyTab=true
- keyTab="/etc/security/keytabs/metron.headless.keytab"
- storeKey=true
- useTicketCache=false
- serviceName="zookeeper"
- principal="metron@EXAMPLE.COM";
- };
- KafkaClient {
- com.sun.security.auth.module.Krb5LoginModule required
- useKeyTab=true
- keyTab="/etc/security/keytabs/metron.headless.keytab"
- storeKey=true
- useTicketCache=false
- serviceName="kafka"
- principal="metron@EXAMPLE.COM";
- };
- EOF
- ```
-
-18. Create a storm.yaml with jaas file info. Set the array of nimbus hosts accordingly.
- ```
- cat << EOF > storm.yaml
- nimbus.seeds : ['node1']
- java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
- storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
- EOF
- ```
-
-19. Create an auxiliary storm configuration json file in the metron user\u2019s home directory. Note the login config option in the file points to our custom client_jaas.conf.
- ```
- cat << EOF > ~/storm-config.json
- {
- "topology.worker.childopts" : "-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
- }
- EOF
- ```
-
-20. Setup enrichment and indexing.
-
- a. Modify enrichment.properties as root located at `${METRON_HOME}/config/enrichment.properties`
- ```
- if [[ $EUID -ne 0 ]]; then
- echo -e "\nERROR:\tYou must be root to run these commands. You may need to type exit."
- else
- sed -i 's/kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/' ${METRON_HOME}/config/enrichment.properties
- sed -i 's/topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/' ${METRON_HOME}/config/enrichment.properties
- fi
- ```
-
- b. Modify elasticsearch.properties as root located at `${METRON_HOME}/config/elasticsearch.properties`
- ```
- if [[ $EUID -ne 0 ]]; then
- echo -e "\nERROR:\tYou must be root to run these commands. You may need to type exit."
- else
- sed -i 's/kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/' ${METRON_HOME}/config/elasticsearch.properties
- sed -i 's/topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/' ${METRON_HOME}/config/elasticsearch.properties
- fi
- ```
-
-21. Kinit with the metron user again
- ```
- su metron
- cd
- kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
- ```
-
-22. Restart the parser topologies. Be sure to pass in the new parameter, \u201c-ksp\u201d or \u201c--kafka_security_protocol.\u201d Run this from the metron home directory.
- ```
- for parser in bro snort; do
- ${METRON_HOME}/bin/start_parser_topology.sh -z ${ZOOKEEPER}:2181 -s ${parser} -ksp SASL_PLAINTEXT -e storm-config.json
- done
- ```
-
-23. Now restart the enrichment and indexing topologies.
- ```
- ${METRON_HOME}/bin/start_enrichment_topology.sh
- ${METRON_HOME}/bin/start_elasticsearch_topology.sh
- ```
-
-24. Push some sample data to one of the parser topics. E.g for bro we took raw data from [incubator-metron/metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput](../../metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput)
+2. Push some sample data to one of the parser topics. E.g for bro we took raw data from [incubator-metron/metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput](../../metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput)
```
cat sample-bro.txt | ${HDP_HOME}/kafka-broker/bin/kafka-console-producer.sh --broker-list ${BROKERLIST}:6667 --security-protocol SASL_PLAINTEXT --topic bro
```
-25. Wait a few moments for data to flow through the system and then check for data in the Elasticsearch indexes. Replace bro with whichever parser type you\u2019ve chosen.
+3. Wait a few moments for data to flow through the system and then check for data in the Elasticsearch indexes. Replace bro with whichever parser type you\u2019ve chosen.
```
curl -XGET "${ZOOKEEPER}:9200/bro*/_search"
curl -XGET "${ZOOKEEPER}:9200/bro*/_count"
```
-26. You should have data flowing from the parsers all the way through to the indexes. This completes the Kerberization instructions
+4. You should have data flowing from the parsers all the way through to the indexes. This completes the Kerberization instructions
-### Other useful commands:
+### Other useful commands
#### Kerberos
Unsure of your Kerberos principal associated with a keytab? There are a couple ways to get this. One is via the list of principals that Ambari provides via downloadable csv. If you didn\u2019t download this list, you can also check the principal manually by running the following against the keytab.
```
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/vagrant/readme-images/ambari-storm-site-properties.png
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/readme-images/ambari-storm-site-properties.png b/metron-deployment/vagrant/readme-images/ambari-storm-site-properties.png
deleted file mode 100755
index e0050a2..0000000
Binary files a/metron-deployment/vagrant/readme-images/ambari-storm-site-properties.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/vagrant/readme-images/ambari-storm-site.png
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/readme-images/ambari-storm-site.png b/metron-deployment/vagrant/readme-images/ambari-storm-site.png
deleted file mode 100755
index 5ff2d24..0000000
Binary files a/metron-deployment/vagrant/readme-images/ambari-storm-site.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/vagrant/readme-images/custom-storm-site-final.png
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/readme-images/custom-storm-site-final.png b/metron-deployment/vagrant/readme-images/custom-storm-site-final.png
deleted file mode 100755
index 9b383d5..0000000
Binary files a/metron-deployment/vagrant/readme-images/custom-storm-site-final.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/vagrant/readme-images/enable-kerberos-configure-kerberos.png
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/readme-images/enable-kerberos-configure-kerberos.png b/metron-deployment/vagrant/readme-images/enable-kerberos-configure-kerberos.png
deleted file mode 100755
index 212c64b..0000000
Binary files a/metron-deployment/vagrant/readme-images/enable-kerberos-configure-kerberos.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/vagrant/readme-images/enable-kerberos-started.png
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/readme-images/enable-kerberos-started.png b/metron-deployment/vagrant/readme-images/enable-kerberos-started.png
deleted file mode 100755
index 96adb51..0000000
Binary files a/metron-deployment/vagrant/readme-images/enable-kerberos-started.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/vagrant/readme-images/enable-kerberos.png
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/readme-images/enable-kerberos.png b/metron-deployment/vagrant/readme-images/enable-kerberos.png
deleted file mode 100755
index bb46923..0000000
Binary files a/metron-deployment/vagrant/readme-images/enable-kerberos.png and /dev/null differ
[10/26] incubator-metron git commit: METRON-799 The MPack should
function in a kerberized cluster (dlyle via justinleet) closes
apache/incubator-metron#518
Posted by ma...@apache.org.
METRON-799 The MPack should function in a kerberized cluster (dlyle via justinleet) closes apache/incubator-metron#518
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/55062fb7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/55062fb7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/55062fb7
Branch: refs/heads/Metron_0.4.0
Commit: 55062fb79a85722c909906874e0772f51b18953d
Parents: 187ef37
Author: David Lyle <dl...@gmail.com>
Authored: Thu Apr 13 11:23:40 2017 -0400
Committer: leet <le...@apache.org>
Committed: Thu Apr 20 16:03:05 2017 -0400
----------------------------------------------------------------------
.../METRON/CURRENT/configuration/metron-env.xml | 19 ++++----
.../METRON/CURRENT/kerberos.json | 22 ++++++++-
.../common-services/METRON/CURRENT/metainfo.xml | 14 ++++++
.../package/scripts/enrichment_commands.py | 12 -----
.../package/scripts/enrichment_master.py | 25 +++++++++-
.../CURRENT/package/scripts/metron_client.py | 49 ++++++++++++++++++++
.../CURRENT/package/scripts/metron_security.py | 9 ++++
.../CURRENT/package/scripts/metron_service.py | 23 +++++++--
.../package/scripts/params/status_params.py | 5 +-
.../METRON/CURRENT/service_advisor.py | 5 --
10 files changed, 145 insertions(+), 38 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/55062fb7/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
index 334e4f3..d09614b 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
@@ -128,21 +128,20 @@
<name>metron_principal_name</name>
<display-name>Metron Principal Name</display-name>
<description>Metron principal name</description>
- </property>
- <property>
- <name>metron_keytab</name>
- <display-name>Metron Keytab</display-name>
- <description>Metron keytab path</description>
- </property>
- <property>
- <name>metron_service_principal_name</name>
- <display-name>Metron Service Principal Name</display-name>
- <description>Metron service principal name</description>
+ <property-type>KERBEROS_PRINCIPAL</property-type>
+ <value/>
+ <value-attributes>
+ <empty-value-valid>true</empty-value-valid>
+ </value-attributes>
</property>
<property>
<name>metron_service_keytab</name>
<display-name>Metron Service User Keytab</display-name>
<description>Metron Service user keytab path</description>
+ <value/>
+ <value-attributes>
+ <empty-value-valid>true</empty-value-valid>
+ </value-attributes>
</property>
<property>
<name>global-json</name>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/55062fb7/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/kerberos.json
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/kerberos.json b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/kerberos.json
index e1a847b..cba1847 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/kerberos.json
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/kerberos.json
@@ -21,11 +21,29 @@
"name": "${cluster-env/user_group}",
"access": "r"
},
- "configuration": "metron-env/metron_keytab"
+ "configuration": "metron-env/metron_service_keytab"
}
}
+ ],
+ "configurations": [
+ {
+ "storm-site": {
+ "nimbus.credential.renewers.classes": "['org.apache.storm.security.auth.kerberos.AutoTGT']",
+ "supervisor.run.worker.as.user": "true"
+ }
+ }
+ ],
+ "components": [
+ {
+ "name": "METRON_PARSERS"
+ },
+ {
+ "name": "METRON_ENRICHMENT_MASTER"
+ },
+ {
+ "name": "METRON_INDEXING"
+ }
]
}
]
}
-
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/55062fb7/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml
index bbe7187..25237f5 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml
@@ -174,6 +174,20 @@
</customCommand>
</customCommands>
</component>
+
+ <component>
+ <name>METRON_CLIENT</name>
+ <displayName>METRON Client</displayName>
+ <category>CLIENT</category>
+ <cardinality>1+</cardinality>
+ <versionAdvertised>true</versionAdvertised>
+ <commandScript>
+ <script>scripts/metron_client.py</script>
+ <scriptType>PYTHON</scriptType>
+ <timeout>1200</timeout>
+ </commandScript>
+ </component>
+
</components>
<themes>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/55062fb7/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py
index aa53391..4fda07b 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py
@@ -21,8 +21,6 @@ from resource_management.core.logger import Logger
from resource_management.core.resources.system import Execute, File
import metron_service
-import metron_security
-
# Wrap major operations and functionality in this class
class EnrichmentCommands:
@@ -143,11 +141,6 @@ class EnrichmentCommands:
-s {1} \
-z {2}"""
Logger.info('Starting ' + self.__enrichment_topology)
- if self.__params.security_enabled:
- metron_security.kinit(self.__params.kinit_path_local,
- self.__params.metron_keytab_path,
- self.__params.metron_principal_name,
- execute_user=self.__params.metron_user)
Execute(start_cmd_template.format(self.__params.metron_home, self.__enrichment_topology, self.__params.zookeeper_quorum),
user=self.__params.metron_user)
@@ -156,11 +149,6 @@ class EnrichmentCommands:
def stop_enrichment_topology(self):
Logger.info('Stopping ' + self.__enrichment_topology)
stop_cmd = 'storm kill ' + self.__enrichment_topology
- if self.__params.security_enabled:
- metron_security.kinit(self.__params.kinit_path_local,
- self.__params.metron_keytab_path,
- self.__params.metron_principal_name,
- execute_user=self.__params.metron_user)
Execute(stop_cmd, user=self.__params.metron_user)
Logger.info('Done stopping enrichment topologies')
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/55062fb7/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_master.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_master.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_master.py
index e8cb8d5..4ab81d4 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_master.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_master.py
@@ -24,14 +24,13 @@ from resource_management.core.logger import Logger
from enrichment_commands import EnrichmentCommands
from metron_security import storm_security_setup
import metron_service
-
+import metron_security
class Enrichment(Script):
def install(self, env):
from params import params
env.set_params(params)
self.install_packages(env)
- self.configure(env)
def configure(self, env, upgrade_type=None, config_dir=None):
from params import params
@@ -52,6 +51,13 @@ class Enrichment(Script):
env.set_params(params)
self.configure(env)
commands = EnrichmentCommands(params)
+
+ if params.security_enabled:
+ metron_security.kinit(params.kinit_path_local,
+ params.metron_keytab_path,
+ params.metron_principal_name,
+ execute_user=params.metron_user)
+
metron_service.load_global_config(params)
if not commands.is_kafka_configured():
@@ -69,8 +75,16 @@ class Enrichment(Script):
def stop(self, env, upgrade_type=None):
from params import params
+
env.set_params(params)
commands = EnrichmentCommands(params)
+
+ if params.security_enabled:
+ metron_security.kinit(params.kinit_path_local,
+ params.metron_keytab_path,
+ params.metron_principal_name,
+ execute_user=params.metron_user)
+
commands.stop_enrichment_topology()
def status(self, env):
@@ -78,6 +92,13 @@ class Enrichment(Script):
env.set_params(status_params)
commands = EnrichmentCommands(status_params)
+ if status_params.security_enabled:
+ metron_security.kinit(status_params.kinit_path_local,
+ status_params.metron_keytab_path,
+ status_params.metron_principal_name,
+ execute_user=status_params.metron_user)
+
+
if not commands.is_topology_active(env):
raise ComponentIsNotRunning()
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/55062fb7/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_client.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_client.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_client.py
new file mode 100644
index 0000000..65831c9
--- /dev/null
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_client.py
@@ -0,0 +1,49 @@
+"""
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+"""
+
+from resource_management.libraries.script.script import Script
+from resource_management.core.exceptions import ClientComponentHasNoStatus
+from resource_management.core.resources.system import Directory
+from metron_security import storm_security_setup
+
+class MetronClient(Script):
+
+ def install(self, env):
+ from params import params
+ env.set_params(params)
+ self.configure(env)
+
+ def configure(self, env):
+ from params import params
+ env.set_params(params)
+ storm_security_setup(params)
+
+ def start(self, env, upgrade_type=None):
+ from params import params
+ env.set_params(params)
+
+ def stop(self, env, upgrade_type=None):
+ from params import params
+ env.set_params(params)
+
+ def status(self, env):
+ raise ClientComponentHasNoStatus()
+
+if __name__ == "__main__":
+ MetronClient().execute()
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/55062fb7/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_security.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_security.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_security.py
index 057339a..4f04daf 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_security.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_security.py
@@ -30,6 +30,15 @@ def storm_security_setup(params):
# Ambari's Directory doesn't do tilde expansion.
metron_storm_dir_tilde = '~' + params.metron_user + '/.storm'
metron_storm_dir = os.path.expanduser(metron_storm_dir_tilde)
+
+
+ Directory(params.metron_home,
+ mode=0755,
+ owner=params.metron_user,
+ group=params.metron_group,
+ create_parents=True
+ )
+
Directory(metron_storm_dir,
mode=0755,
owner=params.metron_user,
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/55062fb7/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py
index b213947..f2cb88d 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py
@@ -22,6 +22,7 @@ from resource_management.core.resources.system import Directory, File
from resource_management.core.resources.system import Execute
from resource_management.core.source import InlineTemplate
from resource_management.libraries.functions import format as ambari_format
+from resource_management.libraries.functions.get_user_call_output import get_user_call_output
from metron_security import kinit
def init_config():
@@ -40,8 +41,7 @@ def get_running_topologies(params):
# Want to sudo to the metron user and kinit as them so we aren't polluting root with Metron's Kerberos tickets.
# This is becuase we need to run a command with a return as the metron user. Sigh
negotiate = '--negotiate -u : ' if params.security_enabled else ''
- sudo = ambari_format('sudo -u {metron_user} ') if params.security_enabled else ''
- cmd = ambari_format(sudo + 'curl --max-time 3 ' + negotiate + '{storm_rest_addr}/api/v1/topology/summary')
+ cmd = ambari_format('curl --max-time 3 ' + negotiate + '{storm_rest_addr}/api/v1/topology/summary')
if params.security_enabled:
kinit(params.kinit_path_local,
@@ -50,8 +50,23 @@ def get_running_topologies(params):
execute_user=params.metron_user)
Logger.info('Running cmd: ' + cmd)
- proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
- (stdout, stderr) = proc.communicate()
+ return_code, stdout, sdterr = get_user_call_output(cmd, user=params.metron_user)
+
+ try:
+ stormjson = json.loads(stdout)
+ except ValueError, e:
+ Logger.info('Stdout: ' + str(stdout))
+ Logger.info('Stderr: ' + str(stderr))
+ Logger.exception(str(e))
+ return {}
+
+ topologiesDict = {}
+
+ for topology in stormjson['topologies']:
+ topologiesDict[topology['name']] = topology['status']
+
+ Logger.info("Topologies: " + str(topologiesDict))
+ return topologiesDict
try:
stormjson = json.loads(stdout)
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/55062fb7/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py
index 65e1eaf..154b3de 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py
@@ -73,6 +73,5 @@ tmp_dir = Script.get_tmp_dir()
metron_user = config['configurations']['metron-env']['metron_user']
-if security_enabled:
- metron_principal_name = config['configurations']['metron-env']['metron_principal_name']
- metron_keytab_path = config['configurations']['metron-env']['metron_keytab']
+metron_principal_name = config['configurations']['metron-env']['metron_principal_name']
+metron_keytab_path = config['configurations']['metron-env']['metron_service_keytab']
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/55062fb7/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py
index c16e3e6..23aebb4 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py
@@ -156,10 +156,5 @@ class METRON${metron.short.version}ServiceAdvisor(service_advisor.ServiceAdvisor
storm_site_desired_values = {
"topology.classpath" : "/etc/hbase/conf:/etc/hadoop/conf"
}
- if is_secured:
- storm_site_desired_values.update({
- "nimbus.credential.renewers.classes":"['org.apache.storm.security.auth.kerberos.AutoTGT']",
- "supervisor.run.worker.as.user":"true"
- })
return storm_site_desired_values
[09/26] incubator-metron git commit: METRON-799 The MPack should
function in a kerberized cluster (justinleet) closes
apache/incubator-metron#518
Posted by ma...@apache.org.
METRON-799 The MPack should function in a kerberized cluster (justinleet) closes apache/incubator-metron#518
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/187ef373
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/187ef373
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/187ef373
Branch: refs/heads/Metron_0.4.0
Commit: 187ef373a6cc0f89696349acc35ddeddbfd836ef
Parents: df44f07
Author: justinleet <ju...@gmail.com>
Authored: Wed Mar 15 14:55:06 2017 -0400
Committer: leet <le...@apache.org>
Committed: Thu Apr 20 16:02:30 2017 -0400
----------------------------------------------------------------------
.../METRON/CURRENT/configuration/metron-env.xml | 25 +++-
.../METRON/CURRENT/kerberos.json | 31 +++++
.../package/scripts/enrichment_commands.py | 127 +++++++++++++------
.../package/scripts/enrichment_master.py | 12 +-
.../package/scripts/indexing_commands.py | 81 +++++++-----
.../CURRENT/package/scripts/indexing_master.py | 12 ++
.../CURRENT/package/scripts/metron_security.py | 74 +++++++++++
.../CURRENT/package/scripts/metron_service.py | 88 ++++++++++++-
.../package/scripts/params/params_linux.py | 44 ++++++-
.../package/scripts/params/status_params.py | 26 +++-
.../CURRENT/package/scripts/parser_commands.py | 84 +++++++-----
.../CURRENT/package/scripts/parser_master.py | 7 +
.../package/templates/client_jaas.conf.j2 | 44 +++++++
.../package/templates/enrichment.properties.j2 | 38 +++---
.../CURRENT/package/templates/storm.config.j2 | 22 ++++
.../CURRENT/package/templates/storm.yaml.j2 | 21 +++
.../METRON/CURRENT/service_advisor.py | 16 ++-
metron-deployment/vagrant/Kerberos-setup.md | 2 +-
.../src/main/config/elasticsearch.properties | 1 +
.../src/main/flux/enrichment/remote.yaml | 1 +
.../src/main/flux/indexing/remote.yaml | 1 +
.../integration/IndexingIntegrationTest.java | 1 +
.../metron-solr/src/main/config/solr.properties | 1 +
23 files changed, 628 insertions(+), 131 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/187ef373/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
index 218e8e5..334e4f3 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
@@ -125,6 +125,26 @@
<value></value>
</property>
<property>
+ <name>metron_principal_name</name>
+ <display-name>Metron Principal Name</display-name>
+ <description>Metron principal name</description>
+ </property>
+ <property>
+ <name>metron_keytab</name>
+ <display-name>Metron Keytab</display-name>
+ <description>Metron keytab path</description>
+ </property>
+ <property>
+ <name>metron_service_principal_name</name>
+ <display-name>Metron Service Principal Name</display-name>
+ <description>Metron service principal name</description>
+ </property>
+ <property>
+ <name>metron_service_keytab</name>
+ <display-name>Metron Service User Keytab</display-name>
+ <description>Metron Service user keytab path</description>
+ </property>
+ <property>
<name>global-json</name>
<display-name>global.json template</display-name>
<description>This is the jinja template for global.json file</description>
@@ -148,12 +168,13 @@
##### Storm #####
indexing.workers=1
indexing.executors=0
-topology.worker.childopts=
+topology.worker.childopts={{topology_worker_childopts}}
+topology.auto-credentials={{topology_auto_credentials}}
##### Kafka #####
kafka.zk={{ zookeeper_quorum }}
kafka.broker={{ kafka_brokers }}
kafka.start=UNCOMMITTED_EARLIEST
-kafka.security.protocol=PLAINTEXT
+kafka.security.protocol={{kafka_security_protocol}}
storm.auto.credentials=[]
##### Indexing #####
index.input.topic=indexing
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/187ef373/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/kerberos.json
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/kerberos.json b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/kerberos.json
new file mode 100644
index 0000000..e1a847b
--- /dev/null
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/kerberos.json
@@ -0,0 +1,31 @@
+{
+ "services": [
+ {
+ "name": "METRON",
+ "identities": [
+ {
+ "name": "metron",
+ "principal": {
+ "value": "${metron-env/metron_user}@${realm}",
+ "type" : "user",
+ "configuration": "metron-env/metron_principal_name",
+ "local_username": "${metron-env/metron_user}"
+ },
+ "keytab": {
+ "file": "${keytab_dir}/metron.headless.keytab",
+ "owner": {
+ "name": "${metron-env/metron_user}",
+ "access": "r"
+ },
+ "group": {
+ "name": "${cluster-env/user_group}",
+ "access": "r"
+ },
+ "configuration": "metron-env/metron_keytab"
+ }
+ }
+ ]
+ }
+ ]
+}
+
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/187ef373/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py
index 3d7eb66..aa53391 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py
@@ -21,6 +21,7 @@ from resource_management.core.logger import Logger
from resource_management.core.resources.system import Execute, File
import metron_service
+import metron_security
# Wrap major operations and functionality in this class
@@ -28,10 +29,10 @@ class EnrichmentCommands:
__params = None
__enrichment_topology = None
__enrichment_topic = None
- __enrichment_error_topic = None
- __threat_intel_error_topic = None
__kafka_configured = False
+ __kafka_acl_configured = False
__hbase_configured = False
+ __hbase_acl_configured = False
__geo_configured = False
def __init__(self, params):
@@ -41,28 +42,50 @@ class EnrichmentCommands:
self.__enrichment_topology = params.metron_enrichment_topology
self.__enrichment_topic = params.metron_enrichment_topic
self.__kafka_configured = os.path.isfile(self.__params.enrichment_kafka_configured_flag_file)
+ self.__kafka_acl_configured = os.path.isfile(self.__params.enrichment_kafka_acl_configured_flag_file)
self.__hbase_configured = os.path.isfile(self.__params.enrichment_hbase_configured_flag_file)
+ self.__hbase_acl_configured = os.path.isfile(self.__params.enrichment_hbase_acl_configured_flag_file)
self.__geo_configured = os.path.isfile(self.__params.enrichment_geo_configured_flag_file)
def is_kafka_configured(self):
return self.__kafka_configured
+ def is_kafka_acl_configured(self):
+ return self.__kafka_acl_configured
+
def set_kafka_configured(self):
Logger.info("Setting Kafka Configured to True")
File(self.__params.enrichment_kafka_configured_flag_file,
content="",
owner=self.__params.metron_user,
- mode=0775)
+ mode=0755)
+
+ def set_kafka_acl_configured(self):
+ Logger.info("Setting Kafka ACL Configured to True")
+ File(self.__params.enrichment_kafka_acl_configured_flag_file,
+ content="",
+ owner=self.__params.metron_user,
+ mode=0755)
def is_hbase_configured(self):
return self.__hbase_configured
+ def is_hbase_acl_configured(self):
+ return self.__hbase_acl_configured
+
def set_hbase_configured(self):
Logger.info("Setting HBase Configured to True")
File(self.__params.enrichment_hbase_configured_flag_file,
content="",
owner=self.__params.metron_user,
- mode=0775)
+ mode=0755)
+
+ def set_hbase_acl_configured(self):
+ Logger.info("Setting HBase ACL Configured to True")
+ File(self.__params.enrichment_hbase_acl_configured_flag_file,
+ content="",
+ owner=self.__params.metron_user,
+ mode=0755)
def is_geo_configured(self):
return self.__geo_configured
@@ -72,7 +95,7 @@ class EnrichmentCommands:
File(self.__params.enrichment_geo_configured_flag_file,
content="",
owner=self.__params.metron_user,
- mode=0775)
+ mode=0755)
def init_geo(self):
Logger.info("Creating HDFS location for GeoIP database")
@@ -80,8 +103,8 @@ class EnrichmentCommands:
type="directory",
action="create_on_execute",
owner=self.__params.metron_user,
- group=self.__params.hadoop_group,
- mode=0775,
+ group=self.__params.metron_group,
+ mode=0755,
)
Logger.info("Creating and loading GeoIp database")
@@ -100,47 +123,45 @@ class EnrichmentCommands:
self.set_geo_configured()
def init_kafka_topics(self):
- Logger.info('Creating Kafka topics')
- command_template = """{0}/kafka-topics.sh \
- --zookeeper {1} \
- --create \
- --topic {2} \
- --partitions {3} \
- --replication-factor {4} \
- --config retention.bytes={5}"""
- num_partitions = 1
- replication_factor = 1
- retention_gigabytes = int(self.__params.metron_topic_retention)
- retention_bytes = retention_gigabytes * 1024 * 1024 * 1024
-
- Logger.info("Creating topics for enrichment")
- topics = [self.__enrichment_topic]
- for topic in topics:
- Logger.info("Creating topic'{0}'".format(topic))
- Execute(command_template.format(self.__params.kafka_bin_dir,
- self.__params.zookeeper_quorum,
- topic,
- num_partitions,
- replication_factor,
- retention_bytes))
-
- Logger.info("Done creating Kafka topics")
+ Logger.info('Creating Kafka topics for enrichment')
+ # All errors go to indexing topics, so create it here if it's not already
+ metron_service.init_kafka_topics(self.__params, [self.__enrichment_topic, self.__params.metron_error_topic])
self.set_kafka_configured()
+ def init_kafka_acls(self):
+ Logger.info('Creating Kafka ACls for enrichment')
+ # Enrichment topic names matches group
+ metron_service.init_kafka_acls(self.__params,
+ [self.__enrichment_topic, self.__params.metron_error_topic],
+ [self.__enrichment_topic])
+
+ self.set_kafka_acl_configured()
+
def start_enrichment_topology(self):
Logger.info("Starting Metron enrichment topology: {0}".format(self.__enrichment_topology))
start_cmd_template = """{0}/bin/start_enrichment_topology.sh \
-s {1} \
-z {2}"""
Logger.info('Starting ' + self.__enrichment_topology)
- Execute(start_cmd_template.format(self.__params.metron_home, self.__enrichment_topology, self.__params.zookeeper_quorum))
+ if self.__params.security_enabled:
+ metron_security.kinit(self.__params.kinit_path_local,
+ self.__params.metron_keytab_path,
+ self.__params.metron_principal_name,
+ execute_user=self.__params.metron_user)
+ Execute(start_cmd_template.format(self.__params.metron_home, self.__enrichment_topology, self.__params.zookeeper_quorum),
+ user=self.__params.metron_user)
Logger.info('Finished starting enrichment topology')
def stop_enrichment_topology(self):
Logger.info('Stopping ' + self.__enrichment_topology)
stop_cmd = 'storm kill ' + self.__enrichment_topology
- Execute(stop_cmd)
+ if self.__params.security_enabled:
+ metron_security.kinit(self.__params.kinit_path_local,
+ self.__params.metron_keytab_path,
+ self.__params.metron_principal_name,
+ execute_user=self.__params.metron_user)
+ Execute(stop_cmd, user=self.__params.metron_user)
Logger.info('Done stopping enrichment topologies')
def restart_enrichment_topology(self, env):
@@ -166,7 +187,7 @@ class EnrichmentCommands:
env.set_params(self.__params)
active = True
- topologies = metron_service.get_running_topologies()
+ topologies = metron_service.get_running_topologies(self.__params)
is_running = False
if self.__enrichment_topology in topologies:
is_running = topologies[self.__enrichment_topology] in ['ACTIVE', 'REBALANCING']
@@ -175,20 +196,48 @@ class EnrichmentCommands:
def create_hbase_tables(self):
Logger.info("Creating HBase Tables")
- add_enrichment_cmd = "echo \"create '{0}','{1}'\" | hbase shell -n".format(self.__params.enrichment_table, self.__params.enrichment_cf)
+ cmd = "echo \"create '{0}','{1}'\" | hbase shell -n"
+ add_enrichment_cmd = cmd.format(self.__params.enrichment_table, self.__params.enrichment_cf)
Execute(add_enrichment_cmd,
tries=3,
try_sleep=5,
logoutput=False,
- path='/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'
+ path='/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin',
+ user=self.__params.hbase_user
)
- add_threatintel_cmd = "echo \"create '{0}','{1}'\" | hbase shell -n".format(self.__params.threatintel_table, self.__params.threatintel_cf)
+ add_threatintel_cmd = cmd.format(self.__params.threatintel_table, self.__params.threatintel_cf)
Execute(add_threatintel_cmd,
tries=3,
try_sleep=5,
logoutput=False,
- path='/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'
+ path='/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin',
+ user=self.__params.hbase_user
)
+
Logger.info("Done creating HBase Tables")
self.set_hbase_configured()
+
+ def set_hbase_acls(self):
+ Logger.info("Setting HBase ACLs")
+ cmd = "echo \"grant '{0}', 'RW', '{1}'\" | hbase shell -n"
+ add_enrichment_acl_cmd = cmd.format(self.__params.metron_user, self.__params.enrichment_table)
+ Execute(add_enrichment_acl_cmd,
+ tries=3,
+ try_sleep=5,
+ logoutput=False,
+ path='/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin',
+ user=self.__params.hbase_user
+ )
+
+ add_threatintel_acl_cmd = cmd.format(self.__params.metron_user, self.__params.threatintel_table)
+ Execute(add_threatintel_acl_cmd,
+ tries=3,
+ try_sleep=5,
+ logoutput=False,
+ path='/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin',
+ user=self.__params.hbase_user
+ )
+
+ Logger.info("Done setting HBase ACLs")
+ self.set_hbase_acl_configured()
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/187ef373/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_master.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_master.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_master.py
index 1e734b4..e8cb8d5 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_master.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_master.py
@@ -15,13 +15,14 @@ limitations under the License.
"""
from resource_management.core.exceptions import ComponentIsNotRunning
-from resource_management.core.logger import Logger
from resource_management.core.resources.system import File
from resource_management.core.source import Template
from resource_management.libraries.functions.format import format
from resource_management.libraries.script import Script
+from resource_management.core.logger import Logger
from enrichment_commands import EnrichmentCommands
+from metron_security import storm_security_setup
import metron_service
@@ -36,22 +37,31 @@ class Enrichment(Script):
from params import params
env.set_params(params)
+ Logger.info("Running enrichment configure")
File(format("{metron_config_path}/enrichment.properties"),
content=Template("enrichment.properties.j2"),
owner=params.metron_user,
group=params.metron_group
)
+ Logger.info("Calling security setup")
+ storm_security_setup(params)
+
def start(self, env, upgrade_type=None):
from params import params
env.set_params(params)
+ self.configure(env)
commands = EnrichmentCommands(params)
metron_service.load_global_config(params)
if not commands.is_kafka_configured():
commands.init_kafka_topics()
+ if params.security_enabled and not commands.is_kafka_acl_configured():
+ commands.init_kafka_acls()
if not commands.is_hbase_configured():
commands.create_hbase_tables()
+ if params.security_enabled and not commands.is_hbase_acl_configured():
+ commands.set_hbase_acls()
if not commands.is_geo_configured():
commands.init_geo()
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/187ef373/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_commands.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_commands.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_commands.py
index 819085f..da94c9d 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_commands.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_commands.py
@@ -22,6 +22,7 @@ from resource_management.core.logger import Logger
from resource_management.core.resources.system import Execute, File
import metron_service
+import metron_security
# Wrap major operations and functionality in this class
@@ -29,6 +30,8 @@ class IndexingCommands:
__params = None
__indexing = None
__configured = False
+ __acl_configured = False
+ __hdfs_perm_configured = False
def __init__(self, params):
if params is None:
@@ -36,66 +39,86 @@ class IndexingCommands:
self.__params = params
self.__indexing = params.metron_indexing_topology
self.__configured = os.path.isfile(self.__params.indexing_configured_flag_file)
+ self.__acl_configured = os.path.isfile(self.__params.indexing_acl_configured_flag_file)
def is_configured(self):
return self.__configured
+ def is_acl_configured(self):
+ return self.__acl_configured
+
+ def is_hdfs_perm_configured(self):
+ return self.__hdfs_perm_configured
+
def set_configured(self):
File(self.__params.indexing_configured_flag_file,
content="",
owner=self.__params.metron_user,
- mode=0775)
+ mode=0755)
+
+ def set_acl_configured(self):
+ File(self.__params.indexing_acl_configured_flag_file,
+ content="",
+ owner=self.__params.metron_user,
+ mode=0755)
+
+ def set_hdfs_perm_configured(self):
+ File(self.__params.indexing_hdfs_perm_configured_flag_file,
+ content="",
+ owner=self.__params.metron_user,
+ mode=0755)
def init_kafka_topics(self):
- Logger.info('Creating Kafka topics')
- command_template = """{0}/kafka-topics.sh \
- --zookeeper {1} \
- --create \
- --topic {2} \
- --partitions {3} \
- --replication-factor {4} \
- --config retention.bytes={5}"""
- num_partitions = 1
- replication_factor = 1
- retention_gigabytes = int(self.__params.metron_topic_retention)
- retention_bytes = retention_gigabytes * 1024 * 1024 * 1024
- Logger.info("Creating topics for indexing")
-
- Logger.info("Creating topic'{0}'".format(self.__indexing))
- Execute(command_template.format(self.__params.kafka_bin_dir,
- self.__params.zookeeper_quorum,
- self.__indexing,
- num_partitions,
- replication_factor,
- retention_bytes))
- Logger.info("Done creating Kafka topics")
+ Logger.info('Creating Kafka topics for indexing')
+ metron_service.init_kafka_topics(self.__params, [self.__indexing])
+
+ def init_kafka_acls(self):
+ Logger.info('Creating Kafka ACLs')
+ # Indexed topic names matches the group
+ metron_service.init_kafka_acls(self.__params, [self.__indexing], [self.__indexing])
def init_hdfs_dir(self):
- Logger.info('Creating HDFS indexing directory')
+ Logger.info('Setting up HDFS indexing directory')
+
+ # Non Kerberized Metron runs under 'storm', requiring write under the 'hadoop' group.
+ # Kerberized Metron runs under it's own user.
+ ownership = 0755 if self.__params.security_enabled else 0775
+ Logger.info('HDFS indexing directory ownership is: ' + str(ownership))
self.__params.HdfsResource(self.__params.metron_apps_indexed_hdfs_dir,
type="directory",
action="create_on_execute",
owner=self.__params.metron_user,
group=self.__params.hadoop_group,
- mode=0775,
+ mode=ownership,
)
Logger.info('Done creating HDFS indexing directory')
-
def start_indexing_topology(self):
Logger.info("Starting Metron indexing topology: {0}".format(self.__indexing))
start_cmd_template = """{0}/bin/start_elasticsearch_topology.sh \
-s {1} \
-z {2}"""
Logger.info('Starting ' + self.__indexing)
- Execute(start_cmd_template.format(self.__params.metron_home, self.__indexing, self.__params.zookeeper_quorum))
+ if self.__params.security_enabled:
+ metron_security.kinit(self.__params.kinit_path_local,
+ self.__params.metron_keytab_path,
+ self.__params.metron_principal_name,
+ execute_user=self.__params.metron_user)
+ Execute(start_cmd_template.format(self.__params.metron_home, self.__indexing, self.__params.zookeeper_quorum),
+ user=self.__params.metron_user)
Logger.info('Finished starting indexing topology')
def stop_indexing_topology(self):
Logger.info('Stopping ' + self.__indexing)
stop_cmd = 'storm kill ' + self.__indexing
- Execute(stop_cmd)
+ if self.__params.security_enabled:
+ metron_security.kinit(self.__params.kinit_path_local,
+ self.__params.metron_keytab_path,
+ self.__params.metron_principal_name,
+ execute_user=self.__params.metron_user)
+ Execute(stop_cmd,
+ user=self.__params.metron_user)
Logger.info('Done stopping indexing topologies')
def restart_indexing_topology(self, env):
@@ -121,7 +144,7 @@ class IndexingCommands:
def is_topology_active(self, env):
env.set_params(self.__params)
active = True
- topologies = metron_service.get_running_topologies()
+ topologies = metron_service.get_running_topologies(self.__params)
is_running = False
if self.__indexing in topologies:
is_running = topologies[self.__indexing] in ['ACTIVE', 'REBALANCING']
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/187ef373/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py
index 8189e3f..f208f3a 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py
@@ -23,6 +23,7 @@ from resource_management.core.source import StaticFile
from resource_management.libraries.functions import format as ambari_format
from resource_management.libraries.script import Script
+from metron_security import storm_security_setup
import metron_service
from indexing_commands import IndexingCommands
@@ -46,6 +47,17 @@ class Indexing(Script):
commands.init_kafka_topics()
commands.init_hdfs_dir()
commands.set_configured()
+ if params.security_enabled and not commands.is_hdfs_perm_configured():
+ # If we Kerberize the cluster, we need to call this again, to remove write perms from hadoop group
+ # If we start off Kerberized, it just does the same thing twice.
+ commands.init_hdfs_dir()
+ commands.set_hdfs_perm_configured()
+ if params.security_enabled and not commands.is_acl_configured():
+ commands.init_kafka_acls()
+ commands.set_acl_configured()
+
+ Logger.info("Calling security setup")
+ storm_security_setup(params)
def start(self, env, upgrade_type=None):
from params import params
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/187ef373/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_security.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_security.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_security.py
new file mode 100644
index 0000000..057339a
--- /dev/null
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_security.py
@@ -0,0 +1,74 @@
+"""
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+"""
+
+import os.path
+from resource_management.core.source import Template
+from resource_management.core.resources.system import Directory, File
+from resource_management.core import global_lock
+from resource_management.core.logger import Logger
+from resource_management.core.resources.system import Execute
+from resource_management.libraries.functions import format as ambari_format
+
+
+# Convenience function for ensuring home dirs are setup consistently.
+def storm_security_setup(params):
+ if params.security_enabled:
+ # I don't think there's an Ambari way to get a user's local home dir , so have Python perform tilde expansion.
+ # Ambari's Directory doesn't do tilde expansion.
+ metron_storm_dir_tilde = '~' + params.metron_user + '/.storm'
+ metron_storm_dir = os.path.expanduser(metron_storm_dir_tilde)
+ Directory(metron_storm_dir,
+ mode=0755,
+ owner=params.metron_user,
+ group=params.metron_group
+ )
+
+ File(ambari_format('{client_jaas_path}'),
+ content=Template('client_jaas.conf.j2'),
+ owner=params.metron_user,
+ group=params.metron_group,
+ mode=0755
+ )
+
+ File(metron_storm_dir + '/storm.yaml',
+ content=Template('storm.yaml.j2'),
+ owner=params.metron_user,
+ group=params.metron_group,
+ mode=0755
+ )
+
+ File(metron_storm_dir + '/storm.config',
+ content=Template('storm.config.j2'),
+ owner=params.metron_user,
+ group=params.metron_group,
+ mode=0755
+ )
+
+
+def kinit(kinit_path_local, keytab_path, principal_name, execute_user=None):
+ # prevent concurrent kinit
+ kinit_lock = global_lock.get_lock(global_lock.LOCK_TYPE_KERBEROS)
+ kinit_lock.acquire()
+ kinitcmd = "{0} -kt {1} {2}; ".format(kinit_path_local, keytab_path, principal_name)
+ Logger.info("kinit command: " + kinitcmd + " as user: " + str(execute_user))
+ try:
+ if execute_user is None:
+ Execute(kinitcmd)
+ else:
+ Execute(kinitcmd, user=execute_user)
+ finally:
+ kinit_lock.release()
+
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/187ef373/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py
index 8415460..b213947 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py
@@ -22,6 +22,7 @@ from resource_management.core.resources.system import Directory, File
from resource_management.core.resources.system import Execute
from resource_management.core.source import InlineTemplate
from resource_management.libraries.functions import format as ambari_format
+from metron_security import kinit
def init_config():
Logger.info('Loading config into ZooKeeper')
@@ -31,16 +32,33 @@ def init_config():
)
-def get_running_topologies():
+def get_running_topologies(params):
Logger.info('Getting Running Storm Topologies from Storm REST Server')
- cmd = ambari_format('curl --max-time 3 {storm_rest_addr}/api/v1/topology/summary')
+ Logger.info('Security enabled? ' + str(params.security_enabled))
+
+ # Want to sudo to the metron user and kinit as them so we aren't polluting root with Metron's Kerberos tickets.
+ # This is becuase we need to run a command with a return as the metron user. Sigh
+ negotiate = '--negotiate -u : ' if params.security_enabled else ''
+ sudo = ambari_format('sudo -u {metron_user} ') if params.security_enabled else ''
+ cmd = ambari_format(sudo + 'curl --max-time 3 ' + negotiate + '{storm_rest_addr}/api/v1/topology/summary')
+
+ if params.security_enabled:
+ kinit(params.kinit_path_local,
+ params.metron_keytab_path,
+ params.metron_principal_name,
+ execute_user=params.metron_user)
+
+ Logger.info('Running cmd: ' + cmd)
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
(stdout, stderr) = proc.communicate()
try:
stormjson = json.loads(stdout)
- except ValueError:
+ except ValueError, e:
+ Logger.info('Stdout: ' + str(stdout))
+ Logger.info('Stderr: ' + str(stderr))
+ Logger.exception(str(e))
return {}
topologiesDict = {}
@@ -73,3 +91,67 @@ def load_global_config(params):
content=InlineTemplate(params.global_properties_template))
init_config()
+
+
+def init_kafka_topics(params, topics):
+ Logger.info('Creating Kafka topics')
+
+ # Create the topics. All the components need indexing (for errors), so we pass '--if-not-exists'.
+ command_template = """{0}/kafka-topics.sh \
+ --zookeeper {1} \
+ --create \
+ --if-not-exists \
+ --topic {2} \
+ --partitions {3} \
+ --replication-factor {4} \
+ --config retention.bytes={5}"""
+
+ num_partitions = 1
+ replication_factor = 1
+ retention_gigabytes = int(params.metron_topic_retention)
+ retention_bytes = retention_gigabytes * 1024 * 1024 * 1024
+ for topic in topics:
+ Logger.info("Creating topic'{0}'".format(topic))
+ Execute(command_template.format(params.kafka_bin_dir,
+ params.zookeeper_quorum,
+ topic,
+ num_partitions,
+ replication_factor,
+ retention_bytes),
+ user=params.kafka_user)
+ Logger.info("Done creating Kafka topics")
+
+
+def init_kafka_acls(params, topics, groups):
+ Logger.info('Creating Kafka ACLs')
+
+ acl_template = """{0}/kafka-acls.sh \
+ --authorizer kafka.security.auth.SimpleAclAuthorizer \
+ --authorizer-properties zookeeper.connect={1} \
+ --add \
+ --allow-principal User:{2} \
+ --topic {3}"""
+
+ for topic in topics:
+ Logger.info("Creating ACL for topic '{0}'".format(topic))
+ Execute(acl_template.format(params.kafka_bin_dir,
+ params.zookeeper_quorum,
+ params.metron_user,
+ topic),
+ user=params.kafka_user)
+
+ acl_template = """{0}/kafka-acls.sh \
+ --authorizer kafka.security.auth.SimpleAclAuthorizer \
+ --authorizer-properties zookeeper.connect={1} \
+ --add \
+ --allow-principal User:{2} \
+ --group {3}"""
+
+ for group in groups:
+ Logger.info("Creating ACL for group '{0}'".format(group))
+ Execute(acl_template.format(params.kafka_bin_dir,
+ params.zookeeper_quorum,
+ params.metron_user,
+ group),
+ user=params.kafka_user)
+ Logger.info("Done creating Kafka ACLs")
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/187ef373/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
index a4bfcbd..f2ab12b 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
@@ -21,14 +21,12 @@ limitations under the License.
import functools
import os
-from ambari_commons.os_check import OSCheck
from resource_management.libraries.functions import conf_select
from resource_management.libraries.functions import format
from resource_management.libraries.functions import get_kinit_path
from resource_management.libraries.functions import stack_select
from resource_management.libraries.functions.default import default
from resource_management.libraries.functions.get_not_managed_resources import get_not_managed_resources
-from resource_management.libraries.functions.is_empty import is_empty
from resource_management.libraries.resources.hdfs_resource import HdfsResource
from resource_management.libraries.script import Script
@@ -44,16 +42,21 @@ parsers = status_params.parsers
geoip_url = config['configurations']['metron-env']['geoip_url']
geoip_hdfs_dir = "/apps/metron/geo/default/"
metron_indexing_topology = status_params.metron_indexing_topology
-metron_user = config['configurations']['metron-env']['metron_user']
+metron_user = status_params.metron_user
metron_group = config['configurations']['metron-env']['metron_group']
metron_config_path = metron_home + '/config'
metron_zookeeper_config_dir = status_params.metron_zookeeper_config_dir
metron_zookeeper_config_path = status_params.metron_zookeeper_config_path
parsers_configured_flag_file = status_params.parsers_configured_flag_file
+parsers_acl_configured_flag_file = status_params.parsers_acl_configured_flag_file
enrichment_kafka_configured_flag_file = status_params.enrichment_kafka_configured_flag_file
+enrichment_kafka_acl_configured_flag_file = status_params.enrichment_kafka_acl_configured_flag_file
enrichment_hbase_configured_flag_file = status_params.enrichment_hbase_configured_flag_file
+enrichment_hbase_acl_configured_flag_file = status_params.enrichment_hbase_acl_configured_flag_file
enrichment_geo_configured_flag_file = status_params.enrichment_geo_configured_flag_file
indexing_configured_flag_file = status_params.indexing_configured_flag_file
+indexing_acl_configured_flag_file = status_params.indexing_acl_configured_flag_file
+indexing_hdfs_perm_configured_flag_file = status_params.indexing_hdfs_perm_configured_flag_file
global_json_template = config['configurations']['metron-env']['global-json']
global_properties_template = config['configurations']['metron-env']['elasticsearch-properties']
@@ -153,8 +156,10 @@ enrichment_cf = status_params.enrichment_cf
threatintel_table = status_params.threatintel_table
threatintel_cf = status_params.threatintel_cf
+# Kafka Topics
metron_enrichment_topology = status_params.metron_enrichment_topology
metron_enrichment_topic = status_params.metron_enrichment_topic
+metron_error_topic = 'indexing'
# ES Templates
bro_index_path = tmp_dir + "/bro_index.template"
@@ -164,3 +169,36 @@ error_index_path = tmp_dir + "/error_index.template"
# Zeppelin Notebooks
metron_config_zeppelin_path = format("{metron_config_path}/zeppelin")
+
+# kafka_security
+kafka_security_protocol = config['configurations']['kafka-broker'].get('security.inter.broker.protocol', 'PLAINTEXT')
+
+kafka_user = config['configurations']['kafka-env']['kafka_user']
+storm_user = config['configurations']['storm-env']['storm_user']
+
+# HBase user table creation and ACLs
+hbase_user = config['configurations']['hbase-env']['hbase_user']
+
+# Security
+security_enabled = status_params.security_enabled
+client_jaas_path = metron_home + '/client_jaas.conf'
+client_jaas_arg = '-Djava.security.auth.login.config=' + metron_home + '/client_jaas.conf'
+topology_worker_childopts = client_jaas_arg if security_enabled else ''
+topology_auto_credentials = config['configurations']['storm-site'].get('nimbus.credential.renewers.classes', [])
+# Needed for storm.config, because it needs Java String
+topology_auto_credentials_double_quotes = str(topology_auto_credentials).replace("'", '"')
+
+if security_enabled:
+ hostname_lowercase = config['hostname'].lower()
+ metron_principal_name = status_params.metron_principal_name
+ metron_keytab_path = status_params.metron_keytab_path
+ kinit_path_local = status_params.kinit_path_local
+
+ hbase_principal_name = config['configurations']['hbase-env']['hbase_principal_name']
+ hbase_keytab_path = config['configurations']['hbase-env']['hbase_user_keytab']
+
+ kafka_principal_raw = config['configurations']['kafka-env']['kafka_principal_name']
+ kafka_principal_name = kafka_principal_raw.replace('_HOST', hostname_lowercase)
+ kafka_keytab_path = config['configurations']['kafka-env']['kafka_keytab']
+
+ nimbus_seeds = config['configurations']['storm-site']['nimbus.seeds']
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/187ef373/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py
index 83b4fa4..65e1eaf 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py
@@ -18,18 +18,22 @@ limitations under the License.
"""
-from ambari_commons import OSCheck
-from resource_management.libraries.functions import format
from resource_management.libraries.script import Script
+from resource_management.libraries.functions import get_kinit_path
+from resource_management.libraries.functions import default, format
+from resource_management.libraries.functions.version import format_stack_version
config = Script.get_config()
+metron_user = config['configurations']['metron-env']['metron_user']
+
# Parsers
parsers = config['configurations']['metron-env']['parsers']
metron_home = config['configurations']['metron-env']['metron_home']
metron_zookeeper_config_dir = config['configurations']['metron-env']['metron_zookeeper_config_dir']
metron_zookeeper_config_path = format('{metron_home}/{metron_zookeeper_config_dir}')
parsers_configured_flag_file = metron_zookeeper_config_path + '/../metron_parsers_configured'
+parsers_acl_configured_flag_file = metron_zookeeper_config_path + '/../metron_parsers_acl_configured'
# Enrichment
metron_enrichment_topology = 'enrichment'
@@ -43,10 +47,14 @@ threatintel_cf = 't'
# Indexing
metron_indexing_topology = config['configurations']['metron-env']['metron_indexing_topology']
indexing_configured_flag_file = metron_zookeeper_config_path + '/../metron_indexing_configured'
+indexing_acl_configured_flag_file = metron_zookeeper_config_path + '/../metron_indexing_acl_configured'
+indexing_hdfs_perm_configured_flag_file = metron_zookeeper_config_path + '/../metron_indexing_hdfs_perm_configured'
# Enrichment
enrichment_kafka_configured_flag_file = metron_zookeeper_config_path + '/../metron_enrichment_kafka_configured'
+enrichment_kafka_acl_configured_flag_file = metron_zookeeper_config_path + '/../metron_enrichment_kafka_acl_configured'
enrichment_hbase_configured_flag_file = metron_zookeeper_config_path + '/../metron_enrichment_hbase_configured'
+enrichment_hbase_acl_configured_flag_file = metron_zookeeper_config_path + '/../metron_enrichment_hbase_acl_configured'
enrichment_geo_configured_flag_file = metron_zookeeper_config_path + '/../metron_enrichment_geo_configured'
# Storm
@@ -54,3 +62,17 @@ storm_rest_addr = config['configurations']['metron-env']['storm_rest_addr']
# Zeppelin
zeppelin_server_url = config['configurations']['metron-env']['zeppelin_server_url']
+
+# Security
+stack_version_unformatted = str(config['hostLevelParams']['stack_version'])
+stack_version_formatted = format_stack_version(stack_version_unformatted)
+hostname = config['hostname']
+security_enabled = config['configurations']['cluster-env']['security_enabled']
+kinit_path_local = get_kinit_path(default('/configurations/kerberos-env/executable_search_paths', None))
+tmp_dir = Script.get_tmp_dir()
+
+metron_user = config['configurations']['metron-env']['metron_user']
+
+if security_enabled:
+ metron_principal_name = config['configurations']['metron-env']['metron_principal_name']
+ metron_keytab_path = config['configurations']['metron-env']['metron_keytab']
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/187ef373/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/parser_commands.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/parser_commands.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/parser_commands.py
index bd3ad2c..965502a 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/parser_commands.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/parser_commands.py
@@ -27,6 +27,7 @@ from resource_management.core.logger import Logger
from resource_management.core.resources.system import Execute, File
import metron_service
+import metron_security
# Wrap major operations and functionality in this class
@@ -34,6 +35,7 @@ class ParserCommands:
__params = None
__parser_list = None
__configured = False
+ __acl_configured = False
def __init__(self, params):
if params is None:
@@ -41,6 +43,7 @@ class ParserCommands:
self.__params = params
self.__parser_list = self.__get_parsers(params)
self.__configured = os.path.isfile(self.__params.parsers_configured_flag_file)
+ self.__acl_configured = os.path.isfile(self.__params.parsers_acl_configured_flag_file)
# get list of parsers
def __get_parsers(self, params):
@@ -49,21 +52,31 @@ class ParserCommands:
def is_configured(self):
return self.__configured
+ def is_acl_configured(self):
+ return self.__acl_configured
+
def set_configured(self):
File(self.__params.parsers_configured_flag_file,
content="",
owner=self.__params.metron_user,
- mode=0775)
+ mode=0755)
+
+ def set_acl_configured(self):
+ File(self.__params.parsers_acl_configured_flag_file,
+ content="",
+ owner=self.__params.metron_user,
+ mode=0755)
def init_parsers(self):
Logger.info(
"Copying grok patterns from local directory '{0}' to HDFS '{1}'".format(self.__params.local_grok_patterns_dir,
self.__params.hdfs_grok_patterns_dir))
+
self.__params.HdfsResource(self.__params.hdfs_grok_patterns_dir,
type="directory",
action="create_on_execute",
owner=self.__params.metron_user,
- mode=0775,
+ mode=0755,
source=self.__params.local_grok_patterns_dir)
Logger.info("Done initializing parser configuration")
@@ -72,39 +85,47 @@ class ParserCommands:
return self.__parser_list
def init_kafka_topics(self):
- Logger.info('Creating Kafka topics')
- command_template = """{0}/kafka-topics.sh \
- --zookeeper {1} \
- --create \
- --topic {2} \
- --partitions {3} \
- --replication-factor {4} \
- --config retention.bytes={5}"""
- num_partitions = 1
- replication_factor = 1
- retention_gigabytes = int(self.__params.metron_topic_retention)
- retention_bytes = retention_gigabytes * 1024 * 1024 * 1024
- Logger.info("Creating main topics for parsers")
- for parser_name in self.get_parser_list():
- Logger.info("Creating topic'{0}'".format(parser_name))
- Execute(command_template.format(self.__params.kafka_bin_dir,
- self.__params.zookeeper_quorum,
- parser_name,
- num_partitions,
- replication_factor,
- retention_bytes))
- Logger.info("Done creating Kafka topics")
+ Logger.info('Creating Kafka topics for parsers')
+ # All errors go to indexing topics, so create it here if it's not already
+ # Getting topics this way is a bit awkward, but I don't want to append to actual list, so copy it
+ topics = list(self.get_parser_list())
+ topics.append(self.__params.metron_error_topic)
+ metron_service.init_kafka_topics(self.__params, topics)
+
+ def init_kafka_acls(self):
+ Logger.info('Creating Kafka ACLs for parsers')
+
+ # Getting topics this way is a bit awkward, but I don't want to modify the actual list, so copy it
+ topics = list(self.get_parser_list())
+ topics.append(self.__params.metron_error_topic)
+ # Parser group is the parser name + '_parser'
+ metron_service.init_kafka_acls(self.__params,
+ topics,
+ [parser + '_parser' for parser in self.get_parser_list()])
def start_parser_topologies(self):
Logger.info("Starting Metron parser topologies: {0}".format(self.get_parser_list()))
start_cmd_template = """{0}/bin/start_parser_topology.sh \
-k {1} \
-z {2} \
- -s {3}"""
+ -s {3} \
+ -ksp {4}"""
+ if self.__params.security_enabled:
+ # Append the extra configs needed for secured cluster.
+ start_cmd_template = start_cmd_template + ' -e ~' + self.__params.metron_user + '/.storm/storm.config'
+ if self.__params.security_enabled:
+ metron_security.kinit(self.__params.kinit_path_local,
+ self.__params.metron_keytab_path,
+ self.__params.metron_principal_name,
+ execute_user=self.__params.metron_user)
for parser in self.get_parser_list():
Logger.info('Starting ' + parser)
- Execute(start_cmd_template.format(self.__params.metron_home, self.__params.kafka_brokers,
- self.__params.zookeeper_quorum, parser))
+ Execute(start_cmd_template.format(self.__params.metron_home,
+ self.__params.kafka_brokers,
+ self.__params.zookeeper_quorum,
+ parser,
+ self.__params.kafka_security_protocol),
+ user=self.__params.metron_user)
Logger.info('Finished starting parser topologies')
@@ -113,7 +134,12 @@ class ParserCommands:
for parser in self.get_parser_list():
Logger.info('Stopping ' + parser)
stop_cmd = 'storm kill ' + parser
- Execute(stop_cmd)
+ if self.__params.security_enabled:
+ metron_security.kinit(self.__params.kinit_path_local,
+ self.__params.metron_keytab_path,
+ self.__params.metron_principal_name,
+ execute_user=self.__params.metron_user)
+ Execute(stop_cmd, user=self.__params.metron_user)
Logger.info('Done stopping parser topologies')
def restart_parser_topologies(self, env):
@@ -144,7 +170,7 @@ class ParserCommands:
def topologies_running(self, env):
env.set_params(self.__params)
all_running = True
- topologies = metron_service.get_running_topologies()
+ topologies = metron_service.get_running_topologies(self.__params)
for parser in self.get_parser_list():
parser_found = False
is_running = False
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/187ef373/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/parser_master.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/parser_master.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/parser_master.py
index 2721d13..6f43de4 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/parser_master.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/parser_master.py
@@ -21,6 +21,7 @@ from resource_management.core.exceptions import ComponentIsNotRunning
from resource_management.core.logger import Logger
from resource_management.libraries.script import Script
+from metron_security import storm_security_setup
import metron_service
from parser_commands import ParserCommands
@@ -45,6 +46,12 @@ class ParserMaster(Script):
commands.init_parsers()
commands.init_kafka_topics()
commands.set_configured()
+ if params.security_enabled and not commands.is_acl_configured():
+ commands.init_kafka_acls()
+ commands.set_acl_configured()
+
+ Logger.info("Calling security setup")
+ storm_security_setup(params)
def start(self, env, upgrade_type=None):
from params import params
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/187ef373/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/client_jaas.conf.j2
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/client_jaas.conf.j2 b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/client_jaas.conf.j2
new file mode 100644
index 0000000..ff2498c
--- /dev/null
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/client_jaas.conf.j2
@@ -0,0 +1,44 @@
+{#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#}
+
+StormClient {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=false
+ storeKey=false
+ useTicketCache=true
+ serviceName="nimbus"
+ principal="{{metron_principal_name}}";
+};
+Client {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ keyTab="{{metron_keytab_path}}"
+ storeKey=true
+ useTicketCache=false
+ serviceName="zookeeper"
+ principal="{{metron_principal_name}}";
+};
+KafkaClient {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ keyTab="{{metron_keytab_path}}"
+ storeKey=true
+ useTicketCache=false
+ serviceName="kafka"
+ principal="{{metron_principal_name}}";
+};
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/187ef373/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/enrichment.properties.j2
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/enrichment.properties.j2 b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/enrichment.properties.j2
index 508bce9..bdafb45 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/enrichment.properties.j2
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/enrichment.properties.j2
@@ -1,29 +1,31 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
+{#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#}
##### Kafka #####
kafka.zk={{zookeeper_quorum}}
kafka.broker={{kafka_brokers}}
-kafka.security.protocol=PLAINTEXT
-topology.worker.childopts=
+kafka.security.protocol={{kafka_security_protocol}}
+topology.worker.childopts={{topology_worker_childopts}}
+topology.auto-credentials={{topology_auto_credentials}}
enrichment.output.topic=indexing
-enrichment.error.topic=enrichments_error
-threat.intel.error.topic=threatintel_error
+enrichment.error.topic=indexing
+threat.intel.error.topic=indexing
##### Metrics #####
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/187ef373/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/storm.config.j2
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/storm.config.j2 b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/storm.config.j2
new file mode 100644
index 0000000..caf1221
--- /dev/null
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/storm.config.j2
@@ -0,0 +1,22 @@
+{#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#}
+
+{
+ "topology.worker.childopts" : "-Djava.security.auth.login.config={{metron_home}}/client_jaas.conf",
+ "topology.auto-credentials" : {{topology_auto_credentials_double_quotes}}
+}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/187ef373/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/storm.yaml.j2
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/storm.yaml.j2 b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/storm.yaml.j2
new file mode 100644
index 0000000..62ec934
--- /dev/null
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/storm.yaml.j2
@@ -0,0 +1,21 @@
+{#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#}
+
+nimbus.seeds : {{nimbus_seeds}}
+java.security.auth.login.config : '{{metron_home}}/client_jaas.conf'
+storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/187ef373/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py
index 8aff88c..c16e3e6 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py
@@ -97,6 +97,8 @@ class METRON${metron.short.version}ServiceAdvisor(service_advisor.ServiceAdvisor
return items
def getServiceConfigurationRecommendations(self, configurations, clusterData, services, hosts):
+ is_secured = self.isSecurityEnabled(services)
+
#Suggest Storm Rest URL
if "storm-site" in services["configurations"]:
stormUIServerHost = self.getComponentHostNames(services, "STORM", "STORM_UI_SERVER")[0]
@@ -108,7 +110,7 @@ class METRON${metron.short.version}ServiceAdvisor(service_advisor.ServiceAdvisor
storm_site = services["configurations"]["storm-site"]["properties"]
putStormSiteProperty = self.putProperty(configurations, "storm-site", services)
- for property, desired_value in self.getSTORMSiteDesiredValues().iteritems():
+ for property, desired_value in self.getSTORMSiteDesiredValues(is_secured).iteritems():
if property not in storm_site:
putStormSiteProperty(property, desired_value)
elif property == "topology.classpath" and storm_site[property] != desired_value:
@@ -128,11 +130,13 @@ class METRON${metron.short.version}ServiceAdvisor(service_advisor.ServiceAdvisor
putMetronEnvProperty("zeppelin_server_url", zeppelinServerUrl)
def validateSTORMSiteConfigurations(self, properties, recommendedDefaults, configurations, services, hosts):
+ # Determine if the cluster is secured
+ is_secured = self.isSecurityEnabled(services)
storm_site = properties
validationItems = []
- for property, desired_value in self.getSTORMSiteDesiredValues().iteritems():
+ for property, desired_value in self.getSTORMSiteDesiredValues(is_secured).iteritems():
if property not in storm_site :
message = "Metron requires this property to be set to the recommended value of " + desired_value
item = self.getErrorItem(message) if property == "topology.classpath" else self.getWarnItem(message)
@@ -147,11 +151,15 @@ class METRON${metron.short.version}ServiceAdvisor(service_advisor.ServiceAdvisor
return self.toConfigurationValidationProblems(validationItems, "storm-site")
- def getSTORMSiteDesiredValues(self):
+ def getSTORMSiteDesiredValues(self, is_secured):
storm_site_desired_values = {
"topology.classpath" : "/etc/hbase/conf:/etc/hadoop/conf"
}
+ if is_secured:
+ storm_site_desired_values.update({
+ "nimbus.credential.renewers.classes":"['org.apache.storm.security.auth.kerberos.AutoTGT']",
+ "supervisor.run.worker.as.user":"true"
+ })
return storm_site_desired_values
-
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/187ef373/metron-deployment/vagrant/Kerberos-setup.md
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/Kerberos-setup.md b/metron-deployment/vagrant/Kerberos-setup.md
index a66da8a..27a56b0 100644
--- a/metron-deployment/vagrant/Kerberos-setup.md
+++ b/metron-deployment/vagrant/Kerberos-setup.md
@@ -1,5 +1,5 @@
# Setting Up Kerberos in Vagrant Full Dev
-**Note:** These are manual instructions for Kerberizing Metron Storm topologies from Kafka to Kafka. This does not cover the Ambari MPack, sensor connections, or MAAS.
+**Note:** These are instructions for Kerberizing Metron Storm topologies from Kafka to Kafka. This does not cover the sensor connections or MAAS.
1. Build full dev and ssh into the machine
```
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/187ef373/metron-platform/metron-elasticsearch/src/main/config/elasticsearch.properties
----------------------------------------------------------------------
diff --git a/metron-platform/metron-elasticsearch/src/main/config/elasticsearch.properties b/metron-platform/metron-elasticsearch/src/main/config/elasticsearch.properties
index 47a5092..d5efcb2 100644
--- a/metron-platform/metron-elasticsearch/src/main/config/elasticsearch.properties
+++ b/metron-platform/metron-elasticsearch/src/main/config/elasticsearch.properties
@@ -18,6 +18,7 @@
indexing.workers=1
indexing.executors=0
topology.worker.childopts=
+topology.auto-credentials=['']
##### Kafka #####
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/187ef373/metron-platform/metron-enrichment/src/main/flux/enrichment/remote.yaml
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/src/main/flux/enrichment/remote.yaml b/metron-platform/metron-enrichment/src/main/flux/enrichment/remote.yaml
index 82863ed..e4f119e 100644
--- a/metron-platform/metron-enrichment/src/main/flux/enrichment/remote.yaml
+++ b/metron-platform/metron-enrichment/src/main/flux/enrichment/remote.yaml
@@ -19,6 +19,7 @@ config:
topology.workers: 1
topology.acker.executors: 0
topology.worker.childopts: ${topology.worker.childopts}
+ topology.auto-credentials: ${topology.auto-credentials}
components:
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/187ef373/metron-platform/metron-indexing/src/main/flux/indexing/remote.yaml
----------------------------------------------------------------------
diff --git a/metron-platform/metron-indexing/src/main/flux/indexing/remote.yaml b/metron-platform/metron-indexing/src/main/flux/indexing/remote.yaml
index 25ecdab..a8615fb 100644
--- a/metron-platform/metron-indexing/src/main/flux/indexing/remote.yaml
+++ b/metron-platform/metron-indexing/src/main/flux/indexing/remote.yaml
@@ -20,6 +20,7 @@ config:
topology.workers: ${indexing.workers}
topology.acker.executors: ${indexing.executors}
topology.worker.childopts: ${topology.worker.childopts}
+ topology.auto-credentials: ${topology.auto-credentials}
components:
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/187ef373/metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/integration/IndexingIntegrationTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/integration/IndexingIntegrationTest.java b/metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/integration/IndexingIntegrationTest.java
index cc7d7e3..ff786cf 100644
--- a/metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/integration/IndexingIntegrationTest.java
+++ b/metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/integration/IndexingIntegrationTest.java
@@ -128,6 +128,7 @@ public abstract class IndexingIntegrationTest extends BaseIntegrationTest {
setProperty("index.input.topic", Constants.INDEXING_TOPIC);
setProperty("index.error.topic", ERROR_TOPIC);
setProperty("index.date.format", dateFormat);
+ setProperty("topology.auto-credentials", "[]");
//HDFS settings
setProperty("bolt.hdfs.rotation.policy", TimedRotationPolicy.class.getCanonicalName());
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/187ef373/metron-platform/metron-solr/src/main/config/solr.properties
----------------------------------------------------------------------
diff --git a/metron-platform/metron-solr/src/main/config/solr.properties b/metron-platform/metron-solr/src/main/config/solr.properties
index e20480d..f0eca6c 100644
--- a/metron-platform/metron-solr/src/main/config/solr.properties
+++ b/metron-platform/metron-solr/src/main/config/solr.properties
@@ -18,6 +18,7 @@
indexing.workers=1
indexing.executors=0
topology.worker.childopts=
+topology.auto-credentials=['']
##### Kafka #####
[18/26] incubator-metron git commit: METRON-883 Capture Bro Plugin
Enhancements from bro/bro-plugins (nickwallen) closes
apache/incubator-metron#545
Posted by ma...@apache.org.
METRON-883 Capture Bro Plugin Enhancements from bro/bro-plugins (nickwallen) closes apache/incubator-metron#545
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/19e0e715
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/19e0e715
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/19e0e715
Branch: refs/heads/Metron_0.4.0
Commit: 19e0e715d30670612b0ad826ed133ba138bdc20f
Parents: fbce3b5
Author: nickwallen <ni...@nickallen.org>
Authored: Tue Apr 25 19:42:28 2017 -0400
Committer: nickallen <ni...@apache.org>
Committed: Tue Apr 25 19:42:28 2017 -0400
----------------------------------------------------------------------
metron-sensors/bro-plugin-kafka/README.md | 106 +++++++++++++------
.../bro-plugin-kafka/cmake/FindLibRDKafka.cmake | 30 +++---
.../bro-plugin-kafka/cmake/FindOpenSSL.cmake | 2 +
.../scripts/Bro/Kafka/__load__.bro | 2 +
.../scripts/Bro/Kafka/logs-to-kafka.bro | 3 +-
.../bro-plugin-kafka/src/KafkaWriter.cc | 85 +++++++++------
.../bro-plugin-kafka/src/KafkaWriter.h | 15 +++
7 files changed, 163 insertions(+), 80 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/19e0e715/metron-sensors/bro-plugin-kafka/README.md
----------------------------------------------------------------------
diff --git a/metron-sensors/bro-plugin-kafka/README.md b/metron-sensors/bro-plugin-kafka/README.md
index 6d0582e..31b1f54 100644
--- a/metron-sensors/bro-plugin-kafka/README.md
+++ b/metron-sensors/bro-plugin-kafka/README.md
@@ -1,60 +1,98 @@
-Bro Logging Output to Kafka
+Logging Bro Output to Kafka
===========================
-A Bro log writer that sends logging output to Kafka. This provides a convenient
-means for tools in the Hadoop ecosystem, such as Storm, Spark, and others, to
-process the data generated by Bro.
+A Bro log writer that sends logging output to Kafka. This provides a convenient means for tools in the Hadoop ecosystem, such as Storm, Spark, and others, to process the data generated by Bro.
Installation
------------
-Install librdkafka (https://github.com/edenhill/librdkafka), a native client
-library for Kafka. This plugin has been tested against the latest release of
-librdkafka, which at the time of this writing is v0.9.4. In order to support interacting
-with a kerberized kafka, you will need libsasl2 installed
+1. Install [librdkafka](https://github.com/edenhill/librdkafka), a native client library for Kafka. This plugin has been tested against the latest release of librdkafka, which at the time of this writing is v0.9.4.
-```
-# curl -L https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz | tar xvz
-# cd librdkafka-0.9.4/
-# ./configure --enable-sasl
-# make
-# sudo make install
-```
+ In order to use this plugin within a kerberized Kafka environment, you will also need `libsasl2` installed and will need to pass `--enable-sasl` to the `configure` script.
-Then compile this Bro plugin using the following commands.
+ ```
+ curl -L https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz | tar xvz
+ cd librdkafka-0.9.4/
+ ./configure --enable-sasl
+ make
+ sudo make install
+ ```
-```
-# ./configure --bro-dist=$BRO_SRC
-# make
-# sudo make install
-```
+1. Build the plugin using the following commands.
-Run the following command to ensure that the plugin was installed successfully.
+ ```
+ ./configure --bro-dist=$BRO_SRC
+ make
+ sudo make install
+ ```
-```
-# bro -N Bro::Kafka
-Bro::Kafka - Writes logs to Kafka (dynamic, version 0.1)
-```
+1. Run the following command to ensure that the plugin was installed successfully.
+
+ ```
+ $ bro -N Bro::Kafka
+ Bro::Kafka - Writes logs to Kafka (dynamic, version 0.1)
+ ```
Activation
----------
-The easiest way to enable Kafka output is to load the plugin's
-`logs-to-kafka.bro` script. If you are using BroControl, the following lines
-added to local.bro will activate it.
+The following examples highlight different ways that the plugin can be used. Simply add Bro script to your `local.bro` file (for example, `/usr/share/bro/site/local.bro`) as shown to activate the plugin.
+
+### Example 1
+
+The goal in this example is to send all HTTP and DNS records to a Kafka topic named `bro`.
+ * Any configuration value accepted by librdkafka can be added to the `kafka_conf` configuration table.
+ * By defining `topic_name` all records will be sent to the same Kafka topic.
```
@load Bro/Kafka/logs-to-kafka.bro
-redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG);
+redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG);
redef Kafka::topic_name = "bro";
redef Kafka::kafka_conf = table(
["metadata.broker.list"] = "localhost:9092"
);
```
-This example will send all HTTP, DNS, and Conn logs to a Kafka broker running on
-the localhost to a topic called `bro`. Any configuration value accepted by
-librdkafka can be added to the `kafka_conf` configuration table.
+### Example 2
+
+It is also possible to send each log stream to a uniquely named topic. The goal in this example is to send all HTTP records to a Kafka topic named `http` and all DNS records to a separate Kafka topic named `dns`.
+ * The `topic_name` value must be set to an empty string.
+ * The `$path` value of Bro's Log Writer mechanism is used to define the topic name.
+ * Any configuration value accepted by librdkafka can be added to the `$config` configuration table.
+ * Each log writer accepts a separate configuration table.
+
+```
+@load Bro/Kafka/logs-to-kafka.bro
+redef Kafka::topic_name = "";
+redef Kafka::tag_json = T;
+
+event bro_init()
+{
+ # handles HTTP
+ local http_filter: Log::Filter = [
+ $name = "kafka-http",
+ $writer = Log::WRITER_KAFKAWRITER,
+ $config = table(
+ ["stream_id"] = "HTTP::LOG",
+ ["metadata.broker.list"] = "localhost:9092"
+ ),
+ $path = "http"
+ ];
+ Log::add_filter(HTTP::LOG, http_filter);
+
+ # handles DNS
+ local dns_filter: Log::Filter = [
+ $name = "kafka-dns",
+ $writer = Log::WRITER_KAFKAWRITER,
+ $config = table(
+ ["stream_id"] = "DNS::LOG",
+ ["metadata.broker.list"] = "localhost:9092"
+ ),
+ $path = "dns"
+ ];
+ Log::add_filter(DNS::LOG, dns_filter);
+}
+```
Settings
--------
@@ -147,7 +185,7 @@ For an environment where the following is true:
The kafka topic `bro` has been given permission for the `metron` user to
write:
```
-# login using the metron user
+# login using the metron user
kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
${KAFKA_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:metron --topic bro
```
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/19e0e715/metron-sensors/bro-plugin-kafka/cmake/FindLibRDKafka.cmake
----------------------------------------------------------------------
diff --git a/metron-sensors/bro-plugin-kafka/cmake/FindLibRDKafka.cmake b/metron-sensors/bro-plugin-kafka/cmake/FindLibRDKafka.cmake
index c64d8f9..904bfff 100644
--- a/metron-sensors/bro-plugin-kafka/cmake/FindLibRDKafka.cmake
+++ b/metron-sensors/bro-plugin-kafka/cmake/FindLibRDKafka.cmake
@@ -16,34 +16,36 @@
#
find_path(LibRDKafka_ROOT_DIR
- NAMES include/librdkafka/rdkafkacpp.h
+ NAMES include/librdkafka/rdkafkacpp.h
)
find_library(LibRDKafka_LIBRARIES
- NAMES rdkafka++
- HINTS ${LibRDKafka_ROOT_DIR}/lib
+ NAMES rdkafka++
+ HINTS ${LibRDKafka_ROOT_DIR}/lib
+ PATH_SUFFIXES ${CMAKE_LIBRARY_ARCHITECTURE}
)
find_library(LibRDKafka_C_LIBRARIES
- NAMES rdkafka
- HINTS ${LibRDKafka_ROT_DIR}/lib
+ NAMES rdkafka
+ HINTS ${LibRDKafka_ROT_DIR}/lib
+ PATH_SUFFIXES ${CMAKE_LIBRARY_ARCHITECTURE}
)
find_path(LibRDKafka_INCLUDE_DIR
- NAMES librdkafka/rdkafkacpp.h
- HINTS ${LibRDKafka_ROOT_DIR}/include
+ NAMES librdkafka/rdkafkacpp.h
+ HINTS ${LibRDKafka_ROOT_DIR}/include
)
include(FindPackageHandleStandardArgs)
find_package_handle_standard_args(LibRDKafka DEFAULT_MSG
- LibRDKafka_LIBRARIES
- LibRDKafka_C_LIBRARIES
- LibRDKafka_INCLUDE_DIR
+ LibRDKafka_LIBRARIES
+ LibRDKafka_C_LIBRARIES
+ LibRDKafka_INCLUDE_DIR
)
mark_as_advanced(
- LibRDKafka_ROOT_DIR
- LibRDKafka_LIBRARIES
- LibRDKafka_C_LIBRARIES
- LibRDKafka_INCLUDE_DIR
+ LibRDKafka_ROOT_DIR
+ LibRDKafka_LIBRARIES
+ LibRDKafka_C_LIBRARIES
+ LibRDKafka_INCLUDE_DIR
)
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/19e0e715/metron-sensors/bro-plugin-kafka/cmake/FindOpenSSL.cmake
----------------------------------------------------------------------
diff --git a/metron-sensors/bro-plugin-kafka/cmake/FindOpenSSL.cmake b/metron-sensors/bro-plugin-kafka/cmake/FindOpenSSL.cmake
index 5ed955c..58af5c7 100644
--- a/metron-sensors/bro-plugin-kafka/cmake/FindOpenSSL.cmake
+++ b/metron-sensors/bro-plugin-kafka/cmake/FindOpenSSL.cmake
@@ -47,11 +47,13 @@ find_path(OpenSSL_INCLUDE_DIR
find_library(OpenSSL_SSL_LIBRARY
NAMES ssl ssleay32 ssleay32MD
HINTS ${OpenSSL_ROOT_DIR}/lib
+ PATH_SUFFIXES ${CMAKE_LIBRARY_ARCHITECTURE}
)
find_library(OpenSSL_CRYPTO_LIBRARY
NAMES crypto
HINTS ${OpenSSL_ROOT_DIR}/lib
+ PATH_SUFFIXES ${CMAKE_LIBRARY_ARCHITECTURE}
)
set(OpenSSL_LIBRARIES ${OpenSSL_SSL_LIBRARY} ${OpenSSL_CRYPTO_LIBRARY}
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/19e0e715/metron-sensors/bro-plugin-kafka/scripts/Bro/Kafka/__load__.bro
----------------------------------------------------------------------
diff --git a/metron-sensors/bro-plugin-kafka/scripts/Bro/Kafka/__load__.bro b/metron-sensors/bro-plugin-kafka/scripts/Bro/Kafka/__load__.bro
index 12295a9..1df1136 100644
--- a/metron-sensors/bro-plugin-kafka/scripts/Bro/Kafka/__load__.bro
+++ b/metron-sensors/bro-plugin-kafka/scripts/Bro/Kafka/__load__.bro
@@ -17,3 +17,5 @@
# This is loaded when a user activates the plugin. Include scripts here that should be
# loaded automatically at that point.
#
+
+@load ./init.bro
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/19e0e715/metron-sensors/bro-plugin-kafka/scripts/Bro/Kafka/logs-to-kafka.bro
----------------------------------------------------------------------
diff --git a/metron-sensors/bro-plugin-kafka/scripts/Bro/Kafka/logs-to-kafka.bro b/metron-sensors/bro-plugin-kafka/scripts/Bro/Kafka/logs-to-kafka.bro
index 84e390c..d62e03f 100644
--- a/metron-sensors/bro-plugin-kafka/scripts/Bro/Kafka/logs-to-kafka.bro
+++ b/metron-sensors/bro-plugin-kafka/scripts/Bro/Kafka/logs-to-kafka.bro
@@ -35,7 +35,8 @@ event bro_init() &priority=-5
{
local filter: Log::Filter = [
$name = fmt("kafka-%s", stream_id),
- $writer = Log::WRITER_KAFKAWRITER
+ $writer = Log::WRITER_KAFKAWRITER,
+ $config = table(["stream_id"] = fmt("%s", stream_id))
];
Log::add_filter(stream_id, filter);
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/19e0e715/metron-sensors/bro-plugin-kafka/src/KafkaWriter.cc
----------------------------------------------------------------------
diff --git a/metron-sensors/bro-plugin-kafka/src/KafkaWriter.cc b/metron-sensors/bro-plugin-kafka/src/KafkaWriter.cc
index 79a85ed..951a60c 100644
--- a/metron-sensors/bro-plugin-kafka/src/KafkaWriter.cc
+++ b/metron-sensors/bro-plugin-kafka/src/KafkaWriter.cc
@@ -22,9 +22,35 @@ using namespace writer;
KafkaWriter::KafkaWriter(WriterFrontend* frontend): WriterBackend(frontend), formatter(NULL), producer(NULL), topic(NULL)
{
- // TODO do we need this??
- topic_name.assign((const char*)BifConst::Kafka::topic_name->Bytes(),
- BifConst::Kafka::topic_name->Len());
+ // need thread-local copies of all user-defined settings coming from
+ // bro scripting land. accessing these is not thread-safe and 'DoInit'
+ // is potentially accessed from multiple threads.
+
+ // tag_json - thread local copy
+ tag_json = BifConst::Kafka::tag_json;
+
+ // topic name - thread local copy
+ topic_name.assign(
+ (const char*)BifConst::Kafka::topic_name->Bytes(),
+ BifConst::Kafka::topic_name->Len());
+
+ // kafka_conf - thread local copy
+ Val* val = BifConst::Kafka::kafka_conf->AsTableVal();
+ IterCookie* c = val->AsTable()->InitForIteration();
+ HashKey* k;
+ TableEntryVal* v;
+ while ((v = val->AsTable()->NextEntry(k, c))) {
+
+ // fetch the key and value
+ ListVal* index = val->AsTableVal()->RecoverIndex(k);
+ string key = index->Index(0)->AsString()->CheckString();
+ string val = v->Value()->AsString()->CheckString();
+ kafka_conf.insert (kafka_conf.begin(), pair<string, string> (key, val));
+
+ // cleanup
+ Unref(index);
+ delete k;
+ }
}
KafkaWriter::~KafkaWriter()
@@ -32,6 +58,11 @@ KafkaWriter::~KafkaWriter()
bool KafkaWriter::DoInit(const WriterInfo& info, int num_fields, const threading::Field* const* fields)
{
+ // if no global 'topic_name' is defined, use the log stream's 'path'
+ if(topic_name.empty()) {
+ topic_name = info.path;
+ }
+
// initialize the formatter
if(BifConst::Kafka::tag_json) {
formatter = new threading::formatter::TaggedJSON(info.path, this, threading::formatter::JSON::TS_EPOCH);
@@ -39,8 +70,7 @@ bool KafkaWriter::DoInit(const WriterInfo& info, int num_fields, const threading
formatter = new threading::formatter::JSON(this, threading::formatter::JSON::TS_EPOCH);
}
- // kafka global configuration
- string err;
+ // is debug enabled
string debug;
debug.assign((const char*)BifConst::Kafka::debug->Bytes(), BifConst::Kafka::debug->Len());
bool is_debug(!debug.empty());
@@ -53,41 +83,31 @@ bool KafkaWriter::DoInit(const WriterInfo& info, int num_fields, const threading
else {
reporter->Info( "Debug is turned off.");
}
+
+ // kafka global configuration
+ string err;
conf = RdKafka::Conf::create(RdKafka::Conf::CONF_GLOBAL);
// apply the user-defined settings to kafka
- Val* val = BifConst::Kafka::kafka_conf->AsTableVal();
- IterCookie* c = val->AsTable()->InitForIteration();
- HashKey* k;
- TableEntryVal* v;
- while ((v = val->AsTable()->NextEntry(k, c))) {
-
- // fetch the key and value
- ListVal* index = val->AsTableVal()->RecoverIndex(k);
- string key = index->Index(0)->AsString()->CheckString();
- string val = v->Value()->AsString()->CheckString();
-
- if(is_debug) {
- reporter->Info("Setting '%s'='%s'", key.c_str(), val.c_str());
- }
- // apply setting to kafka
- if (RdKafka::Conf::CONF_OK != conf->set(key, val, err)) {
- reporter->Error("Failed to set '%s'='%s': %s", key.c_str(), val.c_str(), err.c_str());
- return false;
- }
-
- // cleanup
- Unref(index);
- delete k;
+ map<string,string>::iterator i;
+ for (i = kafka_conf.begin(); i != kafka_conf.end(); ++i) {
+ string key = i->first;
+ string val = i->second;
+
+ // apply setting to kafka
+ if (RdKafka::Conf::CONF_OK != conf->set(key, val, err)) {
+ reporter->Error("Failed to set '%s'='%s': %s", key.c_str(), val.c_str(), err.c_str());
+ return false;
+ }
}
if(is_debug) {
string key("debug");
string val(debug);
- if (RdKafka::Conf::CONF_OK != conf->set(key, val, err)) {
+ if (RdKafka::Conf::CONF_OK != conf->set(key, val, err)) {
reporter->Error("Failed to set '%s'='%s': %s", key.c_str(), val.c_str(), err.c_str());
return false;
- }
+ }
}
// create kafka producer
@@ -104,9 +124,11 @@ bool KafkaWriter::DoInit(const WriterInfo& info, int num_fields, const threading
reporter->Error("Failed to create topic handle: %s", err.c_str());
return false;
}
+
if(is_debug) {
reporter->Info("Successfully created producer.");
}
+
return true;
}
@@ -130,8 +152,9 @@ bool KafkaWriter::DoFinish(double network_time)
// successful only if all messages delivered
if (producer->outq_len() == 0) {
- reporter->Error("Unable to deliver %0d message(s)", producer->outq_len());
success = true;
+ } else {
+ reporter->Error("Unable to deliver %0d message(s)", producer->outq_len());
}
delete topic;
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/19e0e715/metron-sensors/bro-plugin-kafka/src/KafkaWriter.h
----------------------------------------------------------------------
diff --git a/metron-sensors/bro-plugin-kafka/src/KafkaWriter.h b/metron-sensors/bro-plugin-kafka/src/KafkaWriter.h
index 7e77bc0..ad3e03f 100644
--- a/metron-sensors/bro-plugin-kafka/src/KafkaWriter.h
+++ b/metron-sensors/bro-plugin-kafka/src/KafkaWriter.h
@@ -28,6 +28,17 @@
#include "kafka.bif.h"
#include "TaggedJSON.h"
+namespace RdKafka {
+ class Conf;
+ class Producer;
+ class Topic;
+}
+
+namespace threading {
+ namespace formatter {
+ class Formatter;
+}}
+
namespace logging { namespace writer {
/**
@@ -54,6 +65,10 @@ protected:
virtual bool DoHeartbeat(double network_time, double current_time);
private:
+ static const string default_topic_key;
+ string stream_id;
+ bool tag_json;
+ map<string, string> kafka_conf;
string topic_name;
threading::formatter::Formatter *formatter;
RdKafka::Producer* producer;
[24/26] incubator-metron git commit: METRON-870: Add filtering by
packet payload to the pcap query closes apache/incubator-metron#541
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/bf2528fd/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/filter/fixed/FixedPcapFilter.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/filter/fixed/FixedPcapFilter.java b/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/filter/fixed/FixedPcapFilter.java
index a7751b4..001b500 100644
--- a/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/filter/fixed/FixedPcapFilter.java
+++ b/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/filter/fixed/FixedPcapFilter.java
@@ -21,6 +21,7 @@ package org.apache.metron.pcap.filter.fixed;
import com.google.common.base.Joiner;
import org.apache.hadoop.conf.Configuration;
import org.apache.metron.common.Constants;
+import org.apache.metron.common.dsl.MapVariableResolver;
import org.apache.metron.common.dsl.VariableResolver;
import org.apache.metron.pcap.PacketInfo;
import org.apache.metron.pcap.PcapHelper;
@@ -28,69 +29,123 @@ import org.apache.metron.pcap.filter.PcapFilter;
import org.apache.metron.pcap.filter.PcapFilterConfigurator;
import org.apache.metron.pcap.filter.PcapFilters;
import org.apache.metron.pcap.filter.PcapFieldResolver;
+import org.apache.metron.pcap.pattern.ByteArrayMatchingUtil;
+import javax.xml.bind.DatatypeConverter;
import java.util.EnumMap;
import java.util.Map;
+import java.util.concurrent.ExecutionException;
public class FixedPcapFilter implements PcapFilter {
- public static class Configurator implements PcapFilterConfigurator<EnumMap<Constants.Fields, String>> {
+ public static class Configurator implements PcapFilterConfigurator<Map<String, String>> {
@Override
- public void addToConfig(EnumMap<Constants.Fields, String> fields, Configuration conf) {
- for (Map.Entry<Constants.Fields, String> kv : fields.entrySet()) {
- conf.set(kv.getKey().getName(), kv.getValue());
+ public void addToConfig(Map<String, String> fields, Configuration conf) {
+ for (Map.Entry<String, String> kv : fields.entrySet()) {
+ conf.set(kv.getKey(), kv.getValue());
}
conf.set(PCAP_FILTER_NAME_CONF, PcapFilters.FIXED.name());
}
@Override
- public String queryToString(EnumMap<Constants.Fields, String> fields) {
+ public String queryToString(Map<String, String> fields) {
return (fields == null ? "" : Joiner.on("_").join(fields.values()));
}
}
+ private String packetFilter;
private String srcAddr;
private Integer srcPort;
private String dstAddr;
private Integer dstPort;
private String protocol;
private boolean includesReverseTraffic = false;
+ private boolean doHeaderFiltering = false;
@Override
public void configure(Iterable<Map.Entry<String, String>> config) {
for (Map.Entry<String, String> kv : config) {
if (kv.getKey().equals(Constants.Fields.DST_ADDR.getName())) {
+ System.out.println("Processing: " + kv.getKey() + " => " + kv.getValue());
this.dstAddr = kv.getValue();
+ doHeaderFiltering = true;
}
if (kv.getKey().equals(Constants.Fields.SRC_ADDR.getName())) {
+ System.out.println("Processing: " + kv.getKey() + " => " + kv.getValue());
this.srcAddr = kv.getValue();
+ doHeaderFiltering = true;
}
if (kv.getKey().equals(Constants.Fields.DST_PORT.getName())) {
+ System.out.println("Processing: " + kv.getKey() + " => " + kv.getValue());
this.dstPort = Integer.parseInt(kv.getValue());
+ doHeaderFiltering = true;
}
if (kv.getKey().equals(Constants.Fields.SRC_PORT.getName())) {
+ System.out.println("Processing: " + kv.getKey() + " => " + kv.getValue());
this.srcPort = Integer.parseInt(kv.getValue());
+ doHeaderFiltering = true;
}
if (kv.getKey().equals(Constants.Fields.PROTOCOL.getName())) {
+ System.out.println("Processing: " + kv.getKey() + " => " + kv.getValue());
this.protocol = kv.getValue();
+ doHeaderFiltering = true;
}
if (kv.getKey().equals(Constants.Fields.INCLUDES_REVERSE_TRAFFIC.getName())) {
+ System.out.println("Processing: " + kv.getKey() + " => " + kv.getValue());
this.includesReverseTraffic = Boolean.parseBoolean(kv.getValue());
}
+ if(kv.getKey().equals(PcapHelper.PacketFields.PACKET_FILTER.getName())) {
+ System.out.println("Processing: " + kv.getKey() + " => " + kv.getValue());
+ this.packetFilter = kv.getValue();
+ }
}
}
@Override
public boolean test(PacketInfo pi) {
- VariableResolver resolver = new PcapFieldResolver(packetToFields(pi));
+ Map<String, Object> fields = packetToFields(pi);
+ VariableResolver resolver = new MapVariableResolver(fields);
String srcAddrIn = (String) resolver.resolve(Constants.Fields.SRC_ADDR.getName());
Integer srcPortIn = (Integer) resolver.resolve(Constants.Fields.SRC_PORT.getName());
String dstAddrIn = (String) resolver.resolve(Constants.Fields.DST_ADDR.getName());
Integer dstPortIn = (Integer) resolver.resolve(Constants.Fields.DST_PORT.getName());
+
String protocolIn = "" + resolver.resolve(Constants.Fields.PROTOCOL.getName());
+ if(!doHeaderFiltering || testHeader(srcAddrIn, srcPortIn, dstAddrIn, dstPortIn, protocolIn)) {
+ //if we don't do header filtering *or* if we have tested the header and decided it's a match
+ if(packetFilter != null) {
+ //and we have a packet filter, then we need to filter the packet
+ byte[] data = (byte[])resolver.resolve(PcapHelper.PacketFields.PACKET_DATA.getName());
+ try {
+ return ByteArrayMatchingUtil.INSTANCE.match(packetFilter, data);
+ } catch (ExecutionException e) {
+ throw new IllegalStateException("Unable to perform binary filter: " + packetFilter + " on " + DatatypeConverter.printHexBinary(data), e);
+ }
+ }
+ else if(!doHeaderFiltering){
+ //otherwise we aren't doing packet filtering either, so we aren't doing any filtering, then we should
+ //pass the test
+ return true;
+ }
+ else {
+ //and if we *are* doing header filtering and not packet filtering, then we want to pass the test
+ return true;
+ }
+ }
+ else {
+ //in this case we're doing header filtering and we failed the header filter test.
+ return false;
+ }
+ }
+ private boolean testHeader( String srcAddrIn
+ , Integer srcPortIn
+ , String dstAddrIn
+ , Integer dstPortIn
+ , String protocolIn
+ ) {
if (areMatch(protocol, protocolIn)) {
if (matchesSourceAndDestination(srcAddrIn, srcPortIn, dstAddrIn, dstPortIn)) {
return true;
@@ -102,7 +157,7 @@ public class FixedPcapFilter implements PcapFilter {
}
private boolean areMatch(Integer filter, Integer input) {
- return filter == null || areMatch(Integer.toUnsignedString(filter), Integer.toUnsignedString(input));
+ return filter == null || areMatch(Integer.toUnsignedString(filter), input == null?null:Integer.toUnsignedString(input));
}
private boolean areMatch(String filter, String input) {
@@ -113,7 +168,7 @@ public class FixedPcapFilter implements PcapFilter {
}
}
- protected EnumMap<Constants.Fields, Object> packetToFields(PacketInfo pi) {
+ protected Map<String, Object> packetToFields(PacketInfo pi) {
return PcapHelper.packetToFields(pi);
}
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/bf2528fd/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/filter/query/QueryPcapFilter.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/filter/query/QueryPcapFilter.java b/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/filter/query/QueryPcapFilter.java
index 28a8b93..d6fa203 100644
--- a/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/filter/query/QueryPcapFilter.java
+++ b/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/filter/query/QueryPcapFilter.java
@@ -21,6 +21,7 @@ package org.apache.metron.pcap.filter.query;
import org.apache.hadoop.conf.Configuration;
import org.apache.metron.common.Constants;
import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.MapVariableResolver;
import org.apache.metron.common.dsl.StellarFunctions;
import org.apache.metron.common.stellar.StellarPredicateProcessor;
import org.apache.metron.common.dsl.VariableResolver;
@@ -69,12 +70,12 @@ public class QueryPcapFilter implements PcapFilter {
@Override
public boolean test(PacketInfo input) {
- EnumMap<Constants.Fields, Object> fields = packetToFields(input);
- VariableResolver resolver = new PcapFieldResolver(fields);
+ Map<String, Object> fields = packetToFields(input);
+ VariableResolver resolver = new MapVariableResolver(fields);
return predicateProcessor.parse(queryString, resolver, StellarFunctions.FUNCTION_RESOLVER(), Context.EMPTY_CONTEXT());
}
- protected EnumMap<Constants.Fields, Object> packetToFields(PacketInfo pi) {
+ protected Map<String, Object> packetToFields(PacketInfo pi) {
return PcapHelper.packetToFields(pi);
}
}
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/bf2528fd/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/pattern/ByteArrayMatcherFunction.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/pattern/ByteArrayMatcherFunction.java b/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/pattern/ByteArrayMatcherFunction.java
new file mode 100644
index 0000000..a4a74f3
--- /dev/null
+++ b/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/pattern/ByteArrayMatcherFunction.java
@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.pcap.pattern;
+
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+import org.apache.metron.common.dsl.StellarFunction;
+
+import javax.xml.bind.DatatypeConverter;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+
+@Stellar(namespace="BYTEARRAY"
+ ,name="MATCHER"
+ ,description = "Determine if a regex defined sequence of bytes (or strings) exists in a byte array."
+ ,params = {
+ "binary_regex - Regex defining what to look for in the byte array. Note syntax guide for binary regex is at https://github.com/nishihatapalmer/byteseek/blob/master/sequencesyntax.md"
+ ,"data - The byte array to evaluate."
+ }
+ ,returns="result: Boolean indicating whether or not the byte array is a match."
+ )
+public class ByteArrayMatcherFunction implements StellarFunction {
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+ if(args.size() != 2) {
+ return new IllegalStateException("Expected 2 arguments: regex and data");
+ }
+ String regex = (String)args.get(0);
+ byte[] data = (byte[])args.get(1);
+ try {
+ return ByteArrayMatchingUtil.INSTANCE.match(regex, data);
+ }
+ catch (ExecutionException e) {
+ throw new IllegalStateException("Unable to process " + regex + " against " + DatatypeConverter.printHexBinary(data));
+ }
+ }
+
+ @Override
+ public void initialize(Context context) {
+
+ }
+
+ @Override
+ public boolean isInitialized() {
+ return true;
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/bf2528fd/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/pattern/ByteArrayMatchingUtil.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/pattern/ByteArrayMatchingUtil.java b/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/pattern/ByteArrayMatchingUtil.java
new file mode 100644
index 0000000..bd57b7c
--- /dev/null
+++ b/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/pattern/ByteArrayMatchingUtil.java
@@ -0,0 +1,64 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.pcap.pattern;
+
+import com.google.common.cache.CacheBuilder;
+import com.google.common.cache.CacheLoader;
+import com.google.common.cache.LoadingCache;
+import net.byteseek.compiler.CompileException;
+import net.byteseek.compiler.matcher.MatcherCompilerUtils;
+import net.byteseek.compiler.matcher.SequenceMatcherCompiler;
+import net.byteseek.matcher.sequence.SequenceMatcher;
+import net.byteseek.searcher.Searcher;
+import net.byteseek.searcher.sequence.horspool.BoyerMooreHorspoolSearcher;
+import net.byteseek.searcher.sequence.horspool.HorspoolFinalFlagSearcher;
+
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+
+public enum ByteArrayMatchingUtil {
+ INSTANCE;
+ private LoadingCache<String, Searcher<SequenceMatcher>> sequenceMatchers = CacheBuilder.newBuilder()
+ .maximumSize(1000)
+ .expireAfterWrite(10, TimeUnit.MINUTES)
+ .build(
+ new CacheLoader<String, Searcher<SequenceMatcher>>() {
+ public Searcher<SequenceMatcher> load(String pattern) throws Exception {
+ return new HorspoolFinalFlagSearcher(compile(pattern));
+ }
+ });
+ private SequenceMatcherCompiler compiler = new SequenceMatcherCompiler();
+
+ private SequenceMatcher compile(String pattern) throws CompileException {
+
+ return compiler.compile(pattern);
+ }
+
+ public boolean match(String pattern, byte[] data) throws ExecutionException {
+ if(pattern == null) {
+ return false;
+ }
+ Searcher<SequenceMatcher> searcher = sequenceMatchers.get(pattern);
+ if(data == null) {
+ return false;
+ }
+ else {
+ return !searcher.searchForwards(data).isEmpty();
+ }
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/bf2528fd/metron-platform/metron-pcap/src/test/java/org/apache/metron/pcap/filter/fixed/FixedPcapFilterTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap/src/test/java/org/apache/metron/pcap/filter/fixed/FixedPcapFilterTest.java b/metron-platform/metron-pcap/src/test/java/org/apache/metron/pcap/filter/fixed/FixedPcapFilterTest.java
index b75b9c8..af2afd3 100644
--- a/metron-platform/metron-pcap/src/test/java/org/apache/metron/pcap/filter/fixed/FixedPcapFilterTest.java
+++ b/metron-platform/metron-pcap/src/test/java/org/apache/metron/pcap/filter/fixed/FixedPcapFilterTest.java
@@ -22,7 +22,7 @@ import org.apache.metron.common.Constants;
import org.junit.Assert;
import org.junit.Test;
-import java.util.EnumMap;
+import java.util.LinkedHashMap;
import static org.hamcrest.CoreMatchers.equalTo;
@@ -30,12 +30,12 @@ public class FixedPcapFilterTest {
@Test
public void string_representation_of_query_gets_formatted() throws Exception {
- final EnumMap<Constants.Fields, String> fields = new EnumMap<Constants.Fields, String>(Constants.Fields.class) {{
- put(Constants.Fields.SRC_ADDR, "src_ip");
- put(Constants.Fields.SRC_PORT, "0");
- put(Constants.Fields.DST_ADDR, "dst_ip");
- put(Constants.Fields.DST_PORT, "1");
- put(Constants.Fields.INCLUDES_REVERSE_TRAFFIC, "false");
+ final LinkedHashMap<String, String> fields = new LinkedHashMap<String, String>() {{
+ put(Constants.Fields.SRC_ADDR.getName(), "src_ip");
+ put(Constants.Fields.SRC_PORT.getName(), "0");
+ put(Constants.Fields.DST_ADDR.getName(), "dst_ip");
+ put(Constants.Fields.DST_PORT.getName(), "1");
+ put(Constants.Fields.INCLUDES_REVERSE_TRAFFIC.getName(), "false");
}};
String actual = new FixedPcapFilter.Configurator().queryToString(fields);
String expected = "src_ip_0_dst_ip_1_false";
@@ -45,7 +45,7 @@ public class FixedPcapFilterTest {
@Test
public void string_representation_of_empty_fields_empty() throws Exception {
{
- final EnumMap<Constants.Fields, String> fields = new EnumMap<Constants.Fields, String>(Constants.Fields.class);
+ final LinkedHashMap<String, String> fields = new LinkedHashMap<String, String>();
String actual = new FixedPcapFilter.Configurator().queryToString(fields);
String expected = "";
Assert.assertThat("string representation did not match", actual, equalTo(expected));
@@ -56,9 +56,9 @@ public class FixedPcapFilterTest {
Assert.assertThat("string representation did not match", actual, equalTo(expected));
}
{
- final EnumMap<Constants.Fields, String> fields = new EnumMap<Constants.Fields, String>(Constants.Fields.class) {{
- put(Constants.Fields.SRC_ADDR, "");
- put(Constants.Fields.SRC_PORT, "");
+ final LinkedHashMap<String, String> fields = new LinkedHashMap<String, String>() {{
+ put(Constants.Fields.SRC_ADDR.getName(), "");
+ put(Constants.Fields.SRC_PORT.getName(), "");
}};
String actual = new FixedPcapFilter.Configurator().queryToString(fields);
String expected = "_";
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/bf2528fd/metron-platform/metron-pcap/src/test/java/org/apache/metron/pcap/pattern/ByteArrayMatchingUtilTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap/src/test/java/org/apache/metron/pcap/pattern/ByteArrayMatchingUtilTest.java b/metron-platform/metron-pcap/src/test/java/org/apache/metron/pcap/pattern/ByteArrayMatchingUtilTest.java
new file mode 100644
index 0000000..37b3f3f
--- /dev/null
+++ b/metron-platform/metron-pcap/src/test/java/org/apache/metron/pcap/pattern/ByteArrayMatchingUtilTest.java
@@ -0,0 +1,133 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.pcap.pattern;
+
+import com.google.common.collect.ImmutableMap;
+import org.apache.metron.common.utils.StellarProcessorUtils;
+import org.apache.metron.pcap.pattern.ByteArrayMatchingUtil;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import javax.xml.bind.DatatypeConverter;
+import java.util.*;
+import java.util.concurrent.ExecutionException;
+import java.util.function.Predicate;
+
+@RunWith(Parameterized.class)
+public class ByteArrayMatchingUtilTest {
+ public static byte[] DEADBEEF = new byte[] {(byte) 0xde, (byte) 0xad, (byte) 0xbe, (byte) 0xef};
+ public static byte[] DEADBEEF_DONUTHOLE = new byte[] {(byte) 0xde, (byte) 0xad, (byte)0x00, (byte)0x00, (byte) 0xbe, (byte) 0xef};
+ public static byte[] ALLFS = new byte[] {(byte) 0xff, (byte) 0xff, (byte) 0xff, (byte) 0xff};
+ public static byte[] REALPACKET = DatatypeConverter.parseHexBinary("d4c3b2a1020004000000000000000000ffff0000010000000dfbea560923090062000000620000000a002700000008002796a47a0800451000547cdf40004006b7e9c0a84279c0a842010016d9ef129035948b92f427801801f5061200000101080a0014f5d511bdf9915bf6b70140db6d4fb551229ef07d2f56abd814bc56420489ca38e7faf8cec3d4");
+
+ interface Evaluator {
+ boolean evaluate(String pattern, byte[] data);
+ }
+
+ public enum EvaluationStrategy implements Evaluator{
+ STELLAR_WITH_VARIABLES((pattern, data) -> {
+ Map<String, Object> args = new HashMap<>();
+ args.put("pattern", pattern);
+ args.put("data", data);
+ return (boolean) StellarProcessorUtils.run("BYTEARRAY_MATCHER(pattern, data)" , args);
+ }
+ ),
+ STELLAR_WITH_PATTERN_STRING((pattern, data) -> {
+ Map<String, Object> args = new HashMap<>();
+ args.put("data", data);
+ return (boolean) StellarProcessorUtils.run(String.format("BYTEARRAY_MATCHER('%s', data)", pattern) , args);
+ }
+ )
+ , UTIL((pattern, data) -> {
+ try {
+ return ByteArrayMatchingUtil.INSTANCE.match(pattern, data);
+ } catch (ExecutionException e) {
+ throw new IllegalArgumentException(e);
+ }
+ })
+ ;
+ Evaluator evaluator;
+ EvaluationStrategy(Evaluator evaluator) {
+ this.evaluator = evaluator;
+ }
+ @Override
+ public boolean evaluate(String pattern, byte[] data) {
+ return evaluator.evaluate(pattern, data);
+ }
+ }
+ private EvaluationStrategy strategy = null;
+ public ByteArrayMatchingUtilTest(EvaluationStrategy strategy) {
+ this.strategy = strategy;
+ }
+
+ @Parameterized.Parameters
+ public static Collection<Object[]> strategies() {
+ List<Object[]> strategies = new ArrayList<>();
+ for(EvaluationStrategy s : EvaluationStrategy.values()) {
+ strategies.add(new Object[] { s });
+ }
+ return strategies;
+ }
+
+ @Test
+ public void testStringMatch() throws ExecutionException {
+ Assert.assertTrue(strategy.evaluate("`metron`", "metron".getBytes()));
+ Assert.assertTrue(strategy.evaluate("`metron`", "metron example".getBytes()));
+ Assert.assertTrue(strategy.evaluate("`metron`", "edward metron example".getBytes()));
+ Assert.assertFalse(strategy.evaluate("`metron`", "apache".getBytes()));
+ }
+
+ @Test
+ public void testBytesMatch() throws ExecutionException {
+ Assert.assertTrue(strategy.evaluate("2f56abd814bc56420489ca38e7faf8cec3d4", REALPACKET));
+ Assert.assertTrue(strategy.evaluate("2f56..14bc56420489ca38e7faf8cec3d4", REALPACKET));
+ Assert.assertTrue(strategy.evaluate("(2f56)(.){2}(14bc56420489ca38e7faf8cec3d4)", REALPACKET));
+ Assert.assertFalse(strategy.evaluate("(3f56)(.){2}(14bc56420489ca38e7faf8cec3d4)", REALPACKET));
+ Assert.assertFalse(strategy.evaluate("3f56abd814bc56420489ca38e7faf8cec3d4", REALPACKET));
+ Assert.assertTrue(strategy.evaluate("deadbeef", join(DEADBEEF, "metron".getBytes())));
+ Assert.assertTrue(strategy.evaluate("deadbeef", join(DEADBEEF, "metron".getBytes())));
+ Assert.assertTrue(strategy.evaluate("deadbeef `metron`", join(DEADBEEF, "metron".getBytes())));
+ Assert.assertTrue(strategy.evaluate("deadbeef `metron`", join(DEADBEEF, "metronjones".getBytes())));
+ Assert.assertTrue(strategy.evaluate("deadbeef `metron`", join(DEADBEEF, "metronjones".getBytes(), DEADBEEF)));
+ Assert.assertTrue(strategy.evaluate("([ff]){4}", ALLFS));
+ Assert.assertFalse(strategy.evaluate("([ff]){6}", ALLFS));
+ Assert.assertTrue(strategy.evaluate("[^ff]", new byte[] { (byte)0x00 }));
+ Assert.assertTrue(strategy.evaluate("&01", new byte[] { (byte)0x07 }));
+ Assert.assertFalse(strategy.evaluate("&01", new byte[] { (byte)0x00 }));
+ Assert.assertTrue(strategy.evaluate("&01", new byte[] { (byte)0x00, (byte)0x01 }));
+ Assert.assertTrue(strategy.evaluate("(dead).{2}(beef)", DEADBEEF_DONUTHOLE));
+ }
+
+ public byte[] join(byte[]... array) {
+ byte[] ret;
+ int size = 0;
+ for(int i = 0;i < array.length;++i) {
+ size += array[i].length;
+ }
+ ret = new byte[size];
+ int j = 0;
+ for(int i = 0;i < array.length;++i) {
+ for(int k = 0;k < array[i].length;++k,++j) {
+ ret[j] = array[i][k];
+ }
+ }
+ return ret;
+ }
+}
[19/26] incubator-metron git commit: METRON-888 Update README to show
requirement of Ambari 2.4.2 (dlyle via justinleet) closes
apache/incubator-metron#548
Posted by ma...@apache.org.
METRON-888 Update README to show requirement of Ambari 2.4.2 (dlyle via justinleet) closes apache/incubator-metron#548
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/cac696d7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/cac696d7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/cac696d7
Branch: refs/heads/Metron_0.4.0
Commit: cac696d7dbf9bec57cc9a59f599bded48186ca70
Parents: 19e0e71
Author: dlyle <dl...@gmail.com>
Authored: Wed Apr 26 09:16:33 2017 -0400
Committer: leet <le...@apache.org>
Committed: Wed Apr 26 09:16:33 2017 -0400
----------------------------------------------------------------------
metron-deployment/README.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/cac696d7/metron-deployment/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/README.md b/metron-deployment/README.md
index 4d285f4..adb48fd 100644
--- a/metron-deployment/README.md
+++ b/metron-deployment/README.md
@@ -59,7 +59,7 @@ This will set up
- Optional Kibana
### Prerequisites
-- A cluster managed by Ambari 2.4
+- A cluster managed by Ambari 2.4.2+
- Metron RPMs available on the cluster in the /localrepo directory. See [RPM](#rpm) for further information.
### Building Management Pack
[14/26] incubator-metron git commit: METRON-868 Fix documentation on
building RPMs (justinleet) closes apache/incubator-metron#538
Posted by ma...@apache.org.
METRON-868 Fix documentation on building RPMs (justinleet) closes apache/incubator-metron#538
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/bec84db7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/bec84db7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/bec84db7
Branch: refs/heads/Metron_0.4.0
Commit: bec84db776a08b55594bf2ac78613d58b0a65b90
Parents: 8fe52c9
Author: justinleet <ju...@gmail.com>
Authored: Mon Apr 24 11:51:15 2017 -0400
Committer: leet <le...@apache.org>
Committed: Mon Apr 24 11:51:15 2017 -0400
----------------------------------------------------------------------
metron-deployment/README.md | 12 ++++++++++++
metron-deployment/pom.xml | 14 --------------
2 files changed, 12 insertions(+), 14 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/bec84db7/metron-deployment/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/README.md b/metron-deployment/README.md
index 6f1653b..4d285f4 100644
--- a/metron-deployment/README.md
+++ b/metron-deployment/README.md
@@ -119,6 +119,18 @@ Components in the RPMs:
- Docker. The image detailed in: `metron-deployment/packaging/docker/rpm-docker/README.md` will automatically be built (or rebuilt if necessary).
- Artifacts for metron-platform have been produced. E.g. `mvn clean package -DskipTests` in `metron-platform`
+The artifacts are required because there is a dependency on modules not expressed via Maven (we grab the resulting assemblies, but don't need the jars). These are
+- metron-common
+- metron-data-management
+- metron-elasticsearch
+- metron-enrichment
+- metron-indexing
+- metron-parsers
+- metron-pcap-backend
+- metron-solr
+- metron-profiler
+- metron-config
+
### Building RPMs
From `metron-deployment` run
```
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/bec84db7/metron-deployment/pom.xml
----------------------------------------------------------------------
diff --git a/metron-deployment/pom.xml b/metron-deployment/pom.xml
index 86194a1..fa7e1f9 100644
--- a/metron-deployment/pom.xml
+++ b/metron-deployment/pom.xml
@@ -46,18 +46,4 @@
</modules>
</profile>
</profiles>
- <dependencies>
- <dependency>
- <groupId>org.apache.metron</groupId>
- <artifactId>metron-platform</artifactId>
- <version>${project.parent.version}</version>
- <type>pom</type>
- </dependency>
- <dependency>
- <groupId>org.apache.metron</groupId>
- <artifactId>metron-analytics</artifactId>
- <version>${project.parent.version}</version>
- <type>pom</type>
- </dependency>
- </dependencies>
</project>
[06/26] incubator-metron git commit: METRON-842 Add dynamic templates
for risk score fields (cestella via nickwallen) closes
apache/incubator-metron#523
Posted by ma...@apache.org.
METRON-842 Add dynamic templates for risk score fields (cestella via nickwallen) closes apache/incubator-metron#523
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/2ecabaa8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/2ecabaa8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/2ecabaa8
Branch: refs/heads/Metron_0.4.0
Commit: 2ecabaa8c0e7dab929726f8da48b98de623953ab
Parents: 58d5318
Author: cestella <ce...@gmail.com>
Authored: Wed Apr 19 08:55:02 2017 -0400
Committer: nickallen <ni...@apache.org>
Committed: Wed Apr 19 08:55:02 2017 -0400
----------------------------------------------------------------------
.../CURRENT/package/files/bro_index.template | 181 +++++++++++--------
.../CURRENT/package/files/snort_index.template | 171 ++++++++++--------
.../CURRENT/package/files/yaf_index.template | 181 +++++++++++--------
.../triage/ThreatTriageProcessor.java | 1 -
4 files changed, 307 insertions(+), 227 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/2ecabaa8/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/bro_index.template
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/bro_index.template b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/bro_index.template
index 951d967..50eae30 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/bro_index.template
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/bro_index.template
@@ -6,93 +6,120 @@
"enabled": true
},
"dynamic_templates": [
- {
- "geo_location_point": {
- "match": "enrichments:geo:*:location_point",
- "match_mapping_type": "*",
- "mapping": {
- "type": "geo_point"
- }
+ {
+ "geo_location_point": {
+ "match": "enrichments:geo:*:location_point",
+ "match_mapping_type": "*",
+ "mapping": {
+ "type": "geo_point"
}
- },
- {
- "geo_country": {
- "match": "enrichments:geo:*:country",
- "match_mapping_type": "*",
- "mapping": {
- "type": "string",
- "index": "not_analyzed"
- }
+ }
+ },
+ {
+ "geo_country": {
+ "match": "enrichments:geo:*:country",
+ "match_mapping_type": "*",
+ "mapping": {
+ "type": "string",
+ "index": "not_analyzed"
}
- },
- {
- "geo_city": {
- "match": "enrichments:geo:*:city",
- "match_mapping_type": "*",
- "mapping": {
- "type": "string",
- "index": "not_analyzed"
- }
+ }
+ },
+ {
+ "geo_city": {
+ "match": "enrichments:geo:*:city",
+ "match_mapping_type": "*",
+ "mapping": {
+ "type": "string",
+ "index": "not_analyzed"
}
- },
- {
- "geo_location_id": {
- "match": "enrichments:geo:*:locID",
- "match_mapping_type": "*",
- "mapping": {
- "type": "string",
- "index": "not_analyzed"
- }
+ }
+ },
+ {
+ "geo_location_id": {
+ "match": "enrichments:geo:*:locID",
+ "match_mapping_type": "*",
+ "mapping": {
+ "type": "string",
+ "index": "not_analyzed"
}
- },
- {
- "geo_dma_code": {
- "match": "enrichments:geo:*:dmaCode",
- "match_mapping_type": "*",
- "mapping": {
- "type": "string",
- "index": "not_analyzed"
- }
+ }
+ },
+ {
+ "geo_dma_code": {
+ "match": "enrichments:geo:*:dmaCode",
+ "match_mapping_type": "*",
+ "mapping": {
+ "type": "string",
+ "index": "not_analyzed"
}
- },
- {
- "geo_postal_code": {
- "match": "enrichments:geo:*:postalCode",
- "match_mapping_type": "*",
- "mapping": {
- "type": "string",
- "index": "not_analyzed"
- }
+ }
+ },
+ {
+ "geo_postal_code": {
+ "match": "enrichments:geo:*:postalCode",
+ "match_mapping_type": "*",
+ "mapping": {
+ "type": "string",
+ "index": "not_analyzed"
}
- },
- {
- "geo_latitude": {
- "match": "enrichments:geo:*:latitude",
- "match_mapping_type": "*",
- "mapping": {
- "type": "float"
- }
+ }
+ },
+ {
+ "geo_latitude": {
+ "match": "enrichments:geo:*:latitude",
+ "match_mapping_type": "*",
+ "mapping": {
+ "type": "float"
}
- },
- {
- "geo_longitude": {
- "match": "enrichments:geo:*:longitude",
- "match_mapping_type": "*",
- "mapping": {
- "type": "float"
- }
+ }
+ },
+ {
+ "geo_longitude": {
+ "match": "enrichments:geo:*:longitude",
+ "match_mapping_type": "*",
+ "mapping": {
+ "type": "float"
}
- },
- {
- "timestamps": {
- "match": "*:ts",
- "match_mapping_type": "*",
- "mapping": {
- "type": "date",
- "format": "epoch_millis"
- }
+ }
+ },
+ {
+ "timestamps": {
+ "match": "*:ts",
+ "match_mapping_type": "*",
+ "mapping": {
+ "type": "date",
+ "format": "epoch_millis"
}
}
+ },
+ {
+ "threat_triage_score": {
+ "mapping": {
+ "type": "float"
+ },
+ "match": "threat.triage.rules:*:score",
+ "match_mapping_type": "*"
+ }
+ },
+ {
+ "threat_triage_reason": {
+ "mapping": {
+ "type": "string"
+ },
+ "match": "threat.triage.rules:*:reason",
+ "match_mapping_type": "*"
+ }
+ },
+ {
+ "threat_triage_name": {
+ "mapping": {
+ "type": "string"
+ },
+ "match": "threat.triage.rules:*:name",
+ "match_mapping_type": "*"
+ }
+ }
],
"properties": {
"timestamp": {
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/2ecabaa8/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/snort_index.template
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/snort_index.template b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/snort_index.template
index bf943df..2311cf2 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/snort_index.template
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/snort_index.template
@@ -6,91 +6,118 @@
"enabled": true
},
"dynamic_templates": [
- {
- "geo_location_point": {
- "match": "enrichments:geo:*:location_point",
- "match_mapping_type": "*",
- "mapping": {
- "type": "geo_point"
- }
+ {
+ "geo_location_point": {
+ "match": "enrichments:geo:*:location_point",
+ "match_mapping_type": "*",
+ "mapping": {
+ "type": "geo_point"
}
- },
- {
- "geo_country": {
- "match": "enrichments:geo:*:country",
- "match_mapping_type": "*",
- "mapping": {
- "type": "string",
- "index": "not_analyzed"
- }
+ }
+ },
+ {
+ "geo_country": {
+ "match": "enrichments:geo:*:country",
+ "match_mapping_type": "*",
+ "mapping": {
+ "type": "string",
+ "index": "not_analyzed"
}
- },
- {
- "geo_city": {
- "match": "enrichments:geo:*:city",
- "match_mapping_type": "*",
- "mapping": {
- "type": "string",
- "index": "not_analyzed"
- }
+ }
+ },
+ {
+ "geo_city": {
+ "match": "enrichments:geo:*:city",
+ "match_mapping_type": "*",
+ "mapping": {
+ "type": "string",
+ "index": "not_analyzed"
}
- },
- {
- "geo_location_id": {
- "match": "enrichments:geo:*:locID",
- "match_mapping_type": "*",
- "mapping": {
- "type": "string",
- "index": "not_analyzed"
- }
+ }
+ },
+ {
+ "geo_location_id": {
+ "match": "enrichments:geo:*:locID",
+ "match_mapping_type": "*",
+ "mapping": {
+ "type": "string",
+ "index": "not_analyzed"
}
- },
- {
- "geo_dma_code": {
- "match": "enrichments:geo:*:dmaCode",
- "match_mapping_type": "*",
- "mapping": {
- "type": "string",
- "index": "not_analyzed"
- }
+ }
+ },
+ {
+ "geo_dma_code": {
+ "match": "enrichments:geo:*:dmaCode",
+ "match_mapping_type": "*",
+ "mapping": {
+ "type": "string",
+ "index": "not_analyzed"
}
- },
- {
- "geo_postal_code": {
- "match": "enrichments:geo:*:postalCode",
- "match_mapping_type": "*",
- "mapping": {
- "type": "string",
- "index": "not_analyzed"
- }
+ }
+ },
+ {
+ "geo_postal_code": {
+ "match": "enrichments:geo:*:postalCode",
+ "match_mapping_type": "*",
+ "mapping": {
+ "type": "string",
+ "index": "not_analyzed"
}
- },
- {
- "geo_latitude": {
- "match": "enrichments:geo:*:latitude",
- "match_mapping_type": "*",
- "mapping": {
- "type": "float"
- }
+ }
+ },
+ {
+ "geo_latitude": {
+ "match": "enrichments:geo:*:latitude",
+ "match_mapping_type": "*",
+ "mapping": {
+ "type": "float"
}
- },
+ }
+ },
+ {
+ "geo_longitude": {
+ "match": "enrichments:geo:*:longitude",
+ "match_mapping_type": "*",
+ "mapping": {
+ "type": "float"
+ }
+ }
+ },
+ {
+ "timestamps": {
+ "match": "*:ts",
+ "match_mapping_type": "*",
+ "mapping": {
+ "type": "date",
+ "format": "epoch_millis"
+ }
+ }
+ },
+ {
+ "threat_triage_score": {
+ "mapping": {
+ "type": "float"
+ },
+ "match": "threat.triage.rules:*:score",
+ "match_mapping_type": "*"
+ }
+ },
{
- "geo_longitude": {
- "match": "enrichments:geo:*:longitude",
- "match_mapping_type": "*",
+ "threat_triage_reason": {
"mapping": {
- "type": "float"
- }
+ "type": "string"
+ },
+ "match": "threat.triage.rules:*:reason",
+ "match_mapping_type": "*"
}
},
{
- "timestamps": {
- "match": "*:ts",
- "match_mapping_type": "*",
+ "threat_triage_name": {
"mapping": {
- "type": "date",
- "format": "epoch_millis"
- }
+ "type": "string"
+ },
+ "match": "threat.triage.rules:*:name",
+ "match_mapping_type": "*"
}
}
],
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/2ecabaa8/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/yaf_index.template
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/yaf_index.template b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/yaf_index.template
index 7743afc..bd90929 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/yaf_index.template
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/yaf_index.template
@@ -6,93 +6,120 @@
"enabled": true
},
"dynamic_templates": [
- {
- "geo_location_point": {
- "match": "enrichments:geo:*:location_point",
- "match_mapping_type": "*",
- "mapping": {
- "type": "geo_point"
- }
+ {
+ "geo_location_point": {
+ "match": "enrichments:geo:*:location_point",
+ "match_mapping_type": "*",
+ "mapping": {
+ "type": "geo_point"
}
- },
- {
- "geo_country": {
- "match": "enrichments:geo:*:country",
- "match_mapping_type": "*",
- "mapping": {
- "type": "string",
- "index": "not_analyzed"
- }
+ }
+ },
+ {
+ "geo_country": {
+ "match": "enrichments:geo:*:country",
+ "match_mapping_type": "*",
+ "mapping": {
+ "type": "string",
+ "index": "not_analyzed"
}
- },
- {
- "geo_city": {
- "match": "enrichments:geo:*:city",
- "match_mapping_type": "*",
- "mapping": {
- "type": "string",
- "index": "not_analyzed"
- }
+ }
+ },
+ {
+ "geo_city": {
+ "match": "enrichments:geo:*:city",
+ "match_mapping_type": "*",
+ "mapping": {
+ "type": "string",
+ "index": "not_analyzed"
}
- },
- {
- "geo_location_id": {
- "match": "enrichments:geo:*:locID",
- "match_mapping_type": "*",
- "mapping": {
- "type": "string",
- "index": "not_analyzed"
- }
+ }
+ },
+ {
+ "geo_location_id": {
+ "match": "enrichments:geo:*:locID",
+ "match_mapping_type": "*",
+ "mapping": {
+ "type": "string",
+ "index": "not_analyzed"
}
- },
- {
- "geo_dma_code": {
- "match": "enrichments:geo:*:dmaCode",
- "match_mapping_type": "*",
- "mapping": {
- "type": "string",
- "index": "not_analyzed"
- }
+ }
+ },
+ {
+ "geo_dma_code": {
+ "match": "enrichments:geo:*:dmaCode",
+ "match_mapping_type": "*",
+ "mapping": {
+ "type": "string",
+ "index": "not_analyzed"
}
- },
- {
- "geo_postal_code": {
- "match": "enrichments:geo:*:postalCode",
- "match_mapping_type": "*",
- "mapping": {
- "type": "string",
- "index": "not_analyzed"
- }
+ }
+ },
+ {
+ "geo_postal_code": {
+ "match": "enrichments:geo:*:postalCode",
+ "match_mapping_type": "*",
+ "mapping": {
+ "type": "string",
+ "index": "not_analyzed"
}
- },
- {
- "geo_latitude": {
- "match": "enrichments:geo:*:latitude",
- "match_mapping_type": "*",
- "mapping": {
- "type": "float"
- }
+ }
+ },
+ {
+ "geo_latitude": {
+ "match": "enrichments:geo:*:latitude",
+ "match_mapping_type": "*",
+ "mapping": {
+ "type": "float"
}
- },
- {
- "geo_longitude": {
- "match": "enrichments:geo:*:longitude",
- "match_mapping_type": "*",
- "mapping": {
- "type": "float"
- }
+ }
+ },
+ {
+ "geo_longitude": {
+ "match": "enrichments:geo:*:longitude",
+ "match_mapping_type": "*",
+ "mapping": {
+ "type": "float"
}
- },
- {
- "timestamps": {
- "match": "*:ts",
- "match_mapping_type": "*",
- "mapping": {
- "type": "date",
- "format": "epoch_millis"
- }
+ }
+ },
+ {
+ "timestamps": {
+ "match": "*:ts",
+ "match_mapping_type": "*",
+ "mapping": {
+ "type": "date",
+ "format": "epoch_millis"
}
}
+ },
+ {
+ "threat_triage_score": {
+ "mapping": {
+ "type": "float"
+ },
+ "match": "threat.triage.rules:*:score",
+ "match_mapping_type": "*"
+ }
+ },
+ {
+ "threat_triage_reason": {
+ "mapping": {
+ "type": "string"
+ },
+ "match": "threat.triage.rules:*:reason",
+ "match_mapping_type": "*"
+ }
+ },
+ {
+ "threat_triage_name": {
+ "mapping": {
+ "type": "string"
+ },
+ "match": "threat.triage.rules:*:name",
+ "match_mapping_type": "*"
+ }
+ }
],
"properties": {
"timestamp": {
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/2ecabaa8/metron-platform/metron-enrichment/src/main/java/org/apache/metron/threatintel/triage/ThreatTriageProcessor.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/threatintel/triage/ThreatTriageProcessor.java b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/threatintel/triage/ThreatTriageProcessor.java
index 4d22081..8b09e85 100644
--- a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/threatintel/triage/ThreatTriageProcessor.java
+++ b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/threatintel/triage/ThreatTriageProcessor.java
@@ -96,7 +96,6 @@ public class ThreatTriageProcessor implements Function<Map, ThreatScore> {
Aggregators aggregators = threatTriageConfig.getAggregator();
List<Number> allScores = threatScore.getRuleScores().stream().map(score -> score.getRule().getScore()).collect(Collectors.toList());
Double aggregateScore = aggregators.aggregate(allScores, threatTriageConfig.getAggregationConfig());
-
// return the overall threat score
threatScore.setScore(aggregateScore);
return threatScore;
[05/26] incubator-metron git commit: METRON-862 Add Mattf gpg public
key to KEYS file (mattf-horton) closes apache/incubator-metron#536
Posted by ma...@apache.org.
METRON-862 Add Mattf gpg public key to KEYS file (mattf-horton) closes apache/incubator-metron#536
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/58d5318a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/58d5318a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/58d5318a
Branch: refs/heads/Metron_0.4.0
Commit: 58d5318a0a9a40ea33c63d83885c11115eacd32c
Parents: 2ecc55b
Author: mattf-horton <mf...@hortonworks.com>
Authored: Tue Apr 18 22:57:04 2017 -0700
Committer: mattf <ma...@apache.org>
Committed: Tue Apr 18 22:57:04 2017 -0700
----------------------------------------------------------------------
KEYS | 59 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 59 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/58d5318a/KEYS
----------------------------------------------------------------------
diff --git a/KEYS b/KEYS
index 8381e96..e56a6fa 100644
--- a/KEYS
+++ b/KEYS
@@ -925,3 +925,62 @@ GttU/Cj6RskiqDovmrDO8JFTOkOP1PJmWRPFq9knsKZY1bW6+AW3Ti12ZYTy6NDf
A+GKk3KmvkTtI9roji8+rMEEXg==
=xP63
-----END PGP PUBLIC KEY BLOCK-----
+
+pub rsa4096/4169AA27ECB31663 2011-07-31 [SCEA]
+ Key fingerprint = 7854 36A7 8258 6B71 829C 67A0 4169 AA27 ECB3 1663
+uid Matthew Foley (CODE SIGNING KEY) <ma...@apache.org>
+sig 0DFF492D8EE2F25C 2012-10-10 Mahadev Konar (CODE SIGNING KEY) <ma...@apache.org>
+sig 1209E7F13D0C92B9 2012-10-09 Owen O'Malley (Code signing) <om...@apache.org>
+sig 3 4169AA27ECB31663 2011-07-31 Matthew Foley (CODE SIGNING KEY) <ma...@apache.org>
+-----BEGIN PGP PUBLIC KEY BLOCK-----
+Comment: GPGTools - http://gpgtools.org
+
+mQINBE414+MBEAC3vlzZlQysSp5m9fuVgQ/b9VPPh/B0gwJ+KRSP9UHGBjYWS3cu
+mLW4BBC4VmwbU8KvssD6FUJ2o8P2kkRF5Cid6MAANkyndn/h59p1DG/a6KgbCFi3
+MG2RQVhX0PUHcqOhAceeK13SgAQ4omLHKyKKq0k+80wg7mh43OWxwyrFWCl5o74Z
+9yfhytWaJZroUIVbhe3hirGspS/eUoeFbJ3IyXqUvGorQWq/cZX2Cu5nDAYyZOxD
+iuQaTnlS7trfes9CvSoq4GRDotgf7l+ELhhBz9VsFBn+xOu0Xaq8GmLBUSM/6cZJ
+YUhwUxAHF9Z42VZXwTnSeLNX7gCGw3keA9afbG6Osm8GtNT7nIqlX4QVCnogC6HK
+Im4H2tVfxbi6T96p/Km42VMCajBvVpvRPKqqLRr85qpCQzbM7/UUd8BO8ZAMMjbX
+Q9yRJxAFDrWc/uIgeic8EYcPYrF0mBQMDL1JCAujjzD1sDdcT4fdih1nzPsgWwQN
+9cmSPdgw83ANgewulZGDOOQ5we8NEVuBYezmfA2uQE7r2ztm+Xq3cyR7uMS1tFfl
+e4lL/Cf6HDYiVNyPLF32yagytmFm7Rv33vRXKDR51DFXMVV3r4PUf3kAnH/tCRkA
+juWSSl6l5/WsilN8/2uVjaa1lO0FcccDSHb/zAhKIpf3Orkz1IbTtVC/vQARAQAB
+tDNNYXR0aGV3IEZvbGV5IChDT0RFIFNJR05JTkcgS0VZKSA8bWF0dGZAYXBhY2hl
+Lm9yZz6JAhwEEAECAAYFAlB1IHEACgkQDf9JLY7i8lxhOA/9E66dNXHbZRK/1D4z
+GgpjR0PhzWEdRLfu/BO6V3oB7BTISmdBT7QDGp/4nRBZiiJwLWWbIRyNyrxJ1hiW
+VJSE1GxF+5lDD+oKkDqwVanKHzDvWb54CHvZKjY1xhenYrgc5sHJQ3EnrWoqBBUP
+awhB/QohEQ5K5MT/4VCqR7UJuckE6Ny2rndCwid+jBY6RUpOlSJupsAOpqvHdqui
+pa/oHddadhKBrCgLlB6hFlRmHBn3bJqigPMIs3JVQ34ufBVR0guHdRV6uFoZsgce
+GAgJDLnPlHbzaatGprpV4UpwO8uIWdo4K0pcVYu1opFDkekwHF254WaH3jYo4KQ4
+hGvt4x5YIogr89rf9XPRCog/VYg4xoX+cfoAkhpmAVKPAA7aOOJoxOMjGZt8pCEI
+8UjwRv9QR3w7oyS8QDy8b3jisKmUaHEU/h5bWXOQj4kjMk+zavTNtgSytV25Cy+b
+kBOJFSKBKB5JWFUGvhg27IPnyACA9DlHmoMyutIsyP5EMHLCXSkOMtcPoS18lyf3
+iyutdKCXOuzDxBHJU2zqitcrn0IQEquCBZ1oyGK4cS6j0tzZ7Pl7es1y4ddXw8Ae
+GDf/IwfpeR/5ybYlDX2W141CUeQuRrpXbQAgCPjjypQROECBXnMG+Cb7uM1/qBl6
+/60s51h3ZDKaW/RYQ3Tvyv0d4TGJAhwEEAEKAAYFAlB0jZEACgkQEgnn8T0MkrlQ
+sw//Yzy56eX0DoIbuqRMEHD0ZK57ioVUFJbBmhwn99M3aBhHg0Uh0w+JRjPC/u3A
+1Rfy98veTNAfjVqdYbjk3Qx87/vaF1ykQ0/3SO6s4CDPNFGc/ZpeJUc71Z1aR3Vh
+iD6MBTGt2q7L4iyeBfzypvHhpjMzUQWAAqaUPrPXi0TSOqv509FHIkR223IEGMct
+FsL79vmtZVqEZHfH926q2xGaMmmxhJL6JExRjuWnr1omIA2rbl2d5fGe9eZwLr54
+KipmP0Z110YPQ8xW2zUtyPCe+zjNWEOIArOqYUoItcLgQEIYomn6fyCPX3yXW8zU
+TZso4NLX4d5yo14FLigAyMOpElRQ7rauKkr6KRMjhxefJiE2n+L1q2wE8mg1YTsv
+RT7SoLvR63sgUHhij0Ub2dmJIqSXzMceZqFnFLF1xn3UqH0+cp6KsuHbLyk6Qc3I
+grmqT1eq/WSixT0X0O1Gym7XAkQIHopOn30ezGt/Tm3wqgqM8PWISwBz0paIP3b+
+idXQwWlEimYLCpqYDNO3ioLroIOJqwN1EG6ZlfNNae1Z80Nq3fV7YSkhAPEO4WRt
+LnQYJFug+yXnJoaIUXzA/bqxXE/kza5ERkDiOUxRurH63jDjfz7/L9FfJzx+xuBk
+gJPb9a8iG5aCPds4L0ewCVg28Us1NN5tqCAeiaOKlyAT0QGJAjcEEwEKACEFAk41
+4+MCGy8FCwkIBwMFFQoJCAsFFgIDAQACHgECF4AACgkQQWmqJ+yzFmNezA//eB1I
+WH8ozD6BL6OuyXGQ9xgRJExZo+bgfNceZeiLumHLN6HB4R2lk1f22bBB2uVmKk1K
+5DKTzCIncZJeZrnaVyKkKw83dRPlIOLt14Hv6+v8+FzM/DRphzMT2Q31EgkIswxz
+6Ro6mvHA8iqfgO/HAXZRlz+OyipHdEIIzCtTWAiNlHuUx+JYu4nTIYFL4+/Js44g
+3vqJMNezOIhwvOJV9Jk5GL4A8ATkseEZ6Xq6Ihcf7RHS+40xaquVgvNuspr7Rb8N
+IMl0XaQMkJYoHOKUUUTfmGYIgFEYdgrGeh0C2m0Xdiz46MdhZJnBCu/92rMSs3uT
+BS/fo1KYDv7OLbCG3asXVsFgYIdFLeJ0qN1uXTq2kfpCq9hEqbAof08vgXHWGrhH
+MC17F4I9HrjTXUWEEDzQ8wK3xL4p884p/J4TqX0uI6RpI90E3medr5H1yIqO7ka3
+vkd/cbkuZuCthrOJq3gOfaRXS46EZ4rLAvoQKUiw1ZznjjYzI6ErJc1iD10B4/CD
+1i2J74lmGHjGgyDDIUQSoce2EED7n09uU8lUSZRCZQVyroAd5DSAdUmjIlVmWB7H
+3xKMq29kVahbafB4PyOQEmCxdQbss5bcUsXNmei2xdzx2XnBOxvfwcztJ2DCcRTf
+7wQCgib6Orvh57hVFTFWa116lpkU+rV3RUFC8RI=
+=MZyO
+-----END PGP PUBLIC KEY BLOCK-----
[03/26] incubator-metron git commit: METRON-856 Ansible rpm build
wipes out prior binary build (merrimanr) closes apache/incubator-metron#533
Posted by ma...@apache.org.
METRON-856 Ansible rpm build wipes out prior binary build (merrimanr) closes apache/incubator-metron#533
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/84f336b3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/84f336b3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/84f336b3
Branch: refs/heads/Metron_0.4.0
Commit: 84f336b372be271abb563b1a19f2af462fe6de2d
Parents: eb08d14
Author: merrimanr <me...@gmail.com>
Authored: Tue Apr 18 15:33:27 2017 -0500
Committer: merrimanr <me...@gmail.com>
Committed: Tue Apr 18 15:33:27 2017 -0500
----------------------------------------------------------------------
metron-deployment/roles/metron-builder/tasks/main.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/84f336b3/metron-deployment/roles/metron-builder/tasks/main.yml
----------------------------------------------------------------------
diff --git a/metron-deployment/roles/metron-builder/tasks/main.yml b/metron-deployment/roles/metron-builder/tasks/main.yml
index eb8717a..d5fb517 100644
--- a/metron-deployment/roles/metron-builder/tasks/main.yml
+++ b/metron-deployment/roles/metron-builder/tasks/main.yml
@@ -21,6 +21,6 @@
run_once: true
- name: Build Metron RPMs
- local_action: shell cd {{ metron_build_dir }} && mvn clean package -DskipTests -P build-rpms
+ local_action: shell cd {{ metron_build_dir }}/metron-deployment && mvn clean package -DskipTests -P build-rpms
become: false
run_once: true
[26/26] incubator-metron git commit: METRON-861: Allow JVM args to be
passed to CLI utilities closes apache/incubator-metron#534
Posted by ma...@apache.org.
METRON-861: Allow JVM args to be passed to CLI utilities closes apache/incubator-metron#534
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/68bd6c52
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/68bd6c52
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/68bd6c52
Branch: refs/heads/Metron_0.4.0
Commit: 68bd6c52055b34460ff1823a31daae1f60985e1a
Parents: bf2528f
Author: cstella <ce...@gmail.com>
Authored: Thu Apr 27 12:23:52 2017 -0400
Committer: cstella <ce...@gmail.com>
Committed: Thu Apr 27 12:23:52 2017 -0400
----------------------------------------------------------------------
.../metron-maas-service/src/main/scripts/maas_deploy.sh | 1 +
.../metron-maas-service/src/main/scripts/maas_service.sh | 1 +
metron-platform/metron-common/src/main/scripts/stellar | 2 +-
.../metron-common/src/main/scripts/zk_load_configs.sh | 2 +-
.../metron-data-management/src/main/scripts/flatfile_loader.sh | 4 ++--
.../src/main/scripts/geo_enrichment_load.sh | 3 ++-
.../metron-enrichment/src/main/scripts/latency_summarizer.sh | 2 +-
.../metron-pcap-backend/src/main/scripts/pcap_inspector.sh | 2 +-
.../metron-pcap-backend/src/main/scripts/pcap_query.sh | 1 +
9 files changed, 11 insertions(+), 7 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68bd6c52/metron-analytics/metron-maas-service/src/main/scripts/maas_deploy.sh
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-maas-service/src/main/scripts/maas_deploy.sh b/metron-analytics/metron-maas-service/src/main/scripts/maas_deploy.sh
index 1d0a9f0..e9d123b 100755
--- a/metron-analytics/metron-maas-service/src/main/scripts/maas_deploy.sh
+++ b/metron-analytics/metron-maas-service/src/main/scripts/maas_deploy.sh
@@ -31,6 +31,7 @@ export HBASE_HOME=${HBASE_HOME:-/usr/hdp/current/hbase-client}
export METRON_VERSION=${project.version}
export METRON_HOME=/usr/metron/$METRON_VERSION
export DM_JAR=metron-maas-service-$METRON_VERSION-uber.jar
+export HADOOP_OPTS="$HADOOP_OPTS $METRON_JVMFLAGS"
CP=$METRON_HOME/lib/$DM_JAR
HADOOP_CLASSPATH=$(echo $CP )
for jar in $(echo $HADOOP_CLASSPATH | sed 's/:/ /g');do
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68bd6c52/metron-analytics/metron-maas-service/src/main/scripts/maas_service.sh
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-maas-service/src/main/scripts/maas_service.sh b/metron-analytics/metron-maas-service/src/main/scripts/maas_service.sh
index 75736c1..a2c8ac2 100755
--- a/metron-analytics/metron-maas-service/src/main/scripts/maas_service.sh
+++ b/metron-analytics/metron-maas-service/src/main/scripts/maas_service.sh
@@ -31,6 +31,7 @@ export HBASE_HOME=${HBASE_HOME:-/usr/hdp/current/hbase-client}
export METRON_VERSION=${project.version}
export METRON_HOME=/usr/metron/$METRON_VERSION
export DM_JAR=metron-maas-service-$METRON_VERSION-uber.jar
+export HADOOP_OPTS="$HADOOP_OPTS $METRON_JVMFLAGS"
CP=$METRON_HOME/lib/$DM_JAR
HADOOP_CLASSPATH=$(echo $CP )
for jar in $(echo $HADOOP_CLASSPATH | sed 's/:/ /g');do
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68bd6c52/metron-platform/metron-common/src/main/scripts/stellar
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/scripts/stellar b/metron-platform/metron-common/src/main/scripts/stellar
index 8cc48c9..2145350 100644
--- a/metron-platform/metron-common/src/main/scripts/stellar
+++ b/metron-platform/metron-common/src/main/scripts/stellar
@@ -31,4 +31,4 @@ fi
export HBASE_CONFIGS=/etc/hbase/conf
export METRON_VERSION=${project.version}
export METRON_HOME=/usr/metron/$METRON_VERSION
-java -cp "$HBASE_CONFIGS:$METRON_HOME/lib/*" org.apache.metron.common.stellar.shell.StellarShell "$@"
+java $JVMFLAGS -cp "$HBASE_CONFIGS:$METRON_HOME/lib/*" org.apache.metron.common.stellar.shell.StellarShell "$@"
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68bd6c52/metron-platform/metron-common/src/main/scripts/zk_load_configs.sh
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/scripts/zk_load_configs.sh b/metron-platform/metron-common/src/main/scripts/zk_load_configs.sh
index 39e5e3e..2380d13 100755
--- a/metron-platform/metron-common/src/main/scripts/zk_load_configs.sh
+++ b/metron-platform/metron-common/src/main/scripts/zk_load_configs.sh
@@ -33,5 +33,5 @@ export JAR=metron-parsers-$METRON_VERSION-uber.jar
export HBASE_HOME=${HBASE_HOME:-/usr/hdp/current/hbase-client}
CP=$METRON_HOME/lib/$JAR:${HBASE_HOME}/lib/hbase-server.jar:`${HBASE_HOME}/bin/hbase classpath`
-java -cp $CP $CLASSNAME "$@"
+java $METRON_JVMFLAGS -cp $CP $CLASSNAME "$@"
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68bd6c52/metron-platform/metron-data-management/src/main/scripts/flatfile_loader.sh
----------------------------------------------------------------------
diff --git a/metron-platform/metron-data-management/src/main/scripts/flatfile_loader.sh b/metron-platform/metron-data-management/src/main/scripts/flatfile_loader.sh
index b9e2746..f92d418 100755
--- a/metron-platform/metron-data-management/src/main/scripts/flatfile_loader.sh
+++ b/metron-platform/metron-data-management/src/main/scripts/flatfile_loader.sh
@@ -32,7 +32,7 @@ export METRON_HOME=/usr/metron/$METRON_VERSION
export CLASSNAME="org.apache.metron.dataloads.nonbulk.flatfile.SimpleEnrichmentFlatFileLoader"
export DM_JAR=${project.artifactId}-$METRON_VERSION.jar
export HBASE_HOME=${HBASE_HOME:-/usr/hdp/current/hbase-client}
-
+export HADOOP_OPTS="$HADOOP_OPTS $METRON_JVMFLAGS"
if [ $(which hadoop) ]
then
HADOOP_CLASSPATH=${HBASE_HOME}/lib/hbase-server.jar:`${HBASE_HOME}/bin/hbase classpath`
@@ -46,6 +46,6 @@ then
else
echo "Warning: Metron cannot find the hadoop client on this node. This means that loading via Map Reduce will NOT function."
CP=$METRON_HOME/lib/$DM_JAR:/usr/metron/${METRON_VERSION}/lib/taxii-1.1.0.1.jar:`${HBASE_HOME}/bin/hbase classpath`
- java -cp $CP $CLASSNAME "$@"
+ java $METRON_JVMFLAGS -cp $CP $CLASSNAME "$@"
fi
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68bd6c52/metron-platform/metron-data-management/src/main/scripts/geo_enrichment_load.sh
----------------------------------------------------------------------
diff --git a/metron-platform/metron-data-management/src/main/scripts/geo_enrichment_load.sh b/metron-platform/metron-data-management/src/main/scripts/geo_enrichment_load.sh
index 3b6ab37..a2d5448 100644
--- a/metron-platform/metron-data-management/src/main/scripts/geo_enrichment_load.sh
+++ b/metron-platform/metron-data-management/src/main/scripts/geo_enrichment_load.sh
@@ -38,4 +38,5 @@ export HADOOP_CLASSPATH
export METRON_VERSION=${project.version}
export METRON_HOME=/usr/metron/$METRON_VERSION
export DM_JAR=${project.artifactId}-$METRON_VERSION.jar
-hadoop jar $METRON_HOME/lib/$DM_JAR org.apache.metron.dataloads.nonbulk.geo.GeoEnrichmentLoader "$@"
\ No newline at end of file
+export HADOOP_OPTS="$HADOOP_OPTS $METRON_JVMFLAGS"
+hadoop jar $METRON_HOME/lib/$DM_JAR org.apache.metron.dataloads.nonbulk.geo.GeoEnrichmentLoader "$@"
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68bd6c52/metron-platform/metron-enrichment/src/main/scripts/latency_summarizer.sh
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/src/main/scripts/latency_summarizer.sh b/metron-platform/metron-enrichment/src/main/scripts/latency_summarizer.sh
index ea92c61..8d20a48 100755
--- a/metron-platform/metron-enrichment/src/main/scripts/latency_summarizer.sh
+++ b/metron-platform/metron-enrichment/src/main/scripts/latency_summarizer.sh
@@ -29,4 +29,4 @@ fi
export METRON_VERSION=${project.version}
export METRON_HOME=/usr/metron/$METRON_VERSION
export TOPOLOGIES_JAR=Metron-Topologies-$METRON_VERSION.jar
-java -cp $METRON_HOME/lib/$TOPOLOGIES_JAR org.apache.metron.enrichment.cli.LatencySummarizer "$@"
+java $METRON_JVMFLAGS -cp $METRON_HOME/lib/$TOPOLOGIES_JAR org.apache.metron.enrichment.cli.LatencySummarizer "$@"
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68bd6c52/metron-platform/metron-pcap-backend/src/main/scripts/pcap_inspector.sh
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/main/scripts/pcap_inspector.sh b/metron-platform/metron-pcap-backend/src/main/scripts/pcap_inspector.sh
index a5a24ae..438152c 100755
--- a/metron-platform/metron-pcap-backend/src/main/scripts/pcap_inspector.sh
+++ b/metron-platform/metron-pcap-backend/src/main/scripts/pcap_inspector.sh
@@ -30,5 +30,5 @@ fi
export METRON_VERSION=${project.version}
export METRON_HOME=/usr/metron/$METRON_VERSION
export TOPOLOGIES_JAR=${project.artifactId}-$METRON_VERSION.jar
-
+export HADOOP_OPTS="$HADOOP_OPTS $METRON_JVMFLAGS"
yarn jar $METRON_HOME/lib/$TOPOLOGIES_JAR org.apache.metron.utils.PcapInspector "$@"
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68bd6c52/metron-platform/metron-pcap-backend/src/main/scripts/pcap_query.sh
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/main/scripts/pcap_query.sh b/metron-platform/metron-pcap-backend/src/main/scripts/pcap_query.sh
index c09aa03..206e2b8 100755
--- a/metron-platform/metron-pcap-backend/src/main/scripts/pcap_query.sh
+++ b/metron-platform/metron-pcap-backend/src/main/scripts/pcap_query.sh
@@ -30,5 +30,6 @@ fi
export METRON_VERSION=${project.version}
export METRON_HOME=/usr/metron/$METRON_VERSION
export PCAP_BACKEND_JAR=${project.artifactId}-$METRON_VERSION.jar
+export HADOOP_OPTS="$HADOOP_OPTS $METRON_JVMFLAGS"
yarn jar $METRON_HOME/lib/$PCAP_BACKEND_JAR org.apache.metron.pcap.query.PcapCli "$@"
[16/26] incubator-metron git commit: METRON-857 Metron should one
unified docker build image (ottobackwards) closes apache/incubator-metron#543
Posted by ma...@apache.org.
METRON-857 Metron should one unified docker build image (ottobackwards) closes apache/incubator-metron#543
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/8c264f8e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/8c264f8e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/8c264f8e
Branch: refs/heads/Metron_0.4.0
Commit: 8c264f8e7345620c2a6c155d8d32e396ce30ec91
Parents: b6786aa
Author: ottobackwards <ot...@gmail.com>
Authored: Mon Apr 24 17:40:47 2017 -0400
Committer: otto <ot...@apache.org>
Committed: Mon Apr 24 17:40:47 2017 -0400
----------------------------------------------------------------------
.../packaging/docker/ansible-docker/Dockerfile | 15 +-
.../packaging/docker/rpm-docker/pom.xml | 137 +++++++++++++------
2 files changed, 101 insertions(+), 51 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/8c264f8e/metron-deployment/packaging/docker/ansible-docker/Dockerfile
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/docker/ansible-docker/Dockerfile b/metron-deployment/packaging/docker/ansible-docker/Dockerfile
index 1037bca..825e8a7 100644
--- a/metron-deployment/packaging/docker/ansible-docker/Dockerfile
+++ b/metron-deployment/packaging/docker/ansible-docker/Dockerfile
@@ -14,17 +14,17 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-FROM centos:6.6
+FROM centos:centos6
MAINTAINER Apache Metron
RUN yum install -y tar
RUN yum install -y wget
RUN yum groupinstall -y "Development tools"
RUN yum install -y zlib-dev openssl-devel sqlite-devel bzip2-devel libffi-devel
-RUN wget https://www.python.org/ftp/python/2.7.10/Python-2.7.10.tgz -O /usr/src/Python-2.7.10.tgz
+RUN wget https://www.python.org/ftp/python/2.7.11/Python-2.7.11.tgz -O /usr/src/Python-2.7.11.tgz
WORKDIR /usr/src
-RUN tar xvf Python-2.7.10.tgz
-WORKDIR /usr/src/Python-2.7.10
+RUN tar xvf Python-2.7.11.tgz
+WORKDIR /usr/src/Python-2.7.11
RUN ./configure
RUN make altinstall
WORKDIR /usr/src
@@ -41,9 +41,10 @@ RUN yum install -y java-1.8.0-openjdk java-1.8.0-openjdk-devel
RUN yum install -y which
RUN yum install -y nss
WORKDIR /usr/src
-RUN wget http://apache.cs.utah.edu/maven/maven-3/3.2.5/binaries/apache-maven-3.2.5-bin.tar.gz
-RUN tar xzvf apache-maven-3.2.5-bin.tar.gz
-RUN mv apache-maven-3.2.5 /opt/maven
+RUN wget http://apache.cs.utah.edu/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz
+RUN tar xzvf apache-maven-3.3.9-bin.tar.gz
+RUN mv apache-maven-3.3.9 /opt/maven
RUN ln -s /opt/maven/bin/mvn /usr/bin/mvn
+RUN yum -y install asciidoc rpm-build rpm2cpio tar unzip xmlto zip rpmlint && yum clean all
WORKDIR /root
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/8c264f8e/metron-deployment/packaging/docker/rpm-docker/pom.xml
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/docker/rpm-docker/pom.xml b/metron-deployment/packaging/docker/rpm-docker/pom.xml
index aaede8d..dd1a995 100644
--- a/metron-deployment/packaging/docker/rpm-docker/pom.xml
+++ b/metron-deployment/packaging/docker/rpm-docker/pom.xml
@@ -89,50 +89,6 @@
</configuration>
</plugin>
<plugin>
- <groupId>org.codehaus.mojo</groupId>
- <artifactId>exec-maven-plugin</artifactId>
- <version>1.5.0</version>
- <executions>
- <execution>
- <id>docker-build</id>
- <phase>package</phase>
- <goals>
- <goal>exec</goal>
- </goals>
- <configuration>
- <executable>docker</executable>
- <arguments>
- <argument>build</argument>
- <argument>-f</argument>
- <argument>Dockerfile</argument>
- <argument>-t</argument>
- <argument>${rpm.docker.tag}</argument>
- <argument>.</argument>
- </arguments>
- </configuration>
- </execution>
- <execution>
- <id>rpm-build</id>
- <phase>package</phase>
- <goals>
- <goal>exec</goal>
- </goals>
- <configuration>
- <executable>docker</executable>
- <arguments>
- <argument>run</argument>
- <argument>-v</argument>
- <argument>${project.basedir}:/root</argument>
- <argument>${rpm.docker.tag}:latest</argument>
- <argument>/bin/bash</argument>
- <argument>-c</argument>
- <argument>./build.sh ${project.version}</argument>
- </arguments>
- </configuration>
- </execution>
- </executions>
- </plugin>
- <plugin>
<artifactId>maven-resources-plugin</artifactId>
<version>3.0.1</version>
<executions>
@@ -227,4 +183,97 @@
</plugin>
</plugins>
</build>
+ <profiles>
+ <profile>
+ <id>InDocker</id>
+ <activation>
+ <file>
+ <!-- If this file exists, then we are running in docker already -->
+ <exists>/.dockerenv</exists>
+ </file>
+ </activation>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>exec-maven-plugin</artifactId>
+ <version>1.5.0</version>
+ <executions>
+ <execution>
+ <id>rpm-build</id>
+ <phase>package</phase>
+ <goals>
+ <goal>exec</goal>
+ </goals>
+ <configuration>
+ <executable>/bin/bash</executable>
+ <arguments>
+ <argument>-c</argument>
+ <argument>./build.sh ${project.version}</argument>
+ </arguments>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ <profile>
+ <id>NeedsDocker</id>
+ <activation>
+ <file>
+ <!-- if this file doesn't exist, then we need to run docker -->
+ <missing>/.dockerenv</missing>
+ </file>
+ </activation>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>exec-maven-plugin</artifactId>
+ <version>1.5.0</version>
+ <executions>
+ <execution>
+ <id>docker-build</id>
+ <phase>package</phase>
+ <goals>
+ <goal>exec</goal>
+ </goals>
+ <configuration>
+ <executable>docker</executable>
+ <arguments>
+ <argument>build</argument>
+ <argument>-f</argument>
+ <argument>Dockerfile</argument>
+ <argument>-t</argument>
+ <argument>${rpm.docker.tag}</argument>
+ <argument>.</argument>
+ </arguments>
+ </configuration>
+ </execution>
+ <execution>
+ <id>rpm-build</id>
+ <phase>package</phase>
+ <goals>
+ <goal>exec</goal>
+ </goals>
+ <configuration>
+ <executable>docker</executable>
+ <arguments>
+ <argument>run</argument>
+ <argument>-v</argument>
+ <argument>${project.basedir}:/root</argument>
+ <argument>${rpm.docker.tag}:latest</argument>
+ <argument>/bin/bash</argument>
+ <argument>-c</argument>
+ <argument>./build.sh ${project.version}</argument>
+ </arguments>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ </profiles>
</project>
[13/26] incubator-metron git commit: METRON-866 Quickdev and Fulldev
readme misses Docker as prerequisite (nickwallen) closes
apache/incubator-metron#544
Posted by ma...@apache.org.
METRON-866 Quickdev and Fulldev readme misses Docker as prerequisite (nickwallen) closes apache/incubator-metron#544
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/8fe52c9a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/8fe52c9a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/8fe52c9a
Branch: refs/heads/Metron_0.4.0
Commit: 8fe52c9a126bbfbb82472f48e1f162f128e35694
Parents: cb44090
Author: nickwallen <ni...@nickallen.org>
Authored: Mon Apr 24 11:36:58 2017 -0400
Committer: nickallen <ni...@apache.org>
Committed: Mon Apr 24 11:36:58 2017 -0400
----------------------------------------------------------------------
.../vagrant/full-dev-platform/README.md | 38 ++++++++---------
.../vagrant/quick-dev-platform/README.md | 44 +++++++++++---------
2 files changed, 43 insertions(+), 39 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/8fe52c9a/metron-deployment/vagrant/full-dev-platform/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/full-dev-platform/README.md b/metron-deployment/vagrant/full-dev-platform/README.md
index 8bc78b3..177b905 100644
--- a/metron-deployment/vagrant/full-dev-platform/README.md
+++ b/metron-deployment/vagrant/full-dev-platform/README.md
@@ -13,7 +13,9 @@ Getting Started
The computer used to deploy Apache Metron will need to have the following components installed.
- [Ansible](https://github.com/ansible/ansible) (2.0.0.2 or 2.2.2.0)
+ - [Docker](https://www.docker.com/community-edition)
- [Vagrant](https://www.vagrantup.com) 1.8.1
+ - [Vagrant Hostmanager Plugin](https://github.com/devopsgroup-io/vagrant-hostmanager) `vagrant plugin install vagrant-hostmanager`
- [Virtualbox](https://virtualbox.org) 5.0.16
- Python 2.7.11
- Maven 3.3.9
@@ -24,35 +26,31 @@ Any platform that supports these tools is suitable, but the following instructio
1. Install Homebrew by following the instructions at [Homebrew](http://brew.sh/).
-2. Run the following command in a terminal to install all of the required tools.
+1. Run the following command in a terminal to install all of the required tools.
- ```
- brew cask install vagrant virtualbox java
- brew install maven git
- ```
+ ```
+ brew cask install vagrant virtualbox java docker
+ brew install maven git
+ ```
-3. Install Ansible by following the instructions [here](http://docs.ansible.com/ansible/intro_installation.html#latest-releases-via-pip).
+1. Install Ansible by following the instructions [here](http://docs.ansible.com/ansible/intro_installation.html#latest-releases-via-pip).
### Deploy Metron
-1. Install Vagrant Hostmanager.
+1. Ensure that the Docker service is running.
- ```
- vagrant plugin install vagrant-hostmanager
- ```
+1. Deploy Metron
-2. Deploy Metron
+ ```
+ cd metron-deployment/vagrant/full-dev-platform
+ vagrant up
+ ```
- ```
- cd metron-deployment/vagrant/full-dev-platform
- vagrant up
- ```
+ Should the process fail before completing the deployment, the following command will continue the deployment process without re-instantiating the host.
- Should the process fail before completing the deployment, the following command will continue the deployment process without re-instantiating the host.
-
- ```
- vagrant provision
- ```
+ ```
+ vagrant provision
+ ```
### Explore Metron
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/8fe52c9a/metron-deployment/vagrant/quick-dev-platform/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/quick-dev-platform/README.md b/metron-deployment/vagrant/quick-dev-platform/README.md
index 71960ef..ca1168b 100644
--- a/metron-deployment/vagrant/quick-dev-platform/README.md
+++ b/metron-deployment/vagrant/quick-dev-platform/README.md
@@ -15,38 +15,44 @@ Getting Started
As with the Full Development Platform (`metron-deployment/vagrant/full-dev-platform`), the computer used to deploy Apache Metron will need the following components installed.
- [Ansible](https://github.com/ansible/ansible) (2.0.0.2 or 2.2.2.0)
+ - [Docker](https://www.docker.com/community-edition)
- [Vagrant](https://www.vagrantup.com) 1.8.1
+ - [Vagrant Hostmanager Plugin](https://github.com/devopsgroup-io/vagrant-hostmanager) `vagrant plugin install vagrant-hostmanager`
- [Virtualbox](https://virtualbox.org) 5.0.16
- Python 2.7.11
- Maven 3.3.9
-### Deploy Metron
+#### macOS
+
+ Any platform that supports these tools is suitable, but the following instructions cover installation on macOS. The easiest means of installing these tools on a Mac is to use the excellent [Homebrew](http://brew.sh/) project.
+
+ 1. Install Homebrew by following the instructions at [Homebrew](http://brew.sh/).
-1. Build Metron
+ 1. Run the following command in a terminal to install all of the required tools.
- ```
- cd incubator-metron
- mvn clean package -DskipTests
- ```
+ ```
+ brew cask install vagrant virtualbox java docker
+ brew install maven git
+ ```
-2. Install Vagrant Hostmanager.
+ 1. Install Ansible by following the instructions [here](http://docs.ansible.com/ansible/intro_installation.html#latest-releases-via-pip).
+
+### Deploy Metron
- ```
- vagrant plugin install vagrant-hostmanager
- ```
+1. Ensure that the Docker service is running.
-3. Deploy Metron
+1. Deploy Metron
- ```
- cd metron-deployment/vagrant/quick-dev-platform
- vagrant up
- ```
+ ```
+ cd metron-deployment/vagrant/quick-dev-platform
+ vagrant up
+ ```
- Should the process fail before completing the deployment, the following command will continue the deployment process without re-instantiating the host.
+ Should the process fail before completing the deployment, the following command will continue the deployment process without re-instantiating the host.
- ```
- vagrant provision
- ```
+ ```
+ vagrant provision
+ ```
### Explore Metron
[15/26] incubator-metron git commit: METRON-864 typo in indexing
readme (ctramnitz via JonZeolla) closes apache/incubator-metron#537
Posted by ma...@apache.org.
METRON-864 typo in indexing readme (ctramnitz via JonZeolla) closes apache/incubator-metron#537
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/b6786aaf
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/b6786aaf
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/b6786aaf
Branch: refs/heads/Metron_0.4.0
Commit: b6786aaf030122e83b82ad63c828f68f3754d8ec
Parents: bec84db
Author: ctramnitz <ct...@users.noreply.github.com>
Authored: Mon Apr 24 16:43:47 2017 -0400
Committer: jonzeolla <jo...@apache.org>
Committed: Mon Apr 24 16:43:47 2017 -0400
----------------------------------------------------------------------
metron-platform/metron-indexing/README.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/b6786aaf/metron-platform/metron-indexing/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-indexing/README.md b/metron-platform/metron-indexing/README.md
index f243eef..a536d87 100644
--- a/metron-platform/metron-indexing/README.md
+++ b/metron-platform/metron-indexing/README.md
@@ -125,7 +125,7 @@ topology. As such, make sure that the number of partitions in
the kafka topic is sufficient to handle the throughput that you expect.
## Indexing Topology
-The enrichment topology as started by the `$METRON_HOME/bin/start_elasticsearch_topology.sh`
+The `indexing` topology as started by the `$METRON_HOME/bin/start_elasticsearch_topology.sh`
or `$METRON_HOME/bin/start_solr_topology.sh`
script uses a default of one executor per bolt. In a real production system, this should
be customized by modifying the flux file in
@@ -147,4 +147,4 @@ These files must be added to the metron.spec file and the RPMs rebuilt to be ava
The notebook files will be found on the server in `$METRON_HOME/config/zeppelin`
-The Ambari Management Pack has a custom action to load these templates, ZEPPELIN_DASHBOARD_INSTALL, that will import them into Zeppelin.
\ No newline at end of file
+The Ambari Management Pack has a custom action to load these templates, ZEPPELIN_DASHBOARD_INSTALL, that will import them into Zeppelin.
[21/26] incubator-metron git commit: METRON-892 platform_info.sh
should have the version of docker (ottobackwards) closes
apache/incubator-metron#551
Posted by ma...@apache.org.
METRON-892 platform_info.sh should have the version of docker (ottobackwards) closes apache/incubator-metron#551
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/58fa1f69
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/58fa1f69
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/58fa1f69
Branch: refs/heads/Metron_0.4.0
Commit: 58fa1f69cf14436b54414f660ff11c2c48c99c6f
Parents: 6e1e322
Author: ottobackwards <ot...@gmail.com>
Authored: Wed Apr 26 10:03:42 2017 -0400
Committer: otto <ot...@apache.org>
Committed: Wed Apr 26 10:03:42 2017 -0400
----------------------------------------------------------------------
metron-deployment/scripts/platform-info.sh | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/58fa1f69/metron-deployment/scripts/platform-info.sh
----------------------------------------------------------------------
diff --git a/metron-deployment/scripts/platform-info.sh b/metron-deployment/scripts/platform-info.sh
index 79ebcac..f84d822 100755
--- a/metron-deployment/scripts/platform-info.sh
+++ b/metron-deployment/scripts/platform-info.sh
@@ -59,6 +59,10 @@ python --version 2>&1
echo "--"
mvn --version
+# docker
+echo "--"
+docker --version
+
# operating system
echo "--"
uname -a
@@ -73,7 +77,7 @@ case "${OSTYPE}" in
cat /proc/cpuinfo | grep -i '^processor' | wc -l | awk '{print "Total Physical Processors: " $0}'
cat /proc/cpuinfo | grep -i cores | cut -d: -f2 | awk '{corecount+=$1} END {print "Total cores: " corecount}'
echo "Disk information:"
- df -h | grep "^/"
+ df -h | grep "^/"
;;
darwin*)
sysctl hw.memsize | awk '{print "Total System Memory = " $2/1048576 " MB"}'
@@ -82,10 +86,9 @@ case "${OSTYPE}" in
sysctl hw.physicalcpu | cut -d: -f2 | awk '{print "Total Physical Processors:" $0}'
sysctl machdep.cpu | grep 'machdep.cpu.core_count' | cut -d: -f2 | cut -d\@ -f2 | awk '{print "Total cores:" $0}'
echo "Disk information:"
- df -h | grep "^/"
+ df -h | grep "^/"
;;
*)
echo "Unable to detect system resources for ${OSTYPE}"
;;
esac
-
[20/26] incubator-metron git commit: METRON-873: Stellar string
literals do not support quote escaping this closes
apache/incubator-metron#542
Posted by ma...@apache.org.
METRON-873: Stellar string literals do not support quote escaping this closes apache/incubator-metron#542
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/6e1e322e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/6e1e322e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/6e1e322e
Branch: refs/heads/Metron_0.4.0
Commit: 6e1e322eff001882153900f840b8387bc2a19c40
Parents: cac696d
Author: cstella <ce...@gmail.com>
Authored: Wed Apr 26 09:44:40 2017 -0400
Committer: cstella <ce...@gmail.com>
Committed: Wed Apr 26 09:44:40 2017 -0400
----------------------------------------------------------------------
metron-platform/metron-common/README.md | 5 +
.../metron/common/stellar/generated/Stellar.g4 | 5 +-
.../metron/common/stellar/StellarCompiler.java | 5 +-
.../common/stellar/generated/StellarLexer.java | 316 ++++++++++---------
.../metron/common/stellar/StellarTest.java | 16 +
5 files changed, 186 insertions(+), 161 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/6e1e322e/metron-platform/metron-common/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/README.md b/metron-platform/metron-common/README.md
index de3a1b7..10800d5 100644
--- a/metron-platform/metron-common/README.md
+++ b/metron-platform/metron-common/README.md
@@ -20,6 +20,11 @@ created to do simple computation and transformation.
The query language supports the following:
* Referencing fields in the enriched JSON
+* String literals are quoted with either `'` or `"`.
+* String literals support escaping for `'`, `"`, `\t`, `\r`, `\n`, and backslash
+ * The literal `'\'foo\''` would represent `'foo'`
+ * The literal `"\"foo\""` would represent `"foo"`
+ * The literal `'foo \\ bar'` would represent `foo \ bar`
* Simple boolean operations: `and`, `not`, `or`
* Simple arithmetic operations: `*`, `/`, `+`, `-` on real numbers or integers
* Simple comparison operations `<`, `>`, `<=`, `>=`
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/6e1e322e/metron-platform/metron-common/src/main/antlr4/org/apache/metron/common/stellar/generated/Stellar.g4
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/antlr4/org/apache/metron/common/stellar/generated/Stellar.g4 b/metron-platform/metron-common/src/main/antlr4/org/apache/metron/common/stellar/generated/Stellar.g4
index 75f5761..8602fde 100644
--- a/metron-platform/metron-common/src/main/antlr4/org/apache/metron/common/stellar/generated/Stellar.g4
+++ b/metron-platform/metron-common/src/main/antlr4/org/apache/metron/common/stellar/generated/Stellar.g4
@@ -100,8 +100,8 @@ IDENTIFIER : IDENTIFIER_START
;
STRING_LITERAL :
- DOUBLE_QUOTE SCHAR* DOUBLE_QUOTE
- | SINGLE_QUOTE SCHAR* SINGLE_QUOTE
+ SINGLE_QUOTE (~('\'' | '\\') | '\\' ('\'' | '\\' | 'r' | 'n' | 't'))* SINGLE_QUOTE
+ |DOUBLE_QUOTE (~('"' | '\\') | '\\' ('"' | '\\' | 'r' | 'n' | 't'))* DOUBLE_QUOTE
;
// COMMENT and WS are stripped from the output token stream by sending
@@ -114,7 +114,6 @@ WS : [ \r\t\u000C\n]+ -> skip;
fragment ZERO: '0';
fragment FIRST_DIGIT: '1'..'9';
fragment DIGIT: '0'..'9';
-fragment SCHAR: ~['"\\\r\n];
fragment D: ('d'|'D');
fragment E: ('e'|'E');
fragment F: ('f'|'F');
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/6e1e322e/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/StellarCompiler.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/StellarCompiler.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/StellarCompiler.java
index 044b25a..b289f48 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/StellarCompiler.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/StellarCompiler.java
@@ -17,6 +17,7 @@
*/
package org.apache.metron.common.stellar;
+import org.apache.commons.lang3.StringEscapeUtils;
import org.apache.metron.common.dsl.Context;
import org.apache.metron.common.dsl.Token;
import org.apache.metron.common.dsl.VariableResolver;
@@ -263,7 +264,9 @@ public class StellarCompiler extends StellarBaseListener {
@Override
public void exitStringLiteral(StellarParser.StringLiteralContext ctx) {
- expression.tokenDeque.push(new Token<>(ctx.getText().substring(1, ctx.getText().length() - 1), String.class));
+ String rawToken = ctx.getText();
+ String literal = StringEscapeUtils.UNESCAPE_JSON.translate(rawToken);
+ expression.tokenDeque.push(new Token<>(literal.substring(1, literal.length()-1), String.class));
}
@Override
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/6e1e322e/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/generated/StellarLexer.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/generated/StellarLexer.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/generated/StellarLexer.java
index 9f8e0d1..49bbc45 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/generated/StellarLexer.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/generated/StellarLexer.java
@@ -55,7 +55,7 @@ public class StellarLexer extends Lexer {
"MUL", "LBRACE", "RBRACE", "LBRACKET", "RBRACKET", "LPAREN", "RPAREN",
"NIN", "EXISTS", "EXPONENT", "INT_LITERAL", "DOUBLE_LITERAL", "FLOAT_LITERAL",
"LONG_LITERAL", "IDENTIFIER", "STRING_LITERAL", "COMMENT", "WS", "ZERO",
- "FIRST_DIGIT", "DIGIT", "SCHAR", "D", "E", "F", "L", "EOL", "IDENTIFIER_START",
+ "FIRST_DIGIT", "DIGIT", "D", "E", "F", "L", "EOL", "IDENTIFIER_START",
"IDENTIFIER_MIDDLE", "IDENTIFIER_END"
};
@@ -135,164 +135,166 @@ public class StellarLexer extends Lexer {
"\4\32\t\32\4\33\t\33\4\34\t\34\4\35\t\35\4\36\t\36\4\37\t\37\4 \t \4!"+
"\t!\4\"\t\"\4#\t#\4$\t$\4%\t%\4&\t&\4\'\t\'\4(\t(\4)\t)\4*\t*\4+\t+\4"+
",\t,\4-\t-\4.\t.\4/\t/\4\60\t\60\4\61\t\61\4\62\t\62\4\63\t\63\4\64\t"+
- "\64\4\65\t\65\4\66\t\66\4\67\t\67\48\t8\49\t9\3\2\3\2\3\2\3\2\5\2x\n\2"+
- "\3\3\3\3\3\3\3\4\3\4\3\5\3\5\3\6\3\6\3\7\3\7\3\b\3\b\3\b\3\b\3\b\3\b\3"+
- "\b\3\b\5\b\u008d\n\b\3\t\3\t\3\t\3\t\3\t\3\t\5\t\u0095\n\t\3\n\3\n\3\n"+
- "\3\n\3\n\3\n\5\n\u009d\n\n\3\13\3\13\3\13\3\13\3\13\3\13\3\13\3\13\5\13"+
- "\u00a7\n\13\3\f\3\f\3\f\3\f\3\f\3\f\3\f\3\f\3\f\3\f\5\f\u00b3\n\f\3\r"+
- "\3\r\3\r\3\16\3\16\3\16\3\17\3\17\3\20\3\20\3\20\3\21\3\21\3\22\3\22\3"+
- "\22\3\23\3\23\3\24\3\24\3\25\3\25\3\25\3\25\5\25\u00cd\n\25\3\26\3\26"+
- "\3\26\3\26\3\26\3\26\3\26\3\26\5\26\u00d7\n\26\3\27\3\27\3\27\3\27\3\27"+
- "\3\27\3\27\3\27\5\27\u00e1\n\27\3\30\3\30\3\30\3\30\3\30\3\30\3\30\3\30"+
- "\5\30\u00eb\n\30\3\31\3\31\3\32\3\32\3\33\3\33\3\34\3\34\3\35\3\35\3\36"+
- "\3\36\3\37\3\37\3 \3 \3!\3!\3\"\3\"\3#\3#\3#\3#\3#\3#\3#\3#\3#\3#\3#\3"+
- "#\5#\u010d\n#\3$\3$\3$\3$\3$\3$\3$\3$\3$\3$\3$\3$\5$\u011b\n$\3%\3%\3"+
- "%\5%\u0120\n%\3%\6%\u0123\n%\r%\16%\u0124\3&\5&\u0128\n&\3&\3&\5&\u012c"+
- "\n&\3&\3&\7&\u0130\n&\f&\16&\u0133\13&\5&\u0135\n&\3\'\3\'\3\'\7\'\u013a"+
- "\n\'\f\'\16\'\u013d\13\'\3\'\5\'\u0140\n\'\3\'\5\'\u0143\n\'\3\'\3\'\6"+
- "\'\u0147\n\'\r\'\16\'\u0148\3\'\5\'\u014c\n\'\3\'\5\'\u014f\n\'\3\'\3"+
- "\'\3\'\5\'\u0154\n\'\3\'\3\'\5\'\u0158\n\'\3\'\3\'\5\'\u015c\n\'\3(\3"+
- "(\3(\7(\u0161\n(\f(\16(\u0164\13(\3(\5(\u0167\n(\3(\3(\3(\5(\u016c\n("+
- "\3(\3(\6(\u0170\n(\r(\16(\u0171\3(\5(\u0175\n(\3(\3(\3(\3(\5(\u017b\n"+
- "(\3(\3(\5(\u017f\n(\3)\3)\3)\3*\3*\3*\7*\u0187\n*\f*\16*\u018a\13*\3*"+
- "\3*\5*\u018e\n*\3+\3+\7+\u0192\n+\f+\16+\u0195\13+\3+\3+\3+\3+\7+\u019b"+
- "\n+\f+\16+\u019e\13+\3+\3+\5+\u01a2\n+\3,\3,\3,\3,\6,\u01a8\n,\r,\16,"+
- "\u01a9\3,\3,\5,\u01ae\n,\3,\3,\3-\6-\u01b3\n-\r-\16-\u01b4\3-\3-\3.\3"+
- ".\3/\3/\3\60\3\60\3\61\3\61\3\62\3\62\3\63\3\63\3\64\3\64\3\65\3\65\3"+
- "\66\3\66\3\67\3\67\38\38\39\39\3\u01a9\2:\3\3\5\4\7\5\t\6\13\7\r\b\17"+
+ "\64\4\65\t\65\4\66\t\66\4\67\t\67\48\t8\3\2\3\2\3\2\3\2\5\2v\n\2\3\3\3"+
+ "\3\3\3\3\4\3\4\3\5\3\5\3\6\3\6\3\7\3\7\3\b\3\b\3\b\3\b\3\b\3\b\3\b\3\b"+
+ "\5\b\u008b\n\b\3\t\3\t\3\t\3\t\3\t\3\t\5\t\u0093\n\t\3\n\3\n\3\n\3\n\3"+
+ "\n\3\n\5\n\u009b\n\n\3\13\3\13\3\13\3\13\3\13\3\13\3\13\3\13\5\13\u00a5"+
+ "\n\13\3\f\3\f\3\f\3\f\3\f\3\f\3\f\3\f\3\f\3\f\5\f\u00b1\n\f\3\r\3\r\3"+
+ "\r\3\16\3\16\3\16\3\17\3\17\3\20\3\20\3\20\3\21\3\21\3\22\3\22\3\22\3"+
+ "\23\3\23\3\24\3\24\3\25\3\25\3\25\3\25\5\25\u00cb\n\25\3\26\3\26\3\26"+
+ "\3\26\3\26\3\26\3\26\3\26\5\26\u00d5\n\26\3\27\3\27\3\27\3\27\3\27\3\27"+
+ "\3\27\3\27\5\27\u00df\n\27\3\30\3\30\3\30\3\30\3\30\3\30\3\30\3\30\5\30"+
+ "\u00e9\n\30\3\31\3\31\3\32\3\32\3\33\3\33\3\34\3\34\3\35\3\35\3\36\3\36"+
+ "\3\37\3\37\3 \3 \3!\3!\3\"\3\"\3#\3#\3#\3#\3#\3#\3#\3#\3#\3#\3#\3#\5#"+
+ "\u010b\n#\3$\3$\3$\3$\3$\3$\3$\3$\3$\3$\3$\3$\5$\u0119\n$\3%\3%\3%\5%"+
+ "\u011e\n%\3%\6%\u0121\n%\r%\16%\u0122\3&\5&\u0126\n&\3&\3&\5&\u012a\n"+
+ "&\3&\3&\7&\u012e\n&\f&\16&\u0131\13&\5&\u0133\n&\3\'\3\'\3\'\7\'\u0138"+
+ "\n\'\f\'\16\'\u013b\13\'\3\'\5\'\u013e\n\'\3\'\5\'\u0141\n\'\3\'\3\'\6"+
+ "\'\u0145\n\'\r\'\16\'\u0146\3\'\5\'\u014a\n\'\3\'\5\'\u014d\n\'\3\'\3"+
+ "\'\3\'\5\'\u0152\n\'\3\'\3\'\5\'\u0156\n\'\3\'\3\'\5\'\u015a\n\'\3(\3"+
+ "(\3(\7(\u015f\n(\f(\16(\u0162\13(\3(\5(\u0165\n(\3(\3(\3(\5(\u016a\n("+
+ "\3(\3(\6(\u016e\n(\r(\16(\u016f\3(\5(\u0173\n(\3(\3(\3(\3(\5(\u0179\n"+
+ "(\3(\3(\5(\u017d\n(\3)\3)\3)\3*\3*\3*\7*\u0185\n*\f*\16*\u0188\13*\3*"+
+ "\3*\5*\u018c\n*\3+\3+\3+\3+\7+\u0192\n+\f+\16+\u0195\13+\3+\3+\3+\3+\3"+
+ "+\3+\7+\u019d\n+\f+\16+\u01a0\13+\3+\3+\5+\u01a4\n+\3,\3,\3,\3,\6,\u01aa"+
+ "\n,\r,\16,\u01ab\3,\3,\5,\u01b0\n,\3,\3,\3-\6-\u01b5\n-\r-\16-\u01b6\3"+
+ "-\3-\3.\3.\3/\3/\3\60\3\60\3\61\3\61\3\62\3\62\3\63\3\63\3\64\3\64\3\65"+
+ "\3\65\3\66\3\66\3\67\3\67\38\38\3\u01ab\29\3\3\5\4\7\5\t\6\13\7\r\b\17"+
"\t\21\n\23\13\25\f\27\r\31\16\33\17\35\20\37\21!\22#\23%\24\'\25)\26+"+
"\27-\30/\31\61\32\63\33\65\34\67\359\36;\37= ?!A\"C#E$G%I&K\'M(O)Q*S+"+
- "U,W-Y.[\2]\2_\2a\2c\2e\2g\2i\2k\2m\2o\2q\2\3\2\13\5\2\13\f\16\17\"\"\7"+
- "\2\f\f\17\17$$))^^\4\2FFff\4\2GGgg\4\2HHhh\4\2NNnn\6\2&&C\\aac|\b\2\60"+
- "\60\62<C\\^^aac|\b\2\60\60\62;C\\^^aac|\u01f3\2\3\3\2\2\2\2\5\3\2\2\2"+
- "\2\7\3\2\2\2\2\t\3\2\2\2\2\13\3\2\2\2\2\r\3\2\2\2\2\17\3\2\2\2\2\21\3"+
- "\2\2\2\2\23\3\2\2\2\2\25\3\2\2\2\2\27\3\2\2\2\2\31\3\2\2\2\2\33\3\2\2"+
- "\2\2\35\3\2\2\2\2\37\3\2\2\2\2!\3\2\2\2\2#\3\2\2\2\2%\3\2\2\2\2\'\3\2"+
- "\2\2\2)\3\2\2\2\2+\3\2\2\2\2-\3\2\2\2\2/\3\2\2\2\2\61\3\2\2\2\2\63\3\2"+
- "\2\2\2\65\3\2\2\2\2\67\3\2\2\2\29\3\2\2\2\2;\3\2\2\2\2=\3\2\2\2\2?\3\2"+
- "\2\2\2A\3\2\2\2\2C\3\2\2\2\2E\3\2\2\2\2G\3\2\2\2\2I\3\2\2\2\2K\3\2\2\2"+
- "\2M\3\2\2\2\2O\3\2\2\2\2Q\3\2\2\2\2S\3\2\2\2\2U\3\2\2\2\2W\3\2\2\2\2Y"+
- "\3\2\2\2\3w\3\2\2\2\5y\3\2\2\2\7|\3\2\2\2\t~\3\2\2\2\13\u0080\3\2\2\2"+
- "\r\u0082\3\2\2\2\17\u008c\3\2\2\2\21\u0094\3\2\2\2\23\u009c\3\2\2\2\25"+
- "\u00a6\3\2\2\2\27\u00b2\3\2\2\2\31\u00b4\3\2\2\2\33\u00b7\3\2\2\2\35\u00ba"+
- "\3\2\2\2\37\u00bc\3\2\2\2!\u00bf\3\2\2\2#\u00c1\3\2\2\2%\u00c4\3\2\2\2"+
- "\'\u00c6\3\2\2\2)\u00cc\3\2\2\2+\u00d6\3\2\2\2-\u00e0\3\2\2\2/\u00ea\3"+
- "\2\2\2\61\u00ec\3\2\2\2\63\u00ee\3\2\2\2\65\u00f0\3\2\2\2\67\u00f2\3\2"+
- "\2\29\u00f4\3\2\2\2;\u00f6\3\2\2\2=\u00f8\3\2\2\2?\u00fa\3\2\2\2A\u00fc"+
- "\3\2\2\2C\u00fe\3\2\2\2E\u010c\3\2\2\2G\u011a\3\2\2\2I\u011c\3\2\2\2K"+
- "\u0134\3\2\2\2M\u015b\3\2\2\2O\u017e\3\2\2\2Q\u0180\3\2\2\2S\u018d\3\2"+
- "\2\2U\u01a1\3\2\2\2W\u01a3\3\2\2\2Y\u01b2\3\2\2\2[\u01b8\3\2\2\2]\u01ba"+
- "\3\2\2\2_\u01bc\3\2\2\2a\u01be\3\2\2\2c\u01c0\3\2\2\2e\u01c2\3\2\2\2g"+
- "\u01c4\3\2\2\2i\u01c6\3\2\2\2k\u01c8\3\2\2\2m\u01ca\3\2\2\2o\u01cc\3\2"+
- "\2\2q\u01ce\3\2\2\2st\7k\2\2tx\7p\2\2uv\7K\2\2vx\7P\2\2ws\3\2\2\2wu\3"+
- "\2\2\2x\4\3\2\2\2yz\7/\2\2z{\7@\2\2{\6\3\2\2\2|}\7$\2\2}\b\3\2\2\2~\177"+
- "\7)\2\2\177\n\3\2\2\2\u0080\u0081\7.\2\2\u0081\f\3\2\2\2\u0082\u0083\7"+
- "\60\2\2\u0083\16\3\2\2\2\u0084\u0085\7c\2\2\u0085\u0086\7p\2\2\u0086\u008d"+
- "\7f\2\2\u0087\u0088\7(\2\2\u0088\u008d\7(\2\2\u0089\u008a\7C\2\2\u008a"+
- "\u008b\7P\2\2\u008b\u008d\7F\2\2\u008c\u0084\3\2\2\2\u008c\u0087\3\2\2"+
- "\2\u008c\u0089\3\2\2\2\u008d\20\3\2\2\2\u008e\u008f\7q\2\2\u008f\u0095"+
- "\7t\2\2\u0090\u0091\7~\2\2\u0091\u0095\7~\2\2\u0092\u0093\7Q\2\2\u0093"+
- "\u0095\7T\2\2\u0094\u008e\3\2\2\2\u0094\u0090\3\2\2\2\u0094\u0092\3\2"+
- "\2\2\u0095\22\3\2\2\2\u0096\u0097\7p\2\2\u0097\u0098\7q\2\2\u0098\u009d"+
- "\7v\2\2\u0099\u009a\7P\2\2\u009a\u009b\7Q\2\2\u009b\u009d\7V\2\2\u009c"+
- "\u0096\3\2\2\2\u009c\u0099\3\2\2\2\u009d\24\3\2\2\2\u009e\u009f\7v\2\2"+
- "\u009f\u00a0\7t\2\2\u00a0\u00a1\7w\2\2\u00a1\u00a7\7g\2\2\u00a2\u00a3"+
- "\7V\2\2\u00a3\u00a4\7T\2\2\u00a4\u00a5\7W\2\2\u00a5\u00a7\7G\2\2\u00a6"+
- "\u009e\3\2\2\2\u00a6\u00a2\3\2\2\2\u00a7\26\3\2\2\2\u00a8\u00a9\7h\2\2"+
- "\u00a9\u00aa\7c\2\2\u00aa\u00ab\7n\2\2\u00ab\u00ac\7u\2\2\u00ac\u00b3"+
- "\7g\2\2\u00ad\u00ae\7H\2\2\u00ae\u00af\7C\2\2\u00af\u00b0\7N\2\2\u00b0"+
- "\u00b1\7U\2\2\u00b1\u00b3\7G\2\2\u00b2\u00a8\3\2\2\2\u00b2\u00ad\3\2\2"+
- "\2\u00b3\30\3\2\2\2\u00b4\u00b5\7?\2\2\u00b5\u00b6\7?\2\2\u00b6\32\3\2"+
- "\2\2\u00b7\u00b8\7#\2\2\u00b8\u00b9\7?\2\2\u00b9\34\3\2\2\2\u00ba\u00bb"+
- "\7>\2\2\u00bb\36\3\2\2\2\u00bc\u00bd\7>\2\2\u00bd\u00be\7?\2\2\u00be "+
- "\3\2\2\2\u00bf\u00c0\7@\2\2\u00c0\"\3\2\2\2\u00c1\u00c2\7@\2\2\u00c2\u00c3"+
- "\7?\2\2\u00c3$\3\2\2\2\u00c4\u00c5\7A\2\2\u00c5&\3\2\2\2\u00c6\u00c7\7"+
- "<\2\2\u00c7(\3\2\2\2\u00c8\u00c9\7K\2\2\u00c9\u00cd\7H\2\2\u00ca\u00cb"+
- "\7k\2\2\u00cb\u00cd\7h\2\2\u00cc\u00c8\3\2\2\2\u00cc\u00ca\3\2\2\2\u00cd"+
- "*\3\2\2\2\u00ce\u00cf\7V\2\2\u00cf\u00d0\7J\2\2\u00d0\u00d1\7G\2\2\u00d1"+
- "\u00d7\7P\2\2\u00d2\u00d3\7v\2\2\u00d3\u00d4\7j\2\2\u00d4\u00d5\7g\2\2"+
- "\u00d5\u00d7\7p\2\2\u00d6\u00ce\3\2\2\2\u00d6\u00d2\3\2\2\2\u00d7,\3\2"+
- "\2\2\u00d8\u00d9\7G\2\2\u00d9\u00da\7N\2\2\u00da\u00db\7U\2\2\u00db\u00e1"+
- "\7G\2\2\u00dc\u00dd\7g\2\2\u00dd\u00de\7n\2\2\u00de\u00df\7u\2\2\u00df"+
- "\u00e1\7g\2\2\u00e0\u00d8\3\2\2\2\u00e0\u00dc\3\2\2\2\u00e1.\3\2\2\2\u00e2"+
- "\u00e3\7p\2\2\u00e3\u00e4\7w\2\2\u00e4\u00e5\7n\2\2\u00e5\u00eb\7n\2\2"+
- "\u00e6\u00e7\7P\2\2\u00e7\u00e8\7W\2\2\u00e8\u00e9\7N\2\2\u00e9\u00eb"+
- "\7N\2\2\u00ea\u00e2\3\2\2\2\u00ea\u00e6\3\2\2\2\u00eb\60\3\2\2\2\u00ec"+
- "\u00ed\7/\2\2\u00ed\62\3\2\2\2\u00ee\u00ef\7-\2\2\u00ef\64\3\2\2\2\u00f0"+
- "\u00f1\7\61\2\2\u00f1\66\3\2\2\2\u00f2\u00f3\7,\2\2\u00f38\3\2\2\2\u00f4"+
- "\u00f5\7}\2\2\u00f5:\3\2\2\2\u00f6\u00f7\7\177\2\2\u00f7<\3\2\2\2\u00f8"+
- "\u00f9\7]\2\2\u00f9>\3\2\2\2\u00fa\u00fb\7_\2\2\u00fb@\3\2\2\2\u00fc\u00fd"+
- "\7*\2\2\u00fdB\3\2\2\2\u00fe\u00ff\7+\2\2\u00ffD\3\2\2\2\u0100\u0101\7"+
- "p\2\2\u0101\u0102\7q\2\2\u0102\u0103\7v\2\2\u0103\u0104\7\"\2\2\u0104"+
- "\u0105\7k\2\2\u0105\u010d\7p\2\2\u0106\u0107\7P\2\2\u0107\u0108\7Q\2\2"+
- "\u0108\u0109\7V\2\2\u0109\u010a\7\"\2\2\u010a\u010b\7K\2\2\u010b\u010d"+
- "\7P\2\2\u010c\u0100\3\2\2\2\u010c\u0106\3\2\2\2\u010dF\3\2\2\2\u010e\u010f"+
- "\7g\2\2\u010f\u0110\7z\2\2\u0110\u0111\7k\2\2\u0111\u0112\7u\2\2\u0112"+
- "\u0113\7v\2\2\u0113\u011b\7u\2\2\u0114\u0115\7G\2\2\u0115\u0116\7Z\2\2"+
- "\u0116\u0117\7K\2\2\u0117\u0118\7U\2\2\u0118\u0119\7V\2\2\u0119\u011b"+
- "\7U\2\2\u011a\u010e\3\2\2\2\u011a\u0114\3\2\2\2\u011bH\3\2\2\2\u011c\u011f"+
- "\5e\63\2\u011d\u0120\5\63\32\2\u011e\u0120\5\61\31\2\u011f\u011d\3\2\2"+
- "\2\u011f\u011e\3\2\2\2\u011f\u0120\3\2\2\2\u0120\u0122\3\2\2\2\u0121\u0123"+
- "\5_\60\2\u0122\u0121\3\2\2\2\u0123\u0124\3\2\2\2\u0124\u0122\3\2\2\2\u0124"+
- "\u0125\3\2\2\2\u0125J\3\2\2\2\u0126\u0128\5\61\31\2\u0127\u0126\3\2\2"+
- "\2\u0127\u0128\3\2\2\2\u0128\u0129\3\2\2\2\u0129\u0135\5[.\2\u012a\u012c"+
- "\5\61\31\2\u012b\u012a\3\2\2\2\u012b\u012c\3\2\2\2\u012c\u012d\3\2\2\2"+
- "\u012d\u0131\5]/\2\u012e\u0130\5_\60\2\u012f\u012e\3\2\2\2\u0130\u0133"+
- "\3\2\2\2\u0131\u012f\3\2\2\2\u0131\u0132\3\2\2\2\u0132\u0135\3\2\2\2\u0133"+
- "\u0131\3\2\2\2\u0134\u0127\3\2\2\2\u0134\u012b\3\2\2\2\u0135L\3\2\2\2"+
- "\u0136\u0137\5K&\2\u0137\u013b\5\r\7\2\u0138\u013a\5_\60\2\u0139\u0138"+
- "\3\2\2\2\u013a\u013d\3\2\2\2\u013b\u0139\3\2\2\2\u013b\u013c\3\2\2\2\u013c"+
- "\u013f\3\2\2\2\u013d\u013b\3\2\2\2\u013e\u0140\5I%\2\u013f\u013e\3\2\2"+
- "\2\u013f\u0140\3\2\2\2\u0140\u0142\3\2\2\2\u0141\u0143\5c\62\2\u0142\u0141"+
- "\3\2\2\2\u0142\u0143\3\2\2\2\u0143\u015c\3\2\2\2\u0144\u0146\5\r\7\2\u0145"+
- "\u0147\5_\60\2\u0146\u0145\3\2\2\2\u0147\u0148\3\2\2\2\u0148\u0146\3\2"+
- "\2\2\u0148\u0149\3\2\2\2\u0149\u014b\3\2\2\2\u014a\u014c\5I%\2\u014b\u014a"+
- "\3\2\2\2\u014b\u014c\3\2\2\2\u014c\u014e\3\2\2\2\u014d\u014f\5c\62\2\u014e"+
- "\u014d\3\2\2\2\u014e\u014f\3\2\2\2\u014f\u015c\3\2\2\2\u0150\u0151\5K"+
- "&\2\u0151\u0153\5I%\2\u0152\u0154\5c\62\2\u0153\u0152\3\2\2\2\u0153\u0154"+
- "\3\2\2\2\u0154\u015c\3\2\2\2\u0155\u0157\5K&\2\u0156\u0158\5I%\2\u0157"+
- "\u0156\3\2\2\2\u0157\u0158\3\2\2\2\u0158\u0159\3\2\2\2\u0159\u015a\5c"+
- "\62\2\u015a\u015c\3\2\2\2\u015b\u0136\3\2\2\2\u015b\u0144\3\2\2\2\u015b"+
- "\u0150\3\2\2\2\u015b\u0155\3\2\2\2\u015cN\3\2\2\2\u015d\u015e\5K&\2\u015e"+
- "\u0162\5\r\7\2\u015f\u0161\5_\60\2\u0160\u015f\3\2\2\2\u0161\u0164\3\2"+
- "\2\2\u0162\u0160\3\2\2\2\u0162\u0163\3\2\2\2\u0163\u0166\3\2\2\2\u0164"+
- "\u0162\3\2\2\2\u0165\u0167\5I%\2\u0166\u0165\3\2\2\2\u0166\u0167\3\2\2"+
- "\2\u0167\u0168\3\2\2\2\u0168\u0169\5g\64\2\u0169\u017f\3\2\2\2\u016a\u016c"+
- "\5\61\31\2\u016b\u016a\3\2\2\2\u016b\u016c\3\2\2\2\u016c\u016d\3\2\2\2"+
- "\u016d\u016f\5\r\7\2\u016e\u0170\5_\60\2\u016f\u016e\3\2\2\2\u0170\u0171"+
- "\3\2\2\2\u0171\u016f\3\2\2\2\u0171\u0172\3\2\2\2\u0172\u0174\3\2\2\2\u0173"+
- "\u0175\5I%\2\u0174\u0173\3\2\2\2\u0174\u0175\3\2\2\2\u0175\u0176\3\2\2"+
- "\2\u0176\u0177\5g\64\2\u0177\u017f\3\2\2\2\u0178\u017a\5K&\2\u0179\u017b"+
- "\5I%\2\u017a\u0179\3\2\2\2\u017a\u017b\3\2\2\2\u017b\u017c\3\2\2\2\u017c"+
- "\u017d\5g\64\2\u017d\u017f\3\2\2\2\u017e\u015d\3\2\2\2\u017e\u016b\3\2"+
- "\2\2\u017e\u0178\3\2\2\2\u017fP\3\2\2\2\u0180\u0181\5K&\2\u0181\u0182"+
- "\5i\65\2\u0182R\3\2\2\2\u0183\u018e\5m\67\2\u0184\u0188\5m\67\2\u0185"+
- "\u0187\5o8\2\u0186\u0185\3\2\2\2\u0187\u018a\3\2\2\2\u0188\u0186\3\2\2"+
- "\2\u0188\u0189\3\2\2\2\u0189\u018b\3\2\2\2\u018a\u0188\3\2\2\2\u018b\u018c"+
- "\5q9\2\u018c\u018e\3\2\2\2\u018d\u0183\3\2\2\2\u018d\u0184\3\2\2\2\u018e"+
- "T\3\2\2\2\u018f\u0193\5\7\4\2\u0190\u0192\5a\61\2\u0191\u0190\3\2\2\2"+
- "\u0192\u0195\3\2\2\2\u0193\u0191\3\2\2\2\u0193\u0194\3\2\2\2\u0194\u0196"+
- "\3\2\2\2\u0195\u0193\3\2\2\2\u0196\u0197\5\7\4\2\u0197\u01a2\3\2\2\2\u0198"+
- "\u019c\5\t\5\2\u0199\u019b\5a\61\2\u019a\u0199\3\2\2\2\u019b\u019e\3\2"+
- "\2\2\u019c\u019a\3\2\2\2\u019c\u019d\3\2\2\2\u019d\u019f\3\2\2\2\u019e"+
- "\u019c\3\2\2\2\u019f\u01a0\5\t\5\2\u01a0\u01a2\3\2\2\2\u01a1\u018f\3\2"+
- "\2\2\u01a1\u0198\3\2\2\2\u01a2V\3\2\2\2\u01a3\u01a4\7\61\2\2\u01a4\u01a5"+
- "\7\61\2\2\u01a5\u01a7\3\2\2\2\u01a6\u01a8\13\2\2\2\u01a7\u01a6\3\2\2\2"+
- "\u01a8\u01a9\3\2\2\2\u01a9\u01aa\3\2\2\2\u01a9\u01a7\3\2\2\2\u01aa\u01ad"+
- "\3\2\2\2\u01ab\u01ae\5k\66\2\u01ac\u01ae\7\2\2\3\u01ad\u01ab\3\2\2\2\u01ad"+
- "\u01ac\3\2\2\2\u01ae\u01af\3\2\2\2\u01af\u01b0\b,\2\2\u01b0X\3\2\2\2\u01b1"+
- "\u01b3\t\2\2\2\u01b2\u01b1\3\2\2\2\u01b3\u01b4\3\2\2\2\u01b4\u01b2\3\2"+
- "\2\2\u01b4\u01b5\3\2\2\2\u01b5\u01b6\3\2\2\2\u01b6\u01b7\b-\2\2\u01b7"+
- "Z\3\2\2\2\u01b8\u01b9\7\62\2\2\u01b9\\\3\2\2\2\u01ba\u01bb\4\63;\2\u01bb"+
- "^\3\2\2\2\u01bc\u01bd\4\62;\2\u01bd`\3\2\2\2\u01be\u01bf\n\3\2\2\u01bf"+
- "b\3\2\2\2\u01c0\u01c1\t\4\2\2\u01c1d\3\2\2\2\u01c2\u01c3\t\5\2\2\u01c3"+
- "f\3\2\2\2\u01c4\u01c5\t\6\2\2\u01c5h\3\2\2\2\u01c6\u01c7\t\7\2\2\u01c7"+
- "j\3\2\2\2\u01c8\u01c9\7\f\2\2\u01c9l\3\2\2\2\u01ca\u01cb\t\b\2\2\u01cb"+
- "n\3\2\2\2\u01cc\u01cd\t\t\2\2\u01cdp\3\2\2\2\u01ce\u01cf\t\n\2\2\u01cf"+
- "r\3\2\2\2-\2w\u008c\u0094\u009c\u00a6\u00b2\u00cc\u00d6\u00e0\u00ea\u010c"+
- "\u011a\u011f\u0124\u0127\u012b\u0131\u0134\u013b\u013f\u0142\u0148\u014b"+
- "\u014e\u0153\u0157\u015b\u0162\u0166\u016b\u0171\u0174\u017a\u017e\u0188"+
- "\u018d\u0193\u019c\u01a1\u01a9\u01ad\u01b4\3\b\2\2";
+ "U,W-Y.[\2]\2_\2a\2c\2e\2g\2i\2k\2m\2o\2\3\2\16\4\2))^^\7\2))^^ppttvv\4"+
+ "\2$$^^\7\2$$^^ppttvv\5\2\13\f\16\17\"\"\4\2FFff\4\2GGgg\4\2HHhh\4\2NN"+
+ "nn\6\2&&C\\aac|\b\2\60\60\62<C\\^^aac|\b\2\60\60\62;C\\^^aac|\u01f6\2"+
+ "\3\3\2\2\2\2\5\3\2\2\2\2\7\3\2\2\2\2\t\3\2\2\2\2\13\3\2\2\2\2\r\3\2\2"+
+ "\2\2\17\3\2\2\2\2\21\3\2\2\2\2\23\3\2\2\2\2\25\3\2\2\2\2\27\3\2\2\2\2"+
+ "\31\3\2\2\2\2\33\3\2\2\2\2\35\3\2\2\2\2\37\3\2\2\2\2!\3\2\2\2\2#\3\2\2"+
+ "\2\2%\3\2\2\2\2\'\3\2\2\2\2)\3\2\2\2\2+\3\2\2\2\2-\3\2\2\2\2/\3\2\2\2"+
+ "\2\61\3\2\2\2\2\63\3\2\2\2\2\65\3\2\2\2\2\67\3\2\2\2\29\3\2\2\2\2;\3\2"+
+ "\2\2\2=\3\2\2\2\2?\3\2\2\2\2A\3\2\2\2\2C\3\2\2\2\2E\3\2\2\2\2G\3\2\2\2"+
+ "\2I\3\2\2\2\2K\3\2\2\2\2M\3\2\2\2\2O\3\2\2\2\2Q\3\2\2\2\2S\3\2\2\2\2U"+
+ "\3\2\2\2\2W\3\2\2\2\2Y\3\2\2\2\3u\3\2\2\2\5w\3\2\2\2\7z\3\2\2\2\t|\3\2"+
+ "\2\2\13~\3\2\2\2\r\u0080\3\2\2\2\17\u008a\3\2\2\2\21\u0092\3\2\2\2\23"+
+ "\u009a\3\2\2\2\25\u00a4\3\2\2\2\27\u00b0\3\2\2\2\31\u00b2\3\2\2\2\33\u00b5"+
+ "\3\2\2\2\35\u00b8\3\2\2\2\37\u00ba\3\2\2\2!\u00bd\3\2\2\2#\u00bf\3\2\2"+
+ "\2%\u00c2\3\2\2\2\'\u00c4\3\2\2\2)\u00ca\3\2\2\2+\u00d4\3\2\2\2-\u00de"+
+ "\3\2\2\2/\u00e8\3\2\2\2\61\u00ea\3\2\2\2\63\u00ec\3\2\2\2\65\u00ee\3\2"+
+ "\2\2\67\u00f0\3\2\2\29\u00f2\3\2\2\2;\u00f4\3\2\2\2=\u00f6\3\2\2\2?\u00f8"+
+ "\3\2\2\2A\u00fa\3\2\2\2C\u00fc\3\2\2\2E\u010a\3\2\2\2G\u0118\3\2\2\2I"+
+ "\u011a\3\2\2\2K\u0132\3\2\2\2M\u0159\3\2\2\2O\u017c\3\2\2\2Q\u017e\3\2"+
+ "\2\2S\u018b\3\2\2\2U\u01a3\3\2\2\2W\u01a5\3\2\2\2Y\u01b4\3\2\2\2[\u01ba"+
+ "\3\2\2\2]\u01bc\3\2\2\2_\u01be\3\2\2\2a\u01c0\3\2\2\2c\u01c2\3\2\2\2e"+
+ "\u01c4\3\2\2\2g\u01c6\3\2\2\2i\u01c8\3\2\2\2k\u01ca\3\2\2\2m\u01cc\3\2"+
+ "\2\2o\u01ce\3\2\2\2qr\7k\2\2rv\7p\2\2st\7K\2\2tv\7P\2\2uq\3\2\2\2us\3"+
+ "\2\2\2v\4\3\2\2\2wx\7/\2\2xy\7@\2\2y\6\3\2\2\2z{\7$\2\2{\b\3\2\2\2|}\7"+
+ ")\2\2}\n\3\2\2\2~\177\7.\2\2\177\f\3\2\2\2\u0080\u0081\7\60\2\2\u0081"+
+ "\16\3\2\2\2\u0082\u0083\7c\2\2\u0083\u0084\7p\2\2\u0084\u008b\7f\2\2\u0085"+
+ "\u0086\7(\2\2\u0086\u008b\7(\2\2\u0087\u0088\7C\2\2\u0088\u0089\7P\2\2"+
+ "\u0089\u008b\7F\2\2\u008a\u0082\3\2\2\2\u008a\u0085\3\2\2\2\u008a\u0087"+
+ "\3\2\2\2\u008b\20\3\2\2\2\u008c\u008d\7q\2\2\u008d\u0093\7t\2\2\u008e"+
+ "\u008f\7~\2\2\u008f\u0093\7~\2\2\u0090\u0091\7Q\2\2\u0091\u0093\7T\2\2"+
+ "\u0092\u008c\3\2\2\2\u0092\u008e\3\2\2\2\u0092\u0090\3\2\2\2\u0093\22"+
+ "\3\2\2\2\u0094\u0095\7p\2\2\u0095\u0096\7q\2\2\u0096\u009b\7v\2\2\u0097"+
+ "\u0098\7P\2\2\u0098\u0099\7Q\2\2\u0099\u009b\7V\2\2\u009a\u0094\3\2\2"+
+ "\2\u009a\u0097\3\2\2\2\u009b\24\3\2\2\2\u009c\u009d\7v\2\2\u009d\u009e"+
+ "\7t\2\2\u009e\u009f\7w\2\2\u009f\u00a5\7g\2\2\u00a0\u00a1\7V\2\2\u00a1"+
+ "\u00a2\7T\2\2\u00a2\u00a3\7W\2\2\u00a3\u00a5\7G\2\2\u00a4\u009c\3\2\2"+
+ "\2\u00a4\u00a0\3\2\2\2\u00a5\26\3\2\2\2\u00a6\u00a7\7h\2\2\u00a7\u00a8"+
+ "\7c\2\2\u00a8\u00a9\7n\2\2\u00a9\u00aa\7u\2\2\u00aa\u00b1\7g\2\2\u00ab"+
+ "\u00ac\7H\2\2\u00ac\u00ad\7C\2\2\u00ad\u00ae\7N\2\2\u00ae\u00af\7U\2\2"+
+ "\u00af\u00b1\7G\2\2\u00b0\u00a6\3\2\2\2\u00b0\u00ab\3\2\2\2\u00b1\30\3"+
+ "\2\2\2\u00b2\u00b3\7?\2\2\u00b3\u00b4\7?\2\2\u00b4\32\3\2\2\2\u00b5\u00b6"+
+ "\7#\2\2\u00b6\u00b7\7?\2\2\u00b7\34\3\2\2\2\u00b8\u00b9\7>\2\2\u00b9\36"+
+ "\3\2\2\2\u00ba\u00bb\7>\2\2\u00bb\u00bc\7?\2\2\u00bc \3\2\2\2\u00bd\u00be"+
+ "\7@\2\2\u00be\"\3\2\2\2\u00bf\u00c0\7@\2\2\u00c0\u00c1\7?\2\2\u00c1$\3"+
+ "\2\2\2\u00c2\u00c3\7A\2\2\u00c3&\3\2\2\2\u00c4\u00c5\7<\2\2\u00c5(\3\2"+
+ "\2\2\u00c6\u00c7\7K\2\2\u00c7\u00cb\7H\2\2\u00c8\u00c9\7k\2\2\u00c9\u00cb"+
+ "\7h\2\2\u00ca\u00c6\3\2\2\2\u00ca\u00c8\3\2\2\2\u00cb*\3\2\2\2\u00cc\u00cd"+
+ "\7V\2\2\u00cd\u00ce\7J\2\2\u00ce\u00cf\7G\2\2\u00cf\u00d5\7P\2\2\u00d0"+
+ "\u00d1\7v\2\2\u00d1\u00d2\7j\2\2\u00d2\u00d3\7g\2\2\u00d3\u00d5\7p\2\2"+
+ "\u00d4\u00cc\3\2\2\2\u00d4\u00d0\3\2\2\2\u00d5,\3\2\2\2\u00d6\u00d7\7"+
+ "G\2\2\u00d7\u00d8\7N\2\2\u00d8\u00d9\7U\2\2\u00d9\u00df\7G\2\2\u00da\u00db"+
+ "\7g\2\2\u00db\u00dc\7n\2\2\u00dc\u00dd\7u\2\2\u00dd\u00df\7g\2\2\u00de"+
+ "\u00d6\3\2\2\2\u00de\u00da\3\2\2\2\u00df.\3\2\2\2\u00e0\u00e1\7p\2\2\u00e1"+
+ "\u00e2\7w\2\2\u00e2\u00e3\7n\2\2\u00e3\u00e9\7n\2\2\u00e4\u00e5\7P\2\2"+
+ "\u00e5\u00e6\7W\2\2\u00e6\u00e7\7N\2\2\u00e7\u00e9\7N\2\2\u00e8\u00e0"+
+ "\3\2\2\2\u00e8\u00e4\3\2\2\2\u00e9\60\3\2\2\2\u00ea\u00eb\7/\2\2\u00eb"+
+ "\62\3\2\2\2\u00ec\u00ed\7-\2\2\u00ed\64\3\2\2\2\u00ee\u00ef\7\61\2\2\u00ef"+
+ "\66\3\2\2\2\u00f0\u00f1\7,\2\2\u00f18\3\2\2\2\u00f2\u00f3\7}\2\2\u00f3"+
+ ":\3\2\2\2\u00f4\u00f5\7\177\2\2\u00f5<\3\2\2\2\u00f6\u00f7\7]\2\2\u00f7"+
+ ">\3\2\2\2\u00f8\u00f9\7_\2\2\u00f9@\3\2\2\2\u00fa\u00fb\7*\2\2\u00fbB"+
+ "\3\2\2\2\u00fc\u00fd\7+\2\2\u00fdD\3\2\2\2\u00fe\u00ff\7p\2\2\u00ff\u0100"+
+ "\7q\2\2\u0100\u0101\7v\2\2\u0101\u0102\7\"\2\2\u0102\u0103\7k\2\2\u0103"+
+ "\u010b\7p\2\2\u0104\u0105\7P\2\2\u0105\u0106\7Q\2\2\u0106\u0107\7V\2\2"+
+ "\u0107\u0108\7\"\2\2\u0108\u0109\7K\2\2\u0109\u010b\7P\2\2\u010a\u00fe"+
+ "\3\2\2\2\u010a\u0104\3\2\2\2\u010bF\3\2\2\2\u010c\u010d\7g\2\2\u010d\u010e"+
+ "\7z\2\2\u010e\u010f\7k\2\2\u010f\u0110\7u\2\2\u0110\u0111\7v\2\2\u0111"+
+ "\u0119\7u\2\2\u0112\u0113\7G\2\2\u0113\u0114\7Z\2\2\u0114\u0115\7K\2\2"+
+ "\u0115\u0116\7U\2\2\u0116\u0117\7V\2\2\u0117\u0119\7U\2\2\u0118\u010c"+
+ "\3\2\2\2\u0118\u0112\3\2\2\2\u0119H\3\2\2\2\u011a\u011d\5c\62\2\u011b"+
+ "\u011e\5\63\32\2\u011c\u011e\5\61\31\2\u011d\u011b\3\2\2\2\u011d\u011c"+
+ "\3\2\2\2\u011d\u011e\3\2\2\2\u011e\u0120\3\2\2\2\u011f\u0121\5_\60\2\u0120"+
+ "\u011f\3\2\2\2\u0121\u0122\3\2\2\2\u0122\u0120\3\2\2\2\u0122\u0123\3\2"+
+ "\2\2\u0123J\3\2\2\2\u0124\u0126\5\61\31\2\u0125\u0124\3\2\2\2\u0125\u0126"+
+ "\3\2\2\2\u0126\u0127\3\2\2\2\u0127\u0133\5[.\2\u0128\u012a\5\61\31\2\u0129"+
+ "\u0128\3\2\2\2\u0129\u012a\3\2\2\2\u012a\u012b\3\2\2\2\u012b\u012f\5]"+
+ "/\2\u012c\u012e\5_\60\2\u012d\u012c\3\2\2\2\u012e\u0131\3\2\2\2\u012f"+
+ "\u012d\3\2\2\2\u012f\u0130\3\2\2\2\u0130\u0133\3\2\2\2\u0131\u012f\3\2"+
+ "\2\2\u0132\u0125\3\2\2\2\u0132\u0129\3\2\2\2\u0133L\3\2\2\2\u0134\u0135"+
+ "\5K&\2\u0135\u0139\5\r\7\2\u0136\u0138\5_\60\2\u0137\u0136\3\2\2\2\u0138"+
+ "\u013b\3\2\2\2\u0139\u0137\3\2\2\2\u0139\u013a\3\2\2\2\u013a\u013d\3\2"+
+ "\2\2\u013b\u0139\3\2\2\2\u013c\u013e\5I%\2\u013d\u013c\3\2\2\2\u013d\u013e"+
+ "\3\2\2\2\u013e\u0140\3\2\2\2\u013f\u0141\5a\61\2\u0140\u013f\3\2\2\2\u0140"+
+ "\u0141\3\2\2\2\u0141\u015a\3\2\2\2\u0142\u0144\5\r\7\2\u0143\u0145\5_"+
+ "\60\2\u0144\u0143\3\2\2\2\u0145\u0146\3\2\2\2\u0146\u0144\3\2\2\2\u0146"+
+ "\u0147\3\2\2\2\u0147\u0149\3\2\2\2\u0148\u014a\5I%\2\u0149\u0148\3\2\2"+
+ "\2\u0149\u014a\3\2\2\2\u014a\u014c\3\2\2\2\u014b\u014d\5a\61\2\u014c\u014b"+
+ "\3\2\2\2\u014c\u014d\3\2\2\2\u014d\u015a\3\2\2\2\u014e\u014f\5K&\2\u014f"+
+ "\u0151\5I%\2\u0150\u0152\5a\61\2\u0151\u0150\3\2\2\2\u0151\u0152\3\2\2"+
+ "\2\u0152\u015a\3\2\2\2\u0153\u0155\5K&\2\u0154\u0156\5I%\2\u0155\u0154"+
+ "\3\2\2\2\u0155\u0156\3\2\2\2\u0156\u0157\3\2\2\2\u0157\u0158\5a\61\2\u0158"+
+ "\u015a\3\2\2\2\u0159\u0134\3\2\2\2\u0159\u0142\3\2\2\2\u0159\u014e\3\2"+
+ "\2\2\u0159\u0153\3\2\2\2\u015aN\3\2\2\2\u015b\u015c\5K&\2\u015c\u0160"+
+ "\5\r\7\2\u015d\u015f\5_\60\2\u015e\u015d\3\2\2\2\u015f\u0162\3\2\2\2\u0160"+
+ "\u015e\3\2\2\2\u0160\u0161\3\2\2\2\u0161\u0164\3\2\2\2\u0162\u0160\3\2"+
+ "\2\2\u0163\u0165\5I%\2\u0164\u0163\3\2\2\2\u0164\u0165\3\2\2\2\u0165\u0166"+
+ "\3\2\2\2\u0166\u0167\5e\63\2\u0167\u017d\3\2\2\2\u0168\u016a\5\61\31\2"+
+ "\u0169\u0168\3\2\2\2\u0169\u016a\3\2\2\2\u016a\u016b\3\2\2\2\u016b\u016d"+
+ "\5\r\7\2\u016c\u016e\5_\60\2\u016d\u016c\3\2\2\2\u016e\u016f\3\2\2\2\u016f"+
+ "\u016d\3\2\2\2\u016f\u0170\3\2\2\2\u0170\u0172\3\2\2\2\u0171\u0173\5I"+
+ "%\2\u0172\u0171\3\2\2\2\u0172\u0173\3\2\2\2\u0173\u0174\3\2\2\2\u0174"+
+ "\u0175\5e\63\2\u0175\u017d\3\2\2\2\u0176\u0178\5K&\2\u0177\u0179\5I%\2"+
+ "\u0178\u0177\3\2\2\2\u0178\u0179\3\2\2\2\u0179\u017a\3\2\2\2\u017a\u017b"+
+ "\5e\63\2\u017b\u017d\3\2\2\2\u017c\u015b\3\2\2\2\u017c\u0169\3\2\2\2\u017c"+
+ "\u0176\3\2\2\2\u017dP\3\2\2\2\u017e\u017f\5K&\2\u017f\u0180\5g\64\2\u0180"+
+ "R\3\2\2\2\u0181\u018c\5k\66\2\u0182\u0186\5k\66\2\u0183\u0185\5m\67\2"+
+ "\u0184\u0183\3\2\2\2\u0185\u0188\3\2\2\2\u0186\u0184\3\2\2\2\u0186\u0187"+
+ "\3\2\2\2\u0187\u0189\3\2\2\2\u0188\u0186\3\2\2\2\u0189\u018a\5o8\2\u018a"+
+ "\u018c\3\2\2\2\u018b\u0181\3\2\2\2\u018b\u0182\3\2\2\2\u018cT\3\2\2\2"+
+ "\u018d\u0193\5\t\5\2\u018e\u0192\n\2\2\2\u018f\u0190\7^\2\2\u0190\u0192"+
+ "\t\3\2\2\u0191\u018e\3\2\2\2\u0191\u018f\3\2\2\2\u0192\u0195\3\2\2\2\u0193"+
+ "\u0191\3\2\2\2\u0193\u0194\3\2\2\2\u0194\u0196\3\2\2\2\u0195\u0193\3\2"+
+ "\2\2\u0196\u0197\5\t\5\2\u0197\u01a4\3\2\2\2\u0198\u019e\5\7\4\2\u0199"+
+ "\u019d\n\4\2\2\u019a\u019b\7^\2\2\u019b\u019d\t\5\2\2\u019c\u0199\3\2"+
+ "\2\2\u019c\u019a\3\2\2\2\u019d\u01a0\3\2\2\2\u019e\u019c\3\2\2\2\u019e"+
+ "\u019f\3\2\2\2\u019f\u01a1\3\2\2\2\u01a0\u019e\3\2\2\2\u01a1\u01a2\5\7"+
+ "\4\2\u01a2\u01a4\3\2\2\2\u01a3\u018d\3\2\2\2\u01a3\u0198\3\2\2\2\u01a4"+
+ "V\3\2\2\2\u01a5\u01a6\7\61\2\2\u01a6\u01a7\7\61\2\2\u01a7\u01a9\3\2\2"+
+ "\2\u01a8\u01aa\13\2\2\2\u01a9\u01a8\3\2\2\2\u01aa\u01ab\3\2\2\2\u01ab"+
+ "\u01ac\3\2\2\2\u01ab\u01a9\3\2\2\2\u01ac\u01af\3\2\2\2\u01ad\u01b0\5i"+
+ "\65\2\u01ae\u01b0\7\2\2\3\u01af\u01ad\3\2\2\2\u01af\u01ae\3\2\2\2\u01b0"+
+ "\u01b1\3\2\2\2\u01b1\u01b2\b,\2\2\u01b2X\3\2\2\2\u01b3\u01b5\t\6\2\2\u01b4"+
+ "\u01b3\3\2\2\2\u01b5\u01b6\3\2\2\2\u01b6\u01b4\3\2\2\2\u01b6\u01b7\3\2"+
+ "\2\2\u01b7\u01b8\3\2\2\2\u01b8\u01b9\b-\2\2\u01b9Z\3\2\2\2\u01ba\u01bb"+
+ "\7\62\2\2\u01bb\\\3\2\2\2\u01bc\u01bd\4\63;\2\u01bd^\3\2\2\2\u01be\u01bf"+
+ "\4\62;\2\u01bf`\3\2\2\2\u01c0\u01c1\t\7\2\2\u01c1b\3\2\2\2\u01c2\u01c3"+
+ "\t\b\2\2\u01c3d\3\2\2\2\u01c4\u01c5\t\t\2\2\u01c5f\3\2\2\2\u01c6\u01c7"+
+ "\t\n\2\2\u01c7h\3\2\2\2\u01c8\u01c9\7\f\2\2\u01c9j\3\2\2\2\u01ca\u01cb"+
+ "\t\13\2\2\u01cbl\3\2\2\2\u01cc\u01cd\t\f\2\2\u01cdn\3\2\2\2\u01ce\u01cf"+
+ "\t\r\2\2\u01cfp\3\2\2\2/\2u\u008a\u0092\u009a\u00a4\u00b0\u00ca\u00d4"+
+ "\u00de\u00e8\u010a\u0118\u011d\u0122\u0125\u0129\u012f\u0132\u0139\u013d"+
+ "\u0140\u0146\u0149\u014c\u0151\u0155\u0159\u0160\u0164\u0169\u016f\u0172"+
+ "\u0178\u017c\u0186\u018b\u0191\u0193\u019c\u019e\u01a3\u01ab\u01af\u01b6"+
+ "\3\b\2\2";
public static final ATN _ATN =
new ATNDeserializer().deserialize(_serializedATN.toCharArray());
static {
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/6e1e322e/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarTest.java b/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarTest.java
index 4c0a5e9..d7a33d4 100644
--- a/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarTest.java
+++ b/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarTest.java
@@ -64,6 +64,22 @@ public class StellarTest {
Assert.assertTrue(numFound > 0);
}
+ @Test
+ public void testEscapedLiterals() {
+ Assert.assertEquals("'bar'", run("\"'bar'\"", new HashMap<>()));
+ Assert.assertEquals("'BAR'", run("TO_UPPER('\\'bar\\'')", new HashMap<>()));
+ Assert.assertEquals("\"bar\"", run("\"\\\"bar\\\"\"", new HashMap<>()));
+ Assert.assertEquals("\"bar\"", run("'\"bar\"'", new HashMap<>()));
+ Assert.assertEquals("\"BAR\"", run("TO_UPPER(\"\\\"bar\\\"\")", new HashMap<>()));
+ Assert.assertEquals("bar \\ foo", run("'bar \\\\ foo'", new HashMap<>()));
+ Assert.assertEquals("bar \\\\ foo", run("'bar \\\\\\\\ foo'", new HashMap<>()));
+ Assert.assertEquals("bar\nfoo", run("'bar\\nfoo'", new HashMap<>()));
+ Assert.assertEquals("bar\n\nfoo", run("'bar\\n\\nfoo'", new HashMap<>()));
+ Assert.assertEquals("bar\tfoo", run("'bar\\tfoo'", new HashMap<>()));
+ Assert.assertEquals("bar\t\tfoo", run("'bar\\t\\tfoo'", new HashMap<>()));
+ Assert.assertEquals("bar\rfoo", run("'bar\\rfoo'", new HashMap<>()));
+ Assert.assertEquals("'bar'", run("'\\'bar\\''", new HashMap<>()));
+ }
@Test
public void testVariableResolution() {
[22/26] incubator-metron git commit: METRON-857 Metron should one
unified docker build image (ottobackwards via merrimanr) closes
apache/incubator-metron#552
Posted by ma...@apache.org.
METRON-857 Metron should one unified docker build image (ottobackwards via merrimanr) closes apache/incubator-metron#552
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/ffe2d934
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/ffe2d934
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/ffe2d934
Branch: refs/heads/Metron_0.4.0
Commit: ffe2d934089ba6c63513759a9a740c1e34660132
Parents: 58fa1f6
Author: ottobackwards <ot...@gmail.com>
Authored: Wed Apr 26 09:24:07 2017 -0500
Committer: merrimanr <me...@gmail.com>
Committed: Wed Apr 26 09:24:07 2017 -0500
----------------------------------------------------------------------
.../packaging/docker/rpm-docker/pom.xml | 57 ++++++++++++++------
1 file changed, 42 insertions(+), 15 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ffe2d934/metron-deployment/packaging/docker/rpm-docker/pom.xml
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/docker/rpm-docker/pom.xml b/metron-deployment/packaging/docker/rpm-docker/pom.xml
index dd1a995..ec22223 100644
--- a/metron-deployment/packaging/docker/rpm-docker/pom.xml
+++ b/metron-deployment/packaging/docker/rpm-docker/pom.xml
@@ -164,21 +164,6 @@
</resources>
</configuration>
</execution>
- <execution>
- <id>copy-rpms-target</id>
- <phase>package</phase>
- <goals>
- <goal>copy-resources</goal>
- </goals>
- <configuration>
- <outputDirectory>${project.basedir}/target/RPMS</outputDirectory>
- <resources>
- <resource>
- <directory>RPMS</directory>
- </resource>
- </resources>
- </configuration>
- </execution>
</executions>
</plugin>
</plugins>
@@ -215,6 +200,27 @@
</execution>
</executions>
</plugin>
+ <plugin>
+ <artifactId>maven-resources-plugin</artifactId>
+ <version>3.0.1</version>
+ <executions>
+ <execution>
+ <id>copy-rpms-target</id>
+ <phase>package</phase>
+ <goals>
+ <goal>copy-resources</goal>
+ </goals>
+ <configuration>
+ <outputDirectory>${project.basedir}/target/RPMS</outputDirectory>
+ <resources>
+ <resource>
+ <directory>RPMS</directory>
+ </resource>
+ </resources>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
</plugins>
</build>
</profile>
@@ -272,6 +278,27 @@
</execution>
</executions>
</plugin>
+ <plugin>
+ <artifactId>maven-resources-plugin</artifactId>
+ <version>3.0.1</version>
+ <executions>
+ <execution>
+ <id>copy-rpms-target</id>
+ <phase>package</phase>
+ <goals>
+ <goal>copy-resources</goal>
+ </goals>
+ <configuration>
+ <outputDirectory>${project.basedir}/target/RPMS</outputDirectory>
+ <resources>
+ <resource>
+ <directory>RPMS</directory>
+ </resource>
+ </resources>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
</plugins>
</build>
</profile>
[02/26] incubator-metron git commit: METRON-849: Add 'reason' to
threat triage risk level rules doc (mmiklavc) closes
apache/incubator-metron#529
Posted by ma...@apache.org.
METRON-849: Add 'reason' to threat triage risk level rules doc (mmiklavc) closes apache/incubator-metron#529
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/eb08d145
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/eb08d145
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/eb08d145
Branch: refs/heads/Metron_0.4.0
Commit: eb08d145f00166ab73316714f96fdb8d53de0f79
Parents: e6992d5
Author: mmiklavc <mi...@gmail.com>
Authored: Tue Apr 18 14:16:06 2017 -0600
Committer: Michael Miklavcic <mi...@gmail.com>
Committed: Tue Apr 18 14:16:06 2017 -0600
----------------------------------------------------------------------
metron-platform/metron-enrichment/README.md | 16 +++++++++-------
1 file changed, 9 insertions(+), 7 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/eb08d145/metron-platform/metron-enrichment/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/README.md b/metron-platform/metron-enrichment/README.md
index d08929b..a789f0f 100644
--- a/metron-platform/metron-enrichment/README.md
+++ b/metron-platform/metron-enrichment/README.md
@@ -1,4 +1,4 @@
-#Enrichment
+# Enrichment
## Introduction
@@ -29,7 +29,7 @@ There are two types of configurations at the moment, `global` and
See the "[Global Configuration](../metron-common)" section.
-##Sensor Enrichment Configuration
+## Sensor Enrichment Configuration
The sensor specific configuration is intended to configure the
individual enrichments and threat intelligence enrichments for a given
@@ -41,7 +41,7 @@ The configuration is a complex JSON object with the following top level fields:
* `enrichment` : A complex JSON object representing the configuration of the enrichments
* `threatIntel` : A complex JSON object representing the configuration of the threat intelligence enrichments
-###The `enrichment` Configuration
+### The `enrichment` Configuration
| Field | Description | Example |
@@ -106,7 +106,7 @@ The other way in which the stellar enrichment is somewhat more complex is in how
```
We have a group called `numeric` whose stellar statements will be executed sequentially. In parallel to that, we have the group of stellar statements under the group `text` executing. The intent here is to allow you to not force higher latency operations to be done sequentially. You can use any name for your groupings you like. Be aware that the configuration is a map and duplicate configuration keys' values are not combined, so the duplicate configuration value will be overwritten.
-###The `threatIntel` Configuration
+### The `threatIntel` Configuration
| Field | Description | Example |
|------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------|
@@ -131,6 +131,7 @@ A risk level rule is of the following format:
* `comment` : A comment describing the rule
* `rule` : The rule, represented as a Stellar statement
* `score` : Associated threat triage score for the rule
+* `reason` : Reason the rule tripped. Can be represented as a Stellar statement
An example of a rule is as follows:
```
@@ -138,8 +139,9 @@ An example of a rule is as follows:
{
"name" : "is internal"
, "comment" : "determines if the destination is internal."
- , rule" : "IN_SUBNET(ip_dst_addr, '192.168.0.0/24')"
- , "score" : 10
+ , "rule" : "IN_SUBNET(ip_dst_addr, '192.168.0.0/24')"
+ , "score" : 10
+ , "reason" : "FORMAT('%s is internal', ip_dst_addr)"
}
]
```
@@ -150,7 +152,7 @@ The supported aggregation functions are:
* `MEAN` : The mean of all of the associated values for matching queries
* `POSITIVE_MEAN` : The mean of the positive associated values for the matching queries.
-###Example Configuration
+### Example Configuration
An example configuration for the YAF sensor is as follows:
```json
[07/26] incubator-metron git commit: METRON-821 Minor fixes in full
dev kerberos setup instructions (JonZeolla) closes
apache/incubator-metron#510
Posted by ma...@apache.org.
METRON-821 Minor fixes in full dev kerberos setup instructions (JonZeolla) closes apache/incubator-metron#510
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/29d7cb37
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/29d7cb37
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/29d7cb37
Branch: refs/heads/Metron_0.4.0
Commit: 29d7cb378835b819b326660ed59aca4becbf7099
Parents: 2ecabaa
Author: JonZeolla <ze...@gmail.com>
Authored: Wed Apr 19 21:32:58 2017 -0400
Committer: jonzeolla <jo...@apache.org>
Committed: Wed Apr 19 21:32:58 2017 -0400
----------------------------------------------------------------------
.github/PULL_REQUEST_TEMPLATE.md | 2 +-
metron-deployment/vagrant/Kerberos-setup.md | 313 ++++++++++++-----------
2 files changed, 166 insertions(+), 149 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/29d7cb37/.github/PULL_REQUEST_TEMPLATE.md
----------------------------------------------------------------------
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
index 56ca8f3..7d5ce2c 100644
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -40,5 +40,5 @@ In order to streamline the review of the contribution we ask you follow these gu
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
-It is also recommened that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
+It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/29d7cb37/metron-deployment/vagrant/Kerberos-setup.md
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/Kerberos-setup.md b/metron-deployment/vagrant/Kerberos-setup.md
index 1fce650..a66da8a 100644
--- a/metron-deployment/vagrant/Kerberos-setup.md
+++ b/metron-deployment/vagrant/Kerberos-setup.md
@@ -2,60 +2,59 @@
**Note:** These are manual instructions for Kerberizing Metron Storm topologies from Kafka to Kafka. This does not cover the Ambari MPack, sensor connections, or MAAS.
1. Build full dev and ssh into the machine
- ```
-cd incubator-metron/metron-deployment/vagrant/full-dev-platform
-vagrant up
-vagrant ssh
- ```
+ ```
+ cd incubator-metron/metron-deployment/vagrant/full-dev-platform
+ vagrant up
+ vagrant ssh
+ ```
2. Export env vars. Replace *node1* with the appropriate hosts if running anywhere other than full-dev Vagrant.
- ```
-# execute as root
-sudo su -
-export ZOOKEEPER=node1
-export BROKERLIST=node1
-export HDP_HOME="/usr/hdp/current"
-export METRON_VERSION="0.4.0"
-export METRON_HOME="/usr/metron/${METRON_VERSION}"
- ```
+ ```
+ # execute as root
+ sudo su -
+ export ZOOKEEPER=node1
+ export BROKERLIST=node1
+ export HDP_HOME="/usr/hdp/current"
+ export METRON_VERSION="0.4.0"
+ export METRON_HOME="/usr/metron/${METRON_VERSION}"
+ ```
3. Stop all topologies - we will restart them again once Kerberos has been enabled.
- ```
-for topology in bro snort enrichment indexing; do storm kill $topology; done
- ```
+ ```
+ for topology in bro snort enrichment indexing; do storm kill $topology; done
+ ```
4. Setup Kerberos
- ```
-# Note: if you copy/paste this full set of commands, the kdb5_util command will not run as expected, so run the commands individually to ensure they all execute
-# set 'node1' to the correct host for your kdc
-yum -y install krb5-server krb5-libs krb5-workstation
-sed -i 's/kerberos.example.com/node1/g' /etc/krb5.conf
-cp /etc/krb5.conf /var/lib/ambari-server/resources/scripts
-# This step takes a moment. It creates the kerberos database.
-kdb5_util create -s
-/etc/rc.d/init.d/krb5kdc start
-/etc/rc.d/init.d/kadmin start
-chkconfig krb5kdc on
-chkconfig kadmin on
- ```
+ ```
+ # Note: if you copy/paste this full set of commands, the kdb5_util command will not run as expected, so run the commands individually to ensure they all execute
+ # set 'node1' to the correct host for your kdc
+ yum -y install krb5-server krb5-libs krb5-workstation
+ sed -i 's/kerberos.example.com/node1/g' /etc/krb5.conf
+ /bin/cp -f /etc/krb5.conf /var/lib/ambari-server/resources/scripts
+ # This step takes a moment. It creates the kerberos database.
+ kdb5_util create -s
+ /etc/rc.d/init.d/krb5kdc start
+ /etc/rc.d/init.d/kadmin start
+ chkconfig krb5kdc on
+ chkconfig kadmin on
+ ```
5. Setup the admin and metron user principals. You'll kinit as the metron user when running topologies. Make sure to remember the passwords.
- ```
-kadmin.local -q "addprinc admin/admin"
-kadmin.local -q "addprinc metron"
- ```
+ ```
+ kadmin.local -q "addprinc admin/admin"
+ kadmin.local -q "addprinc metron"
+ ```
6. Create the metron user HDFS home directory
- ```
-sudo -u hdfs hdfs dfs -mkdir /user/metron && \
-sudo -u hdfs hdfs dfs -chown metron:hdfs /user/metron && \
-sudo -u hdfs hdfs dfs -chmod 770 /user/metron
- ```
+ ```
+ sudo -u hdfs hdfs dfs -mkdir /user/metron && \
+ sudo -u hdfs hdfs dfs -chown metron:hdfs /user/metron && \
+ sudo -u hdfs hdfs dfs -chmod 770 /user/metron
+ ```
-7. In Ambari, setup Storm to run with Kerberos and run worker jobs as the submitting user:
+7. In [Ambari](http://node1:8080), setup Storm to run with Kerberos and run worker jobs as the submitting user:
a. Add the following properties to custom storm-site:
-
```
topology.auto-credentials=['org.apache.storm.security.auth.kerberos.AutoTGT']
nimbus.credential.renewers.classes=['org.apache.storm.security.auth.kerberos.AutoTGT']
@@ -87,147 +86,159 @@ sudo -u hdfs hdfs dfs -chmod 770 /user/metron
9. Setup Metron keytab
- ```
-kadmin.local -q "ktadd -k metron.headless.keytab metron@EXAMPLE.COM" && \
-cp metron.headless.keytab /etc/security/keytabs && \
-chown metron:hadoop /etc/security/keytabs/metron.headless.keytab && \
-chmod 440 /etc/security/keytabs/metron.headless.keytab
- ```
+ ```
+ kadmin.local -q "ktadd -k metron.headless.keytab metron@EXAMPLE.COM" && \
+ cp metron.headless.keytab /etc/security/keytabs && \
+ chown metron:hadoop /etc/security/keytabs/metron.headless.keytab && \
+ chmod 440 /etc/security/keytabs/metron.headless.keytab
+ ```
10. Kinit with the metron user
- ```
-kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
- ```
+ ```
+ kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+ ```
11. First create any additional Kafka topics you will need. We need to create the topics before adding the required ACLs. The current full dev installation will deploy bro, snort, enrichments, and indexing only. e.g.
- ```
-${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper ${ZOOKEEPER}:2181 --create --topic yaf --partitions 1 --replication-factor 1
- ```
+ ```
+ ${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper ${ZOOKEEPER}:2181 --create --topic yaf --partitions 1 --replication-factor 1
+ ```
12. Setup Kafka ACLs for the topics
- ```
-export KERB_USER=metron;
-for topic in bro enrichments indexing snort; do
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --topic ${topic};
-done;
- ```
+ ```
+ export KERB_USER=metron
+ for topic in bro enrichments indexing snort; do
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --topic ${topic}
+ done
+ ```
13. Setup Kafka ACLs for the consumer groups
- ```
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group bro_parser;
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group snort_parser;
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group yaf_parser;
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group enrichments;
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group indexing;
- ```
+ ```
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group bro_parser
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group snort_parser
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group yaf_parser
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group enrichments
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group indexing
+ ```
14. Add metron user to the Kafka cluster ACL
- ```
-/usr/hdp/current/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --cluster kafka-cluster
- ```
+ ```
+ ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --cluster kafka-cluster
+ ```
15. We also need to grant permissions to the HBase tables. Kinit as the hbase user and add ACLs for metron.
- ```
-kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-metron_cluster@EXAMPLE.COM
-echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
-echo "grant 'metron', 'RW', 'enrichment'" | hbase shell
- ```
+ ```
+ kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-metron_cluster@EXAMPLE.COM
+ echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
+ echo "grant 'metron', 'RW', 'enrichment'" | hbase shell
+ ```
16. Create a \u201c.storm\u201d directory in the metron user\u2019s home directory and switch to that directory.
- ```
-su metron && cd ~/
-mkdir .storm
-cd .storm
- ```
+ ```
+ su metron
+ mkdir ~/.storm
+ cd ~/.storm
+ ```
17. Create a custom client jaas file. This should look identical to the Storm client jaas file located in /etc/storm/conf/client_jaas.conf except for the addition of a Client stanza. The Client stanza is used for Zookeeper. All quotes and semicolons are necessary.
- ```
-[metron@node1 .storm]$ cat client_jaas.conf
-StormClient {
- com.sun.security.auth.module.Krb5LoginModule required
- useTicketCache=true
- renewTicket=true
- serviceName="nimbus";
-};
-Client {
- com.sun.security.auth.module.Krb5LoginModule required
- useKeyTab=true
- keyTab="/etc/security/keytabs/metron.headless.keytab"
- storeKey=true
- useTicketCache=false
- serviceName="zookeeper"
- principal="metron@EXAMPLE.COM";
-};
-KafkaClient {
- com.sun.security.auth.module.Krb5LoginModule required
- useKeyTab=true
- keyTab="/etc/security/keytabs/metron.headless.keytab"
- storeKey=true
- useTicketCache=false
- serviceName="kafka"
- principal="metron@EXAMPLE.COM";
-};
- ```
+ ```
+ cat << EOF > client_jaas.conf
+ StormClient {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useTicketCache=true
+ renewTicket=true
+ serviceName="nimbus";
+ };
+ Client {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ keyTab="/etc/security/keytabs/metron.headless.keytab"
+ storeKey=true
+ useTicketCache=false
+ serviceName="zookeeper"
+ principal="metron@EXAMPLE.COM";
+ };
+ KafkaClient {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ keyTab="/etc/security/keytabs/metron.headless.keytab"
+ storeKey=true
+ useTicketCache=false
+ serviceName="kafka"
+ principal="metron@EXAMPLE.COM";
+ };
+ EOF
+ ```
18. Create a storm.yaml with jaas file info. Set the array of nimbus hosts accordingly.
- ```
-[metron@node1 .storm]$ cat storm.yaml
-nimbus.seeds : ['node1']
-java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
-storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
- ```
+ ```
+ cat << EOF > storm.yaml
+ nimbus.seeds : ['node1']
+ java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
+ storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
+ EOF
+ ```
19. Create an auxiliary storm configuration json file in the metron user\u2019s home directory. Note the login config option in the file points to our custom client_jaas.conf.
- ```
-cd /home/metron
-[metron@node1 ~]$ cat storm-config.json
-{
- "topology.worker.childopts" : "-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
-}
- ```
+ ```
+ cat << EOF > ~/storm-config.json
+ {
+ "topology.worker.childopts" : "-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
+ }
+ EOF
+ ```
20. Setup enrichment and indexing.
- a. Modify enrichment.properties - `${METRON_HOME}/config/enrichment.properties`
-
+ a. Modify enrichment.properties as root located at `${METRON_HOME}/config/enrichment.properties`
```
- kafka.security.protocol=PLAINTEXTSASL
- topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf
+ if [[ $EUID -ne 0 ]]; then
+ echo -e "\nERROR:\tYou must be root to run these commands. You may need to type exit."
+ else
+ sed -i 's/kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/' ${METRON_HOME}/config/enrichment.properties
+ sed -i 's/topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/' ${METRON_HOME}/config/enrichment.properties
+ fi
```
- b. Modify elasticsearch.properties - `${METRON_HOME}/config/elasticsearch.properties`
-
+ b. Modify elasticsearch.properties as root located at `${METRON_HOME}/config/elasticsearch.properties`
```
- kafka.security.protocol=PLAINTEXTSASL
- topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf
+ if [[ $EUID -ne 0 ]]; then
+ echo -e "\nERROR:\tYou must be root to run these commands. You may need to type exit."
+ else
+ sed -i 's/kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/' ${METRON_HOME}/config/elasticsearch.properties
+ sed -i 's/topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/' ${METRON_HOME}/config/elasticsearch.properties
+ fi
```
21. Kinit with the metron user again
- ```
-kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
- ```
+ ```
+ su metron
+ cd
+ kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+ ```
22. Restart the parser topologies. Be sure to pass in the new parameter, \u201c-ksp\u201d or \u201c--kafka_security_protocol.\u201d Run this from the metron home directory.
- ```
-for parser in bro snort; do ${METRON_HOME}/bin/start_parser_topology.sh -z ${ZOOKEEPER}:2181 -s ${parser} -ksp SASL_PLAINTEXT -e storm-config.json; done
- ```
+ ```
+ for parser in bro snort; do
+ ${METRON_HOME}/bin/start_parser_topology.sh -z ${ZOOKEEPER}:2181 -s ${parser} -ksp SASL_PLAINTEXT -e storm-config.json
+ done
+ ```
23. Now restart the enrichment and indexing topologies.
- ```
-${METRON_HOME}/bin/start_enrichment_topology.sh
-${METRON_HOME}/bin/start_elasticsearch_topology.sh
- ```
-
-24. Push some sample data to one of the parser topics. E.g for yaf we took raw data from [incubator-metron/metron-platform/metron-integration-test/src/main/sample/data/yaf/raw/YafExampleOutput](../../metron-platform/metron-integration-test/src/main/sample/data/yaf/raw/YafExampleOutput)
- ```
-cat sample-yaf.txt | ${HDP_HOME}/kafka-broker/bin/kafka-console-producer.sh --broker-list ${BROKERLIST}:6667 --security-protocol SASL_PLAINTEXT --topic yaf
- ```
-
-25. Wait a few moments for data to flow through the system and then check for data in the Elasticsearch indexes. Replace yaf with whichever parser type you\u2019ve chosen.
- ```
-curl -XGET "${ZOOKEEPER}:9200/yaf*/_search"
-curl -XGET "${ZOOKEEPER}:9200/yaf*/_count"
- ```
+ ```
+ ${METRON_HOME}/bin/start_enrichment_topology.sh
+ ${METRON_HOME}/bin/start_elasticsearch_topology.sh
+ ```
+
+24. Push some sample data to one of the parser topics. E.g for bro we took raw data from [incubator-metron/metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput](../../metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput)
+ ```
+ cat sample-bro.txt | ${HDP_HOME}/kafka-broker/bin/kafka-console-producer.sh --broker-list ${BROKERLIST}:6667 --security-protocol SASL_PLAINTEXT --topic bro
+ ```
+
+25. Wait a few moments for data to flow through the system and then check for data in the Elasticsearch indexes. Replace bro with whichever parser type you\u2019ve chosen.
+ ```
+ curl -XGET "${ZOOKEEPER}:9200/bro*/_search"
+ curl -XGET "${ZOOKEEPER}:9200/bro*/_count"
+ ```
26. You should have data flowing from the parsers all the way through to the indexes. This completes the Kerberization instructions
@@ -263,5 +274,11 @@ cat sample-yaf.txt | ${HDP_HOME}/kafka-broker/bin/kafka-console-producer.sh --br
${HDP_HOME}/kafka-broker/bin/kafka-console-consumer.sh --zookeeper ${ZOOKEEPER}:2181 --security-protocol PLAINTEXTSASL --topic yaf
```
+##### Modify the sensor-stubs to send logs via SASL
+```
+sed -i 's/node1:6667 --topic/node1:6667 --security-protocol PLAINTEXTSASL --topic/' /opt/sensor-stubs/bin/start-*-stub
+for sensorstub in bro snort; do service sensor-stubs stop $sensorstub; service sensor-stubs start $sensorstub; done
+```
+
#### References
* [https://github.com/apache/storm/blob/master/SECURITY.md](https://github.com/apache/storm/blob/master/SECURITY.md)
[04/26] incubator-metron git commit: METRON-848 MPack'
s metron_theme.json is malformed closes apache/incubator-metron#527
Posted by ma...@apache.org.
METRON-848 MPack's metron_theme.json is malformed closes apache/incubator-metron#527
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/2ecc55ba
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/2ecc55ba
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/2ecc55ba
Branch: refs/heads/Metron_0.4.0
Commit: 2ecc55ba17591b9216a6cf47d116c53608b9e59f
Parents: 84f336b
Author: David Lyle <dl...@gmail.com>
Authored: Tue Apr 18 18:43:47 2017 -0400
Committer: David Lyle <dl...@gmail.com>
Committed: Tue Apr 18 18:43:47 2017 -0400
----------------------------------------------------------------------
.../METRON/CURRENT/themes/metron_theme.json | 137 ++++++++-----------
1 file changed, 60 insertions(+), 77 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/2ecc55ba/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/themes/metron_theme.json
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/themes/metron_theme.json b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/themes/metron_theme.json
index 9d0f56c..20aca46 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/themes/metron_theme.json
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/themes/metron_theme.json
@@ -1,88 +1,71 @@
{
"configuration": {
- "layouts": [
- {
- "name": "default",
- "tabs": [
- {
- "name": "metron_general",
- "display-name": "Index Settings",
- "layout": {
- "tab-columns": "1",
- "tab-rows": "1",
- "sections": [
- {
- "name": "section-general",
- "row-index": "0",
- "column-index": "0",
- "row-span": "1",
- "column-span": "1",
- "section-columns": "1",
- "section-rows": "1",
- "subsections": [
- {
- "name": "subsection-general-indexing",
- "row-index": "0",
- "column-index": "0",
- "row-span": "1",
- "column-span": "1"
- }
- ]
- }
- ]
- }
- }
- ],
+ "layouts": [{
+ "name": "default",
+ "tabs": [{
+ "name": "metron_general",
+ "display-name": "Index Settings",
+ "layout": {
+ "tab-columns": "1",
+ "tab-rows": "1",
+ "sections": [{
+ "name": "section-general",
+ "row-index": "0",
+ "column-index": "0",
+ "row-span": "1",
+ "column-span": "1",
+ "section-columns": "1",
+ "section-rows": "1",
+ "subsections": [{
+ "name": "subsection-general-indexing",
+ "row-index": "0",
+ "column-index": "0",
+ "row-span": "1",
+ "column-span": "1"
+ }]
+ }]
+ }
+ }]
+ }],
"placement": {
"configuration-layout": "default",
- "configs": [
- {
- "config": "metron-env/metron_enrichment_db_port",
- "subsection-name": "subsection-general-database"
- },
- {
- "config": "metron-env/es_hosts",
- "subsection-name": "subsection-general-indexing"
- },
- {
- "config": "metron-env/es_binary_port",
- "subsection-name": "subsection-general-indexing"
- },
- {
- "config": "metron-env/es_http_port",
- "subsection-name": "subsection-general-indexing"
- },
- {
- "config": "metron-env/es_cluster_name",
- "subsection-name": "subsection-general-indexing"
- }
- ]
- },
- "widgets": [
- {
+ "configs": [{
+ "config": "metron-env/metron_enrichment_db_port",
+ "subsection-name": "subsection-general-database"
+ }, {
"config": "metron-env/es_hosts",
- "widget": {
- "type": "text-field"
- }
- },
- {
+ "subsection-name": "subsection-general-indexing"
+ }, {
"config": "metron-env/es_binary_port",
- "widget": {
- "type": "text-field"
- }
- },
- {
+ "subsection-name": "subsection-general-indexing"
+ }, {
"config": "metron-env/es_http_port",
- "widget": {
- "type": "text-field"
- }
- },
- {
+ "subsection-name": "subsection-general-indexing"
+ }, {
"config": "metron-env/es_cluster_name",
- "widget": {
- "type": "text-field"
- }
+ "subsection-name": "subsection-general-indexing"
+ }]
+ },
+ "widgets": [{
+ "config": "metron-env/es_hosts",
+ "widget": {
+ "type": "text-field"
+ }
+ }, {
+ "config": "metron-env/es_binary_port",
+ "widget": {
+ "type": "text-field"
+ }
+ }, {
+ "config": "metron-env/es_http_port",
+ "widget": {
+ "type": "text-field"
+ }
+ }, {
+ "config": "metron-env/es_cluster_name",
+ "widget": {
+ "type": "text-field"
}
- ]
+ }]
}
}
[17/26] incubator-metron git commit: METRON-835 Use Profiler with
Kerberos (nickwallen) closes apache/incubator-metron#521
Posted by ma...@apache.org.
METRON-835 Use Profiler with Kerberos (nickwallen) closes apache/incubator-metron#521
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/fbce3b5f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/fbce3b5f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/fbce3b5f
Branch: refs/heads/Metron_0.4.0
Commit: fbce3b5f9e396be997c315f39c6749d39b9105a3
Parents: 8c264f8
Author: nickwallen <ni...@nickallen.org>
Authored: Tue Apr 25 11:20:28 2017 -0400
Committer: nickallen <ni...@apache.org>
Committed: Tue Apr 25 11:20:28 2017 -0400
----------------------------------------------------------------------
metron-deployment/vagrant/Kerberos-setup.md | 397 ++++++++++++++++++++---
1 file changed, 345 insertions(+), 52 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/fbce3b5f/metron-deployment/vagrant/Kerberos-setup.md
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/Kerberos-setup.md b/metron-deployment/vagrant/Kerberos-setup.md
index f02cc5f..759d63d 100644
--- a/metron-deployment/vagrant/Kerberos-setup.md
+++ b/metron-deployment/vagrant/Kerberos-setup.md
@@ -1,48 +1,112 @@
-# Setting Up Kerberos in Vagrant Full Dev
-**Note:** These are instructions for Kerberizing Metron Storm topologies from Kafka to Kafka. This does not cover the sensor connections or MAAS.
-General Kerberization notes can be found in the metron-deployment [README.md](../README.md)
+Kerberos Setup
+==============
-## Setup the KDC
+This document provides instructions for kerberizing Metron's Vagrant-based development environments; "Quick Dev" and "Full Dev". These instructions do not cover the Ambari MPack or sensors. General Kerberization notes can be found in the metron-deployment [README.md](../README.md).
-1. Build full dev and ssh into the machine
- ```
- cd incubator-metron/metron-deployment/vagrant/full-dev-platform
- vagrant up
- vagrant ssh
- ```
+* [Setup](#setup)
+* [Create a KDC](#create-a-kdc)
+* [Enable Kerberos](#enable-kerberos)
+* [Kafka Authorization](#kafka-authorization)
+* [HBase Authorization](#hbase-authorization)
+* [Storm Authorization](#storm-authorization)
+* [Start Metron](#start-metron)
+
+Setup
+-----
+
+1. Deploy a Vagrant development environment; either [Full Dev](full-dev-platform) or [Quick Dev](quick-dev-platform).
+
+1. Export the following environment variables. These need to be set for the remainder of the instructions. Replace `node1` with the appropriate hosts, if you are running Metron anywhere other than Vagrant.
-2. Export env vars. Replace *node1* with the appropriate hosts if running anywhere other than full-dev Vagrant.
```
- # execute as root
- sudo su -
- export ZOOKEEPER=node1
- export BROKERLIST=node1
+ export ZOOKEEPER=node1:2181
+ export ELASTICSEARCH=node1:9200
+ export KAFKA=node1:6667
+
export HDP_HOME="/usr/hdp/current"
+ export KAFKA_HOME="${HDP_HOME}/kafka-broker"
export METRON_VERSION="0.4.0"
export METRON_HOME="/usr/metron/${METRON_VERSION}"
```
-3. Setup Kerberos
- ```
- # Note: if you copy/paste this full set of commands, the kdb5_util command will not run as expected, so run the commands individually to ensure they all execute
- # set 'node1' to the correct host for your kdc
- yum -y install krb5-server krb5-libs krb5-workstation
- sed -i 's/kerberos.example.com/node1/g' /etc/krb5.conf
- /bin/cp -f /etc/krb5.conf /var/lib/ambari-server/resources/scripts
- # This step takes a moment. It creates the kerberos database.
- kdb5_util create -s
- /etc/rc.d/init.d/krb5kdc start
- /etc/rc.d/init.d/kadmin start
- chkconfig krb5kdc on
- chkconfig kadmin on
- ```
+1. Execute the following commands as root.
+
+ ```
+ sudo su -
+ ```
+
+1. Stop all Metron topologies. They will be restarted again once Kerberos has been enabled.
+
+ ```
+ for topology in bro snort enrichment indexing; do
+ storm kill $topology;
+ done
+ ```
+
+1. Create the `metron` user's home directory in HDFS.
+
+ ```
+ sudo -u hdfs hdfs dfs -mkdir /user/metron
+ sudo -u hdfs hdfs dfs -chown metron:hdfs /user/metron
+ sudo -u hdfs hdfs dfs -chmod 770 /user/metron
+ ```
+
+Create a KDC
+------------
+
+1. Install dependencies.
+
+ ```
+ yum -y install krb5-server krb5-libs krb5-workstation
+ ```
+
+1. Define the host, `node1`, as the KDC.
+
+ ```
+ sed -i 's/kerberos.example.com/node1/g' /etc/krb5.conf
+ cp -f /etc/krb5.conf /var/lib/ambari-server/resources/scripts
+ ```
+
+1. Do not copy/paste this full set of commands as the `kdb5_util` command will not run as expected. Run the commands individually to ensure they all execute. This step takes a moment. It creates the kerberos database.
+
+ ```
+ kdb5_util create -s
+
+ /etc/rc.d/init.d/krb5kdc start
+ chkconfig krb5kdc on
+
+ /etc/rc.d/init.d/kadmin start
+ chkconfig kadmin on
+ ```
+
+1. Setup the `admin` and `metron` principals. You'll `kinit` as the `metron` principal when running topologies. Make sure to remember the passwords.
+
+ ```
+ kadmin.local -q "addprinc admin/admin"
+ kadmin.local -q "addprinc metron"
+ ```
+
+Enable Kerberos
+---------------
+
+1. In [Ambari](http://node1:8080), setup Storm to use Kerberos and run worker jobs as the submitting user.
+
+ a. Add the following properties to the custom storm-site:
-4. Setup the admin user principal. You'll kinit as the metron user when running topologies. Make sure to remember the password.
```
- kadmin.local -q "addprinc admin/admin"
+ topology.auto-credentials=['org.apache.storm.security.auth.kerberos.AutoTGT']
+ nimbus.credential.renewers.classes=['org.apache.storm.security.auth.kerberos.AutoTGT']
+ supervisor.run.worker.as.user=true
```
-## Ambari Setup
+ b. In the Storm config section in Ambari, choose \u201cAdd Property\u201d under custom storm-site:
+
+ 
+
+ c. In the dialog window, choose the \u201cbulk property add mode\u201d toggle button and add the below values:
+
+ 
+
1. Kerberize the cluster via Ambari. More detailed documentation can be found [here](http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html).
a. For this exercise, choose existing MIT KDC (this is what we setup and installed in the previous steps.)
@@ -55,35 +119,252 @@ General Kerberization notes can be found in the metron-deployment [README.md](..
- c. Click through to \u201cStart and Test Services.\u201d Let the cluster spin up.
+ c. Click through to \u201cStart and Test Services.\u201d Let the cluster spin up, but don't worry about starting up Metron via Ambari - we're going to run the parsers manually against the rest of the Hadoop cluster Kerberized. The wizard will fail at starting Metron, but this is OK. Click \u201ccontinue.\u201d When you\u2019re finished, the custom storm-site should look similar to the following:
+
+ 
+
+1. Create a Metron keytab
+
+ ```
+ kadmin.local -q "ktadd -k metron.headless.keytab metron@EXAMPLE.COM"
+ cp metron.headless.keytab /etc/security/keytabs
+ chown metron:hadoop /etc/security/keytabs/metron.headless.keytab
+ chmod 440 /etc/security/keytabs/metron.headless.keytab
+ ```
+
+Kafka Authorization
+-------------------
+
+1. Acquire a Kerberos ticket using the `metron` principal.
+
+ ```
+ kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+ ```
+
+1. Create any additional Kafka topics that you will need. We need to create the topics before adding the required ACLs. The current full dev installation will deploy bro, snort, enrichments, and indexing only. For example, you may want to add a topic for 'yaf' telemetry.
+
+ ```
+ ${KAFKA_HOME}/bin/kafka-topics.sh \
+ --zookeeper ${ZOOKEEPER} \
+ --create \
+ --topic yaf \
+ --partitions 1 \
+ --replication-factor 1
+ ```
+
+1. Setup Kafka ACLs for the `bro`, `snort`, `enrichments`, and `indexing` topics. Run the same command against any additional topics that you might be using; for example `yaf`.
+
+ ```
+ export KERB_USER=metron
+
+ for topic in bro snort enrichments indexing; do
+ ${KAFKA_HOME}/bin/kafka-acls.sh \
+ --authorizer kafka.security.auth.SimpleAclAuthorizer \
+ --authorizer-properties zookeeper.connect=${ZOOKEEPER} \
+ --add \
+ --allow-principal User:${KERB_USER} \
+ --topic ${topic}
+ done
+ ```
+
+1. Setup Kafka ACLs for the consumer groups. This command sets the ACLs for Bro, Snort, YAF, Enrichments, Indexing, and the Profiler. Execute the same command for any additional Parsers that you may be running.
+
+ ```
+ export KERB_USER=metron
+
+ for group in bro_parser snort_parser yaf_parser enrichments indexing profiler; do
+ ${KAFKA_HOME}/bin/kafka-acls.sh \
+ --authorizer kafka.security.auth.SimpleAclAuthorizer \
+ --authorizer-properties zookeeper.connect=${ZOOKEEPER} \
+ --add \
+ --allow-principal User:${KERB_USER} \
+ --group ${group}
+ done
+ ```
+
+1. Add the `metron` principal to the `kafka-cluster` ACL.
+
+ ```
+ ${KAFKA_HOME}/bin/kafka-acls.sh \
+ --authorizer kafka.security.auth.SimpleAclAuthorizer \
+ --authorizer-properties zookeeper.connect=${ZOOKEEPER} \
+ --add \
+ --allow-principal User:${KERB_USER} \
+ --cluster kafka-cluster
+ ```
+
+HBase Authorization
+-------------------
+
+1. Acquire a Kerberos ticket using the `hbase` principal
+
+ ```
+ kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-metron_cluster@EXAMPLE.COM
+ ```
+
+1. Grant permissions for the HBase tables used in Metron.
+
+ ```
+ echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
+ echo "grant 'metron', 'RW', 'enrichment'" | hbase shell
+ ```
+
+1. If you are using the Profiler, do the same for its HBase table.
+
+ ```
+ echo "create 'profiler', 'P'" | hbase shell
+ echo "grant 'metron', 'RW', 'profiler', 'P'" | hbase shell
+ ```
+
+Storm Authorization
+-------------------
+
+1. Switch to the `metron` user and acquire a Kerberos ticket for the `metron` principal.
+
+ ```
+ su metron
+ kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+ ```
+
+1. Create the directory `/home/metron/.storm` and switch to that directory.
-## Push Data
-1. Kinit with the metron user
```
- kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+ mkdir /home/metron/.storm
+ cd /home/metron/.storm
+ ```
+
+1. Create a client JAAS file at `/home/metron/.storm/client_jaas.conf`. This should look identical to the Storm client JAAS file located at `/etc/storm/conf/client_jaas.conf` except for the addition of a `Client` stanza. The `Client` stanza is used for Zookeeper. All quotes and semicolons are necessary.
+
```
+ cat << EOF > client_jaas.conf
+ StormClient {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useTicketCache=true
+ renewTicket=true
+ serviceName="nimbus";
+ };
+ Client {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ keyTab="/etc/security/keytabs/metron.headless.keytab"
+ storeKey=true
+ useTicketCache=false
+ serviceName="zookeeper"
+ principal="metron@EXAMPLE.COM";
+ };
+ KafkaClient {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ keyTab="/etc/security/keytabs/metron.headless.keytab"
+ storeKey=true
+ useTicketCache=false
+ serviceName="kafka"
+ principal="metron@EXAMPLE.COM";
+ };
+ EOF
+ ```
+
+1. Create a YAML file at `/home/metron/.storm/storm.yaml`. This should point to the client JAAS file. Set the array of nimbus hosts accordingly.
-2. Push some sample data to one of the parser topics. E.g for bro we took raw data from [incubator-metron/metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput](../../metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput)
```
- cat sample-bro.txt | ${HDP_HOME}/kafka-broker/bin/kafka-console-producer.sh --broker-list ${BROKERLIST}:6667 --security-protocol SASL_PLAINTEXT --topic bro
+ cat << EOF > /home/metron/.storm/storm.yaml
+ nimbus.seeds : ['node1']
+ java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
+ storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
+ EOF
```
-3. Wait a few moments for data to flow through the system and then check for data in the Elasticsearch indexes. Replace bro with whichever parser type you\u2019ve chosen.
+1. Create an auxiliary storm configuration file at `/home/metron/storm-config.json`. Note the login config option in the file points to the client JAAS file.
+
```
- curl -XGET "${ZOOKEEPER}:9200/bro*/_search"
- curl -XGET "${ZOOKEEPER}:9200/bro*/_count"
+ cat << EOF > /home/metron/storm-config.json
+ {
+ "topology.worker.childopts" : "-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
+ }
+ EOF
```
-4. You should have data flowing from the parsers all the way through to the indexes. This completes the Kerberization instructions
+1. Configure the Enrichment, Indexing and the Profiler topologies to use the client JAAS file. Add the following properties to each of the topology properties files.
+
+ ```
+ kafka.security.protocol=PLAINTEXTSASL
+ topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf
+ ```
+
+ * `${METRON_HOME}/config/enrichment.properties`
+ * `${METRON_HOME}/config/elasticsearch.properties`
+ * `${METRON_HOME}/config/profiler.properties`
+
+ Use the following command to automate this step.
+
+ ```
+ for file in enrichment.properties elasticsearch.properties profiler.properties; do
+ echo ${file}
+ sed -i "s/^kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/" "${METRON_HOME}/config/${file}"
+ sed -i "s/^topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/" "${METRON_HOME}/config/${file}"
+ done
+ ```
+
+Start Metron
+------------
+
+1. Switch to the `metron` user and acquire a Kerberos ticket for the `metron` principal.
+
+ ```
+ su metron
+ kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+ ```
+
+1. Restart the parser topologies. Be sure to pass in the new parameter, `-ksp` or `--kafka_security_protocol`. The following command will start only the Bro and Snort topologies. Execute the same command for any other Parsers that you may need, for example `yaf`.
+
+ ```
+ for parser in bro snort; do
+ ${METRON_HOME}/bin/start_parser_topology.sh \
+ -z ${ZOOKEEPER} \
+ -s ${parser} \
+ -ksp SASL_PLAINTEXT \
+ -e /home/metron/storm-config.json;
+ done
+ ```
+
+1. Restart the Enrichment and Indexing topologies.
+
+ ```
+ ${METRON_HOME}/bin/start_enrichment_topology.sh
+ ${METRON_HOME}/bin/start_elasticsearch_topology.sh
+ ```
+
+1. Push some sample data to one of the parser topics. E.g for Bro we took raw data from [incubator-metron/metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput](../../metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput)
+
+ ```
+ cat sample-bro.txt | ${KAFKA_HOME}/kafka-broker/bin/kafka-console-producer.sh \
+ --broker-list ${KAFKA} \
+ --security-protocol SASL_PLAINTEXT \
+ --topic bro
+ ```
+
+1. Wait a few moments for data to flow through the system and then check for data in the Elasticsearch indices. Replace yaf with whichever parser type you\u2019ve chosen.
+
+ ```
+ curl -XGET "${ELASTICSEARCH}/bro*/_search"
+ curl -XGET "${ELASTICSEARCH}/bro*/_count"
+ ```
+
+1. You should have data flowing from the parsers all the way through to the indexes. This completes the Kerberization instructions
+
+More Information
+----------------
+
+### Kerberos
-### Other useful commands
-#### Kerberos
Unsure of your Kerberos principal associated with a keytab? There are a couple ways to get this. One is via the list of principals that Ambari provides via downloadable csv. If you didn\u2019t download this list, you can also check the principal manually by running the following against the keytab.
+
```
klist -kt /etc/security/keytabs/<keytab-file-name>
```
E.g.
+
```
klist -kt /etc/security/keytabs/hbase.headless.keytab
Keytab name: FILE:/etc/security/keytabs/hbase.headless.keytab
@@ -96,23 +377,35 @@ KVNO Timestamp Principal
1 03/28/17 19:29:36 hbase-metron_cluster@EXAMPLE.COM
```
-#### Kafka with Kerberos enabled
+### Kafka with Kerberos enabled
+
+#### Write data to a topic with SASL
-##### Write data to a topic with SASL
```
-cat sample-yaf.txt | ${HDP_HOME}/kafka-broker/bin/kafka-console-producer.sh --broker-list ${BROKERLIST}:6667 --security-protocol PLAINTEXTSASL --topic yaf
+cat sample-yaf.txt | ${KAFKA_HOME}/bin/kafka-console-producer.sh \
+ --broker-list ${KAFKA} \
+ --security-protocol PLAINTEXTSASL \
+ --topic yaf
```
-##### View topic data from latest offset with SASL
+#### View topic data from latest offset with SASL
+
```
-${HDP_HOME}/kafka-broker/bin/kafka-console-consumer.sh --zookeeper ${ZOOKEEPER}:2181 --security-protocol PLAINTEXTSASL --topic yaf
+${KAFKA_HOME}/bin/kafka-console-consumer.sh \
+ --zookeeper ${ZOOKEEPER} \
+ --security-protocol PLAINTEXTSASL \
+ --topic yaf
```
-##### Modify the sensor-stubs to send logs via SASL
+#### Modify the sensor-stubs to send logs via SASL
```
sed -i 's/node1:6667 --topic/node1:6667 --security-protocol PLAINTEXTSASL --topic/' /opt/sensor-stubs/bin/start-*-stub
-for sensorstub in bro snort; do service sensor-stubs stop $sensorstub; service sensor-stubs start $sensorstub; done
+for sensorstub in bro snort; do
+ service sensor-stubs stop ${sensorstub};
+ service sensor-stubs start ${sensorstub};
+done
```
-#### References
+### References
+
* [https://github.com/apache/storm/blob/master/SECURITY.md](https://github.com/apache/storm/blob/master/SECURITY.md)
[12/26] incubator-metron git commit: METRON-867: In the event that we
graduate, remove incubating from the website and documentation closes
apache/incubator-metron#539
Posted by ma...@apache.org.
METRON-867: In the event that we graduate, remove incubating from the website and documentation closes apache/incubator-metron#539
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/cb44090b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/cb44090b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/cb44090b
Branch: refs/heads/Metron_0.4.0
Commit: cb44090b0a3f7df6950cf27d43ebb399d5005403
Parents: 41b5b10
Author: cstella <ce...@gmail.com>
Authored: Sun Apr 23 21:16:47 2017 -0400
Committer: cstella <ce...@gmail.com>
Committed: Sun Apr 23 21:16:47 2017 -0400
----------------------------------------------------------------------
.github/PULL_REQUEST_TEMPLATE.md | 2 +-
README.md | 6 +++---
metron-analytics/metron-statistics/HLLP.md | 2 +-
metron-analytics/pom.xml | 2 +-
metron-deployment/amazon-ec2/README.md | 2 +-
.../extra_modules/ambari_service_state.py | 2 +-
.../docker/rpm-docker/SPECS/metron.spec | 2 +-
.../roles/deployment-report/tasks/main.yml | 2 +-
metron-deployment/roles/monit/README.md | 2 +-
metron-interface/pom.xml | 2 +-
metron-platform/README.md | 2 +-
metron-platform/pom.xml | 2 +-
metron-sensors/bro-plugin-kafka/MAINTAINER | 2 +-
pom.xml | 2 +-
site-book/README.md | 2 +-
site-book/bin/generate-md.sh | 2 +-
site-book/pom.xml | 2 +-
.../src-resources/templates/site.xml.template | 16 ++++------------
site/_includes/footer.html | 6 ------
site/_includes/primary-nav-items.html | 3 ---
site/about/index.md | 4 ++--
site/community/index.md | 14 +++++++-------
site/develop/bylaws.md | 8 ++++----
site/develop/index.md | 10 +++++-----
site/documentation/index.md | 4 ++--
site/help/index.md | 6 +++---
site/img/metron_evolution.png | Bin 263188 -> 457377 bytes
site/index.html | 8 ++++----
site/js/top-form.js | 5 +----
site/news/index.html | 2 +-
30 files changed, 52 insertions(+), 72 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/cb44090b/.github/PULL_REQUEST_TEMPLATE.md
----------------------------------------------------------------------
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
index 7d5ce2c..6c1f2a5 100644
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -4,7 +4,7 @@
## Pull Request Checklist
-Thank you for submitting a contribution to Apache Metron (Incubating).
+Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/cb44090b/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 6e30209..09f72c3 100644
--- a/README.md
+++ b/README.md
@@ -1,6 +1,6 @@
[](https://travis-ci.org/apache/incubator-metron)
-# Apache Metron (Incubating)
+# Apache Metron
Metron integrates a variety of open source big data technologies in order
to offer a centralized tool for security monitoring and analysis. Metron
@@ -9,7 +9,7 @@ storage, advanced behavioral analytics and data enrichment, while applying
the most current threat intelligence information to security telemetry
within a single platform.
-For the latest information, please visit our website at http://metron.incubator.apache.org/
+For the latest information, please visit our website at http://metron.apache.org/
Metron can be divided into 4 areas:
@@ -52,7 +52,7 @@ enable rapid detection and rapid response for advanced security threats.
# Obtaining Metron
-To obtain a release of Metron, please visit http://metron.incubator.apache.org/documentation/#releases
+To obtain a release of Metron, please visit http://metron.apache.org/documentation/#releases
This repository is a collection of submodules for convenience which is regularly
updated to point to the latest versions. Github provides multiple ways to obtain
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/cb44090b/metron-analytics/metron-statistics/HLLP.md
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-statistics/HLLP.md b/metron-analytics/metron-statistics/HLLP.md
index a34e542..c645253 100644
--- a/metron-analytics/metron-statistics/HLLP.md
+++ b/metron-analytics/metron-statistics/HLLP.md
@@ -30,7 +30,7 @@ manipulated during the test: cardinality, sparse set precision, and normal set p
### Choosing p, sp
There is a tradeoff when choosing values for p and sp. A higher value for sp means a lower error rate for instances when the sparse set is being used. However, a higher value for sp also requires
more memory, which means the algorithm will cut over to the dense representation more quickly. The Google paper (listed above) recommends settings of p=25, sp=14 for optimal memory use and
-algorithm accuracy, and these are the defaults provided by the Apache Metron (Incubating) implementation in Stellar. You'll notice in the tests below, memory consumption tops out around 12KB
+algorithm accuracy, and these are the defaults provided by the Apache Metron implementation in Stellar. You'll notice in the tests below, memory consumption tops out around 12KB
for this setting, regardless of cardinality size.
### Key
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/cb44090b/metron-analytics/pom.xml
----------------------------------------------------------------------
diff --git a/metron-analytics/pom.xml b/metron-analytics/pom.xml
index cd01184..552de67 100644
--- a/metron-analytics/pom.xml
+++ b/metron-analytics/pom.xml
@@ -25,7 +25,7 @@
<version>0.4.0</version>
</parent>
<description>Stream analytics for Metron</description>
- <url>https://metron.incubator.apache.org/</url>
+ <url>https://metron.apache.org/</url>
<scm>
<connection>scm:git:https://git-wip-us.apache.org/repos/asf/incubator-metron.git</connection>
<developerConnection>scm:git:https://git-wip-us.apache.org/repos/asf/incubator-metron.git</developerConnection>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/cb44090b/metron-deployment/amazon-ec2/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/amazon-ec2/README.md b/metron-deployment/amazon-ec2/README.md
index 6ffc9fa..b28efdd 100644
--- a/metron-deployment/amazon-ec2/README.md
+++ b/metron-deployment/amazon-ec2/README.md
@@ -96,7 +96,7 @@ Having successfully created your Amazon Web Services account, hopefully you will
" Metron @ http://ec2-52-37-255-142.us-west-2.compute.amazonaws.com:5000",
" Ambari @ http://ec2-52-37-225-202.us-west-2.compute.amazonaws.com:8080",
" Sensors @ ec2-52-37-225-202.us-west-2.compute.amazonaws.com on tap0",
- "For additional information, see https://metron.incubator.apache.org/'"
+ "For additional information, see https://metron.apache.org/'"
]
}
```
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/cb44090b/metron-deployment/extra_modules/ambari_service_state.py
----------------------------------------------------------------------
diff --git a/metron-deployment/extra_modules/ambari_service_state.py b/metron-deployment/extra_modules/ambari_service_state.py
index 46d63f2..6d44e8d 100644
--- a/metron-deployment/extra_modules/ambari_service_state.py
+++ b/metron-deployment/extra_modules/ambari_service_state.py
@@ -20,7 +20,7 @@ DOCUMENTATION = '''
---
module: ambari_service_state
version_added: "2.1"
-author: Apache Metron (Incubating : https://github.com/apache/incubator-metron )
+author: Apache Metron (https://metron.apache.org)
short_description: Start/Stop/Change Service or Component State
description:
- Start/Stop/Change Service or Component State
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/cb44090b/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec b/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
index 593cf5c..b1ba637 100644
--- a/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
+++ b/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
@@ -19,7 +19,7 @@
%define full_version %{version}%{?_prerelease}
%define prerelease_fmt %{?_prerelease:.%{_prerelease}}
%define vendor_version %{?_vendor_version}%{!?_vendor_version: UNKNOWN}
-%define url http://metron.incubator.apache.org/
+%define url http://metron.apache.org/
%define base_name metron
%define name %{base_name}-%{vendor_version}
%define versioned_app_name %{base_name}-%{version}
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/cb44090b/metron-deployment/roles/deployment-report/tasks/main.yml
----------------------------------------------------------------------
diff --git a/metron-deployment/roles/deployment-report/tasks/main.yml b/metron-deployment/roles/deployment-report/tasks/main.yml
index de76805..ac84ee8 100644
--- a/metron-deployment/roles/deployment-report/tasks/main.yml
+++ b/metron-deployment/roles/deployment-report/tasks/main.yml
@@ -32,6 +32,6 @@
- " Sensor Status @ http://{{ groups.sensors[0] }}:2812"
- " Zookeeper @ {{ zookeeper_url }}"
- " Kafka @ {{ kafka_broker_url }}"
- - For additional information, see https://metron.incubator.apache.org/'
+ - For additional information, see https://metron.apache.org/'
- debug: var=Success
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/cb44090b/metron-deployment/roles/monit/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/roles/monit/README.md b/metron-deployment/roles/monit/README.md
index 3ce8717..bf52c5d 100644
--- a/metron-deployment/roles/monit/README.md
+++ b/metron-deployment/roles/monit/README.md
@@ -17,7 +17,7 @@ This role will leverage Monit as a process watchdog to manage sensors, topologie
" Ambari @ http://ec2-52-39-4-93.us-west-2.compute.amazonaws.com:8080",
" Sensor Status @ http://ec2-52-39-4-93.us-west-2.compute.amazonaws.com:2812",
" Topology Status @ http://ec2-52-39-130-62.us-west-2.compute.amazonaws.com:2812",
- "For additional information, see https://metron.incubator.apache.org/'"
+ "For additional information, see https://metron.apache.org/'"
]
}
```
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/cb44090b/metron-interface/pom.xml
----------------------------------------------------------------------
diff --git a/metron-interface/pom.xml b/metron-interface/pom.xml
index d664808..fa7e95b 100644
--- a/metron-interface/pom.xml
+++ b/metron-interface/pom.xml
@@ -24,7 +24,7 @@
<version>0.4.0</version>
</parent>
<description>Interfaces for Metron</description>
- <url>https://metron.incubator.apache.org/</url>
+ <url>https://metron.apache.org/</url>
<scm>
<connection>scm:git:https://git-wip-us.apache.org/repos/asf/incubator-metron.git</connection>
<developerConnection>scm:git:https://git-wip-us.apache.org/repos/asf/incubator-metron.git</developerConnection>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/cb44090b/metron-platform/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/README.md b/metron-platform/README.md
index db34025..650541f 100644
--- a/metron-platform/README.md
+++ b/metron-platform/README.md
@@ -23,7 +23,7 @@ We are still in the process of merging/porting additional features from our prod
# metron-platform
-Extensible set of Storm topologies and topology attributes for streaming, enriching, indexing, and storing telemetry in Hadoop. General information on Metron is available at https://metron.incubator.apache.org/
+Extensible set of Storm topologies and topology attributes for streaming, enriching, indexing, and storing telemetry in Hadoop. General information on Metron is available at https://metron.apache.org/
# Documentation
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/cb44090b/metron-platform/pom.xml
----------------------------------------------------------------------
diff --git a/metron-platform/pom.xml b/metron-platform/pom.xml
index 33969c5..68d9be8 100644
--- a/metron-platform/pom.xml
+++ b/metron-platform/pom.xml
@@ -25,7 +25,7 @@
<version>0.4.0</version>
</parent>
<description>Stream analytics for Metron</description>
- <url>https://metron.incubator.apache.org/</url>
+ <url>https://metron.apache.org/</url>
<scm>
<connection>scm:git:https://git-wip-us.apache.org/repos/asf/incubator-metron.git</connection>
<developerConnection>scm:git:https://git-wip-us.apache.org/repos/asf/incubator-metron.git</developerConnection>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/cb44090b/metron-sensors/bro-plugin-kafka/MAINTAINER
----------------------------------------------------------------------
diff --git a/metron-sensors/bro-plugin-kafka/MAINTAINER b/metron-sensors/bro-plugin-kafka/MAINTAINER
index 0ddaed7..bd46089 100644
--- a/metron-sensors/bro-plugin-kafka/MAINTAINER
+++ b/metron-sensors/bro-plugin-kafka/MAINTAINER
@@ -15,4 +15,4 @@
# limitations under the License.
#
-Apache Metron <us...@metron.incubator.apache.org>
+Apache Metron <us...@metron.apache.org>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/cb44090b/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index a86905e..96885d6 100644
--- a/pom.xml
+++ b/pom.xml
@@ -23,7 +23,7 @@
<packaging>pom</packaging>
<name>Metron</name>
<description>Metron Top Level Project</description>
- <url>https://metron.incubator.apache.org/</url>
+ <url>https://metron.apache.org/</url>
<modules>
<module>metron-analytics</module>
<module>metron-platform</module>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/cb44090b/site-book/README.md
----------------------------------------------------------------------
diff --git a/site-book/README.md b/site-book/README.md
index 93d3a31..d1060a0 100644
--- a/site-book/README.md
+++ b/site-book/README.md
@@ -6,7 +6,7 @@ Metron's Site Book is an attempt at producing documentation that is:
- Tied to code versions
- Highly local to the code being documented
-The idea is that a release manager would build the site-book (following the instructions below), then publish it from the public [Metron site](http://metron.incubator.apache.org/) as the docs for the new released version. Older site-book versions should remain available for users that need them.
+The idea is that a release manager would build the site-book (following the instructions below), then publish it from the public [Metron site](http://metron.apache.org/) as the docs for the new released version. Older site-book versions should remain available for users that need them.
To build the book, do the following:
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/cb44090b/site-book/bin/generate-md.sh
----------------------------------------------------------------------
diff --git a/site-book/bin/generate-md.sh b/site-book/bin/generate-md.sh
index 0859b7c..d428fbc 100755
--- a/site-book/bin/generate-md.sh
+++ b/site-book/bin/generate-md.sh
@@ -280,7 +280,7 @@ sed -n -e "1,/${BEGIN_TAG}/ p" "$TEMPLATES_DIR"/site.xml.template > ../site.xml
# Now start inserting menu tree items
# top level of markdown tree is special
if [ -e index.md ] ; then
- echo "<item name='Metron' href='index.html' title='Apache Metron - Incubating' collapse='false'>" >> ../site.xml
+ echo "<item name='Metron' href='index.html' title='Apache Metron' collapse='false'>" >> ../site.xml
item0_exists=1
else
item0_exists=0
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/cb44090b/site-book/pom.xml
----------------------------------------------------------------------
diff --git a/site-book/pom.xml b/site-book/pom.xml
index 2ca7046..2f04c71 100644
--- a/site-book/pom.xml
+++ b/site-book/pom.xml
@@ -25,7 +25,7 @@
<version>0.4.0</version>
</parent>
<description>User Documentation for Metron</description>
- <url>https://metron.incubator.apache.org/</url>
+ <url>https://metron.apache.org/</url>
<scm>
<connection>scm:git:https://git-wip-us.apache.org/repos/asf/incubator-metron.git</connection>
<developerConnection>scm:git:https://git-wip-us.apache.org/repos/asf/incubator-metron.git</developerConnection>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/cb44090b/site-book/src/site/src-resources/templates/site.xml.template
----------------------------------------------------------------------
diff --git a/site-book/src/site/src-resources/templates/site.xml.template b/site-book/src/site/src-resources/templates/site.xml.template
index 85a61e1..8523562 100644
--- a/site-book/src/site/src-resources/templates/site.xml.template
+++ b/site-book/src/site/src-resources/templates/site.xml.template
@@ -28,28 +28,20 @@
<custom>
<fluidoSkin>
- <project>Apache Metron - Incubating</project>
+ <project>Apache Metron</project>
<topBarEnabled>false</topBarEnabled>
<sideBarEnabled>true</sideBarEnabled>
</fluidoSkin>
</custom>
<bannerLeft>
- <name>Apache Metron - Incubating</name>
+ <name>Apache Metron</name>
<src>images/metron-logo.png</src>
- <href>http://metron.incubator.apache.org/</href>
+ <href>http://metron.apache.org/</href>
<width>148px</width>
<height>48px</height>
</bannerLeft>
- <bannerRight>
- <name>Apache Incubating</name>
- <src>images/ApacheIncubating_Logo.png</src>
- <href>http://incubator.apache.org/</href>
- <width>192px</width>
- <height>48px</height>
- </bannerRight>
-
<publishDate position="right"/>
<version position="right"/>
@@ -62,7 +54,7 @@
<breadcrumbs position="left">
<item name="Apache" href="http://www.apache.org"/>
- <item name="Metron-Incubating" title="Apache Metron - Incubating" href="http://metron.incubator.apache.org/"/>
+ <item name="Metron" title="Apache Metron" href="http://metron.apache.org/"/>
<item name="Documentation" title="Metron Docs" href="index.html"/>
</breadcrumbs>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/cb44090b/site/_includes/footer.html
----------------------------------------------------------------------
diff --git a/site/_includes/footer.html b/site/_includes/footer.html
index 8e375ca..55c47a9 100644
--- a/site/_includes/footer.html
+++ b/site/_includes/footer.html
@@ -1,12 +1,6 @@
</section>
<section class="intro-block darken" id="docshome">
- <div class="text-center">
- <img class="v-middle img82" src="/img/incubator_logo.png" alt="Apache" style="margin:0 auto;left: 0px;background: #FFFFFF">
- </div>
- <br>
<div class="content-960 hover-btn text-center disclaimer">
- <b>Disclaimer:</b> Apache Metron is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Hortonworks. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
- <br>
The contents of this website are � 2016 Apache Software Foundation under the terms of the Apache License v2. Apache Metron and its logo are trademarks of the Apache Software Foundation.
</div>
</section>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/cb44090b/site/_includes/primary-nav-items.html
----------------------------------------------------------------------
diff --git a/site/_includes/primary-nav-items.html b/site/_includes/primary-nav-items.html
index 764f276..2799976 100644
--- a/site/_includes/primary-nav-items.html
+++ b/site/_includes/primary-nav-items.html
@@ -12,7 +12,4 @@
<li>
<button class="button-default button-green" style="line-height:43px;"> <a>Download </a></button>
</li>
- <li class="apache-logo">
- <button id="apache-website" style="cursor: pointer;"> <img src="/img/incubator_logo.png" alt="Apache" style="height:43px;background: #FFFFFF;"> </button>
- </li>
</ul>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/cb44090b/site/about/index.md
----------------------------------------------------------------------
diff --git a/site/about/index.md b/site/about/index.md
index 69f644f..03901c6 100644
--- a/site/about/index.md
+++ b/site/about/index.md
@@ -1,6 +1,6 @@
---
layout: default
-title: Apache Metron (Incubating) About
+title: Apache Metron About
---
<section class="hero-second-level no-padding">
@@ -36,7 +36,7 @@ title: Apache Metron (Incubating) About
<h2>Evolution</h2>
</div>
<div>
- <img class="img82" src="/img/metron_evolution.png" alt="What Apache Metron (Incubating) Does" style="margin:0 auto;left: 0px;display:block;">
+ <img class="img82" src="/img/metron_evolution.png" alt="What Apache Metron Does" style="margin:0 auto;left: 0px;display:block;">
</div>
<div class="hover-btn text-center">
<a class="button-default" href="https://cwiki.apache.org/confluence/display/METRON/Evolution+of+Apache+Metron" target="new">LEARN MORE</a>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/cb44090b/site/community/index.md
----------------------------------------------------------------------
diff --git a/site/community/index.md b/site/community/index.md
index d1664c5..ac61708 100644
--- a/site/community/index.md
+++ b/site/community/index.md
@@ -1,6 +1,6 @@
---
layout: default
-title: Apache Metron (Incubating) Community
+title: Apache Metron Community
---
<section class="hero-second-level no-padding">
@@ -94,7 +94,7 @@ title: Apache Metron (Incubating) Community
<h2>Community Resources</h2>
</div>
<div class="content-960 hover-btn text-center">
- <p>Apache Metron (Incubating) provides user and development mailing lists, a Github repository, a JIRA site, and list of community members to enable you to learn more about Metron. Feel free to join any of the mailing lists and the Metron IRC channel, and view the code on Github.</p>
+ <p>Apache Metron provides user and development mailing lists, a Github repository, a JIRA site, and list of community members to enable you to learn more about Metron. Feel free to join any of the mailing lists and the Metron IRC channel, and view the code on Github.</p>
<a class="button-default" href="https://cwiki.apache.org/confluence/display/METRON/Community+Resources#CommunityResources-ApacheMetronCommunityResources" target="_blank">LEARN MORE</a>
</div>
</section>
@@ -104,7 +104,7 @@ title: Apache Metron (Incubating) Community
<h2>Get Involved</h2>
</div>
<div class="content-960 hover-btn text-center">
- <p> Do you want to contribute to Apache Metron (Incubating)? Follow these simple steps to subscribe to the user and development mailing lists, familiarize yourself with the Metron code, set up your development environment, and start contributing.</p>
+ <p> Do you want to contribute to Apache Metron? Follow these simple steps to subscribe to the user and development mailing lists, familiarize yourself with the Metron code, set up your development environment, and start contributing.</p>
<a class="button-default" href="https://cwiki.apache.org/confluence/display/METRON/Community+Resources#CommunityResources-JointheCommunity-GetingStarted" target="_blank">Take Me There</a>
</div>
</section>
@@ -115,12 +115,12 @@ title: Apache Metron (Incubating) Community
</div>
<div class="content-960 hover-btn text-center">
<h4> General & Public Discussion </h4>
- [<a href="mailto:user-subscribe@metron.incubator.apache.org">Subscribe</a>]
- [<a href="mailto:user-unsubscribe@metron.incubator.apache.org">Unsubscribe</a>]
+ [<a href="mailto:user-subscribe@metron.apache.org">Subscribe</a>]
+ [<a href="mailto:user-unsubscribe@metron.apache.org">Unsubscribe</a>]
[<a href="http://mail-archives.apache.org/mod_mbox/incubator-metron-user/">Archives</a>]
<h4> Code & Documentation Change </h4>
- [<a href="mailto:dev-subscribe@metron.incubator.apache.org">Subscribe</a>]
- [<a href="mailto:dev-unsubscribe@metron.incubator.apache.org">Unsubscribe</a>]
+ [<a href="mailto:dev-subscribe@metron.apache.org">Subscribe</a>]
+ [<a href="mailto:dev-unsubscribe@metron.apache.org">Unsubscribe</a>]
[<a href="http://mail-archives.apache.org/mod_mbox/incubator-metron-dev/">Archives</a>]
</div>
</section>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/cb44090b/site/develop/bylaws.md
----------------------------------------------------------------------
diff --git a/site/develop/bylaws.md b/site/develop/bylaws.md
index 9d81a1b..a8fc5fb 100644
--- a/site/develop/bylaws.md
+++ b/site/develop/bylaws.md
@@ -1,11 +1,11 @@
---
layout: page
-title: Apache Metron (Incubating) Bylaws
+title: Apache Metron Bylaws
---
## Introduction
-This document defines the bylaws under which the Apache Metron (Incubating) project
+This document defines the bylaws under which the Apache Metron project
operates. It defines the roles and responsibilities of the project,
who may vote, how voting works, how conflicts are resolved, etc.
@@ -32,8 +32,8 @@ it expects its members to follow. In particular:
Metron is typical of Apache projects in that it operates under a set of
principles, known collectively as the "Apache Way". If you are new to
-Apache development, please refer to the [Incubator
-project](http://incubator.apache.org/) for more
+Apache development, please refer to
+[this](http://www.apache.org/foundation/how-it-works.html) for more
information on how Apache projects operate.
## Roles and Responsibilities
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/cb44090b/site/develop/index.md
----------------------------------------------------------------------
diff --git a/site/develop/index.md b/site/develop/index.md
index d8378a6..ca917fa 100644
--- a/site/develop/index.md
+++ b/site/develop/index.md
@@ -29,16 +29,16 @@ George Vetticaden | gvetticaden | PMC
There are several development mailing lists for Metron
-* [dev@metron.incubator.apache.org](mailto:dev@metron.incubator.apache.org) - Development discussions
+* [dev@metron.apache.org](mailto:dev@metron.apache.org) - Development discussions
with archive [here](https://mail-archives.apache.org/mod_mbox/incubator-metron-dev/)
-* [issues@metron.incubator.apache.org](mailto:issues@metron.incubator.apache.org) - Bug tracking
+* [issues@metron.apache.org](mailto:issues@metron.apache.org) - Bug tracking
with archive [here](https://mail-archives.apache.org/mod_mbox/incubator-metron-issues/)
-* [commits@metron.incubator.apache.org](mailto:commits@metron.incubator.apache.org) - Git tracking
+* [commits@metron.apache.org](mailto:commits@metron.apache.org) - Git tracking
with archive [here](https://mail-archives.apache.org/mod_mbox/incubator-metron-commits/)
You can subscribe to the lists by sending email to
-*list*-subscribe@metron.incubator.apache.org and unsubscribe by sending email to
-*list*-unsubscribe@metron.incubator.apache.org.
+*list*-subscribe@metron.apache.org and unsubscribe by sending email to
+*list*-unsubscribe@metron.apache.org.
## Source code
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/cb44090b/site/documentation/index.md
----------------------------------------------------------------------
diff --git a/site/documentation/index.md b/site/documentation/index.md
index 28660c8..d41894f 100644
--- a/site/documentation/index.md
+++ b/site/documentation/index.md
@@ -1,6 +1,6 @@
---
layout: default
-title: Apache Metron (Incubating) Documentation
+title: Apache Metron Documentation
---
<section class="hero-second-level no-padding">
@@ -35,7 +35,7 @@ title: Apache Metron (Incubating) Documentation
<h2>Quick Start</h2>
</div>
<div class="content-960 hover-btn text-center">
- <p>The Quick Start installation fully automates the provisioning and deployment of Apache Metron (Incubating) and all necessary prerequisites on a single, virtualized host running on VirtualBox.</p>
+ <p>The Quick Start installation fully automates the provisioning and deployment of Apache Metron and all necessary prerequisites on a single, virtualized host running on VirtualBox.</p>
<br>
<p> This image is designed for quick deployment of a single node Metron cluster running on VirtualBox. This platform is ideal for use by Metron developers. It uses a base image that has been pre-loaded with Ambari and HDP.</p>
<a class="button-default" href="https://cwiki.apache.org/confluence/display/METRON/Quick+Start" target="_blank">LEARN MORE</a>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/cb44090b/site/help/index.md
----------------------------------------------------------------------
diff --git a/site/help/index.md b/site/help/index.md
index bea4306..c91f4e1 100644
--- a/site/help/index.md
+++ b/site/help/index.md
@@ -8,16 +8,16 @@ Need help with Metron? Try these resources.
## Mailing Lists
The best option is to send email to the user list
-[user@metron.incubator.apache.org](mailto:user@metron.incubator.apache.org), which is for user
+[user@metron.apache.org](mailto:user@metron.apache.org), which is for user
discussions. All of the historic traffic is available in the
[archive](http://mail-archives.apache.org/mod_mbox/incubator-metron-user/). To
subscribe to the user list, please send email to
-[user-subscribe@metron.incubator.apache.org](mailto:user-subscribe@metron.incubator.apache.org).
+[user-subscribe@metron.apache.org](mailto:user-subscribe@metron.apache.org).
## Bug Reports
Please file any issues you encounter or fixes you'd like on the
-[Apache Metron (Incubating) Jira](https://issues.apache.org/jira/browse/metron). We welcome
+[Apache Metron Jira](https://issues.apache.org/jira/browse/metron). We welcome
patches!
## StackOverflow
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/cb44090b/site/img/metron_evolution.png
----------------------------------------------------------------------
diff --git a/site/img/metron_evolution.png b/site/img/metron_evolution.png
index 78471bd..e11fb93 100644
Binary files a/site/img/metron_evolution.png and b/site/img/metron_evolution.png differ
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/cb44090b/site/index.html
----------------------------------------------------------------------
diff --git a/site/index.html b/site/index.html
index 23805fc..4800e4d 100644
--- a/site/index.html
+++ b/site/index.html
@@ -1,6 +1,6 @@
---
layout: default
-title: Apache Metron (Incubating) Big Data Security
+title: Apache Metron Big Data Security
overview: true
---
@@ -40,15 +40,15 @@ overview: true
<section class="hero-second-level no-padding" id="what">
<div class="bg-img">
- <img src="/img/metron_sky.png" alt="What Apache Metron (Incubating) Does" style="width: 100%; left: 0px;">
+ <img src="/img/metron_sky.png" alt="What Apache Metron Does" style="width: 100%; left: 0px;">
</div>
<div class="v-middle-wrapper">
<div class="v-middle-inner">
<div class="v-middle text-center">
- <h1>What Apache Metron (Incubating) Does </h1>
+ <h1>What Apache Metron Does </h1>
<br />
- <p>Apache Metron (Incubating) provides a scalable advanced security analytics framework built with the Hadoop Community evolving from the Cisco OpenSOC Project. A cyber security application framework that provides organizations the ability to detect cyber anomalies and enable organizations to rapidly respond to identified anomalies.</p>
+ <p>Apache Metron provides a scalable advanced security analytics framework built with the Hadoop Community evolving from the Cisco OpenSOC Project. A cyber security application framework that provides organizations the ability to detect cyber anomalies and enable organizations to rapidly respond to identified anomalies.</p>
<div class="hover-btn text-center">
<a class="button-default" href="https://cwiki.apache.org/confluence/display/METRON/About+Metron" target="_blank"> MORE</a>
</div>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/cb44090b/site/js/top-form.js
----------------------------------------------------------------------
diff --git a/site/js/top-form.js b/site/js/top-form.js
index bd4072b..f078db0 100644
--- a/site/js/top-form.js
+++ b/site/js/top-form.js
@@ -27,9 +27,6 @@ $(document).ready(function () {
window.location = window.location.origin + "/documentation/#releases";
});
- $('.main-menu #apache-website').on('click', function() {
- window.open("http://incubator.apache.org/", "_blank");
- });
/*
* Top-form UI/UX effects
@@ -149,4 +146,4 @@ $(document).ready(function () {
}
}).resize();
});
-});
\ No newline at end of file
+});
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/cb44090b/site/news/index.html
----------------------------------------------------------------------
diff --git a/site/news/index.html b/site/news/index.html
index 30365a7..9b28e72 100644
--- a/site/news/index.html
+++ b/site/news/index.html
@@ -1,6 +1,6 @@
---
layout: default
-title: Apache Metron (Incubating) Community
+title: Apache Metron Community
---
<section class="hero-second-level no-padding">
[08/26] incubator-metron git commit: METRON-634 Mpack bug fixes and
improvements (not related to singlenode install) (mattf-horton) closes
apache/incubator-metron#532
Posted by ma...@apache.org.
METRON-634 Mpack bug fixes and improvements (not related to singlenode install) (mattf-horton) closes apache/incubator-metron#532
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/df44f077
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/df44f077
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/df44f077
Branch: refs/heads/Metron_0.4.0
Commit: df44f077a97d0495fea511f5d9f900e4d3a25147
Parents: 29d7cb3
Author: mattf-horton <mf...@hortonworks.com>
Authored: Wed Apr 19 21:43:05 2017 -0700
Committer: mattf <ma...@apache.org>
Committed: Wed Apr 19 21:43:05 2017 -0700
----------------------------------------------------------------------
.../2.3.3/configuration/elastic-site.xml | 33 ++++++++-------
.../2.3.3/configuration/elastic-sysconfig.xml | 8 +++-
.../ELASTICSEARCH/2.3.3/metainfo.xml | 8 +++-
.../2.3.3/package/scripts/elastic.py | 6 +--
.../2.3.3/package/scripts/elastic_master.py | 8 ++--
.../2.3.3/package/scripts/elastic_slave.py | 8 ++--
.../2.3.3/package/scripts/params.py | 4 +-
.../2.3.3/quicklinks/quicklinks.json | 43 ++++++++++++++++++++
.../METRON/CURRENT/configuration/metron-env.xml | 2 +-
9 files changed, 87 insertions(+), 33 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/df44f077/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/configuration/elastic-site.xml
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/configuration/elastic-site.xml b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/configuration/elastic-site.xml
index cbcebdd..fd29e63 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/configuration/elastic-site.xml
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/configuration/elastic-site.xml
@@ -24,7 +24,7 @@
<property>
<name>cluster_name</name>
<value>metron</value>
- <description>Cluster name identifies your cluster</description>
+ <description>Elasticsearch Cluster Name identifies your Elasticsearch subsystem</description>
</property>
<property>
<name>masters_also_are_datanodes</name>
@@ -37,24 +37,24 @@
<property>
<name>zen_discovery_ping_unicast_hosts</name>
<!--Ideally this gets populated by the list of master eligible nodes (as an acceptable default). Unsure how to do this.-->
+ <!--Also need to document whether should list masters only, or all ES nodes. I think this one is all nodes, but previous inline comment said Masters.-->
<value></value>
- <description>Unicast discovery list of hosts to act as gossip routers, in comma separated format.</description>
+ <description>Unicast discovery list of hosts to act as gossip routers, in comma separated format: eshost1,eshost2</description>
</property>
<property>
<name>index_number_of_shards</name>
<value>4</value>
- <description>Set the number of shards (splits) of an index</description>
+ <description>Set the number of shards (splits) of an index. Changes are not effective after index creation. Usually set to 1 for single-node install.</description>
</property>
<property>
<name>index_number_of_replicas</name>
<value>2</value>
- <description>Set the number of replicas (additional copies) of an index</description>
+ <description>Set the number of replicas (copies in addition to the first) of an index. Usually set to 0 for single-node install.</description>
</property>
- <!-- Logging Configurations -->
<property>
<name>path_data</name>
<value>"/opt/lmm/es_data"</value>
- <description>Path to directory where to store index data allocated for this node. e.g. "/mnt/first", "/mnt/second"</description>
+ <description>Comma-separated list of directories where to store index data allocated for each node: "/mnt/first","/mnt/second". Number of paths should relate to number of shards, and preferably should be on separate physical volumes.</description>
</property>
<property>
<name>http_cors_enabled</name>
@@ -64,22 +64,21 @@
<type>string</type>
</value-attributes>
</property>
- <!-- Discovery -->
- <property>
- <name>transport_tcp_port</name>
- <value>9300-9400</value>
- <description>Set a custom port for the node to node communication</description>
- </property>
<property>
<name>http_port</name>
<value>9200-9300</value>
<description>Set a custom port to listen for HTTP traffic</description>
</property>
- <!-- Discovery -->
+ <property>
+ <name>transport_tcp_port</name>
+ <value>9300-9400</value>
+ <description>Set a custom port for the node to node communication</description>
+ </property>
+ <!-- Multi-node Discovery -->
<property>
<name>discovery_zen_ping_multicast_enabled</name>
<value>false</value>
- <description>master eligible nodes</description>
+ <description>Whether to use multicast</description>
</property>
<property>
<name>discovery_zen_ping_timeout</name>
@@ -190,7 +189,7 @@
</property>
<property>
<name>network_host</name>
- <value>_lo_,_eth0_</value>
- <description>Network interface(s) will bind to. </description>
+ <value>"_lo:ipv4_","_eth0:ipv4_"</value>
+ <description>Network interface(s) ES will bind to within each node. Confirm names via ifconfig. Should have entries for primary external and loopback interfaces, with :ipv4 annotation and quote marks around each entry: "_lo:ipv4_","_eth0:ipv4_"</description>
</property>
-</configuration>
\ No newline at end of file
+</configuration>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/df44f077/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/configuration/elastic-sysconfig.xml
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/configuration/elastic-sysconfig.xml b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/configuration/elastic-sysconfig.xml
index ae26498..44ecf05 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/configuration/elastic-sysconfig.xml
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/configuration/elastic-sysconfig.xml
@@ -58,7 +58,7 @@
<!--/etc/sysconfig/elasticsearch-->
<property>
<name>content</name>
- <description>This is the jinja template for elastic-env.sh file</description>
+ <description>This is the jinja template for elastic sysconfig file (/etc/sysconfig/elasticsearch)</description>
<value>
# Directory where the Elasticsearch binary distribution resides
ES_HOME={{elastic_home}}
@@ -89,6 +89,12 @@ CONF_DIR={{conf_dir}}
# This setting only works for the init script, but has to be configured separately for systemd startup
ES_USER={{elastic_user}}
+# Elasticsearch pid directory
+PID_DIR={{pid_dir}}
+
+# JAVA_HOME must be provided here for OS that use systemd service launch
+JAVA_HOME={{java64_home}}
+
# Additional Java OPTS
ES_JAVA_OPTS="-verbose:gc -Xloggc:{{log_dir}}/elasticsearch_gc.log -XX:-CMSConcurrentMTEnabled \
-XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps \
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/df44f077/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/metainfo.xml
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/metainfo.xml b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/metainfo.xml
index 53909a2..4373e38 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/metainfo.xml
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/metainfo.xml
@@ -68,6 +68,12 @@
<config-type>elastic-sysconfig</config-type>
</configuration-dependencies>
<restartRequiredAfterChange>true</restartRequiredAfterChange>
+ <quickLinksConfigurations>
+ <quickLinksConfiguration>
+ <fileName>quicklinks.json</fileName>
+ <default>true</default>
+ </quickLinksConfiguration>
+ </quickLinksConfigurations>
</service>
</services>
-</metainfo>
\ No newline at end of file
+</metainfo>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/df44f077/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/scripts/elastic.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/scripts/elastic.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/scripts/elastic.py
index 2e638de..c44d5ef 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/scripts/elastic.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/scripts/elastic.py
@@ -33,7 +33,7 @@ def elastic():
data_path[:] = [x.replace('"', '') for x in data_path]
directories = [params.log_dir, params.pid_dir, params.conf_dir]
- directories = directories + data_path
+ directories = directories + data_path + ["{0}/scripts".format(params.conf_dir)]
Directory(directories,
create_parents=True,
@@ -57,11 +57,11 @@ def elastic():
"elasticsearch.master.yaml.j2",
configurations=configurations),
owner=params.elastic_user,
- group=params.elastic_user
+ group=params.elastic_group
)
print "Master sysconfig: /etc/sysconfig/elasticsearch"
- File(format("/etc/sysconfig/elasticsearch"),
+ File("/etc/sysconfig/elasticsearch",
owner="root",
group="root",
content=InlineTemplate(params.sysconfig_template)
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/df44f077/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/scripts/elastic_master.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/scripts/elastic_master.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/scripts/elastic_master.py
index 706610e..3cd63c3 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/scripts/elastic_master.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/scripts/elastic_master.py
@@ -40,7 +40,7 @@ class Elasticsearch(Script):
def stop(self, env, upgrade_type=None):
import params
env.set_params(params)
- stop_cmd = format("service elasticsearch stop")
+ stop_cmd = "service elasticsearch stop"
print 'Stop the Master'
Execute(stop_cmd)
@@ -49,14 +49,14 @@ class Elasticsearch(Script):
env.set_params(params)
self.configure(env)
- start_cmd = format("service elasticsearch start")
+ start_cmd = "service elasticsearch start"
print 'Start the Master'
Execute(start_cmd)
def status(self, env):
import params
env.set_params(params)
- status_cmd = format("service elasticsearch status")
+ status_cmd = "service elasticsearch status"
print 'Status of the Master'
Execute(status_cmd)
@@ -64,7 +64,7 @@ class Elasticsearch(Script):
import params
env.set_params(params)
self.configure(env)
- restart_cmd = format("service elasticsearch restart")
+ restart_cmd = "service elasticsearch restart"
print 'Restarting the Master'
Execute(restart_cmd)
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/df44f077/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/scripts/elastic_slave.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/scripts/elastic_slave.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/scripts/elastic_slave.py
index 7f7f21a..633ddd9 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/scripts/elastic_slave.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/scripts/elastic_slave.py
@@ -39,7 +39,7 @@ class Elasticsearch(Script):
def stop(self, env, upgrade_type=None):
import params
env.set_params(params)
- stop_cmd = format("service elasticsearch stop")
+ stop_cmd = "service elasticsearch stop"
print 'Stop the Slave'
Execute(stop_cmd)
@@ -47,14 +47,14 @@ class Elasticsearch(Script):
import params
env.set_params(params)
self.configure(env)
- start_cmd = format("service elasticsearch start")
+ start_cmd = "service elasticsearch start"
print 'Start the Slave'
Execute(start_cmd)
def status(self, env):
import params
env.set_params(params)
- status_cmd = format("service elasticsearch status")
+ status_cmd = "service elasticsearch status"
print 'Status of the Slave'
Execute(status_cmd)
@@ -62,7 +62,7 @@ class Elasticsearch(Script):
import params
env.set_params(params)
self.configure(env)
- restart_cmd = format("service elasticsearch restart")
+ restart_cmd = "service elasticsearch restart"
print 'Restarting the Slave'
Execute(restart_cmd)
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/df44f077/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/scripts/params.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/scripts/params.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/scripts/params.py
index aea81fb..00929a7 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/scripts/params.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/scripts/params.py
@@ -35,8 +35,8 @@ max_map_count = config['configurations']['elastic-sysconfig']['max_map_count']
elastic_user = config['configurations']['elastic-env']['elastic_user']
elastic_group = config['configurations']['elastic-env']['elastic_group']
log_dir = config['configurations']['elastic-env']['elastic_log_dir']
-pid_dir = '/var/run/elasticsearch'
-pid_file = '/var/run/elasticsearch/elasticsearch.pid'
+pid_dir = config['configurations']['elastic-env']['elastic_pid_dir']
+
hostname = config['hostname']
java64_home = config['hostLevelParams']['java_home']
elastic_env_sh_template = config['configurations']['elastic-env']['content']
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/df44f077/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/quicklinks/quicklinks.json
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/quicklinks/quicklinks.json b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/quicklinks/quicklinks.json
new file mode 100644
index 0000000..909828b
--- /dev/null
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/quicklinks/quicklinks.json
@@ -0,0 +1,43 @@
+{
+ "name": "default",
+ "description": "default quick links configuration",
+ "configuration": {
+ "protocol":
+ {
+ "type":"HTTP_ONLY"
+ },
+
+ "links": [
+ {
+ "name": "es_health_link",
+ "label": "Elasticsearch Health",
+ "requires_user_name": "false",
+ "component_name": "ES_MASTER",
+ "url":"%@://%@:%@/_cat/health?v",
+ "port":{
+ "http_property": "http_port",
+ "http_default_port": "9200",
+ "https_property": "http_port",
+ "https_default_port": "9200",
+ "regex": "^(\\d+)",
+ "site": "elastic-site"
+ }
+ },
+ {
+ "name": "es_indices_link",
+ "label": "Elasticsearch Indexes",
+ "requires_user_name": "false",
+ "component_name": "ES_MASTER",
+ "url":"%@://%@:%@/_cat/indices?v",
+ "port":{
+ "http_property": "http_port",
+ "http_default_port": "9200",
+ "https_property": "http_port",
+ "https_default_port": "9200",
+ "regex": "^(\\d+)",
+ "site": "elastic-site"
+ }
+ }
+ ]
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/df44f077/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
index 90e0757..218e8e5 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
@@ -97,7 +97,7 @@
<property require-input="true">
<name>es_hosts</name>
<value></value>
- <description>Comma delimited list of Elasticsearch Hosts. (eshost1,eshost2)</description>
+ <description>Comma delimited list of Elasticsearch Master Hosts: eshost1,eshost2</description>
<display-name>Elasticsearch Hosts</display-name>
</property>
<property>
[25/26] incubator-metron git commit: METRON-870: Add filtering by
packet payload to the pcap query closes apache/incubator-metron#541
Posted by ma...@apache.org.
METRON-870: Add filtering by packet payload to the pcap query closes apache/incubator-metron#541
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/bf2528fd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/bf2528fd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/bf2528fd
Branch: refs/heads/Metron_0.4.0
Commit: bf2528fd3f4d474164ddf497459cd2421df3e4bb
Parents: 5dd8788
Author: cstella <ce...@gmail.com>
Authored: Thu Apr 27 12:20:45 2017 -0400
Committer: cstella <ce...@gmail.com>
Committed: Thu Apr 27 12:20:45 2017 -0400
----------------------------------------------------------------------
dependencies_with_url.csv | 1 +
.../pcapservice/PcapReceiverImplRestEasy.java | 24 ++-
.../PcapReceiverImplRestEasyTest.java | 85 ++++----
.../org/apache/metron/common/Constants.java | 7 +-
.../stellar/GeoEnrichmentFunctions.java | 2 +-
metron-platform/metron-pcap-backend/README.md | 29 ++-
.../metron/pcap/query/FixedCliConfig.java | 12 +-
.../metron/pcap/query/FixedCliParser.java | 15 +-
.../org/apache/metron/utils/PcapInspector.java | 11 +-
.../apache/metron/pcap/FixedPcapFilterTest.java | 211 ++++++++++---------
.../apache/metron/pcap/QueryPcapFilterTest.java | 121 +++++------
.../PcapTopologyIntegrationTest.java | 42 +++-
.../apache/metron/pcap/query/PcapCliTest.java | 47 +++--
metron-platform/metron-pcap/pom.xml | 19 ++
.../java/org/apache/metron/pcap/PacketInfo.java | 25 ++-
.../java/org/apache/metron/pcap/PcapHelper.java | 38 +++-
.../metron/pcap/filter/PcapFieldResolver.java | 8 +-
.../apache/metron/pcap/filter/PcapFilter.java | 2 +-
.../pcap/filter/fixed/FixedPcapFilter.java | 71 ++++++-
.../pcap/filter/query/QueryPcapFilter.java | 7 +-
.../pcap/pattern/ByteArrayMatcherFunction.java | 63 ++++++
.../pcap/pattern/ByteArrayMatchingUtil.java | 64 ++++++
.../pcap/filter/fixed/FixedPcapFilterTest.java | 22 +-
.../pcap/pattern/ByteArrayMatchingUtilTest.java | 133 ++++++++++++
24 files changed, 754 insertions(+), 305 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/bf2528fd/dependencies_with_url.csv
----------------------------------------------------------------------
diff --git a/dependencies_with_url.csv b/dependencies_with_url.csv
index 053f11a..93a19b7 100644
--- a/dependencies_with_url.csv
+++ b/dependencies_with_url.csv
@@ -293,5 +293,6 @@ org.atteo.classindex:classindex:jar:3.3:compile,ASLv2,https://github.com/atteo/c
com.squareup.okhttp:okhttp:jar:2.4.0:compile,ASLv2,https://github.com/square/okhttp
com.squareup.okio:okio:jar:1.4.0:compile,ASLv2,https://github.com/square/okhttp
org.htrace:htrace-core:jar:3.0.4:compile,ASLv2,http://htrace.incubator.apache.org/
+net.byteseek:byteseek:jar:2.0.3:compile,BSD,https://github.com/nishihatapalmer/byteseek
org.springframework.security.kerberos:spring-security-kerberos-client:jar:1.0.1.RELEASE:compile,ASLv2,https://github.com/spring-projects/spring-security-kerberos
org.springframework.security.kerberos:spring-security-kerberos-core:jar:1.0.1.RELEASE:compile,ASLv2,https://github.com/spring-projects/spring-security-kerberos
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/bf2528fd/metron-platform/metron-api/src/main/java/org/apache/metron/pcapservice/PcapReceiverImplRestEasy.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-api/src/main/java/org/apache/metron/pcapservice/PcapReceiverImplRestEasy.java b/metron-platform/metron-api/src/main/java/org/apache/metron/pcapservice/PcapReceiverImplRestEasy.java
index 1f3c03e..9c58813 100644
--- a/metron-platform/metron-api/src/main/java/org/apache/metron/pcapservice/PcapReceiverImplRestEasy.java
+++ b/metron-platform/metron-api/src/main/java/org/apache/metron/pcapservice/PcapReceiverImplRestEasy.java
@@ -27,6 +27,7 @@ import org.apache.log4j.Logger;
import org.apache.metron.common.Constants;
import org.apache.metron.common.hadoop.SequenceFileIterable;
import org.apache.metron.common.utils.timestamp.TimestampConverters;
+import org.apache.metron.pcap.PcapHelper;
import org.apache.metron.pcap.filter.fixed.FixedPcapFilter;
import org.apache.metron.pcap.filter.query.QueryPcapFilter;
import org.apache.metron.pcap.mr.PcapJob;
@@ -37,10 +38,7 @@ import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import java.io.IOException;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.EnumMap;
-import java.util.List;
+import java.util.*;
@Path("/")
public class PcapReceiverImplRestEasy {
@@ -197,6 +195,7 @@ public class PcapReceiverImplRestEasy {
@DefaultValue("-1") @QueryParam ("endTime")long endTime,
@DefaultValue("10") @QueryParam ("numReducers")int numReducers,
@DefaultValue("false") @QueryParam ("includeReverseTraffic") boolean includeReverseTraffic,
+ @DefaultValue("") @QueryParam ("packetFilter") String packetFilter,
@Context HttpServletResponse servlet_response)
throws IOException {
@@ -225,23 +224,26 @@ public class PcapReceiverImplRestEasy {
//convert to nanoseconds since the epoch
startTime = TimestampConverters.MILLISECONDS.toNanoseconds(startTime);
endTime = TimestampConverters.MILLISECONDS.toNanoseconds(endTime);
- EnumMap<Constants.Fields, String> query = new EnumMap<Constants.Fields, String>(Constants.Fields.class) {{
+ Map<String, String> query = new HashMap<String, String>() {{
if(srcIp != null) {
- put(Constants.Fields.SRC_ADDR, srcIp);
+ put(Constants.Fields.SRC_ADDR.getName(), srcIp);
}
if(dstIp != null) {
- put(Constants.Fields.DST_ADDR, dstIp);
+ put(Constants.Fields.DST_ADDR.getName(), dstIp);
}
if(srcPort != null) {
- put(Constants.Fields.SRC_PORT, srcPort);
+ put(Constants.Fields.SRC_PORT.getName(), srcPort);
}
if(dstPort != null) {
- put(Constants.Fields.DST_PORT, dstPort);
+ put(Constants.Fields.DST_PORT.getName(), dstPort);
}
if(protocol != null) {
- put(Constants.Fields.PROTOCOL, protocol);
+ put(Constants.Fields.PROTOCOL.getName(), protocol);
+ }
+ put(Constants.Fields.INCLUDES_REVERSE_TRAFFIC.getName(), "" + includeReverseTrafficF);
+ if(!org.apache.commons.lang3.StringUtils.isEmpty(packetFilter)) {
+ put(PcapHelper.PacketFields.PACKET_FILTER.getName(), packetFilter);
}
- put(Constants.Fields.INCLUDES_REVERSE_TRAFFIC, "" + includeReverseTrafficF);
}};
if(LOGGER.isDebugEnabled()) {
LOGGER.debug("Query received: " + Joiner.on(",").join(query.entrySet()));
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/bf2528fd/metron-platform/metron-api/src/test/java/org/apache/metron/pcapservice/PcapReceiverImplRestEasyTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-api/src/test/java/org/apache/metron/pcapservice/PcapReceiverImplRestEasyTest.java b/metron-platform/metron-api/src/test/java/org/apache/metron/pcapservice/PcapReceiverImplRestEasyTest.java
index dba87cf..ea6db70 100644
--- a/metron-platform/metron-api/src/test/java/org/apache/metron/pcapservice/PcapReceiverImplRestEasyTest.java
+++ b/metron-platform/metron-api/src/test/java/org/apache/metron/pcapservice/PcapReceiverImplRestEasyTest.java
@@ -24,6 +24,7 @@ import org.apache.hadoop.fs.Path;
import org.apache.metron.common.Constants;
import org.apache.metron.common.hadoop.SequenceFileIterable;
import org.apache.metron.common.utils.timestamp.TimestampConverters;
+import org.apache.metron.pcap.PcapHelper;
import org.apache.metron.pcap.filter.PcapFilterConfigurator;
import org.apache.metron.pcap.mr.PcapJob;
import org.junit.Assert;
@@ -32,6 +33,7 @@ import org.junit.Test;
import java.io.IOException;
import java.util.EnumMap;
+import java.util.Map;
public class PcapReceiverImplRestEasyTest {
@@ -65,7 +67,7 @@ public class PcapReceiverImplRestEasyTest {
}
}
- final MockQueryHandler<EnumMap<Constants.Fields, String>> fixedQueryHandler = new MockQueryHandler<EnumMap<Constants.Fields, String>>();
+ final MockQueryHandler<Map<String, String>> fixedQueryHandler = new MockQueryHandler<>();
final MockQueryHandler<String> queryQueryHandler = new MockQueryHandler<String>();
PcapReceiverImplRestEasy fixedRestEndpoint = new PcapReceiverImplRestEasy() {{
this.queryUtil = fixedQueryHandler;
@@ -89,32 +91,35 @@ public class PcapReceiverImplRestEasyTest {
String dstPort = "100";
long startTime = 100;
long endTime = 1000;
+ String query = "`blah`";
{
boolean includeReverseTraffic = false;
- fixedRestEndpoint.getPcapsByIdentifiers(srcIp, dstIp, protocol, srcPort, dstPort, startTime, endTime, 10, includeReverseTraffic, null);
+ fixedRestEndpoint.getPcapsByIdentifiers(srcIp, dstIp, protocol, srcPort, dstPort, startTime, endTime, 10, includeReverseTraffic, query, null);
Assert.assertEquals(new Path(ConfigurationUtil.getPcapOutputPath()), fixedQueryHandler.basePath);
Assert.assertEquals(new Path(ConfigurationUtil.getTempQueryOutputPath()), fixedQueryHandler.baseOutputPath);
- Assert.assertEquals(srcIp, fixedQueryHandler.fields.get(Constants.Fields.SRC_ADDR));
- Assert.assertEquals(dstIp, fixedQueryHandler.fields.get(Constants.Fields.DST_ADDR));
- Assert.assertEquals(srcPort, fixedQueryHandler.fields.get(Constants.Fields.SRC_PORT));
- Assert.assertEquals(dstPort, fixedQueryHandler.fields.get(Constants.Fields.DST_PORT));
+ Assert.assertEquals(srcIp, fixedQueryHandler.fields.get(Constants.Fields.SRC_ADDR.getName()));
+ Assert.assertEquals(dstIp, fixedQueryHandler.fields.get(Constants.Fields.DST_ADDR.getName()));
+ Assert.assertEquals(srcPort, fixedQueryHandler.fields.get(Constants.Fields.SRC_PORT.getName()));
+ Assert.assertEquals(dstPort, fixedQueryHandler.fields.get(Constants.Fields.DST_PORT.getName()));
Assert.assertEquals(TimestampConverters.MILLISECONDS.toNanoseconds(startTime), fixedQueryHandler.beginNS);
Assert.assertEquals(TimestampConverters.MILLISECONDS.toNanoseconds(endTime), fixedQueryHandler.endNS);
- Assert.assertEquals(includeReverseTraffic, Boolean.parseBoolean(fixedQueryHandler.fields.get(Constants.Fields.INCLUDES_REVERSE_TRAFFIC)));
+ Assert.assertEquals(includeReverseTraffic, Boolean.parseBoolean(fixedQueryHandler.fields.get(Constants.Fields.INCLUDES_REVERSE_TRAFFIC.getName())));
+ Assert.assertEquals(query, fixedQueryHandler.fields.get(PcapHelper.PacketFields.PACKET_FILTER.getName()));
}
{
boolean includeReverseTraffic = true;
- fixedRestEndpoint.getPcapsByIdentifiers(srcIp, dstIp, protocol, srcPort, dstPort, startTime, endTime, 10, includeReverseTraffic, null);
+ fixedRestEndpoint.getPcapsByIdentifiers(srcIp, dstIp, protocol, srcPort, dstPort, startTime, endTime, 10, includeReverseTraffic, query, null);
Assert.assertEquals(new Path(ConfigurationUtil.getPcapOutputPath()), fixedQueryHandler.basePath);
Assert.assertEquals(new Path(ConfigurationUtil.getTempQueryOutputPath()), fixedQueryHandler.baseOutputPath);
- Assert.assertEquals(srcIp, fixedQueryHandler.fields.get(Constants.Fields.SRC_ADDR));
- Assert.assertEquals(dstIp, fixedQueryHandler.fields.get(Constants.Fields.DST_ADDR));
- Assert.assertEquals(srcPort, fixedQueryHandler.fields.get(Constants.Fields.SRC_PORT));
- Assert.assertEquals(dstPort, fixedQueryHandler.fields.get(Constants.Fields.DST_PORT));
+ Assert.assertEquals(srcIp, fixedQueryHandler.fields.get(Constants.Fields.SRC_ADDR.getName()));
+ Assert.assertEquals(dstIp, fixedQueryHandler.fields.get(Constants.Fields.DST_ADDR.getName()));
+ Assert.assertEquals(srcPort, fixedQueryHandler.fields.get(Constants.Fields.SRC_PORT.getName()));
+ Assert.assertEquals(dstPort, fixedQueryHandler.fields.get(Constants.Fields.DST_PORT.getName()));
Assert.assertEquals(TimestampConverters.MILLISECONDS.toNanoseconds(startTime), fixedQueryHandler.beginNS);
Assert.assertEquals(TimestampConverters.MILLISECONDS.toNanoseconds(endTime), fixedQueryHandler.endNS);
- Assert.assertEquals(includeReverseTraffic, Boolean.parseBoolean(fixedQueryHandler.fields.get(Constants.Fields.INCLUDES_REVERSE_TRAFFIC)));
+ Assert.assertEquals(includeReverseTraffic, Boolean.parseBoolean(fixedQueryHandler.fields.get(Constants.Fields.INCLUDES_REVERSE_TRAFFIC.getName())));
+ Assert.assertEquals(query, fixedQueryHandler.fields.get(PcapHelper.PacketFields.PACKET_FILTER.getName()));
}
}
@@ -141,16 +146,18 @@ public class PcapReceiverImplRestEasyTest {
long startTime = 100;
long endTime = 1000;
boolean includeReverseTraffic = false;
- fixedRestEndpoint.getPcapsByIdentifiers(srcIp, dstIp, protocol, srcPort, dstPort, startTime, endTime, 10, includeReverseTraffic, null);
+ String query = "`metron`";
+ fixedRestEndpoint.getPcapsByIdentifiers(srcIp, dstIp, protocol, srcPort, dstPort, startTime, endTime, 10, includeReverseTraffic, query, null);
Assert.assertEquals(new Path(ConfigurationUtil.getPcapOutputPath()), fixedQueryHandler.basePath);
Assert.assertEquals(new Path(ConfigurationUtil.getTempQueryOutputPath()), fixedQueryHandler.baseOutputPath);
- Assert.assertEquals(srcIp, fixedQueryHandler.fields.get(Constants.Fields.SRC_ADDR));
- Assert.assertEquals(dstIp, fixedQueryHandler.fields.get(Constants.Fields.DST_ADDR));
- Assert.assertEquals(srcPort, fixedQueryHandler.fields.get(Constants.Fields.SRC_PORT));
- Assert.assertEquals(dstPort, fixedQueryHandler.fields.get(Constants.Fields.DST_PORT));
+ Assert.assertEquals(srcIp, fixedQueryHandler.fields.get(Constants.Fields.SRC_ADDR.getName()));
+ Assert.assertEquals(dstIp, fixedQueryHandler.fields.get(Constants.Fields.DST_ADDR.getName()));
+ Assert.assertEquals(srcPort, fixedQueryHandler.fields.get(Constants.Fields.SRC_PORT.getName()));
+ Assert.assertEquals(dstPort, fixedQueryHandler.fields.get(Constants.Fields.DST_PORT.getName()));
Assert.assertEquals(TimestampConverters.MILLISECONDS.toNanoseconds(startTime), fixedQueryHandler.beginNS);
Assert.assertEquals(TimestampConverters.MILLISECONDS.toNanoseconds(endTime), fixedQueryHandler.endNS);
- Assert.assertEquals(includeReverseTraffic, Boolean.parseBoolean(fixedQueryHandler.fields.get(Constants.Fields.INCLUDES_REVERSE_TRAFFIC)));
+ Assert.assertEquals(includeReverseTraffic, Boolean.parseBoolean(fixedQueryHandler.fields.get(Constants.Fields.INCLUDES_REVERSE_TRAFFIC.getName())));
+ Assert.assertEquals(query, fixedQueryHandler.fields.get(PcapHelper.PacketFields.PACKET_FILTER.getName()));
}
@Test
@@ -162,17 +169,19 @@ public class PcapReceiverImplRestEasyTest {
String dstPort = "100";
long startTime = 100;
long endTime = 1000;
+ String query = null;
boolean includeReverseTraffic = false;
- fixedRestEndpoint.getPcapsByIdentifiers(srcIp, dstIp, protocol, srcPort, dstPort, startTime, endTime, 10, includeReverseTraffic, null);
+ fixedRestEndpoint.getPcapsByIdentifiers(srcIp, dstIp, protocol, srcPort, dstPort, startTime, endTime, 10, includeReverseTraffic, query, null);
Assert.assertEquals(new Path(ConfigurationUtil.getPcapOutputPath()), fixedQueryHandler.basePath);
Assert.assertEquals(new Path(ConfigurationUtil.getTempQueryOutputPath()), fixedQueryHandler.baseOutputPath);
- Assert.assertEquals(srcIp, fixedQueryHandler.fields.get(Constants.Fields.SRC_ADDR));
- Assert.assertEquals(dstIp, fixedQueryHandler.fields.get(Constants.Fields.DST_ADDR));
- Assert.assertEquals(srcPort, fixedQueryHandler.fields.get(Constants.Fields.SRC_PORT));
- Assert.assertEquals(dstPort, fixedQueryHandler.fields.get(Constants.Fields.DST_PORT));
+ Assert.assertEquals(srcIp, fixedQueryHandler.fields.get(Constants.Fields.SRC_ADDR.getName()));
+ Assert.assertEquals(dstIp, fixedQueryHandler.fields.get(Constants.Fields.DST_ADDR.getName()));
+ Assert.assertEquals(srcPort, fixedQueryHandler.fields.get(Constants.Fields.SRC_PORT.getName()));
+ Assert.assertEquals(dstPort, fixedQueryHandler.fields.get(Constants.Fields.DST_PORT.getName()));
Assert.assertEquals(TimestampConverters.MILLISECONDS.toNanoseconds(startTime), fixedQueryHandler.beginNS);
Assert.assertEquals(TimestampConverters.MILLISECONDS.toNanoseconds(endTime), fixedQueryHandler.endNS);
- Assert.assertEquals(includeReverseTraffic, Boolean.parseBoolean(fixedQueryHandler.fields.get(Constants.Fields.INCLUDES_REVERSE_TRAFFIC)));
+ Assert.assertEquals(includeReverseTraffic, Boolean.parseBoolean(fixedQueryHandler.fields.get(Constants.Fields.INCLUDES_REVERSE_TRAFFIC.getName())));
+ Assert.assertEquals(query, fixedQueryHandler.fields.get(PcapHelper.PacketFields.PACKET_FILTER.getName()));
}
@Test
@@ -185,17 +194,19 @@ public class PcapReceiverImplRestEasyTest {
long startTime = -1;
long endTime = 1000;
{
+ String query = "";
boolean includeReverseTraffic = false;
- fixedRestEndpoint.getPcapsByIdentifiers(srcIp, dstIp, protocol, srcPort, dstPort, startTime, endTime, 10, includeReverseTraffic, null);
+ fixedRestEndpoint.getPcapsByIdentifiers(srcIp, dstIp, protocol, srcPort, dstPort, startTime, endTime, 10, includeReverseTraffic, query, null);
Assert.assertEquals(new Path(ConfigurationUtil.getPcapOutputPath()), fixedQueryHandler.basePath);
Assert.assertEquals(new Path(ConfigurationUtil.getTempQueryOutputPath()), fixedQueryHandler.baseOutputPath);
- Assert.assertEquals(srcIp, fixedQueryHandler.fields.get(Constants.Fields.SRC_ADDR));
- Assert.assertEquals(dstIp, fixedQueryHandler.fields.get(Constants.Fields.DST_ADDR));
- Assert.assertEquals(srcPort, fixedQueryHandler.fields.get(Constants.Fields.SRC_PORT));
- Assert.assertEquals(dstPort, fixedQueryHandler.fields.get(Constants.Fields.DST_PORT));
+ Assert.assertEquals(srcIp, fixedQueryHandler.fields.get(Constants.Fields.SRC_ADDR.getName()));
+ Assert.assertEquals(dstIp, fixedQueryHandler.fields.get(Constants.Fields.DST_ADDR.getName()));
+ Assert.assertEquals(srcPort, fixedQueryHandler.fields.get(Constants.Fields.SRC_PORT.getName() ));
+ Assert.assertEquals(dstPort, fixedQueryHandler.fields.get(Constants.Fields.DST_PORT.getName()));
Assert.assertEquals(0, fixedQueryHandler.beginNS);
Assert.assertEquals(TimestampConverters.MILLISECONDS.toNanoseconds(endTime), fixedQueryHandler.endNS);
- Assert.assertEquals(includeReverseTraffic, Boolean.parseBoolean(fixedQueryHandler.fields.get(Constants.Fields.INCLUDES_REVERSE_TRAFFIC)));
+ Assert.assertEquals(includeReverseTraffic, Boolean.parseBoolean(fixedQueryHandler.fields.get(Constants.Fields.INCLUDES_REVERSE_TRAFFIC.getName())));
+ Assert.assertEquals(null, fixedQueryHandler.fields.get(PcapHelper.PacketFields.PACKET_FILTER.getName()));
}
{
String query = "ip_src_addr == 'srcIp' and ip_src_port == '80' and ip_dst_addr == 'dstIp' and ip_dst_port == '100' and protocol == 'protocol'";
@@ -219,16 +230,16 @@ public class PcapReceiverImplRestEasyTest {
long endTime = -1;
{
boolean includeReverseTraffic = false;
- fixedRestEndpoint.getPcapsByIdentifiers(srcIp, dstIp, protocol, srcPort, dstPort, startTime, endTime, 10, includeReverseTraffic, null);
+ fixedRestEndpoint.getPcapsByIdentifiers(srcIp, dstIp, protocol, srcPort, dstPort, startTime, endTime, 10, includeReverseTraffic, null, null);
Assert.assertEquals(new Path(ConfigurationUtil.getPcapOutputPath()), fixedQueryHandler.basePath);
Assert.assertEquals(new Path(ConfigurationUtil.getTempQueryOutputPath()), fixedQueryHandler.baseOutputPath);
- Assert.assertEquals(srcIp, fixedQueryHandler.fields.get(Constants.Fields.SRC_ADDR));
- Assert.assertEquals(dstIp, fixedQueryHandler.fields.get(Constants.Fields.DST_ADDR));
- Assert.assertEquals(srcPort, fixedQueryHandler.fields.get(Constants.Fields.SRC_PORT));
- Assert.assertEquals(dstPort, fixedQueryHandler.fields.get(Constants.Fields.DST_PORT));
+ Assert.assertEquals(srcIp, fixedQueryHandler.fields.get(Constants.Fields.SRC_ADDR.getName()));
+ Assert.assertEquals(dstIp, fixedQueryHandler.fields.get(Constants.Fields.DST_ADDR.getName()));
+ Assert.assertEquals(srcPort, fixedQueryHandler.fields.get(Constants.Fields.SRC_PORT.getName()));
+ Assert.assertEquals(dstPort, fixedQueryHandler.fields.get(Constants.Fields.DST_PORT.getName()));
Assert.assertEquals(0, fixedQueryHandler.beginNS);
Assert.assertTrue(fixedQueryHandler.endNS > 0);
- Assert.assertEquals(includeReverseTraffic, Boolean.parseBoolean(fixedQueryHandler.fields.get(Constants.Fields.INCLUDES_REVERSE_TRAFFIC)));
+ Assert.assertEquals(includeReverseTraffic, Boolean.parseBoolean(fixedQueryHandler.fields.get(Constants.Fields.INCLUDES_REVERSE_TRAFFIC.getName())));
}
{
String query = "ip_src_addr == 'srcIp' and ip_src_port == '80' and ip_dst_addr == 'dstIp' and ip_dst_port == '100' and protocol == 'protocol'";
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/bf2528fd/metron-platform/metron-common/src/main/java/org/apache/metron/common/Constants.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/Constants.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/Constants.java
index 1dc73da..c2ede49 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/Constants.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/Constants.java
@@ -17,7 +17,9 @@
*/
package org.apache.metron.common;
+import java.util.ArrayList;
import java.util.HashMap;
+import java.util.List;
import java.util.Map;
public class Constants {
@@ -33,7 +35,10 @@ public class Constants {
public static final String SIMPLE_HBASE_THREAT_INTEL = "hbaseThreatIntel";
public static final String GUID = "guid";
- public static enum Fields {
+ public interface Field {
+ String getName();
+ }
+ public enum Fields implements Field {
SRC_ADDR("ip_src_addr")
,SRC_PORT("ip_src_port")
,DST_ADDR("ip_dst_addr")
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/bf2528fd/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/stellar/GeoEnrichmentFunctions.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/stellar/GeoEnrichmentFunctions.java b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/stellar/GeoEnrichmentFunctions.java
index 11e024e..42913b2 100644
--- a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/stellar/GeoEnrichmentFunctions.java
+++ b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/stellar/GeoEnrichmentFunctions.java
@@ -36,7 +36,7 @@ public class GeoEnrichmentFunctions {
,namespace="GEO"
,description="Look up an IPV4 address and returns geographic information about it"
,params = {
- "ip - The IPV4 address to lookup" +
+ "ip - The IPV4 address to lookup",
"fields - Optional list of GeoIP fields to grab. Options are locID, country, city, postalCode, dmaCode, latitude, longitude, location_point"
}
,returns = "If a Single field is requested a string of the field, If multiple fields a map of string of the fields, and null otherwise"
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/bf2528fd/metron-platform/metron-pcap-backend/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/README.md b/metron-platform/metron-pcap-backend/README.md
index e1a2683..5b554b5 100644
--- a/metron-platform/metron-pcap-backend/README.md
+++ b/metron-platform/metron-pcap-backend/README.md
@@ -44,11 +44,13 @@ sequence files.
## Configuration
The configuration file for the Flux topology is located at
-`$METRON_HOME/config/etc/env/pcap.properties` and the possible options
+`$METRON_HOME/config/pcap.properties` and the possible options
are as follows:
* `spout.kafka.topic.pcap` : The kafka topic to listen to
+* `storm.auto.credentials` : The kerberos ticket renewal. If running on a kerberized cluster, this should be `['org.apache.storm.security.auth.kerberos.AutoTGT']`
+* `kafka.security.protocol` : The security protocol to use for kafka. This should be `PLAINTEXT` for a non-kerberized cluster and probably `SASL_PLAINTEXT` for a kerberized cluster.
* `kafka.zk` : The comma separated zookeeper quorum (i.e. host:2181,host2:2181)
-* `kafka.pcap.start` : One of `START`, `END`, `WHERE_I_LEFT_OFF` representing where to start listening on the queue.
+* `kafka.pcap.start` : One of `EARLIEST`, `LATEST`, `UNCOMMITTED_EARLIEST`, `UNCOMMITTED_LATEST` representing where to start listening on the queue.
* `kafka.pcap.numPackets` : The number of packets to keep in one file.
* `kafka.pcap.maxTimeMS` : The number of packets to keep in one file in terms of duration (in milliseconds). For instance, you may only want to keep an hour's worth of packets in a given file.
* `kafka.pcap.ts_scheme` : One of `FROM_KEY` or `FROM_VALUE`. You really only want `FROM_KEY` as that fits the current tooling. `FROM_VALUE` assumes that fully headerized packets are coming in on the value, which is legacy.
@@ -78,7 +80,7 @@ usage: PcapInspector
### Query Filter Utility
This tool exposes the two methods for filtering PCAP data via a command line tool:
- fixed
-- query (Metron Stellar)
+- query (via Stellar)
The tool is executed via
```
@@ -97,6 +99,7 @@ usage: Fixed filter options
and end_time. Default is to use time in
millis since the epoch.
-dp,--ip_dst_port <arg> Destination port
+ -pf,--packet_filter <arg> Packet filter regex
-et,--end_time <arg> Packet end time range. Default is current
system time.
-nr,--num_reducers <arg> The number of reducers to use. Default
@@ -127,3 +130,23 @@ usage: Query filter options
-q,--query <arg> Query string to use as a filter
-st,--start_time <arg> (required) Packet start time range.
```
+
+The Query filter's `--query` argument specifies the Stellar expression to
+execute on each packet. To interact with the packet, a few variables are exposed:
+* `packet` : The packet data (a `byte[]`)
+* `ip_src_addr` : The source address for the packet (a `String`)
+* `ip_src_port` : The source port for the packet (an `Integer`)
+* `ip_dst_addr` : The destination address for the packet (a `String`)
+* `ip_dst_port` : The destination port for the packet (an `Integer`)
+
+#### Binary Regex
+
+Filtering can be done both by the packet header as well as via a binary regular expression
+which can be run on the packet payload itself. This filter can be specified via:
+* The `-pf` or `--packet_filter` options for the fixed query filter
+* The `BYTEARRAY_MATCHER(pattern, data)` Stellar function.
+The first argument is the regex pattern and the second argument is the data.
+The packet data will be exposed via the`packet` variable in Stellar.
+
+The format of this regular expression is described [here](https://github.com/nishihatapalmer/byteseek/blob/master/sequencesyntax.md).
+
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/bf2528fd/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/FixedCliConfig.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/FixedCliConfig.java b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/FixedCliConfig.java
index 897e0fd..df653e1 100644
--- a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/FixedCliConfig.java
+++ b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/FixedCliConfig.java
@@ -20,24 +20,26 @@ package org.apache.metron.pcap.query;
import org.apache.metron.common.Constants;
import java.util.EnumMap;
+import java.util.LinkedHashMap;
+import java.util.Map;
public class FixedCliConfig extends CliConfig {
- private EnumMap<Constants.Fields, String> fixedFields;
+ private Map<String, String> fixedFields;
public FixedCliConfig() {
- this.fixedFields = new EnumMap<>(Constants.Fields.class);
+ this.fixedFields = new LinkedHashMap<>();
}
- public EnumMap<Constants.Fields, String> getFixedFields() {
+ public Map<String, String> getFixedFields() {
return fixedFields;
}
- public void setFixedFields(EnumMap<Constants.Fields, String> fixedFields) {
+ public void setFixedFields(Map<String, String> fixedFields) {
this.fixedFields = fixedFields;
}
- public void putFixedField(Constants.Fields key, String value) {
+ public void putFixedField(String key, String value) {
String trimmedVal = value != null ? value.trim() : null;
if (!isNullOrEmpty(trimmedVal)) {
this.fixedFields.put(key, value);
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/bf2528fd/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/FixedCliParser.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/FixedCliParser.java b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/FixedCliParser.java
index 1123cad..fda8692 100644
--- a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/FixedCliParser.java
+++ b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/FixedCliParser.java
@@ -21,6 +21,7 @@ import org.apache.commons.cli.CommandLine;
import org.apache.commons.cli.Options;
import org.apache.commons.cli.ParseException;
import org.apache.metron.common.Constants;
+import org.apache.metron.pcap.PcapHelper;
public class FixedCliParser extends CliParser {
private Options fixedOptions;
@@ -36,6 +37,7 @@ public class FixedCliParser extends CliParser {
options.addOption(newOption("sp", "ip_src_port", true, "Source port"));
options.addOption(newOption("dp", "ip_dst_port", true, "Destination port"));
options.addOption(newOption("p", "protocol", true, "IP Protocol"));
+ options.addOption(newOption("pf", "packet_filter", true, "Packet Filter regex"));
options.addOption(newOption("ir", "include_reverse", false, "Indicates if filter should check swapped src/dest addresses and IPs"));
return options;
}
@@ -51,12 +53,13 @@ public class FixedCliParser extends CliParser {
CommandLine commandLine = getParser().parse(fixedOptions, args);
FixedCliConfig config = new FixedCliConfig();
super.parse(commandLine, config);
- config.putFixedField(Constants.Fields.SRC_ADDR, commandLine.getOptionValue("ip_src_addr"));
- config.putFixedField(Constants.Fields.DST_ADDR, commandLine.getOptionValue("ip_dst_addr"));
- config.putFixedField(Constants.Fields.SRC_PORT, commandLine.getOptionValue("ip_src_port"));
- config.putFixedField(Constants.Fields.DST_PORT, commandLine.getOptionValue("ip_dst_port"));
- config.putFixedField(Constants.Fields.PROTOCOL, commandLine.getOptionValue("protocol"));
- config.putFixedField(Constants.Fields.INCLUDES_REVERSE_TRAFFIC, Boolean.toString(commandLine.hasOption("include_reverse")));
+ config.putFixedField(Constants.Fields.SRC_ADDR.getName(), commandLine.getOptionValue("ip_src_addr"));
+ config.putFixedField(Constants.Fields.DST_ADDR.getName(), commandLine.getOptionValue("ip_dst_addr"));
+ config.putFixedField(Constants.Fields.SRC_PORT.getName(), commandLine.getOptionValue("ip_src_port"));
+ config.putFixedField(Constants.Fields.DST_PORT.getName(), commandLine.getOptionValue("ip_dst_port"));
+ config.putFixedField(Constants.Fields.PROTOCOL.getName(), commandLine.getOptionValue("protocol"));
+ config.putFixedField(Constants.Fields.INCLUDES_REVERSE_TRAFFIC.getName(), Boolean.toString(commandLine.hasOption("include_reverse")));
+ config.putFixedField(PcapHelper.PacketFields.PACKET_FILTER.getName(), commandLine.getOptionValue("packet_filter"));
return config;
}
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/bf2528fd/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/utils/PcapInspector.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/utils/PcapInspector.java b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/utils/PcapInspector.java
index ef11af8..f460db3 100644
--- a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/utils/PcapInspector.java
+++ b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/utils/PcapInspector.java
@@ -35,10 +35,7 @@ import javax.annotation.Nullable;
import java.io.IOException;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
-import java.util.ArrayList;
-import java.util.Date;
-import java.util.EnumMap;
-import java.util.List;
+import java.util.*;
public class PcapInspector {
private static abstract class OptionHandler implements Function<String, Option> {}
@@ -143,13 +140,13 @@ public class PcapInspector {
long millis = Long.divideUnsigned(key.get(), 1000000);
String ts = DATE_FORMAT.format(new Date(millis));
for(PacketInfo pi : PcapHelper.toPacketInfo(value.copyBytes())) {
- EnumMap<Constants.Fields, Object> result = PcapHelper.packetToFields(pi);
+ Map<String, Object> result = PcapHelper.packetToFields(pi);
List<String> fieldResults = new ArrayList<String>() {{
add("TS: " + ts);
}};
for(Constants.Fields field : Constants.Fields.values()) {
- if(result.containsKey(field)) {
- fieldResults.add(field.getName() + ": " + result.get(field));
+ if(result.containsKey(field.getName())) {
+ fieldResults.add(field.getName() + ": " + result.get(field.getName()));
}
}
System.out.println(Joiner.on(",").join(fieldResults));
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/bf2528fd/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/FixedPcapFilterTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/FixedPcapFilterTest.java b/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/FixedPcapFilterTest.java
index 218d143..84969d3 100644
--- a/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/FixedPcapFilterTest.java
+++ b/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/FixedPcapFilterTest.java
@@ -24,29 +24,30 @@ import org.apache.metron.pcap.filter.fixed.FixedPcapFilter;
import org.junit.Assert;
import org.junit.Test;
-import java.util.EnumMap;
+import java.util.HashMap;
+import java.util.Map;
public class FixedPcapFilterTest {
@Test
public void testTrivialEquality() throws Exception {
Configuration config = new Configuration();
- final EnumMap<Constants.Fields, String> fields = new EnumMap<Constants.Fields, String>(Constants.Fields.class) {{
- put(Constants.Fields.SRC_ADDR, "src_ip");
- put(Constants.Fields.SRC_PORT, "0");
- put(Constants.Fields.DST_ADDR, "dst_ip");
- put(Constants.Fields.DST_PORT, "1");
- put(Constants.Fields.INCLUDES_REVERSE_TRAFFIC, "false");
+ final Map<String, String> fields = new HashMap<String, String>() {{
+ put(Constants.Fields.SRC_ADDR.getName(), "src_ip");
+ put(Constants.Fields.SRC_PORT.getName(), "0");
+ put(Constants.Fields.DST_ADDR.getName(), "dst_ip");
+ put(Constants.Fields.DST_PORT.getName(), "1");
+ put(Constants.Fields.INCLUDES_REVERSE_TRAFFIC.getName(), "false");
}};
new FixedPcapFilter.Configurator().addToConfig(fields, config);
{
FixedPcapFilter filter = new FixedPcapFilter() {
@Override
- protected EnumMap<Constants.Fields, Object> packetToFields(PacketInfo pi) {
- return new EnumMap<Constants.Fields, Object>(Constants.Fields.class) {{
- put(Constants.Fields.SRC_ADDR, "src_ip");
- put(Constants.Fields.SRC_PORT, 0);
- put(Constants.Fields.DST_ADDR, "dst_ip");
- put(Constants.Fields.DST_PORT, 1);
+ protected Map<String, Object> packetToFields(PacketInfo pi) {
+ return new HashMap<String, Object>() {{
+ put(Constants.Fields.SRC_ADDR.getName(), "src_ip");
+ put(Constants.Fields.SRC_PORT.getName(), 0);
+ put(Constants.Fields.DST_ADDR.getName(), "dst_ip");
+ put(Constants.Fields.DST_PORT.getName(), 1);
}};
}
};
@@ -58,23 +59,23 @@ public class FixedPcapFilterTest {
@Test
public void testReverseTraffic() throws Exception {
Configuration config = new Configuration();
- final EnumMap<Constants.Fields, String> fields = new EnumMap<Constants.Fields, String>(Constants.Fields.class) {{
- put(Constants.Fields.SRC_ADDR, "src_ip");
- put(Constants.Fields.SRC_PORT, "0");
- put(Constants.Fields.DST_ADDR, "dst_ip");
- put(Constants.Fields.DST_PORT, "1");
- put(Constants.Fields.INCLUDES_REVERSE_TRAFFIC, "true");
+ final Map<String, String> fields = new HashMap<String, String>() {{
+ put(Constants.Fields.SRC_ADDR.getName(), "src_ip");
+ put(Constants.Fields.SRC_PORT.getName(), "0");
+ put(Constants.Fields.DST_ADDR.getName(), "dst_ip");
+ put(Constants.Fields.DST_PORT.getName(), "1");
+ put(Constants.Fields.INCLUDES_REVERSE_TRAFFIC.getName(), "true");
}};
new FixedPcapFilter.Configurator().addToConfig(fields, config);
{
FixedPcapFilter filter = new FixedPcapFilter() {
@Override
- protected EnumMap<Constants.Fields, Object> packetToFields(PacketInfo pi) {
- return new EnumMap<Constants.Fields, Object>(Constants.Fields.class) {{
- put(Constants.Fields.SRC_ADDR, "src_ip");
- put(Constants.Fields.SRC_PORT, 0);
- put(Constants.Fields.DST_ADDR, "dst_ip");
- put(Constants.Fields.DST_PORT, 1);
+ protected Map<String, Object> packetToFields(PacketInfo pi) {
+ return new HashMap<String, Object>() {{
+ put(Constants.Fields.SRC_ADDR.getName(), "src_ip");
+ put(Constants.Fields.SRC_PORT.getName(), 0);
+ put(Constants.Fields.DST_ADDR.getName(), "dst_ip");
+ put(Constants.Fields.DST_PORT.getName(), 1);
}};
}
};
@@ -85,12 +86,12 @@ public class FixedPcapFilterTest {
{
FixedPcapFilter filter = new FixedPcapFilter() {
@Override
- protected EnumMap<Constants.Fields, Object> packetToFields(PacketInfo pi) {
- return new EnumMap<Constants.Fields, Object>(Constants.Fields.class) {{
- put(Constants.Fields.SRC_ADDR, "dst_ip");
- put(Constants.Fields.SRC_PORT, 1);
- put(Constants.Fields.DST_ADDR, "src_ip");
- put(Constants.Fields.DST_PORT, 0);
+ protected Map<String, Object> packetToFields(PacketInfo pi) {
+ return new HashMap<String, Object>() {{
+ put(Constants.Fields.SRC_ADDR.getName(), "dst_ip");
+ put(Constants.Fields.SRC_PORT.getName(), 1);
+ put(Constants.Fields.DST_ADDR.getName(), "src_ip");
+ put(Constants.Fields.DST_PORT.getName(), 0);
}};
}
};
@@ -101,12 +102,12 @@ public class FixedPcapFilterTest {
{
FixedPcapFilter filter = new FixedPcapFilter() {
@Override
- protected EnumMap<Constants.Fields, Object> packetToFields(PacketInfo pi) {
- return new EnumMap<Constants.Fields, Object>(Constants.Fields.class) {{
- put(Constants.Fields.SRC_ADDR, "dst_ip");
- put(Constants.Fields.SRC_PORT, 0);
- put(Constants.Fields.DST_ADDR, "src_ip");
- put(Constants.Fields.DST_PORT, 1);
+ protected Map<String, Object> packetToFields(PacketInfo pi) {
+ return new HashMap<String, Object>() {{
+ put(Constants.Fields.SRC_ADDR.getName(), "dst_ip");
+ put(Constants.Fields.SRC_PORT.getName(), 0);
+ put(Constants.Fields.DST_ADDR.getName(), "src_ip");
+ put(Constants.Fields.DST_PORT.getName(), 1);
}};
}
};
@@ -117,22 +118,22 @@ public class FixedPcapFilterTest {
@Test
public void testMissingDstAddr() throws Exception {
Configuration config = new Configuration();
- final EnumMap<Constants.Fields, String> fields = new EnumMap<Constants.Fields, String>(Constants.Fields.class) {{
- put(Constants.Fields.SRC_ADDR, "src_ip");
- put(Constants.Fields.SRC_PORT, "0");
- put(Constants.Fields.DST_PORT, "1");
- put(Constants.Fields.INCLUDES_REVERSE_TRAFFIC, "false");
+ final HashMap<String, String> fields = new HashMap<String, String>() {{
+ put(Constants.Fields.SRC_ADDR.getName(), "src_ip");
+ put(Constants.Fields.SRC_PORT.getName(), "0");
+ put(Constants.Fields.DST_PORT.getName(), "1");
+ put(Constants.Fields.INCLUDES_REVERSE_TRAFFIC.getName(), "false");
}};
new FixedPcapFilter.Configurator().addToConfig(fields, config);
{
FixedPcapFilter filter = new FixedPcapFilter() {
@Override
- protected EnumMap<Constants.Fields, Object> packetToFields(PacketInfo pi) {
- return new EnumMap<Constants.Fields, Object>(Constants.Fields.class) {{
- put(Constants.Fields.SRC_ADDR, "src_ip");
- put(Constants.Fields.SRC_PORT, 0);
- put(Constants.Fields.DST_ADDR, "dst_ip");
- put(Constants.Fields.DST_PORT, 1);
+ protected HashMap<String, Object> packetToFields(PacketInfo pi) {
+ return new HashMap<String, Object>() {{
+ put(Constants.Fields.SRC_ADDR.getName(), "src_ip");
+ put(Constants.Fields.SRC_PORT.getName(), 0);
+ put(Constants.Fields.DST_ADDR.getName(), "dst_ip");
+ put(Constants.Fields.DST_PORT.getName(), 1);
}};
}
};
@@ -143,12 +144,12 @@ public void testMissingDstAddr() throws Exception {
{
FixedPcapFilter filter = new FixedPcapFilter() {
@Override
- protected EnumMap<Constants.Fields, Object> packetToFields(PacketInfo pi) {
- return new EnumMap<Constants.Fields, Object>(Constants.Fields.class) {{
- put(Constants.Fields.SRC_ADDR, "src_ip1");
- put(Constants.Fields.SRC_PORT, 0);
- put(Constants.Fields.DST_ADDR, "dst_ip");
- put(Constants.Fields.DST_PORT, 1);
+ protected HashMap<String, Object> packetToFields(PacketInfo pi) {
+ return new HashMap<String, Object>() {{
+ put(Constants.Fields.SRC_ADDR.getName(), "src_ip1");
+ put(Constants.Fields.SRC_PORT.getName(), 0);
+ put(Constants.Fields.DST_ADDR.getName(), "dst_ip");
+ put(Constants.Fields.DST_PORT.getName(), 1);
}};
}
};
@@ -159,22 +160,22 @@ public void testMissingDstAddr() throws Exception {
@Test
public void testMissingDstPort() throws Exception {
Configuration config = new Configuration();
- final EnumMap<Constants.Fields, String> fields = new EnumMap<Constants.Fields, String>(Constants.Fields.class) {{
- put(Constants.Fields.SRC_ADDR, "src_ip");
- put(Constants.Fields.SRC_PORT, "0");
- put(Constants.Fields.DST_ADDR, "dst_ip");
- put(Constants.Fields.INCLUDES_REVERSE_TRAFFIC, "false");
+ final HashMap<String, String> fields = new HashMap<String, String>() {{
+ put(Constants.Fields.SRC_ADDR.getName(), "src_ip");
+ put(Constants.Fields.SRC_PORT.getName(), "0");
+ put(Constants.Fields.DST_ADDR.getName(), "dst_ip");
+ put(Constants.Fields.INCLUDES_REVERSE_TRAFFIC.getName(), "false");
}};
new FixedPcapFilter.Configurator().addToConfig(fields, config);
{
FixedPcapFilter filter = new FixedPcapFilter() {
@Override
- protected EnumMap<Constants.Fields, Object> packetToFields(PacketInfo pi) {
- return new EnumMap<Constants.Fields, Object>(Constants.Fields.class) {{
- put(Constants.Fields.SRC_ADDR, "src_ip");
- put(Constants.Fields.SRC_PORT, 0);
- put(Constants.Fields.DST_ADDR, "dst_ip");
- put(Constants.Fields.DST_PORT, 1);
+ protected HashMap<String, Object> packetToFields(PacketInfo pi) {
+ return new HashMap<String, Object>() {{
+ put(Constants.Fields.SRC_ADDR.getName(), "src_ip");
+ put(Constants.Fields.SRC_PORT.getName(), 0);
+ put(Constants.Fields.DST_ADDR.getName(), "dst_ip");
+ put(Constants.Fields.DST_PORT.getName(), 1);
}};
}
};
@@ -185,12 +186,12 @@ public void testMissingDstAddr() throws Exception {
{
FixedPcapFilter filter = new FixedPcapFilter() {
@Override
- protected EnumMap<Constants.Fields, Object> packetToFields(PacketInfo pi) {
- return new EnumMap<Constants.Fields, Object>(Constants.Fields.class) {{
- put(Constants.Fields.SRC_ADDR, "src_ip");
- put(Constants.Fields.SRC_PORT, 0);
- put(Constants.Fields.DST_ADDR, "dst_ip");
- put(Constants.Fields.DST_PORT, 100);
+ protected HashMap<String, Object> packetToFields(PacketInfo pi) {
+ return new HashMap<String, Object>() {{
+ put(Constants.Fields.SRC_ADDR.getName(), "src_ip");
+ put(Constants.Fields.SRC_PORT.getName(), 0);
+ put(Constants.Fields.DST_ADDR.getName(), "dst_ip");
+ put(Constants.Fields.DST_PORT.getName(), 100);
}};
}
};
@@ -201,12 +202,12 @@ public void testMissingDstAddr() throws Exception {
{
FixedPcapFilter filter = new FixedPcapFilter() {
@Override
- protected EnumMap<Constants.Fields, Object> packetToFields(PacketInfo pi) {
- return new EnumMap<Constants.Fields, Object>(Constants.Fields.class) {{
- put(Constants.Fields.SRC_ADDR, "src_ip");
- put(Constants.Fields.SRC_PORT, 100);
- put(Constants.Fields.DST_ADDR, "dst_ip");
- put(Constants.Fields.DST_PORT, 100);
+ protected HashMap<String, Object> packetToFields(PacketInfo pi) {
+ return new HashMap<String, Object>() {{
+ put(Constants.Fields.SRC_ADDR.getName(), "src_ip");
+ put(Constants.Fields.SRC_PORT.getName(), 100);
+ put(Constants.Fields.DST_ADDR.getName(), "dst_ip");
+ put(Constants.Fields.DST_PORT.getName(), 100);
}};
}
};
@@ -217,22 +218,22 @@ public void testMissingDstAddr() throws Exception {
@Test
public void testMissingSrcAddr() throws Exception {
Configuration config = new Configuration();
- final EnumMap<Constants.Fields, String> fields = new EnumMap<Constants.Fields, String>(Constants.Fields.class) {{
- put(Constants.Fields.SRC_PORT, "0");
- put(Constants.Fields.DST_ADDR, "dst_ip");
- put(Constants.Fields.DST_PORT, "1");
- put(Constants.Fields.INCLUDES_REVERSE_TRAFFIC, "false");
+ final HashMap<String, String> fields = new HashMap<String, String>() {{
+ put(Constants.Fields.SRC_PORT.getName(), "0");
+ put(Constants.Fields.DST_ADDR.getName(), "dst_ip");
+ put(Constants.Fields.DST_PORT.getName(), "1");
+ put(Constants.Fields.INCLUDES_REVERSE_TRAFFIC.getName(), "false");
}};
new FixedPcapFilter.Configurator().addToConfig(fields, config);
{
FixedPcapFilter filter = new FixedPcapFilter() {
@Override
- protected EnumMap<Constants.Fields, Object> packetToFields(PacketInfo pi) {
- return new EnumMap<Constants.Fields, Object>(Constants.Fields.class) {{
- put(Constants.Fields.SRC_ADDR, "src_ip");
- put(Constants.Fields.SRC_PORT, 0);
- put(Constants.Fields.DST_ADDR, "dst_ip");
- put(Constants.Fields.DST_PORT, 1);
+ protected HashMap<String, Object> packetToFields(PacketInfo pi) {
+ return new HashMap<String, Object>() {{
+ put(Constants.Fields.SRC_ADDR.getName(), "src_ip");
+ put(Constants.Fields.SRC_PORT.getName(), 0);
+ put(Constants.Fields.DST_ADDR.getName(), "dst_ip");
+ put(Constants.Fields.DST_PORT.getName(), 1);
}};
}
};
@@ -243,22 +244,22 @@ public void testMissingDstAddr() throws Exception {
@Test
public void testMissingSrcPort() throws Exception {
Configuration config = new Configuration();
- final EnumMap<Constants.Fields, String> fields = new EnumMap<Constants.Fields, String>(Constants.Fields.class) {{
- put(Constants.Fields.SRC_ADDR, "src_ip");
- put(Constants.Fields.DST_ADDR, "dst_ip");
- put(Constants.Fields.DST_PORT, "1");
- put(Constants.Fields.INCLUDES_REVERSE_TRAFFIC, "false");
+ final HashMap<String, String> fields = new HashMap<String, String>() {{
+ put(Constants.Fields.SRC_ADDR.getName(), "src_ip");
+ put(Constants.Fields.DST_ADDR.getName(), "dst_ip");
+ put(Constants.Fields.DST_PORT.getName(), "1");
+ put(Constants.Fields.INCLUDES_REVERSE_TRAFFIC.getName(), "false");
}};
new FixedPcapFilter.Configurator().addToConfig(fields, config);
{
FixedPcapFilter filter = new FixedPcapFilter() {
@Override
- protected EnumMap<Constants.Fields, Object> packetToFields(PacketInfo pi) {
- return new EnumMap<Constants.Fields, Object>(Constants.Fields.class) {{
- put(Constants.Fields.SRC_ADDR, "src_ip");
- put(Constants.Fields.SRC_PORT, 0);
- put(Constants.Fields.DST_ADDR, "dst_ip");
- put(Constants.Fields.DST_PORT, 1);
+ protected HashMap<String, Object> packetToFields(PacketInfo pi) {
+ return new HashMap<String, Object>() {{
+ put(Constants.Fields.SRC_ADDR.getName(), "src_ip");
+ put(Constants.Fields.SRC_PORT.getName(), 0);
+ put(Constants.Fields.DST_ADDR.getName(), "dst_ip");
+ put(Constants.Fields.DST_PORT.getName(), 1);
}};
}
};
@@ -269,12 +270,12 @@ public void testMissingDstAddr() throws Exception {
{
FixedPcapFilter filter = new FixedPcapFilter() {
@Override
- protected EnumMap<Constants.Fields, Object> packetToFields(PacketInfo pi) {
- return new EnumMap<Constants.Fields, Object>(Constants.Fields.class) {{
- put(Constants.Fields.SRC_ADDR, "src_ip");
- put(Constants.Fields.SRC_PORT, 100);
- put(Constants.Fields.DST_ADDR, "dst_ip");
- put(Constants.Fields.DST_PORT, 1);
+ protected HashMap<String, Object> packetToFields(PacketInfo pi) {
+ return new HashMap<String, Object>() {{
+ put(Constants.Fields.SRC_ADDR.getName(), "src_ip");
+ put(Constants.Fields.SRC_PORT.getName(), 100);
+ put(Constants.Fields.DST_ADDR.getName(), "dst_ip");
+ put(Constants.Fields.DST_PORT.getName(), 1);
}};
}
};
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/bf2528fd/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/QueryPcapFilterTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/QueryPcapFilterTest.java b/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/QueryPcapFilterTest.java
index 5bb5d4a..7e3d55c 100644
--- a/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/QueryPcapFilterTest.java
+++ b/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/QueryPcapFilterTest.java
@@ -26,6 +26,7 @@ import org.junit.Assert;
import org.junit.Test;
import java.util.EnumMap;
+import java.util.HashMap;
public class QueryPcapFilterTest {
@@ -37,12 +38,12 @@ public class QueryPcapFilterTest {
{
PcapFilter filter = new QueryPcapFilter() {
@Override
- protected EnumMap<Constants.Fields, Object> packetToFields(PacketInfo pi) {
- return new EnumMap<Constants.Fields, Object>(Constants.Fields.class) {{
- put(Constants.Fields.SRC_ADDR, "src_ip");
- put(Constants.Fields.SRC_PORT, 0);
- put(Constants.Fields.DST_ADDR, "dst_ip");
- put(Constants.Fields.DST_PORT, 1);
+ protected HashMap<String, Object> packetToFields(PacketInfo pi) {
+ return new HashMap<String, Object>() {{
+ put(Constants.Fields.SRC_ADDR.getName(), "src_ip");
+ put(Constants.Fields.SRC_PORT.getName(), 0);
+ put(Constants.Fields.DST_ADDR.getName(), "dst_ip");
+ put(Constants.Fields.DST_PORT.getName(), 1);
}};
}
};
@@ -59,12 +60,12 @@ public class QueryPcapFilterTest {
{
PcapFilter filter = new QueryPcapFilter() {
@Override
- protected EnumMap<Constants.Fields, Object> packetToFields(PacketInfo pi) {
- return new EnumMap<Constants.Fields, Object>(Constants.Fields.class) {{
- put(Constants.Fields.SRC_ADDR, "src_ip");
- put(Constants.Fields.SRC_PORT, 0);
- put(Constants.Fields.DST_ADDR, "dst_ip");
- put(Constants.Fields.DST_PORT, 1);
+ protected HashMap<String, Object> packetToFields(PacketInfo pi) {
+ return new HashMap<String, Object>() {{
+ put(Constants.Fields.SRC_ADDR.getName(), "src_ip");
+ put(Constants.Fields.SRC_PORT.getName(), 0);
+ put(Constants.Fields.DST_ADDR.getName(), "dst_ip");
+ put(Constants.Fields.DST_PORT.getName(), 1);
}};
}
};
@@ -81,12 +82,12 @@ public class QueryPcapFilterTest {
{
QueryPcapFilter filter = new QueryPcapFilter() {
@Override
- protected EnumMap<Constants.Fields, Object> packetToFields(PacketInfo pi) {
- return new EnumMap<Constants.Fields, Object>(Constants.Fields.class) {{
- put(Constants.Fields.SRC_ADDR, "src_ip");
- put(Constants.Fields.SRC_PORT, 0);
- put(Constants.Fields.DST_ADDR, "dst_ip");
- put(Constants.Fields.DST_PORT, 1);
+ protected HashMap<String, Object> packetToFields(PacketInfo pi) {
+ return new HashMap<String, Object>() {{
+ put(Constants.Fields.SRC_ADDR.getName(), "src_ip");
+ put(Constants.Fields.SRC_PORT.getName(), 0);
+ put(Constants.Fields.DST_ADDR.getName(), "dst_ip");
+ put(Constants.Fields.DST_PORT.getName(), 1);
}};
}
};
@@ -97,12 +98,12 @@ public class QueryPcapFilterTest {
{
QueryPcapFilter filter = new QueryPcapFilter() {
@Override
- protected EnumMap<Constants.Fields, Object> packetToFields(PacketInfo pi) {
- return new EnumMap<Constants.Fields, Object>(Constants.Fields.class) {{
- put(Constants.Fields.SRC_ADDR, "src_ip_no_match");
- put(Constants.Fields.SRC_PORT, 0);
- put(Constants.Fields.DST_ADDR, "dst_ip");
- put(Constants.Fields.DST_PORT, 1);
+ protected HashMap<String, Object> packetToFields(PacketInfo pi) {
+ return new HashMap<String, Object>() {{
+ put(Constants.Fields.SRC_ADDR.getName(), "src_ip_no_match");
+ put(Constants.Fields.SRC_PORT.getName(), 0);
+ put(Constants.Fields.DST_ADDR.getName(), "dst_ip");
+ put(Constants.Fields.DST_PORT.getName(), 1);
}};
}
};
@@ -119,12 +120,12 @@ public class QueryPcapFilterTest {
{
QueryPcapFilter filter = new QueryPcapFilter() {
@Override
- protected EnumMap<Constants.Fields, Object> packetToFields(PacketInfo pi) {
- return new EnumMap<Constants.Fields, Object>(Constants.Fields.class) {{
- put(Constants.Fields.SRC_ADDR, "src_ip");
- put(Constants.Fields.SRC_PORT, 0);
- put(Constants.Fields.DST_ADDR, "dst_ip");
- put(Constants.Fields.DST_PORT, 1);
+ protected HashMap<String, Object> packetToFields(PacketInfo pi) {
+ return new HashMap<String, Object>() {{
+ put(Constants.Fields.SRC_ADDR.getName(), "src_ip");
+ put(Constants.Fields.SRC_PORT.getName(), 0);
+ put(Constants.Fields.DST_ADDR.getName(), "dst_ip");
+ put(Constants.Fields.DST_PORT.getName(), 1);
}};
}
};
@@ -135,12 +136,12 @@ public class QueryPcapFilterTest {
{
QueryPcapFilter filter = new QueryPcapFilter() {
@Override
- protected EnumMap<Constants.Fields, Object> packetToFields(PacketInfo pi) {
- return new EnumMap<Constants.Fields, Object>(Constants.Fields.class) {{
- put(Constants.Fields.SRC_ADDR, "src_ip");
- put(Constants.Fields.SRC_PORT, 0);
- put(Constants.Fields.DST_ADDR, "dst_ip");
- put(Constants.Fields.DST_PORT, 100);
+ protected HashMap<String, Object> packetToFields(PacketInfo pi) {
+ return new HashMap<String, Object>() {{
+ put(Constants.Fields.SRC_ADDR.getName(), "src_ip");
+ put(Constants.Fields.SRC_PORT.getName(), 0);
+ put(Constants.Fields.DST_ADDR.getName(), "dst_ip");
+ put(Constants.Fields.DST_PORT.getName(), 100);
}};
}
};
@@ -151,12 +152,12 @@ public class QueryPcapFilterTest {
{
QueryPcapFilter filter = new QueryPcapFilter() {
@Override
- protected EnumMap<Constants.Fields, Object> packetToFields(PacketInfo pi) {
- return new EnumMap<Constants.Fields, Object>(Constants.Fields.class) {{
- put(Constants.Fields.SRC_ADDR, "src_ip");
- put(Constants.Fields.SRC_PORT, 100);
- put(Constants.Fields.DST_ADDR, "dst_ip");
- put(Constants.Fields.DST_PORT, 100);
+ protected HashMap<String, Object> packetToFields(PacketInfo pi) {
+ return new HashMap<String, Object>() {{
+ put(Constants.Fields.SRC_ADDR.getName(), "src_ip");
+ put(Constants.Fields.SRC_PORT.getName(), 100);
+ put(Constants.Fields.DST_ADDR.getName(), "dst_ip");
+ put(Constants.Fields.DST_PORT.getName(), 100);
}};
}
};
@@ -173,12 +174,12 @@ public class QueryPcapFilterTest {
{
QueryPcapFilter filter = new QueryPcapFilter() {
@Override
- protected EnumMap<Constants.Fields, Object> packetToFields(PacketInfo pi) {
- return new EnumMap<Constants.Fields, Object>(Constants.Fields.class) {{
- put(Constants.Fields.SRC_ADDR, "src_ip");
- put(Constants.Fields.SRC_PORT, 0);
- put(Constants.Fields.DST_ADDR, "dst_ip");
- put(Constants.Fields.DST_PORT, 1);
+ protected HashMap<String, Object> packetToFields(PacketInfo pi) {
+ return new HashMap<String, Object>() {{
+ put(Constants.Fields.SRC_ADDR.getName(), "src_ip");
+ put(Constants.Fields.SRC_PORT.getName(), 0);
+ put(Constants.Fields.DST_ADDR.getName(), "dst_ip");
+ put(Constants.Fields.DST_PORT.getName(), 1);
}};
}
};
@@ -195,12 +196,12 @@ public class QueryPcapFilterTest {
{
QueryPcapFilter filter = new QueryPcapFilter() {
@Override
- protected EnumMap<Constants.Fields, Object> packetToFields(PacketInfo pi) {
- return new EnumMap<Constants.Fields, Object>(Constants.Fields.class) {{
- put(Constants.Fields.SRC_ADDR, "src_ip");
- put(Constants.Fields.SRC_PORT, 0);
- put(Constants.Fields.DST_ADDR, "dst_ip");
- put(Constants.Fields.DST_PORT, 1);
+ protected HashMap<String, Object> packetToFields(PacketInfo pi) {
+ return new HashMap<String, Object>() {{
+ put(Constants.Fields.SRC_ADDR.getName(), "src_ip");
+ put(Constants.Fields.SRC_PORT.getName(), 0);
+ put(Constants.Fields.DST_ADDR.getName(), "dst_ip");
+ put(Constants.Fields.DST_PORT.getName(), 1);
}};
}
};
@@ -211,12 +212,12 @@ public class QueryPcapFilterTest {
{
QueryPcapFilter filter = new QueryPcapFilter() {
@Override
- protected EnumMap<Constants.Fields, Object> packetToFields(PacketInfo pi) {
- return new EnumMap<Constants.Fields, Object>(Constants.Fields.class) {{
- put(Constants.Fields.SRC_ADDR, "src_ip");
- put(Constants.Fields.SRC_PORT, 100);
- put(Constants.Fields.DST_ADDR, "dst_ip");
- put(Constants.Fields.DST_PORT, 1);
+ protected HashMap<String, Object> packetToFields(PacketInfo pi) {
+ return new HashMap<String, Object>() {{
+ put(Constants.Fields.SRC_ADDR.getName(), "src_ip");
+ put(Constants.Fields.SRC_PORT.getName(), 100);
+ put(Constants.Fields.DST_ADDR.getName(), "dst_ip");
+ put(Constants.Fields.DST_PORT.getName(), 1);
}};
}
};
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/bf2528fd/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/integration/PcapTopologyIntegrationTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/integration/PcapTopologyIntegrationTest.java b/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/integration/PcapTopologyIntegrationTest.java
index 84e7574..a869723 100644
--- a/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/integration/PcapTopologyIntegrationTest.java
+++ b/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/integration/PcapTopologyIntegrationTest.java
@@ -109,6 +109,13 @@ public class PcapTopologyIntegrationTest {
BytesWritable value = new BytesWritable();
while (reader.next(key, value)) {
byte[] pcapWithHeader = value.copyBytes();
+ //if you are debugging and want the hex dump of the packets, uncomment the following:
+
+ //for(byte b : pcapWithHeader) {
+ // System.out.print(String.format("%02x", b));
+ //}
+ //System.out.println("");
+
long calculatedTs = PcapHelper.getTimestamp(pcapWithHeader);
{
List<PacketInfo> info = PcapHelper.toPacketInfo(pcapWithHeader);
@@ -274,7 +281,7 @@ public class PcapTopologyIntegrationTest {
, getTimestamp(4, pcapEntries)
, getTimestamp(5, pcapEntries)
, 10
- , new EnumMap<>(Constants.Fields.class)
+ , new HashMap<>()
, new Configuration()
, FileSystem.get(new Configuration())
, new FixedPcapFilter.Configurator()
@@ -307,8 +314,8 @@ public class PcapTopologyIntegrationTest {
, getTimestamp(0, pcapEntries)
, getTimestamp(1, pcapEntries)
, 10
- , new EnumMap<Constants.Fields, String>(Constants.Fields.class) {{
- put(Constants.Fields.DST_ADDR, "207.28.210.1");
+ , new HashMap<String, String>() {{
+ put(Constants.Fields.DST_ADDR.getName(), "207.28.210.1");
}}
, new Configuration()
, FileSystem.get(new Configuration())
@@ -342,8 +349,8 @@ public class PcapTopologyIntegrationTest {
, getTimestamp(0, pcapEntries)
, getTimestamp(1, pcapEntries)
, 10
- , new EnumMap<Constants.Fields, String>(Constants.Fields.class) {{
- put(Constants.Fields.PROTOCOL, "foo");
+ , new HashMap<String, String>() {{
+ put(Constants.Fields.PROTOCOL.getName(), "foo");
}}
, new Configuration()
, FileSystem.get(new Configuration())
@@ -377,7 +384,7 @@ public class PcapTopologyIntegrationTest {
, getTimestamp(0, pcapEntries)
, getTimestamp(pcapEntries.size()-1, pcapEntries) + 1
, 10
- , new EnumMap<>(Constants.Fields.class)
+ , new HashMap<>()
, new Configuration()
, FileSystem.get(new Configuration())
, new FixedPcapFilter.Configurator()
@@ -409,8 +416,8 @@ public class PcapTopologyIntegrationTest {
, getTimestamp(0, pcapEntries)
, getTimestamp(pcapEntries.size()-1, pcapEntries) + 1
, 10
- , new EnumMap<Constants.Fields, String>(Constants.Fields.class) {{
- put(Constants.Fields.DST_PORT, "22");
+ , new HashMap<String, String>() {{
+ put(Constants.Fields.DST_PORT.getName(), "22");
}}
, new Configuration()
, FileSystem.get(new Configuration())
@@ -433,6 +440,25 @@ public class PcapTopologyIntegrationTest {
Assert.assertTrue(baos.toByteArray().length > 0);
}
{
+ //test with query filter and byte array matching
+ Iterable<byte[]> results =
+ job.query(new Path(outDir.getAbsolutePath())
+ , new Path(queryDir.getAbsolutePath())
+ , getTimestamp(0, pcapEntries)
+ , getTimestamp(pcapEntries.size()-1, pcapEntries) + 1
+ , 10
+ , "BYTEARRAY_MATCHER('2f56abd814bc56420489ca38e7faf8cec3d4', packet)"
+ , new Configuration()
+ , FileSystem.get(new Configuration())
+ , new QueryPcapFilter.Configurator()
+ );
+ assertInOrder(results);
+ Assert.assertEquals(1, Iterables.size(results));
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ PcapMerger.merge(baos, Iterables.partition(results, 1).iterator().next());
+ Assert.assertTrue(baos.toByteArray().length > 0);
+ }
+ {
//test with query filter
Iterable<byte[]> results =
job.query(new Path(outDir.getAbsolutePath())
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/bf2528fd/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/query/PcapCliTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/query/PcapCliTest.java b/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/query/PcapCliTest.java
index bad22e4..4f441f1 100644
--- a/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/query/PcapCliTest.java
+++ b/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/query/PcapCliTest.java
@@ -24,6 +24,7 @@ import org.apache.metron.common.Constants;
import org.apache.metron.common.hadoop.SequenceFileIterable;
import org.apache.metron.common.system.Clock;
import org.apache.metron.common.utils.timestamp.TimestampConverters;
+import org.apache.metron.pcap.PcapHelper;
import org.apache.metron.pcap.filter.fixed.FixedPcapFilter;
import org.apache.metron.pcap.filter.query.QueryPcapFilter;
import org.apache.metron.pcap.mr.PcapJob;
@@ -70,7 +71,8 @@ public class PcapCliTest {
"-ip_dst_addr", "192.168.1.2",
"-ip_src_port", "8081",
"-ip_dst_port", "8082",
- "-protocol", "6"
+ "-protocol", "6",
+ "-packet_filter", "`casey`"
};
List<byte[]> pcaps = Arrays.asList(new byte[][]{asBytes("abc"), asBytes("def"), asBytes("ghi")});
Iterator iterator = pcaps.iterator();
@@ -79,13 +81,14 @@ public class PcapCliTest {
Path base_path = new Path(CliParser.BASE_PATH_DEFAULT);
Path base_output_path = new Path(CliParser.BASE_OUTPUT_PATH_DEFAULT);
- EnumMap<Constants.Fields, String> query = new EnumMap<Constants.Fields, String>(Constants.Fields.class) {{
- put(Constants.Fields.SRC_ADDR, "192.168.1.1");
- put(Constants.Fields.DST_ADDR, "192.168.1.2");
- put(Constants.Fields.SRC_PORT, "8081");
- put(Constants.Fields.DST_PORT, "8082");
- put(Constants.Fields.PROTOCOL, "6");
- put(Constants.Fields.INCLUDES_REVERSE_TRAFFIC, "false");
+ HashMap<String, String> query = new HashMap<String, String>() {{
+ put(Constants.Fields.SRC_ADDR.getName(), "192.168.1.1");
+ put(Constants.Fields.DST_ADDR.getName(), "192.168.1.2");
+ put(Constants.Fields.SRC_PORT.getName(), "8081");
+ put(Constants.Fields.DST_PORT.getName(), "8082");
+ put(Constants.Fields.PROTOCOL.getName(), "6");
+ put(Constants.Fields.INCLUDES_REVERSE_TRAFFIC.getName(), "false");
+ put(PcapHelper.PacketFields.PACKET_FILTER.getName(), "`casey`");
}};
when(jobRunner.query(eq(base_path), eq(base_output_path), anyLong(), anyLong(), anyInt(), eq(query), isA(Configuration.class), isA(FileSystem.class), isA(FixedPcapFilter.Configurator.class))).thenReturn(iterable);
@@ -120,13 +123,13 @@ public class PcapCliTest {
Path base_path = new Path("/base/path");
Path base_output_path = new Path("/base/output/path");
- EnumMap<Constants.Fields, String> query = new EnumMap<Constants.Fields, String>(Constants.Fields.class) {{
- put(Constants.Fields.SRC_ADDR, "192.168.1.1");
- put(Constants.Fields.DST_ADDR, "192.168.1.2");
- put(Constants.Fields.SRC_PORT, "8081");
- put(Constants.Fields.DST_PORT, "8082");
- put(Constants.Fields.PROTOCOL, "6");
- put(Constants.Fields.INCLUDES_REVERSE_TRAFFIC, "true");
+ Map<String, String> query = new HashMap<String, String>() {{
+ put(Constants.Fields.SRC_ADDR.getName(), "192.168.1.1");
+ put(Constants.Fields.DST_ADDR.getName(), "192.168.1.2");
+ put(Constants.Fields.SRC_PORT.getName(), "8081");
+ put(Constants.Fields.DST_PORT.getName(), "8082");
+ put(Constants.Fields.PROTOCOL.getName(), "6");
+ put(Constants.Fields.INCLUDES_REVERSE_TRAFFIC.getName(), "true");
}};
when(jobRunner.query(eq(base_path), eq(base_output_path), anyLong(), anyLong(), anyInt(), eq(query), isA(Configuration.class), isA(FileSystem.class), isA(FixedPcapFilter.Configurator.class))).thenReturn(iterable);
@@ -162,13 +165,13 @@ public class PcapCliTest {
Path base_path = new Path("/base/path");
Path base_output_path = new Path("/base/output/path");
- EnumMap<Constants.Fields, String> query = new EnumMap<Constants.Fields, String>(Constants.Fields.class) {{
- put(Constants.Fields.SRC_ADDR, "192.168.1.1");
- put(Constants.Fields.DST_ADDR, "192.168.1.2");
- put(Constants.Fields.SRC_PORT, "8081");
- put(Constants.Fields.DST_PORT, "8082");
- put(Constants.Fields.PROTOCOL, "6");
- put(Constants.Fields.INCLUDES_REVERSE_TRAFFIC, "true");
+ Map<String, String> query = new HashMap<String, String>() {{
+ put(Constants.Fields.SRC_ADDR.getName(), "192.168.1.1");
+ put(Constants.Fields.DST_ADDR.getName(), "192.168.1.2");
+ put(Constants.Fields.SRC_PORT.getName(), "8081");
+ put(Constants.Fields.DST_PORT.getName(), "8082");
+ put(Constants.Fields.PROTOCOL.getName(), "6");
+ put(Constants.Fields.INCLUDES_REVERSE_TRAFFIC.getName(), "true");
}};
long startAsNanos = asNanos("2016-06-13-18:35.00", "yyyy-MM-dd-HH:mm.ss");
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/bf2528fd/metron-platform/metron-pcap/pom.xml
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap/pom.xml b/metron-platform/metron-pcap/pom.xml
index 16a34d1..73f57f0 100644
--- a/metron-platform/metron-pcap/pom.xml
+++ b/metron-platform/metron-pcap/pom.xml
@@ -26,6 +26,18 @@
</properties>
<dependencies>
<dependency>
+ <groupId>net.byteseek</groupId>
+ <artifactId>byteseek</artifactId>
+ <version>2.0.3</version>
+ <exclusions>
+ <!-- This is a LGPL dependency of byteseek, so we need to replace/mimic it minimally -->
+ <exclusion>
+ <groupId>net.sf.trove4j</groupId>
+ <artifactId>trove4j</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
<groupId>org.apache.metron</groupId>
<artifactId>metron-hbase</artifactId>
<version>${project.parent.version}</version>
@@ -126,6 +138,13 @@
<version>${project.parent.version}</version>
</dependency>
<dependency>
+ <groupId>org.apache.metron</groupId>
+ <artifactId>metron-common</artifactId>
+ <version>${project.parent.version}</version>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>${global_hadoop_version}</version>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/bf2528fd/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/PacketInfo.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/PacketInfo.java b/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/PacketInfo.java
index fcaf1b0..f3a7e25 100644
--- a/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/PacketInfo.java
+++ b/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/PacketInfo.java
@@ -37,6 +37,9 @@ import org.apache.metron.pcap.utils.PcapUtils;
*/
public class PacketInfo {
+ /** The packet data */
+ private byte[] packetBytes = null;
+
/** The packetHeader. */
private PacketHeader packetHeader = null;
@@ -158,9 +161,18 @@ public class PacketInfo {
* the tcp packet
* @param udpPacket
* the udp packet
+ * @param packetBytes
+ * the raw packet data
*/
- public PacketInfo(GlobalHeader globalHeader, PacketHeader packetHeader, PcapPacket packet, Ipv4Packet ipv4Packet, TcpPacket tcpPacket,
- UdpPacket udpPacket) {
+ public PacketInfo( GlobalHeader globalHeader
+ , PacketHeader packetHeader
+ , PcapPacket packet
+ , Ipv4Packet ipv4Packet
+ , TcpPacket tcpPacket
+ , UdpPacket udpPacket
+ , byte[] packetBytes
+ ) {
+ this.packetBytes = packetBytes;
this.packetHeader = packetHeader;
this.packet = packet;
this.ipv4Packet = ipv4Packet;
@@ -177,6 +189,15 @@ public class PacketInfo {
public GlobalHeader getGlobalHeader() {
return globalHeader;
}
+ /**
+ * Gets the packet raw data.
+ *
+ *
+ * @return the packet data
+ */
+ public byte[] getPacketBytes() {
+ return packetBytes;
+ }
/**
* Gets the packet header.
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/bf2528fd/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/PcapHelper.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/PcapHelper.java b/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/PcapHelper.java
index e48824f..ebd7ac7 100644
--- a/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/PcapHelper.java
+++ b/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/PcapHelper.java
@@ -39,9 +39,7 @@ import org.krakenapps.pcap.util.ByteOrderConverter;
import java.io.EOFException;
import java.io.IOException;
-import java.util.ArrayList;
-import java.util.EnumMap;
-import java.util.List;
+import java.util.*;
import static org.apache.metron.pcap.Constants.*;
@@ -50,6 +48,22 @@ public class PcapHelper {
public static final int PACKET_HEADER_SIZE = 4*Integer.BYTES;
public static final int GLOBAL_HEADER_SIZE = 24;
private static final Logger LOG = Logger.getLogger(PcapHelper.class);
+
+ public enum PacketFields implements org.apache.metron.common.Constants.Field{
+ PACKET_DATA("packet"),
+ PACKET_FILTER("packet_filter")
+ ;
+ String name;
+ PacketFields(String name) {
+ this.name = name;
+ }
+
+ @Override
+ public String getName() {
+ return name;
+ }
+ }
+
public static ThreadLocal<MetronEthernetDecoder> ETHERNET_DECODER = new ThreadLocal<MetronEthernetDecoder>() {
@Override
protected MetronEthernetDecoder initialValue() {
@@ -190,23 +204,25 @@ public class PcapHelper {
System.arraycopy(packet, 0, ret, offset, packet.length);
return ret;
}
- public static EnumMap<org.apache.metron.common.Constants.Fields, Object> packetToFields(PacketInfo pi) {
- EnumMap<org.apache.metron.common.Constants.Fields, Object> ret = new EnumMap(org.apache.metron.common.Constants.Fields.class);
+
+ public static Map<String, Object> packetToFields(PacketInfo pi) {
+ Map<String, Object> ret = new HashMap<>();
+ ret.put(PacketFields.PACKET_DATA.getName(), pi.getPacketBytes());
if(pi.getTcpPacket() != null) {
if(pi.getTcpPacket().getSourceAddress() != null) {
- ret.put(org.apache.metron.common.Constants.Fields.SRC_ADDR, pi.getTcpPacket().getSourceAddress().getHostAddress());
+ ret.put(org.apache.metron.common.Constants.Fields.SRC_ADDR.getName(), pi.getTcpPacket().getSourceAddress().getHostAddress());
}
if(pi.getTcpPacket().getSource() != null ) {
- ret.put(org.apache.metron.common.Constants.Fields.SRC_PORT, pi.getTcpPacket().getSource().getPort());
+ ret.put(org.apache.metron.common.Constants.Fields.SRC_PORT.getName(), pi.getTcpPacket().getSource().getPort());
}
if(pi.getTcpPacket().getDestinationAddress() != null ) {
- ret.put(org.apache.metron.common.Constants.Fields.DST_ADDR, pi.getTcpPacket().getDestinationAddress().getHostAddress());
+ ret.put(org.apache.metron.common.Constants.Fields.DST_ADDR.getName(), pi.getTcpPacket().getDestinationAddress().getHostAddress());
}
if(pi.getTcpPacket().getDestination() != null ) {
- ret.put(org.apache.metron.common.Constants.Fields.DST_PORT, pi.getTcpPacket().getDestination().getPort());
+ ret.put(org.apache.metron.common.Constants.Fields.DST_PORT.getName(), pi.getTcpPacket().getDestination().getPort());
}
if(pi.getIpv4Packet() != null) {
- ret.put(org.apache.metron.common.Constants.Fields.PROTOCOL, pi.getIpv4Packet().getProtocol());
+ ret.put(org.apache.metron.common.Constants.Fields.PROTOCOL.getName(), pi.getIpv4Packet().getProtocol());
}
}
return ret;
@@ -281,7 +297,7 @@ public class PcapHelper {
}
packetInfoList.add(new PacketInfo(globalHeader, packetHeader, packet,
- ipv4Packet, tcpPacket, udpPacket));
+ ipv4Packet, tcpPacket, udpPacket, pcap));
} catch (NegativeArraySizeException ignored) {
LOG.debug("Ignorable exception while parsing packet.", ignored);
} catch (EOFException eof) { // $codepro.audit.disable logExceptions
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/bf2528fd/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/filter/PcapFieldResolver.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/filter/PcapFieldResolver.java b/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/filter/PcapFieldResolver.java
index 4aeec6c..e5a15e7 100644
--- a/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/filter/PcapFieldResolver.java
+++ b/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/filter/PcapFieldResolver.java
@@ -22,17 +22,19 @@ import org.apache.metron.common.Constants;
import org.apache.metron.common.dsl.VariableResolver;
import java.util.EnumMap;
+import java.util.HashMap;
+import java.util.Map;
public class PcapFieldResolver implements VariableResolver {
- EnumMap<Constants.Fields, Object> fieldsMap = null;
+ Map<String, Object> fieldsMap = new HashMap<>();
- public PcapFieldResolver(EnumMap<Constants.Fields, Object> fieldsMap) {
+ public PcapFieldResolver(Map<String, Object> fieldsMap) {
this.fieldsMap = fieldsMap;
}
@Override
public Object resolve(String variable) {
- return fieldsMap.get(Constants.Fields.fromString(variable));
+ return fieldsMap.get(variable);
}
}
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/bf2528fd/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/filter/PcapFilter.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/filter/PcapFilter.java b/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/filter/PcapFilter.java
index c7168aa..6d775df 100644
--- a/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/filter/PcapFilter.java
+++ b/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/filter/PcapFilter.java
@@ -23,6 +23,6 @@ import org.apache.metron.pcap.PacketInfo;
import java.util.Map;
import java.util.function.Predicate;
-public interface PcapFilter extends Predicate<PacketInfo> {
+public interface PcapFilter extends Predicate<PacketInfo>{
void configure(Iterable<Map.Entry<String, String>> config);
}
[23/26] incubator-metron git commit: METRON-859 Use REST application
with Kerberos (merrimanr) closes apache/incubator-metron#535
Posted by ma...@apache.org.
METRON-859 Use REST application with Kerberos (merrimanr) closes apache/incubator-metron#535
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/5dd87886
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/5dd87886
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/5dd87886
Branch: refs/heads/Metron_0.4.0
Commit: 5dd87886d6288b3f65e4f26e88ef10dcc82895fe
Parents: ffe2d93
Author: merrimanr <me...@gmail.com>
Authored: Wed Apr 26 14:01:02 2017 -0500
Committer: merrimanr <me...@gmail.com>
Committed: Wed Apr 26 14:01:02 2017 -0500
----------------------------------------------------------------------
dependencies_with_url.csv | 2 +
metron-interface/metron-rest/README.md | 32 ++++++-
metron-interface/metron-rest/pom.xml | 6 ++
.../apache/metron/rest/MetronRestConstants.java | 4 +
.../apache/metron/rest/config/HadoopConfig.java | 18 +++-
.../apache/metron/rest/config/KafkaConfig.java | 11 ++-
.../metron/rest/config/RestTemplateConfig.java | 20 ++++-
.../rest/service/impl/StormCLIWrapper.java | 15 ++++
.../src/main/resources/application-vagrant.yml | 6 ++
.../metron/rest/config/HadoopConfigTest.java | 89 ++++++++++++++++++++
.../rest/config/RestTemplateConfigTest.java | 67 +++++++++++++++
.../rest/service/impl/StormCLIWrapperTest.java | 24 ++++++
.../metron-rest/src/test/resources/README.vm | 32 ++++++-
13 files changed, 313 insertions(+), 13 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/5dd87886/dependencies_with_url.csv
----------------------------------------------------------------------
diff --git a/dependencies_with_url.csv b/dependencies_with_url.csv
index 25650a3..053f11a 100644
--- a/dependencies_with_url.csv
+++ b/dependencies_with_url.csv
@@ -293,3 +293,5 @@ org.atteo.classindex:classindex:jar:3.3:compile,ASLv2,https://github.com/atteo/c
com.squareup.okhttp:okhttp:jar:2.4.0:compile,ASLv2,https://github.com/square/okhttp
com.squareup.okio:okio:jar:1.4.0:compile,ASLv2,https://github.com/square/okhttp
org.htrace:htrace-core:jar:3.0.4:compile,ASLv2,http://htrace.incubator.apache.org/
+org.springframework.security.kerberos:spring-security-kerberos-client:jar:1.0.1.RELEASE:compile,ASLv2,https://github.com/spring-projects/spring-security-kerberos
+org.springframework.security.kerberos:spring-security-kerberos-core:jar:1.0.1.RELEASE:compile,ASLv2,https://github.com/spring-projects/spring-security-kerberos
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/5dd87886/metron-interface/metron-rest/README.md
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/README.md b/metron-interface/metron-rest/README.md
index c52b21d..89984ed 100644
--- a/metron-interface/metron-rest/README.md
+++ b/metron-interface/metron-rest/README.md
@@ -22,7 +22,7 @@ This module provides a RESTful API for interacting with Metron.
metron-rest-$METRON_VERSION.jar
```
-1. Create an `application.yml` file with the contents of [application-docker.yml](src/main/resources/application-docker.yml). Substitute the appropriate Metron service urls (Kafka, Zookeeper, Storm, etc.) in properties containing `${docker.host.address}` and update the `spring.datasource.*` properties as needed (see the [Security](#security) section for more details).
+1. Create an `application.yml` file with the contents of [application-vagrant.yml](src/main/resources/application-vagrant.yml). Substitute the appropriate Metron service hosts (Kafka, Zookeeper, Storm, etc.) in properties containing `node1` and update the `spring.datasource.*` properties as needed (see the [Security](#security) section for more details).
1. Start the application with this command:
```
@@ -518,11 +518,35 @@ The metron-rest application will be available at http://localhost:8080/swagger-u
To run the application locally on the Quick Dev host, package the application and scp the archive to node1:
```
mvn clean package
-scp ./target/metron-rest-$METRON_VERSION-archive.tar.gz root@node1:~/
+scp ./target/metron-rest-$METRON_VERSION-archive.tar.gz root@node1:$METRON_HOME
```
-Login to node1 and unarchive the metron-rest application. Start the application on a different port to avoid conflicting with Ambari:
+Login to node1 and unarchive the metron-rest application:
```
-java -jar ./lib/metron-rest-$METRON_VERSION.jar --spring.profiles.active=vagrant,dev --server.port=8082
+ssh root@node1
+cd $METRON_HOME && tar xf ./metron-rest-$METRON_VERSION-archive.tar.gz
+```
+Start the application on a different port to avoid conflicting with Ambari:
+```
+java -jar $METRON_HOME/lib/metron-rest-$METRON_VERSION.jar --spring.profiles.active=vagrant,dev --server.port=8082
+```
+In a cluster with Kerberos enabled, first add metron-rest to the Kafka acls:
+```
+sudo su -
+export ZOOKEEPER=node1
+export BROKERLIST=node1
+export HDP_HOME="/usr/hdp/current"
+export METRON_VERSION="0.4.0"
+export METRON_HOME="/usr/metron/${METRON_VERSION}"
+${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:metron --topic ambari_kafka_service_check
+${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:metron --topic __consumer_offsets
+${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:metron --group metron-rest
+```
+
+Then start the application as the metron user while including references to the jaas and krb5.confg files and enabling kerberos support:
+```
+su metron
+cd ~
+java -Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf -Djava.security.krb5.conf=/etc/krb5.conf -jar $METRON_HOME/lib/metron-rest-$METRON_VERSION.jar --spring.profiles.active=vagrant,dev --server.port=8082 --kerberos.enabled=true
```
The metron-rest application will be available at http://node1:8082/swagger-ui.html#/.
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/5dd87886/metron-interface/metron-rest/pom.xml
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/pom.xml b/metron-interface/metron-rest/pom.xml
index 421b7fe..cc53192 100644
--- a/metron-interface/metron-rest/pom.xml
+++ b/metron-interface/metron-rest/pom.xml
@@ -29,6 +29,7 @@
<curator.version>2.7.1</curator.version>
<powermock.version>1.6.4</powermock.version>
<spring.boot.version>1.4.1.RELEASE</spring.boot.version>
+ <spring.kerberos.version>1.0.1.RELEASE</spring.kerberos.version>
<swagger.version>2.5.0</swagger.version>
<mysql.client.version>5.1.40</mysql.client.version>
<emma.version>1.0-alpha-3</emma.version>
@@ -57,6 +58,11 @@
<artifactId>spring-boot-starter-security</artifactId>
</dependency>
<dependency>
+ <groupId>org.springframework.security.kerberos</groupId>
+ <artifactId>spring-security-kerberos-client</artifactId>
+ <version>${spring.kerberos.version}</version>
+ </dependency>
+ <dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jdbc</artifactId>
</dependency>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/5dd87886/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/MetronRestConstants.java
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/MetronRestConstants.java b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/MetronRestConstants.java
index 8786222..2a147e6 100644
--- a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/MetronRestConstants.java
+++ b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/MetronRestConstants.java
@@ -52,4 +52,8 @@ public class MetronRestConstants {
public static final String HDFS_URL_SPRING_PROPERTY = "hdfs.namenode.url";
public static final String DEFAULT_HDFS_URL = "file:///";
+
+ public static final String KERBEROS_ENABLED_SPRING_PROPERTY = "kerberos.enabled";
+ public static final String KERBEROS_PRINCIPLE_SPRING_PROPERTY = "kerberos.principal";
+ public static final String KERBEROS_KEYTAB_SPRING_PROPERTY = "kerberos.keytab";
}
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/5dd87886/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/HadoopConfig.java
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/HadoopConfig.java b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/HadoopConfig.java
index f4b8bdd..7b1bf2f 100644
--- a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/HadoopConfig.java
+++ b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/HadoopConfig.java
@@ -17,22 +17,36 @@
*/
package org.apache.metron.rest.config;
+import org.apache.hadoop.security.UserGroupInformation;
import org.apache.metron.rest.MetronRestConstants;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.env.Environment;
+import java.io.IOException;
+
@Configuration
public class HadoopConfig {
- @Autowired
private Environment environment;
+ @Autowired
+ public HadoopConfig(Environment environment) {
+ this.environment = environment;
+ }
+
@Bean
- public org.apache.hadoop.conf.Configuration configuration() {
+ public org.apache.hadoop.conf.Configuration configuration() throws IOException {
org.apache.hadoop.conf.Configuration configuration = new org.apache.hadoop.conf.Configuration();
configuration.set("fs.defaultFS", environment.getProperty(MetronRestConstants.HDFS_URL_SPRING_PROPERTY, MetronRestConstants.DEFAULT_HDFS_URL));
+ if (environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)) {
+ configuration.set("hadoop.security.authentication", "KERBEROS");
+ UserGroupInformation.setConfiguration(configuration);
+ String keyTabLocation = environment.getProperty(MetronRestConstants.KERBEROS_KEYTAB_SPRING_PROPERTY);
+ String userPrincipal = environment.getProperty(MetronRestConstants.KERBEROS_PRINCIPLE_SPRING_PROPERTY);
+ UserGroupInformation.loginUserFromKeytab(userPrincipal, keyTabLocation);
+ }
return configuration;
}
}
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/5dd87886/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/KafkaConfig.java
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/KafkaConfig.java b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/KafkaConfig.java
index f6ff73c..309a549 100644
--- a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/KafkaConfig.java
+++ b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/KafkaConfig.java
@@ -36,10 +36,14 @@ import static org.apache.metron.rest.MetronRestConstants.TEST_PROFILE;
@Profile("!" + TEST_PROFILE)
public class KafkaConfig {
- @Autowired
private Environment environment;
@Autowired
+ public KafkaConfig(Environment environment) {
+ this.environment = environment;
+ }
+
+ @Autowired
private ZkClient zkClient;
@Bean
@@ -51,12 +55,15 @@ public class KafkaConfig {
public KafkaConsumer<String, String> kafkaConsumer() {
Properties props = new Properties();
props.put("bootstrap.servers", environment.getProperty(MetronRestConstants.KAFKA_BROKER_URL_SPRING_PROPERTY));
- props.put("group.id", "metron-config");
+ props.put("group.id", "metron-rest");
props.put("enable.auto.commit", "false");
props.put("auto.commit.interval.ms", "1000");
props.put("session.timeout.ms", "30000");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
+ if (environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)) {
+ props.put("security.protocol", "SASL_PLAINTEXT");
+ }
return new KafkaConsumer<>(props);
}
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/5dd87886/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/RestTemplateConfig.java
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/RestTemplateConfig.java b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/RestTemplateConfig.java
index 8fea90c..7443ccd 100644
--- a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/RestTemplateConfig.java
+++ b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/RestTemplateConfig.java
@@ -17,9 +17,13 @@
*/
package org.apache.metron.rest.config;
+import org.apache.metron.rest.MetronRestConstants;
+import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;
+import org.springframework.core.env.Environment;
+import org.springframework.security.kerberos.client.KerberosRestTemplate;
import org.springframework.web.client.RestTemplate;
import static org.apache.metron.rest.MetronRestConstants.TEST_PROFILE;
@@ -28,9 +32,23 @@ import static org.apache.metron.rest.MetronRestConstants.TEST_PROFILE;
@Profile("!" + TEST_PROFILE)
public class RestTemplateConfig {
+ private Environment environment;
+
+ @Autowired
+ public RestTemplateConfig(Environment environment) {
+ this.environment = environment;
+ }
+
@Bean
public RestTemplate restTemplate() {
- return new RestTemplate();
+ if (environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)) {
+ String keyTabLocation = environment.getProperty(MetronRestConstants.KERBEROS_KEYTAB_SPRING_PROPERTY);
+ String userPrincipal = environment.getProperty(MetronRestConstants.KERBEROS_PRINCIPLE_SPRING_PROPERTY);
+ return new KerberosRestTemplate(keyTabLocation, userPrincipal);
+ } else {
+ return new RestTemplate();
+ }
+
}
}
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/5dd87886/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java
index c5569c5..1158721 100644
--- a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java
+++ b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java
@@ -43,26 +43,32 @@ public class StormCLIWrapper {
}
public int startParserTopology(String name) throws RestException {
+ kinit();
return runCommand(getParserStartCommand(name));
}
public int stopParserTopology(String name, boolean stopNow) throws RestException {
+ kinit();
return runCommand(getStopCommand(name, stopNow));
}
public int startEnrichmentTopology() throws RestException {
+ kinit();
return runCommand(getEnrichmentStartCommand());
}
public int stopEnrichmentTopology(boolean stopNow) throws RestException {
+ kinit();
return runCommand(getStopCommand(ENRICHMENT_TOPOLOGY_NAME, stopNow));
}
public int startIndexingTopology() throws RestException {
+ kinit();
return runCommand(getIndexingStartCommand());
}
public int stopIndexingTopology(boolean stopNow) throws RestException {
+ kinit();
return runCommand(getStopCommand(INDEXING_TOPOLOGY_NAME, stopNow));
}
@@ -150,5 +156,14 @@ public class StormCLIWrapper {
return stormClientVersionInstalled;
}
+ protected void kinit() throws RestException {
+ if (environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)) {
+ String keyTabLocation = environment.getProperty(MetronRestConstants.KERBEROS_KEYTAB_SPRING_PROPERTY);
+ String userPrincipal = environment.getProperty(MetronRestConstants.KERBEROS_PRINCIPLE_SPRING_PROPERTY);
+ String[] kinitCommand = {"kinit", "-kt", keyTabLocation, userPrincipal};
+ runCommand(kinitCommand);
+ }
+ }
+
}
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/5dd87886/metron-interface/metron-rest/src/main/resources/application-vagrant.yml
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/main/resources/application-vagrant.yml b/metron-interface/metron-rest/src/main/resources/application-vagrant.yml
index 76bef56..158886f 100644
--- a/metron-interface/metron-rest/src/main/resources/application-vagrant.yml
+++ b/metron-interface/metron-rest/src/main/resources/application-vagrant.yml
@@ -49,3 +49,9 @@ storm:
script.path: /usr/metron/${metron.version}/bin/start_enrichment_topology.sh
indexing:
script.path: /usr/metron/${metron.version}/bin/start_elasticsearch_topology.sh
+
+kerberos:
+ enabled: false
+ principal: metron@EXAMPLE.COM
+ keytab: /etc/security/keytabs/metron.headless.keytab
+
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/5dd87886/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/config/HadoopConfigTest.java
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/config/HadoopConfigTest.java b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/config/HadoopConfigTest.java
new file mode 100644
index 0000000..c262783
--- /dev/null
+++ b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/config/HadoopConfigTest.java
@@ -0,0 +1,89 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.rest.config;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.metron.rest.MetronRestConstants;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.powermock.core.classloader.annotations.PrepareForTest;
+import org.powermock.modules.junit4.PowerMockRunner;
+import org.springframework.core.env.Environment;
+
+import java.io.IOException;
+
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.anyString;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.reset;
+import static org.mockito.Mockito.when;
+
+import static org.powermock.api.mockito.PowerMockito.mockStatic;
+import static org.powermock.api.mockito.PowerMockito.verifyStatic;
+
+@RunWith(PowerMockRunner.class)
+@PrepareForTest({HadoopConfig.class, UserGroupInformation.class})
+public class HadoopConfigTest {
+
+ private Environment environment;
+ private HadoopConfig hadoopConfig;
+
+ @Before
+ public void setUp() throws Exception {
+ environment = mock(Environment.class);
+ hadoopConfig = new HadoopConfig(environment);
+ mockStatic(UserGroupInformation.class);
+ }
+
+ @Test
+ public void configurationShouldReturnProperKerberosConfiguration() throws IOException {
+ when(environment.getProperty(MetronRestConstants.HDFS_URL_SPRING_PROPERTY, MetronRestConstants.DEFAULT_HDFS_URL)).thenReturn("default filesystem");
+ when(environment.getProperty(MetronRestConstants.KERBEROS_KEYTAB_SPRING_PROPERTY)).thenReturn("metron keytabLocation");
+ when(environment.getProperty(MetronRestConstants.KERBEROS_PRINCIPLE_SPRING_PROPERTY)).thenReturn("metron principal");
+
+ when(environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)).thenReturn(true);
+
+ Configuration configuration = hadoopConfig.configuration();
+
+ verifyStatic();
+ UserGroupInformation.setConfiguration(any(Configuration.class));
+ UserGroupInformation.loginUserFromKeytab("metron keytabLocation", "metron principal");
+
+ assertEquals("default filesystem", configuration.get("fs.defaultFS"));
+ assertEquals("KERBEROS", configuration.get("hadoop.security.authentication"));
+ }
+
+ @Test
+ public void configurationShouldReturnProperConfiguration() throws IOException {
+ when(environment.getProperty(MetronRestConstants.HDFS_URL_SPRING_PROPERTY, MetronRestConstants.DEFAULT_HDFS_URL)).thenReturn("default filesystem");
+ when(environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)).thenReturn(false);
+
+ Configuration configuration = hadoopConfig.configuration();
+
+ verifyStatic(never());
+ UserGroupInformation.setConfiguration(any(Configuration.class));
+ UserGroupInformation.loginUserFromKeytab(anyString(), anyString());
+
+ assertEquals("default filesystem", configuration.get("fs.defaultFS"));
+ assertEquals("simple", configuration.get("hadoop.security.authentication"));
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/5dd87886/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/config/RestTemplateConfigTest.java
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/config/RestTemplateConfigTest.java b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/config/RestTemplateConfigTest.java
new file mode 100644
index 0000000..0d9d620
--- /dev/null
+++ b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/config/RestTemplateConfigTest.java
@@ -0,0 +1,67 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.rest.config;
+
+import org.apache.metron.rest.MetronRestConstants;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.powermock.core.classloader.annotations.PrepareForTest;
+import org.powermock.modules.junit4.PowerMockRunner;
+import org.springframework.core.env.Environment;
+import org.springframework.security.kerberos.client.KerberosRestTemplate;
+import org.springframework.web.client.RestTemplate;
+
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.powermock.api.mockito.PowerMockito.verifyNew;
+import static org.powermock.api.mockito.PowerMockito.whenNew;
+
+@RunWith(PowerMockRunner.class)
+@PrepareForTest({RestTemplateConfig.class, KerberosRestTemplate.class, RestTemplate.class})
+public class RestTemplateConfigTest {
+
+ private Environment environment;
+ private RestTemplateConfig restTemplateConfig;
+
+ @Before
+ public void setUp() throws Exception {
+ environment = mock(Environment.class);
+ restTemplateConfig = new RestTemplateConfig(environment);
+ }
+
+ @Test
+ public void restTemplateShouldReturnProperTemplate() throws Exception {
+ when(environment.getProperty(MetronRestConstants.KERBEROS_KEYTAB_SPRING_PROPERTY)).thenReturn("metron keytabLocation");
+ when(environment.getProperty(MetronRestConstants.KERBEROS_PRINCIPLE_SPRING_PROPERTY)).thenReturn("metron principal");
+
+ whenNew(KerberosRestTemplate.class).withParameterTypes(String.class, String.class).withArguments("metron keytabLocation", "metron principal").thenReturn(null);
+ when(environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)).thenReturn(true);
+
+ restTemplateConfig.restTemplate();
+ verifyNew(KerberosRestTemplate.class).withArguments("metron keytabLocation", "metron principal");
+
+ whenNew(RestTemplate.class).withNoArguments().thenReturn(null);
+ when(environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)).thenReturn(false);
+
+ restTemplateConfig.restTemplate();
+ verifyNew(RestTemplate.class).withNoArguments();
+ }
+
+
+}
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/5dd87886/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/StormCLIWrapperTest.java
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/StormCLIWrapperTest.java b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/StormCLIWrapperTest.java
index cb26783..9fb067a 100644
--- a/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/StormCLIWrapperTest.java
+++ b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/StormCLIWrapperTest.java
@@ -24,6 +24,7 @@ import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
import org.junit.runner.RunWith;
+import org.mockito.Mockito;
import org.powermock.core.classloader.annotations.PrepareForTest;
import org.powermock.modules.junit4.PowerMockRunner;
import org.springframework.core.env.Environment;
@@ -37,6 +38,7 @@ import java.util.Map;
import static java.nio.charset.StandardCharsets.UTF_8;
import static org.junit.Assert.assertEquals;
import static org.mockito.Matchers.anyVararg;
+import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.powermock.api.mockito.PowerMockito.mock;
import static org.powermock.api.mockito.PowerMockito.verifyNew;
@@ -73,6 +75,7 @@ public class StormCLIWrapperTest {
when(environment.getProperty(MetronRestConstants.PARSER_SCRIPT_PATH_SPRING_PROPERTY)).thenReturn("/start_parser");
when(environment.getProperty(MetronRestConstants.KAFKA_BROKER_URL_SPRING_PROPERTY)).thenReturn("kafka_broker_url");
when(environment.getProperty(MetronRestConstants.ZK_URL_SPRING_PROPERTY)).thenReturn("zookeeper_url");
+ when(environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)).thenReturn(false);
when(process.exitValue()).thenReturn(0);
assertEquals(0, stormCLIWrapper.startParserTopology("bro"));
@@ -85,6 +88,7 @@ public class StormCLIWrapperTest {
whenNew(ProcessBuilder.class).withParameterTypes(String[].class).withArguments(anyVararg()).thenReturn(processBuilder);
when(processBuilder.start()).thenReturn(process);
+ when(environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)).thenReturn(false);
when(process.exitValue()).thenReturn(0);
assertEquals(0, stormCLIWrapper.stopParserTopology("bro", false));
@@ -97,6 +101,7 @@ public class StormCLIWrapperTest {
whenNew(ProcessBuilder.class).withParameterTypes(String[].class).withArguments(anyVararg()).thenReturn(processBuilder);
when(processBuilder.start()).thenReturn(process);
+ when(environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)).thenReturn(false);
when(process.exitValue()).thenReturn(0);
assertEquals(0, stormCLIWrapper.stopParserTopology("bro", true));
@@ -110,6 +115,7 @@ public class StormCLIWrapperTest {
when(processBuilder.start()).thenReturn(process);
when(environment.getProperty(MetronRestConstants.ENRICHMENT_SCRIPT_PATH_SPRING_PROPERTY)).thenReturn("/start_enrichment");
+ when(environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)).thenReturn(false);
when(process.exitValue()).thenReturn(0);
assertEquals(0, stormCLIWrapper.startEnrichmentTopology());
@@ -123,6 +129,7 @@ public class StormCLIWrapperTest {
whenNew(ProcessBuilder.class).withParameterTypes(String[].class).withArguments(anyVararg()).thenReturn(processBuilder);
when(processBuilder.start()).thenReturn(process);
+ when(environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)).thenReturn(false);
when(process.exitValue()).thenReturn(0);
assertEquals(0, stormCLIWrapper.stopEnrichmentTopology(false));
@@ -136,6 +143,7 @@ public class StormCLIWrapperTest {
when(processBuilder.start()).thenReturn(process);
when(environment.getProperty(MetronRestConstants.INDEXING_SCRIPT_PATH_SPRING_PROPERTY)).thenReturn("/start_indexing");
+ when(environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)).thenReturn(false);
when(process.exitValue()).thenReturn(0);
assertEquals(0, stormCLIWrapper.startIndexingTopology());
@@ -149,6 +157,7 @@ public class StormCLIWrapperTest {
whenNew(ProcessBuilder.class).withParameterTypes(String[].class).withArguments(anyVararg()).thenReturn(processBuilder);
when(processBuilder.start()).thenReturn(process);
+ when(environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)).thenReturn(false);
when(process.exitValue()).thenReturn(0);
assertEquals(0, stormCLIWrapper.stopIndexingTopology(false));
@@ -213,4 +222,19 @@ public class StormCLIWrapperTest {
stormCLIWrapper.stormClientVersionInstalled();
}
+
+ @Test
+ public void kinitShouldRunCommandProperly() throws Exception {
+ whenNew(ProcessBuilder.class).withParameterTypes(String[].class).withArguments(anyVararg()).thenReturn(processBuilder);
+
+ when(processBuilder.start()).thenReturn(process);
+ when(environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)).thenReturn(true);
+ when(environment.getProperty(MetronRestConstants.KERBEROS_KEYTAB_SPRING_PROPERTY)).thenReturn("metron keytabLocation");
+ when(environment.getProperty(MetronRestConstants.KERBEROS_PRINCIPLE_SPRING_PROPERTY)).thenReturn("metron principal");
+ when(process.exitValue()).thenReturn(0);
+
+ stormCLIWrapper.kinit();
+ verify(process, times(1)).waitFor();
+ verifyNew(ProcessBuilder.class).withArguments("kinit", "-kt", "metron keytabLocation", "metron principal");
+ }
}
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/5dd87886/metron-interface/metron-rest/src/test/resources/README.vm
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/test/resources/README.vm b/metron-interface/metron-rest/src/test/resources/README.vm
index ca54e6a..cff7ca7 100644
--- a/metron-interface/metron-rest/src/test/resources/README.vm
+++ b/metron-interface/metron-rest/src/test/resources/README.vm
@@ -22,7 +22,7 @@ This module provides a RESTful API for interacting with Metron.
metron-rest-$METRON_VERSION.jar
```
-1. Create an `application.yml` file with the contents of [application-docker.yml](src/main/resources/application-docker.yml). Substitute the appropriate Metron service urls (Kafka, Zookeeper, Storm, etc.) in properties containing `${docker.host.address}` and update the `spring.datasource.*` properties as needed (see the [Security](#security) section for more details).
+1. Create an `application.yml` file with the contents of [application-vagrant.yml](src/main/resources/application-vagrant.yml). Substitute the appropriate Metron service hosts (Kafka, Zookeeper, Storm, etc.) in properties containing `node1` and update the `spring.datasource.*` properties as needed (see the [Security](#security) section for more details).
1. Start the application with this command:
```
@@ -128,11 +128,35 @@ The metron-rest application will be available at http://localhost:8080/swagger-u
To run the application locally on the Quick Dev host, package the application and scp the archive to node1:
```
mvn clean package
-scp ./target/metron-rest-$METRON_VERSION-archive.tar.gz root@node1:~/
+scp ./target/metron-rest-$METRON_VERSION-archive.tar.gz root@node1:$METRON_HOME
```
-Login to node1 and unarchive the metron-rest application. Start the application on a different port to avoid conflicting with Ambari:
+Login to node1 and unarchive the metron-rest application:
```
-java -jar ./lib/metron-rest-$METRON_VERSION.jar --spring.profiles.active=vagrant,dev --server.port=8082
+ssh root@node1
+cd $METRON_HOME && tar xf ./metron-rest-$METRON_VERSION-archive.tar.gz
+```
+Start the application on a different port to avoid conflicting with Ambari:
+```
+java -jar $METRON_HOME/lib/metron-rest-$METRON_VERSION.jar --spring.profiles.active=vagrant,dev --server.port=8082
+```
+In a cluster with Kerberos enabled, first add metron-rest to the Kafka acls:
+```
+sudo su -
+export ZOOKEEPER=node1
+export BROKERLIST=node1
+export HDP_HOME="/usr/hdp/current"
+export METRON_VERSION="0.4.0"
+export METRON_HOME="/usr/metron/${METRON_VERSION}"
+${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:metron --topic ambari_kafka_service_check
+${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:metron --topic __consumer_offsets
+${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:metron --group metron-rest
+```
+
+Then start the application as the metron user while including references to the jaas and krb5.confg files and enabling kerberos support:
+```
+su metron
+cd ~
+java -Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf -Djava.security.krb5.conf=/etc/krb5.conf -jar $METRON_HOME/lib/metron-rest-$METRON_VERSION.jar --spring.profiles.active=vagrant,dev --server.port=8082 --kerberos.enabled=true
```
The metron-rest application will be available at http://node1:8082/swagger-ui.html#/.