You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by rkarthik29 <gi...@git.apache.org> on 2016/11/02 19:13:25 UTC
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
GitHub user rkarthik29 opened a pull request:
https://github.com/apache/incubator-metron/pull/338
METRON-295: Script parsing bolt
addd support for js,python and groovy parser bolt
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rkarthik29/incubator-metron metron-295
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/338.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #338
----
commit 754a1ba7b579d4507aa728d6df17c535130c80d4
Author: Kathik Narayanan <kn...@hortonworks.com>
Date: 2016-11-01T20:36:33Z
working on js parser
commit f4d0cec86cbf3151791dad660e2e13d64007b71e
Author: Kathik Narayanan <kn...@hortonworks.com>
Date: 2016-11-02T14:24:58Z
renamed JSParser to ScriptPArser
commit 91bf93d436d95877f2633518ca5393510f590d11
Author: Kathik Narayanan <kn...@hortonworks.com>
Date: 2016-11-02T18:44:34Z
script parsing support for js,python and groovy.
commit 2f63772d50cbff4232701309d58fa4c82857634a
Author: Kathik Narayanan <kn...@hortonworks.com>
Date: 2016-11-02T19:02:47Z
Merge branch 'metron-295' of https://github.com/rkarthik29/incubator-metron into metron-295
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/338#discussion_r86225277
--- Diff: metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/GroovyParserTest.java ---
@@ -0,0 +1,81 @@
+package org.apache.metron.parsers;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.adrianwalker.multilinestring.Multiline;
+
+public class GroovyParserTest extends ScriptParserTest {
+
+ /**
+ {
+ "customerId":117,
+ "first_name":"karthik",
+ "last_name":"narayanan",
+ "age":"38",
+ "login-time":"2016-01-28 15:29:48",
+ "ip-address":"216.21.170.221",
+ "os":"windows 10",
+ "device":"Dell Inspiron"
+ }
+ */
+ @Multiline
+ public String result=" {"+
+ "\"source\":\"userlog\","+
+ "\"customerId\":117,"+
+ "\"first_name\":\"karthik\","+
+ "\"last_name\":\"narayanan\","+
+ "\"age\":\"38\","+
+ "\"login-time\":\"2016-01-28 15:29:48\","+
+ "\"ip-address\":\"216.21.170.221\","+
+ "\"os\":\"windows 10\","+
+ "\"device\":\"Dell Inspiron\""+
+ "\"original_string\":\"2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron\""+
+ "}";
+
+ @Override
+ public Map getTestData() {
+ Map testData = new HashMap<String,String>();
+ String input = "2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron";
+ testData.put(input,result);
+ return testData;
+ }
+
+ @Override
+ public String getScriptPath() {
+ // TODO Auto-generated method stub
+ return "../metron-integration-test/src/main/sample/scripts/test.groovy";
+ }
+
+ @Override
+ public String getParseFunction() {
+ // TODO Auto-generated method stub
+ return "parse";
+ }
+
+ @Override
+ public String getLanguage() {
+ // TODO Auto-generated method stub
+ return "groovy";
+ }
+
+ @Override
+ public List<String> getTimeFields() {
+ // TODO Auto-generated method stub
--- End diff --
Remove
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/338#discussion_r86225269
--- Diff: metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/GroovyParserTest.java ---
@@ -0,0 +1,81 @@
+package org.apache.metron.parsers;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.adrianwalker.multilinestring.Multiline;
+
+public class GroovyParserTest extends ScriptParserTest {
+
+ /**
+ {
+ "customerId":117,
+ "first_name":"karthik",
+ "last_name":"narayanan",
+ "age":"38",
+ "login-time":"2016-01-28 15:29:48",
+ "ip-address":"216.21.170.221",
+ "os":"windows 10",
+ "device":"Dell Inspiron"
+ }
+ */
+ @Multiline
+ public String result=" {"+
+ "\"source\":\"userlog\","+
+ "\"customerId\":117,"+
+ "\"first_name\":\"karthik\","+
+ "\"last_name\":\"narayanan\","+
+ "\"age\":\"38\","+
+ "\"login-time\":\"2016-01-28 15:29:48\","+
+ "\"ip-address\":\"216.21.170.221\","+
+ "\"os\":\"windows 10\","+
+ "\"device\":\"Dell Inspiron\""+
+ "\"original_string\":\"2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron\""+
+ "}";
+
+ @Override
+ public Map getTestData() {
+ Map testData = new HashMap<String,String>();
+ String input = "2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron";
+ testData.put(input,result);
+ return testData;
+ }
+
+ @Override
+ public String getScriptPath() {
+ // TODO Auto-generated method stub
+ return "../metron-integration-test/src/main/sample/scripts/test.groovy";
+ }
+
+ @Override
+ public String getParseFunction() {
+ // TODO Auto-generated method stub
+ return "parse";
+ }
+
+ @Override
+ public String getLanguage() {
+ // TODO Auto-generated method stub
+ return "groovy";
+ }
+
+ @Override
+ public List<String> getTimeFields() {
+ // TODO Auto-generated method stub
+ return null;
+ }
+
+ @Override
+ public String getDateFormat() {
+ // TODO Auto-generated method stub
--- End diff --
Remove
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #338: METRON-295: Script parsing bolt
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/338
yeah, as a Minimum Viable, but those limitations need to be spelled out, per language. Actually all the points above may have per language implications
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #338: METRON-295: Script parsing bolt
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/338
@ottobackwards Yeah, that's sensible. As far as the features, I think it's ok for this to support the JVM-supported scripting languages with all their limitations (IMO).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/338#discussion_r86225544
--- Diff: metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/PythonParserTest.java ---
@@ -0,0 +1,81 @@
+package org.apache.metron.parsers;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.adrianwalker.multilinestring.Multiline;
+
+public class PythonParserTest extends ScriptParserTest {
+
+ /**
+ {
+ "customerId":117,
+ "first_name":"karthik",
+ "last_name":"narayanan",
+ "age":"38",
+ "login-time":"2016-01-28 15:29:48",
+ "ip-address":"216.21.170.221",
+ "os":"windows 10",
+ "device":"Dell Inspiron"
+ }
+ */
+ @Multiline
+ public String result=" {"+
+ "\"source\":\"userlog\","+
+ "\"customerId\":117,"+
+ "\"first_name\":\"karthik\","+
+ "\"last_name\":\"narayanan\","+
+ "\"age\":\"38\","+
+ "\"login-time\":\"2016-01-28 15:29:48\","+
+ "\"ip-address\":\"216.21.170.221\","+
+ "\"os\":\"windows 10\","+
+ "\"device\":\"Dell Inspiron\""+
+ "\"original_string\":\"2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron\""+
+ "}";
+
+ @Override
+ public Map getTestData() {
+ Map testData = new HashMap<String,String>();
+ String input = "2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron";
+ testData.put(input,result);
+ return testData;
+ }
+
+ @Override
+ public String getScriptPath() {
+ // TODO Auto-generated method stub
+ return "../metron-integration-test/src/main/sample/scripts/test.py";
+ }
+
+ @Override
+ public String getParseFunction() {
+ // TODO Auto-generated method stub
+ return "parse";
+ }
+
+ @Override
+ public String getLanguage() {
+ // TODO Auto-generated method stub
+ return "python";
+ }
+
+ @Override
+ public List<String> getTimeFields() {
+ // TODO Auto-generated method stub
--- End diff --
Remove
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/338#discussion_r86225562
--- Diff: metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/PythonParserTest.java ---
@@ -0,0 +1,81 @@
+package org.apache.metron.parsers;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.adrianwalker.multilinestring.Multiline;
+
+public class PythonParserTest extends ScriptParserTest {
+
+ /**
+ {
+ "customerId":117,
+ "first_name":"karthik",
+ "last_name":"narayanan",
+ "age":"38",
+ "login-time":"2016-01-28 15:29:48",
+ "ip-address":"216.21.170.221",
+ "os":"windows 10",
+ "device":"Dell Inspiron"
+ }
+ */
+ @Multiline
+ public String result=" {"+
+ "\"source\":\"userlog\","+
+ "\"customerId\":117,"+
+ "\"first_name\":\"karthik\","+
+ "\"last_name\":\"narayanan\","+
+ "\"age\":\"38\","+
+ "\"login-time\":\"2016-01-28 15:29:48\","+
+ "\"ip-address\":\"216.21.170.221\","+
+ "\"os\":\"windows 10\","+
+ "\"device\":\"Dell Inspiron\""+
+ "\"original_string\":\"2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron\""+
+ "}";
+
+ @Override
+ public Map getTestData() {
+ Map testData = new HashMap<String,String>();
+ String input = "2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron";
+ testData.put(input,result);
+ return testData;
+ }
+
+ @Override
+ public String getScriptPath() {
+ // TODO Auto-generated method stub
+ return "../metron-integration-test/src/main/sample/scripts/test.py";
+ }
+
+ @Override
+ public String getParseFunction() {
+ // TODO Auto-generated method stub
+ return "parse";
+ }
+
+ @Override
+ public String getLanguage() {
+ // TODO Auto-generated method stub
+ return "python";
+ }
+
+ @Override
+ public List<String> getTimeFields() {
+ // TODO Auto-generated method stub
+ return null;
+ }
+
+ @Override
+ public String getDateFormat() {
+ // TODO Auto-generated method stub
--- End diff --
Remove
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
Posted by kylerichardson <gi...@git.apache.org>.
Github user kylerichardson commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/338#discussion_r87228757
--- Diff: metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/ScriptParser.java ---
@@ -0,0 +1,189 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.Serializable;
+import java.text.ParseException;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+
+import javax.script.Invocable;
+import javax.script.ScriptEngine;
+import javax.script.ScriptEngineFactory;
+import javax.script.ScriptEngineManager;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.metron.common.Constants;
+import org.apache.metron.parsers.interfaces.MessageParser;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+
+public class ScriptParser implements MessageParser<JSONObject>,Serializable{
+
+ protected static final Logger LOG = LoggerFactory.getLogger(ScriptParser.class);
+ protected String scriptPath;
+ protected ScriptEngine engine;
+ protected String parseFunction;
+ protected String language;
+ protected String commonScript="/scripts/";
+ protected List<String> timeFields = new ArrayList<>();
+ protected String timestampField;
+ protected SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.S z");
+
+ @Override
+ public void configure(Map<String, Object> config) {
+ this.scriptPath=(String) config.get("path");
+ this.parseFunction=(String)config.get("function");
+ this.language=(String)config.get("language");
+ this.commonScript=this.commonScript+language+"/common";
+ if(this.parseFunction==null)
+ this.parseFunction="parse";
+ }
+ //Should this be sent to the interface as a default method?
+ public InputStream openInputStream(String streamName) throws IOException {
+ FileSystem fs = FileSystem.get(new Configuration());
+ Path path = new Path(streamName);
+ if(fs.exists(path)) {
+ return fs.open(path);
+ } else {
+ return getClass().getResourceAsStream(streamName);
+ }
+ }
+
+ @Override
+ public void init() {
+ engine = new ScriptEngineManager().getEngineByName(this.language);
+ try{
+ InputStream commonStream = openInputStream(this.commonScript);
+ if (commonStream == null) {
+ throw new RuntimeException(
+ "Unable to initialize "+this.language+" Parser: Unable to load " + this.commonScript + " from either classpath or HDFS");
+ }
+
+ engine.eval(new InputStreamReader(commonStream));
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Loading parser-specific functions from: " + this.scriptPath);
+ }
+
+ InputStream patterInputStream = openInputStream(this.scriptPath);
+ if (patterInputStream == null) {
+ throw new RuntimeException("Script parser unable to initialize "+this.language+" parser: Unable to load " + this.scriptPath
+ + " from either classpath or HDFS");
+ }
+ engine.eval(new InputStreamReader(patterInputStream));
+ }catch(Throwable e){
+ LOG.error(e.getMessage(), e);
+ throw new RuntimeException(this.language+" Script parser Error: " + e.getMessage(), e);
+ }
+
+ Invocable invocable = (Invocable) engine;
+
+ }
+
+ @Override
+ public List<JSONObject> parse(byte[] rawMessage) {
+ List<JSONObject> messages = new ArrayList<JSONObject>();
+ if(engine==null)
+ init();
+ String originalMessage = null;
+ try{
+ originalMessage = new String(rawMessage, "UTF-8");
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Script parser parsing message: " + originalMessage);
+ }
+ Invocable invocable = (Invocable) engine;
+
+ Object result = invocable.invokeFunction(this.parseFunction,originalMessage);
+ JSONObject message = new JSONObject((Map)result);
+
+ if (message.size() == 0)
+ throw new RuntimeException("Script produced a null message. Original message was: "
+ + originalMessage + " and the parsed message was: " + message + " . Check the function at: "
+ + this.scriptPath);
+
+ message.put("original_string", originalMessage);
--- End diff --
To move toward consistency, can this "original_string" field be updated to use the new `Constants.Fields.ORIGINAL.getName()`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/338#discussion_r86223969
--- Diff: metron-platform/metron-integration-test/.pydevproject ---
@@ -0,0 +1,5 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
--- End diff --
I think this should go in the .gitignore
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/338#discussion_r86225504
--- Diff: metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/PythonParserTest.java ---
@@ -0,0 +1,81 @@
+package org.apache.metron.parsers;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.adrianwalker.multilinestring.Multiline;
+
+public class PythonParserTest extends ScriptParserTest {
+
+ /**
+ {
+ "customerId":117,
+ "first_name":"karthik",
+ "last_name":"narayanan",
+ "age":"38",
+ "login-time":"2016-01-28 15:29:48",
+ "ip-address":"216.21.170.221",
+ "os":"windows 10",
+ "device":"Dell Inspiron"
+ }
+ */
+ @Multiline
+ public String result=" {"+
+ "\"source\":\"userlog\","+
+ "\"customerId\":117,"+
+ "\"first_name\":\"karthik\","+
+ "\"last_name\":\"narayanan\","+
+ "\"age\":\"38\","+
+ "\"login-time\":\"2016-01-28 15:29:48\","+
+ "\"ip-address\":\"216.21.170.221\","+
+ "\"os\":\"windows 10\","+
+ "\"device\":\"Dell Inspiron\""+
+ "\"original_string\":\"2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron\""+
+ "}";
+
+ @Override
+ public Map getTestData() {
+ Map testData = new HashMap<String,String>();
+ String input = "2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron";
+ testData.put(input,result);
+ return testData;
+ }
+
+ @Override
+ public String getScriptPath() {
+ // TODO Auto-generated method stub
--- End diff --
Remove
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
Posted by kylerichardson <gi...@git.apache.org>.
Github user kylerichardson commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/338#discussion_r87221814
--- Diff: metron-platform/metron-parsers/src/main/resources/scripts/groovy/common ---
@@ -0,0 +1,23 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+def MetronMessage(){
+ def message=[:];
+ message["source"]="userlog";
--- End diff --
For the other languages you have passed in a name variable and used it for the source, can we do that here to be consistent?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #338: METRON-295: Script parsing bolt
Posted by randerzander <gi...@git.apache.org>.
Github user randerzander commented on the issue:
https://github.com/apache/incubator-metron/pull/338
@ottobackwards , Python's standard libraries are useful by themselves.
Since first tasks with Metron seem often related to ingesting a new source, Python lowers the barrier to entry over Java or Grok parsers.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/338#discussion_r86224685
--- Diff: metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/ScriptParser.java ---
@@ -0,0 +1,175 @@
+package org.apache.metron.parsers;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.Serializable;
+import java.text.ParseException;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+
+import javax.script.Invocable;
+import javax.script.ScriptEngine;
+import javax.script.ScriptEngineFactory;
+import javax.script.ScriptEngineManager;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.metron.common.Constants;
+import org.apache.metron.parsers.interfaces.MessageParser;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+
+public class ScriptParser implements MessageParser<JSONObject>,Serializable{
+
+ protected static final Logger LOG = LoggerFactory.getLogger(ScriptParser.class);
+ protected String scriptPath;
+ protected ScriptEngine engine;
+ protected String parseFunction;
+ protected String language;
+ protected String commonScript="/scripts/";
+ protected List<String> timeFields = new ArrayList<>();
+ protected String timestampField;
+ protected SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.S z");
+
+ @Override
+ public void configure(Map<String, Object> config) {
+ // TODO Auto-generated method stub
+ this.scriptPath=(String) config.get("path");
+ this.parseFunction=(String)config.get("function");
+ this.language=(String)config.get("language");
+ this.commonScript=this.commonScript+language+"/common";
+ if(this.parseFunction==null)
+ this.parseFunction="parse";
+ }
+ //Should this be sent to the interface as a default method?
+ public InputStream openInputStream(String streamName) throws IOException {
+ FileSystem fs = FileSystem.get(new Configuration());
+ Path path = new Path(streamName);
+ if(fs.exists(path)) {
+ return fs.open(path);
+ } else {
+ return getClass().getResourceAsStream(streamName);
+ }
+ }
+
+ @Override
+ public void init() {
+ // TODO Auto-generated method stub
--- End diff --
probably don't need that
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #338: METRON-295: Script parsing bolt
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/338
https://issues.apache.org/jira/browse/METRON-534
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #338: METRON-295: Script parsing bolt
Posted by rkarthik29 <gi...@git.apache.org>.
Github user rkarthik29 commented on the issue:
https://github.com/apache/incubator-metron/pull/338
this is exactly what we would need to do for python...
http://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/338#discussion_r86225310
--- Diff: metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/GroovyParserTest.java ---
@@ -0,0 +1,81 @@
+package org.apache.metron.parsers;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.adrianwalker.multilinestring.Multiline;
+
+public class GroovyParserTest extends ScriptParserTest {
+
+ /**
+ {
+ "customerId":117,
+ "first_name":"karthik",
+ "last_name":"narayanan",
+ "age":"38",
+ "login-time":"2016-01-28 15:29:48",
+ "ip-address":"216.21.170.221",
+ "os":"windows 10",
+ "device":"Dell Inspiron"
+ }
+ */
+ @Multiline
+ public String result=" {"+
+ "\"source\":\"userlog\","+
+ "\"customerId\":117,"+
+ "\"first_name\":\"karthik\","+
+ "\"last_name\":\"narayanan\","+
+ "\"age\":\"38\","+
+ "\"login-time\":\"2016-01-28 15:29:48\","+
+ "\"ip-address\":\"216.21.170.221\","+
+ "\"os\":\"windows 10\","+
+ "\"device\":\"Dell Inspiron\""+
+ "\"original_string\":\"2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron\""+
+ "}";
+
+ @Override
+ public Map getTestData() {
+ Map testData = new HashMap<String,String>();
+ String input = "2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron";
+ testData.put(input,result);
+ return testData;
+ }
+
+ @Override
+ public String getScriptPath() {
+ // TODO Auto-generated method stub
+ return "../metron-integration-test/src/main/sample/scripts/test.groovy";
+ }
+
+ @Override
+ public String getParseFunction() {
+ // TODO Auto-generated method stub
+ return "parse";
+ }
+
+ @Override
+ public String getLanguage() {
+ // TODO Auto-generated method stub
--- End diff --
Remove
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #338: METRON-295: Script parsing bolt
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/338
What is the workflow or steps around deploying the scripts?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #338: METRON-295: Script parsing bolt
Posted by rkarthik29 <gi...@git.apache.org>.
Github user rkarthik29 commented on the issue:
https://github.com/apache/incubator-metron/pull/338
what happens to the PR now?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/338#discussion_r86225260
--- Diff: metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/GroovyParserTest.java ---
@@ -0,0 +1,81 @@
+package org.apache.metron.parsers;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.adrianwalker.multilinestring.Multiline;
+
+public class GroovyParserTest extends ScriptParserTest {
+
+ /**
+ {
+ "customerId":117,
+ "first_name":"karthik",
+ "last_name":"narayanan",
+ "age":"38",
+ "login-time":"2016-01-28 15:29:48",
+ "ip-address":"216.21.170.221",
+ "os":"windows 10",
+ "device":"Dell Inspiron"
+ }
+ */
+ @Multiline
+ public String result=" {"+
+ "\"source\":\"userlog\","+
+ "\"customerId\":117,"+
+ "\"first_name\":\"karthik\","+
+ "\"last_name\":\"narayanan\","+
+ "\"age\":\"38\","+
+ "\"login-time\":\"2016-01-28 15:29:48\","+
+ "\"ip-address\":\"216.21.170.221\","+
+ "\"os\":\"windows 10\","+
+ "\"device\":\"Dell Inspiron\""+
+ "\"original_string\":\"2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron\""+
+ "}";
+
+ @Override
+ public Map getTestData() {
+ Map testData = new HashMap<String,String>();
+ String input = "2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron";
+ testData.put(input,result);
+ return testData;
+ }
+
+ @Override
+ public String getScriptPath() {
+ // TODO Auto-generated method stub
+ return "../metron-integration-test/src/main/sample/scripts/test.groovy";
+ }
+
+ @Override
+ public String getParseFunction() {
+ // TODO Auto-generated method stub
+ return "parse";
+ }
+
+ @Override
+ public String getLanguage() {
+ // TODO Auto-generated method stub
+ return "groovy";
+ }
+
+ @Override
+ public List<String> getTimeFields() {
+ // TODO Auto-generated method stub
+ return null;
+ }
+
+ @Override
+ public String getDateFormat() {
+ // TODO Auto-generated method stub
+ return "yyyy-MM-dd HH:mm:ss";
+ }
+
+ @Override
+ public String getTimestampField() {
+ // TODO Auto-generated method stub
--- End diff --
Remove
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #338: METRON-295: Script parsing bolt
Posted by rkarthik29 <gi...@git.apache.org>.
Github user rkarthik29 commented on the issue:
https://github.com/apache/incubator-metron/pull/338
I think that should be a separate lira. This feature does a very simple thing, it allows for a use to provide a function in the format JsonObject parse(rawMessage) in a language of his choice. The concept is that he can now use standard library functions to parse the rawMessage into JSon, which metron can then use.
dependencies for external libraries i think will need to be tested. This could be the strategy...
1. For JS script, you could just include the min.js link or just put your script in a http server and refer it in the script.
2. For groovy script, could be as simple as dropping a jar in the proper location, i guess the bolt class path.
3. for python, i think we should be able to put the modules in a modules folder and i think python script can refer to it from there, we will have to look at that packaging.
I think it is a little more complicated and i think a separate would be needed to track it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #338: METRON-295: Script parsing bolt
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/338
There has been a lot of thought and modification around updating external content in #308. It seems to me that some of this should be reused or presumed here. We have a similar issue, the need to update an external reference/execution file from zookeeper.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
Posted by kylerichardson <gi...@git.apache.org>.
Github user kylerichardson commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/338#discussion_r87221525
--- Diff: metron-platform/metron-parsers/pom.xml.orig ---
@@ -0,0 +1,370 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software
+ Foundation (ASF) under one or more contributor license agreements. See the
+ NOTICE file distributed with this work for additional information regarding
+ copyright ownership. The ASF licenses this file to You under the Apache License,
+ Version 2.0 (the "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software distributed
+ under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ OR CONDITIONS OF ANY KIND, either express or implied. See the License for
+ the specific language governing permissions and limitations under the License.
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
--- End diff --
This appears to be a local backup of the POM file. Can it be removed in favor of the git history?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #338: METRON-295: Script parsing bolt
Posted by rkarthik29 <gi...@git.apache.org>.
Github user rkarthik29 commented on the issue:
https://github.com/apache/incubator-metron/pull/338
i am coming in late to this party, but i think the core issue , how would the bolt know that the script has changed so it can reload the parser. I was thinking if the previous bolt could add aversion tag to the raw message.. The parser bolt compares this version or checksum to the last one it had, if different reruns init and stores this value into the bolt. Else just drops the version an processes the raw message. A simple checksum may be. Since the scripts can be really evolved, we could give user more leverage if we allowed them to keep in HDFS or a filesystem. even a rest api. Please let me know if i am way off.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #338: METRON-295: Script parsing bolt
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/338
And the documentation for the feature need to account for the limitations etc
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/338#discussion_r86225349
--- Diff: metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/GroovyParserTest.java ---
@@ -0,0 +1,81 @@
+package org.apache.metron.parsers;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.adrianwalker.multilinestring.Multiline;
+
+public class GroovyParserTest extends ScriptParserTest {
+
+ /**
+ {
+ "customerId":117,
+ "first_name":"karthik",
+ "last_name":"narayanan",
+ "age":"38",
+ "login-time":"2016-01-28 15:29:48",
+ "ip-address":"216.21.170.221",
+ "os":"windows 10",
+ "device":"Dell Inspiron"
+ }
+ */
+ @Multiline
+ public String result=" {"+
+ "\"source\":\"userlog\","+
+ "\"customerId\":117,"+
+ "\"first_name\":\"karthik\","+
+ "\"last_name\":\"narayanan\","+
+ "\"age\":\"38\","+
+ "\"login-time\":\"2016-01-28 15:29:48\","+
+ "\"ip-address\":\"216.21.170.221\","+
+ "\"os\":\"windows 10\","+
+ "\"device\":\"Dell Inspiron\""+
+ "\"original_string\":\"2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron\""+
+ "}";
+
+ @Override
+ public Map getTestData() {
+ Map testData = new HashMap<String,String>();
+ String input = "2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron";
+ testData.put(input,result);
+ return testData;
+ }
+
+ @Override
+ public String getScriptPath() {
+ // TODO Auto-generated method stub
+ return "../metron-integration-test/src/main/sample/scripts/test.groovy";
+ }
+
+ @Override
+ public String getParseFunction() {
+ // TODO Auto-generated method stub
--- End diff --
Remove
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #338: METRON-295: Script parsing bolt
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/338
It may be, of course, that this shouldn't be pulled from zookeeper and should be kept a file. I'm not sure how I feel either way. If we keep it a file, I am unclear how we manage to reparse the script (and collateral) upon change without bouncing the topology.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #338: METRON-295: Script parsing bolt
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/338
You can solve this problem ( and the grok problem ) without storing the world in ZK with tooling. An official set of scripts/web interface for deploying scripts/topologies/grok rules to whatever location. This tooling can change the version, checksum, crc whatever in the configurations in ZK and trigger the reload from disk.
I don't know what the real benefit from storing the scripts or rules in ZK are beyond notification. It seems to me to make things harder.
If there is no other reason, and it is only for triggering change maybe a tool/flag approach would be simpler?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #338: METRON-295: Script parsing bolt
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/338
This is interesting. I would think we would want some tooling like https://github.com/YelpArchive/pyleus around building a jar that contains all the python dependencies for a script for people to do non-trivial things in python.
I am not sure how you would do the same for javascript or groovy.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #338: METRON-295: Script parsing bolt
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/338
How do scripts with dependencies work? - I think there are still questions in the comments here that should be answered
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #338: METRON-295: Script parsing bolt
Posted by rkarthik29 <gi...@git.apache.org>.
Github user rkarthik29 commented on the issue:
https://github.com/apache/incubator-metron/pull/338
on a separate note, One feature that could really be a jira that causes code change, could be the one where we can have callbacks from these scripts, especially JS. Which means my script can call an external rest Api over AJAX and once it gets a response, will call back a function in the bolt with the JSONObject, asynchronous. This can give user tremendous amount of extendibility. May be a way to use a centralized rule engine , for their alerts. I know JS supports it, so should groovy. Not sure about python. Will the bolt support such a thing?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/338#discussion_r86225002
--- Diff: metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/ScriptParser.java ---
@@ -0,0 +1,175 @@
+package org.apache.metron.parsers;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.Serializable;
+import java.text.ParseException;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+
+import javax.script.Invocable;
+import javax.script.ScriptEngine;
+import javax.script.ScriptEngineFactory;
+import javax.script.ScriptEngineManager;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.metron.common.Constants;
+import org.apache.metron.parsers.interfaces.MessageParser;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+
+public class ScriptParser implements MessageParser<JSONObject>,Serializable{
+
+ protected static final Logger LOG = LoggerFactory.getLogger(ScriptParser.class);
+ protected String scriptPath;
+ protected ScriptEngine engine;
+ protected String parseFunction;
+ protected String language;
+ protected String commonScript="/scripts/";
+ protected List<String> timeFields = new ArrayList<>();
+ protected String timestampField;
+ protected SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.S z");
+
+ @Override
+ public void configure(Map<String, Object> config) {
+ // TODO Auto-generated method stub
+ this.scriptPath=(String) config.get("path");
+ this.parseFunction=(String)config.get("function");
+ this.language=(String)config.get("language");
+ this.commonScript=this.commonScript+language+"/common";
+ if(this.parseFunction==null)
+ this.parseFunction="parse";
+ }
+ //Should this be sent to the interface as a default method?
+ public InputStream openInputStream(String streamName) throws IOException {
+ FileSystem fs = FileSystem.get(new Configuration());
--- End diff --
So this is following the pattern of what's currently in Grok. That is being moved to take the grok statements from zookeeper because otherwise you cannot update the parser without bouncing the topology. Should we follow suit here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/338#discussion_r86225178
--- Diff: metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/ScriptParser.java ---
@@ -0,0 +1,175 @@
+package org.apache.metron.parsers;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.Serializable;
+import java.text.ParseException;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+
+import javax.script.Invocable;
+import javax.script.ScriptEngine;
+import javax.script.ScriptEngineFactory;
+import javax.script.ScriptEngineManager;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.metron.common.Constants;
+import org.apache.metron.parsers.interfaces.MessageParser;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+
+public class ScriptParser implements MessageParser<JSONObject>,Serializable{
+
+ protected static final Logger LOG = LoggerFactory.getLogger(ScriptParser.class);
+ protected String scriptPath;
+ protected ScriptEngine engine;
+ protected String parseFunction;
+ protected String language;
+ protected String commonScript="/scripts/";
+ protected List<String> timeFields = new ArrayList<>();
+ protected String timestampField;
+ protected SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.S z");
+
+ @Override
+ public void configure(Map<String, Object> config) {
+ // TODO Auto-generated method stub
+ this.scriptPath=(String) config.get("path");
+ this.parseFunction=(String)config.get("function");
+ this.language=(String)config.get("language");
+ this.commonScript=this.commonScript+language+"/common";
+ if(this.parseFunction==null)
+ this.parseFunction="parse";
+ }
+ //Should this be sent to the interface as a default method?
+ public InputStream openInputStream(String streamName) throws IOException {
+ FileSystem fs = FileSystem.get(new Configuration());
+ Path path = new Path(streamName);
+ if(fs.exists(path)) {
+ return fs.open(path);
+ } else {
+ return getClass().getResourceAsStream(streamName);
+ }
+ }
+
+ @Override
+ public void init() {
+ // TODO Auto-generated method stub
+ engine = new ScriptEngineManager().getEngineByName(this.language);
+ try{
+ InputStream commonStream = openInputStream(this.commonScript);
+ if (commonStream == null) {
+ throw new RuntimeException(
+ "Unable to initialize "+this.language+" Parser: Unable to load " + this.commonScript + " from either classpath or HDFS");
+ }
+
+ engine.eval(new InputStreamReader(commonStream));
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Loading parser-specific functions from: " + this.scriptPath);
+ }
+
+ InputStream patterInputStream = openInputStream(this.scriptPath);
+ if (patterInputStream == null) {
+ throw new RuntimeException("Script parser unable to initialize "+this.language+" parser: Unable to load " + this.scriptPath
+ + " from either classpath or HDFS");
+ }
+ engine.eval(new InputStreamReader(patterInputStream));
+ }catch(Throwable e){
+ LOG.error(e.getMessage(), e);
+ throw new RuntimeException(this.language+" Script parser Error: " + e.getMessage(), e);
+ }
+
+ Invocable invocable = (Invocable) engine;
+
+ }
+
+ @Override
+ public List<JSONObject> parse(byte[] rawMessage) {
+ // TODO Auto-generated method stub
+ List<JSONObject> messages = new ArrayList<JSONObject>();
+ if(engine==null)
+ init();
+ String originalMessage = null;
+ try{
+ originalMessage = new String(rawMessage, "UTF-8");
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("JS parser parsing message: " + originalMessage);
+ }
+ Invocable invocable = (Invocable) engine;
+
+ Object result = invocable.invokeFunction(this.parseFunction,originalMessage);
+ JSONObject message = new JSONObject((Map)result);
+
+ if (message.size() == 0)
+ throw new RuntimeException("Script produced a null message. Original message was: "
+ + originalMessage + " and the parsed message was: " + message + " . Check the function at: "
+ + this.scriptPath);
+
+ message.put("original_string", originalMessage);
+ for (String timeField : timeFields) {
+ String fieldValue = (String) message.get(timeField);
+ if (fieldValue != null) {
+ message.put(timeField, toEpoch(fieldValue));
+ }
+ }
+ if (timestampField != null) {
+ message.put(Constants.Fields.TIMESTAMP.getName(), formatTimestamp(message.get(timestampField)));
+ }
+ messages.add(message);
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Grok parser parsed message: " + message);
+ }
+ }catch (Exception e) {
+ LOG.error(e.getMessage(), e);
+ throw new IllegalStateException("Grok parser Error: " + e.getMessage() + " on " + originalMessage , e);
+ }
+ return messages;
+ }
+
+ @Override
+ public boolean validate(JSONObject message) {
+ // TODO Auto-generated method stub
--- End diff --
don't need that TODO
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #338: METRON-295: Script parsing bolt
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/338
Could you add a section to [here|https://github.com/apache/incubator-metron/tree/master/metron-platform/metron-parsers#grok-parser-adapters] describing the parser's config and limitations/abilities?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/338#discussion_r86225367
--- Diff: metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/GroovyParserTest.java ---
@@ -0,0 +1,81 @@
+package org.apache.metron.parsers;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.adrianwalker.multilinestring.Multiline;
+
+public class GroovyParserTest extends ScriptParserTest {
+
+ /**
+ {
+ "customerId":117,
+ "first_name":"karthik",
+ "last_name":"narayanan",
+ "age":"38",
+ "login-time":"2016-01-28 15:29:48",
+ "ip-address":"216.21.170.221",
+ "os":"windows 10",
+ "device":"Dell Inspiron"
+ }
+ */
+ @Multiline
+ public String result=" {"+
+ "\"source\":\"userlog\","+
+ "\"customerId\":117,"+
+ "\"first_name\":\"karthik\","+
+ "\"last_name\":\"narayanan\","+
+ "\"age\":\"38\","+
+ "\"login-time\":\"2016-01-28 15:29:48\","+
+ "\"ip-address\":\"216.21.170.221\","+
+ "\"os\":\"windows 10\","+
+ "\"device\":\"Dell Inspiron\""+
+ "\"original_string\":\"2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron\""+
+ "}";
+
+ @Override
+ public Map getTestData() {
+ Map testData = new HashMap<String,String>();
+ String input = "2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron";
+ testData.put(input,result);
+ return testData;
+ }
+
+ @Override
+ public String getScriptPath() {
+ // TODO Auto-generated method stub
--- End diff --
Remove
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
Posted by kylerichardson <gi...@git.apache.org>.
Github user kylerichardson commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/338#discussion_r87221966
--- Diff: metron-platform/metron-parsers/src/main/resources/scripts/python/common ---
@@ -0,0 +1,20 @@
+ # Licensed to the Apache Software Foundation (ASF) under one
+ # or more contributor license agreements. See the NOTICE file
+ # distributed with this work for additional information
+ # regarding copyright ownership. The ASF licenses this file
+ # to you under the Apache License, Version 2.0 (the
+ # "License"); you may not use this file except in compliance
+ # with the License. You may obtain a copy of the License at
+ #
+ # http://www.apache.org/licenses/LICENSE-2.0
+ #
+ # Unless required by applicable law or agreed to in writing, software
+ # distributed under the License is distributed on an "AS IS" BASIS,
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ # See the License for the specific language governing permissions and
+ # limitations under the License.
+
+def MetronMessage(name):
+ message={}
+ message["source"]="userlog"
--- End diff --
Can we update this to set the source to the passed in name variable?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
Posted by kylerichardson <gi...@git.apache.org>.
Github user kylerichardson commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/338#discussion_r87226233
--- Diff: metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/ScriptParser.java ---
@@ -0,0 +1,175 @@
+package org.apache.metron.parsers;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.Serializable;
+import java.text.ParseException;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+
+import javax.script.Invocable;
+import javax.script.ScriptEngine;
+import javax.script.ScriptEngineFactory;
+import javax.script.ScriptEngineManager;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.metron.common.Constants;
+import org.apache.metron.parsers.interfaces.MessageParser;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+
+public class ScriptParser implements MessageParser<JSONObject>,Serializable{
+
+ protected static final Logger LOG = LoggerFactory.getLogger(ScriptParser.class);
+ protected String scriptPath;
+ protected ScriptEngine engine;
+ protected String parseFunction;
+ protected String language;
+ protected String commonScript="/scripts/";
+ protected List<String> timeFields = new ArrayList<>();
+ protected String timestampField;
+ protected SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.S z");
+
+ @Override
+ public void configure(Map<String, Object> config) {
+ // TODO Auto-generated method stub
+ this.scriptPath=(String) config.get("path");
+ this.parseFunction=(String)config.get("function");
+ this.language=(String)config.get("language");
+ this.commonScript=this.commonScript+language+"/common";
+ if(this.parseFunction==null)
+ this.parseFunction="parse";
+ }
+ //Should this be sent to the interface as a default method?
+ public InputStream openInputStream(String streamName) throws IOException {
+ FileSystem fs = FileSystem.get(new Configuration());
--- End diff --
I agree with the desire for consistency; however, I would potentially be concerned about the user experience of putting scripts (especially ones with external library dependencies) into zookeeper. We would need to make this as clean as possible.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #338: METRON-295: Script parsing bolt
Posted by james-sirota <gi...@git.apache.org>.
Github user james-sirota commented on the issue:
https://github.com/apache/incubator-metron/pull/338
i agree with @randerzander. We can file a follow-on jira to get library support for Python and other languages in another follow-on Jira to this
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
Posted by james-sirota <gi...@git.apache.org>.
Github user james-sirota commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/338#discussion_r86234084
--- Diff: metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/ScriptParser.java ---
@@ -0,0 +1,175 @@
+package org.apache.metron.parsers;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.Serializable;
+import java.text.ParseException;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+
+import javax.script.Invocable;
+import javax.script.ScriptEngine;
+import javax.script.ScriptEngineFactory;
+import javax.script.ScriptEngineManager;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.metron.common.Constants;
+import org.apache.metron.parsers.interfaces.MessageParser;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+
+public class ScriptParser implements MessageParser<JSONObject>,Serializable{
+
+ protected static final Logger LOG = LoggerFactory.getLogger(ScriptParser.class);
+ protected String scriptPath;
+ protected ScriptEngine engine;
+ protected String parseFunction;
+ protected String language;
+ protected String commonScript="/scripts/";
+ protected List<String> timeFields = new ArrayList<>();
+ protected String timestampField;
+ protected SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.S z");
+
+ @Override
+ public void configure(Map<String, Object> config) {
+ // TODO Auto-generated method stub
+ this.scriptPath=(String) config.get("path");
+ this.parseFunction=(String)config.get("function");
+ this.language=(String)config.get("language");
+ this.commonScript=this.commonScript+language+"/common";
+ if(this.parseFunction==null)
+ this.parseFunction="parse";
+ }
+ //Should this be sent to the interface as a default method?
+ public InputStream openInputStream(String streamName) throws IOException {
+ FileSystem fs = FileSystem.get(new Configuration());
--- End diff --
We should be consistent with this for now. If we change them to live in zookeeper we have to change all the parsers. We can't have some live in HDFS and others live in zookeeper
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/338#discussion_r86225579
--- Diff: metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/PythonParserTest.java ---
@@ -0,0 +1,81 @@
+package org.apache.metron.parsers;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.adrianwalker.multilinestring.Multiline;
+
+public class PythonParserTest extends ScriptParserTest {
+
+ /**
+ {
+ "customerId":117,
+ "first_name":"karthik",
+ "last_name":"narayanan",
+ "age":"38",
+ "login-time":"2016-01-28 15:29:48",
+ "ip-address":"216.21.170.221",
+ "os":"windows 10",
+ "device":"Dell Inspiron"
+ }
+ */
+ @Multiline
+ public String result=" {"+
+ "\"source\":\"userlog\","+
+ "\"customerId\":117,"+
+ "\"first_name\":\"karthik\","+
+ "\"last_name\":\"narayanan\","+
+ "\"age\":\"38\","+
+ "\"login-time\":\"2016-01-28 15:29:48\","+
+ "\"ip-address\":\"216.21.170.221\","+
+ "\"os\":\"windows 10\","+
+ "\"device\":\"Dell Inspiron\""+
+ "\"original_string\":\"2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron\""+
+ "}";
+
+ @Override
+ public Map getTestData() {
+ Map testData = new HashMap<String,String>();
+ String input = "2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron";
+ testData.put(input,result);
+ return testData;
+ }
+
+ @Override
+ public String getScriptPath() {
+ // TODO Auto-generated method stub
+ return "../metron-integration-test/src/main/sample/scripts/test.py";
+ }
+
+ @Override
+ public String getParseFunction() {
+ // TODO Auto-generated method stub
+ return "parse";
+ }
+
+ @Override
+ public String getLanguage() {
+ // TODO Auto-generated method stub
+ return "python";
+ }
+
+ @Override
+ public List<String> getTimeFields() {
+ // TODO Auto-generated method stub
+ return null;
+ }
+
+ @Override
+ public String getDateFormat() {
+ // TODO Auto-generated method stub
+ return "yyyy-MM-dd HH:mm:ss";
+ }
+
+ @Override
+ public String getTimestampField() {
+ // TODO Auto-generated method stub
--- End diff --
Remove
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/338#discussion_r86225520
--- Diff: metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/PythonParserTest.java ---
@@ -0,0 +1,81 @@
+package org.apache.metron.parsers;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.adrianwalker.multilinestring.Multiline;
+
+public class PythonParserTest extends ScriptParserTest {
+
+ /**
+ {
+ "customerId":117,
+ "first_name":"karthik",
+ "last_name":"narayanan",
+ "age":"38",
+ "login-time":"2016-01-28 15:29:48",
+ "ip-address":"216.21.170.221",
+ "os":"windows 10",
+ "device":"Dell Inspiron"
+ }
+ */
+ @Multiline
+ public String result=" {"+
+ "\"source\":\"userlog\","+
+ "\"customerId\":117,"+
+ "\"first_name\":\"karthik\","+
+ "\"last_name\":\"narayanan\","+
+ "\"age\":\"38\","+
+ "\"login-time\":\"2016-01-28 15:29:48\","+
+ "\"ip-address\":\"216.21.170.221\","+
+ "\"os\":\"windows 10\","+
+ "\"device\":\"Dell Inspiron\""+
+ "\"original_string\":\"2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron\""+
+ "}";
+
+ @Override
+ public Map getTestData() {
+ Map testData = new HashMap<String,String>();
+ String input = "2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron";
+ testData.put(input,result);
+ return testData;
+ }
+
+ @Override
+ public String getScriptPath() {
+ // TODO Auto-generated method stub
+ return "../metron-integration-test/src/main/sample/scripts/test.py";
+ }
+
+ @Override
+ public String getParseFunction() {
+ // TODO Auto-generated method stub
--- End diff --
Remove
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #338: METRON-295: Script parsing bolt
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/338
I think the tooling around creating the script parser is very important, although that *could* be tracked in another jira as a follow on. But it may be impactful to this
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #338: METRON-295: Script parsing bolt
Posted by james-sirota <gi...@git.apache.org>.
Github user james-sirota commented on the issue:
https://github.com/apache/incubator-metron/pull/338
By the way sounds like the Travis build fails the RAT check. [ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.12:check (default) on project metron-integration-test: Too many files with unapproved license: 3 See RAT report in: /home/travis/build/apache/incubator-metron/metron-platform/metron-integration-test/target/rat.txt -> [Help 1]
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #338: METRON-295: Script parsing bolt
Posted by james-sirota <gi...@git.apache.org>.
Github user james-sirota commented on the issue:
https://github.com/apache/incubator-metron/pull/338
I will try to verify this on a production deployment with production data on Friday. I'll let you know how it goes
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #338: METRON-295: Script parsing bolt
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/incubator-metron/pull/338
@rkarthik29 One of the advantages of keeping the configuration (scripts in this case) in Zookeeper is that you get notified when something changes. You don't need to track versions. Unfortunately, until #308 gets merged in, the ParserBolt does not notify the parser class that the config has changed and only provides the configuration on startup (hence the need to restart the topology).
I think it makes sense to keep it in HDFS for this PR and accept the current restriction of having to restart the topology for it to take effect. Until we merge #308 or agree on the another approach, I wouldn't worry about script changes being updated on the fly. There has already been a lot of discussion and work put into this so it doesn't make sense to start over. We could even update #308 to include this parser once we get closer to merging that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/338#discussion_r86225456
--- Diff: metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/JSParserTest.java ---
@@ -0,0 +1,81 @@
+package org.apache.metron.parsers;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.adrianwalker.multilinestring.Multiline;
+
+public class JSParserTest extends ScriptParserTest {
+
+ /**
+ {
+ "customerId":117,
+ "first_name":"karthik",
+ "last_name":"narayanan",
+ "age":"38",
+ "login-time":"2016-01-28 15:29:48",
+ "ip-address":"216.21.170.221",
+ "os":"windows 10",
+ "device":"Dell Inspiron"
+ }
+ */
+ @Multiline
+ public String result=" {"+
+ "\"source\":\"userlog\","+
+ "\"customerId\":117,"+
+ "\"first_name\":\"karthik\","+
+ "\"last_name\":\"narayanan\","+
+ "\"age\":\"38\","+
+ "\"login-time\":\"2016-01-28 15:29:48\","+
+ "\"ip-address\":\"216.21.170.221\","+
+ "\"os\":\"windows 10\","+
+ "\"device\":\"Dell Inspiron\""+
+ "\"original_string\":\"2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron\""+
+ "}";
+
+ @Override
+ public Map getTestData() {
+ Map testData = new HashMap<String,String>();
+ String input = "2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron";
+ testData.put(input,result);
+ return testData;
+ }
+
+ @Override
+ public String getScriptPath() {
+ // TODO Auto-generated method stub
+ return "../metron-integration-test/src/main/sample/scripts/test.js";
+ }
+
+ @Override
+ public String getParseFunction() {
+ // TODO Auto-generated method stub
+ return "parse";
+ }
+
+ @Override
+ public String getLanguage() {
+ // TODO Auto-generated method stub
+ return "js";
+ }
+
+ @Override
+ public List<String> getTimeFields() {
+ // TODO Auto-generated method stub
--- End diff --
Remove
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/338#discussion_r86234513
--- Diff: metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/ScriptParser.java ---
@@ -0,0 +1,175 @@
+package org.apache.metron.parsers;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.Serializable;
+import java.text.ParseException;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+
+import javax.script.Invocable;
+import javax.script.ScriptEngine;
+import javax.script.ScriptEngineFactory;
+import javax.script.ScriptEngineManager;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.metron.common.Constants;
+import org.apache.metron.parsers.interfaces.MessageParser;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+
+public class ScriptParser implements MessageParser<JSONObject>,Serializable{
+
+ protected static final Logger LOG = LoggerFactory.getLogger(ScriptParser.class);
+ protected String scriptPath;
+ protected ScriptEngine engine;
+ protected String parseFunction;
+ protected String language;
+ protected String commonScript="/scripts/";
+ protected List<String> timeFields = new ArrayList<>();
+ protected String timestampField;
+ protected SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.S z");
+
+ @Override
+ public void configure(Map<String, Object> config) {
+ // TODO Auto-generated method stub
+ this.scriptPath=(String) config.get("path");
+ this.parseFunction=(String)config.get("function");
+ this.language=(String)config.get("language");
+ this.commonScript=this.commonScript+language+"/common";
+ if(this.parseFunction==null)
+ this.parseFunction="parse";
+ }
+ //Should this be sent to the interface as a default method?
+ public InputStream openInputStream(String streamName) throws IOException {
+ FileSystem fs = FileSystem.get(new Configuration());
--- End diff --
Well, we currently have active work to move the only parser that refers to reference files to zookeeper. We put ourselves in an inconsistent state as soon as that work comes in if we have this pull from HDFS.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/338#discussion_r86225430
--- Diff: metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/JSParserTest.java ---
@@ -0,0 +1,81 @@
+package org.apache.metron.parsers;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.adrianwalker.multilinestring.Multiline;
+
+public class JSParserTest extends ScriptParserTest {
+
+ /**
+ {
+ "customerId":117,
+ "first_name":"karthik",
+ "last_name":"narayanan",
+ "age":"38",
+ "login-time":"2016-01-28 15:29:48",
+ "ip-address":"216.21.170.221",
+ "os":"windows 10",
+ "device":"Dell Inspiron"
+ }
+ */
+ @Multiline
+ public String result=" {"+
+ "\"source\":\"userlog\","+
+ "\"customerId\":117,"+
+ "\"first_name\":\"karthik\","+
+ "\"last_name\":\"narayanan\","+
+ "\"age\":\"38\","+
+ "\"login-time\":\"2016-01-28 15:29:48\","+
+ "\"ip-address\":\"216.21.170.221\","+
+ "\"os\":\"windows 10\","+
+ "\"device\":\"Dell Inspiron\""+
+ "\"original_string\":\"2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron\""+
+ "}";
+
+ @Override
+ public Map getTestData() {
+ Map testData = new HashMap<String,String>();
+ String input = "2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron";
+ testData.put(input,result);
+ return testData;
+ }
+
+ @Override
+ public String getScriptPath() {
+ // TODO Auto-generated method stub
+ return "../metron-integration-test/src/main/sample/scripts/test.js";
+ }
+
+ @Override
+ public String getParseFunction() {
+ // TODO Auto-generated method stub
--- End diff --
Remove
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #338: METRON-295: Script parsing bolt
Posted by rkarthik29 <gi...@git.apache.org>.
Github user rkarthik29 commented on the issue:
https://github.com/apache/incubator-metron/pull/338
The pull request is due to some errors with executing test cases. I am not sure if they are related to the parser, if someone guides me i can look at it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/338#discussion_r86225445
--- Diff: metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/JSParserTest.java ---
@@ -0,0 +1,81 @@
+package org.apache.metron.parsers;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.adrianwalker.multilinestring.Multiline;
+
+public class JSParserTest extends ScriptParserTest {
+
+ /**
+ {
+ "customerId":117,
+ "first_name":"karthik",
+ "last_name":"narayanan",
+ "age":"38",
+ "login-time":"2016-01-28 15:29:48",
+ "ip-address":"216.21.170.221",
+ "os":"windows 10",
+ "device":"Dell Inspiron"
+ }
+ */
+ @Multiline
+ public String result=" {"+
+ "\"source\":\"userlog\","+
+ "\"customerId\":117,"+
+ "\"first_name\":\"karthik\","+
+ "\"last_name\":\"narayanan\","+
+ "\"age\":\"38\","+
+ "\"login-time\":\"2016-01-28 15:29:48\","+
+ "\"ip-address\":\"216.21.170.221\","+
+ "\"os\":\"windows 10\","+
+ "\"device\":\"Dell Inspiron\""+
+ "\"original_string\":\"2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron\""+
+ "}";
+
+ @Override
+ public Map getTestData() {
+ Map testData = new HashMap<String,String>();
+ String input = "2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron";
+ testData.put(input,result);
+ return testData;
+ }
+
+ @Override
+ public String getScriptPath() {
+ // TODO Auto-generated method stub
+ return "../metron-integration-test/src/main/sample/scripts/test.js";
+ }
+
+ @Override
+ public String getParseFunction() {
+ // TODO Auto-generated method stub
+ return "parse";
+ }
+
+ @Override
+ public String getLanguage() {
+ // TODO Auto-generated method stub
--- End diff --
Remove
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/338#discussion_r86225417
--- Diff: metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/JSParserTest.java ---
@@ -0,0 +1,81 @@
+package org.apache.metron.parsers;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.adrianwalker.multilinestring.Multiline;
+
+public class JSParserTest extends ScriptParserTest {
+
+ /**
+ {
+ "customerId":117,
+ "first_name":"karthik",
+ "last_name":"narayanan",
+ "age":"38",
+ "login-time":"2016-01-28 15:29:48",
+ "ip-address":"216.21.170.221",
+ "os":"windows 10",
+ "device":"Dell Inspiron"
+ }
+ */
+ @Multiline
+ public String result=" {"+
+ "\"source\":\"userlog\","+
+ "\"customerId\":117,"+
+ "\"first_name\":\"karthik\","+
+ "\"last_name\":\"narayanan\","+
+ "\"age\":\"38\","+
+ "\"login-time\":\"2016-01-28 15:29:48\","+
+ "\"ip-address\":\"216.21.170.221\","+
+ "\"os\":\"windows 10\","+
+ "\"device\":\"Dell Inspiron\""+
+ "\"original_string\":\"2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron\""+
+ "}";
+
+ @Override
+ public Map getTestData() {
+ Map testData = new HashMap<String,String>();
+ String input = "2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron";
+ testData.put(input,result);
+ return testData;
+ }
+
+ @Override
+ public String getScriptPath() {
+ // TODO Auto-generated method stub
--- End diff --
Remove
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #338: METRON-295: Script parsing bolt
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/338
So that is similar to the pyleus stuff. Although pyleus actually produces jar topologies as a unit. The question is, with only standard library support for any of these languages how useful is this? Does that need to be accounted for now or can it be another jira as you mention. I'm not advocating either way, but if we do say it will be another jira, and there will be further work here, we should explicitly state that and create the jiras to put that plan in place.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/338#discussion_r86225466
--- Diff: metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/JSParserTest.java ---
@@ -0,0 +1,81 @@
+package org.apache.metron.parsers;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.adrianwalker.multilinestring.Multiline;
+
+public class JSParserTest extends ScriptParserTest {
+
+ /**
+ {
+ "customerId":117,
+ "first_name":"karthik",
+ "last_name":"narayanan",
+ "age":"38",
+ "login-time":"2016-01-28 15:29:48",
+ "ip-address":"216.21.170.221",
+ "os":"windows 10",
+ "device":"Dell Inspiron"
+ }
+ */
+ @Multiline
+ public String result=" {"+
+ "\"source\":\"userlog\","+
+ "\"customerId\":117,"+
+ "\"first_name\":\"karthik\","+
+ "\"last_name\":\"narayanan\","+
+ "\"age\":\"38\","+
+ "\"login-time\":\"2016-01-28 15:29:48\","+
+ "\"ip-address\":\"216.21.170.221\","+
+ "\"os\":\"windows 10\","+
+ "\"device\":\"Dell Inspiron\""+
+ "\"original_string\":\"2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron\""+
+ "}";
+
+ @Override
+ public Map getTestData() {
+ Map testData = new HashMap<String,String>();
+ String input = "2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron";
+ testData.put(input,result);
+ return testData;
+ }
+
+ @Override
+ public String getScriptPath() {
+ // TODO Auto-generated method stub
+ return "../metron-integration-test/src/main/sample/scripts/test.js";
+ }
+
+ @Override
+ public String getParseFunction() {
+ // TODO Auto-generated method stub
+ return "parse";
+ }
+
+ @Override
+ public String getLanguage() {
+ // TODO Auto-generated method stub
+ return "js";
+ }
+
+ @Override
+ public List<String> getTimeFields() {
+ // TODO Auto-generated method stub
+ return null;
+ }
+
+ @Override
+ public String getDateFormat() {
+ // TODO Auto-generated method stub
--- End diff --
Remove
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #338: METRON-295: Script parsing bolt
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/338
I'm increasingly becoming of the opinion that the code as it stands is probably the right path for the the moment. When we are through the changes from making the grok parser pull from Zookeeper, we can revise if necessary.
The main issue, as I see it, is as @rkarthik29 stated: how do you know that the scripts changed. And there are now two parsers that need non-trivially sized external files to function properly, so it may make some sense for there to be more concerted thought to how to abstract that properly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #338: METRON-295: Script parsing bolt
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/338
https://github.com/onefoursix/hdfs-inotify-example/blob/master/src/main/java/com/onefoursix/HdfsINotifyExample.java
Could we not investigate using INotify for notification of things in HDFS?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/338#discussion_r86225481
--- Diff: metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/JSParserTest.java ---
@@ -0,0 +1,81 @@
+package org.apache.metron.parsers;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.adrianwalker.multilinestring.Multiline;
+
+public class JSParserTest extends ScriptParserTest {
+
+ /**
+ {
+ "customerId":117,
+ "first_name":"karthik",
+ "last_name":"narayanan",
+ "age":"38",
+ "login-time":"2016-01-28 15:29:48",
+ "ip-address":"216.21.170.221",
+ "os":"windows 10",
+ "device":"Dell Inspiron"
+ }
+ */
+ @Multiline
+ public String result=" {"+
+ "\"source\":\"userlog\","+
+ "\"customerId\":117,"+
+ "\"first_name\":\"karthik\","+
+ "\"last_name\":\"narayanan\","+
+ "\"age\":\"38\","+
+ "\"login-time\":\"2016-01-28 15:29:48\","+
+ "\"ip-address\":\"216.21.170.221\","+
+ "\"os\":\"windows 10\","+
+ "\"device\":\"Dell Inspiron\""+
+ "\"original_string\":\"2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron\""+
+ "}";
+
+ @Override
+ public Map getTestData() {
+ Map testData = new HashMap<String,String>();
+ String input = "2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron";
+ testData.put(input,result);
+ return testData;
+ }
+
+ @Override
+ public String getScriptPath() {
+ // TODO Auto-generated method stub
+ return "../metron-integration-test/src/main/sample/scripts/test.js";
+ }
+
+ @Override
+ public String getParseFunction() {
+ // TODO Auto-generated method stub
+ return "parse";
+ }
+
+ @Override
+ public String getLanguage() {
+ // TODO Auto-generated method stub
+ return "js";
+ }
+
+ @Override
+ public List<String> getTimeFields() {
+ // TODO Auto-generated method stub
+ return null;
+ }
+
+ @Override
+ public String getDateFormat() {
+ // TODO Auto-generated method stub
+ return "yyyy-MM-dd HH:mm:ss";
+ }
+
+ @Override
+ public String getTimestampField() {
+ // TODO Auto-generated method stub
--- End diff --
Remove
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #338: METRON-295: Script parsing bolt
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/338#discussion_r86225531
--- Diff: metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/PythonParserTest.java ---
@@ -0,0 +1,81 @@
+package org.apache.metron.parsers;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.adrianwalker.multilinestring.Multiline;
+
+public class PythonParserTest extends ScriptParserTest {
+
+ /**
+ {
+ "customerId":117,
+ "first_name":"karthik",
+ "last_name":"narayanan",
+ "age":"38",
+ "login-time":"2016-01-28 15:29:48",
+ "ip-address":"216.21.170.221",
+ "os":"windows 10",
+ "device":"Dell Inspiron"
+ }
+ */
+ @Multiline
+ public String result=" {"+
+ "\"source\":\"userlog\","+
+ "\"customerId\":117,"+
+ "\"first_name\":\"karthik\","+
+ "\"last_name\":\"narayanan\","+
+ "\"age\":\"38\","+
+ "\"login-time\":\"2016-01-28 15:29:48\","+
+ "\"ip-address\":\"216.21.170.221\","+
+ "\"os\":\"windows 10\","+
+ "\"device\":\"Dell Inspiron\""+
+ "\"original_string\":\"2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron\""+
+ "}";
+
+ @Override
+ public Map getTestData() {
+ Map testData = new HashMap<String,String>();
+ String input = "2016-01-28 15:29:48|117|karthik|narayanan|38|216.21.170.221|windows 10|Dell Inspiron";
+ testData.put(input,result);
+ return testData;
+ }
+
+ @Override
+ public String getScriptPath() {
+ // TODO Auto-generated method stub
+ return "../metron-integration-test/src/main/sample/scripts/test.py";
+ }
+
+ @Override
+ public String getParseFunction() {
+ // TODO Auto-generated method stub
+ return "parse";
+ }
+
+ @Override
+ public String getLanguage() {
+ // TODO Auto-generated method stub
--- End diff --
Remove
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #338: METRON-295: Script parsing bolt
Posted by rkarthik29 <gi...@git.apache.org>.
Github user rkarthik29 commented on the issue:
https://github.com/apache/incubator-metron/pull/338
@james-sirota I will fix the conflicts once we are ready to merge.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #338: METRON-295: Script parsing bolt
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/338
I don't think the script change is the same as configuration/rule change necessarily.
Maybe what is missing are the requirements?
* does the script have to be editable and trigger reload the same as a configuration
* does the script have to include dependencies such that it doesn't introduce problems maintaining environments across the cluster
* does script X have to support all 'features' ( jython limitations for cpython modules )
Etc etc
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---