You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by devopsec <gi...@git.apache.org> on 2017/02/05 00:24:39 UTC
[GitHub] incubator-metron pull request #439: add stellar external functions feature (...
GitHub user devopsec opened a pull request:
https://github.com/apache/incubator-metron/pull/439
add stellar external functions feature (work in progress) for review
This pull is not for merging to tree, it is for code review.
Please review stellar feature for METRON-571 Issue in the jira:
https://issues.apache.org/jira/browse/METRON-571
I have not used junit testing previously so I put together what I could from watching casey's tutorial but I could use some guidance on implementing the unit tests.
Thanks
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/devopsec/incubator-metron master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/439.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #439
----
commit e9122910919f513e34a6567f74d2f82de86f83e9
Author: devopsec <ty...@gmail.com>
Date: 2017-02-05T00:19:02Z
add stellar external functions feature (work in progress) for review
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #439: add stellar external functions feature (...
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/439#discussion_r99479413
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+ public static class ExecuteScript implements StellarFunction {
+
+ private ThreadedStreamHandler inStream;
+ private ThreadedStreamHandler errStream;
+ private boolean isOnTheList = false;
+
+ @Stellar(name="EXEC_SCRIPT",
+ description = "Executes an external shell function via stellar.",
+ params = {
+ "exec - the executing cmd (ie. bash, sh, python)",
+ "name - name of the script, located in /scripts " +
+ "Do NOT include any special chars except(_), Do include file extension"
+ },
+ returns = "the return value of the function"
+ )
+
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+ String exec = "";
+ String name = "";
+ String path = "";
+
+ // if args are provided, get args, only if in whitelist
+ if (args.size() >= 1) {
+ Object execObj = args.get(0);
+ if (!(execObj instanceof String)) { //check if string
+ return null;
+ }
+ else if (((String) execObj).length() > 0) {
+ exec = (String) execObj;
+ }
+ else {
+ return null;
+ }
+
+ Object nameObj = args.get(1);
+ if (!(nameObj instanceof String)) { //check if string
+ return null;
+ }
+ else if (((String) nameObj).length() > 0) {
+ name = (String) nameObj;
+ }
+ else {
+ return null;
+ }
+
+ if (!Pattern.matches("[0-9A-Za-z.]+", name)) {
+ return null; //if not on whitelist
+ }
+
+ path = "/scripts" + name;
+ try {
+ File script = new File(path);
+ if (!script.exists() || script.isDirectory()) {
+ return null;
+ }
+ }
+ catch (NullPointerException e) {
+ System.err.println("Error: " + e.toString());
+ return null;
+ }
+
+ switch (exec) { //check if matches extension
+ case "bash":
+ if (name.contains(".sh")) {
+ isOnTheList = true;
+ }
+ break;
+ case "sh":
+ if (name.contains(".sh")) {
+ isOnTheList = true;
+ }
+ break;
+ case "python":
+ if (name.contains(".py")) {
+ isOnTheList = true;
+ }
+ break;
+ case "java":
+ if (name.contains(".class")) {
+ isOnTheList = true;
+ // java cmd needs called w/o .class
+ name.replace(".class", "");
+ }
+ break;
+ }
+ }
+ if (!(isOnTheList)) {
+ return null;
+ }
--- End diff --
So are you planning to flesh out your idea of the whitelist? Perhaps a good topic for a design discussion on the dev mailing list.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #439: add stellar external functions feature (...
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/439#discussion_r99493097
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+ public static class ExecuteScript implements StellarFunction {
+
+ private ThreadedStreamHandler inStream;
+ private ThreadedStreamHandler errStream;
+ private boolean isOnTheList = false;
+
+ @Stellar(name="EXEC_SCRIPT",
+ description = "Executes an external shell function via stellar.",
+ params = {
+ "exec - the executing cmd (ie. bash, sh, python)",
+ "name - name of the script, located in /scripts " +
+ "Do NOT include any special chars except(_), Do include file extension"
+ },
+ returns = "the return value of the function"
+ )
+
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+ String exec = "";
+ String name = "";
+ String path = "";
+
+ // if args are provided, get args, only if in whitelist
+ if (args.size() >= 1) {
+ Object execObj = args.get(0);
+ if (!(execObj instanceof String)) { //check if string
+ return null;
+ }
+ else if (((String) execObj).length() > 0) {
+ exec = (String) execObj;
+ }
+ else {
+ return null;
+ }
+
+ Object nameObj = args.get(1);
+ if (!(nameObj instanceof String)) { //check if string
+ return null;
+ }
+ else if (((String) nameObj).length() > 0) {
+ name = (String) nameObj;
+ }
+ else {
+ return null;
+ }
+
+ if (!Pattern.matches("[0-9A-Za-z.]+", name)) {
+ return null; //if not on whitelist
+ }
--- End diff --
Ah, apologies to both of you if I mis-spoke due to not being able to see the jira and any interactions that had already occurred. I will read all that before commenting again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
Re: [GitHub] incubator-metron issue #439: add stellar external
functions feature (work in...
Posted by Tyler Moore <tm...@goflyball.com>.
Sorry for late reply work has been crazy around here too.
Changed name and added a few comments and I am not sure if I mentioned it
but devopsec is just my dev handle / dev name, wasn't sure if that was
apparent.
I will push up another revision later this week, add in your comments and
i'll revise as we go, thanks.
Regards,
Tyler Moore
Software Engineer
Phone: 248-909-2769
Email: tmoore@goflyball.com <mo...@goflyball.com>
On Sat, Feb 4, 2017 at 9:01 PM, JonZeolla <gi...@git.apache.org> wrote:
> Github user JonZeolla commented on the issue:
>
> https://github.com/apache/incubator-metron/pull/439
>
> The title of this PR should be "METRON-571: Add stellar keywords for
> executing local commands".
>
> This was originally my JIRA, I just haven't been able to work on it
> and Tyler had a similar need so he's taking a stab at it. That said, I
> will try to provide some context in response to your comments, but
> @devopsec please correct me if I have misunderstood.
>
>
> ---
> If your project is set up for it, you can reply to this email and have your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, please
> contact infrastructure at infrastructure@apache.org or file a JIRA ticket
> with INFRA.
> ---
>
[GitHub] incubator-metron issue #439: add stellar external functions feature (work in...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on the issue:
https://github.com/apache/incubator-metron/pull/439
The title of this PR should be "METRON-571: Add stellar keywords for executing local commands".
This was originally my JIRA, I just haven't been able to work on it and Tyler had a similar need so he's taking a stab at it. That said, I will try to provide some context in response to your comments, but @devopsec please correct me if I have misunderstood.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #439: add stellar external functions feature (...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/439#discussion_r99479668
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+ public static class ExecuteScript implements StellarFunction {
+
+ private ThreadedStreamHandler inStream;
+ private ThreadedStreamHandler errStream;
+ private boolean isOnTheList = false;
+
+ @Stellar(name="EXEC_SCRIPT",
+ description = "Executes an external shell function via stellar.",
+ params = {
+ "exec - the executing cmd (ie. bash, sh, python)",
+ "name - name of the script, located in /scripts " +
+ "Do NOT include any special chars except(_), Do include file extension"
+ },
+ returns = "the return value of the function"
+ )
+
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+ String exec = "";
+ String name = "";
+ String path = "";
+
+ // if args are provided, get args, only if in whitelist
+ if (args.size() >= 1) {
+ Object execObj = args.get(0);
+ if (!(execObj instanceof String)) { //check if string
+ return null;
+ }
+ else if (((String) execObj).length() > 0) {
+ exec = (String) execObj;
+ }
+ else {
+ return null;
+ }
+
+ Object nameObj = args.get(1);
+ if (!(nameObj instanceof String)) { //check if string
+ return null;
+ }
+ else if (((String) nameObj).length() > 0) {
+ name = (String) nameObj;
+ }
+ else {
+ return null;
+ }
+
+ if (!Pattern.matches("[0-9A-Za-z.]+", name)) {
+ return null; //if not on whitelist
+ }
--- End diff --
This was in response to a comment that I made where I requested we sanitize our inputs by whitelisting reasonable characters. Because a user may want to pass arbitrary information from/about the tuple, such as a URI, as an argument to a script I want to avoid an attacker choosing to go to a URI that could be interpreted as `somethinghere&&wget badness&&/execute/badness#ignore everything else`. Not a perfect example, but it shows the idea.
Once the JIRA is back up I posted links to a few best practice suggestions with code examples that show exactly what I was thinking.
I totally agree that there needs to be strong controls around the directory where any scripts are held, and we need to be sure to run the script with minimal permissions, among other things.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #439: METRON-571 add stellar external function...
Posted by devopsec <gi...@git.apache.org>.
Github user devopsec commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/439#discussion_r99967075
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+ public static class ExecuteScript implements StellarFunction {
+
+ private ThreadedStreamHandler inStream;
+ private ThreadedStreamHandler errStream;
+ private boolean isOnTheList = false;
+
+ @Stellar(name="EXEC_SCRIPT",
+ description = "Executes an external shell function via stellar.",
+ params = {
+ "exec - the executing cmd (ie. bash, sh, python)",
+ "name - name of the script, located in /scripts " +
+ "Do NOT include any special chars except(_), Do include file extension"
+ },
+ returns = "the return value of the function"
+ )
+
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+ String exec = "";
+ String name = "";
+ String path = "";
+
+ // if args are provided, get args, only if in whitelist
+ if (args.size() >= 1) {
+ Object execObj = args.get(0);
+ if (!(execObj instanceof String)) { //check if string
+ return null;
+ }
+ else if (((String) execObj).length() > 0) {
+ exec = (String) execObj;
+ }
+ else {
+ return null;
+ }
+
+ Object nameObj = args.get(1);
+ if (!(nameObj instanceof String)) { //check if string
+ return null;
+ }
+ else if (((String) nameObj).length() > 0) {
+ name = (String) nameObj;
+ }
+ else {
+ return null;
+ }
+
+ if (!Pattern.matches("[0-9A-Za-z.]+", name)) {
+ return null; //if not on whitelist
+ }
+
+ path = "/scripts" + name;
+ try {
+ File script = new File(path);
+ if (!script.exists() || script.isDirectory()) {
+ return null;
+ }
+ }
+ catch (NullPointerException e) {
+ System.err.println("Error: " + e.toString());
+ return null;
+ }
--- End diff --
I did plan on adding real error handling, are there any exceptions that should be thrown up the chain? I am thinking we should allow some of these to be logged in the storm logs or log them all to some location depending on how you think that should be handled
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #439: METRON-571 add stellar external function...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/439#discussion_r105324915
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+ public static class ExecuteScript implements StellarFunction {
+
+ private ThreadedStreamHandler inStream;
+ private ThreadedStreamHandler errStream;
+ private boolean isOnTheList = false;
+
+ @Stellar(name="EXEC_SCRIPT",
+ description = "Executes an external shell function via stellar.",
+ params = {
+ "exec - the executing cmd (ie. bash, sh, python)",
+ "name - name of the script, located in /scripts " +
+ "Do NOT include any special chars except(_), Do include file extension"
+ },
+ returns = "the return value of the function"
+ )
+
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+ String exec = "";
+ String name = "";
+ String path = "";
+
+ // if args are provided, get args, only if in whitelist
+ if (args.size() >= 1) {
+ Object execObj = args.get(0);
+ if (!(execObj instanceof String)) { //check if string
+ return null;
+ }
+ else if (((String) execObj).length() > 0) {
+ exec = (String) execObj;
+ }
+ else {
+ return null;
+ }
+
+ Object nameObj = args.get(1);
+ if (!(nameObj instanceof String)) { //check if string
+ return null;
+ }
+ else if (((String) nameObj).length() > 0) {
+ name = (String) nameObj;
+ }
+ else {
+ return null;
+ }
+
+ if (!Pattern.matches("[0-9A-Za-z.]+", name)) {
+ return null; //if not on whitelist
+ }
+
+ path = "/scripts" + name;
--- End diff --
what file chooser dialog is that?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #439: METRON-571 add stellar external functions featu...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/439
This is missing documentation. There are going to be requirements to the types of scripts that can be called, and they all need to be listed out. A sample script or even a template should be provided.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #439: METRON-571 add stellar external function...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/439#discussion_r105325446
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+ public static class ExecuteScript implements StellarFunction {
+
+ private ThreadedStreamHandler inStream;
+ private ThreadedStreamHandler errStream;
+ private boolean isOnTheList = false;
+
+ @Stellar(name="EXEC_SCRIPT",
+ description = "Executes an external shell function via stellar.",
+ params = {
+ "exec - the executing cmd (ie. bash, sh, python)",
+ "name - name of the script, located in /scripts " +
+ "Do NOT include any special chars except(_), Do include file extension"
+ },
+ returns = "the return value of the function"
+ )
+
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+ String exec = "";
+ String name = "";
+ String path = "";
+
+ // if args are provided, get args, only if in whitelist
+ if (args.size() >= 1) {
+ Object execObj = args.get(0);
+ if (!(execObj instanceof String)) { //check if string
+ return null;
+ }
+ else if (((String) execObj).length() > 0) {
+ exec = (String) execObj;
+ }
+ else {
+ return null;
+ }
+
+ Object nameObj = args.get(1);
+ if (!(nameObj instanceof String)) { //check if string
+ return null;
+ }
+ else if (((String) nameObj).length() > 0) {
+ name = (String) nameObj;
+ }
+ else {
+ return null;
+ }
+
+ if (!Pattern.matches("[0-9A-Za-z.]+", name)) {
+ return null; //if not on whitelist
+ }
+
+ path = "/scripts" + name;
+ try {
+ File script = new File(path);
+ if (!script.exists() || script.isDirectory()) {
+ return null;
+ }
+ }
+ catch (NullPointerException e) {
+ System.err.println("Error: " + e.toString());
+ return null;
+ }
+
+ switch (exec) { //check if matches extension
+ case "bash":
+ if (name.contains(".sh")) {
+ isOnTheList = true;
+ }
+ break;
+ case "sh":
+ if (name.contains(".sh")) {
+ isOnTheList = true;
+ }
+ break;
+ case "python":
+ if (name.contains(".py")) {
+ isOnTheList = true;
+ }
+ break;
+ case "java":
+ if (name.contains(".class")) {
+ isOnTheList = true;
+ // java cmd needs called w/o .class
+ name.replace(".class", "");
+ }
+ break;
+ }
+ }
+ if (!(isOnTheList)) {
+ return null;
+ }
+
+ try { // Create and start Process with ProcessBuilder.
+ ProcessBuilder pb = new ProcessBuilder(exec, name);
+ //Map<String, String> env = pb.environment();
+ //env.put("VAR1", "myValue");
+
+ File log = new File("/scripts/log");
+ pb.redirectErrorStream(true);
+ pb.redirectOutput(ProcessBuilder.Redirect.appendTo(log));
+ Process p = pb.start();
+ int exitCode = p.waitFor();
+
+ assert pb.redirectInput() == ProcessBuilder.Redirect.PIPE;
+ assert pb.redirectOutput().file() == log;
+ assert p.getInputStream().read() == -1;
+
+ return exitCode;
+ }
+ catch (NullPointerException e) {
+ System.err.println("Error: " + e.toString());
+ return null;
+ }
+ catch (IndexOutOfBoundsException e) {
+ System.err.println("Error: " + e.toString());
+ return null;
+ }
+ catch (SecurityException e) {
+ System.err.println("Error: " + e.toString());
+ return null;
+ }
+ catch (IOException e) {
+ System.err.println("Error: " + e.toString());
+ return null;
+ } catch (InterruptedException e) {
+ System.err.println("Error: " + e.toString());
+ return null;
+ }
+ }
+
+ @Override
+ public void initialize(Context context) {
+
+ }
+
+ @Override
+ public boolean isInitialized() {
+ return true;
+ }
+ }
+
+ /**
+ * Special thanks to alvin j. alexander at devdaily.com
+ */
+ class ThreadedStreamHandler extends Thread
+ {
+ InputStream inputStream;
+ String adminPassword;
+ OutputStream outputStream;
+ PrintWriter printWriter;
+ StringBuilder outputBuffer = new StringBuilder();
+ private boolean sudoIsRequested = false;
+
+ /**
+ * A simple constructor for when the sudo command is not necessary.
+ * This constructor will just run the command you provide, without
+ * running sudo before the command, and without expecting a password.
+ *
+ * @param inputStream
+ */
+ ThreadedStreamHandler(InputStream inputStream)
+ {
+ this.inputStream = inputStream;
+ }
+
+ /**
+ * Use this constructor when you want to invoke the 'sudo' command.
+ * The outputStream must not be null. If it is, you'll regret it. :)
+ *
+ * TODO this currently hangs if the admin password given for the sudo command is wrong.
+ *
+ * @param inputStream
+ * @param outputStream
+ * @param adminPassword
+ */
+ ThreadedStreamHandler(InputStream inputStream, OutputStream outputStream, String adminPassword)
+ {
+ this.inputStream = inputStream;
+ this.outputStream = outputStream;
+ this.printWriter = new PrintWriter(outputStream);
+ this.adminPassword = adminPassword;
+ this.sudoIsRequested = true;
+ }
+
+ public void run()
+ {
+ // on mac os x 10.5.x, when i run a 'sudo' command, i need to write
+ // the admin password out immediately; that's why this code is
+ // here.
+ if (sudoIsRequested)
+ {
+ //doSleep(500);
--- End diff --
This is getting into some 'Expect' type stuff, there be dragons here. What if this hangs forever?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #439: add stellar external functions feature (...
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/439#discussion_r99479068
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+ public static class ExecuteScript implements StellarFunction {
+
+ private ThreadedStreamHandler inStream;
+ private ThreadedStreamHandler errStream;
+ private boolean isOnTheList = false;
+
+ @Stellar(name="EXEC_SCRIPT",
+ description = "Executes an external shell function via stellar.",
+ params = {
+ "exec - the executing cmd (ie. bash, sh, python)",
+ "name - name of the script, located in /scripts " +
+ "Do NOT include any special chars except(_), Do include file extension"
+ },
+ returns = "the return value of the function"
+ )
+
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+ String exec = "";
+ String name = "";
+ String path = "";
+
+ // if args are provided, get args, only if in whitelist
+ if (args.size() >= 1) {
+ Object execObj = args.get(0);
+ if (!(execObj instanceof String)) { //check if string
+ return null;
+ }
+ else if (((String) execObj).length() > 0) {
+ exec = (String) execObj;
+ }
+ else {
+ return null;
+ }
+
+ Object nameObj = args.get(1);
+ if (!(nameObj instanceof String)) { //check if string
+ return null;
+ }
+ else if (((String) nameObj).length() > 0) {
+ name = (String) nameObj;
+ }
+ else {
+ return null;
+ }
+
+ if (!Pattern.matches("[0-9A-Za-z.]+", name)) {
+ return null; //if not on whitelist
+ }
--- End diff --
What is the intent of this check? The given regex only checks that there is *at least* one alphanumeric character in the name argument, which doesn't say much. And you said it is okay to use "_". So perhaps you meant "^[\w.]+$". This checks that the string, from beginning to end, contains only "word" characters (A-Za-z_0-9) and period. But this excludes unicode strings for script names.
Again, though, what is the intent? The name of the script is rather irrelevant, unless you actually have a true "white list", that is a dictionary of allowed names. I don't know how we would manage such a list in Metron.
The only reliable security is restricted write permissions on the hard-coded /script directory. That needs to be documented, since violations open a really big trapdoor.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #439: METRON-571 add stellar external function...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/439#discussion_r105328056
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+ public static class ExecuteScript implements StellarFunction {
+
+ private ThreadedStreamHandler inStream;
+ private ThreadedStreamHandler errStream;
+ private boolean isOnTheList = false;
+
+ @Stellar(name="EXEC_SCRIPT",
+ description = "Executes an external shell function via stellar.",
+ params = {
+ "exec - the executing cmd (ie. bash, sh, python)",
+ "name - name of the script, located in /scripts " +
+ "Do NOT include any special chars except(_), Do include file extension"
+ },
+ returns = "the return value of the function"
+ )
+
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+ String exec = "";
+ String name = "";
+ String path = "";
+
+ // if args are provided, get args, only if in whitelist
+ if (args.size() >= 1) {
+ Object execObj = args.get(0);
+ if (!(execObj instanceof String)) { //check if string
+ return null;
+ }
+ else if (((String) execObj).length() > 0) {
+ exec = (String) execObj;
+ }
+ else {
+ return null;
+ }
+
+ Object nameObj = args.get(1);
+ if (!(nameObj instanceof String)) { //check if string
+ return null;
+ }
+ else if (((String) nameObj).length() > 0) {
+ name = (String) nameObj;
+ }
+ else {
+ return null;
+ }
+
+ if (!Pattern.matches("[0-9A-Za-z.]+", name)) {
+ return null; //if not on whitelist
+ }
+
+ path = "/scripts" + name;
+ try {
+ File script = new File(path);
+ if (!script.exists() || script.isDirectory()) {
+ return null;
+ }
+ }
+ catch (NullPointerException e) {
+ System.err.println("Error: " + e.toString());
+ return null;
+ }
+
+ switch (exec) { //check if matches extension
+ case "bash":
+ if (name.contains(".sh")) {
+ isOnTheList = true;
+ }
+ break;
+ case "sh":
+ if (name.contains(".sh")) {
+ isOnTheList = true;
+ }
+ break;
+ case "python":
+ if (name.contains(".py")) {
+ isOnTheList = true;
+ }
+ break;
+ case "java":
+ if (name.contains(".class")) {
+ isOnTheList = true;
+ // java cmd needs called w/o .class
+ name.replace(".class", "");
+ }
+ break;
+ }
+ }
+ if (!(isOnTheList)) {
+ return null;
+ }
+
+ try { // Create and start Process with ProcessBuilder.
+ ProcessBuilder pb = new ProcessBuilder(exec, name);
+ //Map<String, String> env = pb.environment();
+ //env.put("VAR1", "myValue");
+
--- End diff --
What about concurrent commands and executions?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #439: add stellar external functions feature (...
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/439#discussion_r99479152
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+ public static class ExecuteScript implements StellarFunction {
+
+ private ThreadedStreamHandler inStream;
+ private ThreadedStreamHandler errStream;
+ private boolean isOnTheList = false;
+
+ @Stellar(name="EXEC_SCRIPT",
+ description = "Executes an external shell function via stellar.",
+ params = {
+ "exec - the executing cmd (ie. bash, sh, python)",
+ "name - name of the script, located in /scripts " +
+ "Do NOT include any special chars except(_), Do include file extension"
+ },
+ returns = "the return value of the function"
+ )
+
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+ String exec = "";
+ String name = "";
+ String path = "";
+
+ // if args are provided, get args, only if in whitelist
+ if (args.size() >= 1) {
+ Object execObj = args.get(0);
+ if (!(execObj instanceof String)) { //check if string
+ return null;
+ }
+ else if (((String) execObj).length() > 0) {
+ exec = (String) execObj;
+ }
+ else {
+ return null;
+ }
+
+ Object nameObj = args.get(1);
+ if (!(nameObj instanceof String)) { //check if string
+ return null;
+ }
+ else if (((String) nameObj).length() > 0) {
+ name = (String) nameObj;
+ }
+ else {
+ return null;
+ }
+
+ if (!Pattern.matches("[0-9A-Za-z.]+", name)) {
+ return null; //if not on whitelist
+ }
+
+ path = "/scripts" + name;
+ try {
+ File script = new File(path);
+ if (!script.exists() || script.isDirectory()) {
+ return null;
+ }
+ }
+ catch (NullPointerException e) {
+ System.err.println("Error: " + e.toString());
+ return null;
+ }
+
+ switch (exec) { //check if matches extension
+ case "bash":
+ if (name.contains(".sh")) {
+ isOnTheList = true;
+ }
+ break;
+ case "sh":
--- End diff --
add case "bash": to same code block
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #439: METRON-571 add stellar external function...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/439#discussion_r105325529
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+ public static class ExecuteScript implements StellarFunction {
+
+ private ThreadedStreamHandler inStream;
+ private ThreadedStreamHandler errStream;
+ private boolean isOnTheList = false;
+
+ @Stellar(name="EXEC_SCRIPT",
+ description = "Executes an external shell function via stellar.",
+ params = {
+ "exec - the executing cmd (ie. bash, sh, python)",
+ "name - name of the script, located in /scripts " +
+ "Do NOT include any special chars except(_), Do include file extension"
+ },
+ returns = "the return value of the function"
+ )
+
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+ String exec = "";
+ String name = "";
+ String path = "";
+
+ // if args are provided, get args, only if in whitelist
+ if (args.size() >= 1) {
+ Object execObj = args.get(0);
+ if (!(execObj instanceof String)) { //check if string
+ return null;
+ }
+ else if (((String) execObj).length() > 0) {
+ exec = (String) execObj;
+ }
+ else {
+ return null;
+ }
+
+ Object nameObj = args.get(1);
+ if (!(nameObj instanceof String)) { //check if string
+ return null;
+ }
+ else if (((String) nameObj).length() > 0) {
+ name = (String) nameObj;
+ }
+ else {
+ return null;
+ }
+
+ if (!Pattern.matches("[0-9A-Za-z.]+", name)) {
+ return null; //if not on whitelist
+ }
+
+ path = "/scripts" + name;
+ try {
+ File script = new File(path);
+ if (!script.exists() || script.isDirectory()) {
+ return null;
+ }
+ }
+ catch (NullPointerException e) {
+ System.err.println("Error: " + e.toString());
+ return null;
+ }
+
+ switch (exec) { //check if matches extension
+ case "bash":
+ if (name.contains(".sh")) {
+ isOnTheList = true;
+ }
+ break;
+ case "sh":
+ if (name.contains(".sh")) {
+ isOnTheList = true;
+ }
+ break;
+ case "python":
+ if (name.contains(".py")) {
+ isOnTheList = true;
+ }
+ break;
+ case "java":
+ if (name.contains(".class")) {
+ isOnTheList = true;
+ // java cmd needs called w/o .class
+ name.replace(".class", "");
+ }
+ break;
+ }
+ }
+ if (!(isOnTheList)) {
+ return null;
+ }
+
+ try { // Create and start Process with ProcessBuilder.
+ ProcessBuilder pb = new ProcessBuilder(exec, name);
+ //Map<String, String> env = pb.environment();
+ //env.put("VAR1", "myValue");
+
+ File log = new File("/scripts/log");
+ pb.redirectErrorStream(true);
+ pb.redirectOutput(ProcessBuilder.Redirect.appendTo(log));
+ Process p = pb.start();
+ int exitCode = p.waitFor();
+
+ assert pb.redirectInput() == ProcessBuilder.Redirect.PIPE;
+ assert pb.redirectOutput().file() == log;
+ assert p.getInputStream().read() == -1;
+
+ return exitCode;
+ }
+ catch (NullPointerException e) {
+ System.err.println("Error: " + e.toString());
+ return null;
+ }
+ catch (IndexOutOfBoundsException e) {
+ System.err.println("Error: " + e.toString());
+ return null;
+ }
+ catch (SecurityException e) {
+ System.err.println("Error: " + e.toString());
+ return null;
+ }
+ catch (IOException e) {
+ System.err.println("Error: " + e.toString());
+ return null;
+ } catch (InterruptedException e) {
+ System.err.println("Error: " + e.toString());
+ return null;
+ }
+ }
+
+ @Override
+ public void initialize(Context context) {
+
+ }
+
+ @Override
+ public boolean isInitialized() {
+ return true;
+ }
+ }
+
+ /**
+ * Special thanks to alvin j. alexander at devdaily.com
+ */
+ class ThreadedStreamHandler extends Thread
+ {
+ InputStream inputStream;
+ String adminPassword;
+ OutputStream outputStream;
+ PrintWriter printWriter;
+ StringBuilder outputBuffer = new StringBuilder();
+ private boolean sudoIsRequested = false;
+
+ /**
+ * A simple constructor for when the sudo command is not necessary.
+ * This constructor will just run the command you provide, without
+ * running sudo before the command, and without expecting a password.
+ *
+ * @param inputStream
+ */
+ ThreadedStreamHandler(InputStream inputStream)
+ {
+ this.inputStream = inputStream;
+ }
+
+ /**
+ * Use this constructor when you want to invoke the 'sudo' command.
+ * The outputStream must not be null. If it is, you'll regret it. :)
+ *
+ * TODO this currently hangs if the admin password given for the sudo command is wrong.
+ *
+ * @param inputStream
+ * @param outputStream
+ * @param adminPassword
+ */
+ ThreadedStreamHandler(InputStream inputStream, OutputStream outputStream, String adminPassword)
+ {
+ this.inputStream = inputStream;
+ this.outputStream = outputStream;
+ this.printWriter = new PrintWriter(outputStream);
+ this.adminPassword = adminPassword;
+ this.sudoIsRequested = true;
+ }
+
+ public void run()
+ {
+ // on mac os x 10.5.x, when i run a 'sudo' command, i need to write
+ // the admin password out immediately; that's why this code is
+ // here.
+ if (sudoIsRequested)
+ {
+ //doSleep(500);
--- End diff --
I *think* there needs to be a command timeout
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #439: METRON-571 add stellar external function...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/439#discussion_r105325103
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+ public static class ExecuteScript implements StellarFunction {
+
+ private ThreadedStreamHandler inStream;
+ private ThreadedStreamHandler errStream;
+ private boolean isOnTheList = false;
+
+ @Stellar(name="EXEC_SCRIPT",
+ description = "Executes an external shell function via stellar.",
+ params = {
+ "exec - the executing cmd (ie. bash, sh, python)",
+ "name - name of the script, located in /scripts " +
+ "Do NOT include any special chars except(_), Do include file extension"
+ },
+ returns = "the return value of the function"
+ )
+
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+ String exec = "";
+ String name = "";
+ String path = "";
+
+ // if args are provided, get args, only if in whitelist
+ if (args.size() >= 1) {
+ Object execObj = args.get(0);
+ if (!(execObj instanceof String)) { //check if string
+ return null;
+ }
+ else if (((String) execObj).length() > 0) {
+ exec = (String) execObj;
+ }
+ else {
+ return null;
+ }
+
+ Object nameObj = args.get(1);
+ if (!(nameObj instanceof String)) { //check if string
+ return null;
+ }
+ else if (((String) nameObj).length() > 0) {
+ name = (String) nameObj;
+ }
+ else {
+ return null;
+ }
+
+ if (!Pattern.matches("[0-9A-Za-z.]+", name)) {
+ return null; //if not on whitelist
+ }
+
+ path = "/scripts" + name;
+ try {
+ File script = new File(path);
+ if (!script.exists() || script.isDirectory()) {
+ return null;
+ }
+ }
+ catch (NullPointerException e) {
+ System.err.println("Error: " + e.toString());
+ return null;
+ }
+
+ switch (exec) { //check if matches extension
+ case "bash":
+ if (name.contains(".sh")) {
+ isOnTheList = true;
+ }
+ break;
+ case "sh":
+ if (name.contains(".sh")) {
+ isOnTheList = true;
+ }
+ break;
+ case "python":
+ if (name.contains(".py")) {
+ isOnTheList = true;
+ }
--- End diff --
a big switch with lots of repeated code could be a sign for a refactor. this could be replaced with a Map
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #439: add stellar external functions feature (...
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/439#discussion_r99479385
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+ public static class ExecuteScript implements StellarFunction {
+
+ private ThreadedStreamHandler inStream;
+ private ThreadedStreamHandler errStream;
+ private boolean isOnTheList = false;
+
+ @Stellar(name="EXEC_SCRIPT",
+ description = "Executes an external shell function via stellar.",
+ params = {
+ "exec - the executing cmd (ie. bash, sh, python)",
+ "name - name of the script, located in /scripts " +
+ "Do NOT include any special chars except(_), Do include file extension"
+ },
+ returns = "the return value of the function"
+ )
+
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+ String exec = "";
+ String name = "";
+ String path = "";
+
+ // if args are provided, get args, only if in whitelist
+ if (args.size() >= 1) {
+ Object execObj = args.get(0);
+ if (!(execObj instanceof String)) { //check if string
+ return null;
+ }
+ else if (((String) execObj).length() > 0) {
+ exec = (String) execObj;
+ }
+ else {
+ return null;
+ }
+
+ Object nameObj = args.get(1);
+ if (!(nameObj instanceof String)) { //check if string
+ return null;
+ }
+ else if (((String) nameObj).length() > 0) {
+ name = (String) nameObj;
+ }
+ else {
+ return null;
+ }
+
+ if (!Pattern.matches("[0-9A-Za-z.]+", name)) {
+ return null; //if not on whitelist
+ }
+
+ path = "/scripts" + name;
+ try {
+ File script = new File(path);
+ if (!script.exists() || script.isDirectory()) {
+ return null;
+ }
+ }
+ catch (NullPointerException e) {
+ System.err.println("Error: " + e.toString());
+ return null;
+ }
+
+ switch (exec) { //check if matches extension
+ case "bash":
+ if (name.contains(".sh")) {
+ isOnTheList = true;
+ }
+ break;
+ case "sh":
+ if (name.contains(".sh")) {
+ isOnTheList = true;
+ }
+ break;
+ case "python":
+ if (name.contains(".py")) {
+ isOnTheList = true;
+ }
--- End diff --
These are reasonable checks, but you want .endsWith() rather than .contains()
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #439: METRON-571 add stellar external function...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/439#discussion_r105327868
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+ public static class ExecuteScript implements StellarFunction {
+
+ private ThreadedStreamHandler inStream;
+ private ThreadedStreamHandler errStream;
+ private boolean isOnTheList = false;
+
+ @Stellar(name="EXEC_SCRIPT",
+ description = "Executes an external shell function via stellar.",
+ params = {
+ "exec - the executing cmd (ie. bash, sh, python)",
+ "name - name of the script, located in /scripts " +
+ "Do NOT include any special chars except(_), Do include file extension"
+ },
+ returns = "the return value of the function"
+ )
+
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+ String exec = "";
+ String name = "";
+ String path = "";
+
--- End diff --
The ShellFunctions btw, are limited to the management package.
Is that were this needs to live?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #439: add stellar external functions feature (...
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/439#discussion_r99479325
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+ public static class ExecuteScript implements StellarFunction {
+
+ private ThreadedStreamHandler inStream;
+ private ThreadedStreamHandler errStream;
+ private boolean isOnTheList = false;
+
+ @Stellar(name="EXEC_SCRIPT",
+ description = "Executes an external shell function via stellar.",
+ params = {
+ "exec - the executing cmd (ie. bash, sh, python)",
+ "name - name of the script, located in /scripts " +
+ "Do NOT include any special chars except(_), Do include file extension"
+ },
+ returns = "the return value of the function"
+ )
+
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+ String exec = "";
+ String name = "";
+ String path = "";
+
+ // if args are provided, get args, only if in whitelist
+ if (args.size() >= 1) {
+ Object execObj = args.get(0);
+ if (!(execObj instanceof String)) { //check if string
+ return null;
+ }
+ else if (((String) execObj).length() > 0) {
+ exec = (String) execObj;
+ }
+ else {
+ return null;
+ }
+
+ Object nameObj = args.get(1);
+ if (!(nameObj instanceof String)) { //check if string
+ return null;
+ }
+ else if (((String) nameObj).length() > 0) {
+ name = (String) nameObj;
+ }
+ else {
+ return null;
+ }
+
+ if (!Pattern.matches("[0-9A-Za-z.]+", name)) {
+ return null; //if not on whitelist
+ }
+
+ path = "/scripts" + name;
+ try {
+ File script = new File(path);
+ if (!script.exists() || script.isDirectory()) {
+ return null;
+ }
+ }
+ catch (NullPointerException e) {
+ System.err.println("Error: " + e.toString());
+ return null;
+ }
--- End diff --
Rather than returning nulls, here and in many places throughout the code, it is better to actually throw meaningful exceptions, with explanatory text so that the user (assuming benign intent) has some ability to fix their mistakes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #439: add stellar external functions feature (...
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/439#discussion_r99479221
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+ public static class ExecuteScript implements StellarFunction {
+
+ private ThreadedStreamHandler inStream;
+ private ThreadedStreamHandler errStream;
+ private boolean isOnTheList = false;
+
+ @Stellar(name="EXEC_SCRIPT",
+ description = "Executes an external shell function via stellar.",
+ params = {
+ "exec - the executing cmd (ie. bash, sh, python)",
+ "name - name of the script, located in /scripts " +
+ "Do NOT include any special chars except(_), Do include file extension"
+ },
+ returns = "the return value of the function"
+ )
+
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+ String exec = "";
+ String name = "";
+ String path = "";
+
+ // if args are provided, get args, only if in whitelist
+ if (args.size() >= 1) {
+ Object execObj = args.get(0);
+ if (!(execObj instanceof String)) { //check if string
+ return null;
+ }
+ else if (((String) execObj).length() > 0) {
+ exec = (String) execObj;
+ }
+ else {
+ return null;
+ }
+
+ Object nameObj = args.get(1);
+ if (!(nameObj instanceof String)) { //check if string
+ return null;
+ }
+ else if (((String) nameObj).length() > 0) {
+ name = (String) nameObj;
+ }
+ else {
+ return null;
+ }
+
+ if (!Pattern.matches("[0-9A-Za-z.]+", name)) {
+ return null; //if not on whitelist
+ }
+
+ path = "/scripts" + name;
--- End diff --
You need another path delimiter "/" between the directory and the name. Since you're going to create a File object below, I would just use the File(String parent, String child) constructor instead.
Also, "/script" is an absolute path starting at root. This is unlikely to be the desired location for such scripts. I would suggest a configuration parameter for the location, defaulting to the install location + "/script". You would then fetch the location using the usual parameter fetching library routines that you can find in many java files in Metron.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #439: METRON-571 add stellar external function...
Posted by devopsec <gi...@git.apache.org>.
Github user devopsec commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/439#discussion_r99966589
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+ public static class ExecuteScript implements StellarFunction {
+
+ private ThreadedStreamHandler inStream;
+ private ThreadedStreamHandler errStream;
+ private boolean isOnTheList = false;
+
+ @Stellar(name="EXEC_SCRIPT",
+ description = "Executes an external shell function via stellar.",
+ params = {
+ "exec - the executing cmd (ie. bash, sh, python)",
+ "name - name of the script, located in /scripts " +
+ "Do NOT include any special chars except(_), Do include file extension"
+ },
+ returns = "the return value of the function"
+ )
+
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+ String exec = "";
+ String name = "";
+ String path = "";
+
+ // if args are provided, get args, only if in whitelist
+ if (args.size() >= 1) {
+ Object execObj = args.get(0);
+ if (!(execObj instanceof String)) { //check if string
+ return null;
+ }
+ else if (((String) execObj).length() > 0) {
+ exec = (String) execObj;
+ }
+ else {
+ return null;
+ }
+
+ Object nameObj = args.get(1);
+ if (!(nameObj instanceof String)) { //check if string
+ return null;
+ }
+ else if (((String) nameObj).length() > 0) {
+ name = (String) nameObj;
+ }
+ else {
+ return null;
+ }
+
+ if (!Pattern.matches("[0-9A-Za-z.]+", name)) {
+ return null; //if not on whitelist
+ }
+
+ path = "/scripts" + name;
--- End diff --
good point, I'd suggest user specified location then and just have a file chooser dialog open
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #439: METRON-571 add stellar external function...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/439#discussion_r105327240
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+ public static class ExecuteScript implements StellarFunction {
+
+ private ThreadedStreamHandler inStream;
+ private ThreadedStreamHandler errStream;
+ private boolean isOnTheList = false;
+
+ @Stellar(name="EXEC_SCRIPT",
+ description = "Executes an external shell function via stellar.",
+ params = {
+ "exec - the executing cmd (ie. bash, sh, python)",
+ "name - name of the script, located in /scripts " +
+ "Do NOT include any special chars except(_), Do include file extension"
+ },
+ returns = "the return value of the function"
+ )
+
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+ String exec = "";
+ String name = "";
+ String path = "";
+
--- End diff --
Do we need to do:
Optional<Object> console = context.getCapability(CONSOLE, true);
to make sure we have console for this?
see: ShellFunctions.java
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #439: METRON-571 add stellar external function...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/439#discussion_r105325302
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+ public static class ExecuteScript implements StellarFunction {
+
+ private ThreadedStreamHandler inStream;
+ private ThreadedStreamHandler errStream;
+ private boolean isOnTheList = false;
+
+ @Stellar(name="EXEC_SCRIPT",
+ description = "Executes an external shell function via stellar.",
+ params = {
+ "exec - the executing cmd (ie. bash, sh, python)",
+ "name - name of the script, located in /scripts " +
+ "Do NOT include any special chars except(_), Do include file extension"
+ },
+ returns = "the return value of the function"
+ )
+
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+ String exec = "";
+ String name = "";
+ String path = "";
+
+ // if args are provided, get args, only if in whitelist
+ if (args.size() >= 1) {
+ Object execObj = args.get(0);
+ if (!(execObj instanceof String)) { //check if string
+ return null;
+ }
+ else if (((String) execObj).length() > 0) {
+ exec = (String) execObj;
+ }
+ else {
+ return null;
+ }
+
+ Object nameObj = args.get(1);
+ if (!(nameObj instanceof String)) { //check if string
+ return null;
+ }
+ else if (((String) nameObj).length() > 0) {
+ name = (String) nameObj;
+ }
+ else {
+ return null;
+ }
+
+ if (!Pattern.matches("[0-9A-Za-z.]+", name)) {
+ return null; //if not on whitelist
+ }
+
+ path = "/scripts" + name;
+ try {
+ File script = new File(path);
+ if (!script.exists() || script.isDirectory()) {
+ return null;
+ }
+ }
+ catch (NullPointerException e) {
+ System.err.println("Error: " + e.toString());
+ return null;
+ }
+
+ switch (exec) { //check if matches extension
+ case "bash":
+ if (name.contains(".sh")) {
+ isOnTheList = true;
+ }
+ break;
+ case "sh":
+ if (name.contains(".sh")) {
+ isOnTheList = true;
+ }
+ break;
+ case "python":
+ if (name.contains(".py")) {
+ isOnTheList = true;
+ }
+ break;
+ case "java":
+ if (name.contains(".class")) {
+ isOnTheList = true;
+ // java cmd needs called w/o .class
+ name.replace(".class", "");
+ }
+ break;
+ }
+ }
+ if (!(isOnTheList)) {
+ return null;
+ }
+
+ try { // Create and start Process with ProcessBuilder.
+ ProcessBuilder pb = new ProcessBuilder(exec, name);
+ //Map<String, String> env = pb.environment();
+ //env.put("VAR1", "myValue");
+
+ File log = new File("/scripts/log");
+ pb.redirectErrorStream(true);
+ pb.redirectOutput(ProcessBuilder.Redirect.appendTo(log));
--- End diff --
I would not mix error and output
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #439: METRON-571 add stellar external function...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/439#discussion_r105325249
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+ public static class ExecuteScript implements StellarFunction {
+
+ private ThreadedStreamHandler inStream;
+ private ThreadedStreamHandler errStream;
+ private boolean isOnTheList = false;
+
+ @Stellar(name="EXEC_SCRIPT",
+ description = "Executes an external shell function via stellar.",
+ params = {
+ "exec - the executing cmd (ie. bash, sh, python)",
+ "name - name of the script, located in /scripts " +
+ "Do NOT include any special chars except(_), Do include file extension"
+ },
+ returns = "the return value of the function"
+ )
+
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+ String exec = "";
+ String name = "";
+ String path = "";
+
+ // if args are provided, get args, only if in whitelist
+ if (args.size() >= 1) {
+ Object execObj = args.get(0);
+ if (!(execObj instanceof String)) { //check if string
+ return null;
+ }
+ else if (((String) execObj).length() > 0) {
+ exec = (String) execObj;
+ }
+ else {
+ return null;
+ }
+
+ Object nameObj = args.get(1);
+ if (!(nameObj instanceof String)) { //check if string
+ return null;
+ }
+ else if (((String) nameObj).length() > 0) {
+ name = (String) nameObj;
+ }
+ else {
+ return null;
+ }
+
+ if (!Pattern.matches("[0-9A-Za-z.]+", name)) {
+ return null; //if not on whitelist
+ }
+
+ path = "/scripts" + name;
+ try {
+ File script = new File(path);
+ if (!script.exists() || script.isDirectory()) {
+ return null;
+ }
+ }
+ catch (NullPointerException e) {
+ System.err.println("Error: " + e.toString());
+ return null;
+ }
+
+ switch (exec) { //check if matches extension
+ case "bash":
+ if (name.contains(".sh")) {
+ isOnTheList = true;
+ }
+ break;
+ case "sh":
+ if (name.contains(".sh")) {
+ isOnTheList = true;
+ }
+ break;
+ case "python":
+ if (name.contains(".py")) {
+ isOnTheList = true;
+ }
+ break;
+ case "java":
+ if (name.contains(".class")) {
+ isOnTheList = true;
+ // java cmd needs called w/o .class
+ name.replace(".class", "");
+ }
+ break;
+ }
+ }
+ if (!(isOnTheList)) {
+ return null;
+ }
+
+ try { // Create and start Process with ProcessBuilder.
+ ProcessBuilder pb = new ProcessBuilder(exec, name);
+ //Map<String, String> env = pb.environment();
+ //env.put("VAR1", "myValue");
+
--- End diff --
so every command goes to the same log? That things is going to be a mess.
I would think that sooner or later we would want to have this output redirected back to the caller.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---