You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@metron.apache.org by ce...@apache.org on 2018/02/07 16:43:49 UTC
[1/8] metron git commit: METRON-1091 STELLAR Shell: Stand Alone
installation (ottobackwards) closes apache/metron#690
Repository: metron
Updated Branches:
refs/heads/feature/METRON-1416-upgrade-solr f525dcfd7 -> 67d6c0edd
METRON-1091 STELLAR Shell: Stand Alone installation (ottobackwards) closes apache/metron#690
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/124becd1
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/124becd1
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/124becd1
Branch: refs/heads/feature/METRON-1416-upgrade-solr
Commit: 124becd1a35d00e620d4eaed5d5d84c0abbe8ba9
Parents: 644e951
Author: ottobackwards <ot...@gmail.com>
Authored: Tue Jan 30 16:34:16 2018 -0500
Committer: otto <ot...@apache.org>
Committed: Tue Jan 30 16:34:16 2018 -0500
----------------------------------------------------------------------
metron-stellar/stellar-common/NOTICE | 6 +++
metron-stellar/stellar-common/README.md | 43 +++++++++++++++
metron-stellar/stellar-common/pom.xml | 1 +
.../src/main/assembly/assembly.xml | 2 +-
.../src/main/assembly/stand-alone-assembly.xml | 57 ++++++++++++++++++++
.../src/main/scripts/deployed/stellar | 36 +++++++++++++
.../src/main/scripts/stand-alone/stellar | 21 ++++++++
.../stellar-common/src/main/scripts/stellar | 36 -------------
8 files changed, 165 insertions(+), 37 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/124becd1/metron-stellar/stellar-common/NOTICE
----------------------------------------------------------------------
diff --git a/metron-stellar/stellar-common/NOTICE b/metron-stellar/stellar-common/NOTICE
new file mode 100644
index 0000000..1a19f3d
--- /dev/null
+++ b/metron-stellar/stellar-common/NOTICE
@@ -0,0 +1,6 @@
+ Apache Metron
+ Copyright 2015-2018 The Apache Software Foundation
+
+ This product includes software developed at
+ The Apache Software Foundation (http://www.apache.org/).
+
http://git-wip-us.apache.org/repos/asf/metron/blob/124becd1/metron-stellar/stellar-common/README.md
----------------------------------------------------------------------
diff --git a/metron-stellar/stellar-common/README.md b/metron-stellar/stellar-common/README.md
index 98aea77..a2f1242 100644
--- a/metron-stellar/stellar-common/README.md
+++ b/metron-stellar/stellar-common/README.md
@@ -1422,6 +1422,49 @@ Please note that functions are loading lazily in the background and will be unav
%functions
ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENDS_WITH, FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GET, GET_FIRST, GET_LAST, HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH, OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE, PREPEND_IF_MISSING, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW, PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD, STATS_BIN, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS, STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE, STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS, STATS_SUM, STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE, STRING_ENTROPY, SYS
TEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR
```
+## Stellar Shell Stand Alone
+
+The Stellar Shell is also packaged as a stand alone application. It can be unpacked on any supported
+operating system.
+
+> Only the base Stellar functions are available as packaged. Other functions, such as those in metron-profiler and metron-management are not available.
+
+
+```bash
+metron-stellar/stellar-common/target/stellar-common-0.4.3-stand-alone.tar.gz
+```
+
+When unpacked, the following structure will be created:
+
+```bash
+.
+├── bin
+│ └── stellar
+└── lib
+ └── stellar-common-0.4.3-uber.jar
+```
+
+To run the Stellar Shell run the following from the directory you unpacked to:
+
+```bash
+bin/stellar
+```
+
+```bash
+-> % bin/stellar
+Stellar, Go!
+Please note that functions are loading lazily in the background and will be unavailable until loaded fully.
+[Stellar]>>> Functions loaded, you may refer to functions now...
+
+[Stellar]>>> %functions
+ABS, APPEND_IF_MISSING, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, CEILING, CHOMP, CHOP, COS, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, DECODE, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENCODE, ENDS_WITH, EXP, FILL_LEFT, FILL_RIGHT, FILTER, FLOOR, FORMAT, GET, GET_FIRST, GET_LAST, GET_SUPPORTED_ENCODINGS, IN_SUBNET, IS_EMPTY, IS_ENCODING, JOIN, LENGTH, LIST_ADD, LN, LOG10, LOG2, MAP, MAP_EXISTS, MAP_GET, MONTH, PREPEND_IF_MISSING, REDUCE, REGEXP_GROUP_VAL, REGEXP_MATCH, ROUND, SIN, SPLIT, SQRT, STARTS_WITH, STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TAN, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR, ZIP, ZIP_LONGEST
+[Stellar]>>>
+```
+
+By default the shell will have the base Stellar Language commands available. Any jars in the lib directory
+that contain Stellar functions will also be loaded, and their commands will be available to shell, as long
+as their dependencies are satisfied.
+
### Implementation
http://git-wip-us.apache.org/repos/asf/metron/blob/124becd1/metron-stellar/stellar-common/pom.xml
----------------------------------------------------------------------
diff --git a/metron-stellar/stellar-common/pom.xml b/metron-stellar/stellar-common/pom.xml
index a5dd20b..b43fcc1 100644
--- a/metron-stellar/stellar-common/pom.xml
+++ b/metron-stellar/stellar-common/pom.xml
@@ -348,6 +348,7 @@
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<descriptor>src/main/assembly/assembly.xml</descriptor>
+ <descriptor>src/main/assembly/stand-alone-assembly.xml</descriptor>
</configuration>
<executions>
<execution>
http://git-wip-us.apache.org/repos/asf/metron/blob/124becd1/metron-stellar/stellar-common/src/main/assembly/assembly.xml
----------------------------------------------------------------------
diff --git a/metron-stellar/stellar-common/src/main/assembly/assembly.xml b/metron-stellar/stellar-common/src/main/assembly/assembly.xml
index 0b5dfb8..b50be00 100644
--- a/metron-stellar/stellar-common/src/main/assembly/assembly.xml
+++ b/metron-stellar/stellar-common/src/main/assembly/assembly.xml
@@ -19,7 +19,7 @@
<includeBaseDirectory>false</includeBaseDirectory>
<fileSets>
<fileSet>
- <directory>${project.basedir}/src/main/scripts</directory>
+ <directory>${project.basedir}/src/main/scripts/deployed</directory>
<outputDirectory>bin</outputDirectory>
<useDefaultExcludes>true</useDefaultExcludes>
<excludes>
http://git-wip-us.apache.org/repos/asf/metron/blob/124becd1/metron-stellar/stellar-common/src/main/assembly/stand-alone-assembly.xml
----------------------------------------------------------------------
diff --git a/metron-stellar/stellar-common/src/main/assembly/stand-alone-assembly.xml b/metron-stellar/stellar-common/src/main/assembly/stand-alone-assembly.xml
new file mode 100644
index 0000000..490f02e
--- /dev/null
+++ b/metron-stellar/stellar-common/src/main/assembly/stand-alone-assembly.xml
@@ -0,0 +1,57 @@
+<!--
+ Licensed to the Apache Software
+ Foundation (ASF) under one or more contributor license agreements. See the
+ NOTICE file distributed with this work for additional information regarding
+ copyright ownership. The ASF licenses this file to You under the Apache License,
+ Version 2.0 (the "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software distributed
+ under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ OR CONDITIONS OF ANY KIND, either express or implied. See the License for
+ the specific language governing permissions and limitations under the License.
+ -->
+
+<assembly>
+ <id>stand-alone</id>
+ <formats>
+ <format>tar.gz</format>
+ </formats>
+ <includeBaseDirectory>false</includeBaseDirectory>
+ <fileSets>
+ <fileSet>
+ <directory>${project.basedir}/src/main/scripts/stand-alone</directory>
+ <outputDirectory>bin</outputDirectory>
+ <useDefaultExcludes>true</useDefaultExcludes>
+ <excludes>
+ <exclude>**/*.formatted</exclude>
+ <exclude>**/*.filtered</exclude>
+ </excludes>
+ <fileMode>0755</fileMode>
+ <lineEnding>unix</lineEnding>
+ <filtered>true</filtered>
+ </fileSet>
+ <fileSet>
+ <directory>${project.basedir}/target</directory>
+ <includes>
+ <include>${project.artifactId}-${project.version}-uber.jar</include>
+ </includes>
+ <outputDirectory>lib</outputDirectory>
+ <useDefaultExcludes>true</useDefaultExcludes>
+ </fileSet>
+ <fileSet>
+ <directory>${project.basedir}</directory>
+ <includes>
+ <include>README.md</include>
+ </includes>
+ <outputDirectory></outputDirectory>
+ </fileSet>
+ <fileSet>
+ <directory>${project.basedir}/src/main/resources/META-INF</directory>
+ <includes>
+ <include>LICENSE</include>
+ <include>NOTICE</include>
+ </includes>
+ <outputDirectory></outputDirectory>
+ </fileSet>
+ </fileSets>
+</assembly>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/metron/blob/124becd1/metron-stellar/stellar-common/src/main/scripts/deployed/stellar
----------------------------------------------------------------------
diff --git a/metron-stellar/stellar-common/src/main/scripts/deployed/stellar b/metron-stellar/stellar-common/src/main/scripts/deployed/stellar
new file mode 100644
index 0000000..7b0f06d
--- /dev/null
+++ b/metron-stellar/stellar-common/src/main/scripts/deployed/stellar
@@ -0,0 +1,36 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+
+BIGTOP_DEFAULTS_DIR=${BIGTOP_DEFAULTS_DIR-/etc/default}
+[ -n "${BIGTOP_DEFAULTS_DIR}" -a -r ${BIGTOP_DEFAULTS_DIR}/hbase ] && . ${BIGTOP_DEFAULTS_DIR}/hbase
+
+# Autodetect JAVA_HOME if not defined
+if [ -e /usr/libexec/bigtop-detect-javahome ]; then
+ . /usr/libexec/bigtop-detect-javahome
+elif [ -e /usr/lib/bigtop-utils/bigtop-detect-javahome ]; then
+ . /usr/lib/bigtop-utils/bigtop-detect-javahome
+fi
+
+export HBASE_CONFIGS=$(hbase classpath)
+export METRON_VERSION=${project.version}
+export METRON_HOME=/usr/metron/$METRON_VERSION
+export STELLAR_LIB=$(find $METRON_HOME/lib/ -name metron-parsers*.jar)
+export MANAGEMENT_LIB=$(find $METRON_HOME/lib/ -name metron-management*.jar)
+java $JVMFLAGS -cp "${CONTRIB:-$METRON_HOME/contrib}:$STELLAR_LIB:$MANAGEMENT_LIB:$HBASE_CONFIGS" org.apache.metron.stellar.common.shell.cli.StellarShell "$@"
http://git-wip-us.apache.org/repos/asf/metron/blob/124becd1/metron-stellar/stellar-common/src/main/scripts/stand-alone/stellar
----------------------------------------------------------------------
diff --git a/metron-stellar/stellar-common/src/main/scripts/stand-alone/stellar b/metron-stellar/stellar-common/src/main/scripts/stand-alone/stellar
new file mode 100644
index 0000000..7906a32
--- /dev/null
+++ b/metron-stellar/stellar-common/src/main/scripts/stand-alone/stellar
@@ -0,0 +1,21 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+binDir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+java $JVMFLAGS -cp "${binDir}/../lib/*" org.apache.metron.stellar.common.shell.cli.StellarShell "$@"
http://git-wip-us.apache.org/repos/asf/metron/blob/124becd1/metron-stellar/stellar-common/src/main/scripts/stellar
----------------------------------------------------------------------
diff --git a/metron-stellar/stellar-common/src/main/scripts/stellar b/metron-stellar/stellar-common/src/main/scripts/stellar
deleted file mode 100644
index 7b0f06d..0000000
--- a/metron-stellar/stellar-common/src/main/scripts/stellar
+++ /dev/null
@@ -1,36 +0,0 @@
-#!/bin/bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-
-BIGTOP_DEFAULTS_DIR=${BIGTOP_DEFAULTS_DIR-/etc/default}
-[ -n "${BIGTOP_DEFAULTS_DIR}" -a -r ${BIGTOP_DEFAULTS_DIR}/hbase ] && . ${BIGTOP_DEFAULTS_DIR}/hbase
-
-# Autodetect JAVA_HOME if not defined
-if [ -e /usr/libexec/bigtop-detect-javahome ]; then
- . /usr/libexec/bigtop-detect-javahome
-elif [ -e /usr/lib/bigtop-utils/bigtop-detect-javahome ]; then
- . /usr/lib/bigtop-utils/bigtop-detect-javahome
-fi
-
-export HBASE_CONFIGS=$(hbase classpath)
-export METRON_VERSION=${project.version}
-export METRON_HOME=/usr/metron/$METRON_VERSION
-export STELLAR_LIB=$(find $METRON_HOME/lib/ -name metron-parsers*.jar)
-export MANAGEMENT_LIB=$(find $METRON_HOME/lib/ -name metron-management*.jar)
-java $JVMFLAGS -cp "${CONTRIB:-$METRON_HOME/contrib}:$STELLAR_LIB:$MANAGEMENT_LIB:$HBASE_CONFIGS" org.apache.metron.stellar.common.shell.cli.StellarShell "$@"
[8/8] metron git commit: Merge branch 'master' into
feature/METRON-1416-upgrade-solr
Posted by ce...@apache.org.
Merge branch 'master' into feature/METRON-1416-upgrade-solr
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/67d6c0ed
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/67d6c0ed
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/67d6c0ed
Branch: refs/heads/feature/METRON-1416-upgrade-solr
Commit: 67d6c0edd731f528af759ca11de875cbf77f6b2c
Parents: f525dcf 38b7d5e
Author: cstella <ce...@gmail.com>
Authored: Wed Feb 7 11:43:34 2018 -0500
Committer: cstella <ce...@gmail.com>
Committed: Wed Feb 7 11:43:34 2018 -0500
----------------------------------------------------------------------
.../roles/java_jdk/tasks/install_jdk_ubuntu.yml | 2 +-
.../packaging/ambari/metron-mpack/README.md | 14 +
metron-deployment/scripts/platform-info.sh | 4 +-
.../advanced-config-form.component.html | 2 +-
.../advanced-config-form.component.spec.ts | 34 +++
.../advanced-config-form.component.ts | 20 +-
.../src/main/config/rest_application.yml | 3 +-
.../apache/metron/rest/MetronRestConstants.java | 6 +-
.../metron/rest/controller/StormController.java | 81 +++--
.../metron/rest/service/StormAdminService.java | 4 +-
.../service/impl/StormAdminServiceImpl.java | 8 +-
.../rest/service/impl/StormCLIWrapper.java | 16 +-
.../src/main/resources/application-test.yml | 3 +-
.../src/main/resources/application-vagrant.yml | 4 +-
.../StormControllerIntegrationTest.java | 102 ++++---
.../rest/mock/MockStormCLIClientWrapper.java | 93 ++++--
.../metron/rest/mock/MockStormRestTemplate.java | 22 +-
.../service/impl/StormAdminServiceImplTest.java | 8 +-
.../rest/service/impl/StormCLIWrapperTest.java | 17 +-
metron-platform/metron-management/README.md | 32 +-
.../metron/management/ShellFunctions.java | 302 -------------------
.../metron/management/ShellFunctionsTest.java | 171 -----------
metron-stellar/stellar-common/NOTICE | 6 +
metron-stellar/stellar-common/README.md | 74 +++++
metron-stellar/stellar-common/pom.xml | 6 +
.../src/main/assembly/assembly.xml | 2 +-
.../src/main/assembly/stand-alone-assembly.xml | 57 ++++
.../stellar/common/shell/cli/PausableInput.java | 23 +-
.../stellar/dsl/functions/ShellFunctions.java | 301 ++++++++++++++++++
.../src/main/scripts/deployed/stellar | 36 +++
.../src/main/scripts/stand-alone/stellar | 21 ++
.../stellar-common/src/main/scripts/stellar | 36 ---
.../dsl/functions/ShellFunctionsTest.java | 176 +++++++++++
33 files changed, 998 insertions(+), 688 deletions(-)
----------------------------------------------------------------------
[2/8] metron git commit: METRON-1439 Turn off git pager in
platform-info script (justinleet) closes apache/metron#919
Posted by ce...@apache.org.
METRON-1439 Turn off git pager in platform-info script (justinleet) closes apache/metron#919
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/567d106b
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/567d106b
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/567d106b
Branch: refs/heads/feature/METRON-1416-upgrade-solr
Commit: 567d106b1fbe86e828f3c4068024df42b2f4ddc0
Parents: 124becd
Author: justinleet <ju...@gmail.com>
Authored: Thu Feb 1 13:46:55 2018 -0500
Committer: leet <le...@apache.org>
Committed: Thu Feb 1 13:46:55 2018 -0500
----------------------------------------------------------------------
metron-deployment/scripts/platform-info.sh | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/567d106b/metron-deployment/scripts/platform-info.sh
----------------------------------------------------------------------
diff --git a/metron-deployment/scripts/platform-info.sh b/metron-deployment/scripts/platform-info.sh
index 98e0bcf..fd2248a 100755
--- a/metron-deployment/scripts/platform-info.sh
+++ b/metron-deployment/scripts/platform-info.sh
@@ -36,11 +36,11 @@ if [ "$IS_GIT_REPO" == "true" ]; then
# last commit
echo "--"
- git log -n 1
+ git --no-pager log -n 1
# local changes since last commit
echo "--"
- git diff --stat
+ git --no-pager diff --stat
fi
# ansible
[3/8] metron git commit: METRON-1435 Management UI cannot save json
objects in advanced config (merrimanr) closes apache/metron#917
Posted by ce...@apache.org.
METRON-1435 Management UI cannot save json objects in advanced config (merrimanr) closes apache/metron#917
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/3d3c43c7
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/3d3c43c7
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/3d3c43c7
Branch: refs/heads/feature/METRON-1416-upgrade-solr
Commit: 3d3c43c716b0a6d58b57d6d5ba83109bc49661e4
Parents: 567d106
Author: merrimanr <me...@gmail.com>
Authored: Fri Feb 2 08:37:29 2018 -0600
Committer: merrimanr <me...@apache.org>
Committed: Fri Feb 2 08:37:29 2018 -0600
----------------------------------------------------------------------
.../advanced-config-form.component.html | 2 +-
.../advanced-config-form.component.spec.ts | 34 ++++++++++++++++++++
.../advanced-config-form.component.ts | 20 +++++++++++-
3 files changed, 54 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/3d3c43c7/metron-interface/metron-config/src/app/shared/advanced-config-form/advanced-config-form.component.html
----------------------------------------------------------------------
diff --git a/metron-interface/metron-config/src/app/shared/advanced-config-form/advanced-config-form.component.html b/metron-interface/metron-config/src/app/shared/advanced-config-form/advanced-config-form.component.html
index 1a30ee7..945be5f 100644
--- a/metron-interface/metron-config/src/app/shared/advanced-config-form/advanced-config-form.component.html
+++ b/metron-interface/metron-config/src/app/shared/advanced-config-form/advanced-config-form.component.html
@@ -19,7 +19,7 @@
</div>
</div>
<div class="row mx-0">
- <div class="col-md-10 advanced-input"><input type="text" class="form-control" formControlName="{{key}}" [(ngModel)]="config[key]"></div>
+ <div class="col-md-10 advanced-input"><input type="text" class="form-control" formControlName="{{key}}" [ngModel]="displayValue(key)" (ngModelChange)="saveValue(key, $event)"></div>
<div class="col-md-2" (click)="removeConfig(key)">
<i class="fa fa-minus fa-4 icon-button" aria-hidden="true" ></i>
</div>
http://git-wip-us.apache.org/repos/asf/metron/blob/3d3c43c7/metron-interface/metron-config/src/app/shared/advanced-config-form/advanced-config-form.component.spec.ts
----------------------------------------------------------------------
diff --git a/metron-interface/metron-config/src/app/shared/advanced-config-form/advanced-config-form.component.spec.ts b/metron-interface/metron-config/src/app/shared/advanced-config-form/advanced-config-form.component.spec.ts
index a8f0ed0..ed80c54 100644
--- a/metron-interface/metron-config/src/app/shared/advanced-config-form/advanced-config-form.component.spec.ts
+++ b/metron-interface/metron-config/src/app/shared/advanced-config-form/advanced-config-form.component.spec.ts
@@ -136,6 +136,18 @@ describe('Component: AdvancedConfigFormComponent', () => {
expect(component.configForm.controls['field2'].value).toEqual('value2');
expect(component.configForm.controls['field3'].value).toEqual('value3');
+ component.newConfigKey = 'field1';
+ component.newConfigValue = '["newValue1"]';
+ component.saveNewConfig();
+ expect(Object.keys(component.config).length).toEqual(3);
+ expect(component.config['field1']).toEqual(['newValue1']);
+
+ component.newConfigKey = 'field1';
+ component.newConfigValue = '{"key":"newValue1"}';
+ component.saveNewConfig();
+ expect(Object.keys(component.config).length).toEqual(3);
+ expect(component.config['field1']).toEqual({key: 'newValue1'});
+
component.removeConfig('field1');
expect(Object.keys(component.config).length).toEqual(2);
expect(component.config['field2']).toEqual('value2');
@@ -146,6 +158,28 @@ describe('Component: AdvancedConfigFormComponent', () => {
expect(component.configForm.controls['newConfigValue'].value).toEqual('enter value');
expect(component.configForm.controls['field2'].value).toEqual('value2');
expect(component.configForm.controls['field3'].value).toEqual('value3');
+
+
}));
+ it('verify display and save values', async(() => {
+ let component: AdvancedConfigFormComponent = fixture.componentInstance;
+ component.config = {'field1': 'value1', 'field2': 'value2'};
+ component.ngOnInit();
+
+ expect(component.displayValue('field1')).toEqual('value1');
+
+ component.saveValue('field1', '["value1","value2"]');
+ expect(component.config['field1']).toEqual(['value1', 'value2']);
+ expect(component.displayValue('field1')).toEqual('["value1","value2"]');
+
+ component.saveValue('field1', '["value1","value2"');
+ expect(component.config['field1']).toEqual('["value1","value2"');
+ expect(component.displayValue('field1')).toEqual('["value1","value2"');
+
+ component.saveValue('field1', '{"key1":"value1"}');
+ expect(component.config['field1']).toEqual({'key1': 'value1'});
+ expect(component.displayValue('field1')).toEqual('{"key1":"value1"}');
+ }));
+
});
http://git-wip-us.apache.org/repos/asf/metron/blob/3d3c43c7/metron-interface/metron-config/src/app/shared/advanced-config-form/advanced-config-form.component.ts
----------------------------------------------------------------------
diff --git a/metron-interface/metron-config/src/app/shared/advanced-config-form/advanced-config-form.component.ts b/metron-interface/metron-config/src/app/shared/advanced-config-form/advanced-config-form.component.ts
index f363391..5da9d48 100644
--- a/metron-interface/metron-config/src/app/shared/advanced-config-form/advanced-config-form.component.ts
+++ b/metron-interface/metron-config/src/app/shared/advanced-config-form/advanced-config-form.component.ts
@@ -83,7 +83,7 @@ export class AdvancedConfigFormComponent implements OnInit, OnChanges {
}
if (this.newConfigKey !== 'enter field' && this.newConfigValue !== 'enter value') {
let keyExists = this.config[this.newConfigKey] !== undefined;
- this.config[this.newConfigKey] = this.newConfigValue;
+ this.saveValue(this.newConfigKey, this.newConfigValue);
if (keyExists) {
this.newConfigKey = 'enter field';
this.newConfigValue = 'enter value';
@@ -107,4 +107,22 @@ export class AdvancedConfigFormComponent implements OnInit, OnChanges {
this.configForm.removeControl(key);
}
+ displayValue(key: string): string {
+ let value = this.config[key];
+ if (Array.isArray(value) || value instanceof Object) {
+ return JSON.stringify(value);
+ } else {
+ return value;
+ }
+ }
+
+ saveValue(key: string, value: string) {
+ try {
+ this.config[key] = JSON.parse(value);
+ } catch (err) {
+ this.config[key] = value;
+ }
+
+ }
+
}
[4/8] metron git commit: METRON-1438 STELLAR: Move shell functions to
common from metron-management (ottobackwards) closes apache/metron#920
Posted by ce...@apache.org.
METRON-1438 STELLAR: Move shell functions to common from metron-management (ottobackwards) closes apache/metron#920
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/c26abbba
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/c26abbba
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/c26abbba
Branch: refs/heads/feature/METRON-1416-upgrade-solr
Commit: c26abbbaeaeea3551e218cb9aa8ba97b32655958
Parents: 3d3c43c
Author: ottobackwards <ot...@gmail.com>
Authored: Fri Feb 2 09:58:59 2018 -0500
Committer: otto <ot...@apache.org>
Committed: Fri Feb 2 09:58:59 2018 -0500
----------------------------------------------------------------------
metron-platform/metron-management/README.md | 32 +-
.../metron/management/ShellFunctions.java | 302 -------------------
.../metron/management/ShellFunctionsTest.java | 171 -----------
metron-stellar/stellar-common/README.md | 31 ++
metron-stellar/stellar-common/pom.xml | 5 +
.../stellar/common/shell/cli/PausableInput.java | 23 +-
.../stellar/dsl/functions/ShellFunctions.java | 301 ++++++++++++++++++
.../dsl/functions/ShellFunctionsTest.java | 176 +++++++++++
8 files changed, 529 insertions(+), 512 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/c26abbba/metron-platform/metron-management/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-management/README.md b/metron-platform/metron-management/README.md
index 812583c..bf939c2 100644
--- a/metron-platform/metron-management/README.md
+++ b/metron-platform/metron-management/README.md
@@ -35,14 +35,13 @@ project.
* [Functions](#functions)
* [Grok Functions](#grok-functions)
* [File Functions](#file-functions)
- * [Shell Functions](#shell-functions)
* [Configuration Functions](#configuration-functions)
* [Parser Functions](#parser-functions)
* [Indexing Functions](#indexing-functions)
* [Enrichment Functions](#enrichment-functions)
* [Threat Triage Functions](#threat-triage-functions)
* [Examples](#examples)
- * [Iterate to Find a Valid Grok Pattern](#iterate-to-find-a-valid-grok-pattern)
+ * [Iterate to Find a Valid Grok Pattern](#iterate-to-find-a-valid-grok-pattern)
* [Manage Stellar Field Transformations](#manage-stellar-field-transformations)
* [Manage Stellar Enrichments](#manage-stellar-enrichments)
* [Manage Threat Triage Rules](#manage-threat-triage-rules)
@@ -132,35 +131,6 @@ The functions are split roughly into a few sections:
* path - The path of the file
* Returns: true if the file was written and false otherwise.
-### Shell Functions
-
-* `SHELL_EDIT`
- * Description: Open an editor (optionally initialized with text) and return whatever is saved from the editor. The editor to use is pulled from `EDITOR` or `VISUAL` environment variable.
- * Input:
- * string - (Optional) A string whose content is used to initialize the editor.
- * Returns: The content that the editor saved after editor exit.
-* `SHELL_GET_EXPRESSION`
- * Description: Get a stellar expression from a variable
- * Input:
- * variable - variable name
- * Returns: The stellar expression associated with the variable.
-* `SHELL_LIST_VARS`
- * Description: Return the variables in a tabular form
- * Input:
- * wrap : Length of string to wrap the columns
- * Returns: A tabular representation of the variables.
-* `SHELL_MAP2TABLE`
- * Description: Take a map and return a table
- * Input:
- * map - Map
- * Returns: The map in table form
-* `SHELL_VARS2MAP`
- * Description: Take a set of variables and return a map
- * Input:
- * variables* - variable names to use to create map
- * Returns: A map associating the variable name with the stellar expression.
-
-
### Configuration Functions
* `CONFIG_GET`
http://git-wip-us.apache.org/repos/asf/metron/blob/c26abbba/metron-platform/metron-management/src/main/java/org/apache/metron/management/ShellFunctions.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-management/src/main/java/org/apache/metron/management/ShellFunctions.java b/metron-platform/metron-management/src/main/java/org/apache/metron/management/ShellFunctions.java
deleted file mode 100644
index afac7f0..0000000
--- a/metron-platform/metron-management/src/main/java/org/apache/metron/management/ShellFunctions.java
+++ /dev/null
@@ -1,302 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.metron.management;
-
-import com.jakewharton.fliptables.FlipTable;
-import org.apache.commons.io.IOUtils;
-import org.apache.commons.lang3.text.WordUtils;
-import org.apache.metron.stellar.common.shell.VariableResult;
-import org.apache.metron.stellar.common.shell.cli.PausableInput;
-import org.apache.metron.stellar.common.utils.ConversionUtils;
-import org.apache.metron.stellar.dsl.BaseStellarFunction;
-import org.apache.metron.stellar.dsl.Context;
-import org.apache.metron.stellar.dsl.ParseException;
-import org.apache.metron.stellar.dsl.Stellar;
-import org.apache.metron.stellar.dsl.StellarFunction;
-import org.jboss.aesh.console.Console;
-import org.slf4j.LoggerFactory;
-
-import java.io.BufferedReader;
-import java.io.File;
-import java.io.FileReader;
-import java.io.IOException;
-import java.io.PrintWriter;
-import java.lang.invoke.MethodHandles;
-import java.util.HashMap;
-import java.util.LinkedHashMap;
-import java.util.List;
-import java.util.Map;
-import java.util.Optional;
-
-import static org.apache.metron.stellar.dsl.Context.Capabilities.CONSOLE;
-
-public class ShellFunctions {
- private static final org.slf4j.Logger LOG = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
-
- private static Map<String, VariableResult> getVariables(Context context) {
- return (Map<String, VariableResult>) context.getCapability(Context.Capabilities.SHELL_VARIABLES).get();
- }
-
- @Stellar(
- namespace = "SHELL"
- ,name = "MAP2TABLE"
- ,description = "Take a map and return a table"
- ,params = {"map - Map"
- }
- ,returns = "The map in table form"
- )
- public static class Map2Table extends BaseStellarFunction {
-
- @Override
- public Object apply(List<Object> args) {
- if(args.size() < 1) {
- return null;
- }
- Map<Object, Object> map = (Map<Object, Object>) args.get(0);
- if(map == null) {
- map = new HashMap<>();
- }
- String[] headers = {"KEY", "VALUE"};
- String[][] data = new String[map.size()][2];
- int i = 0;
- for(Map.Entry<Object, Object> kv : map.entrySet()) {
- data[i++] = new String[] {kv.getKey().toString(), kv.getValue().toString()};
- }
- return FlipTable.of(headers, data);
- }
- }
-
- @Stellar(
- namespace = "SHELL"
- ,name = "LIST_VARS"
- ,description = "Return the variables in a tabular form"
- ,params = {
- "wrap : Length of string to wrap the columns"
- }
- ,returns = "A tabular representation of the variables."
- )
- public static class ListVars implements StellarFunction {
-
- @Override
- public Object apply(List<Object> args, Context context) throws ParseException {
-
- Map<String, VariableResult> variables = getVariables(context);
- String[] headers = {"VARIABLE", "VALUE", "EXPRESSION"};
- String[][] data = new String[variables.size()][3];
- int wordWrap = -1;
- if(args.size() > 0) {
- wordWrap = ConversionUtils.convert(args.get(0), Integer.class);
- }
- int i = 0;
- for(Map.Entry<String, VariableResult> kv : variables.entrySet()) {
- VariableResult result = kv.getValue();
- data[i++] = new String[] { toWrappedString(kv.getKey().toString(), wordWrap)
- , toWrappedString(result.getResult(), wordWrap)
- , toWrappedString(result.getExpression().get(), wordWrap)
- };
- }
- return FlipTable.of(headers, data);
- }
-
- private static String toWrappedString(Object o, int wrap) {
- String s = "" + o;
- if(wrap <= 0) {
- return s;
- }
- return WordUtils.wrap(s, wrap);
- }
-
- @Override
- public void initialize(Context context) {
-
- }
-
- @Override
- public boolean isInitialized() {
- return true;
- }
- }
-
- @Stellar(
- namespace = "SHELL"
- ,name = "VARS2MAP"
- ,description = "Take a set of variables and return a map"
- ,params = {"variables* - variable names to use to create map "
- }
- ,returns = "A map associating the variable name with the stellar expression."
- )
- public static class Var2Map implements StellarFunction {
-
- @Override
- public Object apply(List<Object> args, Context context) throws ParseException {
- Map<String, VariableResult> variables = getVariables(context);
- LinkedHashMap<String, String> ret = new LinkedHashMap<>();
- for(Object arg : args) {
- if(arg == null) {
- continue;
- }
- String variable = (String)arg;
- VariableResult result = variables.get(variable);
- if(result != null && result.getExpression().isPresent()) {
- ret.put(variable, result.getExpression().orElseGet(() -> ""));
- }
- }
- return ret;
- }
-
- @Override
- public void initialize(Context context) {
-
- }
-
- @Override
- public boolean isInitialized() {
- return true;
- }
- }
-
- @Stellar(
- namespace = "SHELL"
- ,name = "GET_EXPRESSION"
- ,description = "Get a stellar expression from a variable"
- ,params = {"variable - variable name"
- }
- ,returns = "The stellar expression associated with the variable."
- )
- public static class GetExpression implements StellarFunction {
-
- @Override
- public Object apply(List<Object> args, Context context) throws ParseException {
- Map<String, VariableResult> variables = getVariables(context);
- if(args.size() == 0) {
- return null;
- }
- String variable = (String) args.get(0);
- if(variable == null) {
- return null;
- }
- VariableResult result = variables.get(variable);
- if(result != null && result.getExpression().isPresent()) {
- return result.getExpression().get();
- }
- return null;
- }
-
- @Override
- public void initialize(Context context) {
-
- }
-
- @Override
- public boolean isInitialized() {
- return true;
- }
- }
-
- @Stellar(
- namespace = "SHELL"
- ,name = "EDIT"
- ,description = "Open an editor (optionally initialized with text) and return " +
- "whatever is saved from the editor. The editor to use is pulled " +
- "from `EDITOR` or `VISUAL` environment variable."
- ,params = { "string - (Optional) A string whose content is used to initialize the editor."
- }
- ,returns = "The content that the editor saved after editor exit."
- )
- public static class Edit implements StellarFunction {
-
- private String getEditor() {
- // if we have editor in the system properties, it should
- // override the env so we check that first
- String editor = System.getProperty("EDITOR");
- if(org.apache.commons.lang3.StringUtils.isEmpty(editor)) {
- editor = System.getenv().get("EDITOR");
- }
- if(org.apache.commons.lang3.StringUtils.isEmpty(editor)) {
- editor = System.getenv("VISUAL");
- }
- if(org.apache.commons.lang3.StringUtils.isEmpty(editor)) {
- editor = "/bin/vi";
- }
- return editor;
- }
-
- @Override
- public Object apply(List<Object> args, Context context) throws ParseException {
- File outFile = null;
- String editor = getEditor();
- try {
- outFile = File.createTempFile("stellar_shell", "out");
- if(args.size() > 0) {
- String arg = (String)args.get(0);
- try(PrintWriter pw = new PrintWriter(outFile)) {
- IOUtils.write(arg, pw);
- }
- }
- } catch (IOException e) {
- String message = "Unable to create temp file: " + e.getMessage();
- LOG.error(message, e);
- throw new IllegalStateException(message, e);
- }
- Optional<Object> console = context.getCapability(CONSOLE, false);
- try {
- PausableInput.INSTANCE.pause();
- //shut down the IO for the console
- ProcessBuilder processBuilder = new ProcessBuilder(editor, outFile.getAbsolutePath());
- processBuilder.redirectInput(ProcessBuilder.Redirect.INHERIT);
- processBuilder.redirectOutput(ProcessBuilder.Redirect.INHERIT);
- processBuilder.redirectError(ProcessBuilder.Redirect.INHERIT);
- try {
- Process p = processBuilder.start();
- // wait for termination.
- p.waitFor();
- try (BufferedReader br = new BufferedReader(new FileReader(outFile))) {
- String ret = IOUtils.toString(br).trim();
- return ret;
- }
- } catch (Exception e) {
- String message = "Unable to read output: " + e.getMessage();
- LOG.error(message, e);
- return null;
- }
- } finally {
- try {
- PausableInput.INSTANCE.unpause();
- if(console.isPresent()) {
- ((Console)console.get()).pushToInputStream("\b\n");
- }
- } catch (IOException e) {
- LOG.error("Unable to unpause: {}", e.getMessage(), e);
- }
- if(outFile.exists()) {
- outFile.delete();
- }
- }
- }
-
- @Override
- public void initialize(Context context) {
-
- }
-
- @Override
- public boolean isInitialized() {
- return true;
- }
- }
-}
http://git-wip-us.apache.org/repos/asf/metron/blob/c26abbba/metron-platform/metron-management/src/test/java/org/apache/metron/management/ShellFunctionsTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-management/src/test/java/org/apache/metron/management/ShellFunctionsTest.java b/metron-platform/metron-management/src/test/java/org/apache/metron/management/ShellFunctionsTest.java
deleted file mode 100644
index 83c2bce..0000000
--- a/metron-platform/metron-management/src/test/java/org/apache/metron/management/ShellFunctionsTest.java
+++ /dev/null
@@ -1,171 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.metron.management;
-
-import com.google.common.collect.ImmutableMap;
-import org.adrianwalker.multilinestring.Multiline;
-import org.apache.metron.stellar.common.shell.VariableResult;
-import org.apache.metron.stellar.dsl.Context;
-import org.junit.Assert;
-import org.junit.Test;
-
-import java.util.HashMap;
-import java.util.Map;
-import java.util.Optional;
-
-import static org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
-
-public class ShellFunctionsTest {
-
- Map<String, VariableResult> variables = ImmutableMap.of(
- "var1" , VariableResult.withExpression("CASEY", "TO_UPPER('casey')"),
- "var2" , VariableResult.withValue("foo"),
- "var3" , VariableResult.withValue(null),
- "var4" , VariableResult.withExpression(null, "blah")
- );
-
- Context context = new Context.Builder()
- .with(Context.Capabilities.SHELL_VARIABLES , () -> variables)
- .build();
-/**
-╔══════════╤═══════╤════════════╗
-║ VARIABLE │ VALUE │ EXPRESSION ║
-╠══════════╪═══════╪════════════╣
-║ foo │ 2.0 │ 1 + 1 ║
-╚══════════╧═══════╧════════════╝
- **/
- @Multiline
- static String expectedListWithFoo;
-
- @Test
- public void testListVarsWithVars() {
- Map<String, VariableResult> variables = ImmutableMap.of(
- "foo", VariableResult.withExpression(2.0, "1 + 1"));
-
- Context context = new Context.Builder()
- .with(Context.Capabilities.SHELL_VARIABLES , () -> variables)
- .build();
- Object out = run("SHELL_LIST_VARS()", new HashMap<>(), context);
- Assert.assertEquals(expectedListWithFoo, out);
- }
-
-/**
-╔══════════╤═══════╤════════════╗
-║ VARIABLE │ VALUE │ EXPRESSION ║
-╠══════════╧═══════╧════════════╣
-║ (empty) ║
-╚═══════════════════════════════╝
- **/
- @Multiline
- static String expectedEmptyList;
-
- @Test
- public void testListVarsWithoutVars() {
- Context context = new Context.Builder()
- .with(Context.Capabilities.SHELL_VARIABLES, () -> new HashMap<>())
- .build();
- Object out = run("SHELL_LIST_VARS()", new HashMap<>(), context);
- Assert.assertEquals(expectedEmptyList, out);
- }
-/**
-╔════════╤═══════╗
-║ KEY │ VALUE ║
-╠════════╪═══════╣
-║ field1 │ val1 ║
-╟────────┼───────╢
-║ field2 │ val2 ║
-╚════════╧═══════╝
- **/
- @Multiline
- static String expectedMap2Table;
-
- @Test
- public void testMap2Table() {
- Map<String, Object> variables = ImmutableMap.of("map_field", ImmutableMap.of("field1", "val1", "field2", "val2"));
- Context context = Context.EMPTY_CONTEXT();
- Object out = run("SHELL_MAP2TABLE(map_field)", variables, context);
- Assert.assertEquals(expectedMap2Table, out);
- }
- /**
-╔═════╤═══════╗
-║ KEY │ VALUE ║
-╠═════╧═══════╣
-║ (empty) ║
-╚═════════════╝
- **/
- @Multiline
- static String expectedMap2TableNullInput;
-
- @Test
- public void testMap2TableNullInput() {
- Map<String,Object> variables = new HashMap<String,Object>(){{
- put("map_field",null);
- }};
- Context context = Context.EMPTY_CONTEXT();
- Object out = run("SHELL_MAP2TABLE(map_field)", variables, context);
- Assert.assertEquals(expectedMap2TableNullInput, out);
- }
-
- @Test
- public void testMap2TableInsufficientArgs() {
- Map<String, Object> variables = new HashMap<>();
- Context context = Context.EMPTY_CONTEXT();
- Object out = run("SHELL_MAP2TABLE()", variables, context);
- Assert.assertNull(out);
- }
-
- @Test
- public void testVars2Map() {
- Object out = run("SHELL_VARS2MAP('var1', 'var2')", new HashMap<>(), context);
- Assert.assertTrue(out instanceof Map);
- Map<String, String> mapOut = (Map<String, String>)out;
- //second one is null, so we don't want it there.
- Assert.assertEquals(1, mapOut.size());
- Assert.assertEquals("TO_UPPER('casey')", mapOut.get("var1"));
- }
-
- @Test
- public void testVars2MapEmpty() {
- Object out = run("SHELL_VARS2MAP()", new HashMap<>(), context);
- Map<String, String> mapOut = (Map<String, String>)out;
- Assert.assertEquals(0, mapOut.size());
- }
-
- @Test
- public void testGetExpression() {
- Object out = run("SHELL_GET_EXPRESSION('var1')", new HashMap<>(), context);
- Assert.assertTrue(out instanceof String);
- String expression = (String)out;
- //second one is null, so we don't want it there.
- Assert.assertEquals("TO_UPPER('casey')", expression);
- }
-
- @Test
- public void testGetExpressionEmpty() {
- Object out = run("SHELL_GET_EXPRESSION()", new HashMap<>(), context);
- Assert.assertNull(out );
- }
-
- @Test
- public void testEdit() throws Exception {
- System.getProperties().put("EDITOR", "/bin/cat");
- Object out = run("TO_UPPER(SHELL_EDIT(foo))", ImmutableMap.of("foo", "foo"), context);
- Assert.assertEquals("FOO", out);
- }
-
-}
http://git-wip-us.apache.org/repos/asf/metron/blob/c26abbba/metron-stellar/stellar-common/README.md
----------------------------------------------------------------------
diff --git a/metron-stellar/stellar-common/README.md b/metron-stellar/stellar-common/README.md
index a2f1242..078799e 100644
--- a/metron-stellar/stellar-common/README.md
+++ b/metron-stellar/stellar-common/README.md
@@ -243,6 +243,11 @@ Where:
| [ `SET_MERGE`](#set_merge) |
| [ `SET_REMOVE`](#set_remove) |
| [ `SIN`](#sin) |
+| [ `SHELL_EDIT`](#shell_edit) |
+| [ `SHELL_GET_EXPRESSION`](#shell_get_expression) |
+| [ `SHELL_LIST_VARS`](#shell_list_vars) |
+| [ `SHELL_MAP2TABLE`](#shell_map2table) |
+| [ `SHELL_VARS2MAP`](#shell_vars2map) |
| [ `SPLIT`](#split) |
| [ `SQRT`](#sqrt) |
| [ `STARTS_WITH`](#starts_with) |
@@ -920,6 +925,32 @@ Where:
* o - object to add to set
* Returns: A Set
+### `SHELL_EDIT`
+ * Description: Open an editor (optionally initialized with text) and return whatever is saved from the editor. The editor to use is pulled from `EDITOR` or `VISUAL` environment variable.
+ * Input:
+ * string - (Optional) A string whose content is used to initialize the editor.
+ * Returns: The content that the editor saved after editor exit.
+### `SHELL_GET_EXPRESSION`
+ * Description: Get a stellar expression from a variable
+ * Input:
+ * variable - variable name
+ * Returns: The stellar expression associated with the variable.
+### `SHELL_LIST_VARS`
+ * Description: Return the variables in a tabular form
+ * Input:
+ * wrap : Length of string to wrap the columns
+ * Returns: A tabular representation of the variables.
+### `SHELL_MAP2TABLE`
+ * Description: Take a map and return a table
+ * Input:
+ * map - Map
+ * Returns: The map in table form
+### `SHELL_VARS2MAP`
+ * Description: Take a set of variables and return a map
+ * Input:
+ * variables* - variable names to use to create map
+ * Returns: A map associating the variable name with the stellar expression.
+
### `SIN`
* Description: Returns the sine of a number.
* Input:
http://git-wip-us.apache.org/repos/asf/metron/blob/c26abbba/metron-stellar/stellar-common/pom.xml
----------------------------------------------------------------------
diff --git a/metron-stellar/stellar-common/pom.xml b/metron-stellar/stellar-common/pom.xml
index b43fcc1..6b07e68 100644
--- a/metron-stellar/stellar-common/pom.xml
+++ b/metron-stellar/stellar-common/pom.xml
@@ -203,6 +203,11 @@
<version>${global_kryo_serializers_version}</version>
</dependency>
<dependency>
+ <groupId>com.jakewharton.fliptables</groupId>
+ <artifactId>fliptables</artifactId>
+ <version>1.0.2</version>
+ </dependency>
+ <dependency>
<!-- junit dependency added with default scope to allow inclusion of StellarProcessorUtils in main jar.
It is excluded from the uber-jar. -->
<groupId>junit</groupId>
http://git-wip-us.apache.org/repos/asf/metron/blob/c26abbba/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/cli/PausableInput.java
----------------------------------------------------------------------
diff --git a/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/cli/PausableInput.java b/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/cli/PausableInput.java
index fad0115..c72d66f 100644
--- a/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/cli/PausableInput.java
+++ b/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/cli/PausableInput.java
@@ -21,6 +21,7 @@ package org.apache.metron.stellar.common.shell.cli;
import java.io.IOException;
import java.io.InputStream;
+import java.util.concurrent.atomic.AtomicBoolean;
/**
* An input stream which mirrors System.in, but allows you to 'pause' and 'unpause' it.
@@ -36,8 +37,8 @@ import java.io.InputStream;
*
*/
public class PausableInput extends InputStream {
- InputStream in = System.in;
- boolean paused = false;
+ private InputStream in = System.in;
+ private AtomicBoolean paused = new AtomicBoolean(false);
private PausableInput() {
super();
}
@@ -46,7 +47,7 @@ public class PausableInput extends InputStream {
* Stop mirroring stdin
*/
public void pause() {
- paused = true;
+ paused.set(true);
}
/**
@@ -54,8 +55,7 @@ public class PausableInput extends InputStream {
* @throws IOException
*/
public void unpause() throws IOException {
- in.read(new byte[in.available()]);
- paused = false;
+ paused.set(false);
}
public final static PausableInput INSTANCE = new PausableInput();
@@ -76,7 +76,14 @@ public class PausableInput extends InputStream {
*/
@Override
public int read() throws IOException {
-
+ if(paused.get()) {
+ try {
+ Thread.sleep(1000);
+ } catch (InterruptedException e) {
+ e.printStackTrace();
+ }
+ return 0;
+ }
return in.read();
}
@@ -116,7 +123,7 @@ public class PausableInput extends InputStream {
@Override
public int read(byte[] b) throws IOException {
- if(paused) {
+ if(paused.get()) {
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
@@ -187,7 +194,7 @@ public class PausableInput extends InputStream {
*/
@Override
public int read(byte[] b, int off, int len) throws IOException {
- if(paused) {
+ if(paused.get()) {
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
http://git-wip-us.apache.org/repos/asf/metron/blob/c26abbba/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/ShellFunctions.java
----------------------------------------------------------------------
diff --git a/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/ShellFunctions.java b/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/ShellFunctions.java
new file mode 100644
index 0000000..1df4a51
--- /dev/null
+++ b/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/ShellFunctions.java
@@ -0,0 +1,301 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import static org.apache.metron.stellar.dsl.Context.Capabilities.CONSOLE;
+
+import com.jakewharton.fliptables.FlipTable;
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang3.text.WordUtils;
+import org.apache.metron.stellar.common.shell.VariableResult;
+import org.apache.metron.stellar.common.shell.cli.PausableInput;
+import org.apache.metron.stellar.common.utils.ConversionUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Context;
+import org.apache.metron.stellar.dsl.ParseException;
+import org.apache.metron.stellar.dsl.Stellar;
+import org.apache.metron.stellar.dsl.StellarFunction;
+import org.jboss.aesh.console.Console;
+import org.slf4j.LoggerFactory;
+
+public class ShellFunctions {
+ private static final org.slf4j.Logger LOG = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @SuppressWarnings("unchecked")
+ private static Map<String, VariableResult> getVariables(Context context) {
+ return (Map<String, VariableResult>) context.getCapability(Context.Capabilities.SHELL_VARIABLES).get();
+ }
+
+ @Stellar(
+ namespace = "SHELL"
+ ,name = "MAP2TABLE"
+ ,description = "Take a map and return a table"
+ ,params = {"map - Map"
+ }
+ ,returns = "The map in table form"
+ )
+ public static class Map2Table extends BaseStellarFunction {
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public Object apply(List<Object> args) {
+ if (args.size() < 1) {
+ return null;
+ }
+ Map<Object, Object> map = (Map<Object, Object>) args.get(0);
+ if (map == null) {
+ map = new HashMap<>();
+ }
+ String[] headers = {"KEY", "VALUE"};
+ String[][] data = new String[map.size()][2];
+ int i = 0;
+ for (Map.Entry<Object, Object> kv : map.entrySet()) {
+ data[i++] = new String[]{kv.getKey().toString(), kv.getValue().toString()};
+ }
+ return FlipTable.of(headers, data);
+ }
+ }
+
+ @Stellar(
+ namespace = "SHELL"
+ ,name = "LIST_VARS"
+ ,description = "Return the variables in a tabular form"
+ ,params = {
+ "wrap : Length of string to wrap the columns"
+ }
+ ,returns = "A tabular representation of the variables."
+ )
+ public static class ListVars implements StellarFunction {
+
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+ Map<String, VariableResult> variables = getVariables(context);
+ String[] headers = {"VARIABLE", "VALUE", "EXPRESSION"};
+ String[][] data = new String[variables.size()][3];
+ int wordWrap = -1;
+ if (args.size() > 0) {
+ wordWrap = ConversionUtils.convert(args.get(0), Integer.class);
+ }
+ int i = 0;
+ for (Map.Entry<String, VariableResult> kv : variables.entrySet()) {
+ VariableResult result = kv.getValue();
+ data[i++] = new String[]{toWrappedString(kv.getKey(), wordWrap),
+ toWrappedString(result.getResult(), wordWrap),
+ toWrappedString(result.getExpression().get(), wordWrap)};
+ }
+ return FlipTable.of(headers, data);
+ }
+
+ private static String toWrappedString(Object o, int wrap) {
+ String s = "" + o;
+ if(wrap <= 0) {
+ return s;
+ }
+ return WordUtils.wrap(s, wrap);
+ }
+
+ @Override
+ public void initialize(Context context) {
+
+ }
+
+ @Override
+ public boolean isInitialized() {
+ return true;
+ }
+ }
+
+ @Stellar(
+ namespace = "SHELL"
+ ,name = "VARS2MAP"
+ ,description = "Take a set of variables and return a map"
+ ,params = {"variables* - variable names to use to create map "
+ }
+ ,returns = "A map associating the variable name with the stellar expression."
+ )
+ public static class Var2Map implements StellarFunction {
+
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+ Map<String, VariableResult> variables = getVariables(context);
+ LinkedHashMap<String, String> ret = new LinkedHashMap<>();
+ for (Object arg : args) {
+ if (arg == null) {
+ continue;
+ }
+ String variable = (String) arg;
+ VariableResult result = variables.get(variable);
+ if (result != null && result.getExpression().isPresent()) {
+ ret.put(variable, result.getExpression().orElseGet(() -> ""));
+ }
+ }
+ return ret;
+ }
+
+ @Override
+ public void initialize(Context context) {
+
+ }
+
+ @Override
+ public boolean isInitialized() {
+ return true;
+ }
+ }
+
+ @Stellar(
+ namespace = "SHELL"
+ ,name = "GET_EXPRESSION"
+ ,description = "Get a stellar expression from a variable"
+ ,params = {"variable - variable name"
+ }
+ ,returns = "The stellar expression associated with the variable."
+ )
+ public static class GetExpression implements StellarFunction {
+
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+ Map<String, VariableResult> variables = getVariables(context);
+ if (args.size() == 0) {
+ return null;
+ }
+ String variable = (String) args.get(0);
+ if (variable == null) {
+ return null;
+ }
+ VariableResult result = variables.get(variable);
+ if (result != null && result.getExpression().isPresent()) {
+ return result.getExpression().get();
+ }
+ return null;
+ }
+
+ @Override
+ public void initialize(Context context) {
+
+ }
+
+ @Override
+ public boolean isInitialized() {
+ return true;
+ }
+ }
+
+ @Stellar(
+ namespace = "SHELL"
+ ,name = "EDIT"
+ ,description = "Open an editor (optionally initialized with text) and return " +
+ "whatever is saved from the editor. The editor to use is pulled " +
+ "from `EDITOR` or `VISUAL` environment variable."
+ ,params = { "string - (Optional) A string whose content is used to initialize the editor."
+ }
+ ,returns = "The content that the editor saved after editor exit."
+ )
+ public static class Edit implements StellarFunction {
+
+ private String getEditor() {
+ // if we have editor in the system properties, it should
+ // override the env so we check that first
+ String editor = System.getProperty("EDITOR");
+ if(org.apache.commons.lang3.StringUtils.isEmpty(editor)) {
+ editor = System.getenv().get("EDITOR");
+ }
+ if(org.apache.commons.lang3.StringUtils.isEmpty(editor)) {
+ editor = System.getenv("VISUAL");
+ }
+ if(org.apache.commons.lang3.StringUtils.isEmpty(editor)) {
+ editor = "/bin/vi";
+ }
+ return editor;
+ }
+
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+ File outFile = null;
+ String editor = getEditor();
+ try {
+ outFile = File.createTempFile("stellar_shell", "out");
+ if (args.size() > 0) {
+ String arg = (String) args.get(0);
+ try (PrintWriter pw = new PrintWriter(outFile)) {
+ IOUtils.write(arg, pw);
+ }
+ }
+ } catch (IOException e) {
+ String message = "Unable to create temp file: " + e.getMessage();
+ LOG.error(message, e);
+ throw new IllegalStateException(message, e);
+ }
+ Optional<Object> console = context.getCapability(CONSOLE, false);
+ try {
+ PausableInput.INSTANCE.pause();
+ //shut down the IO for the console
+ ProcessBuilder processBuilder = new ProcessBuilder(editor, outFile.getAbsolutePath());
+ processBuilder.redirectInput(ProcessBuilder.Redirect.INHERIT);
+ processBuilder.redirectOutput(ProcessBuilder.Redirect.INHERIT);
+ processBuilder.redirectError(ProcessBuilder.Redirect.INHERIT);
+ try {
+ Process p = processBuilder.start();
+ // wait for termination.
+ p.waitFor();
+ try (BufferedReader br = new BufferedReader(new FileReader(outFile))) {
+ String ret = IOUtils.toString(br).trim();
+ return ret;
+ }
+ } catch (Exception e) {
+ String message = "Unable to read output: " + e.getMessage();
+ LOG.error(message, e);
+ return null;
+ }
+ } finally {
+ try {
+ PausableInput.INSTANCE.unpause();
+ if (console.isPresent()) {
+ ((Console) console.get()).pushToInputStream("\b\n");
+ }
+ } catch (IOException e) {
+ LOG.error("Unable to unpause: {}", e.getMessage(), e);
+ }
+ if (outFile.exists()) {
+ outFile.delete();
+ }
+ }
+ }
+
+ @Override
+ public void initialize(Context context) {
+
+ }
+
+ @Override
+ public boolean isInitialized() {
+ return true;
+ }
+ }
+}
http://git-wip-us.apache.org/repos/asf/metron/blob/c26abbba/metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/ShellFunctionsTest.java
----------------------------------------------------------------------
diff --git a/metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/ShellFunctionsTest.java b/metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/ShellFunctionsTest.java
new file mode 100644
index 0000000..354e0c3
--- /dev/null
+++ b/metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/ShellFunctionsTest.java
@@ -0,0 +1,176 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.collect.ImmutableMap;
+import org.adrianwalker.multilinestring.Multiline;
+import org.apache.metron.stellar.common.shell.VariableResult;
+import org.apache.metron.stellar.common.shell.cli.PausableInput;
+import org.apache.metron.stellar.dsl.Context;
+import org.apache.metron.stellar.dsl.Context.Capabilities;
+import org.jboss.aesh.console.Console;
+import org.jboss.aesh.console.settings.Settings;
+import org.jboss.aesh.console.settings.SettingsBuilder;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+
+import static org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
+
+public class ShellFunctionsTest {
+
+ Map<String, VariableResult> variables = ImmutableMap.of(
+ "var1" , VariableResult.withExpression("CASEY", "TO_UPPER('casey')"),
+ "var2" , VariableResult.withValue("foo"),
+ "var3" , VariableResult.withValue(null),
+ "var4" , VariableResult.withExpression(null, "blah")
+ );
+
+ Context context = new Context.Builder()
+ .with(Context.Capabilities.SHELL_VARIABLES , () -> variables).build();
+
+/**
+╔══════════╤═══════╤════════════╗
+║ VARIABLE │ VALUE │ EXPRESSION ║
+╠══════════╪═══════╪════════════╣
+║ foo │ 2.0 │ 1 + 1 ║
+╚══════════╧═══════╧════════════╝
+ **/
+ @Multiline
+ static String expectedListWithFoo;
+
+ @Test
+ public void testListVarsWithVars() {
+ Map<String, VariableResult> variables = ImmutableMap.of(
+ "foo", VariableResult.withExpression(2.0, "1 + 1"));
+
+ Context context = new Context.Builder()
+ .with(Context.Capabilities.SHELL_VARIABLES , () -> variables)
+ .build();
+ Object out = run("SHELL_LIST_VARS()", new HashMap<>(), context);
+ Assert.assertEquals(expectedListWithFoo, out);
+ }
+
+/**
+╔══════════╤═══════╤════════════╗
+║ VARIABLE │ VALUE │ EXPRESSION ║
+╠══════════╧═══════╧════════════╣
+║ (empty) ║
+╚═══════════════════════════════╝
+ **/
+ @Multiline
+ static String expectedEmptyList;
+
+ @Test
+ public void testListVarsWithoutVars() {
+ Context context = new Context.Builder()
+ .with(Context.Capabilities.SHELL_VARIABLES, () -> new HashMap<>())
+ .build();
+ Object out = run("SHELL_LIST_VARS()", new HashMap<>(), context);
+ Assert.assertEquals(expectedEmptyList, out);
+ }
+/**
+╔════════╤═══════╗
+║ KEY │ VALUE ║
+╠════════╪═══════╣
+║ field1 │ val1 ║
+╟────────┼───────╢
+║ field2 │ val2 ║
+╚════════╧═══════╝
+ **/
+ @Multiline
+ static String expectedMap2Table;
+
+ @Test
+ public void testMap2Table() {
+ Map<String, Object> variables = ImmutableMap.of("map_field", ImmutableMap.of("field1", "val1", "field2", "val2"));
+ Context context = Context.EMPTY_CONTEXT();
+ Object out = run("SHELL_MAP2TABLE(map_field)", variables, context);
+ Assert.assertEquals(expectedMap2Table, out);
+ }
+ /**
+╔═════╤═══════╗
+║ KEY │ VALUE ║
+╠═════╧═══════╣
+║ (empty) ║
+╚═════════════╝
+ **/
+ @Multiline
+ static String expectedMap2TableNullInput;
+
+ @Test
+ public void testMap2TableNullInput() {
+ Map<String,Object> variables = new HashMap<String,Object>(){{
+ put("map_field",null);
+ }};
+ Context context = Context.EMPTY_CONTEXT();
+ Object out = run("SHELL_MAP2TABLE(map_field)", variables, context);
+ Assert.assertEquals(expectedMap2TableNullInput, out);
+ }
+
+ @Test
+ public void testMap2TableInsufficientArgs() {
+ Map<String, Object> variables = new HashMap<>();
+ Context context = Context.EMPTY_CONTEXT();
+ Object out = run("SHELL_MAP2TABLE()", variables, context);
+ Assert.assertNull(out);
+ }
+
+ @Test
+ public void testVars2Map() {
+ Object out = run("SHELL_VARS2MAP('var1', 'var2')", new HashMap<>(), context);
+ Assert.assertTrue(out instanceof Map);
+ Map<String, String> mapOut = (Map<String, String>)out;
+ //second one is null, so we don't want it there.
+ Assert.assertEquals(1, mapOut.size());
+ Assert.assertEquals("TO_UPPER('casey')", mapOut.get("var1"));
+ }
+
+ @Test
+ public void testVars2MapEmpty() {
+ Object out = run("SHELL_VARS2MAP()", new HashMap<>(), context);
+ Map<String, String> mapOut = (Map<String, String>)out;
+ Assert.assertEquals(0, mapOut.size());
+ }
+
+ @Test
+ public void testGetExpression() {
+ Object out = run("SHELL_GET_EXPRESSION('var1')", new HashMap<>(), context);
+ Assert.assertTrue(out instanceof String);
+ String expression = (String)out;
+ //second one is null, so we don't want it there.
+ Assert.assertEquals("TO_UPPER('casey')", expression);
+ }
+
+ @Test
+ public void testGetExpressionEmpty() {
+ Object out = run("SHELL_GET_EXPRESSION()", new HashMap<>(), context);
+ Assert.assertNull(out );
+ }
+
+ @Test
+ public void testEdit() throws Exception {
+ System.getProperties().put("EDITOR", "/bin/cat");
+ Object out = run("TO_UPPER(SHELL_EDIT(foo))", ImmutableMap.of("foo", "foo"), context);
+ Assert.assertEquals("FOO", out);
+ }
+
+}
[5/8] metron git commit: METRON-1443 Missing Critical MPack Install
Instruction for Ubuntu (nickwallen) closes apache/metron#925
Posted by ce...@apache.org.
METRON-1443 Missing Critical MPack Install Instruction for Ubuntu (nickwallen) closes apache/metron#925
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/06305052
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/06305052
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/06305052
Branch: refs/heads/feature/METRON-1416-upgrade-solr
Commit: 06305052a186ff911d204c755804e7225d05bf28
Parents: c26abbb
Author: nickwallen <ni...@nickallen.org>
Authored: Mon Feb 5 09:17:57 2018 -0500
Committer: nickallen <ni...@apache.org>
Committed: Mon Feb 5 09:17:57 2018 -0500
----------------------------------------------------------------------
.../packaging/ambari/metron-mpack/README.md | 14 ++++++++++++++
1 file changed, 14 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/06305052/metron-deployment/packaging/ambari/metron-mpack/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/README.md b/metron-deployment/packaging/ambari/metron-mpack/README.md
index 5179b5f..4b0b073 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/README.md
+++ b/metron-deployment/packaging/ambari/metron-mpack/README.md
@@ -28,6 +28,20 @@ This allows you to easily install Metron using a simple, guided process. This a
* A [Node.js](https://nodejs.org/en/download/package-manager/) repository installed on the host running the Management and Alarm UI.
+* When installing on Ubuntu the Elasticsearch repository must be defined manually. This is NOT defined by the Mpack like it is on CentOS. This is an open bug that needs addressed in the Mpack. See the [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html) documentation for more specific instructions.
+ ```
+ $ cat >/etc/apt/sources.list.d/elasticsearch.list << EOL
+ deb https://packages.elastic.co/curator/5/debian stable main
+ deb https://artifacts.elastic.co/packages/5.x/apt stable main
+ EOL
+
+ $ wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
+
+ $ apt-get update
+ ```
+
+
+
### Quick Start
1. Build the Metron MPack. Execute the following command from the project's root directory.
[6/8] metron git commit: METRON-1442: Split rest end points for
indexing topology into random access indexing and batch indexing this closes
apache/incubator-metron#923
Posted by ce...@apache.org.
METRON-1442: Split rest end points for indexing topology into random access indexing and batch indexing this closes apache/incubator-metron#923
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/fcff0596
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/fcff0596
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/fcff0596
Branch: refs/heads/feature/METRON-1416-upgrade-solr
Commit: fcff0596c7d2b2546d89283fb90fbc8c10b31f1f
Parents: 0630505
Author: MohanDV <mo...@gmail.com>
Authored: Mon Feb 5 09:48:47 2018 -0500
Committer: cstella <ce...@gmail.com>
Committed: Mon Feb 5 09:48:47 2018 -0500
----------------------------------------------------------------------
.../src/main/config/rest_application.yml | 3 +-
.../apache/metron/rest/MetronRestConstants.java | 6 +-
.../metron/rest/controller/StormController.java | 81 +++++++++++----
.../metron/rest/service/StormAdminService.java | 4 +-
.../service/impl/StormAdminServiceImpl.java | 8 +-
.../rest/service/impl/StormCLIWrapper.java | 16 +--
.../src/main/resources/application-test.yml | 3 +-
.../src/main/resources/application-vagrant.yml | 4 +-
.../StormControllerIntegrationTest.java | 102 ++++++++++---------
.../rest/mock/MockStormCLIClientWrapper.java | 93 ++++++++++++-----
.../metron/rest/mock/MockStormRestTemplate.java | 22 ++--
.../service/impl/StormAdminServiceImplTest.java | 8 +-
.../rest/service/impl/StormCLIWrapperTest.java | 17 ++--
13 files changed, 233 insertions(+), 134 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/fcff0596/metron-interface/metron-rest/src/main/config/rest_application.yml
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/main/config/rest_application.yml b/metron-interface/metron-rest/src/main/config/rest_application.yml
index 6e4fb66..4cc51ff 100644
--- a/metron-interface/metron-rest/src/main/config/rest_application.yml
+++ b/metron-interface/metron-rest/src/main/config/rest_application.yml
@@ -46,7 +46,8 @@ storm:
enrichment:
script.path: ${METRON_HOME}/bin/start_enrichment_topology.sh
indexing:
- script.path: ${METRON_HOME}/bin/start_elasticsearch_topology.sh
+ randomaccess.script.path: ${METRON_HOME}/bin/start_elasticsearch_topology.sh
+ batch.script.path: ${METRON_HOME}/bin/start_hdfs_topology.sh
kerberos:
enabled: ${SECURITY_ENABLED}
http://git-wip-us.apache.org/repos/asf/metron/blob/fcff0596/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/MetronRestConstants.java
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/MetronRestConstants.java b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/MetronRestConstants.java
index 4e8d7f2..f18d4cf 100644
--- a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/MetronRestConstants.java
+++ b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/MetronRestConstants.java
@@ -35,10 +35,12 @@ public class MetronRestConstants {
public static final String TOPOLOGY_SUMMARY_URL = "/api/v1/topology/summary";
public static final String TOPOLOGY_URL = "/api/v1/topology";
public static final String ENRICHMENT_TOPOLOGY_NAME = "enrichment";
- public static final String INDEXING_TOPOLOGY_NAME = "indexing";
+ public static final String BATCH_INDEXING_TOPOLOGY_NAME = "batch_indexing";
+ public static final String RANDOM_ACCESS_INDEXING_TOPOLOGY_NAME = "random_access_indexing";
public static final String PARSER_SCRIPT_PATH_SPRING_PROPERTY = "storm.parser.script.path";
public static final String ENRICHMENT_SCRIPT_PATH_SPRING_PROPERTY = "storm.enrichment.script.path";
- public static final String INDEXING_SCRIPT_PATH_SPRING_PROPERTY = "storm.indexing.script.path";
+ public static final String BATCH_INDEXING_SCRIPT_PATH_SPRING_PROPERTY = "storm.indexing.batch.script.path";
+ public static final String RANDOM_ACCESS_INDEXING_SCRIPT_PATH_SPRING_PROPERTY = "storm.indexing.randomaccess.script.path";
public static final String PARSER_TOPOLOGY_OPTIONS_SPRING_PROPERTY = "storm.parser.topology.options";
public static final String KAFKA_SECURITY_PROTOCOL_SPRING_PROPERTY = "kafka.security.protocol";
http://git-wip-us.apache.org/repos/asf/metron/blob/fcff0596/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/StormController.java
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/StormController.java b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/StormController.java
index 292c668..d1af1c5 100644
--- a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/StormController.java
+++ b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/StormController.java
@@ -147,12 +147,12 @@ public class StormController {
return new ResponseEntity<>(stormStatusService.deactivateTopology(MetronRestConstants.ENRICHMENT_TOPOLOGY_NAME), HttpStatus.OK);
}
- @ApiOperation(value = "Retrieves the status of the Storm indexing topology")
+ @ApiOperation(value = "Retrieves the status of the Storm random access indexing topology")
@ApiResponses(value = { @ApiResponse(message = "Returns topology status information", code = 200),
@ApiResponse(message = "Topology is missing", code = 404) })
- @RequestMapping(value = "/indexing", method = RequestMethod.GET)
- ResponseEntity<TopologyStatus> getIndexing() throws RestException {
- TopologyStatus topologyStatus = stormStatusService.getTopologyStatus(MetronRestConstants.INDEXING_TOPOLOGY_NAME);
+ @RequestMapping(value = "/indexing/randomaccess", method = RequestMethod.GET)
+ ResponseEntity<TopologyStatus> getRandomAccessIndexing() throws RestException {
+ TopologyStatus topologyStatus = stormStatusService.getTopologyStatus(MetronRestConstants.RANDOM_ACCESS_INDEXING_TOPOLOGY_NAME);
if (topologyStatus != null) {
return new ResponseEntity<>(topologyStatus, HttpStatus.OK);
} else {
@@ -160,32 +160,32 @@ public class StormController {
}
}
- @ApiOperation(value = "Starts a Storm indexing topology")
+ @ApiOperation(value = "Starts a Storm random access indexing topology")
@ApiResponse(message = "Returns start response message", code = 200)
- @RequestMapping(value = "/indexing/start", method = RequestMethod.GET)
- ResponseEntity<TopologyResponse> startIndexing() throws RestException {
- return new ResponseEntity<>(stormAdminService.startIndexingTopology(), HttpStatus.OK);
+ @RequestMapping(value = "/indexing/randomaccess/start", method = RequestMethod.GET)
+ ResponseEntity<TopologyResponse> startRandomAccessIndexing() throws RestException {
+ return new ResponseEntity<>(stormAdminService.startIndexingTopology(MetronRestConstants.RANDOM_ACCESS_INDEXING_SCRIPT_PATH_SPRING_PROPERTY), HttpStatus.OK);
}
- @ApiOperation(value = "Stops a Storm enrichment topology")
+ @ApiOperation(value = "Stops a Storm random access indexing topology")
@ApiResponse(message = "Returns stop response message", code = 200)
- @RequestMapping(value = "/indexing/stop", method = RequestMethod.GET)
- ResponseEntity<TopologyResponse> stopIndexing(@ApiParam(name="stopNow", value="Stop the topology immediately")@RequestParam(required = false, defaultValue = "false") boolean stopNow) throws RestException {
- return new ResponseEntity<>(stormAdminService.stopIndexingTopology(stopNow), HttpStatus.OK);
+ @RequestMapping(value = "/indexing/randomaccess/stop", method = RequestMethod.GET)
+ ResponseEntity<TopologyResponse> stopRandomAccessIndexing(@ApiParam(name="stopNow", value="Stop the topology immediately")@RequestParam(required = false, defaultValue = "false") boolean stopNow) throws RestException {
+ return new ResponseEntity<>(stormAdminService.stopIndexingTopology(MetronRestConstants.RANDOM_ACCESS_INDEXING_TOPOLOGY_NAME ,stopNow), HttpStatus.OK);
}
- @ApiOperation(value = "Activates a Storm indexing topology")
+ @ApiOperation(value = "Activates a Storm random access indexing topology")
@ApiResponse(message = "Returns activate response message", code = 200)
- @RequestMapping(value = "/indexing/activate", method = RequestMethod.GET)
- ResponseEntity<TopologyResponse> activateIndexing() throws RestException {
- return new ResponseEntity<>(stormStatusService.activateTopology(MetronRestConstants.INDEXING_TOPOLOGY_NAME), HttpStatus.OK);
+ @RequestMapping(value = "/indexing/randomaccess/activate", method = RequestMethod.GET)
+ ResponseEntity<TopologyResponse> activateRandomAccessIndexing() throws RestException {
+ return new ResponseEntity<>(stormStatusService.activateTopology(MetronRestConstants.RANDOM_ACCESS_INDEXING_TOPOLOGY_NAME), HttpStatus.OK);
}
- @ApiOperation(value = "Deactivates a Storm indexing topology")
+ @ApiOperation(value = "Deactivates a Storm random access indexing topology")
@ApiResponse(message = "Returns deactivate response message", code = 200)
- @RequestMapping(value = "/indexing/deactivate", method = RequestMethod.GET)
- ResponseEntity<TopologyResponse> deactivateIndexing() throws RestException {
- return new ResponseEntity<>(stormStatusService.deactivateTopology(MetronRestConstants.INDEXING_TOPOLOGY_NAME), HttpStatus.OK);
+ @RequestMapping(value = "/indexing/randomaccess/deactivate", method = RequestMethod.GET)
+ ResponseEntity<TopologyResponse> deactivateRandomAccessIndexing() throws RestException {
+ return new ResponseEntity<>(stormStatusService.deactivateTopology(MetronRestConstants.RANDOM_ACCESS_INDEXING_TOPOLOGY_NAME), HttpStatus.OK);
}
@ApiOperation(value = "Retrieves information about the Storm command line client")
@@ -195,4 +195,45 @@ public class StormController {
return new ResponseEntity<>(stormAdminService.getStormClientStatus(), HttpStatus.OK);
}
+ @ApiOperation(value = "Retrieves the status of the Storm batch indexing topology")
+ @ApiResponses(value = { @ApiResponse(message = "Returns topology status information", code = 200),
+ @ApiResponse(message = "Topology is missing", code = 404) })
+ @RequestMapping(value = "/indexing/batch", method = RequestMethod.GET)
+ ResponseEntity<TopologyStatus> getBatchIndexing() throws RestException {
+ TopologyStatus topologyStatus = stormStatusService.getTopologyStatus(MetronRestConstants.BATCH_INDEXING_TOPOLOGY_NAME);
+ if (topologyStatus != null) {
+ return new ResponseEntity<>(topologyStatus, HttpStatus.OK);
+ } else {
+ return new ResponseEntity<>(HttpStatus.NOT_FOUND);
+ }
+ }
+
+ @ApiOperation(value = "Starts a Storm batch indexing topology")
+ @ApiResponse(message = "Returns start response message", code = 200)
+ @RequestMapping(value = "/indexing/batch/start", method = RequestMethod.GET)
+ ResponseEntity<TopologyResponse> startBatchIndexing() throws RestException {
+ return new ResponseEntity<>(stormAdminService.startIndexingTopology(MetronRestConstants.BATCH_INDEXING_SCRIPT_PATH_SPRING_PROPERTY), HttpStatus.OK);
+ }
+
+ @ApiOperation(value = "Stops a Storm batch indexing topology")
+ @ApiResponse(message = "Returns stop response message", code = 200)
+ @RequestMapping(value = "/indexing/batch/stop", method = RequestMethod.GET)
+ ResponseEntity<TopologyResponse> stopBatchIndexing(@ApiParam(name="stopNow", value="Stop the topology immediately")@RequestParam(required = false, defaultValue = "false") boolean stopNow) throws RestException {
+ return new ResponseEntity<>(stormAdminService.stopIndexingTopology(MetronRestConstants.BATCH_INDEXING_TOPOLOGY_NAME, stopNow), HttpStatus.OK);
+ }
+
+ @ApiOperation(value = "Activates a Storm batch indexing topology")
+ @ApiResponse(message = "Returns activate response message", code = 200)
+ @RequestMapping(value = "/indexing/batch/activate", method = RequestMethod.GET)
+ ResponseEntity<TopologyResponse> activateBatchIndexing() throws RestException {
+ return new ResponseEntity<>(stormStatusService.activateTopology(MetronRestConstants.BATCH_INDEXING_TOPOLOGY_NAME), HttpStatus.OK);
+ }
+
+ @ApiOperation(value = "Deactivates a Storm batch indexing topology")
+ @ApiResponse(message = "Returns deactivate response message", code = 200)
+ @RequestMapping(value = "/indexing/batch/deactivate", method = RequestMethod.GET)
+ ResponseEntity<TopologyResponse> deactivateBatchIndexing() throws RestException {
+ return new ResponseEntity<>(stormStatusService.deactivateTopology(MetronRestConstants.BATCH_INDEXING_TOPOLOGY_NAME), HttpStatus.OK);
+ }
+
}
http://git-wip-us.apache.org/repos/asf/metron/blob/fcff0596/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/StormAdminService.java
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/StormAdminService.java b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/StormAdminService.java
index 8c1e228..3f6f8ff 100644
--- a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/StormAdminService.java
+++ b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/StormAdminService.java
@@ -32,9 +32,9 @@ public interface StormAdminService {
TopologyResponse stopEnrichmentTopology(boolean stopNow) throws RestException;
- TopologyResponse startIndexingTopology() throws RestException;
+ TopologyResponse startIndexingTopology(String scriptPath) throws RestException;
- TopologyResponse stopIndexingTopology(boolean stopNow) throws RestException;
+ TopologyResponse stopIndexingTopology(String name, boolean stopNow) throws RestException;
Map<String, String> getStormClientStatus() throws RestException;
}
http://git-wip-us.apache.org/repos/asf/metron/blob/fcff0596/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormAdminServiceImpl.java
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormAdminServiceImpl.java b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormAdminServiceImpl.java
index 9bd368f..40b01f1 100644
--- a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormAdminServiceImpl.java
+++ b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormAdminServiceImpl.java
@@ -74,13 +74,13 @@ public class StormAdminServiceImpl implements StormAdminService {
}
@Override
- public TopologyResponse startIndexingTopology() throws RestException {
- return createResponse(stormCLIClientWrapper.startIndexingTopology(), TopologyStatusCode.STARTED, TopologyStatusCode.START_ERROR);
+ public TopologyResponse startIndexingTopology(String scriptPath) throws RestException {
+ return createResponse(stormCLIClientWrapper.startIndexingTopology(scriptPath), TopologyStatusCode.STARTED, TopologyStatusCode.START_ERROR);
}
@Override
- public TopologyResponse stopIndexingTopology(boolean stopNow) throws RestException {
- return createResponse(stormCLIClientWrapper.stopIndexingTopology(stopNow), TopologyStatusCode.STOPPED, TopologyStatusCode.STOP_ERROR);
+ public TopologyResponse stopIndexingTopology(String name, boolean stopNow) throws RestException {
+ return createResponse(stormCLIClientWrapper.stopIndexingTopology(name, stopNow), TopologyStatusCode.STOPPED, TopologyStatusCode.STOP_ERROR);
}
private TopologyResponse createResponse(int responseCode, TopologyStatusCode successMessage, TopologyStatusCode errorMessage) {
http://git-wip-us.apache.org/repos/asf/metron/blob/fcff0596/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java
index fff7390..26049dd 100644
--- a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java
+++ b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java
@@ -37,7 +37,6 @@ import java.util.Map;
import static java.util.stream.Collectors.toList;
import static org.apache.metron.rest.MetronRestConstants.ENRICHMENT_TOPOLOGY_NAME;
-import static org.apache.metron.rest.MetronRestConstants.INDEXING_TOPOLOGY_NAME;
public class StormCLIWrapper {
@@ -70,14 +69,14 @@ public class StormCLIWrapper {
return runCommand(getStopCommand(ENRICHMENT_TOPOLOGY_NAME, stopNow));
}
- public int startIndexingTopology() throws RestException {
+ public int startIndexingTopology(String scriptPath) throws RestException {
kinit();
- return runCommand(getIndexingStartCommand());
+ return runCommand(getIndexingStartCommand(scriptPath));
}
- public int stopIndexingTopology(boolean stopNow) throws RestException {
+ public int stopIndexingTopology(String name, boolean stopNow) throws RestException {
kinit();
- return runCommand(getStopCommand(INDEXING_TOPOLOGY_NAME, stopNow));
+ return runCommand(getStopCommand(name, stopNow));
}
protected int runCommand(String[] command) throws RestException {
@@ -137,9 +136,9 @@ public class StormCLIWrapper {
return command;
}
- protected String[] getIndexingStartCommand() {
+ protected String[] getIndexingStartCommand(String scriptPath) {
String[] command = new String[1];
- command[0] = environment.getProperty(MetronRestConstants.INDEXING_SCRIPT_PATH_SPRING_PROPERTY);
+ command[0] = environment.getProperty(scriptPath);
return command;
}
@@ -166,7 +165,8 @@ public class StormCLIWrapper {
Map<String, String> status = new HashMap<>();
status.put("parserScriptPath", environment.getProperty(MetronRestConstants.PARSER_SCRIPT_PATH_SPRING_PROPERTY));
status.put("enrichmentScriptPath", environment.getProperty(MetronRestConstants.ENRICHMENT_SCRIPT_PATH_SPRING_PROPERTY));
- status.put("indexingScriptPath", environment.getProperty(MetronRestConstants.INDEXING_SCRIPT_PATH_SPRING_PROPERTY));
+ status.put("randomAccessIndexingScriptPath", environment.getProperty(MetronRestConstants.RANDOM_ACCESS_INDEXING_SCRIPT_PATH_SPRING_PROPERTY));
+ status.put("batchIndexingScriptPath", environment.getProperty(MetronRestConstants.BATCH_INDEXING_SCRIPT_PATH_SPRING_PROPERTY));
status.put("stormClientVersionInstalled", stormClientVersionInstalled());
return status;
}
http://git-wip-us.apache.org/repos/asf/metron/blob/fcff0596/metron-interface/metron-rest/src/main/resources/application-test.yml
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/main/resources/application-test.yml b/metron-interface/metron-rest/src/main/resources/application-test.yml
index 749dec4..3cca5e0 100644
--- a/metron-interface/metron-rest/src/main/resources/application-test.yml
+++ b/metron-interface/metron-rest/src/main/resources/application-test.yml
@@ -38,7 +38,8 @@ storm:
enrichment:
script.path: /usr/metron/${metron.version}/bin/start_enrichment_topology.sh
indexing:
- script.path: /usr/metron/${metron.version}/bin/start_elasticsearch_topology.sh
+ randomaccess.script.path: /usr/metron/${metron.version}/bin/start_elasticsearch_topology.sh
+ batch.script.path: /usr/metron/${metron.version}/bin/start_hdfs_topology.sh
search:
max:
http://git-wip-us.apache.org/repos/asf/metron/blob/fcff0596/metron-interface/metron-rest/src/main/resources/application-vagrant.yml
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/main/resources/application-vagrant.yml b/metron-interface/metron-rest/src/main/resources/application-vagrant.yml
index cf2c170..3eea24a 100644
--- a/metron-interface/metron-rest/src/main/resources/application-vagrant.yml
+++ b/metron-interface/metron-rest/src/main/resources/application-vagrant.yml
@@ -49,7 +49,9 @@ storm:
enrichment:
script.path: /usr/metron/${metron.version}/bin/start_enrichment_topology.sh
indexing:
- script.path: /usr/metron/${metron.version}/bin/start_elasticsearch_topology.sh
+ randomaccess.script.path: /usr/metron/${metron.version}/bin/start_elasticsearch_topology.sh
+ batch.script.path: /usr/metron/${metron.version}/bin/start_hdfs_topology.sh
+
kerberos:
enabled: false
http://git-wip-us.apache.org/repos/asf/metron/blob/fcff0596/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/controller/StormControllerIntegrationTest.java
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/controller/StormControllerIntegrationTest.java b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/controller/StormControllerIntegrationTest.java
index 9a6022c..3986413 100644
--- a/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/controller/StormControllerIntegrationTest.java
+++ b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/controller/StormControllerIntegrationTest.java
@@ -17,8 +17,10 @@
*/
package org.apache.metron.rest.controller;
+import org.apache.hadoop.hbase.shaded.com.google.common.collect.ImmutableList;
import org.apache.metron.common.configuration.SensorParserConfig;
import org.apache.metron.integration.utils.TestUtils;
+import org.apache.metron.rest.MetronRestConstants;
import org.apache.metron.rest.model.TopologyStatusCode;
import org.apache.metron.rest.service.GlobalConfigService;
import org.apache.metron.rest.service.SensorParserConfigService;
@@ -33,6 +35,7 @@ import org.springframework.http.MediaType;
import org.springframework.test.context.ActiveProfiles;
import org.springframework.test.context.junit4.SpringRunner;
import org.springframework.test.web.servlet.MockMvc;
+import org.springframework.test.web.servlet.ResultActions;
import org.springframework.test.web.servlet.setup.MockMvcBuilders;
import org.springframework.web.context.WebApplicationContext;
@@ -287,66 +290,69 @@ public class StormControllerIntegrationTest {
.andExpect(jsonPath("$.status").value("SUCCESS"))
.andExpect(jsonPath("$.message").value(TopologyStatusCode.STOPPED.name()));
- this.mockMvc.perform(get(stormUrl + "/indexing").with(httpBasic(user,password)))
- .andExpect(status().isNotFound());
-
- this.mockMvc.perform(get(stormUrl + "/indexing/activate").with(httpBasic(user,password)))
- .andExpect(status().isOk())
- .andExpect(jsonPath("$.status").value("ERROR"))
- .andExpect(jsonPath("$.message").value(TopologyStatusCode.TOPOLOGY_NOT_FOUND.name()));
-
- this.mockMvc.perform(get(stormUrl + "/indexing/deactivate").with(httpBasic(user,password)))
- .andExpect(status().isOk())
- .andExpect(jsonPath("$.status").value("ERROR"))
- .andExpect(jsonPath("$.message").value(TopologyStatusCode.TOPOLOGY_NOT_FOUND.name()));
-
- this.mockMvc.perform(get(stormUrl + "/indexing/stop?stopNow=true").with(httpBasic(user,password)))
- .andExpect(status().isOk())
- .andExpect(jsonPath("$.status").value("ERROR"))
- .andExpect(jsonPath("$.message").value(TopologyStatusCode.STOP_ERROR.toString()));
-
- this.mockMvc.perform(get(stormUrl + "/indexing/start").with(httpBasic(user,password)))
- .andExpect(status().isOk())
- .andExpect(jsonPath("$.status").value("SUCCESS"))
- .andExpect(jsonPath("$.message").value(TopologyStatusCode.STARTED.toString()));
-
- this.mockMvc.perform(get(stormUrl + "/indexing/deactivate").with(httpBasic(user,password)))
- .andExpect(status().isOk())
- .andExpect(jsonPath("$.status").value("SUCCESS"))
- .andExpect(jsonPath("$.message").value(TopologyStatusCode.INACTIVE.name()));
-
- this.mockMvc.perform(get(stormUrl + "/indexing/activate").with(httpBasic(user,password)))
- .andExpect(status().isOk())
- .andExpect(jsonPath("$.status").value("SUCCESS"))
- .andExpect(jsonPath("$.message").value(TopologyStatusCode.ACTIVE.name()));
-
- this.mockMvc.perform(get(stormUrl + "/indexing").with(httpBasic(user,password)))
+ for(String type : ImmutableList.of("randomaccess", "batch")) {
+ this.mockMvc.perform(get(stormUrl + "/indexing/" + type).with(httpBasic(user,password)))
+ .andExpect(status().isNotFound());
+ this.mockMvc.perform(get(stormUrl + "/indexing/" + type + "/activate").with(httpBasic(user, password)))
+ .andExpect(status().isOk())
+ .andExpect(jsonPath("$.status").value("ERROR"))
+ .andExpect(jsonPath("$.message").value(TopologyStatusCode.TOPOLOGY_NOT_FOUND.name()));
+
+ this.mockMvc.perform(get(stormUrl + "/indexing/" + type + "/deactivate").with(httpBasic(user, password)))
+ .andExpect(status().isOk())
+ .andExpect(jsonPath("$.status").value("ERROR"))
+ .andExpect(jsonPath("$.message").value(TopologyStatusCode.TOPOLOGY_NOT_FOUND.name()));
+
+ this.mockMvc.perform(get(stormUrl + "/indexing/" + type + "/stop?stopNow=true").with(httpBasic(user, password)))
+ .andExpect(status().isOk())
+ .andExpect(jsonPath("$.status").value("ERROR"))
+ .andExpect(jsonPath("$.message").value(TopologyStatusCode.STOP_ERROR.toString()));
+
+ this.mockMvc.perform(get(stormUrl + "/indexing/" + type + "/start").with(httpBasic(user, password)))
+ .andExpect(status().isOk())
+ .andExpect(jsonPath("$.status").value("SUCCESS"))
+ .andExpect(jsonPath("$.message").value(TopologyStatusCode.STARTED.toString()));
+
+ ResultActions actions = this.mockMvc.perform(get(stormUrl + "/indexing/" + type + "/deactivate").with(httpBasic(user, password)));
+ actions.andExpect(status().isOk())
+ .andExpect(jsonPath("$.status").value("SUCCESS"))
+ .andExpect(jsonPath("$.message").value(TopologyStatusCode.INACTIVE.name()));
+
+ this.mockMvc.perform(get(stormUrl + "/indexing/" + type + "/activate").with(httpBasic(user, password)))
+ .andExpect(status().isOk())
+ .andExpect(jsonPath("$.status").value("SUCCESS"))
+ .andExpect(jsonPath("$.message").value(TopologyStatusCode.ACTIVE.name()));
+ String topologyName = type.equals("randomaccess")? MetronRestConstants.RANDOM_ACCESS_INDEXING_TOPOLOGY_NAME:MetronRestConstants.BATCH_INDEXING_TOPOLOGY_NAME;
+ this.mockMvc.perform(get(stormUrl + "/indexing/" + type).with(httpBasic(user, password)))
+ .andExpect(status().isOk())
+ .andExpect(content().contentType(MediaType.parseMediaType("application/json;charset=UTF-8")))
+ .andExpect(jsonPath("$.name").value(topologyName))
+ .andExpect(jsonPath("$.id", containsString("indexing")))
+ .andExpect(jsonPath("$.status").value("ACTIVE"))
+ .andExpect(jsonPath("$.latency").exists())
+ .andExpect(jsonPath("$.throughput").exists())
+ .andExpect(jsonPath("$.emitted").exists())
+ .andExpect(jsonPath("$.acked").exists());
+ this.mockMvc.perform(get(stormUrl).with(httpBasic(user,password)))
.andExpect(status().isOk())
.andExpect(content().contentType(MediaType.parseMediaType("application/json;charset=UTF-8")))
- .andExpect(jsonPath("$.name").value("indexing"))
- .andExpect(jsonPath("$.id", containsString("indexing")))
- .andExpect(jsonPath("$.status").value("ACTIVE"))
- .andExpect(jsonPath("$.latency").exists())
- .andExpect(jsonPath("$.throughput").exists())
- .andExpect(jsonPath("$.emitted").exists())
- .andExpect(jsonPath("$.acked").exists());
+ .andExpect(jsonPath("$[?(@.name == '" + topologyName + "' && @.status == 'ACTIVE')]").exists());
- this.mockMvc.perform(get(stormUrl).with(httpBasic(user,password)))
- .andExpect(status().isOk())
- .andExpect(content().contentType(MediaType.parseMediaType("application/json;charset=UTF-8")))
- .andExpect(jsonPath("$[?(@.name == 'indexing' && @.status == 'ACTIVE')]").exists());
-
- this.mockMvc.perform(get(stormUrl + "/indexing/stop").with(httpBasic(user,password)))
+ this.mockMvc.perform(get(stormUrl + "/indexing/" + type + "/stop").with(httpBasic(user,password)))
.andExpect(status().isOk())
.andExpect(jsonPath("$.status").value("SUCCESS"))
.andExpect(jsonPath("$.message").value(TopologyStatusCode.STOPPED.name()));
+ }
+
+
this.mockMvc.perform(get(stormUrl + "/client/status").with(httpBasic(user,password)))
.andExpect(status().isOk())
.andExpect(jsonPath("$.stormClientVersionInstalled").value("1.0.1"))
.andExpect(jsonPath("$.parserScriptPath").value("/usr/metron/" + metronVersion + "/bin/start_parser_topology.sh"))
.andExpect(jsonPath("$.enrichmentScriptPath").value("/usr/metron/" + metronVersion + "/bin/start_enrichment_topology.sh"))
- .andExpect(jsonPath("$.indexingScriptPath").value("/usr/metron/" + metronVersion + "/bin/start_elasticsearch_topology.sh"));
+ .andExpect(jsonPath("$.randomAccessIndexingScriptPath").value("/usr/metron/" + metronVersion + "/bin/start_elasticsearch_topology.sh"))
+ .andExpect(jsonPath("$.batchIndexingScriptPath").value("/usr/metron/" + metronVersion + "/bin/start_hdfs_topology.sh"));
globalConfigService.delete();
sensorParserConfigService.delete("broTest");
http://git-wip-us.apache.org/repos/asf/metron/blob/fcff0596/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/mock/MockStormCLIClientWrapper.java
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/mock/MockStormCLIClientWrapper.java b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/mock/MockStormCLIClientWrapper.java
index dd21095..9018935 100644
--- a/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/mock/MockStormCLIClientWrapper.java
+++ b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/mock/MockStormCLIClientWrapper.java
@@ -17,6 +17,7 @@
*/
package org.apache.metron.rest.mock;
+import org.apache.metron.rest.MetronRestConstants;
import org.apache.metron.rest.RestException;
import org.apache.metron.rest.model.TopologyStatusCode;
import org.apache.metron.rest.service.impl.StormCLIWrapper;
@@ -29,7 +30,8 @@ public class MockStormCLIClientWrapper extends StormCLIWrapper {
private final Map<String, TopologyStatusCode> parsersStatus = new HashMap<>();
private TopologyStatusCode enrichmentStatus = TopologyStatusCode.TOPOLOGY_NOT_FOUND;
- private TopologyStatusCode indexingStatus = TopologyStatusCode.TOPOLOGY_NOT_FOUND;
+ private TopologyStatusCode randomAccessIndexingStatus = TopologyStatusCode.TOPOLOGY_NOT_FOUND;
+ private TopologyStatusCode batchIndexingStatus = TopologyStatusCode.TOPOLOGY_NOT_FOUND;
public Set<String> getParserTopologyNames() {
return parsersStatus.keySet();
@@ -128,45 +130,84 @@ public class MockStormCLIClientWrapper extends StormCLIWrapper {
}
}
- public TopologyStatusCode getIndexingStatus() {
- return indexingStatus;
+ public TopologyStatusCode getIndexingStatus(String name) {
+ return name.equals(MetronRestConstants.BATCH_INDEXING_TOPOLOGY_NAME)?batchIndexingStatus:randomAccessIndexingStatus;
}
@Override
- public int startIndexingTopology() throws RestException {
- if (indexingStatus == TopologyStatusCode.TOPOLOGY_NOT_FOUND) {
- indexingStatus = TopologyStatusCode.ACTIVE;
- return 0;
- } else {
- return 1;
+ public int startIndexingTopology(String scriptPath) throws RestException {
+ if(scriptPath.equals(MetronRestConstants.BATCH_INDEXING_SCRIPT_PATH_SPRING_PROPERTY)) {
+ if (batchIndexingStatus == TopologyStatusCode.TOPOLOGY_NOT_FOUND) {
+ batchIndexingStatus = TopologyStatusCode.ACTIVE;
+ return 0;
+ } else {
+ return 1;
+ }
+ }
+ else {
+ if (randomAccessIndexingStatus == TopologyStatusCode.TOPOLOGY_NOT_FOUND) {
+ randomAccessIndexingStatus = TopologyStatusCode.ACTIVE;
+ return 0;
+ } else {
+ return 1;
+ }
}
}
@Override
- public int stopIndexingTopology(boolean stopNow) throws RestException {
- if (indexingStatus == TopologyStatusCode.ACTIVE) {
- indexingStatus = TopologyStatusCode.TOPOLOGY_NOT_FOUND;
- return 0;
- } else {
- return 1;
+ public int stopIndexingTopology(String name, boolean stopNow) throws RestException {
+ if(name.equals(MetronRestConstants.BATCH_INDEXING_TOPOLOGY_NAME)) {
+ if (batchIndexingStatus == TopologyStatusCode.ACTIVE) {
+ batchIndexingStatus = TopologyStatusCode.TOPOLOGY_NOT_FOUND;
+ return 0;
+ } else {
+ return 1;
+ }
+ }
+ else {
+ if (randomAccessIndexingStatus == TopologyStatusCode.ACTIVE) {
+ randomAccessIndexingStatus = TopologyStatusCode.TOPOLOGY_NOT_FOUND;
+ return 0;
+ } else {
+ return 1;
+ }
}
}
- public int activateIndexingTopology() {
- if (indexingStatus == TopologyStatusCode.INACTIVE || indexingStatus == TopologyStatusCode.ACTIVE) {
- indexingStatus = TopologyStatusCode.ACTIVE;
- return 0;
- } else {
- return 1;
+ public int activateIndexingTopology(String name) {
+ if(name.equals(MetronRestConstants.BATCH_INDEXING_TOPOLOGY_NAME)) {
+ if (batchIndexingStatus == TopologyStatusCode.INACTIVE || batchIndexingStatus == TopologyStatusCode.ACTIVE) {
+ batchIndexingStatus = TopologyStatusCode.ACTIVE;
+ return 0;
+ } else {
+ return 1;
+ }
+ }
+ else {
+ if (randomAccessIndexingStatus == TopologyStatusCode.INACTIVE || randomAccessIndexingStatus == TopologyStatusCode.ACTIVE) {
+ randomAccessIndexingStatus = TopologyStatusCode.ACTIVE;
+ return 0;
+ } else {
+ return 1;
+ }
}
}
- public int deactivateIndexingTopology() {
- if (indexingStatus == TopologyStatusCode.INACTIVE || indexingStatus == TopologyStatusCode.ACTIVE) {
- indexingStatus = TopologyStatusCode.INACTIVE;
- return 0;
+ public int deactivateIndexingTopology(String name) {
+ if (name.equals(MetronRestConstants.BATCH_INDEXING_TOPOLOGY_NAME)) {
+ if (batchIndexingStatus == TopologyStatusCode.INACTIVE || batchIndexingStatus == TopologyStatusCode.ACTIVE) {
+ batchIndexingStatus = TopologyStatusCode.INACTIVE;
+ return 0;
+ } else {
+ return 1;
+ }
} else {
- return 1;
+ if (randomAccessIndexingStatus == TopologyStatusCode.INACTIVE || randomAccessIndexingStatus == TopologyStatusCode.ACTIVE) {
+ randomAccessIndexingStatus = TopologyStatusCode.INACTIVE;
+ return 0;
+ } else {
+ return 1;
+ }
}
}
http://git-wip-us.apache.org/repos/asf/metron/blob/fcff0596/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/mock/MockStormRestTemplate.java
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/mock/MockStormRestTemplate.java b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/mock/MockStormRestTemplate.java
index ccf993d..ef47ac9 100644
--- a/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/mock/MockStormRestTemplate.java
+++ b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/mock/MockStormRestTemplate.java
@@ -53,9 +53,13 @@ public class MockStormRestTemplate extends RestTemplate {
if (enrichmentStatus != TopologyStatusCode.TOPOLOGY_NOT_FOUND) {
topologyStatusList.add(getTopologyStatus("enrichment"));
}
- TopologyStatusCode indexingStatus = mockStormCLIClientWrapper.getIndexingStatus();
- if (indexingStatus != TopologyStatusCode.TOPOLOGY_NOT_FOUND) {
- topologyStatusList.add(getTopologyStatus("indexing"));
+ TopologyStatusCode batchIndexingStatus = mockStormCLIClientWrapper.getIndexingStatus(MetronRestConstants.BATCH_INDEXING_TOPOLOGY_NAME);
+ if (batchIndexingStatus != TopologyStatusCode.TOPOLOGY_NOT_FOUND) {
+ topologyStatusList.add(getTopologyStatus(MetronRestConstants.BATCH_INDEXING_TOPOLOGY_NAME));
+ }
+ TopologyStatusCode randomIndexingStatus = mockStormCLIClientWrapper.getIndexingStatus(MetronRestConstants.RANDOM_ACCESS_INDEXING_TOPOLOGY_NAME);
+ if (randomIndexingStatus != TopologyStatusCode.TOPOLOGY_NOT_FOUND) {
+ topologyStatusList.add(getTopologyStatus(MetronRestConstants.RANDOM_ACCESS_INDEXING_TOPOLOGY_NAME));
}
topologySummary.setTopologies(topologyStatusList.toArray(new TopologyStatus[topologyStatusList.size()]));
response = topologySummary;
@@ -79,8 +83,8 @@ public class MockStormRestTemplate extends RestTemplate {
topologyStatus.setId(name + "-id");
if ("enrichment".equals(name)) {
topologyStatus.setStatus(mockStormCLIClientWrapper.getEnrichmentStatus());
- } else if ("indexing".equals(name)) {
- topologyStatus.setStatus(mockStormCLIClientWrapper.getIndexingStatus());
+ } else if (name.contains("indexing")) {
+ topologyStatus.setStatus(mockStormCLIClientWrapper.getIndexingStatus(name));
} else {
topologyStatus.setStatus(mockStormCLIClientWrapper.getParserStatus(name));
}
@@ -97,16 +101,16 @@ public class MockStormRestTemplate extends RestTemplate {
if (action.equals("activate")) {
if (name.equals("enrichment")) {
returnCode = mockStormCLIClientWrapper.activateEnrichmentTopology();
- } else if (name.equals("indexing")) {
- returnCode = mockStormCLIClientWrapper.activateIndexingTopology();
+ } else if (name.contains("indexing")) {
+ returnCode = mockStormCLIClientWrapper.activateIndexingTopology(name);
} else {
returnCode = mockStormCLIClientWrapper.activateParserTopology(name);
}
} else if (action.equals("deactivate")){
if (name.equals("enrichment")) {
returnCode = mockStormCLIClientWrapper.deactivateEnrichmentTopology();
- } else if (name.equals("indexing")) {
- returnCode = mockStormCLIClientWrapper.deactivateIndexingTopology();
+ } else if (name.contains("indexing")) {
+ returnCode = mockStormCLIClientWrapper.deactivateIndexingTopology(name);
} else {
returnCode = mockStormCLIClientWrapper.deactivateParserTopology(name);
}
http://git-wip-us.apache.org/repos/asf/metron/blob/fcff0596/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/StormAdminServiceImplTest.java
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/StormAdminServiceImplTest.java b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/StormAdminServiceImplTest.java
index d83a74c..65a1bda 100644
--- a/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/StormAdminServiceImplTest.java
+++ b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/StormAdminServiceImplTest.java
@@ -122,22 +122,22 @@ public class StormAdminServiceImplTest {
@Test
public void startIndexingTopologyShouldProperlyReturnSuccessTopologyResponse() throws Exception {
- when(stormCLIClientWrapper.startIndexingTopology()).thenReturn(0);
+ when(stormCLIClientWrapper.startIndexingTopology("random_access_indexing_script_path")).thenReturn(0);
TopologyResponse expected = new TopologyResponse();
expected.setSuccessMessage(TopologyStatusCode.STARTED.toString());
- assertEquals(expected, stormAdminService.startIndexingTopology());
+ assertEquals(expected, stormAdminService.startIndexingTopology("random_access_indexing_script_path"));
}
@Test
public void stopIndexingTopologyShouldProperlyReturnSuccessTopologyResponse() throws Exception {
- when(stormCLIClientWrapper.stopIndexingTopology(false)).thenReturn(0);
+ when(stormCLIClientWrapper.stopIndexingTopology("random_access_indexing", false)).thenReturn(0);
TopologyResponse expected = new TopologyResponse();
expected.setSuccessMessage(TopologyStatusCode.STOPPED.toString());
- assertEquals(expected, stormAdminService.stopIndexingTopology(false));
+ assertEquals(expected, stormAdminService.stopIndexingTopology("random_access_indexing",false));
}
@Test
http://git-wip-us.apache.org/repos/asf/metron/blob/fcff0596/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/StormCLIWrapperTest.java
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/StormCLIWrapperTest.java b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/StormCLIWrapperTest.java
index 73d54d8..60a9790 100644
--- a/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/StormCLIWrapperTest.java
+++ b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/StormCLIWrapperTest.java
@@ -174,11 +174,11 @@ public class StormCLIWrapperTest {
whenNew(ProcessBuilder.class).withParameterTypes(String[].class).withArguments(anyVararg()).thenReturn(processBuilder);
when(processBuilder.start()).thenReturn(process);
- when(environment.getProperty(MetronRestConstants.INDEXING_SCRIPT_PATH_SPRING_PROPERTY)).thenReturn("/start_indexing");
+ when(environment.getProperty(MetronRestConstants.RANDOM_ACCESS_INDEXING_SCRIPT_PATH_SPRING_PROPERTY)).thenReturn("/start_indexing");
when(environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)).thenReturn(false);
when(process.exitValue()).thenReturn(0);
- assertEquals(0, stormCLIWrapper.startIndexingTopology());
+ assertEquals(0, stormCLIWrapper.startIndexingTopology(MetronRestConstants.RANDOM_ACCESS_INDEXING_SCRIPT_PATH_SPRING_PROPERTY));
verify(process).waitFor();
verifyNew(ProcessBuilder.class).withArguments("/start_indexing");
@@ -192,9 +192,9 @@ public class StormCLIWrapperTest {
when(environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)).thenReturn(false);
when(process.exitValue()).thenReturn(0);
- assertEquals(0, stormCLIWrapper.stopIndexingTopology(false));
+ assertEquals(0, stormCLIWrapper.stopIndexingTopology("random_access_indexing", false));
verify(process).waitFor();
- verifyNew(ProcessBuilder.class).withArguments("storm", "kill", MetronRestConstants.INDEXING_TOPOLOGY_NAME);
+ verifyNew(ProcessBuilder.class).withArguments("storm", "kill", MetronRestConstants.RANDOM_ACCESS_INDEXING_TOPOLOGY_NAME);
}
@Test
@@ -209,15 +209,16 @@ public class StormCLIWrapperTest {
when(process.getInputStream()).thenReturn(inputStream);
when(environment.getProperty(MetronRestConstants.PARSER_SCRIPT_PATH_SPRING_PROPERTY)).thenReturn("/start_parser");
when(environment.getProperty(MetronRestConstants.ENRICHMENT_SCRIPT_PATH_SPRING_PROPERTY)).thenReturn("/start_enrichment");
- when(environment.getProperty(MetronRestConstants.INDEXING_SCRIPT_PATH_SPRING_PROPERTY)).thenReturn("/start_indexing");
-
+ when(environment.getProperty(MetronRestConstants.RANDOM_ACCESS_INDEXING_SCRIPT_PATH_SPRING_PROPERTY)).thenReturn("/start_elasticsearch");
+ when(environment.getProperty(MetronRestConstants.BATCH_INDEXING_SCRIPT_PATH_SPRING_PROPERTY)).thenReturn("/start_hdfs");
Map<String, String> actual = stormCLIWrapper.getStormClientStatus();
assertEquals(new HashMap<String, String>() {{
- put("parserScriptPath", "/start_parser");
+ put("randomAccessIndexingScriptPath", "/start_elasticsearch");
put("enrichmentScriptPath", "/start_enrichment");
- put("indexingScriptPath", "/start_indexing");
put("stormClientVersionInstalled", "1.1");
+ put("parserScriptPath", "/start_parser");
+ put("batchIndexingScriptPath", "/start_hdfs");
}}, actual);
verifyNew(ProcessBuilder.class).withArguments("storm", "version");
[7/8] metron git commit: METRON-1446: Fix openjdk issue with Ubuntu
(mmiklavc via mmiklavc) closes apache/metron#926
Posted by ce...@apache.org.
METRON-1446: Fix openjdk issue with Ubuntu (mmiklavc via mmiklavc) closes apache/metron#926
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/38b7d5e3
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/38b7d5e3
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/38b7d5e3
Branch: refs/heads/feature/METRON-1416-upgrade-solr
Commit: 38b7d5e339320e71cd4d35d31b8d118a3e6d0be7
Parents: fcff059
Author: mmiklavc <mi...@gmail.com>
Authored: Tue Feb 6 09:57:15 2018 -0700
Committer: Michael Miklavcic <mi...@gmail.com>
Committed: Tue Feb 6 09:57:15 2018 -0700
----------------------------------------------------------------------
.../ansible/roles/java_jdk/tasks/install_jdk_ubuntu.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/38b7d5e3/metron-deployment/ansible/roles/java_jdk/tasks/install_jdk_ubuntu.yml
----------------------------------------------------------------------
diff --git a/metron-deployment/ansible/roles/java_jdk/tasks/install_jdk_ubuntu.yml b/metron-deployment/ansible/roles/java_jdk/tasks/install_jdk_ubuntu.yml
index 8337b81..4eb0945 100644
--- a/metron-deployment/ansible/roles/java_jdk/tasks/install_jdk_ubuntu.yml
+++ b/metron-deployment/ansible/roles/java_jdk/tasks/install_jdk_ubuntu.yml
@@ -20,7 +20,7 @@
register: jdk_dir
- name: Install openjdk repository
- shell: add-apt-repository ppa:openjdk-r/ppa
+ shell: add-apt-repository ppa:openjdk-r/ppa -y
when: not jdk_dir.stat.exists
- name: Update package cache