You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@metron.apache.org by ce...@apache.org on 2017/02/23 15:55:10 UTC
[01/11] incubator-metron git commit: METRON-716 Add README.md to
site-book (ottobackwards) closes apache/incubator-metron#454
Repository: incubator-metron
Updated Branches:
refs/heads/Metron_0.3.1 f6c253f44 -> 7abd7e8a2
METRON-716 Add README.md to site-book (ottobackwards) closes apache/incubator-metron#454
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/d7147e32
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/d7147e32
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/d7147e32
Branch: refs/heads/Metron_0.3.1
Commit: d7147e32d9f4f279bc779d4e1d39d691b43c3d35
Parents: f6c253f
Author: ottobackwards <ot...@gmail.com>
Authored: Wed Feb 15 06:02:58 2017 -0500
Committer: Otto Fowler <ot...@apache.org>
Committed: Wed Feb 15 06:02:58 2017 -0500
----------------------------------------------------------------------
site-book/README.md | 52 ++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 52 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/d7147e32/site-book/README.md
----------------------------------------------------------------------
diff --git a/site-book/README.md b/site-book/README.md
new file mode 100644
index 0000000..93d3a31
--- /dev/null
+++ b/site-book/README.md
@@ -0,0 +1,52 @@
+# Metron Site-Book documentation
+
+Metron's Site Book is an attempt at producing documentation that is:
+
+- Versioned and reviewed
+- Tied to code versions
+- Highly local to the code being documented
+
+The idea is that a release manager would build the site-book (following the instructions below), then publish it from the public [Metron site](http://metron.incubator.apache.org/) as the docs for the new released version. Older site-book versions should remain available for users that need them.
+
+
+To build the book, do the following:
+
+In any git clone of incubator-metron containing the site-book subdirectory,
+
+```
+cd site-book
+bin/generate-md.sh
+mvn site:site
+```
+
+It only takes a few seconds. You may now view your copy of the book in a browser by opening
+
+```
+file:///your/path/to/incubator-metron/site-book/target/site/index.html
+```
+
+On a Mac, you can just type the following on the command line
+
+```
+open target/site/index.html
+```
+
+##Key Components:
+
+###bin/generate-md.sh
+
+- Copies all .md files from the code directory tree into the site tree
+- Performs some transformations on them
+- Generates the nav tree structure and labels
+
+###bin/fix-md-dialect.py
+
+- Called by 'generate-md.sh'
+- Does transforms within the text of each file
+ - Converts the Github-MD dialect of markdown into the doxia-markdown dialect
+
+###pom.xml and src/site/site.xml
+
+- [Doxia](https://maven.apache.org/doxia/) boilerplate, tweaked for our specific needs
+
+
[06/11] incubator-metron git commit: METRON-720 modify generate-md.sh
to re-throw errors from within 'find' closes apache/incubator-metron#455
Posted by ce...@apache.org.
METRON-720 modify generate-md.sh to re-throw errors from within 'find' closes apache/incubator-metron#455
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/b7cd3ea8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/b7cd3ea8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/b7cd3ea8
Branch: refs/heads/Metron_0.3.1
Commit: b7cd3ea82e27cde52ba81e87086f7ddaf612beaf
Parents: a6299fd
Author: mattf-horton <mf...@hortonworks.com>
Authored: Tue Feb 21 09:23:06 2017 -0500
Committer: cstella <ce...@gmail.com>
Committed: Tue Feb 21 09:23:06 2017 -0500
----------------------------------------------------------------------
.../metron-data-management/README.md | 16 +-
metron-platform/metron-indexing/README.md | 2 +-
site-book/.gitignore | 1 +
site-book/bin/fix-md-dialect.py | 2 +-
site-book/bin/generate-md.sh | 159 ++++++++++++-------
.../image-archive/ApacheIncubating_Logo.png | Bin 11294 -> 0 bytes
.../resources/image-archive/metron-logo.png | Bin 21186 -> 0 bytes
site-book/src/site/site.xml | 124 ---------------
.../images/ApacheIncubating_Logo.png | Bin 0 -> 11294 bytes
.../site/src-resources/images/metron-logo.png | Bin 0 -> 21186 bytes
.../src-resources/templates/site.xml.template | 80 ++++++++++
11 files changed, 189 insertions(+), 195 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/b7cd3ea8/metron-platform/metron-data-management/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-data-management/README.md b/metron-platform/metron-data-management/README.md
index 42e4b91..1bb7994 100644
--- a/metron-platform/metron-data-management/README.md
+++ b/metron-platform/metron-data-management/README.md
@@ -133,7 +133,7 @@ Users also have the ability to transform and filter enrichment and threat intel
As an example, we will be providing a CSV list of top domains as an enrichment and filtering the value metadata, as well as the indicator column, with Stellar expressions.
-````
+```
{
"config" : {
"zk_quorum" : "node1:2181",
@@ -155,7 +155,7 @@ As an example, we will be providing a CSV list of top domains as an enrichment a
},
"extractor" : "CSV"
}
-````
+```
There are 2 property maps that work with full Stellar expressions, and 2 properties that will work with Stellar predicates.
@@ -167,11 +167,11 @@ There are 2 property maps that work with full Stellar expressions, and 2 propert
| indicator_filter | Allows additional filtering with Stellar predicates based on results from the value transformations. In this example, records whose indicator value is empty after removing the TLD will be omitted. |
top-list.csv
-````
+```
1,google.com
2,youtube.com
...
-````
+```
Running a file import with the above data and extractor configuration would result in the following 2 extracted data records:
@@ -182,14 +182,14 @@ Running a file import with the above data and extractor configuration would resu
Similar to the parser framework, providing a Zookeeper quorum via the zk_quorum property will enable Stellar to access properties that reside in the global config.
Expanding on our example above, if the global config looks as follows:
-````
+```
{
"global_property" : "metron-ftw"
}
-````
+```
And we expand our value_tranform:
-````
+```
...
"value_transform" : {
"domain" : "DOMAIN_REMOVE_TLD(domain)",
@@ -197,7 +197,7 @@ And we expand our value_tranform:
},
...
-````
+```
The resulting value data would look like the following:
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/b7cd3ea8/metron-platform/metron-indexing/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-indexing/README.md b/metron-platform/metron-indexing/README.md
index cbe7a76..5296ea0 100644
--- a/metron-platform/metron-indexing/README.md
+++ b/metron-platform/metron-indexing/README.md
@@ -12,7 +12,7 @@ By default, this topology writes out to both HDFS and one of
Elasticsearch and Solr.
Indices are written in batch and the batch size is specified in the
-[Indexing Config](../metron-enrichment) via the `batchSize` parameter.
+[Sensor Indexing Configuration](#sensor-indexing-configuration) via the `batchSize` parameter.
This config is variable by sensor type.
## Indexing Architecture
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/b7cd3ea8/site-book/.gitignore
----------------------------------------------------------------------
diff --git a/site-book/.gitignore b/site-book/.gitignore
index 90e4c61..1a008f8 100644
--- a/site-book/.gitignore
+++ b/site-book/.gitignore
@@ -12,6 +12,7 @@ target/
*.settings
*.metadata
*hbase-site.xml
+site.xml
*.log
*.swp
*.tmp
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/b7cd3ea8/site-book/bin/fix-md-dialect.py
----------------------------------------------------------------------
diff --git a/site-book/bin/fix-md-dialect.py b/site-book/bin/fix-md-dialect.py
index 23ce42a..d594ca2 100755
--- a/site-book/bin/fix-md-dialect.py
+++ b/site-book/bin/fix-md-dialect.py
@@ -65,7 +65,7 @@ def report_error(s) :
print >>sys.stderr, "ERROR: " + s
print >>sys.stderr, "on line: " + str(FNR) + " in file: " + FILENAME
print >>sys.stderr, inputline
- exit -1
+ exit(1)
def trace(msg) :
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/b7cd3ea8/site-book/bin/generate-md.sh
----------------------------------------------------------------------
diff --git a/site-book/bin/generate-md.sh b/site-book/bin/generate-md.sh
index 623e141..14ddb54 100755
--- a/site-book/bin/generate-md.sh
+++ b/site-book/bin/generate-md.sh
@@ -32,6 +32,10 @@
# into a book-like collection. It should perhaps be viewed as a collection of essays,
# since each README.md file is written independently.
+
+## fail fast in the event of a failure of any command in this script
+set -e
+
## This script assumes it is running at $METRON_SOURCE/site-book/bin/
METRON_SOURCE=`cd $(dirname $0); cd ../..; pwd`
@@ -49,7 +53,7 @@ EXCLUSION_LIST=(
## This is a list of resources (eg .png files) needed to render the markdown files.
## Each entry is a file path, relative to $METRON_SOURCE.
-## Note: any images in site-book/src/site/resources/image-archive/ will also be included.
+## Note: any images in site-book/src/site/src-resources/images/ will also be included.
RESOURCE_LIST=(
metron-platform/metron-parsers/parser_arch.png
metron-platform/metron-indexing/indexing_arch.png
@@ -67,6 +71,8 @@ HREF_REWRITE_LIST=(
metron-analytics/metron-maas-service/README.md 's#(maas_arch.png)#(../../images/maas_arch.png)#g'
)
+TEMPLATES_DIR="$METRON_SOURCE/site-book/src/site/src-resources/templates"
+
######################
######################
@@ -77,18 +83,21 @@ HREF_REWRITE_LIST=(
TRACE_ENABLE=0
function trace () {
if (( $TRACE_ENABLE == 1 )) ; then
- echo "$*"
+ echo "$*"
fi # else do nothing
}
TREE_TRACE_ENABLE=0
function tree_trace () {
if (( $TREE_TRACE_ENABLE == 1 )) ; then
- echo "$*"
+ echo "$*"
fi # else do nothing
}
+# file used for storing error messages during re-write routine
+SCRATCH_ERR_FILE_NAME="$METRON_SOURCE/site-book/src/site/errout.dat"
+
# input: cumulative directory_path, indent_level
-# output: items to site.tmp, as lines of text
+# output: items to site.xml, as lines of text
# This function is called recursively as we descend the directory tree
# The cum_dir_path must not have a terminal "/".
function descend () {
@@ -100,38 +109,39 @@ function descend () {
indent=$2
if [ -e "${cum_dir_path}"/index.md ] ; then
- dir_name=`basename "$cum_dir_path"`
- dir_name="${dir_name#metron-}" #remove the "metron-" prefix if present
- dir_name=`get_prettyname "$dir_name"` #capitalize the remainder
- # Is it a leaf node?
- num_peers=`ls -d "${cum_dir_path}"/* |wc -l`
- if (( $num_peers == 1 )) ; then #yes, it's a leaf node, do a closed item
- echo "${INDENTS[$indent]}<item name='${dir_name}' href='${cum_dir_path}/index.html'/>" >> ../site.tmp
- tree_trace "exit descend due to leaf node"
- return #nothing else to process in this directory path
- fi #otherwise carry on with open item and child items at deeper indent
- echo "${INDENTS[$indent]}<item name='${dir_name}' href='${cum_dir_path}/index.html' collapse='true'>" >> ../site.tmp
- open_item_exists=1
- indent=$(( indent + 1 ))
+ dir_name=`basename "$cum_dir_path"`
+ dir_name="${dir_name#metron-}" #remove the "metron-" prefix if present
+ dir_name=`get_prettyname "$dir_name"` #capitalize the remainder
+ # Is it a leaf node?
+ num_peers=`ls -d "${cum_dir_path}"/* |wc -l`
+ if (( $num_peers == 1 )) ; then #yes, it's a leaf node, do a closed item
+ echo "${INDENTS[$indent]}<item name='${dir_name}' href='${cum_dir_path}/index.html'/>" >> ../site.xml
+ tree_trace "exit descend due to leaf node"
+ return #nothing else to process in this directory path
+ fi #otherwise carry on with open item and child items at deeper indent
+ echo "${INDENTS[$indent]}<item name='${dir_name}' href='${cum_dir_path}/index.html' collapse='true'>" >> ../site.xml
+ open_item_exists=1
+ indent=$(( indent + 1 ))
else
- open_item_exists=0
+ open_item_exists=0
fi
for md in "${cum_dir_path}"/*.md ; do
- if [ ! -e "$md" ] ; then continue ; fi #globbing sometimes gives spurious results
- item_name=`basename -s ".md" "$md"` #strip the suffix
- if [ "$item_name" != "index" ] ; then
- echo "${INDENTS[$indent]}<item name='${item_name}' href='${cum_dir_path}/${item_name}.html'/>" >> ../site.tmp
- fi
+ if [ ! -e "$md" ] ; then continue ; fi #globbing sometimes gives spurious results
+ item_name=`basename "$md"`
+ item_name="${item_name%.md}" #strip the extension
+ if [ "$item_name" != "index" ] ; then
+ echo "${INDENTS[$indent]}<item name='${item_name}' href='${cum_dir_path}/${item_name}.html'/>" >> ../site.xml
+ fi
done
for dir in "${cum_dir_path}"/* ; do
- if [ ! -e "$dir" ] ; then continue ; fi #globbing sometimes gives spurious results
- if [ -d "$dir" ] ; then
- descend "$dir" $indent
- fi
+ if [ ! -e "$dir" ] ; then continue ; fi #globbing sometimes gives spurious results
+ if [ -d "$dir" ] ; then
+ descend "$dir" $indent
+ fi
done
if (( open_item_exists == 1 )) ; then
- indent=$(( indent - 1 )) #close the item
- echo "${INDENTS[$indent]}</item>" >> ../site.tmp
+ indent=$(( indent - 1 )) #close the item
+ echo "${INDENTS[$indent]}</item>" >> ../site.xml
fi
tree_trace "exit descend with indent = $indent"
}
@@ -144,17 +154,37 @@ function get_prettyname () {
echo "$(tr '[:lower:]' '[:upper:]' <<< ${1:0:1})${1:1}"
}
+# This function, with the following traps, cleans up before exiting, if interrupted during the re-write routine
+function sig_handle () {
+ exitCode=${1:-0}
+ rm -f "$SCRATCH_ERR_FILE_NAME"
+ echo "ERROR: EARLY TERMINATION with error code $exitCode" ${2:+"due to $2"}
+ exit $exitCode
+}
+trap 'sig_handle 129 SIGHUP' SIGHUP
+trap 'sig_handle 130 SIGINT' SIGINT
+trap 'sig_handle 143 SIGTERM' SIGTERM
+trap 'sig_handle $? ERR' ERR
+
######################
## Proceed
cd "$METRON_SOURCE"
-# Clean up generated directories
-if [ -d "$METRON_SOURCE"/site-book/src/site/markdown ] ; then
+# Validate that the src/site directory is writable for generated content
+if [ ! -w "site-book/src/site" ]; then
+ echo "ERROR: 'site-book/src/site' is not writable" > /dev/stderr
+ exit 126
+fi
+
+# Clean up generated directories and files in src/site/
+if [ -e "$METRON_SOURCE"/site-book/src/site/markdown ] ; then
rm -rf "$METRON_SOURCE"/site-book/src/site/markdown ; fi
-if [ -d "$METRON_SOURCE"/site-book/src/site/resources/images ] ; then
+if [ -e "$METRON_SOURCE"/site-book/src/site/resources/images ] ; then
rm -rf "$METRON_SOURCE"/site-book/src/site/resources/images ; fi
+if [ -e "$METRON_SOURCE"/site-book/src/site/site.xml ] ; then
+ rm -f "$METRON_SOURCE"/site-book/src/site/site.xml; fi
mkdir -p "$METRON_SOURCE"/site-book/src/site/markdown \
"$METRON_SOURCE"/site-book/src/site/resources/images
@@ -178,7 +208,7 @@ tar cvf - "${MD_FILE_LIST[@]}" | ( cd "$METRON_SOURCE"/site-book/src/site/markdo
# Grab the other resources needed
echo " "
echo Collecting additional resource files:
-for r in "${RESOURCE_LIST[@]}" site-book/src/site/resources/image-archive/* ; do
+for r in "${RESOURCE_LIST[@]}" site-book/src/site/src-resources/images/* ; do
if [ ! -e "$r" ] ; then continue ; fi #globbing sometimes gives spurious results
echo ./"$r"
cp "$r" "$METRON_SOURCE"/site-book/src/site/resources/images/
@@ -191,18 +221,18 @@ cd site-book/src/site/markdown
for (( i=0; i<${#HREF_REWRITE_LIST[@]} ; i+=2 )) ; do
echo rewriting href in "${HREF_REWRITE_LIST[$i]}" : "${HREF_REWRITE_LIST[ $(( i + 1 )) ]}"
case "${OSTYPE}" in
- linux*)
- # Linux sed correctly parses lack of argument after -i option
+ linux*)
+ # Linux sed correctly parses lack of argument after -i option
sed -i -e "${HREF_REWRITE_LIST[ $(( i + 1 )) ]}" "${HREF_REWRITE_LIST[$i]}"
- ;;
- darwin*)
+ ;;
+ darwin*)
# MacOS sed needs an empty-string argument after -i option to get the same result
sed -i '' -e "${HREF_REWRITE_LIST[ $(( i + 1 )) ]}" "${HREF_REWRITE_LIST[$i]}"
- ;;
- *)
- echo "ERROR: Unable to determine 'sed' argument list for OS ${OSTYPE}" > /dev/stderr
- exit -1
- ;;
+ ;;
+ *)
+ echo "ERROR: Unable to determine 'sed' argument list for OS ${OSTYPE}" > /dev/stderr
+ exit 126
+ ;;
esac
done
echo " "
@@ -213,15 +243,15 @@ echo " "
echo Renaming \"README\" files to \"index\" files.
if (( `ls -R |grep -c 'index.md'` > 0 )) ; then
echo "ERROR: index.md file exists in tree already, we currently don't handle that"
- exit -1
+ exit 1
fi
find . -name README.md -execdir mv README.md index.md \;
echo " "
-# Insert the tree of generated html files in the LHS menu of the site.xml
+# Insert the tree of generated html files in the LHS nav menu of the site.xml
# The problem is that we want a depth-first listing, with files before subdirectories, and "index" always first.
-# So the following logic is a little complex, but we avoid having to hardwire the tree structure -- which we
-# may go back to in the long run.
+# And we synthesize the page labels in the nav tree from the directory paths.
+# So the following logic is a little complex, but we avoid having to hardwire the tree structure.
BEGIN_TAG="BEGIN_MENU_TREE"
END_TAG="END_MENU_TREE"
@@ -231,12 +261,12 @@ echo "Generating menu tree from directory tree structure"
echo " "
# Copy the first part of the file, up to where the menu tree goes.
-sed -n -e "1,/${BEGIN_TAG}/ p" ../site.xml > ../site.tmp
+sed -n -e "1,/${BEGIN_TAG}/ p" "$TEMPLATES_DIR"/site.xml.template > ../site.xml
# Now start inserting menu tree items
# top level of markdown tree is special
if [ -e index.md ] ; then
- echo "<item name='Metron' href='index.html' title='Apache Metron - Incubating' collapse='false'>" >> ../site.tmp
+ echo "<item name='Metron' href='index.html' title='Apache Metron - Incubating' collapse='false'>" >> ../site.xml
item0_exists=1
else
item0_exists=0
@@ -245,33 +275,40 @@ indent_level=1
for md in *.md ; do
if [ ! -e "$md" ] ; then continue ; fi #globbing sometimes gives spurious results
if [ "$md" != "index.md" ] ; then
- item_name="${md%.*}" #strip the suffix
- echo "${INDENTS[$indent_level]}<item name='${item_name}' href='${item_name}.html' />" >> ../site.tmp
+ item_name="${md%.md}" #strip the extension
+ echo "${INDENTS[$indent_level]}<item name='${item_name}' href='${item_name}.html' />" >> ../site.xml
fi
done
for dir in * ; do
if [ ! -e "$dir" ] ; then continue ; fi #globbing sometimes gives spurious results
if [ -d "$dir" ] ; then
- descend "$dir" $indent_level
+ descend "$dir" $indent_level
fi
done
if (( item0_exists == 1 )) ; then
- echo "</item>" >> ../site.tmp
+ echo "</item>" >> ../site.xml
fi
# Copy the last part of the file, from the end of the menu tree.
-sed -n -e "/${END_TAG}/,"'$ p' ../site.xml >> ../site.tmp
-
-mv ../site.xml ../site.xml.bak
-mv ../site.tmp ../site.xml
+sed -n -e "/${END_TAG}/,"'$ p' "$TEMPLATES_DIR"/site.xml.template >> ../site.xml
echo "Done."
echo " "
echo "Fixing up markdown dialect problems between Github-MD and doxia-markdown:"
-find . -name '*.md' -print -exec python "$METRON_SOURCE"/site-book/bin/fix-md-dialect.py '{}' \;
-echo "Done."
-echo " "
-
+# Detecting errors from a `find -exec` command is difficult. We do it using an intermediary file.
+rm -f "$SCRATCH_ERR_FILE_NAME"
+find . -name '*.md' -print -exec python "$METRON_SOURCE"/site-book/bin/fix-md-dialect.py '{}' \; 2> "$SCRATCH_ERR_FILE_NAME"
+errlines=`wc -l "$SCRATCH_ERR_FILE_NAME"`
+if (( ${errlines% *} > 0 )) ; then
+ echo "ERROR OR ERRORS DETECTED:"
+ cat "$SCRATCH_ERR_FILE_NAME"
+ rm -f "$SCRATCH_ERR_FILE_NAME"
+ exit 1
+else
+ rm -f "$SCRATCH_ERR_FILE_NAME"
+ echo "Done."
+ echo " "
+ exit 0
+fi
-exit 0
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/b7cd3ea8/site-book/src/site/resources/image-archive/ApacheIncubating_Logo.png
----------------------------------------------------------------------
diff --git a/site-book/src/site/resources/image-archive/ApacheIncubating_Logo.png b/site-book/src/site/resources/image-archive/ApacheIncubating_Logo.png
deleted file mode 100644
index 83f096c..0000000
Binary files a/site-book/src/site/resources/image-archive/ApacheIncubating_Logo.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/b7cd3ea8/site-book/src/site/resources/image-archive/metron-logo.png
----------------------------------------------------------------------
diff --git a/site-book/src/site/resources/image-archive/metron-logo.png b/site-book/src/site/resources/image-archive/metron-logo.png
deleted file mode 100644
index a0bc8cb..0000000
Binary files a/site-book/src/site/resources/image-archive/metron-logo.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/b7cd3ea8/site-book/src/site/site.xml
----------------------------------------------------------------------
diff --git a/site-book/src/site/site.xml b/site-book/src/site/site.xml
deleted file mode 100644
index ba96f27..0000000
--- a/site-book/src/site/site.xml
+++ /dev/null
@@ -1,124 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<!--
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-<project name="Metron" xmlns="http://maven.apache.org/DECORATION/1.3.0"
- xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
- xsi:schemaLocation="http://maven.apache.org/DECORATION/1.3.0 http://maven.apache.org/xsd/decoration-1.3.0.xsd">
-
- <skin>
- <groupId>org.apache.maven.skins</groupId>
- <artifactId>maven-fluido-skin</artifactId>
- <version>1.3.0</version>
- </skin>
-
- <custom>
- <fluidoSkin>
- <project>Apache Metron - Incubating</project>
- <topBarEnabled>false</topBarEnabled>
- <sideBarEnabled>true</sideBarEnabled>
- </fluidoSkin>
- </custom>
-
- <bannerLeft>
- <name>Apache Metron - Incubating</name>
- <src>images/metron-logo.png</src>
- <href>http://metron.incubator.apache.org/</href>
- <width>148px</width>
- <height>48px</height>
- </bannerLeft>
-
- <bannerRight>
- <name>Apache Incubating</name>
- <src>images/ApacheIncubating_Logo.png</src>
- <href>http://incubator.apache.org/</href>
- <width>192px</width>
- <height>48px</height>
- </bannerRight>
-
- <publishDate position="right"/>
- <version position="right"/>
-
- <body>
- <head>
- <script type="text/javascript">
- $( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );
- </script>
- </head>
-
- <breadcrumbs position="left">
- <item name="Apache" href="http://www.apache.org"/>
- <item name="Metron-Incubating" title="Apache Metron - Incubating" href="http://metron.incubator.apache.org/"/>
- <item name="Documentation" title="Metron Docs" href="index.html"/>
- </breadcrumbs>
-
- <menu name="User Documentation">
- <!-- BEGIN_MENU_TREE - Do not remove this line, it is used for auto-insert -->
-<item name='Metron' href='index.html' title='Apache Metron - Incubating' collapse='false'>
- <item name='Analytics' href='metron-analytics/index.html' collapse='true'>
- <item name='Maas-service' href='metron-analytics/metron-maas-service/index.html'/>
- <item name='Profiler' href='metron-analytics/metron-profiler/index.html'/>
- <item name='Profiler-client' href='metron-analytics/metron-profiler-client/index.html'/>
- <item name='Statistics' href='metron-analytics/metron-statistics/index.html' collapse='true'>
- <item name='HLLP' href='metron-analytics/metron-statistics/HLLP.html'/>
- </item>
- </item>
- <item name='Deployment' href='metron-deployment/index.html' collapse='true'>
- <item name='Amazon-ec2' href='metron-deployment/amazon-ec2/index.html'/>
- <item name='Ansible-docker' href='metron-deployment/packaging/docker/ansible-docker/index.html'/>
- <item name='Rpm-docker' href='metron-deployment/packaging/docker/rpm-docker/index.html'/>
- <item name='Packer-build' href='metron-deployment/packer-build/index.html'/>
- <item name='Roles' href='metron-deployment/roles/index.html' collapse='true'>
- <item name='Kibana' href='metron-deployment/roles/kibana/index.html'/>
- <item name='Monit' href='metron-deployment/roles/monit/index.html'/>
- <item name='Opentaxii' href='metron-deployment/roles/opentaxii/index.html'/>
- <item name='Pcap_replay' href='metron-deployment/roles/pcap_replay/index.html'/>
- <item name='Sensor-stubs' href='metron-deployment/roles/sensor-stubs/index.html'/>
- <item name='Sensor-test-mode' href='metron-deployment/roles/sensor-test-mode/index.html'/>
- </item>
- <item name='Vagrant' href='metron-deployment/vagrant/index.html' collapse='true'>
- <item name='Codelab-platform' href='metron-deployment/vagrant/codelab-platform/index.html'/>
- <item name='Fastcapa-test-platform' href='metron-deployment/vagrant/fastcapa-test-platform/index.html'/>
- <item name='Full-dev-platform' href='metron-deployment/vagrant/full-dev-platform/index.html'/>
- <item name='Quick-dev-platform' href='metron-deployment/vagrant/quick-dev-platform/index.html'/>
- </item>
- </item>
- <item name='Docker' href='metron-docker/index.html'/>
- <item name='Platform' href='metron-platform/index.html' collapse='true'>
- <item name='Api' href='metron-platform/metron-api/index.html'/>
- <item name='Common' href='metron-platform/metron-common/index.html'/>
- <item name='Data-management' href='metron-platform/metron-data-management/index.html'/>
- <item name='Enrichment' href='metron-platform/metron-enrichment/index.html'/>
- <item name='Indexing' href='metron-platform/metron-indexing/index.html'/>
- <item name='Management' href='metron-platform/metron-management/index.html'/>
- <item name='Parsers' href='metron-platform/metron-parsers/index.html'/>
- <item name='Pcap-backend' href='metron-platform/metron-pcap-backend/index.html'/>
- </item>
- <item name='Sensors' href='metron-sensors/index.html' collapse='true'>
- <item name='Fastcapa' href='metron-sensors/fastcapa/index.html'/>
- <item name='Pycapa' href='metron-sensors/pycapa/index.html'/>
- </item>
-</item>
- <!-- END_MENU_TREE - Do not remove this line, it is used for auto-insert -->
- </menu>
-
- <footer>
- © 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo,
- and the Apache Metron project logo are trademarks of The Apache Software Foundation.
- </footer>
- </body>
-</project>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/b7cd3ea8/site-book/src/site/src-resources/images/ApacheIncubating_Logo.png
----------------------------------------------------------------------
diff --git a/site-book/src/site/src-resources/images/ApacheIncubating_Logo.png b/site-book/src/site/src-resources/images/ApacheIncubating_Logo.png
new file mode 100644
index 0000000..83f096c
Binary files /dev/null and b/site-book/src/site/src-resources/images/ApacheIncubating_Logo.png differ
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/b7cd3ea8/site-book/src/site/src-resources/images/metron-logo.png
----------------------------------------------------------------------
diff --git a/site-book/src/site/src-resources/images/metron-logo.png b/site-book/src/site/src-resources/images/metron-logo.png
new file mode 100644
index 0000000..a0bc8cb
Binary files /dev/null and b/site-book/src/site/src-resources/images/metron-logo.png differ
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/b7cd3ea8/site-book/src/site/src-resources/templates/site.xml.template
----------------------------------------------------------------------
diff --git a/site-book/src/site/src-resources/templates/site.xml.template b/site-book/src/site/src-resources/templates/site.xml.template
new file mode 100644
index 0000000..85a61e1
--- /dev/null
+++ b/site-book/src/site/src-resources/templates/site.xml.template
@@ -0,0 +1,80 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project name="Metron" xmlns="http://maven.apache.org/DECORATION/1.3.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/DECORATION/1.3.0 http://maven.apache.org/xsd/decoration-1.3.0.xsd">
+
+ <skin>
+ <groupId>org.apache.maven.skins</groupId>
+ <artifactId>maven-fluido-skin</artifactId>
+ <version>1.3.0</version>
+ </skin>
+
+ <custom>
+ <fluidoSkin>
+ <project>Apache Metron - Incubating</project>
+ <topBarEnabled>false</topBarEnabled>
+ <sideBarEnabled>true</sideBarEnabled>
+ </fluidoSkin>
+ </custom>
+
+ <bannerLeft>
+ <name>Apache Metron - Incubating</name>
+ <src>images/metron-logo.png</src>
+ <href>http://metron.incubator.apache.org/</href>
+ <width>148px</width>
+ <height>48px</height>
+ </bannerLeft>
+
+ <bannerRight>
+ <name>Apache Incubating</name>
+ <src>images/ApacheIncubating_Logo.png</src>
+ <href>http://incubator.apache.org/</href>
+ <width>192px</width>
+ <height>48px</height>
+ </bannerRight>
+
+ <publishDate position="right"/>
+ <version position="right"/>
+
+ <body>
+ <head>
+ <script type="text/javascript">
+ $( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );
+ </script>
+ </head>
+
+ <breadcrumbs position="left">
+ <item name="Apache" href="http://www.apache.org"/>
+ <item name="Metron-Incubating" title="Apache Metron - Incubating" href="http://metron.incubator.apache.org/"/>
+ <item name="Documentation" title="Metron Docs" href="index.html"/>
+ </breadcrumbs>
+
+ <menu name="User Documentation">
+ <!-- BEGIN_MENU_TREE - Do not remove this line, it is used for auto-insert -->
+ <!-- The nav tree contents are machine generated into this location -->
+ <!-- END_MENU_TREE - Do not remove this line, it is used for auto-insert -->
+ </menu>
+
+ <footer>
+ © 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo,
+ and the Apache Metron project logo are trademarks of The Apache Software Foundation.
+ </footer>
+ </body>
+</project>
[11/11] incubator-metron git commit: METRON-734 Builds failing
because of MaxMind DB transitive dependency (justinleet via cestella) closes
apache/incubator-metron#462
Posted by ce...@apache.org.
METRON-734 Builds failing because of MaxMind DB transitive dependency (justinleet via cestella) closes apache/incubator-metron#462
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/7abd7e8a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/7abd7e8a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/7abd7e8a
Branch: refs/heads/Metron_0.3.1
Commit: 7abd7e8a231c6cbe9ee4ab23a5df1e97344f5212
Parents: 0e8abc4
Author: justinleet <ju...@gmail.com>
Authored: Thu Feb 23 10:40:14 2017 -0500
Committer: cstella <ce...@gmail.com>
Committed: Thu Feb 23 10:40:14 2017 -0500
----------------------------------------------------------------------
metron-platform/metron-enrichment/pom.xml | 10 ++++++++++
1 file changed, 10 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/7abd7e8a/metron-platform/metron-enrichment/pom.xml
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/pom.xml b/metron-platform/metron-enrichment/pom.xml
index 4dca431..7daf297 100644
--- a/metron-platform/metron-enrichment/pom.xml
+++ b/metron-platform/metron-enrichment/pom.xml
@@ -76,6 +76,16 @@
<groupId>com.maxmind.geoip2</groupId>
<artifactId>geoip2</artifactId>
<version>${geoip.version}</version>
+ <exclusions>
+ <exclusion>
+ <artifactId>jackson-core</artifactId>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>jackson-databind</artifactId>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ </exclusion>
+ </exclusions>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
[09/11] incubator-metron git commit: METRON-157 Create CEF Parser
(simonellistonball via kylerichardson) closes apache/incubator-metron#451
Posted by ce...@apache.org.
METRON-157 Create CEF Parser (simonellistonball via kylerichardson) closes apache/incubator-metron#451
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/9e15cb6e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/9e15cb6e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/9e15cb6e
Branch: refs/heads/Metron_0.3.1
Commit: 9e15cb6e24872620ec4bf3c183d15dd6292b153d
Parents: 246acff
Author: simonellistonball <si...@simonellistonball.com>
Authored: Tue Feb 21 15:50:51 2017 -0500
Committer: Kyle Richardson <ky...@apache.org>
Committed: Tue Feb 21 15:50:51 2017 -0500
----------------------------------------------------------------------
metron-platform/metron-parsers/pom.xml | 2 +-
.../apache/metron/parsers/cef/CEFParser.java | 274 ++++++++++++++++++
.../apache/metron/parsers/utils/DateUtils.java | 115 ++++++++
.../metron/parsers/cef/CEFParserTest.java | 277 +++++++++++++++++++
.../org/apache/metron/parsers/cef/adallom.cef | 1 +
.../apache/metron/parsers/cef/adallom.schema | 37 +++
.../org/apache/metron/parsers/cef/cyberark.cef | 1 +
.../org/apache/metron/parsers/cef/cyberark.json | 21 ++
.../apache/metron/parsers/cef/cyberark.schema | 38 +++
.../org/apache/metron/parsers/cef/waf.cef | 1 +
.../org/apache/metron/parsers/cef/waf.schema | 67 +++++
11 files changed, 833 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/9e15cb6e/metron-platform/metron-parsers/pom.xml
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/pom.xml b/metron-platform/metron-parsers/pom.xml
index d8a77a0..3049a71 100644
--- a/metron-platform/metron-parsers/pom.xml
+++ b/metron-platform/metron-parsers/pom.xml
@@ -143,7 +143,7 @@
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
- <version>${global_hbase_guava_version}</version>
+ <version>${global_guava_version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/9e15cb6e/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/cef/CEFParser.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/cef/CEFParser.java b/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/cef/CEFParser.java
new file mode 100644
index 0000000..a765dd8
--- /dev/null
+++ b/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/cef/CEFParser.java
@@ -0,0 +1,274 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.cef;
+
+import java.nio.charset.Charset;
+import java.time.Clock;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.metron.parsers.BasicParser;
+import org.apache.metron.parsers.ParseException;
+import org.apache.metron.parsers.utils.DateUtils;
+import org.apache.metron.parsers.utils.SyslogUtils;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class CEFParser extends BasicParser {
+ private static final long serialVersionUID = 1L;
+
+ protected static final Logger LOG = LoggerFactory.getLogger(CEFParser.class);
+ private static final String HEADER_CAPTURE_PATTERN = "[^\\|]*";
+ private static final String EXTENSION_CAPTURE_PATTERN = "(?<!\\\\)=";
+ private static final Charset UTF_8 = Charset.forName("UTF-8");
+
+ private Pattern p;
+ private Pattern pext;
+
+ public void init() {
+
+ // CEF Headers: Device Vendor|Device Product|Device Version|Device Event
+ // Class ID|Name|Severity
+
+ String syslogTime = "(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\\b +(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) (?!<[0-9])(?:2[0123]|[01]?[0-9]):(?:[0-5][0-9])(?::(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?))(?![0-9])?";
+ String syslogTime5424 = "(?:\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}(?:\\.\\d+)?(?:Z|[+-]\\d{2}:\\d{2}))";
+ String syslogPriority = "<(?:[0-9]+)>";
+ String syslogHost = "[a-z0-9\\.\\\\-_]+";
+
+ StringBuilder sb = new StringBuilder("(?<syslogTime>");
+ sb.append(syslogTime);
+ sb.append("|");
+ sb.append(syslogTime5424);
+ sb.append(")?");
+
+ sb.append("(?<syslogHost>");
+ sb.append(syslogHost);
+ sb.append(")?");
+
+ sb.append("(?<syslogPriority>");
+ sb.append(syslogPriority);
+ sb.append(")?");
+
+ sb.append(".*");
+
+ sb.append("CEF:0\\|");
+
+ headerBlock("DeviceVendor", sb);
+ sb.append("\\|");
+ headerBlock("DeviceProduct", sb);
+ sb.append("\\|");
+ headerBlock("DeviceVersion", sb);
+ sb.append("\\|");
+ headerBlock("DeviceEvent", sb);
+ sb.append("\\|");
+ headerBlock("Name", sb);
+ sb.append("\\|");
+ headerBlock("Severity", sb);
+ sb.append("\\|");
+
+ // extension capture:
+ sb.append("(?<extensions>.*)");
+ String pattern = sb.toString();
+
+ p = Pattern.compile(pattern);
+
+ // key finder for extensions
+ pext = Pattern.compile(EXTENSION_CAPTURE_PATTERN);
+ }
+
+ @SuppressWarnings("unchecked")
+ public List<JSONObject> parse(byte[] rawMessage) {
+ List<JSONObject> messages = new ArrayList<>();
+
+ String cefString = new String(rawMessage, UTF_8);
+
+ Matcher matcher = p.matcher(cefString);
+
+ while (matcher.find()) {
+ JSONObject obj = new JSONObject();
+ if (matcher.matches()) {
+ LOG.info(String.format("Found %d groups", matcher.groupCount()));
+ obj.put("DeviceVendor", matcher.group("DeviceVendor"));
+ obj.put("DeviceProduct", matcher.group("DeviceProduct"));
+ obj.put("DeviceVersion", matcher.group("DeviceVersion"));
+ obj.put("DeviceEvent", matcher.group("DeviceEvent"));
+ obj.put("Name", matcher.group("Name"));
+ obj.put("Severity", standardizeSeverity(matcher.group("Severity")));
+ }
+
+ String ext = matcher.group("extensions");
+ Matcher m = pext.matcher(ext);
+
+ int index = 0;
+ String key = null;
+ String value = null;
+ Map<String, String> labelMap = new HashMap<String, String>();
+
+ while (m.find()) {
+ if (key == null) {
+ key = ext.substring(index, m.start());
+ index = m.end();
+ if (!m.find()) {
+ break;
+ }
+ }
+ value = ext.substring(index, m.start());
+ index = m.end();
+ int v = value.lastIndexOf(" ");
+ if (v > 0) {
+ String temp = value.substring(0, v).trim();
+ if (key.endsWith("Label")) {
+ labelMap.put(key.substring(0, key.length() - 5), temp);
+ } else {
+ obj.put(key, temp);
+ }
+ key = value.substring(v).trim();
+ }
+ }
+ value = ext.substring(index);
+
+ // Build a map of Label extensions to apply later
+ if (key.endsWith("Label")) {
+ labelMap.put(key.substring(0, key.length() - 5), value);
+ } else {
+ obj.put(key, value);
+ }
+
+ // Apply the labels to custom fields
+ for (Entry<String, String> label : labelMap.entrySet()) {
+ mutate(obj, label.getKey(), label.getValue());
+ }
+
+ // Rename standard CEF fields to comply with Metron standards
+ obj = mutate(obj, "dst", "ip_dst_addr");
+ obj = mutate(obj, "dpt", "ip_dst_port");
+ obj = convertToInt(obj, "ip_dst_port");
+
+ obj = mutate(obj, "src", "ip_src_addr");
+ obj = mutate(obj, "spt", "ip_src_port");
+ obj = convertToInt(obj, "ip_src_port");
+
+ obj = mutate(obj, "act", "deviceAction");
+ // applicationProtocol
+ obj = mutate(obj, "app", "protocol");
+
+ obj.put("original_string", cefString);
+
+ // apply timestamp from message if present, using rt, syslog
+ // timestamp,
+ // default to current system time
+
+ if (obj.containsKey("rt")) {
+ String rt = (String) obj.get("rt");
+ try {
+ obj.put("timestamp", DateUtils.parseMultiformat(rt, DateUtils.DATE_FORMATS_CEF));
+ } catch (java.text.ParseException e) {
+ throw new IllegalStateException("rt field present in CEF but cannot be parsed", e);
+ }
+ } else {
+ String logTimestamp = matcher.group("syslogTime");
+ if (!(logTimestamp == null || logTimestamp.isEmpty())) {
+ try {
+ obj.put("timestamp", SyslogUtils.parseTimestampToEpochMillis(logTimestamp, Clock.systemUTC()));
+ } catch (ParseException e) {
+ throw new IllegalStateException("Cannot parse syslog timestamp", e);
+ }
+ } else {
+ obj.put("timestamp", System.currentTimeMillis());
+ }
+ }
+
+ // add the host
+ String host = matcher.group("syslogHost");
+ if (!(host == null || host.isEmpty())) {
+ obj.put("host", host);
+ }
+
+ messages.add(obj);
+ }
+ return messages;
+ }
+
+ @SuppressWarnings("unchecked")
+ private JSONObject convertToInt(JSONObject obj, String key) {
+ if (obj.containsKey(key)) {
+ obj.put(key, Integer.valueOf((String) obj.get(key)));
+ }
+ return obj;
+ }
+
+ private void headerBlock(String name, StringBuilder sb) {
+ sb.append("(?<").append(name).append(">").append(HEADER_CAPTURE_PATTERN).append(")");
+ }
+
+ /**
+ * Maps string based severity in CEF format to integer.
+ *
+ * The strings are mapped according to the CEF 23 specification, taking the
+ * integer value as the value of the range buckets rounded up
+ *
+ * The valid string values are: Unknown, Low, Medium, High, and Very-High.
+ * The valid integer values are: 0-3=Low, 4-6=Medium, 7- 8=High, and
+ * 9-10=Very-High.
+ *
+ * @param severity
+ * String or Integer
+ * @return Integer value mapped from the string
+ */
+ private Integer standardizeSeverity(String severity) {
+ if (severity.length() < 3) {
+ // should be a number
+ return Integer.valueOf(severity);
+ } else {
+ switch (severity) {
+ case "Low":
+ return 2;
+ case "Medium":
+ return 5;
+ case "High":
+ return 8;
+ case "Very-High":
+ return 10;
+ default:
+ return 0;
+ }
+ }
+ }
+
+ @Override
+ public void configure(Map<String, Object> config) {
+ // TODO Auto-generated method stub
+
+ }
+
+ @SuppressWarnings("unchecked")
+ private JSONObject mutate(JSONObject json, String oldKey, String newKey) {
+ if (json.containsKey(oldKey)) {
+ json.put(newKey, json.remove(oldKey));
+ }
+ return json;
+ }
+
+}
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/9e15cb6e/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/utils/DateUtils.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/utils/DateUtils.java b/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/utils/DateUtils.java
new file mode 100644
index 0000000..888649a
--- /dev/null
+++ b/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/utils/DateUtils.java
@@ -0,0 +1,115 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.utils;
+
+import java.text.ParseException;
+import java.text.SimpleDateFormat;
+import java.time.ZonedDateTime;
+import java.util.ArrayList;
+import java.util.Calendar;
+import java.util.Date;
+import java.util.List;
+import java.util.TimeZone;
+import java.util.regex.Pattern;
+
+import org.apache.commons.lang.StringUtils;
+
+/**
+ * Various utilities for parsing and extracting dates
+ *
+ */
+public class DateUtils {
+
+ public static List<SimpleDateFormat> DATE_FORMATS_CEF = new ArrayList<SimpleDateFormat>() {
+ {
+ // as per CEF Spec
+ add(new SimpleDateFormat("MMM dd HH:mm:ss.SSS zzz"));
+ add(new SimpleDateFormat("MMM dd HH:mm:ss.SSS"));
+ add(new SimpleDateFormat("MMM dd HH:mm:ss zzz"));
+ add(new SimpleDateFormat("MMM dd HH:mm:ss"));
+ add(new SimpleDateFormat("MMM dd yyyy HH:mm:ss.SSS zzz"));
+ add(new SimpleDateFormat("MMM dd yyyy HH:mm:ss.SSS"));
+ add(new SimpleDateFormat("MMM dd yyyy HH:mm:ss zzz"));
+ add(new SimpleDateFormat("MMM dd yyyy HH:mm:ss"));
+ // found in the wild
+ add(new SimpleDateFormat("dd MMMM yyyy HH:mm:ss"));
+ }
+ };
+
+ public static List<SimpleDateFormat> DATE_FORMATS_SYSLOG = new ArrayList<SimpleDateFormat>() {
+ {
+ // As specified in https://tools.ietf.org/html/rfc5424
+ add(new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSSZ"));
+
+ // common format per rsyslog defaults e.g. Mar 21 14:05:02
+ add(new SimpleDateFormat("MMM dd HH:mm:ss"));
+ add(new SimpleDateFormat("MMM dd yyyy HH:mm:ss"));
+
+ // additional formats found in the wild
+ add(new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss"));
+ add(new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ssZ"));
+ add(new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS"));
+
+ }
+ };
+
+ Pattern NUMERIC = Pattern.compile("\\b\\d+\\b");
+
+ /**
+ * Parse the data according to a sequence of possible parse patterns.
+ *
+ * If the given date is entirely numeric, it is assumed to be a unix
+ * timestamp.
+ *
+ * If the year is not specified in the date string, use the current year.
+ * Assume that any date more than 4 days in the future is in the past as per
+ * SyslogUtils
+ *
+ * @param candidate
+ * The possible date.
+ * @param validPatterns
+ * A list of SimpleDateFormat instances to try parsing with.
+ * @return A java.util.Date based on the parse result
+ * @throws ParseException
+ */
+ public static long parseMultiformat(String candidate, List<SimpleDateFormat> validPatterns) throws ParseException {
+ if (StringUtils.isNumeric(candidate)) {
+ return Long.valueOf(candidate);
+ } else {
+ for (SimpleDateFormat pattern : validPatterns) {
+ try {
+ Calendar cal = Calendar.getInstance();
+ cal.setTime(pattern.parse(candidate));
+ Calendar current = Calendar.getInstance();
+ if (cal.get(Calendar.YEAR) == 1970) {
+ cal.set(Calendar.YEAR, current.get(Calendar.YEAR));
+ }
+ current.add(Calendar.DAY_OF_MONTH, 4);
+ if (cal.after(current)) {
+ cal.add(Calendar.YEAR, -1);
+ }
+ return cal.getTimeInMillis();
+ } catch (ParseException e) {
+ continue;
+ }
+ }
+ throw new ParseException("Failed to parse any of the given date formats", 0);
+ }
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/9e15cb6e/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/cef/CEFParserTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/cef/CEFParserTest.java b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/cef/CEFParserTest.java
new file mode 100644
index 0000000..88c0f0c
--- /dev/null
+++ b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/cef/CEFParserTest.java
@@ -0,0 +1,277 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.cef;
+
+import java.io.IOException;
+import java.net.URL;
+import java.nio.charset.Charset;
+import java.text.SimpleDateFormat;
+import java.util.Calendar;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+
+import org.json.simple.JSONObject;
+import org.json.simple.parser.JSONParser;
+import org.json.simple.parser.ParseException;
+import org.junit.Assert;
+import org.junit.Test;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.github.fge.jackson.JsonLoader;
+import com.github.fge.jsonschema.core.report.ProcessingReport;
+import com.github.fge.jsonschema.main.JsonSchemaFactory;
+import com.github.fge.jsonschema.main.JsonValidator;
+import com.google.common.io.Resources;
+
+import junit.framework.TestCase;
+
+public class CEFParserTest extends TestCase {
+
+ private static final Charset UTF_8 = Charset.forName("utf-8");
+ private CEFParser parser;
+
+ @Override
+ public void setUp() {
+ parser = new CEFParser();
+ parser.init();
+ }
+
+ @Test
+ public void testInvalid() {
+ List<JSONObject> obj = parse("test test test nonsense\n");
+ assertEquals(0, obj.size());
+ }
+
+ @Test
+ public void testEscaping() {
+ for (JSONObject obj : parse(
+ "Sep 19 08:26:10 host CEF:0|security|threatmanager|1.0|100|detected a \\ in packet|10|src=10.0.0.1 act=blocked a \\ dst=1.1.1.1")) {
+ assertEquals("10.0.0.1", obj.get("ip_src_addr"));
+ assertEquals("blocked a \\", obj.get("deviceAction"));
+ assertEquals("1.1.1.1", obj.get("ip_dst_addr"));
+ }
+ }
+
+ public void testBasicHeader() {
+ for (JSONObject obj : parse(
+ "CEF:0|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2 spt=1232")) {
+ assertEquals("Security", obj.get("DeviceVendor"));
+ assertEquals("threatmanager", obj.get("DeviceProduct"));
+ assertEquals("1.0", obj.get("DeviceVersion"));
+ assertEquals("100", obj.get("DeviceEvent"));
+ assertEquals("worm successfully stopped", obj.get("Name"));
+ assertEquals(10, obj.get("Severity"));
+ }
+ }
+
+ public void testBasicExtensions() {
+ for (JSONObject obj : parse(
+ "CEF:0|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2 spt=1232")) {
+ assertEquals("10.0.0.1", obj.get("ip_src_addr"));
+ assertEquals("2.1.2.2", obj.get("ip_dst_addr"));
+ assertEquals(1232, obj.get("ip_src_port"));
+ }
+ }
+
+ public void testCustomLabelWithSpace() {
+ for (JSONObject obj : parse(
+ "CEF:0|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2 spt=1232 custom=Text with space customLabel=Label with space")) {
+ assertEquals(true, obj.containsKey("Label with space"));
+ assertEquals("Text with space", obj.get("Label with space"));
+ }
+ }
+
+ public void testTimestampPriority() throws java.text.ParseException {
+ long correctTime = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSSz").parse("2016-05-01T09:29:11.356-0400")
+ .getTime();
+
+ SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSSz");
+
+ for (JSONObject obj : parse(
+ "CEF:0|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 rt=May 1 2016 09:29:11.356 -0400 dst=2.1.2.2 spt=1232")) {
+ assertEquals(new Date(correctTime), new Date((long) obj.get("timestamp")));
+ assertEquals(correctTime, obj.get("timestamp"));
+ }
+ for (JSONObject obj : parse(
+ "2016-06-01T09:29:11.356-04:00 host CEF:0|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 rt=May 1 2016 09:29:11.356 -0400 dst=2.1.2.2 spt=1232")) {
+ assertEquals(new Date(correctTime), new Date((long) obj.get("timestamp")));
+ assertEquals(correctTime, obj.get("timestamp"));
+ }
+ for (JSONObject obj : parse(
+ "2016-05-01T09:29:11.356-04:00 host CEF:0|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2 spt=1232")) {
+ assertEquals(new Date(correctTime), new Date((long) obj.get("timestamp")));
+ assertEquals(correctTime, obj.get("timestamp"));
+ }
+ for (JSONObject obj : parse(
+ "CEF:0|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2 spt=1232")) {
+ assertNotNull(obj.get("timestamp"));
+ }
+
+ }
+
+ public void testRtValueAsEpochTimestamp() throws java.text.ParseException {
+ long correctTime = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSSz").parse("2016-05-01T09:29:11.356-0400")
+ .getTime();
+ for (JSONObject obj : parse("CEF:0|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 rt="
+ + String.valueOf(correctTime) + " dst=2.1.2.2 spt=1232")) {
+ assertEquals(new Date(correctTime), new Date((long) obj.get("timestamp")));
+ assertEquals(correctTime, obj.get("timestamp"));
+ }
+ }
+
+ private void runMissingYear(Calendar expected, Calendar input) {
+ SimpleDateFormat sdf = new SimpleDateFormat("MMM dd HH:mm:ss.SSS");
+ for (JSONObject obj : parse("CEF:0|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 rt="
+ + sdf.format(input.getTime()) + " dst=2.1.2.2 spt=1232")) {
+ assertEquals(expected.getTimeInMillis(), obj.get("timestamp"));
+ assertEquals(expected.getTime(), new Date((long) obj.get("timestamp")));
+ }
+ }
+
+ public void testMissingYearFromDate() throws java.text.ParseException {
+ Calendar current = Calendar.getInstance();
+ Calendar correct = Calendar.getInstance();
+
+ correct.setTimeInMillis(current.getTimeInMillis());
+
+ runMissingYear(correct, current);
+ }
+
+ public void testFourDayFutureBecomesPast() {
+ Calendar current = Calendar.getInstance();
+ Calendar correct = Calendar.getInstance();
+
+ current.add(Calendar.DAY_OF_MONTH, 5);
+ // correct.setTime(current.getTime());
+ correct.setTimeInMillis(current.getTimeInMillis());
+ correct.add(Calendar.YEAR, -1);
+
+ runMissingYear(correct, current);
+ }
+
+ public void testCEFParserAdallom() throws Exception {
+ runTest("adallom", Resources.readLines(Resources.getResource(getClass(), "adallom.cef"), UTF_8),
+ Resources.toString(Resources.getResource(getClass(), "adallom.schema"), UTF_8));
+ }
+
+ public void testCEFParserCyberArk() throws Exception {
+ runTest("cyberark", Resources.readLines(Resources.getResource(getClass(), "cyberark.cef"), UTF_8),
+ Resources.toString(Resources.getResource(getClass(), "cyberark.schema"), UTF_8),
+ Resources.toString(Resources.getResource(getClass(), "cyberark.json"), UTF_8));
+ }
+
+ public void testCEFParserWAF() throws Exception {
+ URL waf_url = Resources.getResource(getClass(), "waf.cef");
+ runTest("waf", Resources.readLines(waf_url, UTF_8),
+ Resources.toString(Resources.getResource(getClass(), "waf.schema"), UTF_8));
+ }
+
+ private void runTest(String name, List<String> lines, String schema) throws Exception {
+ runTest(name, lines, schema, "");
+ }
+
+ private void runTest(String name, List<String> lines, String schema, String targetJson) throws Exception {
+ for (String inputString : lines) {
+ JSONObject parsed = parse(inputString).get(0);
+ assertNotNull(parsed);
+ assertNotNull(parsed.get("timestamp"));
+ assertTrue((long) parsed.get("timestamp") > 0);
+
+ System.out.println(parsed);
+ JSONParser parser = new JSONParser();
+
+ Map<?, ?> json = null;
+ try {
+ json = (Map<?, ?>) parser.parse(parsed.toJSONString());
+ Assert.assertEquals(true, validateJsonData(schema, json.toString()));
+ } catch (ParseException e) {
+ e.printStackTrace();
+ }
+
+ // test against an explicit json example
+ if (!targetJson.isEmpty()) {
+
+ }
+ }
+ }
+
+ /**
+ * Additional Sample from NiFi test Suite
+ * (https://github.com/apache/nifi/blob/rel/nifi-1.1.1/nifi-nar-bundles/nifi
+ * -standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/
+ * processors/standard/TestParseCEF.java)
+ */
+ private final static String sample = "CEF:0|TestVendor|TestProduct|TestVersion|TestEventClassID|TestName|Low|" +
+ // TimeStamp, String and Long
+ "rt=Feb 09 2015 00:27:43 UTC cn3Label=Test Long cn3=9223372036854775807 " +
+ // FloatPoint and MacAddress
+ "cfp1=1.234 cfp1Label=Test FP Number smac=00:00:0c:07:ac:00 " +
+ // IPv6 and String
+ "c6a3=2001:cdba::3257:9652 c6a3Label=Test IPv6 cs1Label=Test String cs1=test test test chocolate " +
+ // IPv4
+ "destinationTranslatedAddress=123.123.123.123 " +
+ // Date without TZ
+ "deviceCustomDate1=Feb 06 2015 13:27:43 " +
+ // Integer and IP Address (from v4)
+ "dpt=1234 agt=123.123.0.124 dlat=40.366633 " +
+ // A JSON object inside one of CEF's custom Strings
+ "cs2Label=JSON payload "
+ + "cs2={\"test_test_test\": \"chocolate!\", \"what?!?\": \"Simple! test test test chocolate!\"}";
+
+ @Test
+ public void testSuccessfulWhenCEFContainsJSON() throws JsonProcessingException, IOException {
+ List<JSONObject> parse = parse(sample);
+ JSONObject obj = parse.get(0);
+
+ assertEquals("TestVendor", obj.get("DeviceVendor"));
+ assertEquals(1423441663000L, obj.get("timestamp"));
+ assertEquals("9223372036854775807", obj.get("Test Long"));
+ assertEquals(obj.get("Test FP Number"), String.valueOf(1.234F));
+ assertEquals("00:00:0c:07:ac:00", obj.get("smac"));
+ assertEquals("2001:cdba::3257:9652", obj.get("Test IPv6"));
+ assertEquals("test test test chocolate", obj.get("Test String"));
+ assertEquals("123.123.123.123", obj.get("destinationTranslatedAddress"));
+
+ JsonNode inner = new ObjectMapper().readTree((String) obj.get("JSON payload"));
+ Assert.assertEquals("chocolate!", inner.get("test_test_test").asText());
+ }
+
+ protected boolean validateJsonData(final String jsonSchema, final String jsonData) throws Exception {
+ final JsonNode d = JsonLoader.fromString(jsonData);
+ final JsonNode s = JsonLoader.fromString(jsonSchema);
+
+ final JsonSchemaFactory factory = JsonSchemaFactory.byDefault();
+ JsonValidator v = factory.getValidator();
+
+ ProcessingReport report = v.validate(s, d);
+ System.out.println(report);
+
+ return report.toString().contains("success");
+ }
+
+ private List<JSONObject> parse(String string) {
+ List<JSONObject> parse = parser.parse(string.getBytes(Charset.forName("utf-8")));
+ assertNotNull(parse);
+ return parse;
+ }
+
+}
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/9e15cb6e/metron-platform/metron-parsers/src/test/resources/org/apache/metron/parsers/cef/adallom.cef
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/resources/org/apache/metron/parsers/cef/adallom.cef b/metron-platform/metron-parsers/src/test/resources/org/apache/metron/parsers/cef/adallom.cef
new file mode 100644
index 0000000..a35f354
--- /dev/null
+++ b/metron-platform/metron-parsers/src/test/resources/org/apache/metron/parsers/cef/adallom.cef
@@ -0,0 +1 @@
+2016-04-01T09:29:11.356-0400 CEF:0|Adallom|Adallom|1.0|56fe779ee4b0459f4e9a484a|ALERT_CABINET_EVENT_MATCH_AUDIT|0|msg=Activity policy 'User download/view file' was triggered by 'person@example.com' suser=auser@example.com start=1459517280810 end=1459517280810 audits=["AVPR-4oIPeFmuZ3CKKrg","AVPR-wx80cd9PUpAu2aj","AVPR-6XGPeFmuZ3CKKvx","AVPSALn_qE4Kgs_8_yK9","AVPSASW3gw_f3aEvgEmi"] services=["APPID_SXC"] users=["another@example.com"] cs6=https://abcd-remote.console.arc.com/#/alerts/56fe779ee4b0459f4e9a484a cs6Label=consoleUrl
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/9e15cb6e/metron-platform/metron-parsers/src/test/resources/org/apache/metron/parsers/cef/adallom.schema
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/resources/org/apache/metron/parsers/cef/adallom.schema b/metron-platform/metron-parsers/src/test/resources/org/apache/metron/parsers/cef/adallom.schema
new file mode 100644
index 0000000..a91cce0
--- /dev/null
+++ b/metron-platform/metron-parsers/src/test/resources/org/apache/metron/parsers/cef/adallom.schema
@@ -0,0 +1,37 @@
+{
+ "title": "Adallom Schema",
+ "type": "object",
+ "properties": {
+ "original_string": {
+ "type": "string"
+ },
+ "timestamp": {
+ "type": "integer"
+ },
+ "DeviceVendor": {
+ "type": "string"
+ },
+ "DeviceProduct": {
+ "type": "string"
+ },
+ "DeviceVersion": {
+ "type": "string"
+ },
+ "DeviceEvent": {
+ "type": "string"
+ },
+ "Name": {
+ "type": "string"
+ },
+ "Severity": {
+ "type": "integer"
+ },
+ "consoleUrl": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "original_string", "timestamp",
+ "DeviceVendor", "DeviceProduct", "DeviceVersion", "Name", "Severity",
+ "consoleUrl"]
+}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/9e15cb6e/metron-platform/metron-parsers/src/test/resources/org/apache/metron/parsers/cef/cyberark.cef
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/resources/org/apache/metron/parsers/cef/cyberark.cef b/metron-platform/metron-parsers/src/test/resources/org/apache/metron/parsers/cef/cyberark.cef
new file mode 100644
index 0000000..9d4fe6f
--- /dev/null
+++ b/metron-platform/metron-parsers/src/test/resources/org/apache/metron/parsers/cef/cyberark.cef
@@ -0,0 +1 @@
+Mar 21 14:05:02 HHHPVATN1 CEF:0|Cyber-Ark|Vault|7.20.0091|295|Retrieve password|5|act=Retrieve password suser=spilgrim fname=Root\ABC phobos3 - COMP dvc=120.99.70.3 shost=10.44.134.78 dhost= duser= externalId= app= reason= cs1Label="Affected User Name" cs1= cs2Label="Safe Name" cs2=Security Vulnerability Mgmt cs3Label="Device Type" cs3= cs4Label="Database" cs4= cs5Label="Other info" cs5=101.198.70.93 cn1Label="Request Id" cn1= cn2Label="Ticket Id" cn2=Needed to verify config files being pulled msg=Needed to verify config files being pulled
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/9e15cb6e/metron-platform/metron-parsers/src/test/resources/org/apache/metron/parsers/cef/cyberark.json
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/resources/org/apache/metron/parsers/cef/cyberark.json b/metron-platform/metron-parsers/src/test/resources/org/apache/metron/parsers/cef/cyberark.json
new file mode 100644
index 0000000..e900a9a
--- /dev/null
+++ b/metron-platform/metron-parsers/src/test/resources/org/apache/metron/parsers/cef/cyberark.json
@@ -0,0 +1,21 @@
+{
+ "\"Other info\"": "101.198.70.93",
+ "\"Safe Name\"": "Security Vulnerability Mgmt",
+ "\"Ticket Id\"": "Needed to verify config files being pulled ",
+ "deviceAction": "Retrieve password",
+ "deviceAddress": "120.99.70.3",
+ "device_product": "Vault",
+ "device_vendor": "Cyber-Ark",
+ "device_version": "7.20.0091",
+ "event_class_id": "295",
+ "event_name": "Retrieve password",
+ "fileName": "Root\\ABC phobos3 - COMP",
+ "header": "Mar 21 14:05:02 HHHPVATN1 CEF:0",
+ "message": "Needed to verify config files being pulled",
+ "original_string": "Mar 21 14:05:02 HHHPVATN1 CEF:0|Cyber-Ark|Vault|7.20.0091|295|Retrieve password|5|act=Retrieve password suser=spilgrim fname=Root\\ABC phobos3 - COMP dvc=120.99.70.3 shost=10.44.134.78 dhost= duser= externalId= app= reason= cs1Label=\"Affected User Name\" cs1= cs2Label=\"Safe Name\" cs2=Security Vulnerability Mgmt cs3Label=\"Device Type\" cs3= cs4Label=\"Database\" cs4= cs5Label=\"Other info\" cs5=101.198.70.93 cn1Label=\"Request Id\" cn1= cn2Label=\"Ticket Id\" cn2=Needed to verify config files being pulled msg=Needed to verify config files being pulled",
+ "severity": "5",
+ "source.type": "cyberark",
+ "src_hostname": "10.44.134.78",
+ "src_username": "spilgrim",
+ "timestamp": 1458569102000
+}
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/9e15cb6e/metron-platform/metron-parsers/src/test/resources/org/apache/metron/parsers/cef/cyberark.schema
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/resources/org/apache/metron/parsers/cef/cyberark.schema b/metron-platform/metron-parsers/src/test/resources/org/apache/metron/parsers/cef/cyberark.schema
new file mode 100644
index 0000000..5bd1021
--- /dev/null
+++ b/metron-platform/metron-parsers/src/test/resources/org/apache/metron/parsers/cef/cyberark.schema
@@ -0,0 +1,38 @@
+{
+ "title": "CyberArk Schema",
+ "type": "object",
+ "properties": {
+ "ip_src_addr": {
+ "type": "string"
+ },
+ "ip_dst_addr": {
+ "type": "string"
+ },
+ "original_string": {
+ "type": "string"
+ },
+ "timestamp": {
+ "type": "integer"
+ },
+ "DeviceVendor": {
+ "type": "string"
+ },
+ "DeviceProduct": {
+ "type": "string"
+ },
+ "DeviceVersion": {
+ "type": "string"
+ },
+ "DeviceEvent": {
+ "type": "string"
+ },
+ "Name": {
+ "type": "string"
+ },
+ "Severity": {
+ "type": "integer"
+ }
+ },
+ "required": ["original_string", "timestamp",
+ "DeviceVendor", "DeviceProduct", "DeviceVersion", "Name", "Severity"]
+}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/9e15cb6e/metron-platform/metron-parsers/src/test/resources/org/apache/metron/parsers/cef/waf.cef
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/resources/org/apache/metron/parsers/cef/waf.cef b/metron-platform/metron-parsers/src/test/resources/org/apache/metron/parsers/cef/waf.cef
new file mode 100644
index 0000000..86e1d6b
--- /dev/null
+++ b/metron-platform/metron-parsers/src/test/resources/org/apache/metron/parsers/cef/waf.cef
@@ -0,0 +1 @@
+<14>CEF:0|Imperva Inc.|SecureSphere|10.0.0.4_16|ABC - Secure Login.vm Page Rate Limit UK - Source IP||High|act=alert dst=17.43.200.42 dpt=88 duser=${Alert.username} src=10.31.45.69 spt=34435 proto=TCP rt=31 March 2016 13:04:55 cat=Alert cs1= cs1Label=Policy cs2=ABC-Secure cs2Label=ServerGroup cs3=servers_svc cs3Label=ServiceName cs4=server_app cs4Label=ApplicationName cs5=QA cs5Label=Description
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/9e15cb6e/metron-platform/metron-parsers/src/test/resources/org/apache/metron/parsers/cef/waf.schema
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/resources/org/apache/metron/parsers/cef/waf.schema b/metron-platform/metron-parsers/src/test/resources/org/apache/metron/parsers/cef/waf.schema
new file mode 100644
index 0000000..b38485c
--- /dev/null
+++ b/metron-platform/metron-parsers/src/test/resources/org/apache/metron/parsers/cef/waf.schema
@@ -0,0 +1,67 @@
+{
+ "title": "WAF CEF Schema",
+ "type": "object",
+ "properties": {
+ "ip_src_addr": {
+ "type": "string"
+ },
+ "ip_src_port": {
+ "type": "integer"
+ },
+ "ip_dst_addr": {
+ "type": "string"
+ },
+ "ip_dst_port": {
+ "type": "integer"
+ },
+ "original_string": {
+ "type": "string"
+ },
+ "@version": {
+ "type": "string"
+ },
+ "timestamp": {
+ "type": "integer"
+ },
+ "type": {
+ "type": "string"
+ },
+ "DeviceVendor": {
+ "type": "string"
+ },
+ "DeviceProduct": {
+ "type": "string"
+ },
+ "DeviceVersion": {
+ "type": "string"
+ },
+ "DeviceEvent": {
+ "type": "string"
+ },
+ "Name": {
+ "type": "string"
+ },
+ "Severity": {
+ "type": "integer"
+ },
+ "cat": {
+ "type": "string"
+ },
+ "ServerGroup": {
+ "type": "string"
+ },
+ "ServiceName": {
+ "type": "string"
+ },
+ "ApplicationName": {
+ "type": "string"
+ },
+ "Description": {
+ "type": "string"
+ }
+ },
+ "required": ["ip_src_addr", "ip_dst_addr", "ip_src_port", "ip_dst_port", "original_string", "timestamp",
+ "DeviceVendor", "DeviceProduct", "DeviceVersion", "Name", "Severity",
+ "cat",
+ "ServerGroup", "ServiceName", "ApplicationName", "Description"]
+}
\ No newline at end of file
[10/11] incubator-metron git commit: METRON-636 Capture memory and
cpu details as a part of platform-info script (anandsubbu via nickwallen)
closes apache/incubator-metron#400
Posted by ce...@apache.org.
METRON-636 Capture memory and cpu details as a part of platform-info script (anandsubbu via nickwallen) closes apache/incubator-metron#400
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/0e8abc48
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/0e8abc48
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/0e8abc48
Branch: refs/heads/Metron_0.3.1
Commit: 0e8abc48e5f6b29fb382371ad26bb165cf6e5f25
Parents: 9e15cb6
Author: anandsubbu <an...@gmail.com>
Authored: Wed Feb 22 09:01:12 2017 -0500
Committer: Nick Allen <ni...@nickallen.org>
Committed: Wed Feb 22 09:01:12 2017 -0500
----------------------------------------------------------------------
metron-deployment/scripts/platform-info.sh | 27 +++++++++++++++++++++++++
1 file changed, 27 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/0e8abc48/metron-deployment/scripts/platform-info.sh
----------------------------------------------------------------------
diff --git a/metron-deployment/scripts/platform-info.sh b/metron-deployment/scripts/platform-info.sh
index be53e86..79ebcac 100755
--- a/metron-deployment/scripts/platform-info.sh
+++ b/metron-deployment/scripts/platform-info.sh
@@ -62,3 +62,30 @@ mvn --version
# operating system
echo "--"
uname -a
+
+# system resources
+echo "--"
+case "${OSTYPE}" in
+ linux*)
+ cat /proc/meminfo | grep -i MemTotal | awk '{print "Total System Memory = " $2/1024 " MB"}'
+ cat /proc/cpuinfo | egrep 'model\ name' | uniq | cut -d: -f2 | awk '{print "Processor Model:" $0}'
+ cat /proc/cpuinfo | egrep 'cpu\ MHz' | uniq | cut -d: -f2 | awk '{print "Processor Speed:" $0 " MHz"}'
+ cat /proc/cpuinfo | grep -i '^processor' | wc -l | awk '{print "Total Physical Processors: " $0}'
+ cat /proc/cpuinfo | grep -i cores | cut -d: -f2 | awk '{corecount+=$1} END {print "Total cores: " corecount}'
+ echo "Disk information:"
+ df -h | grep "^/"
+ ;;
+ darwin*)
+ sysctl hw.memsize | awk '{print "Total System Memory = " $2/1048576 " MB"}'
+ sysctl machdep.cpu | grep 'machdep.cpu.brand_string' | cut -d: -f2 | cut -d\@ -f1 | awk '{print "Processor Model:" $0}'
+ sysctl machdep.cpu | grep 'machdep.cpu.brand_string' | cut -d: -f2 | cut -d\@ -f2 | awk '{print "Processor Speed:" $0}'
+ sysctl hw.physicalcpu | cut -d: -f2 | awk '{print "Total Physical Processors:" $0}'
+ sysctl machdep.cpu | grep 'machdep.cpu.core_count' | cut -d: -f2 | cut -d\@ -f2 | awk '{print "Total cores:" $0}'
+ echo "Disk information:"
+ df -h | grep "^/"
+ ;;
+ *)
+ echo "Unable to detect system resources for ${OSTYPE}"
+ ;;
+esac
+
[03/11] incubator-metron git commit: METRON-724 Account for `in`
grammar in Stellar Documentation and Unit Tests (ottobackwards) closes
apache/incubator-metron#457
Posted by ce...@apache.org.
METRON-724 Account for `in` grammar in Stellar Documentation and Unit Tests (ottobackwards) closes apache/incubator-metron#457
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/22ea8e32
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/22ea8e32
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/22ea8e32
Branch: refs/heads/Metron_0.3.1
Commit: 22ea8e3205c0f2bfe6c538dc2f10e90f92ff597c
Parents: 2964655
Author: ottobackwards <ot...@gmail.com>
Authored: Fri Feb 17 15:43:29 2017 -0500
Committer: Otto Fowler <ot...@apache.org>
Committed: Fri Feb 17 15:43:29 2017 -0500
----------------------------------------------------------------------
metron-platform/metron-common/README.md | 4 ++--
.../metron/common/stellar/StellarTest.java | 21 ++++++++++++++++++++
2 files changed, 23 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/22ea8e32/metron-platform/metron-common/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/README.md b/metron-platform/metron-common/README.md
index 872b320..067bf8d 100644
--- a/metron-platform/metron-common/README.md
+++ b/metron-platform/metron-common/README.md
@@ -35,9 +35,9 @@ The following keywords need to be single quote escaped in order to be used in St
| | | | | |
| :-----------: | :-----------: | :---------: | :---------: | :---------: |
| not | else | exists | if | then |
-| and | or | == | != | \< |
+| and | or | in | == | != |
| \<= | \> | \>= | \+ | \- |
-| ? | \* | / | , | |
+| \< | ? | \* | / | , |
Using parens such as: "foo" : "\<ok\>" requires escaping; "foo": "\'\<ok\>\'"
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/22ea8e32/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarTest.java b/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarTest.java
index dabf293..6ff3d8a 100644
--- a/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarTest.java
+++ b/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarTest.java
@@ -21,6 +21,7 @@ package org.apache.metron.common.stellar;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
import org.apache.commons.lang3.StringUtils;
+import org.apache.metron.common.dsl.ParseException;
import org.apache.metron.common.dsl.Stellar;
import org.apache.metron.common.dsl.StellarFunction;
import org.junit.Assert;
@@ -288,6 +289,26 @@ public class StellarTest {
}
@Test
+ public void testInNotIN(){
+ HashMap variables = new HashMap<>();
+ boolean thrown = false;
+ try{
+ run("in in ['','in']" ,variables );
+ }catch(ParseException pe) {
+ thrown = true;
+ }
+ Assert.assertTrue(thrown);
+ thrown = false;
+
+ try{
+ Assert.assertEquals(true,run("'in' in ['','in']" ,variables ));
+ }catch(ParseException pe) {
+ thrown = true;
+ }
+ Assert.assertFalse(thrown);
+ }
+
+ @Test
public void testHappyPath() {
String query = "TO_UPPER(TRIM(foo))";
Assert.assertEquals("CASEY", run(query, ImmutableMap.of("foo", "casey ")));
[07/11] incubator-metron git commit: METRON-715: Removed MySQL from
Enrichment Diagram closes apache/incubator-metron#452
Posted by ce...@apache.org.
METRON-715: Removed MySQL from Enrichment Diagram closes apache/incubator-metron#452
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/610146ef
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/610146ef
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/610146ef
Branch: refs/heads/Metron_0.3.1
Commit: 610146efb5f61cf7b7ea04982619a9efa14365ca
Parents: b7cd3ea
Author: Simon Elliston Ball <si...@simonellistonball.com>
Authored: Tue Feb 21 09:41:05 2017 -0500
Committer: cstella <ce...@gmail.com>
Committed: Tue Feb 21 09:41:05 2017 -0500
----------------------------------------------------------------------
.../metron-enrichment/enrichment_arch.png | Bin 181522 -> 113606 bytes
1 file changed, 0 insertions(+), 0 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/610146ef/metron-platform/metron-enrichment/enrichment_arch.png
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/enrichment_arch.png b/metron-platform/metron-enrichment/enrichment_arch.png
index 7bf4b3f..3b8bcdb 100644
Binary files a/metron-platform/metron-enrichment/enrichment_arch.png and b/metron-platform/metron-enrichment/enrichment_arch.png differ
[08/11] incubator-metron git commit: METRON-725 Javadoc is broken by
the use of apiNote (justinleet) closes apache/incubator-metron#458
Posted by ce...@apache.org.
METRON-725 Javadoc is broken by the use of apiNote (justinleet) closes apache/incubator-metron#458
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/246acff0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/246acff0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/246acff0
Branch: refs/heads/Metron_0.3.1
Commit: 246acff0f41f9c569687b1715f7c7115283c1f2a
Parents: 610146e
Author: justinleet <ju...@gmail.com>
Authored: Tue Feb 21 11:15:56 2017 -0500
Committer: leet <le...@apache.org>
Committed: Tue Feb 21 11:15:56 2017 -0500
----------------------------------------------------------------------
.../common/utils/file/ReaderSpliterator.java | 98 +-------------------
1 file changed, 1 insertion(+), 97 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/246acff0/metron-platform/metron-common/src/main/java/org/apache/metron/common/utils/file/ReaderSpliterator.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/utils/file/ReaderSpliterator.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/utils/file/ReaderSpliterator.java
index 20a40fa..9de61d4 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/utils/file/ReaderSpliterator.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/utils/file/ReaderSpliterator.java
@@ -65,18 +65,7 @@ public class ReaderSpliterator implements Spliterator<String> {
throw new IllegalStateException(e);
}
}
- /**
- * If a remaining element exists, performs the given action on it,
- * returning {@code true}; else returns {@code false}. If this
- * Spliterator is {@link #ORDERED} the action is performed on the
- * next element in encounter order. Exceptions thrown by the
- * action are relayed to the caller.
- *
- * @param action The action
- * @return {@code false} if no remaining elements existed
- * upon entry to this method, else {@code true}.
- * @throws NullPointerException if the specified action is null
- */
+
@Override
public boolean tryAdvance(Consumer<? super String> action) {
if (action == null) {
@@ -96,45 +85,6 @@ public class ReaderSpliterator implements Spliterator<String> {
}
}
- /**
- * If this spliterator can be partitioned, returns a Spliterator
- * covering elements, that will, upon return from this method, not
- * be covered by this Spliterator.
- * <p>
- * <p>If this Spliterator is {@link #ORDERED}, the returned Spliterator
- * must cover a strict prefix of the elements.
- * <p>
- * <p>Unless this Spliterator covers an infinite number of elements,
- * repeated calls to {@code trySplit()} must eventually return {@code null}.
- * Upon non-null return:
- * <ul>
- * <li>the value reported for {@code estimateSize()} before splitting,
- * must, after splitting, be greater than or equal to {@code estimateSize()}
- * for this and the returned Spliterator; and</li>
- * <li>if this Spliterator is {@code SUBSIZED}, then {@code estimateSize()}
- * for this spliterator before splitting must be equal to the sum of
- * {@code estimateSize()} for this and the returned Spliterator after
- * splitting.</li>
- * </ul>
- * <p>
- * <p>This method may return {@code null} for any reason,
- * including emptiness, inability to split after traversal has
- * commenced, data structure constraints, and efficiency
- * considerations.
- *
- * @return a {@code Spliterator} covering some portion of the
- * elements, or {@code null} if this spliterator cannot be split
- * @apiNote An ideal {@code trySplit} method efficiently (without
- * traversal) divides its elements exactly in half, allowing
- * balanced parallel computation. Many departures from this ideal
- * remain highly effective; for example, only approximately
- * splitting an approximately balanced tree, or for a tree in
- * which leaf nodes may contain either one or two elements,
- * failing to further split these nodes. However, large
- * deviations in balance and/or overly inefficient {@code
- * trySplit} mechanics typically result in poor parallel
- * performance.
- */
@Override
public Spliterator<String> trySplit() {
final ConsumerWithLookback holder = new ConsumerWithLookback();
@@ -150,52 +100,11 @@ public class ReaderSpliterator implements Spliterator<String> {
return spliterator(batch, 0, j, characteristics() | SIZED);
}
- /**
- * Returns an estimate of the number of elements that would be
- * encountered by a {@link #forEachRemaining} traversal, or returns {@link
- * Long#MAX_VALUE} if infinite, unknown, or too expensive to compute.
- * <p>
- * <p>If this Spliterator is {@link #SIZED} and has not yet been partially
- * traversed or split, or this Spliterator is {@link #SUBSIZED} and has
- * not yet been partially traversed, this estimate must be an accurate
- * count of elements that would be encountered by a complete traversal.
- * Otherwise, this estimate may be arbitrarily inaccurate, but must decrease
- * as specified across invocations of {@link #trySplit}.
- *
- * @return the estimated size, or {@code Long.MAX_VALUE} if infinite,
- * unknown, or too expensive to compute.
- * @apiNote Even an inexact estimate is often useful and inexpensive to compute.
- * For example, a sub-spliterator of an approximately balanced binary tree
- * may return a value that estimates the number of elements to be half of
- * that of its parent; if the root Spliterator does not maintain an
- * accurate count, it could estimate size to be the power of two
- * corresponding to its maximum depth.
- */
@Override
public long estimateSize() {
return Long.MAX_VALUE;
}
- /**
- * Returns a set of characteristics of this Spliterator and its
- * elements. The result is represented as ORed values from {@link
- * #ORDERED}, {@link #DISTINCT}, {@link #SORTED}, {@link #SIZED},
- * {@link #NONNULL}, {@link #IMMUTABLE}, {@link #CONCURRENT},
- * {@link #SUBSIZED}. Repeated calls to {@code characteristics()} on
- * a given spliterator, prior to or in-between calls to {@code trySplit},
- * should always return the same result.
- * <p>
- * <p>If a Spliterator reports an inconsistent set of
- * characteristics (either those returned from a single invocation
- * or across multiple invocations), no guarantees can be made
- * about any computation using this Spliterator.
- *
- * @return a representation of characteristics
- * @apiNote The characteristics of a given spliterator before splitting
- * may differ from the characteristics after splitting. For specific
- * examples see the characteristic values {@link #SIZED}, {@link #SUBSIZED}
- * and {@link #CONCURRENT}.
- */
@Override
public int characteristics() {
return characteristics;
@@ -203,11 +112,6 @@ public class ReaderSpliterator implements Spliterator<String> {
static class ConsumerWithLookback implements Consumer<String> {
String value;
- /**
- * Performs this operation on the given argument.
- *
- * @param string the input argument
- */
@Override
public void accept(String string) {
this.value = string;
[02/11] incubator-metron git commit: METRON-721 Add Github pull
request template to help submitters and reviewers (ottobackwards) closes
apache/incubator-metron#456
Posted by ce...@apache.org.
METRON-721 Add Github pull request template to help submitters and reviewers (ottobackwards) closes apache/incubator-metron#456
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/29646550
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/29646550
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/29646550
Branch: refs/heads/Metron_0.3.1
Commit: 29646550ce3b775668a4439b23677ad86a4dcf80
Parents: d7147e3
Author: ottobackwards <ot...@gmail.com>
Authored: Fri Feb 17 10:38:41 2017 -0500
Committer: Otto Fowler <ot...@apache.org>
Committed: Fri Feb 17 10:38:41 2017 -0500
----------------------------------------------------------------------
.github/PULL_REQUEST_TEMPLATE.md | 40 +++++++++++++++++++++++++++++++++++
1 file changed, 40 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/29646550/.github/PULL_REQUEST_TEMPLATE.md
----------------------------------------------------------------------
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
new file mode 100644
index 0000000..d86305a
--- /dev/null
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -0,0 +1,40 @@
+Thank you for submitting a contribution to Apache Metron (Incubating).
+Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
+Please refer also to our [Build Verification guildlines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
+
+
+In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check
+the following:
+
+### For all changes:
+- [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
+- [ ] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
+- [ ] Has your PR been rebased against the latest commit within the target branch (typically master)?
+
+
+### For code changes:
+- [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
+- [ ] Have you included steps or a guide to how the change may be verified and tested manually?
+- [ ] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via:
+
+```
+mvn -q clean integration-test install && build_utils/verify_licenses.sh
+```
+
+- [ ] Have you written or updated unit tests and or integration tests to verify your changes?
+- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
+- [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
+
+### For documentation related changes:
+- [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via site-book/target/site/index.html.
+
+```
+cd site-book
+bin/generate-md.sh
+mvn site:site
+
+```
+
+### Note:
+Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
+It is also recommened that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
[04/11] incubator-metron git commit: METRON-705: Parallelize the
build in travis to the extent that is obvious closes
apache/incubator-metron#444
Posted by ce...@apache.org.
METRON-705: Parallelize the build in travis to the extent that is obvious closes apache/incubator-metron#444
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/80b8aee6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/80b8aee6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/80b8aee6
Branch: refs/heads/Metron_0.3.1
Commit: 80b8aee6a47047ea84f99037977ec7a4717d15aa
Parents: 22ea8e3
Author: cstella <ce...@gmail.com>
Authored: Tue Feb 21 09:13:00 2017 -0500
Committer: cstella <ce...@gmail.com>
Committed: Tue Feb 21 09:13:00 2017 -0500
----------------------------------------------------------------------
.travis.yml | 7 ++++++-
README.md | 2 +-
pom.xml | 2 ++
3 files changed, 9 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/80b8aee6/.travis.yml
----------------------------------------------------------------------
diff --git a/.travis.yml b/.travis.yml
index 4f1c5e0..9e9f536 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -2,9 +2,14 @@ install: true
language: java
jdk:
- oraclejdk8
+before_install:
+ - wget https://archive.apache.org/dist/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.zip
+ - unzip -qq apache-maven-3.3.9-bin.zip
+ - export M2_HOME=$PWD/apache-maven-3.3.9
+ - export PATH=$M2_HOME/bin:$PATH
script:
- |
- mvn -q integration-test install && build_utils/verify_licenses.sh
+ time mvn -q -T 2C -DskipTests install && time mvn -q -T 2C surefire:test@unit-tests && mvn -q surefire:test@integration-tests && time build_utils/verify_licenses.sh
cache:
directories:
- $HOME/.m2
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/80b8aee6/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 419e9dd..52c6bd3 100644
--- a/README.md
+++ b/README.md
@@ -1,5 +1,5 @@
[](https://travis-ci.org/apache/incubator-metron)
-
+
# Apache Metron (Incubating)
Metron integrates a variety of open source big data technologies in order
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/80b8aee6/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index c57ce87..04c3954 100644
--- a/pom.xml
+++ b/pom.xml
@@ -237,6 +237,8 @@
<configuration>
<excludes>
<exclude>dependencies_with_url.csv</exclude>
+ <!-- In travis we need to pull down maven 3.3.9, so we should exclude it here as it is not our code. -->
+ <exclude>apache-maven-3.3.9/**</exclude>
<exclude>**/*.md</exclude>
<exclude>**/VERSION</exclude>
<exclude>**/*.json</exclude>
[05/11] incubator-metron git commit: METRON-730 Fix links to mailings
list on landing Apache Metron homepage (anandsubbu via cestella) closes
apache/incubator-metron#460
Posted by ce...@apache.org.
METRON-730 Fix links to mailings list on landing Apache Metron homepage (anandsubbu via cestella) closes apache/incubator-metron#460
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/a6299fd7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/a6299fd7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/a6299fd7
Branch: refs/heads/Metron_0.3.1
Commit: a6299fd75378cc8e29673c7f45d4b2c8e7b0437f
Parents: 80b8aee
Author: Anand Subramanian <an...@gmail.com>
Authored: Tue Feb 21 09:18:31 2017 -0500
Committer: cstella <ce...@gmail.com>
Committed: Tue Feb 21 09:18:31 2017 -0500
----------------------------------------------------------------------
site/community/index.md | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/a6299fd7/site/community/index.md
----------------------------------------------------------------------
diff --git a/site/community/index.md b/site/community/index.md
index 697b11f..057b2fe 100644
--- a/site/community/index.md
+++ b/site/community/index.md
@@ -109,13 +109,13 @@ title: Apache Metron (Incubating) Community
</div>
<div class="content-960 hover-btn text-center">
<h4> General & Public Discussion </h4>
- [<a href="mailto:general-subscribe@incubator.apache.org">Subscribe</a>]
- [<a href="mailto:general-unsubscribe@incubator.apache.org">Unsubscribe</a>]
- [<a href="http://mail-archives.apache.org/mod_mbox/incubator-general/">Archives</a>]
+ [<a href="mailto:user-subscribe@metron.incubator.apache.org">Subscribe</a>]
+ [<a href="mailto:user-unsubscribe@metron.incubator.apache.org">Unsubscribe</a>]
+ [<a href="http://mail-archives.apache.org/mod_mbox/incubator-metron-user/">Archives</a>]
<h4> Code & Documentation Change </h4>
- [<a href="mailto:cvs-subscribe@incubator.apache.org">Subscribe</a>]
- [<a href="mailto:cvs-unsubscribe@incubator.apache.org">Unsubscribe</a>]
- [<a href="http://mail-archives.apache.org/mod_mbox/incubator-cvs/">Archives</a>]
+ [<a href="mailto:dev-subscribe@metron.incubator.apache.org">Subscribe</a>]
+ [<a href="mailto:dev-unsubscribe@metron.incubator.apache.org">Unsubscribe</a>]
+ [<a href="http://mail-archives.apache.org/mod_mbox/incubator-metron-dev/">Archives</a>]
</div>
</section>