You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spot.apache.org by nl...@apache.org on 2017/06/12 17:20:29 UTC
[1/2] incubator-spot git commit: Removed --num-executors from
ml_ops.sh and replaced with dynamicAllocation.maxExecutors for maximum number
of executors and stop getting Spark warning about ignoring --num-executors.
Repository: incubator-spot
Updated Branches:
refs/heads/master 12dd9624e -> 5b1bb07e5
Removed --num-executors from ml_ops.sh and replaced with dynamicAllocation.maxExecutors for maximum number of executors and stop getting Spark warning about ignoring --num-executors.
Project: http://git-wip-us.apache.org/repos/asf/incubator-spot/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spot/commit/1499e84a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spot/tree/1499e84a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spot/diff/1499e84a
Branch: refs/heads/master
Commit: 1499e84a576451b57f2cee71a7a9d9770a72ea6f
Parents: c0f80c5
Author: Ricardo Barona <ri...@intel.com>
Authored: Wed May 31 16:37:46 2017 -0500
Committer: Ricardo Barona <ri...@intel.com>
Committed: Wed Jun 7 15:30:57 2017 -0500
----------------------------------------------------------------------
spot-ml/ml_ops.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spot/blob/1499e84a/spot-ml/ml_ops.sh
----------------------------------------------------------------------
diff --git a/spot-ml/ml_ops.sh b/spot-ml/ml_ops.sh
index 7bf98d7..a15e490 100755
--- a/spot-ml/ml_ops.sh
+++ b/spot-ml/ml_ops.sh
@@ -84,10 +84,10 @@ hdfs dfs -rm -R -f ${HDFS_SCORED_CONNECTS}
time spark-submit --class "org.apache.spot.SuspiciousConnects" \
--master yarn-client \
--driver-memory ${SPK_DRIVER_MEM} \
- --num-executors ${SPK_EXEC} \
--conf spark.driver.maxResultSize=${SPK_DRIVER_MAX_RESULTS} \
--conf spark.driver.maxPermSize=512m \
--conf spark.dynamicAllocation.enabled=true \
+ --conf spark.dynamicAllocation.maxExecutors=${SPK_EXEC} \
--conf spark.executor.cores=${SPK_EXEC_CORES} \
--conf spark.executor.memory=${SPK_EXEC_MEM} \
--conf "spark.executor.extraJavaOptions=-XX:MaxPermSize=512M -XX:PermSize=512M" \
[2/2] incubator-spot git commit: Merge branch 'master' into spot-136
Posted by nl...@apache.org.
Merge branch 'master' into spot-136
Project: http://git-wip-us.apache.org/repos/asf/incubator-spot/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spot/commit/5b1bb07e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spot/tree/5b1bb07e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spot/diff/5b1bb07e
Branch: refs/heads/master
Commit: 5b1bb07e5dd85ffcdfbca546849acb8d9ee23a80
Parents: 1499e84 12dd962
Author: nlsegerl <na...@intel.com>
Authored: Mon Jun 12 10:20:20 2017 -0700
Committer: nlsegerl <na...@intel.com>
Committed: Mon Jun 12 10:20:20 2017 -0700
----------------------------------------------------------------------
spot-ml/INSTALL.md | 2 +-
spot-ml/ML_OPS.md | 69 ++++++++++
spot-ml/README.md | 135 ++++---------------
spot-ml/SPOT-ML-JAR.md | 45 +++++++
spot-ml/SUSPICIOUS_CONNECTS_SCHEMA.md | 46 +++++++
spot-ml/ml_ops.sh | 5 +-
.../org/apache/spot/SuspiciousConnects.scala | 38 +++++-
.../dns/DNSSuspiciousConnectsAnalysis.scala | 92 ++++---------
.../dns/model/DNSSuspiciousConnectsModel.scala | 14 +-
.../FlowSuspiciousConnectsAnalysis.scala | 86 ++++--------
.../model/FlowSuspiciousConnectsModel.scala | 10 --
.../proxy/ProxySuspiciousConnectsAnalysis.scala | 78 +++--------
.../proxy/ProxySuspiciousConnectsModel.scala | 10 +-
.../dns/DNSSuspiciousConnectsAnalysisTest.scala | 34 ++---
.../FlowSuspiciousConnectsAnalysisTest.scala | 44 +++---
.../model/FlowSuspiciousConnectsModelTest.scala | 9 --
.../ProxySuspiciousConnectsAnalysisTest.scala | 32 ++---
.../spot/proxy/ProxyWordCreationTest.scala | 7 +-
18 files changed, 347 insertions(+), 409 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spot/blob/5b1bb07e/spot-ml/ml_ops.sh
----------------------------------------------------------------------