You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@amaterasu.apache.org by ya...@apache.org on 2018/06/09 04:24:12 UTC
[incubator-amaterasu] branch master updated (a3f3240 -> 74a1b9c)
This is an automated email from the ASF dual-hosted git repository.
yaniv pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-amaterasu.git.
from a3f3240 Merge pull request #23 from apache/version-0.2.0-incubating-rc3
new b56bf06 Merge tag 'version-0.2.0-incubating-rc3'
new 733084d Merge branch 'master' of github.com:apache/incubator-amaterasu
new 74a1b9c added checking for multiline comments in scala
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../apache/spark/repl/amaterasu/runners/spark/SparkScalaRunner.scala | 3 ++-
.../test/scala/org/apache/amaterasu/spark/SparkScalaRunnerTests.scala | 1 -
2 files changed, 2 insertions(+), 2 deletions(-)
--
To stop receiving notification emails like this one, please contact
yaniv@apache.org.
[incubator-amaterasu] 01/03: Merge tag
'version-0.2.0-incubating-rc3'
Posted by ya...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yaniv pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-amaterasu.git
commit b56bf06428bb0bf7af399c1dd01b8bb0bb242984
Merge: 80c7ea4 ec6525a
Author: Yaniv Rodenski <ro...@gmail.com>
AuthorDate: Sat Jun 9 13:56:18 2018 +1000
Merge tag 'version-0.2.0-incubating-rc3'
.gitignore | 15 +++
.travis.yml | 20 ++-
LICENSE | 48 ++++----
ama-key | 51 --------
ama-key.pub | 1 -
build.gradle | 20 ++-
.../common/configuration/ClusterConfig.scala | 22 ++--
.../common/dataobjects/ActionDataHelper.scala | 16 +++
.../apache/amaterasu/common/utils/FileUtils.scala | 16 +++
executor/build.gradle | 1 +
executor/src/main/resources/runtime.py | 16 +++
executor/src/main/resources/spark_intp.py | 21 +++-
.../runners/spark/PySpark/PySparkRunner.scala | 135 ++++++++-------------
.../runners/spark/SparkRunnersProvider.scala | 28 +++--
.../executor/yarn/executors/ActionsExecutor.scala | 16 +++
.../runners/spark/SparkRunnerHelper.scala | 6 +-
executor/src/test/resources/amaterasu.properties | 1 +
.../src/test/resources/pyspark-with-amacontext.py | 37 ++++++
executor/src/test/resources/runtime.py | 34 +++++-
executor/src/test/resources/simple-pyspark.py | 16 +++
executor/src/test/resources/simple-python-err.py | 16 +++
executor/src/test/resources/simple-python.py | 16 +++
executor/src/test/resources/simple-spark.scala | 16 +++
executor/src/test/resources/spark_intp.py | 64 +++++++---
executor/src/test/resources/step-2.scala | 16 +++
.../amaterasu/spark/SparkScalaRunnerTests.scala | 16 +++
leader/build.gradle | 9 +-
.../org/apache/amaterasu/leader/yarn/Client.java | 14 ++-
.../frameworks/FrameworkProvidersFactory.scala | 16 +++
.../frameworks/spark/SparkSetupProvider.scala | 35 ++++--
.../amaterasu/leader/mesos/MesosJobLauncher.scala | 16 +++
.../leader/mesos/schedulers/JobScheduler.scala | 28 +++--
.../leader/utilities/ActiveReportListener.scala | 16 +++
.../apache/amaterasu/leader/utilities/Args.scala | 17 ++-
.../leader/utilities/BaseJobLauncher.scala | 16 +++
.../leader/utilities/MemoryFormatParser.scala | 16 +++
.../amaterasu/leader/yarn/ApplicationMaster.scala | 65 +++++-----
.../leader/yarn/YarnRMCallbackHandler.scala | 2 +-
leader/src/main/scripts/ama-start-yarn.sh | 24 +++-
leader/src/main/scripts/amaterasu.properties | 16 ++-
leader/src/main/scripts/log4j.properties | 15 +++
.../sdk/frameworks/FrameworkSetupProvider.java | 16 +++
.../configuration/DriverConfiguration.java | 16 +++
43 files changed, 731 insertions(+), 270 deletions(-)
--
To stop receiving notification emails like this one, please contact
yaniv@apache.org.
[incubator-amaterasu] 03/03: added checking for multiline comments
in scala
Posted by ya...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yaniv pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-amaterasu.git
commit 74a1b9c6b95f838f31140e0bc552e85da34c5e77
Author: Yaniv Rodenski <ro...@gmail.com>
AuthorDate: Sat Jun 9 14:23:57 2018 +1000
added checking for multiline comments in scala
---
.../apache/spark/repl/amaterasu/runners/spark/SparkScalaRunner.scala | 3 ++-
.../test/scala/org/apache/amaterasu/spark/SparkScalaRunnerTests.scala | 1 -
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/executor/src/main/scala/org/apache/spark/repl/amaterasu/runners/spark/SparkScalaRunner.scala b/executor/src/main/scala/org/apache/spark/repl/amaterasu/runners/spark/SparkScalaRunner.scala
index a45b8c0..56a04cf 100755
--- a/executor/src/main/scala/org/apache/spark/repl/amaterasu/runners/spark/SparkScalaRunner.scala
+++ b/executor/src/main/scala/org/apache/spark/repl/amaterasu/runners/spark/SparkScalaRunner.scala
@@ -61,7 +61,8 @@ class SparkScalaRunner(var env: Environment,
for (line <- source.getLines()) {
- if (!line.isEmpty) {
+ // ignoring empty or commented lines
+ if (!line.isEmpty && !line.trim.startsWith("*") && !line.startsWith("/")) {
outStream.reset()
log.debug(line)
diff --git a/executor/src/test/scala/org/apache/amaterasu/spark/SparkScalaRunnerTests.scala b/executor/src/test/scala/org/apache/amaterasu/spark/SparkScalaRunnerTests.scala
index 9575205..1d79fc9 100755
--- a/executor/src/test/scala/org/apache/amaterasu/spark/SparkScalaRunnerTests.scala
+++ b/executor/src/test/scala/org/apache/amaterasu/spark/SparkScalaRunnerTests.scala
@@ -33,7 +33,6 @@ class SparkScalaRunnerTests extends FlatSpec with Matchers with BeforeAndAfterAl
"SparkScalaRunner" should "execute the simple-spark.scala" in {
-
val sparkRunner =factory.getRunner("spark", "scala").get.asInstanceOf[SparkScalaRunner]
val script = getClass.getResource("/simple-spark.scala").getPath
val sourceCode = Source.fromFile(script).getLines().mkString("\n")
--
To stop receiving notification emails like this one, please contact
yaniv@apache.org.
[incubator-amaterasu] 02/03: Merge branch 'master' of
github.com:apache/incubator-amaterasu
Posted by ya...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yaniv pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-amaterasu.git
commit 733084d33723b0f82870c9a91fd4b6ad95715c28
Merge: b56bf06 a3f3240
Author: Yaniv Rodenski <ro...@gmail.com>
AuthorDate: Sat Jun 9 14:10:40 2018 +1000
Merge branch 'master' of github.com:apache/incubator-amaterasu
executor/src/test/resources/simple-spark.scala | 7 +-
executor/src/test/resources/step-2.scala | 4 +-
...c370548d-1371-4c08-81d6-f2ba72defa12.gz.parquet | Bin 0 -> 583 bytes
...c370548d-1371-4c08-81d6-f2ba72defa12.gz.parquet | Bin 0 -> 574 bytes
.../amaterasu/spark/SparkScalaRunnerTests.scala | 94 +++++++++------------
.../apache/amaterasu/spark/SparkTestsSuite.scala | 6 +-
6 files changed, 49 insertions(+), 62 deletions(-)
--
To stop receiving notification emails like this one, please contact
yaniv@apache.org.