You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@nemo.apache.org by je...@apache.org on 2018/06/12 09:48:32 UTC
[incubator-nemo] branch nemo-63 updated (6122ff6 -> f643199)
This is an automated email from the ASF dual-hosted git repository.
jeongyoon pushed a change to branch nemo-63
in repository https://gitbox.apache.org/repos/asf/incubator-nemo.git.
from 6122ff6 address comments
add 4d3ef80 [NEMO-12] Frontend support for Scala Spark (#28)
new 565ae77 Merge branch 'master' of https://github.com/apache/incubator-nemo into nemo-63
new f643199 nits
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
compiler/frontend/spark/pom.xml | 11 +
.../spark/core/{java => }/JavaSparkContext.java | 3 +-
.../snu/nemo/compiler/frontend/spark/core/RDD.java | 70 --
.../compiler/frontend/spark/core/SparkContext.java | 64 ++
.../spark/core/{java => }/SparkFrontendUtils.java | 111 +++-
.../frontend/spark/core/java/JavaPairRDD.java | 119 ----
.../frontend/spark/core/rdd/JavaPairRDD.java | 707 +++++++++++++++++++++
.../frontend/spark/core/{java => rdd}/JavaRDD.java | 200 +++---
.../source/SparkDatasetBoundedSourceVertex.java | 4 +-
.../source/SparkTextFileBoundedSourceVertex.java | 1 +
.../nemo/compiler/frontend/spark/sql/Dataset.java | 29 +-
.../compiler/frontend/spark/sql/SparkSession.java | 37 +-
.../frontend/spark/transform/CollectTransform.java | 13 +-
.../spark/transform/HDFSTextFileTransform.java | 5 +-
.../spark/transform/LocalTextFileTransform.java | 5 +-
.../frontend/spark/transform/MapTransform.java | 1 +
.../frontend/spark/transform/ReduceTransform.java | 2 +-
.../frontend/spark/core/rdd/PairRDDFunctions.scala | 324 ++++++++++
.../compiler/frontend/spark/core/rdd/RDD.scala | 521 +++++++++++++++
.../edu/snu/nemo/examples/spark/JavaMapReduce.java | 8 +-
.../edu/snu/nemo/examples/spark/JavaSparkPi.java | 4 +-
.../edu/snu/nemo/examples/spark/JavaWordCount.java | 4 +-
.../examples/spark/sql/JavaSparkSQLExample.java | 2 +-
.../spark/{SparkPi.scala => SparkWordCount.scala} | 45 +-
.../{SparkITCase.java => SparkJavaITCase.java} | 5 +-
.../snu/nemo/examples/spark/SparkScalaITCase.java} | 69 +-
26 files changed, 1967 insertions(+), 397 deletions(-)
rename compiler/frontend/spark/src/main/java/edu/snu/nemo/compiler/frontend/spark/core/{java => }/JavaSparkContext.java (95%)
delete mode 100644 compiler/frontend/spark/src/main/java/edu/snu/nemo/compiler/frontend/spark/core/RDD.java
create mode 100644 compiler/frontend/spark/src/main/java/edu/snu/nemo/compiler/frontend/spark/core/SparkContext.java
rename compiler/frontend/spark/src/main/java/edu/snu/nemo/compiler/frontend/spark/core/{java => }/SparkFrontendUtils.java (54%)
delete mode 100644 compiler/frontend/spark/src/main/java/edu/snu/nemo/compiler/frontend/spark/core/java/JavaPairRDD.java
create mode 100644 compiler/frontend/spark/src/main/java/edu/snu/nemo/compiler/frontend/spark/core/rdd/JavaPairRDD.java
rename compiler/frontend/spark/src/main/java/edu/snu/nemo/compiler/frontend/spark/core/{java => rdd}/JavaRDD.java (74%)
create mode 100644 compiler/frontend/spark/src/main/scala/edu/snu/nemo/compiler/frontend/spark/core/rdd/PairRDDFunctions.scala
create mode 100644 compiler/frontend/spark/src/main/scala/edu/snu/nemo/compiler/frontend/spark/core/rdd/RDD.scala
copy examples/spark/src/main/scala/edu/snu/nemo/examples/spark/{SparkPi.scala => SparkWordCount.scala} (56%)
rename examples/spark/src/test/java/edu/snu/nemo/examples/spark/{SparkITCase.java => SparkJavaITCase.java} (98%)
copy examples/{beam/src/test/java/edu/snu/nemo/examples/beam/BroadcastITCase.java => spark/src/test/java/edu/snu/nemo/examples/spark/SparkScalaITCase.java} (53%)
--
To stop receiving notification emails like this one, please contact
jeongyoon@apache.org.
[incubator-nemo] 01/02: Merge branch 'master' of
https://github.com/apache/incubator-nemo into nemo-63
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jeongyoon pushed a commit to branch nemo-63
in repository https://gitbox.apache.org/repos/asf/incubator-nemo.git
commit 565ae770e9710e61c50343b41c2e37da215ad66c
Merge: 6122ff6 4d3ef80
Author: Jeongyoon Eo <je...@gmail.com>
AuthorDate: Tue Jun 12 18:37:49 2018 +0900
Merge branch 'master' of https://github.com/apache/incubator-nemo into nemo-63
compiler/frontend/spark/pom.xml | 11 +
.../spark/core/{java => }/JavaSparkContext.java | 3 +-
.../snu/nemo/compiler/frontend/spark/core/RDD.java | 70 --
.../compiler/frontend/spark/core/SparkContext.java | 64 ++
.../spark/core/{java => }/SparkFrontendUtils.java | 111 +++-
.../frontend/spark/core/java/JavaPairRDD.java | 119 ----
.../frontend/spark/core/rdd/JavaPairRDD.java | 707 +++++++++++++++++++++
.../frontend/spark/core/{java => rdd}/JavaRDD.java | 200 +++---
.../source/SparkDatasetBoundedSourceVertex.java | 4 +-
.../source/SparkTextFileBoundedSourceVertex.java | 1 +
.../nemo/compiler/frontend/spark/sql/Dataset.java | 29 +-
.../compiler/frontend/spark/sql/SparkSession.java | 37 +-
.../frontend/spark/transform/CollectTransform.java | 13 +-
.../spark/transform/HDFSTextFileTransform.java | 5 +-
.../spark/transform/LocalTextFileTransform.java | 5 +-
.../frontend/spark/transform/MapTransform.java | 1 +
.../frontend/spark/transform/ReduceTransform.java | 2 +-
.../frontend/spark/core/rdd/PairRDDFunctions.scala | 324 ++++++++++
.../compiler/frontend/spark/core/rdd/RDD.scala | 521 +++++++++++++++
.../edu/snu/nemo/examples/spark/JavaMapReduce.java | 8 +-
.../edu/snu/nemo/examples/spark/JavaSparkPi.java | 4 +-
.../edu/snu/nemo/examples/spark/JavaWordCount.java | 4 +-
.../examples/spark/sql/JavaSparkSQLExample.java | 2 +-
.../snu/nemo/examples/spark/SparkWordCount.scala | 60 ++
.../{SparkITCase.java => SparkJavaITCase.java} | 5 +-
.../snu/nemo/examples/spark/SparkScalaITCase.java | 80 +++
26 files changed, 2043 insertions(+), 347 deletions(-)
--
To stop receiving notification emails like this one, please contact
jeongyoon@apache.org.
[incubator-nemo] 02/02: nits
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jeongyoon pushed a commit to branch nemo-63
in repository https://gitbox.apache.org/repos/asf/incubator-nemo.git
commit f6431998e41460857d23cac416339a855a42a105
Author: Jeongyoon Eo <je...@gmail.com>
AuthorDate: Tue Jun 12 18:48:16 2018 +0900
nits
---
.../test/java/edu/snu/nemo/examples/spark/SparkScalaITCase.java | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/examples/spark/src/test/java/edu/snu/nemo/examples/spark/SparkScalaITCase.java b/examples/spark/src/test/java/edu/snu/nemo/examples/spark/SparkScalaITCase.java
index 59b0508..5541e58 100644
--- a/examples/spark/src/test/java/edu/snu/nemo/examples/spark/SparkScalaITCase.java
+++ b/examples/spark/src/test/java/edu/snu/nemo/examples/spark/SparkScalaITCase.java
@@ -58,9 +58,9 @@ public final class SparkScalaITCase {
@Test(timeout = TIMEOUT)
public void testWordCount() throws Exception {
- final String inputFileName = "sample_input_wordcount";
- final String outputFileName = "sample_output_wordcount";
- final String testResourceFilename = "test_output_wordcount";
+ final String inputFileName = "sample_input_wordcount_spark";
+ final String outputFileName = "sample_output_wordcount_spark";
+ final String testResourceFilename = "test_output_wordcount_spark";
final String inputFilePath = fileBasePath + inputFileName;
final String outputFilePath = fileBasePath + outputFileName;
@@ -77,4 +77,4 @@ public final class SparkScalaITCase {
ExampleTestUtil.deleteOutputFile(fileBasePath, outputFileName);
}
}
-}
\ No newline at end of file
+}
--
To stop receiving notification emails like this one, please contact
jeongyoon@apache.org.