You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gearpump.apache.org by whjiang <gi...@git.apache.org> on 2016/04/27 11:11:17 UTC
[GitHub] incubator-gearpump pull request: fix GEARPUMP-109: add wordcount J...
GitHub user whjiang opened a pull request:
https://github.com/apache/incubator-gearpump/pull/6
fix GEARPUMP-109: add wordcount Java/Scala and DSL/low level API examples
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/whjiang/incubator-gearpump gearpump-109
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-gearpump/pull/6.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6
----
commit 0ac81302ee8f307b5f10b1946f938de184057974
Author: Weihua Jiang <wh...@outlook.com>
Date: 2016-04-27T09:08:04Z
fix GEARPUMP-109: add wordcount Java/Scala and DSL/low level API examples.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request: fix GEARPUMP-109: add wordcount J...
Posted by manuzhang <gi...@git.apache.org>.
Github user manuzhang commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/6#discussion_r61362813
--- Diff: docs/dev-write-1st-app.md ---
@@ -55,13 +163,58 @@ object Split {
}
```
-Like Split, every processor extends a `TaskActor`. The `onStart` method is called once before any message comes in; `onNext` method is called to process every incoming message. Note that Gearpump employs the message-driven model and that's why Split sends itself a message at the end of `onStart` and `onNext` to trigger next message processing.
+</div>
+
+<div data-lang="java" markdown="1">
+```java
+public class Split extends Task {
+
+ public static String TEXT = "This is a good start for java! bingo! bingo! ";
+
+ public Split(TaskContext taskContext, UserConfig userConf) {
+ super(taskContext, userConf);
+ }
+
+ private Long now() {
+ return System.currentTimeMillis();
+ }
+
+ @Override
+ public void onStart(StartTime startTime) {
+ self().tell(new Message("start", now()), self());
+ }
+
+ @Override
+ public void onNext(Message msg) {
+
+ // Split the TEXT to words
+ String[] words = TEXT.split(" ");
+ for (int i = 0; i < words.length; i++) {
+ context.output(new Message(words[i], now()));
+ }
+ self().tell(new Message("next", now()), self());
+ }
+}
+```
+
+</div>
+
+</div>
+
+Essentially, each processor consists of two descriptions:
+
+1. A `Task` to define the operation.
+
+2. A parallelism level to define the parallelism of this processor. I.e. this processor will have how many tasks running in parallel.
--- End diff --
=> "A parallelism level to define the number of tasks of this processor to run in parallel."
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request: fix GEARPUMP-109: add wordcount J...
Posted by manuzhang <gi...@git.apache.org>.
Github user manuzhang commented on the pull request:
https://github.com/apache/incubator-gearpump/pull/6#issuecomment-215279151
honestly, the title "write your 1st gearpump app" is not as straightforward as "examples". What do you think ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request: fix GEARPUMP-109: add wordcount J...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-gearpump/pull/6
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request: fix GEARPUMP-109: add wordcount J...
Posted by whjiang <gi...@git.apache.org>.
Github user whjiang commented on the pull request:
https://github.com/apache/incubator-gearpump/pull/6#issuecomment-215281967
For the name "write your 1st Gearpump app", I prefer it over "example".
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request: fix GEARPUMP-109: add wordcount J...
Posted by whjiang <gi...@git.apache.org>.
Github user whjiang commented on the pull request:
https://github.com/apache/incubator-gearpump/pull/6#issuecomment-215280345
Hi @manuzhang,
1. for the "/download.html" link issue, I have fixed in PR #2.
2. for the rest, I will try to fix it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request: fix GEARPUMP-109: add wordcount J...
Posted by manuzhang <gi...@git.apache.org>.
Github user manuzhang commented on the pull request:
https://github.com/apache/incubator-gearpump/pull/6#issuecomment-215278640
could you help to fix this ?
```
- _site/gearpump-internals.html
* linking to internal hash #what-happen-when-executor-crash that does not exist (line 252)
- _site/get-gearpump-distribution.html
* internally linking to /download.html, which does not exist (line 144)
htmlproof 2.6.1 | Error: HTML-Proofer found 2 failures!
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request: fix GEARPUMP-109: add wordcount J...
Posted by whjiang <gi...@git.apache.org>.
Github user whjiang commented on the pull request:
https://github.com/apache/incubator-gearpump/pull/6#issuecomment-215295543
I don't think the failure is related to this PR as my PR is only document related.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request: fix GEARPUMP-109: add wordcount J...
Posted by manuzhang <gi...@git.apache.org>.
Github user manuzhang commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/6#discussion_r61362610
--- Diff: docs/dev-write-1st-app.md ---
@@ -14,23 +14,131 @@ Repository and library dependencies can be found at [Maven Setting](maven-settin
### IDE Setup (Optional)
You can get your preferred IDE ready for Gearpump by following [this guide](dev-ide-setup.html).
-### Define Processor(Task) class and Partitioner class
+### Decide which language and API to use for writing
+Gearpump supports two level APIs:
-An application is a Directed Acyclic Graph (DAG) of processors. In the wordcount example, We will firstly define two processors `Split` and `Sum`, and then weave them together.
+1. Low level API, which is more similar to Akka programming, operating on each event. The API document can be found at [Low Level API Doc](http://www.gearpump.io/releases/latest/api/scala/index.html#io.gearpump.streaming.package).
+
+2. High level API (aka DSL), which is operating on streaming instead of individual event. The API document can be found at [DSL API Doc](http://www.gearpump.io/releases/latest/api/scala/index.html#io.gearpump.streaming.dsl.package).
+
+And both APIs have their Java version and Scala version.
+
+So, before you writing our first Gearpump application, you need to decide which API to use and which language to use.
+
+## DSL version for Wordcount
+
+The easiest way to write your streaming application is to write it with Gearpump DSL.
+Below will demostrate how to write WordCount application via Gearpump DSL.
+
+
+<div class="codetabs">
+<div data-lang="scala" markdown="1" >
+
+
+```scala
+/** WordCount with High level DSL */
+object WordCount extends AkkaApp with ArgumentsParser {
+
+ override val options: Array[(String, CLIOption[Any])] = Array.empty
-#### About message type
+ override def main(akkaConf: Config, args: Array[String]): Unit = {
+ val context = ClientContext(akkaConf)
+ val app = StreamApp("dsl", context)
+ val data = "This is a good start, bingo!! bingo!!"
-User are allowed to send message of type Any(Accept any type except Null, Nothing and Unit).
+ //count for each word and output to log
+ app.source(data.lines.toList, 1, "source").
+ // word => (word, count)
+ flatMap(line => line.split("[\\s]+")).map((_, 1)).
+ // (word, count1), (word, count2) => (word, count1 + count2)
+ groupByKey().sum.log
+ val appId = context.submit(app)
+ context.close()
+ }
+}
```
-case class Message(msg: Any, timestamp: TimeStamp = Message.noTimeStamp)
+
+</div>
+
+<div data-lang="java" markdown="1">
+
+```java
+
+/** Java version of WordCount with high level DSL API */
+public class WordCount {
+
+ public static void main(String[] args) throws InterruptedException {
+ main(ClusterConfig.defaultConfig(), args);
+ }
+
+ public static void main(Config akkaConf, String[] args) throws InterruptedException {
+ ClientContext context = new ClientContext(akkaConf);
+ JavaStreamApp app = new JavaStreamApp("JavaDSL", context, UserConfig.empty());
+ List<String> source = Lists.newArrayList("This is a good start, bingo!! bingo!!");
+
+ //create a stream from the string list.
+ JavaStream<String> sentence = app.source(source, 1, UserConfig.empty(), "source");
+
+ //tokenize the strings and create a new stream
+ JavaStream<String> words = sentence.flatMap(new FlatMapFunction<String, String>() {
+ @Override
+ public Iterator<String> apply(String s) {
+ return Lists.newArrayList(s.split("\\s+")).iterator();
+ }
+ }, "flatMap");
+
+ //map each string as (string, 1) pair
+ JavaStream<Tuple2<String, Integer>> ones = words.map(new MapFunction<String, Tuple2<String, Integer>>() {
+ @Override
+ public Tuple2<String, Integer> apply(String s) {
+ return new Tuple2<String, Integer>(s, 1);
+ }
+ }, "map");
+
+ //group by according to string
+ JavaStream<Tuple2<String, Integer>> groupedOnes = ones.groupBy(new GroupByFunction<Tuple2<String, Integer>, String>() {
+ @Override
+ public String apply(Tuple2<String, Integer> tuple) {
+ return tuple._1();
+ }
+ }, 1, "groupBy");
+
+ //for each group, make the sum
+ JavaStream<Tuple2<String, Integer>> wordcount = groupedOnes.reduce(new ReduceFunction<Tuple2<String, Integer>>() {
+ @Override
+ public Tuple2<String, Integer> apply(Tuple2<String, Integer> t1, Tuple2<String, Integer> t2) {
+ return new Tuple2<String, Integer>(t1._1(), t1._2() + t2._2());
+ }
+ }, "reduce");
+
+ //output result using log
+ wordcount.log();
+
+ app.run();
+ context.close();
+ }
+}
```
+</div>
+
+</div>
+
+## Low level API based Wordcount
+
+### Define Processor(Task) class and Partitioner class
+
+An application is a Directed Acyclic Graph (DAG) of processors. In the wordcount example, We will firstly define two processors `Split` and `Sum`, and then weave them together.
+
+
#### Split processor
-In the Split processor, we simply split a predefined text (the content is simplified for conciseness) and send out each split word to Sum.
+In the `Split` processor, we simply split a predefined text (the content is simplified for conciseness) and send out each split word to Sum.
--- End diff --
"Sum" => "`Sum`"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request: fix GEARPUMP-109: add wordcount J...
Posted by manuzhang <gi...@git.apache.org>.
Github user manuzhang commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/6#discussion_r61362383
--- Diff: docs/dev-write-1st-app.md ---
@@ -14,23 +14,131 @@ Repository and library dependencies can be found at [Maven Setting](maven-settin
### IDE Setup (Optional)
You can get your preferred IDE ready for Gearpump by following [this guide](dev-ide-setup.html).
-### Define Processor(Task) class and Partitioner class
+### Decide which language and API to use for writing
+Gearpump supports two level APIs:
-An application is a Directed Acyclic Graph (DAG) of processors. In the wordcount example, We will firstly define two processors `Split` and `Sum`, and then weave them together.
+1. Low level API, which is more similar to Akka programming, operating on each event. The API document can be found at [Low Level API Doc](http://www.gearpump.io/releases/latest/api/scala/index.html#io.gearpump.streaming.package).
+
+2. High level API (aka DSL), which is operating on streaming instead of individual event. The API document can be found at [DSL API Doc](http://www.gearpump.io/releases/latest/api/scala/index.html#io.gearpump.streaming.dsl.package).
+
+And both APIs have their Java version and Scala version.
+
+So, before you writing our first Gearpump application, you need to decide which API to use and which language to use.
--- End diff --
"you writing our first" => "you write your first"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request: fix GEARPUMP-109: add wordcount J...
Posted by whjiang <gi...@git.apache.org>.
Github user whjiang commented on the pull request:
https://github.com/apache/incubator-gearpump/pull/6#issuecomment-215610837
rebase done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request: fix GEARPUMP-109: add wordcount J...
Posted by manuzhang <gi...@git.apache.org>.
Github user manuzhang commented on the pull request:
https://github.com/apache/incubator-gearpump/pull/6#issuecomment-215605674
please rebase
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request: fix GEARPUMP-109: add wordcount J...
Posted by codecov-io <gi...@git.apache.org>.
Github user codecov-io commented on the pull request:
https://github.com/apache/incubator-gearpump/pull/6#issuecomment-215284423
## [Current coverage][cc-pull] is **51.97%**
> Merging [#6][cc-pull] into [master][cc-base-branch] will increase coverage by **+0.60%**
```diff
@@ master #6 diff @@
==========================================
Files 562 562
Lines 9285 9285
Methods 8551 8551
Messages 0 0
Branches 408 408
==========================================
+ Hits 4768 4825 +57
+ Misses 4517 4460 -57
Partials 0 0
```
1. 2 files (not in diff) in `...ts/distributeservice` were modified. [more](https://codecov.io/gh/apache/incubator-gearpump/commit/caec5b9ddd044ba8f8213dfd994150cba967c755/changes?src=pr#6578616D706C65732F64697374726962757465736572766963652F7372632F6D61696E2F7363616C612F696F2F6765617270756D702F6578706572696D656E74732F6469737472696275746573657276696365)
- Misses `+9`
- Hits `-9`
1. 1 files (not in diff) in `examples` were modified. [more](https://codecov.io/gh/apache/incubator-gearpump/commit/caec5b9ddd044ba8f8213dfd994150cba967c755/changes?src=pr#6578616D706C6573)
- Misses `+6`
- Hits `-6`
1. 4 files (not in diff) in `...ala/io/gearpump/util` were modified. [more](https://codecov.io/gh/apache/incubator-gearpump/commit/caec5b9ddd044ba8f8213dfd994150cba967c755/changes?src=pr#636F72652F7372632F6D61696E2F7363616C612F696F2F6765617270756D702F7574696C)
- Misses `-16`
- Hits `+16`
1. 2 files (not in diff) in `...rpump/cluster/client` were modified. [more](https://codecov.io/gh/apache/incubator-gearpump/commit/caec5b9ddd044ba8f8213dfd994150cba967c755/changes?src=pr#636F72652F7372632F6D61696E2F7363616C612F696F2F6765617270756D702F636C75737465722F636C69656E74)
- Misses `-57`
- Hits `+57`
1. 6 files (not in diff) in `.../io/gearpump/cluster` were modified. [more](https://codecov.io/gh/apache/incubator-gearpump/commit/caec5b9ddd044ba8f8213dfd994150cba967c755/changes?src=pr#636F72652F7372632F6D61696E2F7363616C612F696F2F6765617270756D702F636C7573746572)
- Misses `-19`
- Hits `+19`
1. 1 files (not in diff) in `...in/scala/io/gearpump` were modified. [more](https://codecov.io/gh/apache/incubator-gearpump/commit/caec5b9ddd044ba8f8213dfd994150cba967c755/changes?src=pr#636F72652F7372632F6D61696E2F7363616C612F696F2F6765617270756D70)
- Misses `+13`
- Hits `-13`
1. File `.../worker/Worker.scala` (not in diff) was modified. [more](https://codecov.io/gh/apache/incubator-gearpump/commit/caec5b9ddd044ba8f8213dfd994150cba967c755/changes?src=pr#6461656D6F6E2F7372632F6D61696E2F7363616C612F696F2F6765617270756D702F636C75737465722F776F726B65722F576F726B65722E7363616C61)
- Misses `+7`
- Partials `0`
- Hits `-7`
> Powered by [Codecov](https://codecov.io?src=pr). Last updated by caec5b9
[cc-base-branch]: https://codecov.io/gh/apache/incubator-gearpump/branch/master?src=pr
[cc-pull]: https://codecov.io/gh/apache/incubator-gearpump/pull/6?src=pr
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request: fix GEARPUMP-109: add wordcount J...
Posted by manuzhang <gi...@git.apache.org>.
Github user manuzhang commented on the pull request:
https://github.com/apache/incubator-gearpump/pull/6#issuecomment-215301310
LGTM + 1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---