You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gearpump.apache.org by Roshanson <gi...@git.apache.org> on 2016/08/26 10:10:02 UTC
[GitHub] incubator-gearpump pull request #78: refactor example sources task to use Da...
GitHub user Roshanson opened a pull request:
https://github.com/apache/incubator-gearpump/pull/78
refactor example sources task to use DataSourceAPI
Refactor example sources task to use DataSourceAPI.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Roshanson/incubator-gearpump fix-192
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-gearpump/pull/78.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #78
----
commit e70cb18ec8dff8e59448853e32cb9acbf1696c81
Author: Roshanson <73...@qq.com>
Date: 2016-08-26T07:08:07Z
refactor example sources task to use DataSourceAPI
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request #78: [GEARPUMP-192] refactor example sources...
Posted by huafengw <gi...@git.apache.org>.
Github user huafengw commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/78#discussion_r77100932
--- Diff: examples/streaming/wordcount/src/main/scala/org/apache/gearpump/streaming/examples/wordcount/Split.scala ---
@@ -23,29 +23,34 @@ import java.util.concurrent.TimeUnit
import org.apache.gearpump.Message
import org.apache.gearpump.cluster.UserConfig
-import org.apache.gearpump.streaming.source.Watermark
+import org.apache.gearpump.streaming.source.{DataSource, DataSourceTask, Watermark}
import org.apache.gearpump.streaming.task.{Task, TaskContext}
-class Split(taskContext: TaskContext, conf: UserConfig) extends Task(taskContext, conf) {
- import taskContext.output
- override def onStart(startTime: Instant): Unit = {
- self ! Watermark(Instant.now)
- }
+class Split() extends DataSource {
--- End diff --
I prefer `class Split extends DataSource`, no braces.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump issue #78: refactor example sources task to use DataSourc...
Posted by manuzhang <gi...@git.apache.org>.
Github user manuzhang commented on the issue:
https://github.com/apache/incubator-gearpump/pull/78
also, please fix the UT failure
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request #78: [GEARPUMP-192] refactor example sources...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-gearpump/pull/78
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump issue #78: [GEARPUMP-192] refactor example sources task t...
Posted by codecov-io <gi...@git.apache.org>.
Github user codecov-io commented on the issue:
https://github.com/apache/incubator-gearpump/pull/78
## [Current coverage](https://codecov.io/gh/apache/incubator-gearpump/pull/78?src=pr) is 69.63% (diff: 100%)
> Merging [#78](https://codecov.io/gh/apache/incubator-gearpump/pull/78?src=pr) into [master](https://codecov.io/gh/apache/incubator-gearpump/branch/master?src=pr) will increase coverage by **1.02%**
```diff
@@ master #78 diff @@
==========================================
Files 177 177
Lines 5927 5927
Methods 5604 5606 +2
Messages 0 0
Branches 323 321 -2
==========================================
+ Hits 4066 4127 +61
+ Misses 1861 1800 -61
Partials 0 0
```
> Powered by [Codecov](https://codecov.io?src=pr). Last update [529799c...d8f0838](https://codecov.io/gh/apache/incubator-gearpump/compare/529799cc400a72ae9e0d2385044ce1fd5e329bb3...d8f0838ed3e31b844db46e122df56658a0a698a9?src=pr)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump issue #78: [GEARPUMP-192] refactor example sources task t...
Posted by manuzhang <gi...@git.apache.org>.
Github user manuzhang commented on the issue:
https://github.com/apache/incubator-gearpump/pull/78
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump issue #78: refactor example sources task to use DataSourc...
Posted by manuzhang <gi...@git.apache.org>.
Github user manuzhang commented on the issue:
https://github.com/apache/incubator-gearpump/pull/78
please modify the commit comment to the format "[GEARPUMP-XXX] xyz"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request #78: [GEARPUMP-192] refactor example sources...
Posted by manuzhang <gi...@git.apache.org>.
Github user manuzhang commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/78#discussion_r77342296
--- Diff: examples/streaming/wordcount/src/main/scala/org/apache/gearpump/streaming/examples/wordcount/Split.scala ---
@@ -19,34 +19,45 @@
package org.apache.gearpump.streaming.examples.wordcount
import java.time.Instant
-import java.util.concurrent.TimeUnit
import org.apache.gearpump.Message
-import org.apache.gearpump.cluster.UserConfig
-import org.apache.gearpump.streaming.source.{DataSource, DataSourceTask, Watermark}
-import org.apache.gearpump.streaming.task.{Task, TaskContext}
+import org.apache.gearpump.streaming.source.DataSource
+import org.apache.gearpump.streaming.task.TaskContext
+import scala.collection.mutable.ArrayBuffer
-class Split() extends DataSource {
+class Split extends DataSource {
+
+ val result = ArrayBuffer[Message]()
+ var item = -1
+ Split.TEXT_TO_SPLIT.lines.foreach { line =>
+ line.split("[\\s]+").filter(_.nonEmpty).foreach { msg => // => \u4e3a\u533f\u540d\u51fd\u6570,\u4f20\u5165\u4e00\u4e2amsg\u53c2\u6570\u6267\u884c\u53f3\u8fb9\u7684\u64cd\u4f5c
--- End diff --
please remove this comment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request #78: [GEARPUMP-192] refactor example sources...
Posted by Roshanson <gi...@git.apache.org>.
Github user Roshanson commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/78#discussion_r77278370
--- Diff: examples/streaming/wordcount/src/main/scala/org/apache/gearpump/streaming/examples/wordcount/Split.scala ---
@@ -23,29 +23,34 @@ import java.util.concurrent.TimeUnit
import org.apache.gearpump.Message
import org.apache.gearpump.cluster.UserConfig
-import org.apache.gearpump.streaming.source.Watermark
+import org.apache.gearpump.streaming.source.{DataSource, DataSourceTask, Watermark}
import org.apache.gearpump.streaming.task.{Task, TaskContext}
-class Split(taskContext: TaskContext, conf: UserConfig) extends Task(taskContext, conf) {
- import taskContext.output
- override def onStart(startTime: Instant): Unit = {
- self ! Watermark(Instant.now)
- }
+class Split() extends DataSource {
+
+
+ override def open(context: TaskContext, startTime: Instant): Unit = {}
- override def onNext(msg: Message): Unit = {
+
+ override def read(): Message = {
Split.TEXT_TO_SPLIT.lines.foreach { line =>
line.split("[\\s]+").filter(_.nonEmpty).foreach { msg =>
- output(new Message(msg, System.currentTimeMillis()))
+ new Message(msg, System.currentTimeMillis())
}
}
-
- import scala.concurrent.duration._
- taskContext.scheduleOnce(Duration(100, TimeUnit.MILLISECONDS))(self !
- Watermark(Instant.now))
+ Message("message")
}
+
+ override def close(): Unit = {}
+
+ override def getWatermark: Instant = Instant.now()
+
+ Watermark(Instant.now)
--- End diff --
Thanks\uff0ci'm learning how gearpump is working now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request #78: [GEARPUMP-192] refactor example sources...
Posted by huafengw <gi...@git.apache.org>.
Github user huafengw commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/78#discussion_r77101042
--- Diff: examples/streaming/wordcount/src/main/scala/org/apache/gearpump/streaming/examples/wordcount/Split.scala ---
@@ -23,29 +23,34 @@ import java.util.concurrent.TimeUnit
import org.apache.gearpump.Message
import org.apache.gearpump.cluster.UserConfig
-import org.apache.gearpump.streaming.source.Watermark
+import org.apache.gearpump.streaming.source.{DataSource, DataSourceTask, Watermark}
import org.apache.gearpump.streaming.task.{Task, TaskContext}
-class Split(taskContext: TaskContext, conf: UserConfig) extends Task(taskContext, conf) {
- import taskContext.output
- override def onStart(startTime: Instant): Unit = {
- self ! Watermark(Instant.now)
- }
+class Split() extends DataSource {
+
+
+ override def open(context: TaskContext, startTime: Instant): Unit = {}
- override def onNext(msg: Message): Unit = {
+
+ override def read(): Message = {
Split.TEXT_TO_SPLIT.lines.foreach { line =>
line.split("[\\s]+").filter(_.nonEmpty).foreach { msg =>
- output(new Message(msg, System.currentTimeMillis()))
+ new Message(msg, System.currentTimeMillis())
}
}
-
- import scala.concurrent.duration._
- taskContext.scheduleOnce(Duration(100, TimeUnit.MILLISECONDS))(self !
- Watermark(Instant.now))
+ Message("message")
}
+
+ override def close(): Unit = {}
+
+ override def getWatermark: Instant = Instant.now()
+
+ Watermark(Instant.now)
--- End diff --
What's this line of code used for?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---