You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gearpump.apache.org by Roshanson <gi...@git.apache.org> on 2016/11/02 08:45:26 UTC
[GitHub] incubator-gearpump pull request #107: Add hbase-example
GitHub user Roshanson opened a pull request:
https://github.com/apache/incubator-gearpump/pull/107
Add hbase-example
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Roshanson/incubator-gearpump HBase
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-gearpump/pull/107.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #107
----
commit d2eed3a6ebc673898e04cae2c1b3f8a4c364d8c8
Author: Roshanson <73...@qq.com>
Date: 2016-11-02T02:01:46Z
Squashed commit of the following:
commit 68d9027656fd7470c6389cb85ee63181fe64d0f5
Author: Roshanson <73...@qq.com>
Date: Mon Oct 31 15:13:38 2016 +0800
Add connecting to HBase example
commit 8a46511c0ded3d5ec8198ce844246255e935b153
Merge: 02b9bac 7bc5fba
Author: Roshanson <73...@qq.com>
Date: Mon Oct 31 14:30:10 2016 +0800
Merge branch 'master' of https://github.com/apache/incubator-gearpump into InHBase
commit 02b9bac0766f40c7547343084412cf3aef930821
Author: Roshanson <73...@qq.com>
Date: Mon Oct 31 13:50:24 2016 +0800
Add example of connecting to HBase
commit 0e1df27b4e78797ced14690980c9066189af7afc
Author: Roshanson <73...@qq.com>
Date: Mon Oct 31 13:12:32 2016 +0800
Add example of connecting to HBase
commit b611ef7d2a73bd121c7b3dc38b25ff17c43ae04c
Author: Roshanson <73...@qq.com>
Date: Mon Oct 31 13:01:19 2016 +0800
Add example of connection to HBase
commit 6b6f2eee612ce0fe3d5cb5dd25dd1cffd4bc0519
Author: Roshanson <73...@qq.com>
Date: Fri Oct 21 15:55:16 2016 +0800
[GEARPUMP-204]INSERT TO HBASE
commit ac1482db04639b9e7101140a8907e988f40c8f92
Merge: 65e13bd c812097
Author: Roshanson <73...@qq.com>
Date: Tue Sep 27 20:26:14 2016 +0800
Merge remote-tracking branch 'origin/fix-addHBaseUT' into fix-hbasetest
# Conflicts:
# examples/streaming/wordcount/src/main/scala/org/apache/gearpump/streaming/examples/wordcount/Split.scala
# examples/streaming/wordcount/src/main/scala/org/apache/gearpump/streaming/examples/wordcount/WordCount.scala
# external/hbase/src/main/scala/org/apache/gearpump/external/hbase/HBaseSink.scala
# external/hbase/src/test/scala/org/apache/gearpump/external/hbase/HBaseSinkSpec.scala
commit c8120978b8609b41012823707ff04b830f1dfb50
Author: Roshanson <73...@qq.com>
Date: Tue Sep 27 19:54:50 2016 +0800
[GEARPUMP-204]Add unit test for external_hbase module
commit 501dca4861549e27beeb9a6ef89f4bb9902a3249
Author: Roshanson <73...@qq.com>
Date: Tue Sep 27 18:53:25 2016 +0800
[GEARPUMP-204]Add unit test for external_hbase module
commit d52f51da4ccb797c7ef954b396b88735a2bf969e
Author: Roshanson <73...@qq.com>
Date: Fri Sep 23 11:33:33 2016 +0800
[GEARPUMP-204]add unit test for external_hbase module
commit 52d4b8d6ff4e1720147b9c088412625f5312f0a0
Author: Roshanson <doyouta123>
Date: Fri Sep 23 10:41:37 2016 +0800
[GEARPUMP-204]add unit test for external_hbase module
commit 1caf488e88253380c01e319d59957f36694784a3
Author: Roshanson <73...@qq.com>
Date: Thu Sep 22 19:41:31 2016 +0800
[GEARPUMP-204]Add unit test for external_hbase module
commit 2e6ea2c33fdacddff283ea25d93493d1ef30bb4d
Author: Roshanson <doyouta123>
Date: Thu Sep 22 16:49:29 2016 +0800
[GEARPUMP-204]add unit test for external_hbase module
commit 4920f2598721beede24e182205a87b72b9f13afb
Author: roshanson <doyouta123>
Date: Thu Sep 22 16:46:08 2016 +0800
[GEARPUMP-204]add unit test for external_hbase module
commit f07f7feba571ba4a2ba830f8ae794b9dcd8f1e54
Author: roshanson <doyouta123>
Date: Thu Sep 22 15:16:31 2016 +0800
[GEARPUMP-204]add unit test for external_hbase module
commit ff7b930a6d2cd8081bdf2be1f5c2d69c2d144fd8
Author: roshanson <doyouta123>
Date: Thu Sep 22 14:59:19 2016 +0800
[GEARPUMP-204]add unit test for external_hbase module
commit 3086e4535fbbaa1c6e652f8e3cc3fdeb7bf24ead
Author: roshanson <doyouta123>
Date: Thu Sep 22 13:12:49 2016 +0800
[GEARPUMP-204]add unit test for external_hbase module
commit 87364c049fd283c5da6898abf61973eae031a8a7
Author: roshanson <doyouta123>
Date: Thu Sep 22 11:36:25 2016 +0800
[GEARPUMP-204]add unit test for external_hbase module
commit 8ef7dbb0d9b816a595441e66b0f71a041fc5b9f8
Author: roshanson <doyouta123>
Date: Wed Sep 21 13:57:24 2016 +0800
[GEARPUMP-204]add unit test for external_hbase module
commit a14df0d10d03219800a6b1a166ec9d773cfa76b3
Author: roshanson <doyouta123>
Date: Wed Sep 21 12:33:57 2016 +0800
[GEARPUMP-204]add unit test for external_hbase module
commit 99257d02e54bd0fa7ad530487aa1b2282964ca85
Author: roshanson <doyouta123>
Date: Mon Sep 19 17:00:49 2016 +0800
connection to hbase
commit 0097e2527783114a7d8890eca7173e5fb55a6194
Author: Roshanson <73...@qq.com>
Date: Tue Sep 13 21:30:57 2016 +0800
[GEARPUMP-204]Add unit test for external_hbase module
commit 57c7e8817ba9c8683039e050a217904d065f7a13
Author: 736781877@qq.com <doyouta123>
Date: Fri Sep 2 20:31:28 2016 +0800
refactor example sources task to use DataSourceAPI
commit 5db634a5f6cfa2ff198a46019bb8a4fa6b63cfeb
Author: 736781877@qq.com <doyouta123>
Date: Fri Sep 2 20:26:48 2016 +0800
refactor example sources task to use DataSourceAPI
commit bc5c21e52fcdc46f1f4b6b350d4351bc2e1134f7
Author: Roshanson <73...@qq.com>
Date: Fri Sep 2 19:17:33 2016 +0800
refactor example sources task to use DataSourceAPI
commit 65e13bde0d728da62396ac59c73d01b3c5dc1320
Author: roshanson <doyouta123>
Date: Mon Sep 19 17:00:49 2016 +0800
connection to hbase
commit ee7d2acdb50686e781b31c283bf651eeab64b155
Author: Roshanson <73...@qq.com>
Date: Tue Sep 13 21:30:57 2016 +0800
[GEARPUMP-204]Add unit test for external_hbase module
commit 84e884c42196a091d170a175600a91a4858d8ad4
Author: Roshanson <73...@qq.com>
Date: Fri Sep 2 21:25:43 2016 +0800
[GEARPUMP-192] Refactor example sources task to use DataSourceAPI
commit f4c6737198cc0a0a9a1a4ae42b6aece1e6a4eb3c
Author: 736781877@qq.com <doyouta123>
Date: Fri Sep 2 21:16:16 2016 +0800
[GEARPUMP-192] Refactor example sources task to use DataSourceAPI
commit 7191000bd4a97edb6575b092265ca886a61e07b0
Author: 736781877@qq.com <doyouta123>
Date: Fri Sep 2 20:31:28 2016 +0800
refactor example sources task to use DataSourceAPI
commit bc9a2ee818278abc7b0b4bd8b3a42026f3905236
Author: 736781877@qq.com <doyouta123>
Date: Fri Sep 2 20:26:48 2016 +0800
refactor example sources task to use DataSourceAPI
commit d8f0838ed3e31b844db46e122df56658a0a698a9
Author: 736781877@qq.com <doyouta123>
Date: Fri Sep 2 20:24:30 2016 +0800
refactor example sources task to use DataSourceAPI
commit 4ff8453c494773b39acadceb9848aecbdd432859
Author: Roshanson <73...@qq.com>
Date: Fri Sep 2 19:17:33 2016 +0800
refactor example sources task to use DataSourceAPI
commit ff3d1401c307d661307138ff05c29c9f03bc3961
Author: Roshanson <73...@qq.com>
Date: Mon Aug 29 19:32:16 2016 +0800
refactor example sources task to use DataSourceAPI
commit e70cb18ec8dff8e59448853e32cb9acbf1696c81
Author: Roshanson <73...@qq.com>
Date: Fri Aug 26 15:08:07 2016 +0800
refactor example sources task to use DataSourceAPI
commit 384acfbb02b8a0def6acf81fc9e1d0ce889e2d4f
Author: Roshanson <73...@qq.com>
Date: 2016-11-02T08:41:02Z
Add hbase-example
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request #107: [GEARPUMP-233]Add hbase-example
Posted by manuzhang <gi...@git.apache.org>.
Github user manuzhang commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/107#discussion_r87349458
--- Diff: examples/streaming/hbase/src/main/scala/org/apache/gearpump/streaming/examples/hbase/HBaseConn.scala ---
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gearpump.streaming.examples.hbase
+
+import akka.actor.ActorSystem
+import org.apache.gearpump.cluster.UserConfig
+import org.apache.gearpump.cluster.client.ClientContext
+import org.apache.gearpump.cluster.main.{ArgumentsParser, CLIOption, ParseResult}
+import org.apache.gearpump.external.hbase.HBaseSink
+import org.apache.gearpump.streaming.StreamApplication
+import org.apache.gearpump.streaming.partitioner.HashPartitioner
+import org.apache.gearpump.streaming.sink.DataSinkProcessor
+import org.apache.gearpump.streaming.source.DataSourceProcessor
+import org.apache.gearpump.util.Graph.Node
+import org.apache.gearpump.util.{AkkaApp, Graph, LogUtil}
+import org.slf4j.Logger
+
+object HBaseConn extends AkkaApp with ArgumentsParser {
+ private val LOG: Logger = LogUtil.getLogger(getClass)
+ val RUN_FOR_EVER = -1
+
+ override val options: Array[(String, CLIOption[Any])] = Array(
+ "splitNum" -> CLIOption[Int]("<how many sum tasks>", required = false, defaultValue = Some(1)),
+ "sinkNum" -> CLIOption[Int]("<how many sum tasks>", required = false, defaultValue = Some(1))
+ )
+
+ def application(config: ParseResult, system: ActorSystem): StreamApplication = {
+ implicit val actorSystem = system
+
+ val splitNum = config.getInt("splitNum")
+ val sinkNum = config.getInt("sinkNum")
+
+ val split = new Split
+ val sourceProcessor = DataSourceProcessor(split, splitNum, "Split")
+ val sink = HBaseSink(UserConfig.empty, "intel")
--- End diff --
please do not use company name for the table name
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump issue #107: [GEARPUMP-233]Add hbase-example
Posted by manuzhang <gi...@git.apache.org>.
Github user manuzhang commented on the issue:
https://github.com/apache/incubator-gearpump/pull/107
@huafengw what do you think of this ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump issue #107: [GEARPUMP-233]Add hbase-example
Posted by huafengw <gi...@git.apache.org>.
Github user huafengw commented on the issue:
https://github.com/apache/incubator-gearpump/pull/107
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump issue #107: Add hbase-example
Posted by codecov-io <gi...@git.apache.org>.
Github user codecov-io commented on the issue:
https://github.com/apache/incubator-gearpump/pull/107
## [Current coverage](https://codecov.io/gh/apache/incubator-gearpump/pull/107?src=pr) is 71.69% (diff: 100%)
> Merging [#107](https://codecov.io/gh/apache/incubator-gearpump/pull/107?src=pr) into [master](https://codecov.io/gh/apache/incubator-gearpump/branch/master?src=pr) will increase coverage by **0.03%**
```diff
@@ master #107 diff @@
==========================================
Files 186 186
Lines 5985 5985
Methods 5456 5456
Messages 0 0
Branches 529 529
==========================================
+ Hits 4289 4291 +2
+ Misses 1696 1694 -2
Partials 0 0
```
> Powered by [Codecov](https://codecov.io?src=pr). Last update [7bc5fba...384acfb](https://codecov.io/gh/apache/incubator-gearpump/compare/7bc5fba5b5c5166992e04218071a3da4931e47a9...384acfbb02b8a0def6acf81fc9e1d0ce889e2d4f?src=pr)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request #107: Add hbase-example
Posted by huafengw <gi...@git.apache.org>.
Github user huafengw commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/107#discussion_r86284053
--- Diff: examples/streaming/hbase/src/main/scala/org/apache/gearpump/streaming/hbase/HBaseConn.scala ---
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gearpump.streaming.hbase
+
+
+import akka.actor.ActorSystem
+import org.apache.gearpump.cluster.UserConfig
+import org.apache.gearpump.cluster.client.ClientContext
+import org.apache.gearpump.cluster.main.{ArgumentsParser, CLIOption, ParseResult}
+import org.apache.gearpump.streaming.StreamApplication
+import org.apache.gearpump.streaming.partitioner.HashPartitioner
+import org.apache.gearpump.streaming.sink.DataSinkProcessor
+import org.apache.gearpump.streaming.source.DataSourceProcessor
+import org.apache.gearpump.util.Graph.Node
+import org.apache.gearpump.util.{AkkaApp, Graph, LogUtil}
+import org.slf4j.Logger
+
+object HBaseConn extends AkkaApp with ArgumentsParser {
+ private val LOG: Logger = LogUtil.getLogger(getClass)
+ val RUN_FOR_EVER = -1
+
+ override val options: Array[(String, CLIOption[Any])] = Array(
+ "splitNum" -> CLIOption[Int]("<how many sum tasks>", required = false, defaultValue = Some(1)),
+ "sinkNum" -> CLIOption[Int]("<how many sum tasks>", required = false, defaultValue = Some(1))
+ )
+
+ def application(config: ParseResult, system: ActorSystem): StreamApplication = {
+ implicit val actorSystem = system
+
+ val splitNum = config.getInt("splitNum")
+ val sinkNum = config.getInt("sinkNum")
+
+ val split = new Split
+ val sourceProcessor = DataSourceProcessor(split, splitNum, "Split")
+ val sink = new Sink
--- End diff --
HBaseSink is already a DataSink so you don't need to implement this one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump issue #107: Add hbase-example
Posted by manuzhang <gi...@git.apache.org>.
Github user manuzhang commented on the issue:
https://github.com/apache/incubator-gearpump/pull/107
please file a [JIRA issue](http://issues.apache.org/jira/browse/GEARPUMP) for this
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request #107: [GEARPUMP-233]Add hbase-example
Posted by manuzhang <gi...@git.apache.org>.
Github user manuzhang commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/107#discussion_r88181338
--- Diff: project/BuildExternals.scala ---
@@ -80,7 +80,7 @@ object BuildExternals extends sbt.Build {
exclude("log4j", "log4j")
)
))
- .dependsOn (core % "provided", streaming % "test->test; provided")
+ .dependsOn (core, streaming % "test->test; provided")
--- End diff --
the core dependency is always `provided`. please add `core % "provided"` dependency to `example_hbase` project
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request #107: [GEARPUMP-233]Add hbase-example
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-gearpump/pull/107
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump issue #107: [GEARPUMP-233]Add hbase-example
Posted by manuzhang <gi...@git.apache.org>.
Github user manuzhang commented on the issue:
https://github.com/apache/incubator-gearpump/pull/107
please change the commit log to the form `[GEARPUMP-XXX] XYZ`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---