You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samoa.apache.org by pangolulu <gi...@git.apache.org> on 2016/04/14 09:25:00 UTC
[GitHub] incubator-samoa pull request: SAMOA-59, add an adapter for Apache ...
GitHub user pangolulu opened a pull request:
https://github.com/apache/incubator-samoa/pull/54
SAMOA-59, add an adapter for Apache Gearpump
Hi,
This PR is referenced to [SAMOA-59](https://issues.apache.org/jira/browse/SAMOA-59). And how to execute SAMOA with Apache Gearpump, you can follow the README file.
Thanks.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pangolulu/incubator-samoa samoa-gearpump
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-samoa/pull/54.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #54
----
commit 1ff1f4044bf9f7e0ba129394af73f153bd1cb233
Author: pangolulu <gy...@gmail.com>
Date: 2016-04-07T11:45:27Z
SAMOA-59, add an adapter for Apache Gearpump
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-samoa issue #54: SAMOA-59: add an adapter for Apache Gearpump
Posted by manuzhang <gi...@git.apache.org>.
Github user manuzhang commented on the issue:
https://github.com/apache/incubator-samoa/pull/54
@gdfm I upgraded gearpump version to 0.8.1-SNAPSHOT which includes critical bug fixes and package renaming although that requires manual build. Hopefully we'll release gearpump 0.8.1 soon.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-samoa issue #54: SAMOA-59: add an adapter for Apache Gearpump
Posted by manuzhang <gi...@git.apache.org>.
Github user manuzhang commented on the issue:
https://github.com/apache/incubator-samoa/pull/54
@nicolas-kourtellis Thanks for the detailed review. I'll look into each item and get back to you.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-samoa issue #54: SAMOA-59: add an adapter for Apache Gearpump
Posted by manuzhang <gi...@git.apache.org>.
Github user manuzhang commented on the issue:
https://github.com/apache/incubator-samoa/pull/54
@nicolas-kourtellis We haven't made a binary release yet so the binary for 0.8.1 is temporarily hosted at our old github repo https://github.com/gearpump/gearpump/releases/tag/0.8.1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-samoa pull request #54: SAMOA-59: add an adapter for Apache Gearpu...
Posted by gdfm <gi...@git.apache.org>.
Github user gdfm commented on a diff in the pull request:
https://github.com/apache/incubator-samoa/pull/54#discussion_r65855293
--- Diff: pom.xml ---
@@ -127,6 +137,9 @@
<miniball.version>1.0.3</miniball.version>
<samza.version>0.7.0</samza.version>
<flink.version>0.10.1</flink.version>
+ <gearpump.version>0.7.5</gearpump.version>
--- End diff --
Could we use the latest 0.8.0 version?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-samoa issue #54: SAMOA-59: add an adapter for Apache Gearpump
Posted by nicolas-kourtellis <gi...@git.apache.org>.
Github user nicolas-kourtellis commented on the issue:
https://github.com/apache/incubator-samoa/pull/54
Hi manuzhang,
I was trying to test the adapter in local mode but realized that I am having difficulties compiling the gearpump 0.8.1 from source. I installed scala, sbt, etc., following the instructions I found here
http://gearpump.incubator.apache.org/releases/latest/get-gearpump-distribution.html
but it keeps complaining for some things and the build fails. Is it possible you make available the binary version of 0.8.1 so that I can use it directly? Or are you guys planning to make it available for the public soon?
Thanks!
Nicolas
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-samoa issue #54: SAMOA-59: add an adapter for Apache Gearpump
Posted by nicolas-kourtellis <gi...@git.apache.org>.
Github user nicolas-kourtellis commented on the issue:
https://github.com/apache/incubator-samoa/pull/54
Hi @manuzhang,
I managed to get the adapter working. Here are some notes that I would ask you take into consideration:
- There are some inherent difficulties compiling gearpump from source. It would be good to have a compiled version to use directly.
- Assuming this is given (which was my case because @manuzhang provided a compiled version), I managed to get samoa to compile/package with gearpump and run the package.
- However, it would be good for the adapter to be upgraded to the new version of samoa in incubation, which is 0.5.0. But it should be fairly straightforward. This will allow us to test it with some more generators and ML methods added in the recent past.
- Feedback when executing VHT:
=> The engine seems to continue executing the topology long after it has been created, used for the task and finished. Is there any way to pass a signal at the end of the execution to shut it down? (note: not the engine itself, but the topology). It was occupying resources on my computer for no reason at full CPU consumption. I found a manual way to kill it using the command "gear kill -appid X" with X being the id of the task, but I wonder if there is a more automatic way.
=> After I killed the jobs manually, the java processes that were created for the execution (I will assume they are the containers of the topologies) were still alive, just not consuming much resources. Shouldn't they have been terminated and removed? Is there a way to do that?
=> When I run new tasks, they just keep getting added on the engine (which is logical), even though I had killed the other ones earlier.
=> Multiple executions of the same experiment with the same seed for the random generator using the parameter -r which should yield the same random tree, perform differently with respect to accuracy. Is that expected?
=> Using a different seed for the random tree generator (r=1,...,5), the performance of the execution of VHT on local GearPump is fairly low (average over 5 different seeds: 65.39% accuracy) in comparison to running the topology on local Storm (84.046% accuracy). Any explanation why so much reduction in performance?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-samoa issue #54: SAMOA-59: add an adapter for Apache Gearpump
Posted by manuzhang <gi...@git.apache.org>.
Github user manuzhang commented on the issue:
https://github.com/apache/incubator-samoa/pull/54
@nicolas-kourtellis any updates on this PR ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-samoa pull request #54: SAMOA-59: add an adapter for Apache Gearpu...
Posted by manuzhang <gi...@git.apache.org>.
Github user manuzhang commented on a diff in the pull request:
https://github.com/apache/incubator-samoa/pull/54#discussion_r66021866
--- Diff: pom.xml ---
@@ -127,6 +137,9 @@
<miniball.version>1.0.3</miniball.version>
<samza.version>0.7.0</samza.version>
<flink.version>0.10.1</flink.version>
+ <gearpump.version>0.7.5</gearpump.version>
--- End diff --
yes, I'll bump it to 0.8.0
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-samoa issue #54: SAMOA-59: add an adapter for Apache Gearpump
Posted by manuzhang <gi...@git.apache.org>.
Github user manuzhang commented on the issue:
https://github.com/apache/incubator-samoa/pull/54
@nicolas-kourtellis updated to latest release [0.8.1](https://dist.apache.org/repos/dist/release/incubator/gearpump/0.8.1-incubating/). Note this is a source release so you have to build gearpump from source.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-samoa issue #54: SAMOA-59: add an adapter for Apache Gearpump
Posted by nicolas-kourtellis <gi...@git.apache.org>.
Github user nicolas-kourtellis commented on the issue:
https://github.com/apache/incubator-samoa/pull/54
@manuzhang was there any update from your end on the version of gearpump that we can test Samoa?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---