You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samoa.apache.org by bhupeshchawda <gi...@git.apache.org> on 2016/06/08 23:29:25 UTC
[GitHub] incubator-samoa pull request #55: SAMOA-49: Added adapter for Apache Apex
GitHub user bhupeshchawda opened a pull request:
https://github.com/apache/incubator-samoa/pull/55
SAMOA-49: Added adapter for Apache Apex
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/bhupeshchawda/incubator-samoa SAMOA-49-adapter-apache-apex
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-samoa/pull/55.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #55
----
commit d7acb359feef2a5749e46421965d44b9056df8c6
Author: bhupeshchawda <bh...@gmail.com>
Date: 2015-09-22T11:04:14Z
SAMOA-49: Added adapter for Apache Apex
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-samoa issue #55: SAMOA-49: Added adapter for Apache Apex
Posted by bhupeshchawda <gi...@git.apache.org>.
Github user bhupeshchawda commented on the issue:
https://github.com/apache/incubator-samoa/pull/55
@gdfm Did the above comments help you in moving forward? Please let me know if any help is needed towards the PR review.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-samoa issue #55: SAMOA-49: Added adapter for Apache Apex
Posted by bhupeshchawda <gi...@git.apache.org>.
Github user bhupeshchawda commented on the issue:
https://github.com/apache/incubator-samoa/pull/55
@davidyan74 Yes David. I am looking at some optimization using the window length and the number of tuples.
Regarding the serialization changes, ```@Bind``` might not be possible since these classes are only within Samoa code and are only called during runtime from samoa-apex.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-samoa issue #55: SAMOA-49: Added adapter for Apache Apex
Posted by gdfm <gi...@git.apache.org>.
Github user gdfm commented on the issue:
https://github.com/apache/incubator-samoa/pull/55
I am trying to run the code in this PR against Apache Apex on a cluster.
I see the SAMOA application is running, but when I try to visualize the tracker page I get a 404 error and a WebAppException.
Probably there is something wrong in my cluster setup. Could you point me to a configuration guide for Apex?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-samoa issue #55: SAMOA-49: Added adapter for Apache Apex
Posted by davidyan74 <gi...@git.apache.org>.
Github user davidyan74 commented on the issue:
https://github.com/apache/incubator-samoa/pull/55
@bhupeshchawda For the slow execution, we can do some tuning on the timeout window count, the window width and the number of tuples being generated for each window. For the serialization, is it possible to use java serializer for the affected fields by using @FieldSerializer.Bind(JavaSerializer.class) within the Apex operator without changing Samoa code?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-samoa issue #55: SAMOA-49: Added adapter for Apache Apex
Posted by nicolas-kourtellis <gi...@git.apache.org>.
Github user nicolas-kourtellis commented on the issue:
https://github.com/apache/incubator-samoa/pull/55
Hi Bhupehash,
I saw you still had some modifications in the samoa-api,
- ModelAggregatorProcessor.java
- ArffFileStream.java
so I had two quick questions:
1) What is the effect of modifying the two fields (ScheduledFeature and fileReader) into Transient, especially on other DSPEs? I am assuming that since the other DSPEs didn't need these objects to be serializable / serialized, the change shouldn't matter, but I wanted your opinion as well.
2) Do you need to do similar changes to other parts of the samoa-api for the objects to be serialized? i.e., in other classifiers and clusterers in learning?
Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-samoa issue #55: SAMOA-49: Added adapter for Apache Apex
Posted by bhupeshchawda <gi...@git.apache.org>.
Github user bhupeshchawda commented on the issue:
https://github.com/apache/incubator-samoa/pull/55
@nicolas-kourtellis Thanks a lot for your review. Please give me some time to look at the issues and get back with the explanations.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-samoa issue #55: SAMOA-49: Added adapter for Apache Apex
Posted by nicolas-kourtellis <gi...@git.apache.org>.
Github user nicolas-kourtellis commented on the issue:
https://github.com/apache/incubator-samoa/pull/55
Hi Bhupesh,
Thanks for your responses. It makes sense indeed.
One more point regarding execution speed:
- Even though things are faster now after the changes, I wonder if they can be even faster, (always depending on the data).
- Do you think it makes sense to expose any of the windowing parameters needed in Apex on the higher layer (e.g., to be set by the user at command line)? In other words, do you think the optimization for the window length, number of tuples, etc., can be exposed to the user level and tested? Also, any guide on the values you picked?
- I am asking this because I think depending on the type of data inputted, the optimization point maybe different and it would be good to be able to find it at least experimentally.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-samoa issue #55: SAMOA-49: Added adapter for Apache Apex
Posted by bhupeshchawda <gi...@git.apache.org>.
Github user bhupeshchawda commented on the issue:
https://github.com/apache/incubator-samoa/pull/55
Updated the PR to remove Samoa specific changes due to Kryo. Also, the speed should be better now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-samoa issue #55: SAMOA-49: Added adapter for Apache Apex
Posted by bhupeshchawda <gi...@git.apache.org>.
Github user bhupeshchawda commented on the issue:
https://github.com/apache/incubator-samoa/pull/55
Hi Gianmarco,
What is the tracker page you are trying to view?
For Apex, the visualization is provided by DataTorrent in a community edition of its product. You can download it from here: https://www.datatorrent.com/download/. Installation guide is here: http://docs.datatorrent.com/installation/ and details about dtManage (the monitoring UI tool) here: http://docs.datatorrent.com/dtmanage/
Also, Apex runs as a YARN application. You can track the application through the YARN web UI as well. Usually it is running on http://r<esourcemanager>:8088.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-samoa issue #55: SAMOA-49: Added adapter for Apache Apex
Posted by tweise <gi...@git.apache.org>.
Github user tweise commented on the issue:
https://github.com/apache/incubator-samoa/pull/55
@bhupeshchawda We should be able to do serialization without the Kryo related changes. Fields inside operators can be marked with different serializers, what goes over the wire can be encoded using any coders that may already be provided by Samoa or another compatible mechanism by using a different Apex stream codec.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-samoa issue #55: SAMOA-49: Added adapter for Apache Apex
Posted by bhupeshchawda <gi...@git.apache.org>.
Github user bhupeshchawda commented on the issue:
https://github.com/apache/incubator-samoa/pull/55
@gdfm Can you try installing https://www.datatorrent.com/downloads/3.5.0/datatorrent-rts-community-3.5.0.bin. This is the community edition of Datatorrent RTS and will give you access to the visualization.
Also, what is the exact command you are trying to run? Can you use the dump option (-d file) in the SAMOA args? You can then monitor this file to check if the output is generated successfully.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-samoa issue #55: SAMOA-49: Added adapter for Apache Apex
Posted by gdfm <gi...@git.apache.org>.
Github user gdfm commented on the issue:
https://github.com/apache/incubator-samoa/pull/55
I managed to access the YARN logs but I get nothing on the stdout.
When I try to access the AM instead I get a 404 caused by:
org.apache.hadoop.yarn.webapp.WebAppException: /: controller for default not found
at org.apache.hadoop.yarn.webapp.Router.resolveDefault(Router.java:232)
at org.apache.hadoop.yarn.webapp.Router.resolve(Router.java:140)
at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:134)
Seems like I am doing something wrong.
On the other hand, I also tried to run the local mode by changing the property in samoa-apex.properties, and I manage to get the intended output (although there is a lot of debug information to go through and the execution is rather slow).
How can I validate my apex cluster installation?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-samoa issue #55: SAMOA-49: Added adapter for Apache Apex
Posted by bhupeshchawda <gi...@git.apache.org>.
Github user bhupeshchawda commented on the issue:
https://github.com/apache/incubator-samoa/pull/55
Rebased with latest master and squashed commits
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-samoa issue #55: SAMOA-49: Added adapter for Apache Apex
Posted by bhupeshchawda <gi...@git.apache.org>.
Github user bhupeshchawda commented on the issue:
https://github.com/apache/incubator-samoa/pull/55
Hi Nicolas,
Please find my responses below:
1. I think these changes should not affect other DSPEs as these fields must have been either serializable or transient. I verified that these objects are created before being used. So in case of a failure, it might not need to be read from a serialized form and then deserialized.
2. I don't need to make any other changes to samoa-api. I have a fix in the code to handle all other cases. These specific cases were not handled because these serializable classes have non-serializable elements in them which actually should have been transient.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-samoa issue #55: SAMOA-49: Added adapter for Apache Apex
Posted by bhupeshchawda <gi...@git.apache.org>.
Github user bhupeshchawda commented on the issue:
https://github.com/apache/incubator-samoa/pull/55
Hi Nicolas,
I completely agree with your point that the optimization point can change depending on the data and topology. We should indeed allow the user to tune the performance depending on the dataset and the algorithm topology being run.
As suggested, I have added an external config parameter for the user to set the limit on the number of tuples that will be consumed in one window. The instructions are updated in the README.md file. The README, now also contains information regarding more configuration options for the Apex engine.
Please review.
Apologies for the delay.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-samoa pull request #55: SAMOA-49: Added adapter for Apache Apex
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-samoa/pull/55
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-samoa issue #55: SAMOA-49: Added adapter for Apache Apex
Posted by bhupeshchawda <gi...@git.apache.org>.
Github user bhupeshchawda commented on the issue:
https://github.com/apache/incubator-samoa/pull/55
@gdfm
Were you able to get the visualization working?
Any updates on the pr review?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-samoa issue #55: SAMOA-49: Added adapter for Apache Apex
Posted by bhupeshchawda <gi...@git.apache.org>.
Github user bhupeshchawda commented on the issue:
https://github.com/apache/incubator-samoa/pull/55
@nicolas-kourtellis Please find my responses below:
1) The slow execution is a deliberate (although temporary) configuration done in samoa-apex by limiting the number of tuples in an application window. This has to do with the way iteration works in Apex, which is tightly coupled to windowing. In case we don't limit the number of tuples, the tuples in a particular window keep on increasing due to the additional tuples that are fed on the iteration loop back stream. If the number of iterations is large enough, the amount of time taken to process a window of data increases beyond normal behaviour and the operator is killed by the Apex app master. I am working on identifying some workaround either to eliminate this limit, or to optimally set this limit.
2) The execution in local mode of Apex is highly asynchronous with all operators in the topology running in different threads. The local mode of Samoa, on the other hand seems to be synchronous; i.e. the next tuple is processed only when the first one has been processed completely by all operators. I also tried to check executing the local mode of Storm, which also produces different results every time it is run for the same input file.
3) I think this is due to the same reason in (2)
4) Yes, these changes are necessary for Apex to function correctly. Apex relies on Kryo serialization (without any fall back on Java serialization) and hence is necessary for classes to have a default constructor. I think it will be better to have them as part of this PR. May be I can split them into a different commit if that helps?
Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-samoa issue #55: SAMOA-49: Added adapter for Apache Apex
Posted by nicolas-kourtellis <gi...@git.apache.org>.
Github user nicolas-kourtellis commented on the issue:
https://github.com/apache/incubator-samoa/pull/55
Hi bhupeshchawda,
I was checking the adapter on apex local mode and I had some initial comments:
1) The execution of the classification on both a fixed arff file (covTypeNorm) and the random tree generator are much slower (even 5-10 times slower) in comparison to the samoa-local (the basic local mode of Samoa for testing purposes). Can you check if this is a matter of the necessary overhead for the distributed execution or something else is causing the delay? I tested with various numbers for parallelism hint, but the observation holds.
2) The results on covtypenorm should have been deterministic. That is, given that the file is the same used in repetitive runs, the results should be identical across runs (for sure given that parallelism is 1). However, when I run the VHT repetitively on the same arff file, the results are not the same across runs. Is there any fundamental reason why this would be happening? (Again, you can check for this using the local mode for testing).
3) The results with the random tree generator show a good accuracy and similar to the one received with basic local mode of Samoa for testing. But the results on the arff file degrade in performance, lower than the basic local mode of Samoa. Any reason why this is happening? Maybe it has to do with the problem in comment 2?
4) I noticed there are some changes in your PR regarding the samoa-api in about 10 files. These changes seem to be mostly related to Kryo serialization. Can you explain why you needed these done now? Also, do they affect the execution with the other adaptors and DSPEs? Also, maybe it would have made sense to push them in a different PR? Or do they need to be there for Apex to play along in the first place?
Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-samoa issue #55: SAMOA-49: Added adapter for Apache Apex
Posted by nicolas-kourtellis <gi...@git.apache.org>.
Github user nicolas-kourtellis commented on the issue:
https://github.com/apache/incubator-samoa/pull/55
Thanks Bhupesh,
I will check it out as soon as possible in the next few days.
Cheers,
Nicolas
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-samoa issue #55: SAMOA-49: Added adapter for Apache Apex
Posted by bhupeshchawda <gi...@git.apache.org>.
Github user bhupeshchawda commented on the issue:
https://github.com/apache/incubator-samoa/pull/55
Added a section in `samoa-apex/README.md ` on using Apex cli to view details on running applications.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---