You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samoa.apache.org by jayadeepj <gi...@git.apache.org> on 2015/08/03 07:48:41 UTC

[GitHub] incubator-samoa pull request: SAMOA-29: Remove hardcoded value for...

GitHub user jayadeepj opened a pull request:

    https://github.com/apache/incubator-samoa/pull/34

    SAMOA-29: Remove hardcoded value for the duration of Storm local topology execution

    This fix is for SAMOA-29. Instead of the hard-coded value, the PropertiesConfiguration class of Apache commons-configuration is used to dynamically read samoa.storm.local.mode.execution.duration from config file. Other properties excluding local topology execution is left as such. A new method is added in StormSamoaUtils which can be reused to fetch other properties.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/jayadeepj/incubator-samoa master

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-samoa/pull/34.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #34
    
----
commit edde7c1ce33a8540abdf985ad0b75ea42c6e8f22
Author: jayadeepj <ja...@gmail.com>
Date:   2015-07-31T04:38:14Z

    SAMOA-29: Fix storm local execution mode duration

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-samoa pull request: SAMOA-29: Remove hardcoded value for...

Posted by gdfm <gi...@git.apache.org>.
Github user gdfm commented on the pull request:

    https://github.com/apache/incubator-samoa/pull/34#issuecomment-128642121
  
    Hey @jayadeepj, thanks for taking a look at the issue.
    I manually checked the PR and it passes tests (we need to reduce log verbosity so Travis-CI does not bail out).
    The code looks good to me, however I was unable to make it work as expected.
    
    I set "samoa.storm.local.mode.execution.duration=1" in the properties file, and ran 
    ```
    bin/samoa local target/SAMOA-Local-0.4.0-incubating-SNAPSHOT.jar "PrequentialEvaluation -i 10000000"
    ```
    The topology kept on going until the end, while I expected it to be killed after 1 second.
    Any ideas?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-samoa pull request: SAMOA-29: Remove hardcoded value for...

Posted by gdfm <gi...@git.apache.org>.
Github user gdfm commented on the pull request:

    https://github.com/apache/incubator-samoa/pull/34#issuecomment-128662403
  
    @jayadeepj sorry you are right. I confused the two "local" modes.
    I re-tested it with
    ```
    bin/samoa storm target/SAMOA-Storm-0.4.0-incubating-SNAPSHOT.jar "PrequentialEvaluation -i 10000000"
    ```
    And configuration
    ```
    samoa.storm.mode=local
    samoa.storm.numworker=4
    samoa.storm.local.mode.execution.duration=1
    ```
    but I still managed to get to the end of the evaluation.
    ```
    41052 [Thread-9-org.apache.samoa.evaluation.EvaluatorProcessor_1] INFO  org.apache.samoa.evaluation.EvaluatorProcessor - total evaluation time: 35 seconds for 10000000 instances
    ```



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-samoa pull request: SAMOA-29: Remove hardcoded value for...

Posted by gdfm <gi...@git.apache.org>.
Github user gdfm commented on the pull request:

    https://github.com/apache/incubator-samoa/pull/34#issuecomment-128666501
  
    Sorry, my bad again :)
    I had a previous artifact lying around. After cleaning and recompiling it works as expected.
    
    +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-samoa pull request: SAMOA-29: Remove hardcoded value for...

Posted by jayadeepj <gi...@git.apache.org>.
Github user jayadeepj commented on the pull request:

    https://github.com/apache/incubator-samoa/pull/34#issuecomment-129727319
  
    The code is now updated as per @gdfm suggestion to remove the properties from compile-time include. As per this change, the bin/samoa-storm.properties will be looked up during run time for samoa.storm.local.mode.execution.duration for both Storm Local Mode deployment & JUNIT test executions.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-samoa pull request: SAMOA-29: Remove hardcoded value for...

Posted by jayadeepj <gi...@git.apache.org>.
Github user jayadeepj commented on the pull request:

    https://github.com/apache/incubator-samoa/pull/34#issuecomment-128720359
  
    Hi @gdfm, yes you are right. I had made a compile time addition of the property file to the jar in pom.xml. I did that to avoid a code like this
    
    _config = new PropertiesConfiguration(/some/hardcoded/path/to/bin/samoa-storm.properties)_
    
    Also We may not be able to simply omit samoa-storm.properties from the jar, as the  samoa-storm.properties need to be in classpath for the PropertiesConfiguration to pick it up.  The second alternative that is to update the bin/samoa script to add 'bin' directory to the CLASSPATH something like 
    
    _CLASSPATH="$CLASSPATH:$STORM_HOME/lib/*:$BASE_DIR/:$DEPLOYABLE"
    java -cp $CLASSPATH org.apache.samoa.LocalStormDoTask $COMPLETE_ARG $NUM_WORKER_
    
    In this second alternative, we may need to update pom.xml to include a testResources as well for the storm junits. Let me know your thoughts. Will be happy to implement.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-samoa pull request: SAMOA-29: Remove hardcoded value for...

Posted by gdfm <gi...@git.apache.org>.
Github user gdfm commented on the pull request:

    https://github.com/apache/incubator-samoa/pull/34#issuecomment-129140660
  
    @jayadeepj this second alternative seems reasonable.
    In the end, the bin/samoa script already assumes that the properties file are in the same dir (so it knows where they are and can add them to the classpath).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-samoa pull request: SAMOA-29: Remove hardcoded value for...

Posted by gdfm <gi...@git.apache.org>.
Github user gdfm commented on the pull request:

    https://github.com/apache/incubator-samoa/pull/34#issuecomment-128667532
  
    Actually there is still a small problem.
    The jar target/SAMOA-Storm-0.4.0-incubating-SNAPSHOT.jar includes samoa-storm.properties at compile time, and this properties file overrides the one in bin/.
    If you change the properties file in bin/ at runtime you cannot override the compiled-in value.
    Should we simply omit samoa-storm.properties from the jar?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Re: [GitHub] incubator-samoa pull request: SAMOA-29: Remove hardcoded value for...

Posted by Nicolas Kourtellis <nk...@gmail.com>.
+1
Thanks for the addition!

On Mon, Aug 17, 2015 at 10:00 AM, abifet <gi...@git.apache.org> wrote:

> Github user abifet commented on the pull request:
>
>
> https://github.com/apache/incubator-samoa/pull/34#issuecomment-131721175
>
>     +1
>
>
> ---
> If your project is set up for it, you can reply to this email and have your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, please
> contact infrastructure at infrastructure@apache.org or file a JIRA ticket
> with INFRA.
> ---
>



-- 
Nicolas Kourtellis

[GitHub] incubator-samoa pull request: SAMOA-29: Remove hardcoded value for...

Posted by abifet <gi...@git.apache.org>.
Github user abifet commented on the pull request:

    https://github.com/apache/incubator-samoa/pull/34#issuecomment-131721175
  
    +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-samoa pull request: SAMOA-29: Remove hardcoded value for...

Posted by jayadeepj <gi...@git.apache.org>.
Github user jayadeepj commented on the pull request:

    https://github.com/apache/incubator-samoa/pull/34#issuecomment-128649878
  
    Hi @gdfm, Thanks for the review.
    
    I am under the impression this fix is for Storm's Local mode and not the 'Local Test Mode' of SAMOA itself. The fix is made in LocalStormDoTask of samoa-storm (and not the LocalDoTask of samoa-local). I guess you have invoked it bin/samoa local target/SAMOA-Local-0.4.0-incubating-SNAPSHOT.jar ....
    
    I have tested it with bin/samoa storm target/SAMOA-Storm-0.4.0-incubating-SNAPSHOT.jar "PrequentialEvaluation -l classifiers.ensemble.Bagging -s (ArffFileStream -f covtypeNorm.arff) -f 100000".
    
    Though I will run it again & check it to confirm. Please correct if my understanding is wrong.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-samoa pull request: SAMOA-29: Remove hardcoded value for...

Posted by gdfm <gi...@git.apache.org>.
Github user gdfm commented on the pull request:

    https://github.com/apache/incubator-samoa/pull/34#issuecomment-131705879
  
    Great!
    +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-samoa pull request: SAMOA-29: Remove hardcoded value for...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/incubator-samoa/pull/34


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---