You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oodt.apache.org by Radu Manole <ma...@gmail.com> on 2015/08/22 17:50:19 UTC

Review Request 37701: Full Avro RPC impementation in crawler module.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37701/
-----------------------------------------------------------

Review request for oodt, Lewis McGibbney and Chris Mattmann.


Repository: oodt


Description
-------

Modified the DaemonCrawler to use Avro RPC login instead of XML-RPC.


Diffs
-----

  trunk/crawler/pom.xml 1693501 
  trunk/crawler/src/main/avro/types/protocol.avdl PRE-CREATION 
  trunk/crawler/src/main/bin/crawlctl 1693501 
  trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/cli/action/CrawlerLauncherCliAction.java 1693501 
  trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/AvroCrawlDaemon.java PRE-CREATION 
  trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/AvroCrawlDaemonController.java PRE-CREATION 
  trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/CrawlDaemon.java 1693501 
  trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/CrawlDaemonController.java 1693501 
  trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/IntCrawlDaemon.java PRE-CREATION 
  trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/IntCrawlDaemonController.java PRE-CREATION 
  trunk/crawler/src/test/java/org/apache/oodt/cas/crawl/daemon/DummyDaemonCrawler.java PRE-CREATION 
  trunk/crawler/src/test/java/org/apache/oodt/cas/crawl/daemon/TestDaemonCrawler.java PRE-CREATION 

Diff: https://reviews.apache.org/r/37701/diff/


Testing
-------

No test was found for xmlrpc DaemonCrawler logic, so I made a DummyAvroDaemonCrawler class to test only the rpc aspect of daemon crawler. 
and created a class TestDaemonCrawler where I tested the connection and the communication between AvroDaemonCrawler and AvroDaemonCrawlerController.


Thanks,

Radu Manole

Re: Review Request 37701: Full Avro RPC impementation in crawler module.

Posted by Chris Mattmann <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37701/#review96361
-----------------------------------------------------------



trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/AvroCrawlDaemon.java (line 1)
<https://reviews.apache.org/r/37701/#comment151679>

    I wonder if this could be refactored, so that the differences between it and XmlRpc one are minimal.



trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/AvroCrawlDaemonController.java (line 33)
<https://reviews.apache.org/r/37701/#comment151681>

    also maybe refactor to create minor differences between this and XmlRpc one.



trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/CrawlDaemonController.java (line 42)
<https://reviews.apache.org/r/37701/#comment151686>

    How about just change CrawlDaemonController to an interface instead of changing its name?



trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/IntCrawlDaemon.java (line 24)
<https://reviews.apache.org/r/37701/#comment151687>

    maybe just call it CrawlDaemon and then CrawlDaemonController?



trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/IntCrawlDaemonController.java (line 24)
<https://reviews.apache.org/r/37701/#comment151688>

    take the Int prefix off?


- Chris Mattmann


On Aug. 22, 2015, 3:50 p.m., Radu Manole wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37701/
> -----------------------------------------------------------
> 
> (Updated Aug. 22, 2015, 3:50 p.m.)
> 
> 
> Review request for oodt, Lewis McGibbney and Chris Mattmann.
> 
> 
> Repository: oodt
> 
> 
> Description
> -------
> 
> Modified the DaemonCrawler to use Avro RPC login instead of XML-RPC.
> 
> 
> Diffs
> -----
> 
>   trunk/crawler/pom.xml 1693501 
>   trunk/crawler/src/main/avro/types/protocol.avdl PRE-CREATION 
>   trunk/crawler/src/main/bin/crawlctl 1693501 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/cli/action/CrawlerLauncherCliAction.java 1693501 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/AvroCrawlDaemon.java PRE-CREATION 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/AvroCrawlDaemonController.java PRE-CREATION 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/CrawlDaemon.java 1693501 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/CrawlDaemonController.java 1693501 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/IntCrawlDaemon.java PRE-CREATION 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/IntCrawlDaemonController.java PRE-CREATION 
>   trunk/crawler/src/test/java/org/apache/oodt/cas/crawl/daemon/DummyDaemonCrawler.java PRE-CREATION 
>   trunk/crawler/src/test/java/org/apache/oodt/cas/crawl/daemon/TestDaemonCrawler.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/37701/diff/
> 
> 
> Testing
> -------
> 
> No test was found for xmlrpc DaemonCrawler logic, so I made a DummyAvroDaemonCrawler class to test only the rpc aspect of daemon crawler. 
> and created a class TestDaemonCrawler where I tested the connection and the communication between AvroDaemonCrawler and AvroDaemonCrawlerController.
> 
> 
> Thanks,
> 
> Radu Manole
> 
>

Re: Review Request 37701: Full Avro RPC impementation in crawler module.

Posted by Chris Mattmann <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37701/#review96363
-----------------------------------------------------------


Overall pretty good work! See my comments, I think this is close.

- Chris Mattmann


On Aug. 22, 2015, 3:50 p.m., Radu Manole wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37701/
> -----------------------------------------------------------
> 
> (Updated Aug. 22, 2015, 3:50 p.m.)
> 
> 
> Review request for oodt, Lewis McGibbney and Chris Mattmann.
> 
> 
> Repository: oodt
> 
> 
> Description
> -------
> 
> Modified the DaemonCrawler to use Avro RPC login instead of XML-RPC.
> 
> 
> Diffs
> -----
> 
>   trunk/crawler/pom.xml 1693501 
>   trunk/crawler/src/main/avro/types/protocol.avdl PRE-CREATION 
>   trunk/crawler/src/main/bin/crawlctl 1693501 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/cli/action/CrawlerLauncherCliAction.java 1693501 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/AvroCrawlDaemon.java PRE-CREATION 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/AvroCrawlDaemonController.java PRE-CREATION 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/CrawlDaemon.java 1693501 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/CrawlDaemonController.java 1693501 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/IntCrawlDaemon.java PRE-CREATION 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/IntCrawlDaemonController.java PRE-CREATION 
>   trunk/crawler/src/test/java/org/apache/oodt/cas/crawl/daemon/DummyDaemonCrawler.java PRE-CREATION 
>   trunk/crawler/src/test/java/org/apache/oodt/cas/crawl/daemon/TestDaemonCrawler.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/37701/diff/
> 
> 
> Testing
> -------
> 
> No test was found for xmlrpc DaemonCrawler logic, so I made a DummyAvroDaemonCrawler class to test only the rpc aspect of daemon crawler. 
> and created a class TestDaemonCrawler where I tested the connection and the communication between AvroDaemonCrawler and AvroDaemonCrawlerController.
> 
> 
> Thanks,
> 
> Radu Manole
> 
>

Re: Review Request 37701: Full Avro RPC impementation in crawler module.

Posted by Lewis McGibbney <le...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37701/#review97620
-----------------------------------------------------------

Ship it!


Ship It!

- Lewis McGibbney


On Aug. 27, 2015, 12:11 p.m., Radu Manole wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37701/
> -----------------------------------------------------------
> 
> (Updated Aug. 27, 2015, 12:11 p.m.)
> 
> 
> Review request for oodt, Lewis McGibbney and Chris Mattmann.
> 
> 
> Repository: oodt
> 
> 
> Description
> -------
> 
> Modified the DaemonCrawler to use Avro RPC login instead of XML-RPC.
> 
> 
> Diffs
> -----
> 
>   trunk/crawler/pom.xml 1693501 
>   trunk/crawler/src/main/avro/types/protocol.avdl PRE-CREATION 
>   trunk/crawler/src/main/bin/crawlctl 1693501 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/cli/action/CrawlerLauncherCliAction.java 1693501 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/AvroCrawlDaemonController.java PRE-CREATION 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/AvroRpcCrawlerDaemon.java PRE-CREATION 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/CrawlDaemon.java 1693501 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/CrawlDaemonController.java 1693501 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/XmlRpcCrawlDaemonController.java PRE-CREATION 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/XmlRpcCrawlerDaemon.java PRE-CREATION 
>   trunk/crawler/src/test/java/org/apache/oodt/cas/crawl/daemon/DummyDaemonCrawler.java PRE-CREATION 
>   trunk/crawler/src/test/java/org/apache/oodt/cas/crawl/daemon/TestDaemonCrawler.java PRE-CREATION 
>   trunk/mvn/archetypes/radix/src/main/resources/archetype-resources/crawler/src/main/resources/bin/crawlctl 1693501 
>   trunk/pcs/core/src/main/java/org/apache/oodt/pcs/tools/PCSHealthMonitor.java 1693501 
> 
> Diff: https://reviews.apache.org/r/37701/diff/
> 
> 
> Testing
> -------
> 
> No test was found for xmlrpc DaemonCrawler logic, so I made a DummyAvroDaemonCrawler class to test only the rpc aspect of daemon crawler. 
> and created a class TestDaemonCrawler where I tested the connection and the communication between AvroDaemonCrawler and AvroDaemonCrawlerController.
> 
> 
> Thanks,
> 
> Radu Manole
> 
>

Re: Review Request 37701: Full Avro RPC impementation in crawler module.

Posted by Chris Mattmann <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37701/#review97093
-----------------------------------------------------------

Ship it!


Ship It!

- Chris Mattmann


On Aug. 27, 2015, 12:11 p.m., Radu Manole wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37701/
> -----------------------------------------------------------
> 
> (Updated Aug. 27, 2015, 12:11 p.m.)
> 
> 
> Review request for oodt, Lewis McGibbney and Chris Mattmann.
> 
> 
> Repository: oodt
> 
> 
> Description
> -------
> 
> Modified the DaemonCrawler to use Avro RPC login instead of XML-RPC.
> 
> 
> Diffs
> -----
> 
>   trunk/crawler/pom.xml 1693501 
>   trunk/crawler/src/main/avro/types/protocol.avdl PRE-CREATION 
>   trunk/crawler/src/main/bin/crawlctl 1693501 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/cli/action/CrawlerLauncherCliAction.java 1693501 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/AvroCrawlDaemonController.java PRE-CREATION 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/AvroRpcCrawlerDaemon.java PRE-CREATION 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/CrawlDaemon.java 1693501 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/CrawlDaemonController.java 1693501 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/XmlRpcCrawlDaemonController.java PRE-CREATION 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/XmlRpcCrawlerDaemon.java PRE-CREATION 
>   trunk/crawler/src/test/java/org/apache/oodt/cas/crawl/daemon/DummyDaemonCrawler.java PRE-CREATION 
>   trunk/crawler/src/test/java/org/apache/oodt/cas/crawl/daemon/TestDaemonCrawler.java PRE-CREATION 
>   trunk/mvn/archetypes/radix/src/main/resources/archetype-resources/crawler/src/main/resources/bin/crawlctl 1693501 
>   trunk/pcs/core/src/main/java/org/apache/oodt/pcs/tools/PCSHealthMonitor.java 1693501 
> 
> Diff: https://reviews.apache.org/r/37701/diff/
> 
> 
> Testing
> -------
> 
> No test was found for xmlrpc DaemonCrawler logic, so I made a DummyAvroDaemonCrawler class to test only the rpc aspect of daemon crawler. 
> and created a class TestDaemonCrawler where I tested the connection and the communication between AvroDaemonCrawler and AvroDaemonCrawlerController.
> 
> 
> Thanks,
> 
> Radu Manole
> 
>

Re: Review Request 37701: Full Avro RPC impementation in crawler module.

Posted by Radu Manole <ma...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37701/
-----------------------------------------------------------

(Updated Aug. 27, 2015, 12:11 p.m.)


Review request for oodt, Lewis McGibbney and Chris Mattmann.


Changes
-------

Refactor the crawler daemon classes so the difference is only rpc communication.


Repository: oodt


Description
-------

Modified the DaemonCrawler to use Avro RPC login instead of XML-RPC.


Diffs (updated)
-----

  trunk/crawler/pom.xml 1693501 
  trunk/crawler/src/main/avro/types/protocol.avdl PRE-CREATION 
  trunk/crawler/src/main/bin/crawlctl 1693501 
  trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/cli/action/CrawlerLauncherCliAction.java 1693501 
  trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/AvroCrawlDaemonController.java PRE-CREATION 
  trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/AvroRpcCrawlerDaemon.java PRE-CREATION 
  trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/CrawlDaemon.java 1693501 
  trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/CrawlDaemonController.java 1693501 
  trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/XmlRpcCrawlDaemonController.java PRE-CREATION 
  trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/XmlRpcCrawlerDaemon.java PRE-CREATION 
  trunk/crawler/src/test/java/org/apache/oodt/cas/crawl/daemon/DummyDaemonCrawler.java PRE-CREATION 
  trunk/crawler/src/test/java/org/apache/oodt/cas/crawl/daemon/TestDaemonCrawler.java PRE-CREATION 
  trunk/mvn/archetypes/radix/src/main/resources/archetype-resources/crawler/src/main/resources/bin/crawlctl 1693501 
  trunk/pcs/core/src/main/java/org/apache/oodt/pcs/tools/PCSHealthMonitor.java 1693501 

Diff: https://reviews.apache.org/r/37701/diff/


Testing
-------

No test was found for xmlrpc DaemonCrawler logic, so I made a DummyAvroDaemonCrawler class to test only the rpc aspect of daemon crawler. 
and created a class TestDaemonCrawler where I tested the connection and the communication between AvroDaemonCrawler and AvroDaemonCrawlerController.


Thanks,

Radu Manole