You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@oodt.apache.org by Michael Starch <st...@umich.edu> on 2014/12/10 22:41:17 UTC

Re: Review Request 22791: Streaming OODT Changes

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/
-----------------------------------------------------------

(Updated Dec. 10, 2014, 9:41 p.m.)


Review request for oodt, Lewis McGibbney and Chris Mattmann.


Repository: oodt


Description (updated)
-------

This patch contains all the changes needed to add in "streaming oodt" into the oodt svn repository.

There are four main portions:
   -Mesos Framework for Resource Manager (Prototype working)
   -Spark Runner for Workflow Manager (Prototype working)
   -Filemanager "streaming" type (In development)
   -Deployment and cluster management scripts (In development)

Where can this stuff be put so that it is available to use, even while it is in development?


Note: Filemanager work (and corrections here-in) have been moved to sub-patch review: https://reviews.apache.org/r/27172

Resource manager changes sent to: https://reviews.apache.org/r/27773/ and https://reviews.apache.org/r/28917/


Diffs
-----

  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800 
  http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java 1617800 
  http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java 1617800 
  http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java 1617800 
  http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java 1617800 
  http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java 1617800 
  http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java 1617800 
  http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java 1617800 
  http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java 1617800 
  http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java 1617800 
  http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java 1617800 
  http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/VersioningUtils.java 1617800 
  http://svn.apache.org/repos/asf/oodt/trunk/resource/pom.xml 1617800 
  http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManager.java PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManagerFactory.java PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/MesosUtilities.java PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/OODTExecutor.java PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosScheduler.java PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/exception/MesosFrameworkException.java PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/proto/ResourceProto.java PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitor.java PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitorFactory.java PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/Scheduler.java 1617800 
  http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/proto/resc.proto PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/streamer/pom.xml PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/assembly/assembly.xml PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/bin/streamer PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/KafkaPublisher.java PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/Publisher.java PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/InputStreamReader.java PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/Reader.java PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/StreamEmptyException.java PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStream.java.bak PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStreamArcheaic.java PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/system/MultiSourceStreamer.java PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-actions.xml PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-options.xml PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/logging.properties PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/streamer.properties PRE-CREATION 

Diff: https://reviews.apache.org/r/22791/diff/


Testing
-------

Basic functionality tests done for both the resource-manger and workflow manager pieces.  Filemanager have been tested to properly ingest a "GenericStream" type with the lucene catalog only.


Thanks,

Michael Starch

Re: Review Request 22791: Streaming OODT Changes

Posted by Chris Mattmann <ma...@apache.org>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/#review64661
-----------------------------------------------------------

Ship it!


Ship It!

- Chris Mattmann


On Dec. 10, 2014, 9:41 p.m., Michael Starch wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22791/
> -----------------------------------------------------------
> 
> (Updated Dec. 10, 2014, 9:41 p.m.)
> 
> 
> Review request for oodt, Lewis McGibbney and Chris Mattmann.
> 
> 
> Repository: oodt
> 
> 
> Description
> -------
> 
> This patch contains all the changes needed to add in "streaming oodt" into the oodt svn repository.
> 
> There are four main portions:
>    -Mesos Framework for Resource Manager (Prototype working)
>    -Spark Runner for Workflow Manager (Prototype working)
>    -Filemanager "streaming" type (In development)
>    -Deployment and cluster management scripts (In development)
> 
> Where can this stuff be put so that it is available to use, even while it is in development?
> 
> 
> Note: Filemanager work (and corrections here-in) have been moved to sub-patch review: https://reviews.apache.org/r/27172
> 
> Resource manager changes sent to: https://reviews.apache.org/r/27773/ and https://reviews.apache.org/r/28917/
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800 
>   http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java 1617800 
>   http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java 1617800 
>   http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java 1617800 
>   http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java 1617800 
>   http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java 1617800 
>   http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java 1617800 
>   http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java 1617800 
>   http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java 1617800 
>   http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java 1617800 
>   http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java 1617800 
>   http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/VersioningUtils.java 1617800 
>   http://svn.apache.org/repos/asf/oodt/trunk/resource/pom.xml 1617800 
>   http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManager.java PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManagerFactory.java PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/MesosUtilities.java PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/OODTExecutor.java PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosScheduler.java PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/exception/MesosFrameworkException.java PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/proto/ResourceProto.java PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitor.java PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitorFactory.java PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/Scheduler.java 1617800 
>   http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/proto/resc.proto PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/streamer/pom.xml PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/assembly/assembly.xml PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/bin/streamer PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/KafkaPublisher.java PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/Publisher.java PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/InputStreamReader.java PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/Reader.java PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/StreamEmptyException.java PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStream.java.bak PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStreamArcheaic.java PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/system/MultiSourceStreamer.java PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-actions.xml PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-options.xml PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/logging.properties PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/streamer.properties PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/22791/diff/
> 
> 
> Testing
> -------
> 
> Basic functionality tests done for both the resource-manger and workflow manager pieces.  Filemanager have been tested to properly ingest a "GenericStream" type with the lucene catalog only.
> 
> 
> Thanks,
> 
> Michael Starch
> 
>