You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oodt.apache.org by Michael Starch <st...@umich.edu> on 2014/09/04 19:23:12 UTC
Re: Review Request 22791: Streaming OODT Changes
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/
-----------------------------------------------------------
(Updated Sept. 4, 2014, 5:23 p.m.)
Review request for oodt, Lewis McGibbney and Chris Mattmann.
Changes
-------
Chris, Lewis any other comments or can I get a first pass checked in?
Repository: oodt
Description (updated)
-------
This patch contains all the changes needed to add in "streaming oodt" into the oodt svn repository.
There are four main portions:
-Mesos Framework for Resource Manager (Prototype working)
-Spark Runner for Workflow Manager (Prototype working)
-Filemanager "streaming" type (In development)
-Deployment and cluster management scripts (In development)
Where can this stuff be put so that it is available to use, even while it is in development?
Diffs
-----
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/VersioningUtils.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/resource/pom.xml 1617800
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManager.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManagerFactory.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/MesosUtilities.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/OODTExecutor.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosScheduler.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/exception/MesosFrameworkException.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/proto/ResourceProto.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitor.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitorFactory.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/Scheduler.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/proto/resc.proto PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/pom.xml PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/assembly/assembly.xml PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/bin/streamer PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/KafkaPublisher.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/Publisher.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/InputStreamReader.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/Reader.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/StreamEmptyException.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStream.java.bak PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStreamArcheaic.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/system/MultiSourceStreamer.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-actions.xml PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-options.xml PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/logging.properties PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/streamer.properties PRE-CREATION
Diff: https://reviews.apache.org/r/22791/diff/
Testing
-------
Basic functionality tests done for both the resource-manger and workflow manager pieces. Filemanager have been tested to properly ingest a "GenericStream" type with the lucene catalog only.
Thanks,
Michael Starch
Re: Review Request 22791: Streaming OODT Changes
Posted by Chris Mattmann <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/#review64661
-----------------------------------------------------------
Ship it!
Ship It!
- Chris Mattmann
On Dec. 10, 2014, 9:41 p.m., Michael Starch wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22791/
> -----------------------------------------------------------
>
> (Updated Dec. 10, 2014, 9:41 p.m.)
>
>
> Review request for oodt, Lewis McGibbney and Chris Mattmann.
>
>
> Repository: oodt
>
>
> Description
> -------
>
> This patch contains all the changes needed to add in "streaming oodt" into the oodt svn repository.
>
> There are four main portions:
> -Mesos Framework for Resource Manager (Prototype working)
> -Spark Runner for Workflow Manager (Prototype working)
> -Filemanager "streaming" type (In development)
> -Deployment and cluster management scripts (In development)
>
> Where can this stuff be put so that it is available to use, even while it is in development?
>
>
> Note: Filemanager work (and corrections here-in) have been moved to sub-patch review: https://reviews.apache.org/r/27172
>
> Resource manager changes sent to: https://reviews.apache.org/r/27773/ and https://reviews.apache.org/r/28917/
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/VersioningUtils.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/resource/pom.xml 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManager.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManagerFactory.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/MesosUtilities.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/OODTExecutor.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosScheduler.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/exception/MesosFrameworkException.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/proto/ResourceProto.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitor.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitorFactory.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/Scheduler.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/proto/resc.proto PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/pom.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/assembly/assembly.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/bin/streamer PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/KafkaPublisher.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/Publisher.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/InputStreamReader.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/Reader.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/StreamEmptyException.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStream.java.bak PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStreamArcheaic.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/system/MultiSourceStreamer.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-actions.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-options.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/logging.properties PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/streamer.properties PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22791/diff/
>
>
> Testing
> -------
>
> Basic functionality tests done for both the resource-manger and workflow manager pieces. Filemanager have been tested to properly ingest a "GenericStream" type with the lucene catalog only.
>
>
> Thanks,
>
> Michael Starch
>
>
Re: Review Request 22791: Streaming OODT Changes
Posted by Michael Starch <st...@umich.edu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/
-----------------------------------------------------------
(Updated Dec. 10, 2014, 9:41 p.m.)
Review request for oodt, Lewis McGibbney and Chris Mattmann.
Repository: oodt
Description (updated)
-------
This patch contains all the changes needed to add in "streaming oodt" into the oodt svn repository.
There are four main portions:
-Mesos Framework for Resource Manager (Prototype working)
-Spark Runner for Workflow Manager (Prototype working)
-Filemanager "streaming" type (In development)
-Deployment and cluster management scripts (In development)
Where can this stuff be put so that it is available to use, even while it is in development?
Note: Filemanager work (and corrections here-in) have been moved to sub-patch review: https://reviews.apache.org/r/27172
Resource manager changes sent to: https://reviews.apache.org/r/27773/ and https://reviews.apache.org/r/28917/
Diffs
-----
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/VersioningUtils.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/resource/pom.xml 1617800
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManager.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManagerFactory.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/MesosUtilities.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/OODTExecutor.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosScheduler.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/exception/MesosFrameworkException.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/proto/ResourceProto.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitor.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitorFactory.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/Scheduler.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/proto/resc.proto PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/pom.xml PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/assembly/assembly.xml PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/bin/streamer PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/KafkaPublisher.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/Publisher.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/InputStreamReader.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/Reader.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/StreamEmptyException.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStream.java.bak PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStreamArcheaic.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/system/MultiSourceStreamer.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-actions.xml PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-options.xml PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/logging.properties PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/streamer.properties PRE-CREATION
Diff: https://reviews.apache.org/r/22791/diff/
Testing
-------
Basic functionality tests done for both the resource-manger and workflow manager pieces. Filemanager have been tested to properly ingest a "GenericStream" type with the lucene catalog only.
Thanks,
Michael Starch
Re: Review Request 22791: Streaming OODT Changes
Posted by Michael Starch <st...@umich.edu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/
-----------------------------------------------------------
(Updated Oct. 24, 2014, 10:35 p.m.)
Review request for oodt, Lewis McGibbney and Chris Mattmann.
Changes
-------
Breaking out sub patches, so work isn't held up waiting for all issues to be resolved. Corrected work may now be committed, while other work in progress. It will be done in an order that does not leave OODT broken wiaitng for dependant changes to be entered.
Filemanager changes moved to: https://reviews.apache.org/r/27172
Repository: oodt
Description (updated)
-------
This patch contains all the changes needed to add in "streaming oodt" into the oodt svn repository.
There are four main portions:
-Mesos Framework for Resource Manager (Prototype working)
-Spark Runner for Workflow Manager (Prototype working)
-Filemanager "streaming" type (In development)
-Deployment and cluster management scripts (In development)
Where can this stuff be put so that it is available to use, even while it is in development?
Note: Filemanager work (and corrections here-in) have been moved to sub-patch review: https://reviews.apache.org/r/27172
Diffs
-----
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/VersioningUtils.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/resource/pom.xml 1617800
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManager.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManagerFactory.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/MesosUtilities.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/OODTExecutor.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosScheduler.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/exception/MesosFrameworkException.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/proto/ResourceProto.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitor.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitorFactory.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/Scheduler.java 1617800
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/proto/resc.proto PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/pom.xml PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/assembly/assembly.xml PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/bin/streamer PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/KafkaPublisher.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/Publisher.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/InputStreamReader.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/Reader.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/StreamEmptyException.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStream.java.bak PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStreamArcheaic.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/system/MultiSourceStreamer.java PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-actions.xml PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-options.xml PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/logging.properties PRE-CREATION
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/streamer.properties PRE-CREATION
Diff: https://reviews.apache.org/r/22791/diff/
Testing
-------
Basic functionality tests done for both the resource-manger and workflow manager pieces. Filemanager have been tested to properly ingest a "GenericStream" type with the lucene catalog only.
Thanks,
Michael Starch
Re: Review Request 22791: Streaming OODT Changes
Posted by Chris Mattmann <ma...@apache.org>.
> On Sept. 4, 2014, 5:37 p.m., Chris Mattmann wrote:
> > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh, line 1
> > <https://reviews.apache.org/r/22791/diff/2/?file=659631#file659631line1>
> >
> > Mike, instead of directly including cluster-tools in oodt, can you simply make:
> >
> > # A Maven AntRun script (build.xml) or something that downloads the cluster-tools as part of the Resource Manager build? We shouldn't have to maintain these scripts in OODT.
>
> Michael Starch wrote:
> Download from where? Do we have a place to upload unmaintained OPS scripts?
THanks Mike. Ask @paulramirez about how to use Maven AntRun or assembly.xml - you could theoretically reference Mesos if it has a distribution for its scripts in the Central repo - you could ref the dist dependency as a dependency and then unpack them dynamically into a directory. If it doesn't have a Maven dist assembly for the Mesos scripts, then my recommendation would simply be to:
1. Create a build.xml that downloads (e.g., from Mesos trunk or a tag in Apache git) those scripts into whatever OODT build directory you want (inside of Resource Manager probably makes the most sense)
2. Use the Maven Ant-Run plugin to call that build.xml in resource/pom.xml
3. Consider using a Maven profile (e.g., mvn -Pstreaming) to insulate your streaming profile cluster-tools download
- Chris
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/#review52324
-----------------------------------------------------------
On Sept. 4, 2014, 5:23 p.m., Michael Starch wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22791/
> -----------------------------------------------------------
>
> (Updated Sept. 4, 2014, 5:23 p.m.)
>
>
> Review request for oodt, Lewis McGibbney and Chris Mattmann.
>
>
> Repository: oodt
>
>
> Description
> -------
>
> This patch contains all the changes needed to add in "streaming oodt" into the oodt svn repository.
>
> There are four main portions:
> -Mesos Framework for Resource Manager (Prototype working)
> -Spark Runner for Workflow Manager (Prototype working)
> -Filemanager "streaming" type (In development)
> -Deployment and cluster management scripts (In development)
>
> Where can this stuff be put so that it is available to use, even while it is in development?
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/VersioningUtils.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/resource/pom.xml 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManager.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManagerFactory.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/MesosUtilities.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/OODTExecutor.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosScheduler.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/exception/MesosFrameworkException.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/proto/ResourceProto.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitor.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitorFactory.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/Scheduler.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/proto/resc.proto PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/pom.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/assembly/assembly.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/bin/streamer PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/KafkaPublisher.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/Publisher.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/InputStreamReader.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/Reader.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/StreamEmptyException.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStream.java.bak PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStreamArcheaic.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/system/MultiSourceStreamer.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-actions.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-options.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/logging.properties PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/streamer.properties PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22791/diff/
>
>
> Testing
> -------
>
> Basic functionality tests done for both the resource-manger and workflow manager pieces. Filemanager have been tested to properly ingest a "GenericStream" type with the lucene catalog only.
>
>
> Thanks,
>
> Michael Starch
>
>
Re: Review Request 22791: Streaming OODT Changes
Posted by Michael Starch <st...@umich.edu>.
> On Sept. 4, 2014, 5:37 p.m., Chris Mattmann wrote:
> > http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml, line 274
> > <https://reviews.apache.org/r/22791/diff/2/?file=659641#file659641line274>
> >
> > Custom Maven repos are difficult since the Central repository is phasing them out:
> >
> > http://blog.sonatype.com/2010/03/why-external-repos-are-being-phased-out-of-central/
> >
> > If we absolutely need to ref this repo, can you make it a Maven profile, not enabled by default, so that users won't have the issue for this when downloading and building OODT?
These jars exist in the standard repo, so finding them there.
> On Sept. 4, 2014, 5:37 p.m., Chris Mattmann wrote:
> > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java, line 1
> > <https://reviews.apache.org/r/22791/diff/2/?file=659658#file659658line1>
> >
> > All of these files need ALv2 license headers.
Added ALv2 headers.
- Michael
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/#review52324
-----------------------------------------------------------
On Oct. 24, 2014, 10:35 p.m., Michael Starch wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22791/
> -----------------------------------------------------------
>
> (Updated Oct. 24, 2014, 10:35 p.m.)
>
>
> Review request for oodt, Lewis McGibbney and Chris Mattmann.
>
>
> Repository: oodt
>
>
> Description
> -------
>
> This patch contains all the changes needed to add in "streaming oodt" into the oodt svn repository.
>
> There are four main portions:
> -Mesos Framework for Resource Manager (Prototype working)
> -Spark Runner for Workflow Manager (Prototype working)
> -Filemanager "streaming" type (In development)
> -Deployment and cluster management scripts (In development)
>
> Where can this stuff be put so that it is available to use, even while it is in development?
>
>
> Note: Filemanager work (and corrections here-in) have been moved to sub-patch review: https://reviews.apache.org/r/27172
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/VersioningUtils.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/resource/pom.xml 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManager.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManagerFactory.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/MesosUtilities.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/OODTExecutor.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosScheduler.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/exception/MesosFrameworkException.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/proto/ResourceProto.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitor.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitorFactory.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/Scheduler.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/proto/resc.proto PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/pom.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/assembly/assembly.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/bin/streamer PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/KafkaPublisher.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/Publisher.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/InputStreamReader.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/Reader.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/StreamEmptyException.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStream.java.bak PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStreamArcheaic.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/system/MultiSourceStreamer.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-actions.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-options.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/logging.properties PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/streamer.properties PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22791/diff/
>
>
> Testing
> -------
>
> Basic functionality tests done for both the resource-manger and workflow manager pieces. Filemanager have been tested to properly ingest a "GenericStream" type with the lucene catalog only.
>
>
> Thanks,
>
> Michael Starch
>
>
Re: Review Request 22791: Streaming OODT Changes
Posted by Michael Starch <st...@umich.edu>.
> On Sept. 4, 2014, 5:37 p.m., Chris Mattmann wrote:
> > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java, line 82
> > <https://reviews.apache.org/r/22791/diff/2/?file=659644#file659644line82>
> >
> > how about a similarly derived STREAM loc here, perhaps the same as the FILENAME suggestion using UUID?
See: https://reviews.apache.org/r/27172
> On Sept. 4, 2014, 5:37 p.m., Chris Mattmann wrote:
> > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java, line 30
> > <https://reviews.apache.org/r/22791/diff/2/?file=659646#file659646line30>
> >
> > not sure what this change is?
Added guard clause for ProductStructure, so users cannot end up with an unknown structure. All tests that are in violation of this have been fixed. See: https://reviews.apache.org/r/27172
> On Sept. 4, 2014, 5:37 p.m., Chris Mattmann wrote:
> > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java, line 76
> > <https://reviews.apache.org/r/22791/diff/2/?file=659645#file659645line76>
> >
> > rather than silently do something, maybe take this out until there is something to do?
Done see: https://reviews.apache.org/r/27172
- Michael
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/#review52324
-----------------------------------------------------------
On Oct. 24, 2014, 10:35 p.m., Michael Starch wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22791/
> -----------------------------------------------------------
>
> (Updated Oct. 24, 2014, 10:35 p.m.)
>
>
> Review request for oodt, Lewis McGibbney and Chris Mattmann.
>
>
> Repository: oodt
>
>
> Description
> -------
>
> This patch contains all the changes needed to add in "streaming oodt" into the oodt svn repository.
>
> There are four main portions:
> -Mesos Framework for Resource Manager (Prototype working)
> -Spark Runner for Workflow Manager (Prototype working)
> -Filemanager "streaming" type (In development)
> -Deployment and cluster management scripts (In development)
>
> Where can this stuff be put so that it is available to use, even while it is in development?
>
>
> Note: Filemanager work (and corrections here-in) have been moved to sub-patch review: https://reviews.apache.org/r/27172
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/VersioningUtils.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/resource/pom.xml 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManager.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManagerFactory.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/MesosUtilities.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/OODTExecutor.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosScheduler.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/exception/MesosFrameworkException.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/proto/ResourceProto.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitor.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitorFactory.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/Scheduler.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/proto/resc.proto PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/pom.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/assembly/assembly.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/bin/streamer PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/KafkaPublisher.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/Publisher.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/InputStreamReader.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/Reader.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/StreamEmptyException.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStream.java.bak PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStreamArcheaic.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/system/MultiSourceStreamer.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-actions.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-options.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/logging.properties PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/streamer.properties PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22791/diff/
>
>
> Testing
> -------
>
> Basic functionality tests done for both the resource-manger and workflow manager pieces. Filemanager have been tested to properly ingest a "GenericStream" type with the lucene catalog only.
>
>
> Thanks,
>
> Michael Starch
>
>
Re: Review Request 22791: Streaming OODT Changes
Posted by Michael Starch <st...@umich.edu>.
> On Sept. 4, 2014, 5:37 p.m., Chris Mattmann wrote:
> > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh, line 1
> > <https://reviews.apache.org/r/22791/diff/2/?file=659631#file659631line1>
> >
> > Mike, instead of directly including cluster-tools in oodt, can you simply make:
> >
> > # A Maven AntRun script (build.xml) or something that downloads the cluster-tools as part of the Resource Manager build? We shouldn't have to maintain these scripts in OODT.
Download from where? Do we have a place to upload unmaintained OPS scripts?
- Michael
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/#review52324
-----------------------------------------------------------
On Sept. 4, 2014, 5:23 p.m., Michael Starch wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22791/
> -----------------------------------------------------------
>
> (Updated Sept. 4, 2014, 5:23 p.m.)
>
>
> Review request for oodt, Lewis McGibbney and Chris Mattmann.
>
>
> Repository: oodt
>
>
> Description
> -------
>
> This patch contains all the changes needed to add in "streaming oodt" into the oodt svn repository.
>
> There are four main portions:
> -Mesos Framework for Resource Manager (Prototype working)
> -Spark Runner for Workflow Manager (Prototype working)
> -Filemanager "streaming" type (In development)
> -Deployment and cluster management scripts (In development)
>
> Where can this stuff be put so that it is available to use, even while it is in development?
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/VersioningUtils.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/resource/pom.xml 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManager.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManagerFactory.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/MesosUtilities.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/OODTExecutor.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosScheduler.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/exception/MesosFrameworkException.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/proto/ResourceProto.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitor.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitorFactory.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/Scheduler.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/proto/resc.proto PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/pom.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/assembly/assembly.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/bin/streamer PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/KafkaPublisher.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/Publisher.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/InputStreamReader.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/Reader.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/StreamEmptyException.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStream.java.bak PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStreamArcheaic.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/system/MultiSourceStreamer.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-actions.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-options.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/logging.properties PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/streamer.properties PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22791/diff/
>
>
> Testing
> -------
>
> Basic functionality tests done for both the resource-manger and workflow manager pieces. Filemanager have been tested to properly ingest a "GenericStream" type with the lucene catalog only.
>
>
> Thanks,
>
> Michael Starch
>
>
Re: Review Request 22791: Streaming OODT Changes
Posted by Michael Starch <st...@umich.edu>.
> On Sept. 4, 2014, 5:37 p.m., Chris Mattmann wrote:
> > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh, line 1
> > <https://reviews.apache.org/r/22791/diff/2/?file=659631#file659631line1>
> >
> > Mike, instead of directly including cluster-tools in oodt, can you simply make:
> >
> > # A Maven AntRun script (build.xml) or something that downloads the cluster-tools as part of the Resource Manager build? We shouldn't have to maintain these scripts in OODT.
>
> Michael Starch wrote:
> Download from where? Do we have a place to upload unmaintained OPS scripts?
>
> Chris Mattmann wrote:
> THanks Mike. Ask @paulramirez about how to use Maven AntRun or assembly.xml - you could theoretically reference Mesos if it has a distribution for its scripts in the Central repo - you could ref the dist dependency as a dependency and then unpack them dynamically into a directory. If it doesn't have a Maven dist assembly for the Mesos scripts, then my recommendation would simply be to:
>
> 1. Create a build.xml that downloads (e.g., from Mesos trunk or a tag in Apache git) those scripts into whatever OODT build directory you want (inside of Resource Manager probably makes the most sense)
> 2. Use the Maven Ant-Run plugin to call that build.xml in resource/pom.xml
> 3. Consider using a Maven profile (e.g., mvn -Pstreaming) to insulate your streaming profile cluster-tools download
>
> Michael Starch wrote:
> To be clear...these are scripts I created to setup and run our cluster for use under OODT. They currently just handle the supporting Mesos components, but eventually will startup oodt in a streaming-enabled mode as well. The mesos project knows nothing about them. If we do not want them maintained by OODT then I can remove them from the checkin, however; in my opinion the inculsion of these scripts goes towards making a more ops-friendly OODT package. This is one of the current goals OODT is working toward.
>
> Am I missing something here?
>
> Chris Mattmann wrote:
> Hey Mike, sorry about that: those scripts looked like this:
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=tree;f=bin;h=8497ec15f623cb08cd7a690e79e387be51225c34;hb=HEAD
>
> I thought some of those were Mesos scripts, but they are not, my bad! Can you please add the ALv2 license headers to them at the
> top? Then we're good to go on this part of my review, sorry again!
Moved to: https://reviews.apache.org/r/27179/ Apache headers have been added.
- Michael
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/#review52324
-----------------------------------------------------------
On Oct. 24, 2014, 10:35 p.m., Michael Starch wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22791/
> -----------------------------------------------------------
>
> (Updated Oct. 24, 2014, 10:35 p.m.)
>
>
> Review request for oodt, Lewis McGibbney and Chris Mattmann.
>
>
> Repository: oodt
>
>
> Description
> -------
>
> This patch contains all the changes needed to add in "streaming oodt" into the oodt svn repository.
>
> There are four main portions:
> -Mesos Framework for Resource Manager (Prototype working)
> -Spark Runner for Workflow Manager (Prototype working)
> -Filemanager "streaming" type (In development)
> -Deployment and cluster management scripts (In development)
>
> Where can this stuff be put so that it is available to use, even while it is in development?
>
>
> Note: Filemanager work (and corrections here-in) have been moved to sub-patch review: https://reviews.apache.org/r/27172
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/VersioningUtils.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/resource/pom.xml 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManager.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManagerFactory.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/MesosUtilities.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/OODTExecutor.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosScheduler.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/exception/MesosFrameworkException.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/proto/ResourceProto.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitor.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitorFactory.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/Scheduler.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/proto/resc.proto PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/pom.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/assembly/assembly.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/bin/streamer PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/KafkaPublisher.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/Publisher.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/InputStreamReader.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/Reader.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/StreamEmptyException.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStream.java.bak PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStreamArcheaic.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/system/MultiSourceStreamer.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-actions.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-options.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/logging.properties PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/streamer.properties PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22791/diff/
>
>
> Testing
> -------
>
> Basic functionality tests done for both the resource-manger and workflow manager pieces. Filemanager have been tested to properly ingest a "GenericStream" type with the lucene catalog only.
>
>
> Thanks,
>
> Michael Starch
>
>
Re: Review Request 22791: Streaming OODT Changes
Posted by Michael Starch <st...@umich.edu>.
> On Sept. 4, 2014, 5:37 p.m., Chris Mattmann wrote:
> > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh, line 1
> > <https://reviews.apache.org/r/22791/diff/2/?file=659631#file659631line1>
> >
> > Mike, instead of directly including cluster-tools in oodt, can you simply make:
> >
> > # A Maven AntRun script (build.xml) or something that downloads the cluster-tools as part of the Resource Manager build? We shouldn't have to maintain these scripts in OODT.
>
> Michael Starch wrote:
> Download from where? Do we have a place to upload unmaintained OPS scripts?
>
> Chris Mattmann wrote:
> THanks Mike. Ask @paulramirez about how to use Maven AntRun or assembly.xml - you could theoretically reference Mesos if it has a distribution for its scripts in the Central repo - you could ref the dist dependency as a dependency and then unpack them dynamically into a directory. If it doesn't have a Maven dist assembly for the Mesos scripts, then my recommendation would simply be to:
>
> 1. Create a build.xml that downloads (e.g., from Mesos trunk or a tag in Apache git) those scripts into whatever OODT build directory you want (inside of Resource Manager probably makes the most sense)
> 2. Use the Maven Ant-Run plugin to call that build.xml in resource/pom.xml
> 3. Consider using a Maven profile (e.g., mvn -Pstreaming) to insulate your streaming profile cluster-tools download
To be clear...these are scripts I created to setup and run our cluster for use under OODT. They currently just handle the supporting Mesos components, but eventually will startup oodt in a streaming-enabled mode as well. The mesos project knows nothing about them. If we do not want them maintained by OODT then I can remove them from the checkin, however; in my opinion the inculsion of these scripts goes towards making a more ops-friendly OODT package. This is one of the current goals OODT is working toward.
Am I missing something here?
- Michael
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/#review52324
-----------------------------------------------------------
On Sept. 4, 2014, 5:23 p.m., Michael Starch wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22791/
> -----------------------------------------------------------
>
> (Updated Sept. 4, 2014, 5:23 p.m.)
>
>
> Review request for oodt, Lewis McGibbney and Chris Mattmann.
>
>
> Repository: oodt
>
>
> Description
> -------
>
> This patch contains all the changes needed to add in "streaming oodt" into the oodt svn repository.
>
> There are four main portions:
> -Mesos Framework for Resource Manager (Prototype working)
> -Spark Runner for Workflow Manager (Prototype working)
> -Filemanager "streaming" type (In development)
> -Deployment and cluster management scripts (In development)
>
> Where can this stuff be put so that it is available to use, even while it is in development?
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/VersioningUtils.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/resource/pom.xml 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManager.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManagerFactory.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/MesosUtilities.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/OODTExecutor.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosScheduler.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/exception/MesosFrameworkException.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/proto/ResourceProto.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitor.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitorFactory.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/Scheduler.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/proto/resc.proto PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/pom.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/assembly/assembly.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/bin/streamer PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/KafkaPublisher.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/Publisher.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/InputStreamReader.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/Reader.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/StreamEmptyException.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStream.java.bak PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStreamArcheaic.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/system/MultiSourceStreamer.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-actions.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-options.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/logging.properties PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/streamer.properties PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22791/diff/
>
>
> Testing
> -------
>
> Basic functionality tests done for both the resource-manger and workflow manager pieces. Filemanager have been tested to properly ingest a "GenericStream" type with the lucene catalog only.
>
>
> Thanks,
>
> Michael Starch
>
>
Re: Review Request 22791: Streaming OODT Changes
Posted by Michael Starch <st...@umich.edu>.
> On Sept. 4, 2014, 5:37 p.m., Chris Mattmann wrote:
> > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java, line 79
> > <https://reviews.apache.org/r/22791/diff/2/?file=659644#file659644line79>
> >
> > how about instead of NA, for the filename, we call it a "stream-UUID" where we generate a unique stream UUID as the FILENAME field.
Shifted this work to sub-patch: https://reviews.apache.org/r/27172
- Michael
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/#review52324
-----------------------------------------------------------
On Sept. 4, 2014, 5:23 p.m., Michael Starch wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22791/
> -----------------------------------------------------------
>
> (Updated Sept. 4, 2014, 5:23 p.m.)
>
>
> Review request for oodt, Lewis McGibbney and Chris Mattmann.
>
>
> Repository: oodt
>
>
> Description
> -------
>
> This patch contains all the changes needed to add in "streaming oodt" into the oodt svn repository.
>
> There are four main portions:
> -Mesos Framework for Resource Manager (Prototype working)
> -Spark Runner for Workflow Manager (Prototype working)
> -Filemanager "streaming" type (In development)
> -Deployment and cluster management scripts (In development)
>
> Where can this stuff be put so that it is available to use, even while it is in development?
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/VersioningUtils.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/resource/pom.xml 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManager.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManagerFactory.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/MesosUtilities.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/OODTExecutor.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosScheduler.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/exception/MesosFrameworkException.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/proto/ResourceProto.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitor.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitorFactory.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/Scheduler.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/proto/resc.proto PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/pom.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/assembly/assembly.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/bin/streamer PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/KafkaPublisher.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/Publisher.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/InputStreamReader.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/Reader.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/StreamEmptyException.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStream.java.bak PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStreamArcheaic.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/system/MultiSourceStreamer.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-actions.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-options.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/logging.properties PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/streamer.properties PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22791/diff/
>
>
> Testing
> -------
>
> Basic functionality tests done for both the resource-manger and workflow manager pieces. Filemanager have been tested to properly ingest a "GenericStream" type with the lucene catalog only.
>
>
> Thanks,
>
> Michael Starch
>
>
Re: Review Request 22791: Streaming OODT Changes
Posted by Chris Mattmann <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/#review52324
-----------------------------------------------------------
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh
<https://reviews.apache.org/r/22791/#comment91085>
Mike, instead of directly including cluster-tools in oodt, can you simply make:
# A Maven AntRun script (build.xml) or something that downloads the cluster-tools as part of the Resource Manager build? We shouldn't have to maintain these scripts in OODT.
http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml
<https://reviews.apache.org/r/22791/#comment91086>
Custom Maven repos are difficult since the Central repository is phasing them out:
http://blog.sonatype.com/2010/03/why-external-repos-are-being-phased-out-of-central/
If we absolutely need to ref this repo, can you make it a Maven profile, not enabled by default, so that users won't have the issue for this when downloading and building OODT?
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java
<https://reviews.apache.org/r/22791/#comment91087>
how about instead of NA, for the filename, we call it a "stream-UUID" where we generate a unique stream UUID as the FILENAME field.
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java
<https://reviews.apache.org/r/22791/#comment91088>
how about a similarly derived STREAM loc here, perhaps the same as the FILENAME suggestion using UUID?
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java
<https://reviews.apache.org/r/22791/#comment91089>
rather than silently do something, maybe take this out until there is something to do?
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java
<https://reviews.apache.org/r/22791/#comment91090>
not sure what this change is?
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java
<https://reviews.apache.org/r/22791/#comment91091>
All of these files need ALv2 license headers.
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/proto/ResourceProto.java
<https://reviews.apache.org/r/22791/#comment91092>
We shoudln't be checking in generated files - can this simply be generated during the Resmgr build?
http://svn.apache.org/repos/asf/oodt/trunk/streamer/pom.xml
<https://reviews.apache.org/r/22791/#comment91093>
if we don't need this, remove it.
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStream.java.bak
<https://reviews.apache.org/r/22791/#comment91094>
ALv2 headers here and everywhere
http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/streamer.properties
<https://reviews.apache.org/r/22791/#comment91095>
remove JPL specific URLs.
- Chris Mattmann
On Sept. 4, 2014, 5:23 p.m., Michael Starch wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22791/
> -----------------------------------------------------------
>
> (Updated Sept. 4, 2014, 5:23 p.m.)
>
>
> Review request for oodt, Lewis McGibbney and Chris Mattmann.
>
>
> Repository: oodt
>
>
> Description
> -------
>
> This patch contains all the changes needed to add in "streaming oodt" into the oodt svn repository.
>
> There are four main portions:
> -Mesos Framework for Resource Manager (Prototype working)
> -Spark Runner for Workflow Manager (Prototype working)
> -Filemanager "streaming" type (In development)
> -Deployment and cluster management scripts (In development)
>
> Where can this stuff be put so that it is available to use, even while it is in development?
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/VersioningUtils.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/resource/pom.xml 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManager.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManagerFactory.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/MesosUtilities.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/OODTExecutor.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosScheduler.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/exception/MesosFrameworkException.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/proto/ResourceProto.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitor.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitorFactory.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/Scheduler.java 1617800
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/proto/resc.proto PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/pom.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/assembly/assembly.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/bin/streamer PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/KafkaPublisher.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/Publisher.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/InputStreamReader.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/Reader.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/StreamEmptyException.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStream.java.bak PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStreamArcheaic.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/system/MultiSourceStreamer.java PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-actions.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-options.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/logging.properties PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/streamer.properties PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22791/diff/
>
>
> Testing
> -------
>
> Basic functionality tests done for both the resource-manger and workflow manager pieces. Filemanager have been tested to properly ingest a "GenericStream" type with the lucene catalog only.
>
>
> Thanks,
>
> Michael Starch
>
>