Resource Manager Info.

["Iwunze, Michael C (GSFC-4700)[NOAA-JPSS]" <mi...@nasa.gov>]

Hello,

         I am one of the GRAVITE developers that is new to OODT. Currently researching the resource manager component, I saw the user guide online. Do you know of any other locations were I can get more information on this component as well as more examples and how it can be used?

Thanks

Mike

Re: Resource Manager Info.

[Bruce Barkstrom <br...@gmail.com>]

Have you tried drawing up a Gantt chart that shows the
jobs as activities on a time line?  I think that would be
a revealing action.

Bruce B.

On Wed, Apr 18, 2012 at 10:05 AM, Cameron Goodale <si...@gmail.com> wrote:
> Hey Mike,
>
> I don't have any formal documentation on the Resource Manager, but I can
> try to explain how the Snow Data System team is using it to process large
> amounts of MODIS MOD09GA tile data.  From there it might be easier to just
> ask additional questions since the Components are a lot like legos and you
> can swap them around to make several different systems in the end.
>
> On Snow (Snow Data System Project) we have 5 Servers (1 VM, and 4 Work
> Horses with 48 Cores Each).  The Workflow Manager and Resource Manager are
> installed on the VM and it is sort of like a head node where it does all
> the Job Creation and Submission, but it does none of the tile processing.
>  Here is a list of high-level setups/config that enable us to run the
> MOSCAG algorithm across 3 of the 4 Work Horse Machines (the 4th Machine is
> used for another processing pipeline, so we will ignore it for now).
>
>
>   1. On the Head Node (VM) we have a Workflow Manager (WM) configured to
>   Create MODSCAG Workflows.
>   2. On the Head Node we also have a Resource Manager (RM) configured to
>   send jobs out to BatchStubs on the 3 Work Horse Machines.
>   3. On the Head Node the WM is configured to USE the RM.  This means the
>   WM will submit jobs to the RM until the RM has reached Capacity, then the
>   Workflow Manager will queue any additional jobs and wait for the RM to have
>   additional Capacity by completing a Job.
>   4. Our MODSCAG Jobs run in a Quad Core Process, so we have configured
>   the RM to only submit 10 Jobs to each Work Horse machine (hence using 40 of
>   the 48 cores).  This is unique to the image processing code base, your
>   mileage (and core usage) may vary.
>
> So I hope this helps.  Again your configuration might be that you have 100
> 4-core machines that you want the RM to use for job processing, or you may
> want to stand up BatchStubs on your localhost for testing purposes.  At
> that point it really is just a matter of getting the configuration setup
> and restarting the components.
>
> If anyone else in the community has anything to add or comment, that would
> be great.  And Mike if you have any questions at all, don't hesitate to ask.
>
> Best Regards,
>
>
> Cameron
>
>
> On Tue, Apr 17, 2012 at 9:52 AM, Iwunze, Michael C (GSFC-4700)[NOAA-JPSS] <
> michael.iwunze@nasa.gov> wrote:
>
>> Hello,
>>
>>         I am one of the GRAVITE developers that is new to OODT. Currently
>> researching the resource manager component, I saw the user guide online. Do
>> you know of any other locations were I can get more information on this
>> component as well as more examples and how it can be used?
>>
>> Thanks
>>
>> Mike
>>
>
>
>
> --
>
> Sent from a Tin Can attached to a String

Re: Resource Manager Info.

[Cameron Goodale <si...@gmail.com>]

Hey Mike,

I don't have any formal documentation on the Resource Manager, but I can
try to explain how the Snow Data System team is using it to process large
amounts of MODIS MOD09GA tile data.  From there it might be easier to just
ask additional questions since the Components are a lot like legos and you
can swap them around to make several different systems in the end.

On Snow (Snow Data System Project) we have 5 Servers (1 VM, and 4 Work
Horses with 48 Cores Each).  The Workflow Manager and Resource Manager are
installed on the VM and it is sort of like a head node where it does all
the Job Creation and Submission, but it does none of the tile processing.
 Here is a list of high-level setups/config that enable us to run the
MOSCAG algorithm across 3 of the 4 Work Horse Machines (the 4th Machine is
used for another processing pipeline, so we will ignore it for now).


   1. On the Head Node (VM) we have a Workflow Manager (WM) configured to
   Create MODSCAG Workflows.
   2. On the Head Node we also have a Resource Manager (RM) configured to
   send jobs out to BatchStubs on the 3 Work Horse Machines.
   3. On the Head Node the WM is configured to USE the RM.  This means the
   WM will submit jobs to the RM until the RM has reached Capacity, then the
   Workflow Manager will queue any additional jobs and wait for the RM to have
   additional Capacity by completing a Job.
   4. Our MODSCAG Jobs run in a Quad Core Process, so we have configured
   the RM to only submit 10 Jobs to each Work Horse machine (hence using 40 of
   the 48 cores).  This is unique to the image processing code base, your
   mileage (and core usage) may vary.

So I hope this helps.  Again your configuration might be that you have 100
4-core machines that you want the RM to use for job processing, or you may
want to stand up BatchStubs on your localhost for testing purposes.  At
that point it really is just a matter of getting the configuration setup
and restarting the components.

If anyone else in the community has anything to add or comment, that would
be great.  And Mike if you have any questions at all, don't hesitate to ask.

Best Regards,


Cameron


On Tue, Apr 17, 2012 at 9:52 AM, Iwunze, Michael C (GSFC-4700)[NOAA-JPSS] <
michael.iwunze@nasa.gov> wrote:

> Hello,
>
>         I am one of the GRAVITE developers that is new to OODT. Currently
> researching the resource manager component, I saw the user guide online. Do
> you know of any other locations were I can get more information on this
> component as well as more examples and how it can be used?
>
> Thanks
>
> Mike
>



-- 

Sent from a Tin Can attached to a String