You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airavata.apache.org by Emre Brookes <em...@mso.umt.edu> on 2023/01/05 15:58:34 UTC
Re: Airavata MetaSchedular
Hi Eroma,
Hope you had good holidays and Happy 2023!
When can we schedule our first meeting?
Thanks,
Emre
Eroma Abeysinghe wrote:
> Hi Emre,
>
> Yes, We'll meet with you in January to discuss the implementation and
> testing with UltraScan.
> Currently, the initial testing is being planned for the implementation.
>
> Thanks,
> Eroma
>
>
>
> On Thu, Dec 8, 2022 at 9:07 AM Emre Brookes <emre.brookes@mso.umt.edu
> <ma...@mso.umt.edu>> wrote:
>
> Hi Isuru,
>
> Thanks for the updates. Great to see progress is being made.
> I plan to dive into the UltraScan side of this starting in January,
> perhaps we can setup a meeting early January to see what is
> implemented,
> how the new UltraScan microservice can be integrated & begin
> planning/running some tests?
>
> Best wishes,
> Emre
>
>
> Isuru Ranawaka wrote:
> > *EXTERNAL EMAIL*
> >
> > Hi all,
> >
> > We have implemented the initial phase of the Airavata
> Metascheduler.
> > Following is the finalized architecture at this phase.
> >
> > image.png
> > Airavata Experiments can be configured for auto-scheduling and for
> > such Experiments Orchestrator invokes meta scheduler to check
> whether
> > the Experiment is schedulable. If it is schedulable Orchestrator
> > submits the respective Process of the Experiment to Helix Framework
> > for instant scheduling otherwise Experiment and Process are Queued.
> >
> >
> > *Metascheduler Architecture*
> >
> > image.png
> >
> > Process scanner scans for already queued
> Processes
> > periodically and sends those Process information to the
> MetaScheduler
> > Engine. Metascheduler Engine evaluates the chance of successful
> > submission based on configured scheduling policies. If it is
> > submittable MetaScheduler engine invokes Orchestrator APIs to
> initiate
> > the Job Submission for those Processes through Helix Framework.
> >
> > * Compute Resource Monitoring Service
> >
> > Compute resource monitoring service uses Airavata Group Resource
> > Profile information to periodically monitor configured Remote
> Compute
> > Resources and update the status of the Queues in the registry. So
> > the information is consumed by Metacheduler Engine to schedule or
> > reschedule Processes.
> >
> >
> >
> > thanks
> > Isuru
> >
> >
> >
> >
> > On Tue, Oct 4, 2022 at 7:41 AM Isuru Ranawaka
> <irjanith@gmail.com <ma...@gmail.com>
> > <mailto:irjanith@gmail.com <ma...@gmail.com>>> wrote:
> >
> > Hi all,
> >
> > Airavata MetaSchedular is a managed job submission framework for
> > High-Performance Computing resources. Basically, It handles
> >
> > *
> >
> > User-based load balancing of Jobs between resource queues.
> >
> > *
> >
> > Dynamic Job scheduling across compute resources
> >
> > *
> >
> > Job queueing and managed submissions
> >
> > and the following document [1] contains the captured use cases.
> > Please have a look and make suggestions, comments are highly
> > appreciated.
> > I will update the thread as I proceed with the implementation.
> >
> > thanks
> > Isuru
> >
> >
> >
> >
> [1]https://docs.google.com/document/d/1i_cAR7ulNpkEATt0VW4-xoYZyZTQbdnNAJVbnAlMI4I/edit?usp=sharing
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F1i_cAR7ulNpkEATt0VW4-xoYZyZTQbdnNAJVbnAlMI4I%2Fedit%3Fusp%3Dsharing&data=05%7C01%7Cemre.brookes%40mso.umt.edu%7Cbe901cb022944847525b08dad9273cd3%7C68407ce503da49ffaf0a724be0d37c9d%7C0%7C0%7C638061059806579260%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nS4gzBpdmOWF04SLIpJeCwZEAjDsEPipQzUsaVz7L30%3D&reserved=0>
> >
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F1i_cAR7ulNpkEATt0VW4-xoYZyZTQbdnNAJVbnAlMI4I%2Fedit%3Fusp%3Dsharing&data=05%7C01%7Cemre.brookes%40mso.umt.edu%7Cef364bd1eacf40f9084708dad7c05f78%7C68407ce503da49ffaf0a724be0d37c9d%7C0%7C0%7C638059518517375866%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=TakGZiJUBDCypMJMjsD0yNrjayREybgM8JBtQVu9hJM%3D&reserved=0
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F1i_cAR7ulNpkEATt0VW4-xoYZyZTQbdnNAJVbnAlMI4I%2Fedit%3Fusp%3Dsharing&data=05%7C01%7Cemre.brookes%40mso.umt.edu%7Cbe901cb022944847525b08dad9273cd3%7C68407ce503da49ffaf0a724be0d37c9d%7C0%7C0%7C638061059806735502%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=sbHtACYIZgKqCyfBJDM0uuyNkg7xW8c5SgNPkGHcQv4%3D&reserved=0>>
> >
> >
> > --
> > Research Software Engineer
> > Indiana University, IN
> >
> >
> >
> > --
> > Research Software Engineer
> > Indiana University, IN
> >
> >
> > CAUTION: This message originated outside of UT Health San Antonio.
> > Please exercise caution when clicking on links or opening
> attachments.
>
>
>
> --
> Thank You,
> Best Regards,
> Eroma
Re: Airavata MetaSchedular
Posted by Eroma Abeysinghe <er...@gmail.com>.
Hi Emre,
Happy New Year !!!
We are carrying out our planning activities, so let me check our
schedules and send you a time.
Thanks,
Eroma
On Thu, Jan 5, 2023 at 10:58 AM Emre Brookes <em...@mso.umt.edu>
wrote:
> Hi Eroma,
>
> Hope you had good holidays and Happy 2023!
>
> When can we schedule our first meeting?
>
> Thanks,
> Emre
>
>
> Eroma Abeysinghe wrote:
> > Hi Emre,
> >
> > Yes, We'll meet with you in January to discuss the implementation and
> > testing with UltraScan.
> > Currently, the initial testing is being planned for the implementation.
> >
> > Thanks,
> > Eroma
> >
> >
> >
> > On Thu, Dec 8, 2022 at 9:07 AM Emre Brookes <emre.brookes@mso.umt.edu
> > <ma...@mso.umt.edu>> wrote:
> >
> > Hi Isuru,
> >
> > Thanks for the updates. Great to see progress is being made.
> > I plan to dive into the UltraScan side of this starting in January,
> > perhaps we can setup a meeting early January to see what is
> > implemented,
> > how the new UltraScan microservice can be integrated & begin
> > planning/running some tests?
> >
> > Best wishes,
> > Emre
> >
> >
> > Isuru Ranawaka wrote:
> > > *EXTERNAL EMAIL*
> > >
> > > Hi all,
> > >
> > > We have implemented the initial phase of the Airavata
> > Metascheduler.
> > > Following is the finalized architecture at this phase.
> > >
> > > image.png
> > > Airavata Experiments can be configured for auto-scheduling and for
> > > such Experiments Orchestrator invokes meta scheduler to check
> > whether
> > > the Experiment is schedulable. If it is schedulable Orchestrator
> > > submits the respective Process of the Experiment to Helix Framework
> > > for instant scheduling otherwise Experiment and Process are Queued.
> > >
> > >
> > > *Metascheduler Architecture*
> > >
> > > image.png
> > >
> > > Process scanner scans for already queued
> > Processes
> > > periodically and sends those Process information to the
> > MetaScheduler
> > > Engine. Metascheduler Engine evaluates the chance of successful
> > > submission based on configured scheduling policies. If it is
> > > submittable MetaScheduler engine invokes Orchestrator APIs to
> > initiate
> > > the Job Submission for those Processes through Helix Framework.
> > >
> > > * Compute Resource Monitoring Service
> > >
> > > Compute resource monitoring service uses Airavata Group Resource
> > > Profile information to periodically monitor configured Remote
> > Compute
> > > Resources and update the status of the Queues in the registry. So
> > > the information is consumed by Metacheduler Engine to schedule or
> > > reschedule Processes.
> > >
> > >
> > >
> > > thanks
> > > Isuru
> > >
> > >
> > >
> > >
> > > On Tue, Oct 4, 2022 at 7:41 AM Isuru Ranawaka
> > <irjanith@gmail.com <ma...@gmail.com>
> > > <mailto:irjanith@gmail.com <ma...@gmail.com>>> wrote:
> > >
> > > Hi all,
> > >
> > > Airavata MetaSchedular is a managed job submission framework
> for
> > > High-Performance Computing resources. Basically, It handles
> > >
> > > *
> > >
> > > User-based load balancing of Jobs between resource queues.
> > >
> > > *
> > >
> > > Dynamic Job scheduling across compute resources
> > >
> > > *
> > >
> > > Job queueing and managed submissions
> > >
> > > and the following document [1] contains the captured use cases.
> > > Please have a look and make suggestions, comments are highly
> > > appreciated.
> > > I will update the thread as I proceed with the implementation.
> > >
> > > thanks
> > > Isuru
> > >
> > >
> > >
> > >
> > [1]
> https://docs.google.com/document/d/1i_cAR7ulNpkEATt0VW4-xoYZyZTQbdnNAJVbnAlMI4I/edit?usp=sharing
> > <
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F1i_cAR7ulNpkEATt0VW4-xoYZyZTQbdnNAJVbnAlMI4I%2Fedit%3Fusp%3Dsharing&data=05%7C01%7Cemre.brookes%40mso.umt.edu%7Cbe901cb022944847525b08dad9273cd3%7C68407ce503da49ffaf0a724be0d37c9d%7C0%7C0%7C638061059806579260%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nS4gzBpdmOWF04SLIpJeCwZEAjDsEPipQzUsaVz7L30%3D&reserved=0
> >
> > >
> > <
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F1i_cAR7ulNpkEATt0VW4-xoYZyZTQbdnNAJVbnAlMI4I%2Fedit%3Fusp%3Dsharing&data=05%7C01%7Cemre.brookes%40mso.umt.edu%7Cef364bd1eacf40f9084708dad7c05f78%7C68407ce503da49ffaf0a724be0d37c9d%7C0%7C0%7C638059518517375866%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=TakGZiJUBDCypMJMjsD0yNrjayREybgM8JBtQVu9hJM%3D&reserved=0
> > <
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F1i_cAR7ulNpkEATt0VW4-xoYZyZTQbdnNAJVbnAlMI4I%2Fedit%3Fusp%3Dsharing&data=05%7C01%7Cemre.brookes%40mso.umt.edu%7Cbe901cb022944847525b08dad9273cd3%7C68407ce503da49ffaf0a724be0d37c9d%7C0%7C0%7C638061059806735502%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=sbHtACYIZgKqCyfBJDM0uuyNkg7xW8c5SgNPkGHcQv4%3D&reserved=0
> >>
> > >
> > >
> > > --
> > > Research Software Engineer
> > > Indiana University, IN
> > >
> > >
> > >
> > > --
> > > Research Software Engineer
> > > Indiana University, IN
> > >
> > >
> > > CAUTION: This message originated outside of UT Health San Antonio.
> > > Please exercise caution when clicking on links or opening
> > attachments.
> >
> >
> >
> > --
> > Thank You,
> > Best Regards,
> > Eroma
>
>
--
Thank You,
Best Regards,
Eroma