You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@vcl.apache.org by Dmitri Chebotarov <dc...@gmu.edu> on 2013/05/17 21:32:30 UTC
HPC /w VCL
Hi
Is there a manual which describes how make HPC reservations with VCL ?
I see that code/database mentions 'hpc' and 'vclhpc' projects and would like to get more info on the subj if possible.
--
Thank you,
Dmitri Chebotarov
VCL Sys Eng, Engineering & Architectural Support, TSD - Ent Servers & Messaging
223 Aquia Building, Ffx, MSN: 1B5
Phone: (703) 993-6175 | Fax: (703) 993-3404
Re: HPC /w VCL
Posted by Dmitri Chebotarov <dc...@gmu.edu>.
Thank you.
On May 21, 2013, at 11:19 , Aaron Peeler <aa...@ncsu.edu>
wrote:
> There's currently not a description on how to do this.
>
> The HPC run here at NCSU uses LSF and is static. It's static mainly
> because our researchers consume everything and the jobs are usually
> quite long > 2weeks on average.
>
> We do (where) feasible migrate VCL blades to HPC mode during breaks,
> but this is usually for the summer. This is done somewhat by using the
> 'hpc' project listed in the VCL database, by starting a reload of the
> blade with a HPC compute node image. Which triggers a reload by xCAT
> and sets a predifined VLAN in one of the blade center's Ethernet
> Switch module to use the HPC vlan as the primary vlan id.
>
> There are some folks are using VCL to build short term clusters for
> such things. This is done using the clustering in the manage images
> edit image profiles tool. I think UTEP is doing this today in their
> VCL instance/cluster. Several researchers here at NCSU also use the
> image cluster feature to do this, mainly for running hadoop.
>
> Also Andy K. might be able to elaborate more, but I think with his
> work on the Configuration management piece could better orchestrate
> adding and removing nodes from a compute pool.
>
> Aaron
>
>
>
> On Fri, May 17, 2013 at 3:32 PM, Dmitri Chebotarov <dc...@gmu.edu> wrote:
>> Hi
>> Is there a manual which describes how make HPC reservations with VCL ?
>> I see that code/database mentions 'hpc' and 'vclhpc' projects and would like to get more info on the subj if possible.
>>
>> --
>> Thank you,
>>
>> Dmitri Chebotarov
>> VCL Sys Eng, Engineering & Architectural Support, TSD - Ent Servers & Messaging
>> 223 Aquia Building, Ffx, MSN: 1B5
>> Phone: (703) 993-6175 | Fax: (703) 993-3404
>>
>>
>>
>
>
>
> --
> Aaron Peeler
> Program Manager
> Virtual Computing Lab
> NC State University
>
> All electronic mail messages in connection with State business which
> are sent to or received by this account are subject to the NC Public
> Records Law and may be disclosed to third parties.
>
--
Thank you,
Dmitri Chebotarov
VCL Sys Eng, Engineering & Architectural Support, TSD - Ent Servers & Messaging
223 Aquia Building, Ffx, MSN: 1B5
Phone: (703) 993-6175 | Fax: (703) 993-3404
Re: HPC /w VCL
Posted by Aaron Peeler <aa...@ncsu.edu>.
There's currently not a description on how to do this.
The HPC run here at NCSU uses LSF and is static. It's static mainly
because our researchers consume everything and the jobs are usually
quite long > 2weeks on average.
We do (where) feasible migrate VCL blades to HPC mode during breaks,
but this is usually for the summer. This is done somewhat by using the
'hpc' project listed in the VCL database, by starting a reload of the
blade with a HPC compute node image. Which triggers a reload by xCAT
and sets a predifined VLAN in one of the blade center's Ethernet
Switch module to use the HPC vlan as the primary vlan id.
There are some folks are using VCL to build short term clusters for
such things. This is done using the clustering in the manage images
edit image profiles tool. I think UTEP is doing this today in their
VCL instance/cluster. Several researchers here at NCSU also use the
image cluster feature to do this, mainly for running hadoop.
Also Andy K. might be able to elaborate more, but I think with his
work on the Configuration management piece could better orchestrate
adding and removing nodes from a compute pool.
Aaron
On Fri, May 17, 2013 at 3:32 PM, Dmitri Chebotarov <dc...@gmu.edu> wrote:
> Hi
> Is there a manual which describes how make HPC reservations with VCL ?
> I see that code/database mentions 'hpc' and 'vclhpc' projects and would like to get more info on the subj if possible.
>
> --
> Thank you,
>
> Dmitri Chebotarov
> VCL Sys Eng, Engineering & Architectural Support, TSD - Ent Servers & Messaging
> 223 Aquia Building, Ffx, MSN: 1B5
> Phone: (703) 993-6175 | Fax: (703) 993-3404
>
>
>
--
Aaron Peeler
Program Manager
Virtual Computing Lab
NC State University
All electronic mail messages in connection with State business which
are sent to or received by this account are subject to the NC Public
Records Law and may be disclosed to third parties.
Re: HPC /w VCL
Posted by Henry Schaffer <he...@ncsu.edu>.
I think there are two separate areas which this question might be dealing
with.
1) Moving nodes/blades back and forth between HPC (we run our HPC as a
parallel computing, distributed memory system batch computing with LSF as
the queue manager) and VCL. Since we have racks which can do either, we
actually reassign nodes/blades to one pool or the other depending on demand
- and on the suitability of the hardware - we tend to keep the lower
performance hardware in the VCL. I think we actually move a chassis at a
time.
We don't do this for short time changes (e.g. weekends) - so it is
primarily the summer - when the VCL demand is down because of fewer courses
and the graduate students and faculty demand is up - because of fewer
courses. :-) I think we also used to do this for Winter break - but I don't
know if we do that now - the problem being that HPC jobs are running longer
and longer, and Winter break isn't long enough to accommodate many of these
long jobs. Still, this reassignment significantly adds to the % of the time
that the equipment is utilized and so gets more production out of our
investment.
What is involved is primarily reconfiguring the communication switches so
that either:
a) the VCL and its management nodes "see" the chassis and then they start
assigning reservations to it - or in the other direction
b) the LSF "sees" the chassis and assigns it to its queues
This is done by remotely reconfiguring the switches, no physical wiring
changes need to be done.
2) Working *within* the VCL to assign a cluster. This could be used for HPC
production computation, but most often it's used for experimenting with
HPC. Sometimes people want to tinker with the OS - and in our HPC area they
don't get to do that - they get our vanilla Linux OS. Sometimes, e.g. when
developing code, they want to interact with nodes in their cluster to find
out what's going on, and they don't get to do that in our *batch* HPC area.
There are many other "one off" times when this can be useful - and the VCL
allows making a Reservation for a "cluster" if the person has the correct
privileges.
Aaron Peeler has told me that creating this type of VCL cluster is also
called adding subimages and that there is documentation at:
https://cwiki.apache.org/VCL/creating-a-cluster-enviroment.html
I'm not sure what UTEP is doing with their VCL in this cluster area -
perhaps someone from there can fill us in.
Also, if I got anything above wrong or omitted something relevant/important
- please fill us in.
--henry schaffer
On Fri, May 17, 2013 at 3:32 PM, Dmitri Chebotarov <dc...@gmu.edu> wrote:
> Hi
> Is there a manual which describes how make HPC reservations with VCL ?
> I see that code/database mentions 'hpc' and 'vclhpc' projects and would
> like to get more info on the subj if possible.
>
> --
> Thank you,
>
> Dmitri Chebotarov
> VCL Sys Eng, Engineering & Architectural Support, TSD - Ent Servers &
> Messaging
> 223 Aquia Building, Ffx, MSN: 1B5
> Phone: (703) 993-6175 | Fax: (703) 993-3404
>
>
>
>
Re: HPC /w VCL
Posted by Henry Schaffer <he...@ncsu.edu>.
I think there are two separate areas which this question might be dealing
with.
1) Moving nodes/blades back and forth between HPC (we run our HPC as a
parallel computing, distributed memory system batch computing with LSF as
the queue manager) and VCL. Since we have racks which can do either, we
actually reassign nodes/blades to one pool or the other depending on demand
- and on the suitability of the hardware - we tend to keep the lower
performance hardware in the VCL. I think we actually move a chassis at a
time.
We don't do this for short time changes (e.g. weekends) - so it is
primarily the summer - when the VCL demand is down because of fewer courses
and the graduate students and faculty demand is up - because of fewer
courses. :-) I think we also used to do this for Winter break - but I don't
know if we do that now - the problem being that HPC jobs are running longer
and longer, and Winter break isn't long enough to accommodate many of these
long jobs. Still, this reassignment significantly adds to the % of the time
that the equipment is utilized and so gets more production out of our
investment.
What is involved is primarily reconfiguring the communication switches so
that either:
a) the VCL and its management nodes "see" the chassis and then they start
assigning reservations to it - or in the other direction
b) the LSF "sees" the chassis and assigns it to its queues
This is done by remotely reconfiguring the switches, no physical wiring
changes need to be done.
2) Working *within* the VCL to assign a cluster. This could be used for HPC
production computation, but most often it's used for experimenting with
HPC. Sometimes people want to tinker with the OS - and in our HPC area they
don't get to do that - they get our vanilla Linux OS. Sometimes, e.g. when
developing code, they want to interact with nodes in their cluster to find
out what's going on, and they don't get to do that in our *batch* HPC area.
There are many other "one off" times when this can be useful - and the VCL
allows making a Reservation for a "cluster" if the person has the correct
privileges.
Aaron Peeler has told me that creating this type of VCL cluster is also
called adding subimages and that there is documentation at:
https://cwiki.apache.org/VCL/creating-a-cluster-enviroment.html
I'm not sure what UTEP is doing with their VCL in this cluster area -
perhaps someone from there can fill us in.
Also, if I got anything above wrong or omitted something relevant/important
- please fill us in.
--henry schaffer
On Fri, May 17, 2013 at 3:32 PM, Dmitri Chebotarov <dc...@gmu.edu> wrote:
> Hi
> Is there a manual which describes how make HPC reservations with VCL ?
> I see that code/database mentions 'hpc' and 'vclhpc' projects and would
> like to get more info on the subj if possible.
>
> --
> Thank you,
>
> Dmitri Chebotarov
> VCL Sys Eng, Engineering & Architectural Support, TSD - Ent Servers &
> Messaging
> 223 Aquia Building, Ffx, MSN: 1B5
> Phone: (703) 993-6175 | Fax: (703) 993-3404
>
>
>
>