You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@vcl.apache.org by Josh Thompson <jo...@ncsu.edu> on 2009/02/04 22:39:30 UTC
frontend API for block reservations (VCL-78)
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
I'm going to start work on VCL-78 which is an addition to the frontend XML RPC
API to allow the backend to call the frontend for allocating computers for
block reservations. Currently, the block reservations are created in the
frontend, but computers are not allocated to fulfill them because the
requested time(s) could be at any point in the future. So, around 6 hours
before a requested time, the backend picks up that block reservation time
slot, allocates computers for it, and then inserts reload reservations early
enough for the computers to be preloaded by the start of the time slot.
This results in a fair amount of duplicated code, since the frontend currently
handles computer allocation in all other cases.
This is how I am planning on implementing it. Please reply with any comments
on doing it differently or tweaking it somehow. I'm open to any suggestions.
The backend could just call the frontend with the blockTimes id to process,
but I'll make it more generic to be more useful in other cases and require
that the backend specify several items.
function name: XMLRPCblockAllocation
parameters:
imageid - id of the image to be used
start - unix timestamp for the start time (i.e. machines should be prep'd
and ready by this time)
end - unix timestamp for the end time
machinecount - number of computers to allocate
usergroupid - id of user group for checking user access to machines
ignoreprivileges - 0 (false) or 1 (true) - set to 1 to select computers from
any that are mapped to be able to run the image; set to 0 to only select
computers from ones that are both mapped and that users in usergroupid have
been granted access to through the privilege tree. If this is set to 1,
usergroupid is ignored (more on this below).
blockTimeid - id from blockTimes table that this will fulfill or 0 if there
is not an existing block reservation related to this call
returns: the number of machines that were successfully allocated, with 0
indicating a complete failure of not being able to allocate any computers
Description of how it will work:
The frontend will use the normal scheduling functions to allocate computers
and insert them into the blockComputers table. If blockTimeid is 0, new
entries will be created in the blockRequest and blockTimes tables. It will
create reload reservations for the computers early enough for them to be
loaded by the start time based on historical loading times.
The ignoreprivileges flag exists to allow more machines to be available for
fulfilling block reservations than just those that the user group has access
to. The current implementation with the backend doing the processing
functions as if ignoreprivileges were set to 1.
That's pretty much it.
Josh
- --
- -------------------------------
Josh Thompson
Systems Programmer
Virtual Computing Lab (VCL)
North Carolina State University
Josh_Thompson@ncsu.edu
919-515-5323
my GPG/PGP key can be found at pgp.mit.edu
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
iD8DBQFJigsTV/LQcNdtPQMRAvSuAJkBLrJnfgC+VD/MFVcg6HIM3OIb9QCdGVkL
NK5Mjyap5+oeemdm40DEdaQ=
=e3CH
-----END PGP SIGNATURE-----
Re: frontend API for block reservations (VCL-78)
Posted by Aaron Peeler <aa...@ncsu.edu>.
never mind this response
I overlooked the part about keeping the first function which provides the
ability to create a block reservation through the API.
Aaron
--On February 5, 2009 10:33:40 AM -0500 Aaron Peeler
<aa...@ncsu.edu> wrote:
> That would work.
>
> But now you loose the ability to create new block allocations through the
> API. Previously it created one if there was not an existing entry.
>
> So I guess we need to ask is this a desired/separate feature, the ability
> create block reservations through the API.
>
>
>
> --On February 5, 2009 10:18:17 AM -0500 Josh Thompson
> <jo...@ncsu.edu> wrote:
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> For the frontend, one function works as well as two would. However, I
>> can see how it would simplify code on the backend to just pass a
>> blockTimes id since it wouldn't have to determine all of the other
>> information (blockTimes entries include blockrequestids; so, the
>> blockrequestid wouldn't need to be passed as well).
>>
>> So, how about this, keep the function already described except drop
>> the 'blockTimeid' parameter and add this function as well:
>>
>> function: XMLRPCproccessBlockTime
>>
>> parameters:
>> blockTimesid - id from the blockTimes table
>> ignoreprivileges - 0 (false) or 1 (true) - set to 1 to select computers
>> from any that are mapped to be able to run the image; set to 0 to only
>> select computers from ones that are both mapped and that users in the
>> usergroup assigned to this block request have been granted access to
>> through the privilege tree.
>>
>> returns: the number of machines that were successfully allocated, with 0
>> indicating a complete failure of not being able to allocate any computers
>>
>>
>> I just realized a problem in the return part of both functions. What if
>> the function is called too close to the start time to be able to preload
>> the computers by then? Should it just return 0 and not try to load any
>> of them? It may be that the functions need to return an array
>> containing both a return code and the number of computers that were
>> successfully allocated. Thoughts?
>>
>> Josh
>>
>> On Thursday February 05, 2009, Aaron Peeler wrote:
>>> Would it make sense to break this into two functions?
>>>
>>> Have this function XMLRPCblockAllocation with only two parameters
>>> blockrequestid and blockTimeid.
>>>
>>> The second function would be to create the Block Allocation -
>>> XMLRPCcreateBlockAllocation.
>>>
>>> Aaron
>>>
>>> --On February 4, 2009 4:39:30 PM -0500 Josh Thompson
>>>
>>> <jo...@ncsu.edu> wrote:
>>> > -----BEGIN PGP SIGNED MESSAGE-----
>>> > Hash: SHA1
>>> >
>>> > I'm going to start work on VCL-78 which is an addition to the frontend
>>> > XML RPC API to allow the backend to call the frontend for allocating
>>> > computers for block reservations. Currently, the block reservations
>>> > are created in the frontend, but computers are not allocated to
>>> > fulfill them because the requested time(s) could be at any point in
>>> > the future. So, around 6 hours before a requested time, the backend
>>> > picks up that block reservation time slot, allocates computers for
>>> > it, and then inserts reload reservations early enough for the
>>> > computers to be preloaded by the start of the time slot.
>>> >
>>> > This results in a fair amount of duplicated code, since the frontend
>>> > currently handles computer allocation in all other cases.
>>> >
>>> > This is how I am planning on implementing it. Please reply with any
>>> > comments on doing it differently or tweaking it somehow. I'm open to
>>> > any suggestions.
>>> >
>>> > The backend could just call the frontend with the blockTimes id to
>>> > process, but I'll make it more generic to be more useful in other
>>> > cases and require that the backend specify several items.
>>> >
>>> > function name: XMLRPCblockAllocation
>>> >
>>> > parameters:
>>> > imageid - id of the image to be used
>>> > start - unix timestamp for the start time (i.e. machines should be
>>> > prep'd and ready by this time)
>>> > end - unix timestamp for the end time
>>> > machinecount - number of computers to allocate
>>> > usergroupid - id of user group for checking user access to machines
>>> > ignoreprivileges - 0 (false) or 1 (true) - set to 1 to select
>>> > computers from any that are mapped to be able to run the image; set
>>> > to 0 to only select computers from ones that are both mapped and that
>>> > users in usergroupid have been granted access to through the
>>> > privilege tree. If this is set to 1, usergroupid is ignored (more on
>>> > this below). blockTimeid - id from blockTimes table that this will
>>> > fulfill or 0 if there is not an existing block reservation related
>>> > to this call
>>> >
>>> > returns: the number of machines that were successfully allocated, with
>>> > 0 indicating a complete failure of not being able to allocate any
>>> > computers
>>> >
>>> > Description of how it will work:
>>> > The frontend will use the normal scheduling functions to allocate
>>> > computers and insert them into the blockComputers table. If
>>> > blockTimeid is 0, new entries will be created in the blockRequest and
>>> > blockTimes tables. It will create reload reservations for the
>>> > computers early enough for them to be loaded by the start time based
>>> > on historical loading times.
>>> >
>>> > The ignoreprivileges flag exists to allow more machines to be
>>> > available for fulfilling block reservations than just those that the
>>> > user group has access to. The current implementation with the
>>> > backend doing the processing functions as if ignoreprivileges were
>>> > set to 1.
>>> >
>>> > That's pretty much it.
>>> >
>>> > Josh
>>> > - --
>>> > - -------------------------------
>>> > Josh Thompson
>>> > Systems Programmer
>>> > Virtual Computing Lab (VCL)
>>> > North Carolina State University
>>> >
>>> > Josh_Thompson@ncsu.edu
>>> > 919-515-5323
>>> >
>>> > my GPG/PGP key can be found at pgp.mit.edu
>>> > -----BEGIN PGP SIGNATURE-----
>>> > Version: GnuPG v1.4.6 (GNU/Linux)
>>> >
>>> > iD8DBQFJigsTV/LQcNdtPQMRAvSuAJkBLrJnfgC+VD/MFVcg6HIM3OIb9QCdGVkL
>>> > NK5Mjyap5+oeemdm40DEdaQ=
>>> > =e3CH
>>> > -----END PGP SIGNATURE-----
>>>
>>> Aaron Peeler
>>> OIT Advanced Computing
>>> College of Engineering-NCSU
>>> 919.513.4571
>>> http://vcl.ncsu.edu
>>
>>
>>
>> - --
>> - -------------------------------
>> Josh Thompson
>> Systems Programmer
>> Virtual Computing Lab (VCL)
>> North Carolina State University
>>
>> Josh_Thompson@ncsu.edu
>> 919-515-5323
>>
>> my GPG/PGP key can be found at pgp.mit.edu
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1.4.6 (GNU/Linux)
>>
>> iD8DBQFJiwM/V/LQcNdtPQMRAotCAJ9mqHM1vfk/Hj7P+mFwjTShiduUcwCeKyVS
>> Vxd5BgeBEphBMu+RSbSIw84=
>> =09dV
>> -----END PGP SIGNATURE-----
>
>
>
> Aaron Peeler
> OIT Advanced Computing
> College of Engineering-NCSU
> 919.513.4571
> http://vcl.ncsu.edu
Aaron Peeler
OIT Advanced Computing
College of Engineering-NCSU
919.513.4571
http://vcl.ncsu.edu
Re: frontend API for block reservations (VCL-78)
Posted by Aaron Peeler <aa...@ncsu.edu>.
That would work.
But now you loose the ability to create new block allocations through the
API. Previously it created one if there was not an existing entry.
So I guess we need to ask is this a desired/separate feature, the ability
create block reservations through the API.
--On February 5, 2009 10:18:17 AM -0500 Josh Thompson
<jo...@ncsu.edu> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> For the frontend, one function works as well as two would. However, I
> can see how it would simplify code on the backend to just pass a
> blockTimes id since it wouldn't have to determine all of the other
> information (blockTimes entries include blockrequestids; so, the
> blockrequestid wouldn't need to be passed as well).
>
> So, how about this, keep the function already described except drop
> the 'blockTimeid' parameter and add this function as well:
>
> function: XMLRPCproccessBlockTime
>
> parameters:
> blockTimesid - id from the blockTimes table
> ignoreprivileges - 0 (false) or 1 (true) - set to 1 to select computers
> from any that are mapped to be able to run the image; set to 0 to only
> select computers from ones that are both mapped and that users in the
> usergroup assigned to this block request have been granted access to
> through the privilege tree.
>
> returns: the number of machines that were successfully allocated, with 0
> indicating a complete failure of not being able to allocate any computers
>
>
> I just realized a problem in the return part of both functions. What if
> the function is called too close to the start time to be able to preload
> the computers by then? Should it just return 0 and not try to load any
> of them? It may be that the functions need to return an array
> containing both a return code and the number of computers that were
> successfully allocated. Thoughts?
>
> Josh
>
> On Thursday February 05, 2009, Aaron Peeler wrote:
>> Would it make sense to break this into two functions?
>>
>> Have this function XMLRPCblockAllocation with only two parameters
>> blockrequestid and blockTimeid.
>>
>> The second function would be to create the Block Allocation -
>> XMLRPCcreateBlockAllocation.
>>
>> Aaron
>>
>> --On February 4, 2009 4:39:30 PM -0500 Josh Thompson
>>
>> <jo...@ncsu.edu> wrote:
>> > -----BEGIN PGP SIGNED MESSAGE-----
>> > Hash: SHA1
>> >
>> > I'm going to start work on VCL-78 which is an addition to the frontend
>> > XML RPC API to allow the backend to call the frontend for allocating
>> > computers for block reservations. Currently, the block reservations
>> > are created in the frontend, but computers are not allocated to
>> > fulfill them because the requested time(s) could be at any point in
>> > the future. So, around 6 hours before a requested time, the backend
>> > picks up that block reservation time slot, allocates computers for
>> > it, and then inserts reload reservations early enough for the
>> > computers to be preloaded by the start of the time slot.
>> >
>> > This results in a fair amount of duplicated code, since the frontend
>> > currently handles computer allocation in all other cases.
>> >
>> > This is how I am planning on implementing it. Please reply with any
>> > comments on doing it differently or tweaking it somehow. I'm open to
>> > any suggestions.
>> >
>> > The backend could just call the frontend with the blockTimes id to
>> > process, but I'll make it more generic to be more useful in other
>> > cases and require that the backend specify several items.
>> >
>> > function name: XMLRPCblockAllocation
>> >
>> > parameters:
>> > imageid - id of the image to be used
>> > start - unix timestamp for the start time (i.e. machines should be
>> > prep'd and ready by this time)
>> > end - unix timestamp for the end time
>> > machinecount - number of computers to allocate
>> > usergroupid - id of user group for checking user access to machines
>> > ignoreprivileges - 0 (false) or 1 (true) - set to 1 to select
>> > computers from any that are mapped to be able to run the image; set
>> > to 0 to only select computers from ones that are both mapped and that
>> > users in usergroupid have been granted access to through the
>> > privilege tree. If this is set to 1, usergroupid is ignored (more on
>> > this below). blockTimeid - id from blockTimes table that this will
>> > fulfill or 0 if there is not an existing block reservation related
>> > to this call
>> >
>> > returns: the number of machines that were successfully allocated, with
>> > 0 indicating a complete failure of not being able to allocate any
>> > computers
>> >
>> > Description of how it will work:
>> > The frontend will use the normal scheduling functions to allocate
>> > computers and insert them into the blockComputers table. If
>> > blockTimeid is 0, new entries will be created in the blockRequest and
>> > blockTimes tables. It will create reload reservations for the
>> > computers early enough for them to be loaded by the start time based
>> > on historical loading times.
>> >
>> > The ignoreprivileges flag exists to allow more machines to be available
>> > for fulfilling block reservations than just those that the user group
>> > has access to. The current implementation with the backend doing the
>> > processing functions as if ignoreprivileges were set to 1.
>> >
>> > That's pretty much it.
>> >
>> > Josh
>> > - --
>> > - -------------------------------
>> > Josh Thompson
>> > Systems Programmer
>> > Virtual Computing Lab (VCL)
>> > North Carolina State University
>> >
>> > Josh_Thompson@ncsu.edu
>> > 919-515-5323
>> >
>> > my GPG/PGP key can be found at pgp.mit.edu
>> > -----BEGIN PGP SIGNATURE-----
>> > Version: GnuPG v1.4.6 (GNU/Linux)
>> >
>> > iD8DBQFJigsTV/LQcNdtPQMRAvSuAJkBLrJnfgC+VD/MFVcg6HIM3OIb9QCdGVkL
>> > NK5Mjyap5+oeemdm40DEdaQ=
>> > =e3CH
>> > -----END PGP SIGNATURE-----
>>
>> Aaron Peeler
>> OIT Advanced Computing
>> College of Engineering-NCSU
>> 919.513.4571
>> http://vcl.ncsu.edu
>
>
>
> - --
> - -------------------------------
> Josh Thompson
> Systems Programmer
> Virtual Computing Lab (VCL)
> North Carolina State University
>
> Josh_Thompson@ncsu.edu
> 919-515-5323
>
> my GPG/PGP key can be found at pgp.mit.edu
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
>
> iD8DBQFJiwM/V/LQcNdtPQMRAotCAJ9mqHM1vfk/Hj7P+mFwjTShiduUcwCeKyVS
> Vxd5BgeBEphBMu+RSbSIw84=
> =09dV
> -----END PGP SIGNATURE-----
Aaron Peeler
OIT Advanced Computing
College of Engineering-NCSU
919.513.4571
http://vcl.ncsu.edu
Re: frontend API for block reservations (VCL-78)
Posted by Josh Thompson <jo...@ncsu.edu>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On Thu February 5 2009 10:18:17 am Josh Thompson wrote:
> > > function name: XMLRPCblockAllocation
> > >
> > > parameters:
> > > imageid - id of the image to be used
> > > start - unix timestamp for the start time (i.e. machines should be
> > > prep'd and ready by this time)
> > > end - unix timestamp for the end time
> > > machinecount - number of computers to allocate
> > > usergroupid - id of user group for checking user access to machines
> > > ignoreprivileges - 0 (false) or 1 (true) - set to 1 to select
> > > computers from any that are mapped to be able to run the image; set to
> > > 0 to only select computers from ones that are both mapped and that
> > > users in usergroupid have been granted access to through the privilege
> > > tree. If this is set to 1, usergroupid is ignored (more on this
> > > below).
After coding this, I discovered I converted 'start' and 'end' to the mysql
datetime format and never used the unix timestamp format. Does anyone see
any reason to keep the parameters as unix timestamps? If not, I'll specify
that they should be mysql datetime format. I'm think that would be easier
for people to build when calling the API than a unix timestamp.
Josh
- --
- -------------------------------
Josh Thompson
Systems Programmer
Virtual Computing Lab (VCL)
North Carolina State University
Josh_Thompson@ncsu.edu
919-515-5323
my GPG/PGP key can be found at www.keyserver.net
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)
iD8DBQFJr/s1V/LQcNdtPQMRAiMpAJ93GkRA/1jJj0k+X0afyp9I92g0zwCdHDNj
SgWB7+f1ZJ6QO+43IdiSCCM=
=LvnQ
-----END PGP SIGNATURE-----
Re: frontend API for block reservations (VCL-78)
Posted by Andy Kurth <an...@ncsu.edu>.
This is just semantics for now but it might be better to think of the number of
things being allocated as requests rather than machines. This will allow for
block requests of cluster images.
Regarding the return code... it might be useful to return the number of requests
that haven't been successfully allocated for the block time. A return code of 0
would mean all of the resources for the block time have been successfully allocated.
The backend wouldn't need to worry about the total number of requests that are
needed nor would it care how many were successfully allocated during this stage.
It would only have to know if the number which couldn't be allocated is
greater than 0.
This would make it easy to add some failure recovery features in the future. If
the number which hasn't yet been allocated is greater than 0, the backend
process could make multiple attempts by sleeping and looping.
-Andy
Josh Thompson wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> For the frontend, one function works as well as two would. However, I can see
> how it would simplify code on the backend to just pass a blockTimes id since
> it wouldn't have to determine all of the other information (blockTimes
> entries include blockrequestids; so, the blockrequestid wouldn't need to be
> passed as well).
>
> So, how about this, keep the function already described except drop
> the 'blockTimeid' parameter and add this function as well:
>
> function: XMLRPCproccessBlockTime
>
> parameters:
> blockTimesid - id from the blockTimes table
> ignoreprivileges - 0 (false) or 1 (true) - set to 1 to select computers from
> any that are mapped to be able to run the image; set to 0 to only select
> computers from ones that are both mapped and that users in the usergroup
> assigned to this block request have been granted access to through the
> privilege tree.
>
> returns: the number of machines that were successfully allocated, with 0
> indicating a complete failure of not being able to allocate any computers
>
>
> I just realized a problem in the return part of both functions. What if the
> function is called too close to the start time to be able to preload the
> computers by then? Should it just return 0 and not try to load any of them?
> It may be that the functions need to return an array containing both a return
> code and the number of computers that were successfully allocated. Thoughts?
>
> Josh
>
> On Thursday February 05, 2009, Aaron Peeler wrote:
>> Would it make sense to break this into two functions?
>>
>> Have this function XMLRPCblockAllocation with only two parameters
>> blockrequestid and blockTimeid.
>>
>> The second function would be to create the Block Allocation -
>> XMLRPCcreateBlockAllocation.
>>
>> Aaron
>>
>> --On February 4, 2009 4:39:30 PM -0500 Josh Thompson
>>
>> <jo...@ncsu.edu> wrote:
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA1
>>>
>>> I'm going to start work on VCL-78 which is an addition to the frontend
>>> XML RPC API to allow the backend to call the frontend for allocating
>>> computers for block reservations. Currently, the block reservations are
>>> created in the frontend, but computers are not allocated to fulfill them
>>> because the requested time(s) could be at any point in the future. So,
>>> around 6 hours before a requested time, the backend picks up that block
>>> reservation time slot, allocates computers for it, and then inserts
>>> reload reservations early enough for the computers to be preloaded by
>>> the start of the time slot.
>>>
>>> This results in a fair amount of duplicated code, since the frontend
>>> currently handles computer allocation in all other cases.
>>>
>>> This is how I am planning on implementing it. Please reply with any
>>> comments on doing it differently or tweaking it somehow. I'm open to
>>> any suggestions.
>>>
>>> The backend could just call the frontend with the blockTimes id to
>>> process, but I'll make it more generic to be more useful in other cases
>>> and require that the backend specify several items.
>>>
>>> function name: XMLRPCblockAllocation
>>>
>>> parameters:
>>> imageid - id of the image to be used
>>> start - unix timestamp for the start time (i.e. machines should be
>>> prep'd and ready by this time)
>>> end - unix timestamp for the end time
>>> machinecount - number of computers to allocate
>>> usergroupid - id of user group for checking user access to machines
>>> ignoreprivileges - 0 (false) or 1 (true) - set to 1 to select computers
>>> from any that are mapped to be able to run the image; set to 0 to only
>>> select computers from ones that are both mapped and that users in
>>> usergroupid have been granted access to through the privilege tree. If
>>> this is set to 1, usergroupid is ignored (more on this below).
>>> blockTimeid - id from blockTimes table that this will fulfill or 0 if
>>> there is not an existing block reservation related to this call
>>>
>>> returns: the number of machines that were successfully allocated, with 0
>>> indicating a complete failure of not being able to allocate any computers
>>>
>>> Description of how it will work:
>>> The frontend will use the normal scheduling functions to allocate
>>> computers and insert them into the blockComputers table. If blockTimeid
>>> is 0, new entries will be created in the blockRequest and blockTimes
>>> tables. It will create reload reservations for the computers early
>>> enough for them to be loaded by the start time based on historical
>>> loading times.
>>>
>>> The ignoreprivileges flag exists to allow more machines to be available
>>> for fulfilling block reservations than just those that the user group
>>> has access to. The current implementation with the backend doing the
>>> processing functions as if ignoreprivileges were set to 1.
>>>
>>> That's pretty much it.
>>>
>>> Josh
>>> - --
>>> - -------------------------------
>>> Josh Thompson
>>> Systems Programmer
>>> Virtual Computing Lab (VCL)
>>> North Carolina State University
>>>
>>> Josh_Thompson@ncsu.edu
>>> 919-515-5323
>>>
>>> my GPG/PGP key can be found at pgp.mit.edu
>>> -----BEGIN PGP SIGNATURE-----
>>> Version: GnuPG v1.4.6 (GNU/Linux)
>>>
>>> iD8DBQFJigsTV/LQcNdtPQMRAvSuAJkBLrJnfgC+VD/MFVcg6HIM3OIb9QCdGVkL
>>> NK5Mjyap5+oeemdm40DEdaQ=
>>> =e3CH
>>> -----END PGP SIGNATURE-----
>> Aaron Peeler
>> OIT Advanced Computing
>> College of Engineering-NCSU
>> 919.513.4571
>> http://vcl.ncsu.edu
>
>
>
> - --
> - -------------------------------
> Josh Thompson
> Systems Programmer
> Virtual Computing Lab (VCL)
> North Carolina State University
>
> Josh_Thompson@ncsu.edu
> 919-515-5323
>
> my GPG/PGP key can be found at pgp.mit.edu
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
>
> iD8DBQFJiwM/V/LQcNdtPQMRAotCAJ9mqHM1vfk/Hj7P+mFwjTShiduUcwCeKyVS
> Vxd5BgeBEphBMu+RSbSIw84=
> =09dV
> -----END PGP SIGNATURE-----
--
Andy Kurth
Virtual Computing Lab
Office of Information Technology
North Carolina State University
andy_kurth@ncsu.edu
919.513.4090
Re: frontend API for block reservations (VCL-78)
Posted by Josh Thompson <jo...@ncsu.edu>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
For the frontend, one function works as well as two would. However, I can see
how it would simplify code on the backend to just pass a blockTimes id since
it wouldn't have to determine all of the other information (blockTimes
entries include blockrequestids; so, the blockrequestid wouldn't need to be
passed as well).
So, how about this, keep the function already described except drop
the 'blockTimeid' parameter and add this function as well:
function: XMLRPCproccessBlockTime
parameters:
blockTimesid - id from the blockTimes table
ignoreprivileges - 0 (false) or 1 (true) - set to 1 to select computers from
any that are mapped to be able to run the image; set to 0 to only select
computers from ones that are both mapped and that users in the usergroup
assigned to this block request have been granted access to through the
privilege tree.
returns: the number of machines that were successfully allocated, with 0
indicating a complete failure of not being able to allocate any computers
I just realized a problem in the return part of both functions. What if the
function is called too close to the start time to be able to preload the
computers by then? Should it just return 0 and not try to load any of them?
It may be that the functions need to return an array containing both a return
code and the number of computers that were successfully allocated. Thoughts?
Josh
On Thursday February 05, 2009, Aaron Peeler wrote:
> Would it make sense to break this into two functions?
>
> Have this function XMLRPCblockAllocation with only two parameters
> blockrequestid and blockTimeid.
>
> The second function would be to create the Block Allocation -
> XMLRPCcreateBlockAllocation.
>
> Aaron
>
> --On February 4, 2009 4:39:30 PM -0500 Josh Thompson
>
> <jo...@ncsu.edu> wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > I'm going to start work on VCL-78 which is an addition to the frontend
> > XML RPC API to allow the backend to call the frontend for allocating
> > computers for block reservations. Currently, the block reservations are
> > created in the frontend, but computers are not allocated to fulfill them
> > because the requested time(s) could be at any point in the future. So,
> > around 6 hours before a requested time, the backend picks up that block
> > reservation time slot, allocates computers for it, and then inserts
> > reload reservations early enough for the computers to be preloaded by
> > the start of the time slot.
> >
> > This results in a fair amount of duplicated code, since the frontend
> > currently handles computer allocation in all other cases.
> >
> > This is how I am planning on implementing it. Please reply with any
> > comments on doing it differently or tweaking it somehow. I'm open to
> > any suggestions.
> >
> > The backend could just call the frontend with the blockTimes id to
> > process, but I'll make it more generic to be more useful in other cases
> > and require that the backend specify several items.
> >
> > function name: XMLRPCblockAllocation
> >
> > parameters:
> > imageid - id of the image to be used
> > start - unix timestamp for the start time (i.e. machines should be
> > prep'd and ready by this time)
> > end - unix timestamp for the end time
> > machinecount - number of computers to allocate
> > usergroupid - id of user group for checking user access to machines
> > ignoreprivileges - 0 (false) or 1 (true) - set to 1 to select computers
> > from any that are mapped to be able to run the image; set to 0 to only
> > select computers from ones that are both mapped and that users in
> > usergroupid have been granted access to through the privilege tree. If
> > this is set to 1, usergroupid is ignored (more on this below).
> > blockTimeid - id from blockTimes table that this will fulfill or 0 if
> > there is not an existing block reservation related to this call
> >
> > returns: the number of machines that were successfully allocated, with 0
> > indicating a complete failure of not being able to allocate any computers
> >
> > Description of how it will work:
> > The frontend will use the normal scheduling functions to allocate
> > computers and insert them into the blockComputers table. If blockTimeid
> > is 0, new entries will be created in the blockRequest and blockTimes
> > tables. It will create reload reservations for the computers early
> > enough for them to be loaded by the start time based on historical
> > loading times.
> >
> > The ignoreprivileges flag exists to allow more machines to be available
> > for fulfilling block reservations than just those that the user group
> > has access to. The current implementation with the backend doing the
> > processing functions as if ignoreprivileges were set to 1.
> >
> > That's pretty much it.
> >
> > Josh
> > - --
> > - -------------------------------
> > Josh Thompson
> > Systems Programmer
> > Virtual Computing Lab (VCL)
> > North Carolina State University
> >
> > Josh_Thompson@ncsu.edu
> > 919-515-5323
> >
> > my GPG/PGP key can be found at pgp.mit.edu
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v1.4.6 (GNU/Linux)
> >
> > iD8DBQFJigsTV/LQcNdtPQMRAvSuAJkBLrJnfgC+VD/MFVcg6HIM3OIb9QCdGVkL
> > NK5Mjyap5+oeemdm40DEdaQ=
> > =e3CH
> > -----END PGP SIGNATURE-----
>
> Aaron Peeler
> OIT Advanced Computing
> College of Engineering-NCSU
> 919.513.4571
> http://vcl.ncsu.edu
- --
- -------------------------------
Josh Thompson
Systems Programmer
Virtual Computing Lab (VCL)
North Carolina State University
Josh_Thompson@ncsu.edu
919-515-5323
my GPG/PGP key can be found at pgp.mit.edu
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
iD8DBQFJiwM/V/LQcNdtPQMRAotCAJ9mqHM1vfk/Hj7P+mFwjTShiduUcwCeKyVS
Vxd5BgeBEphBMu+RSbSIw84=
=09dV
-----END PGP SIGNATURE-----
Re: frontend API for block reservations (VCL-78)
Posted by Aaron Peeler <aa...@ncsu.edu>.
Another suggestion.
Can you look at staggering the reload start times based on the load of the
management node?
A problem that currently exist on the backend method is that the block
allocation process inserts multiple reloads at a time, which can quickly
overload a management node. For example if a 50 seat block allocation for
an upcoming class or workshop is triggered the backend currently inserts 50
reloads at the same time to start preparing for the upcoming block.
So maybe somewhere around 5-10 concurrent reloads 2-5 minutes apart. Of
course we hit a problem of missing the window needed to get the machines
ready. Maybe the number of concurrent reloads and the reload start delta
could be a result of some calculation of how many nodes are current loading
on that particular management node and an average of the historical load
time of the image. Hope that made sense.
Aaron
--On February 5, 2009 9:16:13 AM -0500 Aaron Peeler <aa...@ncsu.edu>
wrote:
> Would it make sense to break this into two functions?
>
> Have this function XMLRPCblockAllocation with only two parameters
> blockrequestid and blockTimeid.
>
> The second function would be to create the Block Allocation -
> XMLRPCcreateBlockAllocation.
>
> Aaron
>
> --On February 4, 2009 4:39:30 PM -0500 Josh Thompson
> <jo...@ncsu.edu> wrote:
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> I'm going to start work on VCL-78 which is an addition to the frontend
>> XML RPC API to allow the backend to call the frontend for allocating
>> computers for block reservations. Currently, the block reservations are
>> created in the frontend, but computers are not allocated to fulfill them
>> because the requested time(s) could be at any point in the future. So,
>> around 6 hours before a requested time, the backend picks up that block
>> reservation time slot, allocates computers for it, and then inserts
>> reload reservations early enough for the computers to be preloaded by
>> the start of the time slot.
>>
>> This results in a fair amount of duplicated code, since the frontend
>> currently handles computer allocation in all other cases.
>>
>> This is how I am planning on implementing it. Please reply with any
>> comments on doing it differently or tweaking it somehow. I'm open to
>> any suggestions.
>>
>> The backend could just call the frontend with the blockTimes id to
>> process, but I'll make it more generic to be more useful in other cases
>> and require that the backend specify several items.
>>
>> function name: XMLRPCblockAllocation
>>
>> parameters:
>> imageid - id of the image to be used
>> start - unix timestamp for the start time (i.e. machines should be
>> prep'd and ready by this time)
>> end - unix timestamp for the end time
>> machinecount - number of computers to allocate
>> usergroupid - id of user group for checking user access to machines
>> ignoreprivileges - 0 (false) or 1 (true) - set to 1 to select computers
>> from any that are mapped to be able to run the image; set to 0 to only
>> select computers from ones that are both mapped and that users in
>> usergroupid have been granted access to through the privilege tree. If
>> this is set to 1, usergroupid is ignored (more on this below).
>> blockTimeid - id from blockTimes table that this will fulfill or 0 if
>> there is not an existing block reservation related to this call
>>
>> returns: the number of machines that were successfully allocated, with 0
>> indicating a complete failure of not being able to allocate any computers
>>
>> Description of how it will work:
>> The frontend will use the normal scheduling functions to allocate
>> computers and insert them into the blockComputers table. If blockTimeid
>> is 0, new entries will be created in the blockRequest and blockTimes
>> tables. It will create reload reservations for the computers early
>> enough for them to be loaded by the start time based on historical
>> loading times.
>>
>> The ignoreprivileges flag exists to allow more machines to be available
>> for fulfilling block reservations than just those that the user group
>> has access to. The current implementation with the backend doing the
>> processing functions as if ignoreprivileges were set to 1.
>>
>> That's pretty much it.
>>
>> Josh
>> - --
>> - -------------------------------
>> Josh Thompson
>> Systems Programmer
>> Virtual Computing Lab (VCL)
>> North Carolina State University
>>
>> Josh_Thompson@ncsu.edu
>> 919-515-5323
>>
>> my GPG/PGP key can be found at pgp.mit.edu
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1.4.6 (GNU/Linux)
>>
>> iD8DBQFJigsTV/LQcNdtPQMRAvSuAJkBLrJnfgC+VD/MFVcg6HIM3OIb9QCdGVkL
>> NK5Mjyap5+oeemdm40DEdaQ=
>> =e3CH
>> -----END PGP SIGNATURE-----
>
Re: frontend API for block reservations (VCL-78)
Posted by Aaron Peeler <aa...@ncsu.edu>.
Would it make sense to break this into two functions?
Have this function XMLRPCblockAllocation with only two parameters
blockrequestid and blockTimeid.
The second function would be to create the Block Allocation -
XMLRPCcreateBlockAllocation.
Aaron
--On February 4, 2009 4:39:30 PM -0500 Josh Thompson
<jo...@ncsu.edu> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> I'm going to start work on VCL-78 which is an addition to the frontend
> XML RPC API to allow the backend to call the frontend for allocating
> computers for block reservations. Currently, the block reservations are
> created in the frontend, but computers are not allocated to fulfill them
> because the requested time(s) could be at any point in the future. So,
> around 6 hours before a requested time, the backend picks up that block
> reservation time slot, allocates computers for it, and then inserts
> reload reservations early enough for the computers to be preloaded by
> the start of the time slot.
>
> This results in a fair amount of duplicated code, since the frontend
> currently handles computer allocation in all other cases.
>
> This is how I am planning on implementing it. Please reply with any
> comments on doing it differently or tweaking it somehow. I'm open to
> any suggestions.
>
> The backend could just call the frontend with the blockTimes id to
> process, but I'll make it more generic to be more useful in other cases
> and require that the backend specify several items.
>
> function name: XMLRPCblockAllocation
>
> parameters:
> imageid - id of the image to be used
> start - unix timestamp for the start time (i.e. machines should be
> prep'd and ready by this time)
> end - unix timestamp for the end time
> machinecount - number of computers to allocate
> usergroupid - id of user group for checking user access to machines
> ignoreprivileges - 0 (false) or 1 (true) - set to 1 to select computers
> from any that are mapped to be able to run the image; set to 0 to only
> select computers from ones that are both mapped and that users in
> usergroupid have been granted access to through the privilege tree. If
> this is set to 1, usergroupid is ignored (more on this below).
> blockTimeid - id from blockTimes table that this will fulfill or 0 if
> there is not an existing block reservation related to this call
>
> returns: the number of machines that were successfully allocated, with 0
> indicating a complete failure of not being able to allocate any computers
>
> Description of how it will work:
> The frontend will use the normal scheduling functions to allocate
> computers and insert them into the blockComputers table. If blockTimeid
> is 0, new entries will be created in the blockRequest and blockTimes
> tables. It will create reload reservations for the computers early
> enough for them to be loaded by the start time based on historical
> loading times.
>
> The ignoreprivileges flag exists to allow more machines to be available
> for fulfilling block reservations than just those that the user group
> has access to. The current implementation with the backend doing the
> processing functions as if ignoreprivileges were set to 1.
>
> That's pretty much it.
>
> Josh
> - --
> - -------------------------------
> Josh Thompson
> Systems Programmer
> Virtual Computing Lab (VCL)
> North Carolina State University
>
> Josh_Thompson@ncsu.edu
> 919-515-5323
>
> my GPG/PGP key can be found at pgp.mit.edu
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
>
> iD8DBQFJigsTV/LQcNdtPQMRAvSuAJkBLrJnfgC+VD/MFVcg6HIM3OIb9QCdGVkL
> NK5Mjyap5+oeemdm40DEdaQ=
> =e3CH
> -----END PGP SIGNATURE-----
Aaron Peeler
OIT Advanced Computing
College of Engineering-NCSU
919.513.4571
http://vcl.ncsu.edu