You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@vcl.apache.org by Dmitri Chebotarov <dc...@gmu.edu> on 2013/07/17 18:36:02 UTC
VCL front-end and load balancer
Hi
I would like to load balance multiple front-end VCL servers.
It is F5 load balancer. The LB configuration allows to enable session persistency.
Is this will be OK with VCL's front-end? I remember someone mentioned some issues with having multiple front-end servers, but don't remember details.
Thanks.
Re: VCL front-end and load balancer
Posted by Dmitri Chebotarov <dc...@gmu.edu>.
Thank you. I was trying to understand how the resource locking works. So far this what I got:
- with one web-server PHP maintains resource locks in memory (semaphores), which essentially stops multiple users to request the same computer.
- with more then one each web-server has non-shared locks, which presents the problem.
Is memcached could help to share semaphore locks between web-servers? From wikipedia: "...Memcached's APIs provide a giant hash table distributed across multiple machines...".
I'm not sure how much work it would be to implement either solution - fewer change with same result the better...
Thanks.
On Jul 23, 2013, at 16:14 , Aaron Coburn <ac...@amherst.edu>
wrote:
> I agree with Mark.
>
> Stored procedures seem nice, but they are incredibly difficult to document, and if you have to make changes later, they can be a nightmare to debug.
>
> Aaron
>
> On Jul 23, 2013, at 3:58 PM, Mark Gardner <mk...@vt.edu> wrote:
>
>> I would vote against stored procedures. Keep the DB simple.
>>
>> Mark
>>
>> On Tue, Jul 23, 2013 at 9:01 AM, Josh Thompson <jo...@ncsu.edu> wrote:
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA1
>>>
>>> Hi Jim,
>>>
>>> What are you saying would not be a problem? I know having 2 systems trying to
>>> write at the same time would be okay, but the scheduling code has to read a
>>> bunch of data from the database, analyze it to select a computer, and then
>>> write information to the database. The semaphore is needed because the state
>>> of the system needs to remain the same between the reading of the data and the
>>> writing of it. Otherwise, computers that were not assigned during the read
>>> may get assigned before the write happens, which can result in multiple people
>>> getting the same computer. We actually had this happen back in 2004 when we
>>> were having our initial pilot of VCL, which is what prompted adding the
>>> semaphore.
>>>
>>> Another option is to actually move the scheduling code to a stored procedure
>>> in the database. However, that would require someone with much better stored
>>> procedure skills than mine.
>>>
>>> Josh
>>>
>>> On Monday, July 22, 2013 2:36:28 PM James O'Dell wrote:
>>>> Hey Josh,
>>>>
>>>> I've been F5 load balancing, and SSL offloading, Moodle for the last
>>>> couple of years with out a problem.
>>>> Moodle just insists the database engine be ACID compliant
>>>>
>>>> If the database engine is transaction safe (i.e. InnoDB) you shouldn't
>>>> have a problem.
>>>> It's really just a matter of handling the results (i.e. if someone else
>>>> grabs the reservation
>>>> what do you do)
>>>>
>>>> Imho relying on the database seems like the simplest/cleanest approach
>>>>
>>>> __Jim
>>>>
>>>> On 7/22/2013 2:20 PM, Josh Thompson wrote:
>>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>>> Hash: SHA1
>>>>>
>>>>> The problem with load balancing the front end is that there is a semaphore
>>>>> lock (php's built in implementation) around the scheduling code. Without
>>>>> it, 2 users coming in at the same time can get assigned the same
>>>>> computer. So, having multiple frontends would allow the same thing to
>>>>> happen.
>>>>>
>>>>> I've considered just locking a table in the database as a semaphore.
>>>>> However, in my testing of this, if the web server locks a table and then
>>>>> goes offline before unlocking it, the table stays locked. So, done this
>>>>> way, if there were multiple frontends and one went offline while in the
>>>>> middle of the lock, none of them would be able to create reservations. I
>>>>> don't remember what had to be done to release the lock.
>>>>>
>>>>> The option I'd like to use that I've not gotten around to implementing (I
>>>>> believe there is a really old JIRA issue for it) is to add an XMLRPC API
>>>>> function for calling the scheduling code. Then, have one frontend
>>>>> assigned as the one that would handle all of the scheduling. The others
>>>>> would make calls to that one for just the scheduling part, but would do
>>>>> everything else normally. Optionally, there could be an election process
>>>>> so that a new frontend could be selected to do the scheduling if the one
>>>>> that had been doing it went down.
>>>>>
>>>>> Aaron C. listed some good ideas, but I think the above would be more
>>>>> straightforward since it would not involve bringing in other technologies.
>>>>>
>>>>> Josh
>>>>>
>>>>> On Thursday, July 18, 2013 9:49:39 AM Aaron Peeler wrote:
>>>>>> I like both of these approaches, especially 1, but I'm not sure of the
>>>>>> effort it would take to convert.
>>>>>> -AP
>>>>>>
>>>>>> On Wed, Jul 17, 2013 at 2:59 PM, Aaron Coburn <ac...@amherst.edu>
>>> wrote:
>>>>>>> In my opinion, this issue has largely been solved by the NoSQL and
>>>>>>> application messaging communities. But as long as the VCL web server(s)
>>>>>>> use local, file-based semaphores to control resource allocation, this
>>>>>>> will be a hard nut to crack.
>>>>>>>
>>>>>>> If I were going to tackle this problem, I would take one of two general
>>>>>>> approaches:
>>>>>>>
>>>>>>> 1) Use an asynchronous messaging queue (i.e. Apache ActiveMQ or
>>>>>>> something
>>>>>>> like Redis)
>>>>>>>
>>>>>>> In this way, when a reservation is made, the request is pushed onto a
>>>>>>> centralized queue, and some intermediate (i.e. vcld) process will shift
>>>>>>> messages off the queue and assign the reservations. If we used ActiveMQ,
>>>>>>> the php and perl code would communicate with the message broker over the
>>>>>>> STOMP protocol [1]. Redis would also work because it runs in a single
>>>>>>> thread and all operations are atomic -- requests can simply be pushed
>>>>>>> onto a FIFO-type list structure [2].
>>>>>>>
>>>>>>> 2. Use a NoSQL database such as CouchDB or Riak that uses a type of
>>>>>>> optimistic locking model for all writes.
>>>>>>>
>>>>>>> That is, if reservations are stored in such a way that the resource id
>>>>>>> (i.e. computer id) forms the database key, the assignment of a user to a
>>>>>>> particular compute resource requires sending the correct revision_id of
>>>>>>> that resource in order for the write to be successful. If successful, it
>>>>>>> returns an HTTP 200 status code and the client proceeds normally;
>>>>>>> otherwise, it sends a 409 header and it is up to the client to try again
>>>>>>> [3]. It would also be possible to use the existing MySQL database to
>>>>>>> emulate something like #2, but under a heavy load, I imagine that the
>>>>>>> row
>>>>>>> locking could really slow things down. Using Couch would really be much
>>>>>>> more scalable than trying to do everything in MySQL.
>>>>>>>
>>>>>>> Either of these approaches would then allow you to distribute the web
>>>>>>> front end across multiple machines.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Aaron Coburn
>>>>>>>
>>>>>>> [1] http://activemq.apache.org/stomp.html
>>>>>>> [2] http://redis.io/commands/lpush
>>>>>>> [3] http://wiki.apache.org/couchdb/HTTP_Document_API#PUT
>>>>>>>
>>>>>>> On Jul 17, 2013, at 1:06 PM, Aaron Peeler <aa...@ncsu.edu> wrote:
>>>>>>>> This is definitely desired.
>>>>>>>>
>>>>>>>> An additional challenge with multiple web servers is to figure how to
>>>>>>>> sanely lock/unlock the resource to prevent it from getting assigned
>>>>>>>> two separate users that are making requests at the same instant.
>>>>>>>>
>>>>>>>> AP
>>>>>>>>
>>>>>>>> On Wed, Jul 17, 2013 at 12:57 PM, James O'Dell <jo...@fullerton.edu>
>>>>>
>>>>> wrote:
>>>>>>>>> I also tried load balancing the web. I didn't have any success.
>>>>>>>>>
>>>>>>>>> I also tried an SSL offload using the F5
>>>>>>>>>
>>>>>>>>> The SSLoffload didn't work because
>>>>>>>>> an internal VCL check noticed the header wasn't https
>>>>>>>>> and redirected to https. Basically it kept looping
>>>>>>>>>
>>>>>>>>> On 7/17/2013 9:36 AM, Dmitri Chebotarov wrote:
>>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>> I would like to load balance multiple front-end VCL servers.
>>>>>>>>> It is F5 load balancer. The LB configuration allows to enable session
>>>>>>>>> persistency.
>>>>>>>>> Is this will be OK with VCL's front-end? I remember someone mentioned
>>>>>>>>> some
>>>>>>>>> issues with having multiple front-end servers, but don't remember
>>>>>>>>> details.
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Aaron Peeler
>>>>>>>> Program Manager
>>>>>>>> Virtual Computing Lab
>>>>>>>> NC State University
>>>>>>>>
>>>>>>>> All electronic mail messages in connection with State business which
>>>>>>>> are sent to or received by this account are subject to the NC Public
>>>>>>>> Records Law and may be disclosed to third parties.
>>>>>
>>>>> - --
>>>>> - -------------------------------
>>>>> Josh Thompson
>>>>> VCL Developer
>>>>> North Carolina State University
>>>>>
>>>>> my GPG/PGP key can be found at pgp.mit.edu
>>>>>
>>>>> All electronic mail messages in connection with State business which
>>>>> are sent to or received by this account are subject to the NC Public
>>>>> Records Law and may be disclosed to third parties.
>>>>> -----BEGIN PGP SIGNATURE-----
>>>>> Version: GnuPG v2.0.19 (GNU/Linux)
>>>>>
>>>>> iEYEARECAAYFAlHtojYACgkQV/LQcNdtPQNNjwCfQ+PRt/yZXI4tf2YDiH2IZu2m
>>>>> coYAn2QbjICSa5MUR0cR9DIxniZVQFtK
>>>>> =/YZs
>>>>> -----END PGP SIGNATURE-----
>>> - --
>>> - -------------------------------
>>> Josh Thompson
>>> VCL Developer
>>> North Carolina State University
>>>
>>> my GPG/PGP key can be found at pgp.mit.edu
>>>
>>> All electronic mail messages in connection with State business which
>>> are sent to or received by this account are subject to the NC Public
>>> Records Law and may be disclosed to third parties.
>>> -----BEGIN PGP SIGNATURE-----
>>> Version: GnuPG v2.0.19 (GNU/Linux)
>>>
>>> iEYEARECAAYFAlHufrkACgkQV/LQcNdtPQPSagCdHzGSWMPRoJjTYR/nqdJScmW/
>>> k2EAn2cGY5gKBZZgkRsG9ebscAVyOcYa
>>> =oxwt
>>> -----END PGP SIGNATURE-----
>>>
>>
>>
>>
>> --
>> Mark Gardner
>> --
>
>
--
Thank you,
Dmitri Chebotarov
VCL Sys Eng, Engineering & Architectural Support, TSD - Ent Servers & Messaging
223 Aquia Building, Ffx, MSN: 1B5
Phone: (703) 993-6175 | Fax: (703) 993-3404
Re: VCL front-end and load balancer
Posted by Dmitri Chebotarov <dc...@gmu.edu>.
Thank you. I was trying to understand how the resource locking works. So far this what I got:
- with one web-server PHP maintains resource locks in memory (semaphores), which essentially stops multiple users to request the same computer.
- with more then one each web-server has non-shared locks, which presents the problem.
Is memcached could help to share semaphore locks between web-servers? From wikipedia: "...Memcached's APIs provide a giant hash table distributed across multiple machines...".
I'm not sure how much work it would be to implement either solution - fewer change with same result the better...
Thanks.
On Jul 23, 2013, at 16:14 , Aaron Coburn <ac...@amherst.edu>
wrote:
> I agree with Mark.
>
> Stored procedures seem nice, but they are incredibly difficult to document, and if you have to make changes later, they can be a nightmare to debug.
>
> Aaron
>
> On Jul 23, 2013, at 3:58 PM, Mark Gardner <mk...@vt.edu> wrote:
>
>> I would vote against stored procedures. Keep the DB simple.
>>
>> Mark
>>
>> On Tue, Jul 23, 2013 at 9:01 AM, Josh Thompson <jo...@ncsu.edu> wrote:
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA1
>>>
>>> Hi Jim,
>>>
>>> What are you saying would not be a problem? I know having 2 systems trying to
>>> write at the same time would be okay, but the scheduling code has to read a
>>> bunch of data from the database, analyze it to select a computer, and then
>>> write information to the database. The semaphore is needed because the state
>>> of the system needs to remain the same between the reading of the data and the
>>> writing of it. Otherwise, computers that were not assigned during the read
>>> may get assigned before the write happens, which can result in multiple people
>>> getting the same computer. We actually had this happen back in 2004 when we
>>> were having our initial pilot of VCL, which is what prompted adding the
>>> semaphore.
>>>
>>> Another option is to actually move the scheduling code to a stored procedure
>>> in the database. However, that would require someone with much better stored
>>> procedure skills than mine.
>>>
>>> Josh
>>>
>>> On Monday, July 22, 2013 2:36:28 PM James O'Dell wrote:
>>>> Hey Josh,
>>>>
>>>> I've been F5 load balancing, and SSL offloading, Moodle for the last
>>>> couple of years with out a problem.
>>>> Moodle just insists the database engine be ACID compliant
>>>>
>>>> If the database engine is transaction safe (i.e. InnoDB) you shouldn't
>>>> have a problem.
>>>> It's really just a matter of handling the results (i.e. if someone else
>>>> grabs the reservation
>>>> what do you do)
>>>>
>>>> Imho relying on the database seems like the simplest/cleanest approach
>>>>
>>>> __Jim
>>>>
>>>> On 7/22/2013 2:20 PM, Josh Thompson wrote:
>>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>>> Hash: SHA1
>>>>>
>>>>> The problem with load balancing the front end is that there is a semaphore
>>>>> lock (php's built in implementation) around the scheduling code. Without
>>>>> it, 2 users coming in at the same time can get assigned the same
>>>>> computer. So, having multiple frontends would allow the same thing to
>>>>> happen.
>>>>>
>>>>> I've considered just locking a table in the database as a semaphore.
>>>>> However, in my testing of this, if the web server locks a table and then
>>>>> goes offline before unlocking it, the table stays locked. So, done this
>>>>> way, if there were multiple frontends and one went offline while in the
>>>>> middle of the lock, none of them would be able to create reservations. I
>>>>> don't remember what had to be done to release the lock.
>>>>>
>>>>> The option I'd like to use that I've not gotten around to implementing (I
>>>>> believe there is a really old JIRA issue for it) is to add an XMLRPC API
>>>>> function for calling the scheduling code. Then, have one frontend
>>>>> assigned as the one that would handle all of the scheduling. The others
>>>>> would make calls to that one for just the scheduling part, but would do
>>>>> everything else normally. Optionally, there could be an election process
>>>>> so that a new frontend could be selected to do the scheduling if the one
>>>>> that had been doing it went down.
>>>>>
>>>>> Aaron C. listed some good ideas, but I think the above would be more
>>>>> straightforward since it would not involve bringing in other technologies.
>>>>>
>>>>> Josh
>>>>>
>>>>> On Thursday, July 18, 2013 9:49:39 AM Aaron Peeler wrote:
>>>>>> I like both of these approaches, especially 1, but I'm not sure of the
>>>>>> effort it would take to convert.
>>>>>> -AP
>>>>>>
>>>>>> On Wed, Jul 17, 2013 at 2:59 PM, Aaron Coburn <ac...@amherst.edu>
>>> wrote:
>>>>>>> In my opinion, this issue has largely been solved by the NoSQL and
>>>>>>> application messaging communities. But as long as the VCL web server(s)
>>>>>>> use local, file-based semaphores to control resource allocation, this
>>>>>>> will be a hard nut to crack.
>>>>>>>
>>>>>>> If I were going to tackle this problem, I would take one of two general
>>>>>>> approaches:
>>>>>>>
>>>>>>> 1) Use an asynchronous messaging queue (i.e. Apache ActiveMQ or
>>>>>>> something
>>>>>>> like Redis)
>>>>>>>
>>>>>>> In this way, when a reservation is made, the request is pushed onto a
>>>>>>> centralized queue, and some intermediate (i.e. vcld) process will shift
>>>>>>> messages off the queue and assign the reservations. If we used ActiveMQ,
>>>>>>> the php and perl code would communicate with the message broker over the
>>>>>>> STOMP protocol [1]. Redis would also work because it runs in a single
>>>>>>> thread and all operations are atomic -- requests can simply be pushed
>>>>>>> onto a FIFO-type list structure [2].
>>>>>>>
>>>>>>> 2. Use a NoSQL database such as CouchDB or Riak that uses a type of
>>>>>>> optimistic locking model for all writes.
>>>>>>>
>>>>>>> That is, if reservations are stored in such a way that the resource id
>>>>>>> (i.e. computer id) forms the database key, the assignment of a user to a
>>>>>>> particular compute resource requires sending the correct revision_id of
>>>>>>> that resource in order for the write to be successful. If successful, it
>>>>>>> returns an HTTP 200 status code and the client proceeds normally;
>>>>>>> otherwise, it sends a 409 header and it is up to the client to try again
>>>>>>> [3]. It would also be possible to use the existing MySQL database to
>>>>>>> emulate something like #2, but under a heavy load, I imagine that the
>>>>>>> row
>>>>>>> locking could really slow things down. Using Couch would really be much
>>>>>>> more scalable than trying to do everything in MySQL.
>>>>>>>
>>>>>>> Either of these approaches would then allow you to distribute the web
>>>>>>> front end across multiple machines.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Aaron Coburn
>>>>>>>
>>>>>>> [1] http://activemq.apache.org/stomp.html
>>>>>>> [2] http://redis.io/commands/lpush
>>>>>>> [3] http://wiki.apache.org/couchdb/HTTP_Document_API#PUT
>>>>>>>
>>>>>>> On Jul 17, 2013, at 1:06 PM, Aaron Peeler <aa...@ncsu.edu> wrote:
>>>>>>>> This is definitely desired.
>>>>>>>>
>>>>>>>> An additional challenge with multiple web servers is to figure how to
>>>>>>>> sanely lock/unlock the resource to prevent it from getting assigned
>>>>>>>> two separate users that are making requests at the same instant.
>>>>>>>>
>>>>>>>> AP
>>>>>>>>
>>>>>>>> On Wed, Jul 17, 2013 at 12:57 PM, James O'Dell <jo...@fullerton.edu>
>>>>>
>>>>> wrote:
>>>>>>>>> I also tried load balancing the web. I didn't have any success.
>>>>>>>>>
>>>>>>>>> I also tried an SSL offload using the F5
>>>>>>>>>
>>>>>>>>> The SSLoffload didn't work because
>>>>>>>>> an internal VCL check noticed the header wasn't https
>>>>>>>>> and redirected to https. Basically it kept looping
>>>>>>>>>
>>>>>>>>> On 7/17/2013 9:36 AM, Dmitri Chebotarov wrote:
>>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>> I would like to load balance multiple front-end VCL servers.
>>>>>>>>> It is F5 load balancer. The LB configuration allows to enable session
>>>>>>>>> persistency.
>>>>>>>>> Is this will be OK with VCL's front-end? I remember someone mentioned
>>>>>>>>> some
>>>>>>>>> issues with having multiple front-end servers, but don't remember
>>>>>>>>> details.
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Aaron Peeler
>>>>>>>> Program Manager
>>>>>>>> Virtual Computing Lab
>>>>>>>> NC State University
>>>>>>>>
>>>>>>>> All electronic mail messages in connection with State business which
>>>>>>>> are sent to or received by this account are subject to the NC Public
>>>>>>>> Records Law and may be disclosed to third parties.
>>>>>
>>>>> - --
>>>>> - -------------------------------
>>>>> Josh Thompson
>>>>> VCL Developer
>>>>> North Carolina State University
>>>>>
>>>>> my GPG/PGP key can be found at pgp.mit.edu
>>>>>
>>>>> All electronic mail messages in connection with State business which
>>>>> are sent to or received by this account are subject to the NC Public
>>>>> Records Law and may be disclosed to third parties.
>>>>> -----BEGIN PGP SIGNATURE-----
>>>>> Version: GnuPG v2.0.19 (GNU/Linux)
>>>>>
>>>>> iEYEARECAAYFAlHtojYACgkQV/LQcNdtPQNNjwCfQ+PRt/yZXI4tf2YDiH2IZu2m
>>>>> coYAn2QbjICSa5MUR0cR9DIxniZVQFtK
>>>>> =/YZs
>>>>> -----END PGP SIGNATURE-----
>>> - --
>>> - -------------------------------
>>> Josh Thompson
>>> VCL Developer
>>> North Carolina State University
>>>
>>> my GPG/PGP key can be found at pgp.mit.edu
>>>
>>> All electronic mail messages in connection with State business which
>>> are sent to or received by this account are subject to the NC Public
>>> Records Law and may be disclosed to third parties.
>>> -----BEGIN PGP SIGNATURE-----
>>> Version: GnuPG v2.0.19 (GNU/Linux)
>>>
>>> iEYEARECAAYFAlHufrkACgkQV/LQcNdtPQPSagCdHzGSWMPRoJjTYR/nqdJScmW/
>>> k2EAn2cGY5gKBZZgkRsG9ebscAVyOcYa
>>> =oxwt
>>> -----END PGP SIGNATURE-----
>>>
>>
>>
>>
>> --
>> Mark Gardner
>> --
>
>
--
Thank you,
Dmitri Chebotarov
VCL Sys Eng, Engineering & Architectural Support, TSD - Ent Servers & Messaging
223 Aquia Building, Ffx, MSN: 1B5
Phone: (703) 993-6175 | Fax: (703) 993-3404
Re: VCL front-end and load balancer
Posted by Aaron Coburn <ac...@amherst.edu>.
I agree with Mark.
Stored procedures seem nice, but they are incredibly difficult to document, and if you have to make changes later, they can be a nightmare to debug.
Aaron
On Jul 23, 2013, at 3:58 PM, Mark Gardner <mk...@vt.edu> wrote:
> I would vote against stored procedures. Keep the DB simple.
>
> Mark
>
> On Tue, Jul 23, 2013 at 9:01 AM, Josh Thompson <jo...@ncsu.edu> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Hi Jim,
>>
>> What are you saying would not be a problem? I know having 2 systems trying to
>> write at the same time would be okay, but the scheduling code has to read a
>> bunch of data from the database, analyze it to select a computer, and then
>> write information to the database. The semaphore is needed because the state
>> of the system needs to remain the same between the reading of the data and the
>> writing of it. Otherwise, computers that were not assigned during the read
>> may get assigned before the write happens, which can result in multiple people
>> getting the same computer. We actually had this happen back in 2004 when we
>> were having our initial pilot of VCL, which is what prompted adding the
>> semaphore.
>>
>> Another option is to actually move the scheduling code to a stored procedure
>> in the database. However, that would require someone with much better stored
>> procedure skills than mine.
>>
>> Josh
>>
>> On Monday, July 22, 2013 2:36:28 PM James O'Dell wrote:
>>> Hey Josh,
>>>
>>> I've been F5 load balancing, and SSL offloading, Moodle for the last
>>> couple of years with out a problem.
>>> Moodle just insists the database engine be ACID compliant
>>>
>>> If the database engine is transaction safe (i.e. InnoDB) you shouldn't
>>> have a problem.
>>> It's really just a matter of handling the results (i.e. if someone else
>>> grabs the reservation
>>> what do you do)
>>>
>>> Imho relying on the database seems like the simplest/cleanest approach
>>>
>>> __Jim
>>>
>>> On 7/22/2013 2:20 PM, Josh Thompson wrote:
>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>> Hash: SHA1
>>>>
>>>> The problem with load balancing the front end is that there is a semaphore
>>>> lock (php's built in implementation) around the scheduling code. Without
>>>> it, 2 users coming in at the same time can get assigned the same
>>>> computer. So, having multiple frontends would allow the same thing to
>>>> happen.
>>>>
>>>> I've considered just locking a table in the database as a semaphore.
>>>> However, in my testing of this, if the web server locks a table and then
>>>> goes offline before unlocking it, the table stays locked. So, done this
>>>> way, if there were multiple frontends and one went offline while in the
>>>> middle of the lock, none of them would be able to create reservations. I
>>>> don't remember what had to be done to release the lock.
>>>>
>>>> The option I'd like to use that I've not gotten around to implementing (I
>>>> believe there is a really old JIRA issue for it) is to add an XMLRPC API
>>>> function for calling the scheduling code. Then, have one frontend
>>>> assigned as the one that would handle all of the scheduling. The others
>>>> would make calls to that one for just the scheduling part, but would do
>>>> everything else normally. Optionally, there could be an election process
>>>> so that a new frontend could be selected to do the scheduling if the one
>>>> that had been doing it went down.
>>>>
>>>> Aaron C. listed some good ideas, but I think the above would be more
>>>> straightforward since it would not involve bringing in other technologies.
>>>>
>>>> Josh
>>>>
>>>> On Thursday, July 18, 2013 9:49:39 AM Aaron Peeler wrote:
>>>>> I like both of these approaches, especially 1, but I'm not sure of the
>>>>> effort it would take to convert.
>>>>> -AP
>>>>>
>>>>> On Wed, Jul 17, 2013 at 2:59 PM, Aaron Coburn <ac...@amherst.edu>
>> wrote:
>>>>>> In my opinion, this issue has largely been solved by the NoSQL and
>>>>>> application messaging communities. But as long as the VCL web server(s)
>>>>>> use local, file-based semaphores to control resource allocation, this
>>>>>> will be a hard nut to crack.
>>>>>>
>>>>>> If I were going to tackle this problem, I would take one of two general
>>>>>> approaches:
>>>>>>
>>>>>> 1) Use an asynchronous messaging queue (i.e. Apache ActiveMQ or
>>>>>> something
>>>>>> like Redis)
>>>>>>
>>>>>> In this way, when a reservation is made, the request is pushed onto a
>>>>>> centralized queue, and some intermediate (i.e. vcld) process will shift
>>>>>> messages off the queue and assign the reservations. If we used ActiveMQ,
>>>>>> the php and perl code would communicate with the message broker over the
>>>>>> STOMP protocol [1]. Redis would also work because it runs in a single
>>>>>> thread and all operations are atomic -- requests can simply be pushed
>>>>>> onto a FIFO-type list structure [2].
>>>>>>
>>>>>> 2. Use a NoSQL database such as CouchDB or Riak that uses a type of
>>>>>> optimistic locking model for all writes.
>>>>>>
>>>>>> That is, if reservations are stored in such a way that the resource id
>>>>>> (i.e. computer id) forms the database key, the assignment of a user to a
>>>>>> particular compute resource requires sending the correct revision_id of
>>>>>> that resource in order for the write to be successful. If successful, it
>>>>>> returns an HTTP 200 status code and the client proceeds normally;
>>>>>> otherwise, it sends a 409 header and it is up to the client to try again
>>>>>> [3]. It would also be possible to use the existing MySQL database to
>>>>>> emulate something like #2, but under a heavy load, I imagine that the
>>>>>> row
>>>>>> locking could really slow things down. Using Couch would really be much
>>>>>> more scalable than trying to do everything in MySQL.
>>>>>>
>>>>>> Either of these approaches would then allow you to distribute the web
>>>>>> front end across multiple machines.
>>>>>>
>>>>>> Cheers,
>>>>>> Aaron Coburn
>>>>>>
>>>>>> [1] http://activemq.apache.org/stomp.html
>>>>>> [2] http://redis.io/commands/lpush
>>>>>> [3] http://wiki.apache.org/couchdb/HTTP_Document_API#PUT
>>>>>>
>>>>>> On Jul 17, 2013, at 1:06 PM, Aaron Peeler <aa...@ncsu.edu> wrote:
>>>>>>> This is definitely desired.
>>>>>>>
>>>>>>> An additional challenge with multiple web servers is to figure how to
>>>>>>> sanely lock/unlock the resource to prevent it from getting assigned
>>>>>>> two separate users that are making requests at the same instant.
>>>>>>>
>>>>>>> AP
>>>>>>>
>>>>>>> On Wed, Jul 17, 2013 at 12:57 PM, James O'Dell <jo...@fullerton.edu>
>>>>
>>>> wrote:
>>>>>>>> I also tried load balancing the web. I didn't have any success.
>>>>>>>>
>>>>>>>> I also tried an SSL offload using the F5
>>>>>>>>
>>>>>>>> The SSLoffload didn't work because
>>>>>>>> an internal VCL check noticed the header wasn't https
>>>>>>>> and redirected to https. Basically it kept looping
>>>>>>>>
>>>>>>>> On 7/17/2013 9:36 AM, Dmitri Chebotarov wrote:
>>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> I would like to load balance multiple front-end VCL servers.
>>>>>>>> It is F5 load balancer. The LB configuration allows to enable session
>>>>>>>> persistency.
>>>>>>>> Is this will be OK with VCL's front-end? I remember someone mentioned
>>>>>>>> some
>>>>>>>> issues with having multiple front-end servers, but don't remember
>>>>>>>> details.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>
>>>>>>> --
>>>>>>> Aaron Peeler
>>>>>>> Program Manager
>>>>>>> Virtual Computing Lab
>>>>>>> NC State University
>>>>>>>
>>>>>>> All electronic mail messages in connection with State business which
>>>>>>> are sent to or received by this account are subject to the NC Public
>>>>>>> Records Law and may be disclosed to third parties.
>>>>
>>>> - --
>>>> - -------------------------------
>>>> Josh Thompson
>>>> VCL Developer
>>>> North Carolina State University
>>>>
>>>> my GPG/PGP key can be found at pgp.mit.edu
>>>>
>>>> All electronic mail messages in connection with State business which
>>>> are sent to or received by this account are subject to the NC Public
>>>> Records Law and may be disclosed to third parties.
>>>> -----BEGIN PGP SIGNATURE-----
>>>> Version: GnuPG v2.0.19 (GNU/Linux)
>>>>
>>>> iEYEARECAAYFAlHtojYACgkQV/LQcNdtPQNNjwCfQ+PRt/yZXI4tf2YDiH2IZu2m
>>>> coYAn2QbjICSa5MUR0cR9DIxniZVQFtK
>>>> =/YZs
>>>> -----END PGP SIGNATURE-----
>> - --
>> - -------------------------------
>> Josh Thompson
>> VCL Developer
>> North Carolina State University
>>
>> my GPG/PGP key can be found at pgp.mit.edu
>>
>> All electronic mail messages in connection with State business which
>> are sent to or received by this account are subject to the NC Public
>> Records Law and may be disclosed to third parties.
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v2.0.19 (GNU/Linux)
>>
>> iEYEARECAAYFAlHufrkACgkQV/LQcNdtPQPSagCdHzGSWMPRoJjTYR/nqdJScmW/
>> k2EAn2cGY5gKBZZgkRsG9ebscAVyOcYa
>> =oxwt
>> -----END PGP SIGNATURE-----
>>
>
>
>
> --
> Mark Gardner
> --
Re: VCL front-end and load balancer
Posted by Aaron Coburn <ac...@amherst.edu>.
I agree with Mark.
Stored procedures seem nice, but they are incredibly difficult to document, and if you have to make changes later, they can be a nightmare to debug.
Aaron
On Jul 23, 2013, at 3:58 PM, Mark Gardner <mk...@vt.edu> wrote:
> I would vote against stored procedures. Keep the DB simple.
>
> Mark
>
> On Tue, Jul 23, 2013 at 9:01 AM, Josh Thompson <jo...@ncsu.edu> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Hi Jim,
>>
>> What are you saying would not be a problem? I know having 2 systems trying to
>> write at the same time would be okay, but the scheduling code has to read a
>> bunch of data from the database, analyze it to select a computer, and then
>> write information to the database. The semaphore is needed because the state
>> of the system needs to remain the same between the reading of the data and the
>> writing of it. Otherwise, computers that were not assigned during the read
>> may get assigned before the write happens, which can result in multiple people
>> getting the same computer. We actually had this happen back in 2004 when we
>> were having our initial pilot of VCL, which is what prompted adding the
>> semaphore.
>>
>> Another option is to actually move the scheduling code to a stored procedure
>> in the database. However, that would require someone with much better stored
>> procedure skills than mine.
>>
>> Josh
>>
>> On Monday, July 22, 2013 2:36:28 PM James O'Dell wrote:
>>> Hey Josh,
>>>
>>> I've been F5 load balancing, and SSL offloading, Moodle for the last
>>> couple of years with out a problem.
>>> Moodle just insists the database engine be ACID compliant
>>>
>>> If the database engine is transaction safe (i.e. InnoDB) you shouldn't
>>> have a problem.
>>> It's really just a matter of handling the results (i.e. if someone else
>>> grabs the reservation
>>> what do you do)
>>>
>>> Imho relying on the database seems like the simplest/cleanest approach
>>>
>>> __Jim
>>>
>>> On 7/22/2013 2:20 PM, Josh Thompson wrote:
>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>> Hash: SHA1
>>>>
>>>> The problem with load balancing the front end is that there is a semaphore
>>>> lock (php's built in implementation) around the scheduling code. Without
>>>> it, 2 users coming in at the same time can get assigned the same
>>>> computer. So, having multiple frontends would allow the same thing to
>>>> happen.
>>>>
>>>> I've considered just locking a table in the database as a semaphore.
>>>> However, in my testing of this, if the web server locks a table and then
>>>> goes offline before unlocking it, the table stays locked. So, done this
>>>> way, if there were multiple frontends and one went offline while in the
>>>> middle of the lock, none of them would be able to create reservations. I
>>>> don't remember what had to be done to release the lock.
>>>>
>>>> The option I'd like to use that I've not gotten around to implementing (I
>>>> believe there is a really old JIRA issue for it) is to add an XMLRPC API
>>>> function for calling the scheduling code. Then, have one frontend
>>>> assigned as the one that would handle all of the scheduling. The others
>>>> would make calls to that one for just the scheduling part, but would do
>>>> everything else normally. Optionally, there could be an election process
>>>> so that a new frontend could be selected to do the scheduling if the one
>>>> that had been doing it went down.
>>>>
>>>> Aaron C. listed some good ideas, but I think the above would be more
>>>> straightforward since it would not involve bringing in other technologies.
>>>>
>>>> Josh
>>>>
>>>> On Thursday, July 18, 2013 9:49:39 AM Aaron Peeler wrote:
>>>>> I like both of these approaches, especially 1, but I'm not sure of the
>>>>> effort it would take to convert.
>>>>> -AP
>>>>>
>>>>> On Wed, Jul 17, 2013 at 2:59 PM, Aaron Coburn <ac...@amherst.edu>
>> wrote:
>>>>>> In my opinion, this issue has largely been solved by the NoSQL and
>>>>>> application messaging communities. But as long as the VCL web server(s)
>>>>>> use local, file-based semaphores to control resource allocation, this
>>>>>> will be a hard nut to crack.
>>>>>>
>>>>>> If I were going to tackle this problem, I would take one of two general
>>>>>> approaches:
>>>>>>
>>>>>> 1) Use an asynchronous messaging queue (i.e. Apache ActiveMQ or
>>>>>> something
>>>>>> like Redis)
>>>>>>
>>>>>> In this way, when a reservation is made, the request is pushed onto a
>>>>>> centralized queue, and some intermediate (i.e. vcld) process will shift
>>>>>> messages off the queue and assign the reservations. If we used ActiveMQ,
>>>>>> the php and perl code would communicate with the message broker over the
>>>>>> STOMP protocol [1]. Redis would also work because it runs in a single
>>>>>> thread and all operations are atomic -- requests can simply be pushed
>>>>>> onto a FIFO-type list structure [2].
>>>>>>
>>>>>> 2. Use a NoSQL database such as CouchDB or Riak that uses a type of
>>>>>> optimistic locking model for all writes.
>>>>>>
>>>>>> That is, if reservations are stored in such a way that the resource id
>>>>>> (i.e. computer id) forms the database key, the assignment of a user to a
>>>>>> particular compute resource requires sending the correct revision_id of
>>>>>> that resource in order for the write to be successful. If successful, it
>>>>>> returns an HTTP 200 status code and the client proceeds normally;
>>>>>> otherwise, it sends a 409 header and it is up to the client to try again
>>>>>> [3]. It would also be possible to use the existing MySQL database to
>>>>>> emulate something like #2, but under a heavy load, I imagine that the
>>>>>> row
>>>>>> locking could really slow things down. Using Couch would really be much
>>>>>> more scalable than trying to do everything in MySQL.
>>>>>>
>>>>>> Either of these approaches would then allow you to distribute the web
>>>>>> front end across multiple machines.
>>>>>>
>>>>>> Cheers,
>>>>>> Aaron Coburn
>>>>>>
>>>>>> [1] http://activemq.apache.org/stomp.html
>>>>>> [2] http://redis.io/commands/lpush
>>>>>> [3] http://wiki.apache.org/couchdb/HTTP_Document_API#PUT
>>>>>>
>>>>>> On Jul 17, 2013, at 1:06 PM, Aaron Peeler <aa...@ncsu.edu> wrote:
>>>>>>> This is definitely desired.
>>>>>>>
>>>>>>> An additional challenge with multiple web servers is to figure how to
>>>>>>> sanely lock/unlock the resource to prevent it from getting assigned
>>>>>>> two separate users that are making requests at the same instant.
>>>>>>>
>>>>>>> AP
>>>>>>>
>>>>>>> On Wed, Jul 17, 2013 at 12:57 PM, James O'Dell <jo...@fullerton.edu>
>>>>
>>>> wrote:
>>>>>>>> I also tried load balancing the web. I didn't have any success.
>>>>>>>>
>>>>>>>> I also tried an SSL offload using the F5
>>>>>>>>
>>>>>>>> The SSLoffload didn't work because
>>>>>>>> an internal VCL check noticed the header wasn't https
>>>>>>>> and redirected to https. Basically it kept looping
>>>>>>>>
>>>>>>>> On 7/17/2013 9:36 AM, Dmitri Chebotarov wrote:
>>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> I would like to load balance multiple front-end VCL servers.
>>>>>>>> It is F5 load balancer. The LB configuration allows to enable session
>>>>>>>> persistency.
>>>>>>>> Is this will be OK with VCL's front-end? I remember someone mentioned
>>>>>>>> some
>>>>>>>> issues with having multiple front-end servers, but don't remember
>>>>>>>> details.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>
>>>>>>> --
>>>>>>> Aaron Peeler
>>>>>>> Program Manager
>>>>>>> Virtual Computing Lab
>>>>>>> NC State University
>>>>>>>
>>>>>>> All electronic mail messages in connection with State business which
>>>>>>> are sent to or received by this account are subject to the NC Public
>>>>>>> Records Law and may be disclosed to third parties.
>>>>
>>>> - --
>>>> - -------------------------------
>>>> Josh Thompson
>>>> VCL Developer
>>>> North Carolina State University
>>>>
>>>> my GPG/PGP key can be found at pgp.mit.edu
>>>>
>>>> All electronic mail messages in connection with State business which
>>>> are sent to or received by this account are subject to the NC Public
>>>> Records Law and may be disclosed to third parties.
>>>> -----BEGIN PGP SIGNATURE-----
>>>> Version: GnuPG v2.0.19 (GNU/Linux)
>>>>
>>>> iEYEARECAAYFAlHtojYACgkQV/LQcNdtPQNNjwCfQ+PRt/yZXI4tf2YDiH2IZu2m
>>>> coYAn2QbjICSa5MUR0cR9DIxniZVQFtK
>>>> =/YZs
>>>> -----END PGP SIGNATURE-----
>> - --
>> - -------------------------------
>> Josh Thompson
>> VCL Developer
>> North Carolina State University
>>
>> my GPG/PGP key can be found at pgp.mit.edu
>>
>> All electronic mail messages in connection with State business which
>> are sent to or received by this account are subject to the NC Public
>> Records Law and may be disclosed to third parties.
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v2.0.19 (GNU/Linux)
>>
>> iEYEARECAAYFAlHufrkACgkQV/LQcNdtPQPSagCdHzGSWMPRoJjTYR/nqdJScmW/
>> k2EAn2cGY5gKBZZgkRsG9ebscAVyOcYa
>> =oxwt
>> -----END PGP SIGNATURE-----
>>
>
>
>
> --
> Mark Gardner
> --
Re: VCL front-end and load balancer
Posted by Mark Gardner <mk...@vt.edu>.
I would vote against stored procedures. Keep the DB simple.
Mark
On Tue, Jul 23, 2013 at 9:01 AM, Josh Thompson <jo...@ncsu.edu> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi Jim,
>
> What are you saying would not be a problem? I know having 2 systems trying to
> write at the same time would be okay, but the scheduling code has to read a
> bunch of data from the database, analyze it to select a computer, and then
> write information to the database. The semaphore is needed because the state
> of the system needs to remain the same between the reading of the data and the
> writing of it. Otherwise, computers that were not assigned during the read
> may get assigned before the write happens, which can result in multiple people
> getting the same computer. We actually had this happen back in 2004 when we
> were having our initial pilot of VCL, which is what prompted adding the
> semaphore.
>
> Another option is to actually move the scheduling code to a stored procedure
> in the database. However, that would require someone with much better stored
> procedure skills than mine.
>
> Josh
>
> On Monday, July 22, 2013 2:36:28 PM James O'Dell wrote:
>> Hey Josh,
>>
>> I've been F5 load balancing, and SSL offloading, Moodle for the last
>> couple of years with out a problem.
>> Moodle just insists the database engine be ACID compliant
>>
>> If the database engine is transaction safe (i.e. InnoDB) you shouldn't
>> have a problem.
>> It's really just a matter of handling the results (i.e. if someone else
>> grabs the reservation
>> what do you do)
>>
>> Imho relying on the database seems like the simplest/cleanest approach
>>
>> __Jim
>>
>> On 7/22/2013 2:20 PM, Josh Thompson wrote:
>> > -----BEGIN PGP SIGNED MESSAGE-----
>> > Hash: SHA1
>> >
>> > The problem with load balancing the front end is that there is a semaphore
>> > lock (php's built in implementation) around the scheduling code. Without
>> > it, 2 users coming in at the same time can get assigned the same
>> > computer. So, having multiple frontends would allow the same thing to
>> > happen.
>> >
>> > I've considered just locking a table in the database as a semaphore.
>> > However, in my testing of this, if the web server locks a table and then
>> > goes offline before unlocking it, the table stays locked. So, done this
>> > way, if there were multiple frontends and one went offline while in the
>> > middle of the lock, none of them would be able to create reservations. I
>> > don't remember what had to be done to release the lock.
>> >
>> > The option I'd like to use that I've not gotten around to implementing (I
>> > believe there is a really old JIRA issue for it) is to add an XMLRPC API
>> > function for calling the scheduling code. Then, have one frontend
>> > assigned as the one that would handle all of the scheduling. The others
>> > would make calls to that one for just the scheduling part, but would do
>> > everything else normally. Optionally, there could be an election process
>> > so that a new frontend could be selected to do the scheduling if the one
>> > that had been doing it went down.
>> >
>> > Aaron C. listed some good ideas, but I think the above would be more
>> > straightforward since it would not involve bringing in other technologies.
>> >
>> > Josh
>> >
>> > On Thursday, July 18, 2013 9:49:39 AM Aaron Peeler wrote:
>> >> I like both of these approaches, especially 1, but I'm not sure of the
>> >> effort it would take to convert.
>> >> -AP
>> >>
>> >> On Wed, Jul 17, 2013 at 2:59 PM, Aaron Coburn <ac...@amherst.edu>
> wrote:
>> >>> In my opinion, this issue has largely been solved by the NoSQL and
>> >>> application messaging communities. But as long as the VCL web server(s)
>> >>> use local, file-based semaphores to control resource allocation, this
>> >>> will be a hard nut to crack.
>> >>>
>> >>> If I were going to tackle this problem, I would take one of two general
>> >>> approaches:
>> >>>
>> >>> 1) Use an asynchronous messaging queue (i.e. Apache ActiveMQ or
>> >>> something
>> >>> like Redis)
>> >>>
>> >>> In this way, when a reservation is made, the request is pushed onto a
>> >>> centralized queue, and some intermediate (i.e. vcld) process will shift
>> >>> messages off the queue and assign the reservations. If we used ActiveMQ,
>> >>> the php and perl code would communicate with the message broker over the
>> >>> STOMP protocol [1]. Redis would also work because it runs in a single
>> >>> thread and all operations are atomic -- requests can simply be pushed
>> >>> onto a FIFO-type list structure [2].
>> >>>
>> >>> 2. Use a NoSQL database such as CouchDB or Riak that uses a type of
>> >>> optimistic locking model for all writes.
>> >>>
>> >>> That is, if reservations are stored in such a way that the resource id
>> >>> (i.e. computer id) forms the database key, the assignment of a user to a
>> >>> particular compute resource requires sending the correct revision_id of
>> >>> that resource in order for the write to be successful. If successful, it
>> >>> returns an HTTP 200 status code and the client proceeds normally;
>> >>> otherwise, it sends a 409 header and it is up to the client to try again
>> >>> [3]. It would also be possible to use the existing MySQL database to
>> >>> emulate something like #2, but under a heavy load, I imagine that the
>> >>> row
>> >>> locking could really slow things down. Using Couch would really be much
>> >>> more scalable than trying to do everything in MySQL.
>> >>>
>> >>> Either of these approaches would then allow you to distribute the web
>> >>> front end across multiple machines.
>> >>>
>> >>> Cheers,
>> >>> Aaron Coburn
>> >>>
>> >>> [1] http://activemq.apache.org/stomp.html
>> >>> [2] http://redis.io/commands/lpush
>> >>> [3] http://wiki.apache.org/couchdb/HTTP_Document_API#PUT
>> >>>
>> >>> On Jul 17, 2013, at 1:06 PM, Aaron Peeler <aa...@ncsu.edu> wrote:
>> >>>> This is definitely desired.
>> >>>>
>> >>>> An additional challenge with multiple web servers is to figure how to
>> >>>> sanely lock/unlock the resource to prevent it from getting assigned
>> >>>> two separate users that are making requests at the same instant.
>> >>>>
>> >>>> AP
>> >>>>
>> >>>> On Wed, Jul 17, 2013 at 12:57 PM, James O'Dell <jo...@fullerton.edu>
>> >
>> > wrote:
>> >>>>> I also tried load balancing the web. I didn't have any success.
>> >>>>>
>> >>>>> I also tried an SSL offload using the F5
>> >>>>>
>> >>>>> The SSLoffload didn't work because
>> >>>>> an internal VCL check noticed the header wasn't https
>> >>>>> and redirected to https. Basically it kept looping
>> >>>>>
>> >>>>> On 7/17/2013 9:36 AM, Dmitri Chebotarov wrote:
>> >>>>>
>> >>>>> Hi
>> >>>>>
>> >>>>> I would like to load balance multiple front-end VCL servers.
>> >>>>> It is F5 load balancer. The LB configuration allows to enable session
>> >>>>> persistency.
>> >>>>> Is this will be OK with VCL's front-end? I remember someone mentioned
>> >>>>> some
>> >>>>> issues with having multiple front-end servers, but don't remember
>> >>>>> details.
>> >>>>>
>> >>>>> Thanks.
>> >>>>
>> >>>> --
>> >>>> Aaron Peeler
>> >>>> Program Manager
>> >>>> Virtual Computing Lab
>> >>>> NC State University
>> >>>>
>> >>>> All electronic mail messages in connection with State business which
>> >>>> are sent to or received by this account are subject to the NC Public
>> >>>> Records Law and may be disclosed to third parties.
>> >
>> > - --
>> > - -------------------------------
>> > Josh Thompson
>> > VCL Developer
>> > North Carolina State University
>> >
>> > my GPG/PGP key can be found at pgp.mit.edu
>> >
>> > All electronic mail messages in connection with State business which
>> > are sent to or received by this account are subject to the NC Public
>> > Records Law and may be disclosed to third parties.
>> > -----BEGIN PGP SIGNATURE-----
>> > Version: GnuPG v2.0.19 (GNU/Linux)
>> >
>> > iEYEARECAAYFAlHtojYACgkQV/LQcNdtPQNNjwCfQ+PRt/yZXI4tf2YDiH2IZu2m
>> > coYAn2QbjICSa5MUR0cR9DIxniZVQFtK
>> > =/YZs
>> > -----END PGP SIGNATURE-----
> - --
> - -------------------------------
> Josh Thompson
> VCL Developer
> North Carolina State University
>
> my GPG/PGP key can be found at pgp.mit.edu
>
> All electronic mail messages in connection with State business which
> are sent to or received by this account are subject to the NC Public
> Records Law and may be disclosed to third parties.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.19 (GNU/Linux)
>
> iEYEARECAAYFAlHufrkACgkQV/LQcNdtPQPSagCdHzGSWMPRoJjTYR/nqdJScmW/
> k2EAn2cGY5gKBZZgkRsG9ebscAVyOcYa
> =oxwt
> -----END PGP SIGNATURE-----
>
--
Mark Gardner
--
Re: VCL front-end and load balancer
Posted by Mark Gardner <mk...@vt.edu>.
I would vote against stored procedures. Keep the DB simple.
Mark
On Tue, Jul 23, 2013 at 9:01 AM, Josh Thompson <jo...@ncsu.edu> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi Jim,
>
> What are you saying would not be a problem? I know having 2 systems trying to
> write at the same time would be okay, but the scheduling code has to read a
> bunch of data from the database, analyze it to select a computer, and then
> write information to the database. The semaphore is needed because the state
> of the system needs to remain the same between the reading of the data and the
> writing of it. Otherwise, computers that were not assigned during the read
> may get assigned before the write happens, which can result in multiple people
> getting the same computer. We actually had this happen back in 2004 when we
> were having our initial pilot of VCL, which is what prompted adding the
> semaphore.
>
> Another option is to actually move the scheduling code to a stored procedure
> in the database. However, that would require someone with much better stored
> procedure skills than mine.
>
> Josh
>
> On Monday, July 22, 2013 2:36:28 PM James O'Dell wrote:
>> Hey Josh,
>>
>> I've been F5 load balancing, and SSL offloading, Moodle for the last
>> couple of years with out a problem.
>> Moodle just insists the database engine be ACID compliant
>>
>> If the database engine is transaction safe (i.e. InnoDB) you shouldn't
>> have a problem.
>> It's really just a matter of handling the results (i.e. if someone else
>> grabs the reservation
>> what do you do)
>>
>> Imho relying on the database seems like the simplest/cleanest approach
>>
>> __Jim
>>
>> On 7/22/2013 2:20 PM, Josh Thompson wrote:
>> > -----BEGIN PGP SIGNED MESSAGE-----
>> > Hash: SHA1
>> >
>> > The problem with load balancing the front end is that there is a semaphore
>> > lock (php's built in implementation) around the scheduling code. Without
>> > it, 2 users coming in at the same time can get assigned the same
>> > computer. So, having multiple frontends would allow the same thing to
>> > happen.
>> >
>> > I've considered just locking a table in the database as a semaphore.
>> > However, in my testing of this, if the web server locks a table and then
>> > goes offline before unlocking it, the table stays locked. So, done this
>> > way, if there were multiple frontends and one went offline while in the
>> > middle of the lock, none of them would be able to create reservations. I
>> > don't remember what had to be done to release the lock.
>> >
>> > The option I'd like to use that I've not gotten around to implementing (I
>> > believe there is a really old JIRA issue for it) is to add an XMLRPC API
>> > function for calling the scheduling code. Then, have one frontend
>> > assigned as the one that would handle all of the scheduling. The others
>> > would make calls to that one for just the scheduling part, but would do
>> > everything else normally. Optionally, there could be an election process
>> > so that a new frontend could be selected to do the scheduling if the one
>> > that had been doing it went down.
>> >
>> > Aaron C. listed some good ideas, but I think the above would be more
>> > straightforward since it would not involve bringing in other technologies.
>> >
>> > Josh
>> >
>> > On Thursday, July 18, 2013 9:49:39 AM Aaron Peeler wrote:
>> >> I like both of these approaches, especially 1, but I'm not sure of the
>> >> effort it would take to convert.
>> >> -AP
>> >>
>> >> On Wed, Jul 17, 2013 at 2:59 PM, Aaron Coburn <ac...@amherst.edu>
> wrote:
>> >>> In my opinion, this issue has largely been solved by the NoSQL and
>> >>> application messaging communities. But as long as the VCL web server(s)
>> >>> use local, file-based semaphores to control resource allocation, this
>> >>> will be a hard nut to crack.
>> >>>
>> >>> If I were going to tackle this problem, I would take one of two general
>> >>> approaches:
>> >>>
>> >>> 1) Use an asynchronous messaging queue (i.e. Apache ActiveMQ or
>> >>> something
>> >>> like Redis)
>> >>>
>> >>> In this way, when a reservation is made, the request is pushed onto a
>> >>> centralized queue, and some intermediate (i.e. vcld) process will shift
>> >>> messages off the queue and assign the reservations. If we used ActiveMQ,
>> >>> the php and perl code would communicate with the message broker over the
>> >>> STOMP protocol [1]. Redis would also work because it runs in a single
>> >>> thread and all operations are atomic -- requests can simply be pushed
>> >>> onto a FIFO-type list structure [2].
>> >>>
>> >>> 2. Use a NoSQL database such as CouchDB or Riak that uses a type of
>> >>> optimistic locking model for all writes.
>> >>>
>> >>> That is, if reservations are stored in such a way that the resource id
>> >>> (i.e. computer id) forms the database key, the assignment of a user to a
>> >>> particular compute resource requires sending the correct revision_id of
>> >>> that resource in order for the write to be successful. If successful, it
>> >>> returns an HTTP 200 status code and the client proceeds normally;
>> >>> otherwise, it sends a 409 header and it is up to the client to try again
>> >>> [3]. It would also be possible to use the existing MySQL database to
>> >>> emulate something like #2, but under a heavy load, I imagine that the
>> >>> row
>> >>> locking could really slow things down. Using Couch would really be much
>> >>> more scalable than trying to do everything in MySQL.
>> >>>
>> >>> Either of these approaches would then allow you to distribute the web
>> >>> front end across multiple machines.
>> >>>
>> >>> Cheers,
>> >>> Aaron Coburn
>> >>>
>> >>> [1] http://activemq.apache.org/stomp.html
>> >>> [2] http://redis.io/commands/lpush
>> >>> [3] http://wiki.apache.org/couchdb/HTTP_Document_API#PUT
>> >>>
>> >>> On Jul 17, 2013, at 1:06 PM, Aaron Peeler <aa...@ncsu.edu> wrote:
>> >>>> This is definitely desired.
>> >>>>
>> >>>> An additional challenge with multiple web servers is to figure how to
>> >>>> sanely lock/unlock the resource to prevent it from getting assigned
>> >>>> two separate users that are making requests at the same instant.
>> >>>>
>> >>>> AP
>> >>>>
>> >>>> On Wed, Jul 17, 2013 at 12:57 PM, James O'Dell <jo...@fullerton.edu>
>> >
>> > wrote:
>> >>>>> I also tried load balancing the web. I didn't have any success.
>> >>>>>
>> >>>>> I also tried an SSL offload using the F5
>> >>>>>
>> >>>>> The SSLoffload didn't work because
>> >>>>> an internal VCL check noticed the header wasn't https
>> >>>>> and redirected to https. Basically it kept looping
>> >>>>>
>> >>>>> On 7/17/2013 9:36 AM, Dmitri Chebotarov wrote:
>> >>>>>
>> >>>>> Hi
>> >>>>>
>> >>>>> I would like to load balance multiple front-end VCL servers.
>> >>>>> It is F5 load balancer. The LB configuration allows to enable session
>> >>>>> persistency.
>> >>>>> Is this will be OK with VCL's front-end? I remember someone mentioned
>> >>>>> some
>> >>>>> issues with having multiple front-end servers, but don't remember
>> >>>>> details.
>> >>>>>
>> >>>>> Thanks.
>> >>>>
>> >>>> --
>> >>>> Aaron Peeler
>> >>>> Program Manager
>> >>>> Virtual Computing Lab
>> >>>> NC State University
>> >>>>
>> >>>> All electronic mail messages in connection with State business which
>> >>>> are sent to or received by this account are subject to the NC Public
>> >>>> Records Law and may be disclosed to third parties.
>> >
>> > - --
>> > - -------------------------------
>> > Josh Thompson
>> > VCL Developer
>> > North Carolina State University
>> >
>> > my GPG/PGP key can be found at pgp.mit.edu
>> >
>> > All electronic mail messages in connection with State business which
>> > are sent to or received by this account are subject to the NC Public
>> > Records Law and may be disclosed to third parties.
>> > -----BEGIN PGP SIGNATURE-----
>> > Version: GnuPG v2.0.19 (GNU/Linux)
>> >
>> > iEYEARECAAYFAlHtojYACgkQV/LQcNdtPQNNjwCfQ+PRt/yZXI4tf2YDiH2IZu2m
>> > coYAn2QbjICSa5MUR0cR9DIxniZVQFtK
>> > =/YZs
>> > -----END PGP SIGNATURE-----
> - --
> - -------------------------------
> Josh Thompson
> VCL Developer
> North Carolina State University
>
> my GPG/PGP key can be found at pgp.mit.edu
>
> All electronic mail messages in connection with State business which
> are sent to or received by this account are subject to the NC Public
> Records Law and may be disclosed to third parties.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.19 (GNU/Linux)
>
> iEYEARECAAYFAlHufrkACgkQV/LQcNdtPQPSagCdHzGSWMPRoJjTYR/nqdJScmW/
> k2EAn2cGY5gKBZZgkRsG9ebscAVyOcYa
> =oxwt
> -----END PGP SIGNATURE-----
>
--
Mark Gardner
--
Re: VCL front-end and load balancer
Posted by Josh Thompson <jo...@ncsu.edu>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi Jim,
What are you saying would not be a problem? I know having 2 systems trying to
write at the same time would be okay, but the scheduling code has to read a
bunch of data from the database, analyze it to select a computer, and then
write information to the database. The semaphore is needed because the state
of the system needs to remain the same between the reading of the data and the
writing of it. Otherwise, computers that were not assigned during the read
may get assigned before the write happens, which can result in multiple people
getting the same computer. We actually had this happen back in 2004 when we
were having our initial pilot of VCL, which is what prompted adding the
semaphore.
Another option is to actually move the scheduling code to a stored procedure
in the database. However, that would require someone with much better stored
procedure skills than mine.
Josh
On Monday, July 22, 2013 2:36:28 PM James O'Dell wrote:
> Hey Josh,
>
> I've been F5 load balancing, and SSL offloading, Moodle for the last
> couple of years with out a problem.
> Moodle just insists the database engine be ACID compliant
>
> If the database engine is transaction safe (i.e. InnoDB) you shouldn't
> have a problem.
> It's really just a matter of handling the results (i.e. if someone else
> grabs the reservation
> what do you do)
>
> Imho relying on the database seems like the simplest/cleanest approach
>
> __Jim
>
> On 7/22/2013 2:20 PM, Josh Thompson wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > The problem with load balancing the front end is that there is a semaphore
> > lock (php's built in implementation) around the scheduling code. Without
> > it, 2 users coming in at the same time can get assigned the same
> > computer. So, having multiple frontends would allow the same thing to
> > happen.
> >
> > I've considered just locking a table in the database as a semaphore.
> > However, in my testing of this, if the web server locks a table and then
> > goes offline before unlocking it, the table stays locked. So, done this
> > way, if there were multiple frontends and one went offline while in the
> > middle of the lock, none of them would be able to create reservations. I
> > don't remember what had to be done to release the lock.
> >
> > The option I'd like to use that I've not gotten around to implementing (I
> > believe there is a really old JIRA issue for it) is to add an XMLRPC API
> > function for calling the scheduling code. Then, have one frontend
> > assigned as the one that would handle all of the scheduling. The others
> > would make calls to that one for just the scheduling part, but would do
> > everything else normally. Optionally, there could be an election process
> > so that a new frontend could be selected to do the scheduling if the one
> > that had been doing it went down.
> >
> > Aaron C. listed some good ideas, but I think the above would be more
> > straightforward since it would not involve bringing in other technologies.
> >
> > Josh
> >
> > On Thursday, July 18, 2013 9:49:39 AM Aaron Peeler wrote:
> >> I like both of these approaches, especially 1, but I'm not sure of the
> >> effort it would take to convert.
> >> -AP
> >>
> >> On Wed, Jul 17, 2013 at 2:59 PM, Aaron Coburn <ac...@amherst.edu>
wrote:
> >>> In my opinion, this issue has largely been solved by the NoSQL and
> >>> application messaging communities. But as long as the VCL web server(s)
> >>> use local, file-based semaphores to control resource allocation, this
> >>> will be a hard nut to crack.
> >>>
> >>> If I were going to tackle this problem, I would take one of two general
> >>> approaches:
> >>>
> >>> 1) Use an asynchronous messaging queue (i.e. Apache ActiveMQ or
> >>> something
> >>> like Redis)
> >>>
> >>> In this way, when a reservation is made, the request is pushed onto a
> >>> centralized queue, and some intermediate (i.e. vcld) process will shift
> >>> messages off the queue and assign the reservations. If we used ActiveMQ,
> >>> the php and perl code would communicate with the message broker over the
> >>> STOMP protocol [1]. Redis would also work because it runs in a single
> >>> thread and all operations are atomic -- requests can simply be pushed
> >>> onto a FIFO-type list structure [2].
> >>>
> >>> 2. Use a NoSQL database such as CouchDB or Riak that uses a type of
> >>> optimistic locking model for all writes.
> >>>
> >>> That is, if reservations are stored in such a way that the resource id
> >>> (i.e. computer id) forms the database key, the assignment of a user to a
> >>> particular compute resource requires sending the correct revision_id of
> >>> that resource in order for the write to be successful. If successful, it
> >>> returns an HTTP 200 status code and the client proceeds normally;
> >>> otherwise, it sends a 409 header and it is up to the client to try again
> >>> [3]. It would also be possible to use the existing MySQL database to
> >>> emulate something like #2, but under a heavy load, I imagine that the
> >>> row
> >>> locking could really slow things down. Using Couch would really be much
> >>> more scalable than trying to do everything in MySQL.
> >>>
> >>> Either of these approaches would then allow you to distribute the web
> >>> front end across multiple machines.
> >>>
> >>> Cheers,
> >>> Aaron Coburn
> >>>
> >>> [1] http://activemq.apache.org/stomp.html
> >>> [2] http://redis.io/commands/lpush
> >>> [3] http://wiki.apache.org/couchdb/HTTP_Document_API#PUT
> >>>
> >>> On Jul 17, 2013, at 1:06 PM, Aaron Peeler <aa...@ncsu.edu> wrote:
> >>>> This is definitely desired.
> >>>>
> >>>> An additional challenge with multiple web servers is to figure how to
> >>>> sanely lock/unlock the resource to prevent it from getting assigned
> >>>> two separate users that are making requests at the same instant.
> >>>>
> >>>> AP
> >>>>
> >>>> On Wed, Jul 17, 2013 at 12:57 PM, James O'Dell <jo...@fullerton.edu>
> >
> > wrote:
> >>>>> I also tried load balancing the web. I didn't have any success.
> >>>>>
> >>>>> I also tried an SSL offload using the F5
> >>>>>
> >>>>> The SSLoffload didn't work because
> >>>>> an internal VCL check noticed the header wasn't https
> >>>>> and redirected to https. Basically it kept looping
> >>>>>
> >>>>> On 7/17/2013 9:36 AM, Dmitri Chebotarov wrote:
> >>>>>
> >>>>> Hi
> >>>>>
> >>>>> I would like to load balance multiple front-end VCL servers.
> >>>>> It is F5 load balancer. The LB configuration allows to enable session
> >>>>> persistency.
> >>>>> Is this will be OK with VCL's front-end? I remember someone mentioned
> >>>>> some
> >>>>> issues with having multiple front-end servers, but don't remember
> >>>>> details.
> >>>>>
> >>>>> Thanks.
> >>>>
> >>>> --
> >>>> Aaron Peeler
> >>>> Program Manager
> >>>> Virtual Computing Lab
> >>>> NC State University
> >>>>
> >>>> All electronic mail messages in connection with State business which
> >>>> are sent to or received by this account are subject to the NC Public
> >>>> Records Law and may be disclosed to third parties.
> >
> > - --
> > - -------------------------------
> > Josh Thompson
> > VCL Developer
> > North Carolina State University
> >
> > my GPG/PGP key can be found at pgp.mit.edu
> >
> > All electronic mail messages in connection with State business which
> > are sent to or received by this account are subject to the NC Public
> > Records Law and may be disclosed to third parties.
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v2.0.19 (GNU/Linux)
> >
> > iEYEARECAAYFAlHtojYACgkQV/LQcNdtPQNNjwCfQ+PRt/yZXI4tf2YDiH2IZu2m
> > coYAn2QbjICSa5MUR0cR9DIxniZVQFtK
> > =/YZs
> > -----END PGP SIGNATURE-----
- --
- -------------------------------
Josh Thompson
VCL Developer
North Carolina State University
my GPG/PGP key can be found at pgp.mit.edu
All electronic mail messages in connection with State business which
are sent to or received by this account are subject to the NC Public
Records Law and may be disclosed to third parties.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
iEYEARECAAYFAlHufrkACgkQV/LQcNdtPQPSagCdHzGSWMPRoJjTYR/nqdJScmW/
k2EAn2cGY5gKBZZgkRsG9ebscAVyOcYa
=oxwt
-----END PGP SIGNATURE-----
Re: VCL front-end and load balancer
Posted by Josh Thompson <jo...@ncsu.edu>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi Jim,
What are you saying would not be a problem? I know having 2 systems trying to
write at the same time would be okay, but the scheduling code has to read a
bunch of data from the database, analyze it to select a computer, and then
write information to the database. The semaphore is needed because the state
of the system needs to remain the same between the reading of the data and the
writing of it. Otherwise, computers that were not assigned during the read
may get assigned before the write happens, which can result in multiple people
getting the same computer. We actually had this happen back in 2004 when we
were having our initial pilot of VCL, which is what prompted adding the
semaphore.
Another option is to actually move the scheduling code to a stored procedure
in the database. However, that would require someone with much better stored
procedure skills than mine.
Josh
On Monday, July 22, 2013 2:36:28 PM James O'Dell wrote:
> Hey Josh,
>
> I've been F5 load balancing, and SSL offloading, Moodle for the last
> couple of years with out a problem.
> Moodle just insists the database engine be ACID compliant
>
> If the database engine is transaction safe (i.e. InnoDB) you shouldn't
> have a problem.
> It's really just a matter of handling the results (i.e. if someone else
> grabs the reservation
> what do you do)
>
> Imho relying on the database seems like the simplest/cleanest approach
>
> __Jim
>
> On 7/22/2013 2:20 PM, Josh Thompson wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > The problem with load balancing the front end is that there is a semaphore
> > lock (php's built in implementation) around the scheduling code. Without
> > it, 2 users coming in at the same time can get assigned the same
> > computer. So, having multiple frontends would allow the same thing to
> > happen.
> >
> > I've considered just locking a table in the database as a semaphore.
> > However, in my testing of this, if the web server locks a table and then
> > goes offline before unlocking it, the table stays locked. So, done this
> > way, if there were multiple frontends and one went offline while in the
> > middle of the lock, none of them would be able to create reservations. I
> > don't remember what had to be done to release the lock.
> >
> > The option I'd like to use that I've not gotten around to implementing (I
> > believe there is a really old JIRA issue for it) is to add an XMLRPC API
> > function for calling the scheduling code. Then, have one frontend
> > assigned as the one that would handle all of the scheduling. The others
> > would make calls to that one for just the scheduling part, but would do
> > everything else normally. Optionally, there could be an election process
> > so that a new frontend could be selected to do the scheduling if the one
> > that had been doing it went down.
> >
> > Aaron C. listed some good ideas, but I think the above would be more
> > straightforward since it would not involve bringing in other technologies.
> >
> > Josh
> >
> > On Thursday, July 18, 2013 9:49:39 AM Aaron Peeler wrote:
> >> I like both of these approaches, especially 1, but I'm not sure of the
> >> effort it would take to convert.
> >> -AP
> >>
> >> On Wed, Jul 17, 2013 at 2:59 PM, Aaron Coburn <ac...@amherst.edu>
wrote:
> >>> In my opinion, this issue has largely been solved by the NoSQL and
> >>> application messaging communities. But as long as the VCL web server(s)
> >>> use local, file-based semaphores to control resource allocation, this
> >>> will be a hard nut to crack.
> >>>
> >>> If I were going to tackle this problem, I would take one of two general
> >>> approaches:
> >>>
> >>> 1) Use an asynchronous messaging queue (i.e. Apache ActiveMQ or
> >>> something
> >>> like Redis)
> >>>
> >>> In this way, when a reservation is made, the request is pushed onto a
> >>> centralized queue, and some intermediate (i.e. vcld) process will shift
> >>> messages off the queue and assign the reservations. If we used ActiveMQ,
> >>> the php and perl code would communicate with the message broker over the
> >>> STOMP protocol [1]. Redis would also work because it runs in a single
> >>> thread and all operations are atomic -- requests can simply be pushed
> >>> onto a FIFO-type list structure [2].
> >>>
> >>> 2. Use a NoSQL database such as CouchDB or Riak that uses a type of
> >>> optimistic locking model for all writes.
> >>>
> >>> That is, if reservations are stored in such a way that the resource id
> >>> (i.e. computer id) forms the database key, the assignment of a user to a
> >>> particular compute resource requires sending the correct revision_id of
> >>> that resource in order for the write to be successful. If successful, it
> >>> returns an HTTP 200 status code and the client proceeds normally;
> >>> otherwise, it sends a 409 header and it is up to the client to try again
> >>> [3]. It would also be possible to use the existing MySQL database to
> >>> emulate something like #2, but under a heavy load, I imagine that the
> >>> row
> >>> locking could really slow things down. Using Couch would really be much
> >>> more scalable than trying to do everything in MySQL.
> >>>
> >>> Either of these approaches would then allow you to distribute the web
> >>> front end across multiple machines.
> >>>
> >>> Cheers,
> >>> Aaron Coburn
> >>>
> >>> [1] http://activemq.apache.org/stomp.html
> >>> [2] http://redis.io/commands/lpush
> >>> [3] http://wiki.apache.org/couchdb/HTTP_Document_API#PUT
> >>>
> >>> On Jul 17, 2013, at 1:06 PM, Aaron Peeler <aa...@ncsu.edu> wrote:
> >>>> This is definitely desired.
> >>>>
> >>>> An additional challenge with multiple web servers is to figure how to
> >>>> sanely lock/unlock the resource to prevent it from getting assigned
> >>>> two separate users that are making requests at the same instant.
> >>>>
> >>>> AP
> >>>>
> >>>> On Wed, Jul 17, 2013 at 12:57 PM, James O'Dell <jo...@fullerton.edu>
> >
> > wrote:
> >>>>> I also tried load balancing the web. I didn't have any success.
> >>>>>
> >>>>> I also tried an SSL offload using the F5
> >>>>>
> >>>>> The SSLoffload didn't work because
> >>>>> an internal VCL check noticed the header wasn't https
> >>>>> and redirected to https. Basically it kept looping
> >>>>>
> >>>>> On 7/17/2013 9:36 AM, Dmitri Chebotarov wrote:
> >>>>>
> >>>>> Hi
> >>>>>
> >>>>> I would like to load balance multiple front-end VCL servers.
> >>>>> It is F5 load balancer. The LB configuration allows to enable session
> >>>>> persistency.
> >>>>> Is this will be OK with VCL's front-end? I remember someone mentioned
> >>>>> some
> >>>>> issues with having multiple front-end servers, but don't remember
> >>>>> details.
> >>>>>
> >>>>> Thanks.
> >>>>
> >>>> --
> >>>> Aaron Peeler
> >>>> Program Manager
> >>>> Virtual Computing Lab
> >>>> NC State University
> >>>>
> >>>> All electronic mail messages in connection with State business which
> >>>> are sent to or received by this account are subject to the NC Public
> >>>> Records Law and may be disclosed to third parties.
> >
> > - --
> > - -------------------------------
> > Josh Thompson
> > VCL Developer
> > North Carolina State University
> >
> > my GPG/PGP key can be found at pgp.mit.edu
> >
> > All electronic mail messages in connection with State business which
> > are sent to or received by this account are subject to the NC Public
> > Records Law and may be disclosed to third parties.
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v2.0.19 (GNU/Linux)
> >
> > iEYEARECAAYFAlHtojYACgkQV/LQcNdtPQNNjwCfQ+PRt/yZXI4tf2YDiH2IZu2m
> > coYAn2QbjICSa5MUR0cR9DIxniZVQFtK
> > =/YZs
> > -----END PGP SIGNATURE-----
- --
- -------------------------------
Josh Thompson
VCL Developer
North Carolina State University
my GPG/PGP key can be found at pgp.mit.edu
All electronic mail messages in connection with State business which
are sent to or received by this account are subject to the NC Public
Records Law and may be disclosed to third parties.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
iEYEARECAAYFAlHufrkACgkQV/LQcNdtPQPSagCdHzGSWMPRoJjTYR/nqdJScmW/
k2EAn2cGY5gKBZZgkRsG9ebscAVyOcYa
=oxwt
-----END PGP SIGNATURE-----
Re: VCL front-end and load balancer
Posted by Josh Thompson <jo...@ncsu.edu>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Aaron,
I really like where you are going with this. I talked it through with Andy,
and we came up with the following that is based on your initial ideas. One
issue that needed to be addressed is that there could be an unusual race
condition between doing the update where it gets a lock on the computer and
doing the write to the request/reservation tables such that the lock could
expire for a web server that was severely bogged down and a fast running web
server could get a lock for the same computer but the bogged down web server
would still make a reservation for the computer.
Create a table named 'semaphore' with this structure:
`computerid` smallint unsigned NOT NULL,
`imageid` smallint unsigned NOT NULL,
`imagerevisionid` mediumint unsigned NOT NULL,
`managementnodeid` smallint unsigned NOT NULL,
`expires` timestamp NOT NULL,
`procid` varchar(255) NOT NULL,
PRIMARY KEY (`computer_id`)
expires - time the semaphore for that computer will expire
procid - concatenation of the web server's IP and the process id
Now, we modify the allocComputer function to accept additional information so
it knows the imageid and imagerevisionid and to loop through the available
computers passed to it to attempt inserting an entry in the semaphore table
until it successfully does so. The insert would need to be something like
this:
INSERT INTO semaphore
SELECT c.id, $imageid, $imagerevisionid, $managementnodeid
NOW() + , '$procid'
FROM computer c LEFT JOIN semaphore s ON (c.id = s.computerid)
WHERE c.id = $computerid
AND (s.expires IS NULL OR s.expires < NOW())
LIMIT 1
Next, we modify the addRequest function to actually insert data into the
reservation table from the semaphore table so that we don't have a race
condition. The insert would look something like this:
INSERT INTO reservation
SELECT $requestid, s.computerid, s.imageid, s.imagerevisionid,
s.managementnodeid
FROM semaphore
WHERE expires > NOW() AND procid = '$procid'
The semaphore entries would either not be created for calls to isAvailable
that are only intended to check resource available to report to the user, or
they would need to be deleted at the current places where the semUnlock
function is called. Also, there would need to be a something added to delete
expired entries in the semaphore table. This could be done in a number of
places in the frontend or the backend or even using a mysql trigger function.
I need to head out for today, but wanted to go ahead and get this out. I'll
think it through more and do some testing on it in the next few days.
Josh
On Tuesday, July 23, 2013 3:01:18 PM Aaron Coburn wrote:
> The problem with shared locks on a distributed filesystem is that, if you
> have additional servers trying to share the same lock, you may actually
> achieve worse performance -- that is, there would, potentially, be an even
> higher level of contention over a single resource while also dealing with,
> potentially, inconsistent levels of latency across a network.
>
> If the desire to support multiple web servers is motivated by a need to
> scale out the infrastructure, the best way to do that would be to avoid
> locks altogether.
>
> Also, my concern with the JIRA issue related to using the XML-RPC interface,
> is that it introduces a single point of failure on the part of the web
> servers. That is, if you have ten web servers running, and all are pointing
> the XML-RPC calls to node 1, then what happens when node1 goes down?
>
> In terms of what I had suggested re: NoSQL, this can be emulated in MySQL in
> a fairly straight forward manner. I still really like the Async Messaging
> approach because it would reduce the need to continuously poll the MySQL db
> and it could be used in a lot of other ways, too, but here is one way to do
> this without adding new dependencies to the current infrastructure:
>
> First, you would need something like a 'pending_actions' table, structured
> like this:
>
> `computer_id` INT NOT NULL,
> `timestamp` timestamp NOT NULL default CURRENT_TIMESTAMP on update
> CURRENT_TIMESTAMP, `revision` varchar(255) NOT NULL,
> `data` text,
> PRIMARY KEY (`computer_id`)
>
> Then the action of "acquiring a semaphore" would happen on each computer
> (assume a lock is automatically released after 100 seconds -- that value is
> arbitrary):
>
> SELECT computer_id, revision
> FROM pending_actions
> WHERE computer_id IN(X,Y,Z) AND NOW() - timestamp > {100};
>
> In the PHP code, I would store a mapping from computer_id to 'revision'
> value, while also generating a new, random value for the new revision --
> using a new revision value will cause any other concurrent attempts to grab
> this resource to fail.
>
> ... Loop over results ...
>
> * generate new random revision value
>
> UPDATE pending_actions
> SET revision = {random, unique value}
> WHERE computer_id = {X} AND revision = {Y}
>
> * check if update succeeded via `mysql_affected_rows()`
>
> * if so: put the new revision value in the computer_id -> revision mapping
> otherwise: exclude the computer from the set of "locked" resources
>
> ... End of loop ...
>
> In the PHP code, I would know that the update (i.e. "locking") operation
> succeeded on each resource if the `mysql_affected_rows()` function returns
> a positive integer. If it returns 0, then some other process had
> successfully "locked" a particular computer before I did. In that case, the
> code can either try to use another computer or tell the user to try again.
>
> Now, for all the computers that were successfully "locked", the code has a
> period of time (i.e. 100 seconds) to do what it typically does between
> 'semLock()' and 'semUnlock()'
>
> Then, in place of semUnlock(), the code could simply set the timestamp to 0
> or sometime in the distant past:
>
> UPDATE pending_actions
> SET timestamp = 0
> WHERE computer_id = {X} AND revision = {Y}
>
> The key point is just that any UPDATE statements must use both the
> computer_id AND revision values.
>
> If I am not mistaken, this type of code could be a simple drop-in
> replacement for semLock() / semUnlock(). There would clearly need to be a
> way to populate the pending_actions table in the first place, though INSERT
> IGNORE could be used for that.
>
> -Aaron C
>
>
> On Jul 23, 2013, at 9:26 AM, Dmitri Chebotarov <dc...@gmu.edu>
>
> wrote:
> > Is it possible to use shared filesystem to store/manage semaphore locks?
> >
> > On Jul 22, 2013, at 17:20 , Josh Thompson <jo...@ncsu.edu> wrote:
> >> -----BEGIN PGP SIGNED MESSAGE-----
> >> Hash: SHA1
> >>
> >> The problem with load balancing the front end is that there is a
> >> semaphore
> >> lock (php's built in implementation) around the scheduling code. Without
> >> it, 2 users coming in at the same time can get assigned the same
> >> computer. So, having multiple frontends would allow the same thing to
> >> happen.
> >>
> >> I've considered just locking a table in the database as a semaphore.
> >> However, in my testing of this, if the web server locks a table and then
> >> goes offline before unlocking it, the table stays locked. So, done this
> >> way, if there were multiple frontends and one went offline while in the
> >> middle of the lock, none of them would be able to create reservations.
> >> I don't remember what had to be done to release the lock.
> >>
> >> The option I'd like to use that I've not gotten around to implementing (I
> >> believe there is a really old JIRA issue for it) is to add an XMLRPC API
> >> function for calling the scheduling code. Then, have one frontend
> >> assigned as the one that would handle all of the scheduling. The others
> >> would make calls to that one for just the scheduling part, but would do
> >> everything else normally. Optionally, there could be an election
> >> process so that a new frontend could be selected to do the scheduling if
> >> the one that had been doing it went down.
> >>
> >> Aaron C. listed some good ideas, but I think the above would be more
> >> straightforward since it would not involve bringing in other
> >> technologies.
> >>
> >> Josh
> >>
> >> On Thursday, July 18, 2013 9:49:39 AM Aaron Peeler wrote:
> >>> I like both of these approaches, especially 1, but I'm not sure of the
> >>> effort it would take to convert.
> >>> -AP
> >>>
> >>> On Wed, Jul 17, 2013 at 2:59 PM, Aaron Coburn <ac...@amherst.edu>
wrote:
> >>>> In my opinion, this issue has largely been solved by the NoSQL and
> >>>> application messaging communities. But as long as the VCL web server(s)
> >>>> use local, file-based semaphores to control resource allocation, this
> >>>> will be a hard nut to crack.
> >>>>
> >>>> If I were going to tackle this problem, I would take one of two general
> >>>> approaches:
> >>>>
> >>>> 1) Use an asynchronous messaging queue (i.e. Apache ActiveMQ or
> >>>> something
> >>>> like Redis)
> >>>>
> >>>> In this way, when a reservation is made, the request is pushed onto a
> >>>> centralized queue, and some intermediate (i.e. vcld) process will shift
> >>>> messages off the queue and assign the reservations. If we used
> >>>> ActiveMQ,
> >>>> the php and perl code would communicate with the message broker over
> >>>> the
> >>>> STOMP protocol [1]. Redis would also work because it runs in a single
> >>>> thread and all operations are atomic -- requests can simply be pushed
> >>>> onto a FIFO-type list structure [2].
> >>>>
> >>>> 2. Use a NoSQL database such as CouchDB or Riak that uses a type of
> >>>> optimistic locking model for all writes.
> >>>>
> >>>> That is, if reservations are stored in such a way that the resource id
> >>>> (i.e. computer id) forms the database key, the assignment of a user to
> >>>> a
> >>>> particular compute resource requires sending the correct revision_id of
> >>>> that resource in order for the write to be successful. If successful,
> >>>> it
> >>>> returns an HTTP 200 status code and the client proceeds normally;
> >>>> otherwise, it sends a 409 header and it is up to the client to try
> >>>> again
> >>>> [3]. It would also be possible to use the existing MySQL database to
> >>>> emulate something like #2, but under a heavy load, I imagine that the
> >>>> row
> >>>> locking could really slow things down. Using Couch would really be much
> >>>> more scalable than trying to do everything in MySQL.
> >>>>
> >>>> Either of these approaches would then allow you to distribute the web
> >>>> front end across multiple machines.
> >>>>
> >>>> Cheers,
> >>>> Aaron Coburn
> >>>>
> >>>> [1] http://activemq.apache.org/stomp.html
> >>>> [2] http://redis.io/commands/lpush
> >>>> [3] http://wiki.apache.org/couchdb/HTTP_Document_API#PUT
> >>>>
> >>>> On Jul 17, 2013, at 1:06 PM, Aaron Peeler <aa...@ncsu.edu>
wrote:
> >>>>> This is definitely desired.
> >>>>>
> >>>>> An additional challenge with multiple web servers is to figure how to
> >>>>> sanely lock/unlock the resource to prevent it from getting assigned
> >>>>> two separate users that are making requests at the same instant.
> >>>>>
> >>>>> AP
> >>>>>
> >>>>> On Wed, Jul 17, 2013 at 12:57 PM, James O'Dell <jo...@fullerton.edu>
> >>
> >> wrote:
> >>>>>> I also tried load balancing the web. I didn't have any success.
> >>>>>>
> >>>>>> I also tried an SSL offload using the F5
> >>>>>>
> >>>>>> The SSLoffload didn't work because
> >>>>>> an internal VCL check noticed the header wasn't https
> >>>>>> and redirected to https. Basically it kept looping
> >>>>>>
> >>>>>> On 7/17/2013 9:36 AM, Dmitri Chebotarov wrote:
> >>>>>>
> >>>>>> Hi
> >>>>>>
> >>>>>> I would like to load balance multiple front-end VCL servers.
> >>>>>> It is F5 load balancer. The LB configuration allows to enable session
> >>>>>> persistency.
> >>>>>> Is this will be OK with VCL's front-end? I remember someone mentioned
> >>>>>> some
> >>>>>> issues with having multiple front-end servers, but don't remember
> >>>>>> details.
> >>>>>>
> >>>>>> Thanks.
> >>>>>
> >>>>> --
> >>>>> Aaron Peeler
> >>>>> Program Manager
> >>>>> Virtual Computing Lab
> >>>>> NC State University
> >>>>>
> >>>>> All electronic mail messages in connection with State business which
> >>>>> are sent to or received by this account are subject to the NC Public
> >>>>> Records Law and may be disclosed to third parties.
> >>
> >> - --
> >> - -------------------------------
> >> Josh Thompson
> >> VCL Developer
> >> North Carolina State University
> >>
> >> my GPG/PGP key can be found at pgp.mit.edu
> >>
> >> All electronic mail messages in connection with State business which
> >> are sent to or received by this account are subject to the NC Public
> >> Records Law and may be disclosed to third parties.
> >> -----BEGIN PGP SIGNATURE-----
> >> Version: GnuPG v2.0.19 (GNU/Linux)
> >>
> >> iEYEARECAAYFAlHtojYACgkQV/LQcNdtPQNNjwCfQ+PRt/yZXI4tf2YDiH2IZu2m
> >> coYAn2QbjICSa5MUR0cR9DIxniZVQFtK
> >> =/YZs
> >> -----END PGP SIGNATURE-----
> >
> > --
> > Thank you,
> >
> > Dmitri Chebotarov
> > VCL Sys Eng, Engineering & Architectural Support, TSD - Ent Servers &
> > Messaging 223 Aquia Building, Ffx, MSN: 1B5
> > Phone: (703) 993-6175 | Fax: (703) 993-3404
- --
- -------------------------------
Josh Thompson
VCL Developer
North Carolina State University
my GPG/PGP key can be found at pgp.mit.edu
All electronic mail messages in connection with State business which
are sent to or received by this account are subject to the NC Public
Records Law and may be disclosed to third parties.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
iEYEARECAAYFAlHu5/YACgkQV/LQcNdtPQNuGwCcDCHRphWiibzsLQEIfMRzxde8
B0MAmweorld+LI9w24PaecxA3cT7V0K9
=wrJ/
-----END PGP SIGNATURE-----
Re: VCL front-end and load balancer
Posted by Josh Thompson <jo...@ncsu.edu>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Aaron,
I really like where you are going with this. I talked it through with Andy,
and we came up with the following that is based on your initial ideas. One
issue that needed to be addressed is that there could be an unusual race
condition between doing the update where it gets a lock on the computer and
doing the write to the request/reservation tables such that the lock could
expire for a web server that was severely bogged down and a fast running web
server could get a lock for the same computer but the bogged down web server
would still make a reservation for the computer.
Create a table named 'semaphore' with this structure:
`computerid` smallint unsigned NOT NULL,
`imageid` smallint unsigned NOT NULL,
`imagerevisionid` mediumint unsigned NOT NULL,
`managementnodeid` smallint unsigned NOT NULL,
`expires` timestamp NOT NULL,
`procid` varchar(255) NOT NULL,
PRIMARY KEY (`computer_id`)
expires - time the semaphore for that computer will expire
procid - concatenation of the web server's IP and the process id
Now, we modify the allocComputer function to accept additional information so
it knows the imageid and imagerevisionid and to loop through the available
computers passed to it to attempt inserting an entry in the semaphore table
until it successfully does so. The insert would need to be something like
this:
INSERT INTO semaphore
SELECT c.id, $imageid, $imagerevisionid, $managementnodeid
NOW() + , '$procid'
FROM computer c LEFT JOIN semaphore s ON (c.id = s.computerid)
WHERE c.id = $computerid
AND (s.expires IS NULL OR s.expires < NOW())
LIMIT 1
Next, we modify the addRequest function to actually insert data into the
reservation table from the semaphore table so that we don't have a race
condition. The insert would look something like this:
INSERT INTO reservation
SELECT $requestid, s.computerid, s.imageid, s.imagerevisionid,
s.managementnodeid
FROM semaphore
WHERE expires > NOW() AND procid = '$procid'
The semaphore entries would either not be created for calls to isAvailable
that are only intended to check resource available to report to the user, or
they would need to be deleted at the current places where the semUnlock
function is called. Also, there would need to be a something added to delete
expired entries in the semaphore table. This could be done in a number of
places in the frontend or the backend or even using a mysql trigger function.
I need to head out for today, but wanted to go ahead and get this out. I'll
think it through more and do some testing on it in the next few days.
Josh
On Tuesday, July 23, 2013 3:01:18 PM Aaron Coburn wrote:
> The problem with shared locks on a distributed filesystem is that, if you
> have additional servers trying to share the same lock, you may actually
> achieve worse performance -- that is, there would, potentially, be an even
> higher level of contention over a single resource while also dealing with,
> potentially, inconsistent levels of latency across a network.
>
> If the desire to support multiple web servers is motivated by a need to
> scale out the infrastructure, the best way to do that would be to avoid
> locks altogether.
>
> Also, my concern with the JIRA issue related to using the XML-RPC interface,
> is that it introduces a single point of failure on the part of the web
> servers. That is, if you have ten web servers running, and all are pointing
> the XML-RPC calls to node 1, then what happens when node1 goes down?
>
> In terms of what I had suggested re: NoSQL, this can be emulated in MySQL in
> a fairly straight forward manner. I still really like the Async Messaging
> approach because it would reduce the need to continuously poll the MySQL db
> and it could be used in a lot of other ways, too, but here is one way to do
> this without adding new dependencies to the current infrastructure:
>
> First, you would need something like a 'pending_actions' table, structured
> like this:
>
> `computer_id` INT NOT NULL,
> `timestamp` timestamp NOT NULL default CURRENT_TIMESTAMP on update
> CURRENT_TIMESTAMP, `revision` varchar(255) NOT NULL,
> `data` text,
> PRIMARY KEY (`computer_id`)
>
> Then the action of "acquiring a semaphore" would happen on each computer
> (assume a lock is automatically released after 100 seconds -- that value is
> arbitrary):
>
> SELECT computer_id, revision
> FROM pending_actions
> WHERE computer_id IN(X,Y,Z) AND NOW() - timestamp > {100};
>
> In the PHP code, I would store a mapping from computer_id to 'revision'
> value, while also generating a new, random value for the new revision --
> using a new revision value will cause any other concurrent attempts to grab
> this resource to fail.
>
> ... Loop over results ...
>
> * generate new random revision value
>
> UPDATE pending_actions
> SET revision = {random, unique value}
> WHERE computer_id = {X} AND revision = {Y}
>
> * check if update succeeded via `mysql_affected_rows()`
>
> * if so: put the new revision value in the computer_id -> revision mapping
> otherwise: exclude the computer from the set of "locked" resources
>
> ... End of loop ...
>
> In the PHP code, I would know that the update (i.e. "locking") operation
> succeeded on each resource if the `mysql_affected_rows()` function returns
> a positive integer. If it returns 0, then some other process had
> successfully "locked" a particular computer before I did. In that case, the
> code can either try to use another computer or tell the user to try again.
>
> Now, for all the computers that were successfully "locked", the code has a
> period of time (i.e. 100 seconds) to do what it typically does between
> 'semLock()' and 'semUnlock()'
>
> Then, in place of semUnlock(), the code could simply set the timestamp to 0
> or sometime in the distant past:
>
> UPDATE pending_actions
> SET timestamp = 0
> WHERE computer_id = {X} AND revision = {Y}
>
> The key point is just that any UPDATE statements must use both the
> computer_id AND revision values.
>
> If I am not mistaken, this type of code could be a simple drop-in
> replacement for semLock() / semUnlock(). There would clearly need to be a
> way to populate the pending_actions table in the first place, though INSERT
> IGNORE could be used for that.
>
> -Aaron C
>
>
> On Jul 23, 2013, at 9:26 AM, Dmitri Chebotarov <dc...@gmu.edu>
>
> wrote:
> > Is it possible to use shared filesystem to store/manage semaphore locks?
> >
> > On Jul 22, 2013, at 17:20 , Josh Thompson <jo...@ncsu.edu> wrote:
> >> -----BEGIN PGP SIGNED MESSAGE-----
> >> Hash: SHA1
> >>
> >> The problem with load balancing the front end is that there is a
> >> semaphore
> >> lock (php's built in implementation) around the scheduling code. Without
> >> it, 2 users coming in at the same time can get assigned the same
> >> computer. So, having multiple frontends would allow the same thing to
> >> happen.
> >>
> >> I've considered just locking a table in the database as a semaphore.
> >> However, in my testing of this, if the web server locks a table and then
> >> goes offline before unlocking it, the table stays locked. So, done this
> >> way, if there were multiple frontends and one went offline while in the
> >> middle of the lock, none of them would be able to create reservations.
> >> I don't remember what had to be done to release the lock.
> >>
> >> The option I'd like to use that I've not gotten around to implementing (I
> >> believe there is a really old JIRA issue for it) is to add an XMLRPC API
> >> function for calling the scheduling code. Then, have one frontend
> >> assigned as the one that would handle all of the scheduling. The others
> >> would make calls to that one for just the scheduling part, but would do
> >> everything else normally. Optionally, there could be an election
> >> process so that a new frontend could be selected to do the scheduling if
> >> the one that had been doing it went down.
> >>
> >> Aaron C. listed some good ideas, but I think the above would be more
> >> straightforward since it would not involve bringing in other
> >> technologies.
> >>
> >> Josh
> >>
> >> On Thursday, July 18, 2013 9:49:39 AM Aaron Peeler wrote:
> >>> I like both of these approaches, especially 1, but I'm not sure of the
> >>> effort it would take to convert.
> >>> -AP
> >>>
> >>> On Wed, Jul 17, 2013 at 2:59 PM, Aaron Coburn <ac...@amherst.edu>
wrote:
> >>>> In my opinion, this issue has largely been solved by the NoSQL and
> >>>> application messaging communities. But as long as the VCL web server(s)
> >>>> use local, file-based semaphores to control resource allocation, this
> >>>> will be a hard nut to crack.
> >>>>
> >>>> If I were going to tackle this problem, I would take one of two general
> >>>> approaches:
> >>>>
> >>>> 1) Use an asynchronous messaging queue (i.e. Apache ActiveMQ or
> >>>> something
> >>>> like Redis)
> >>>>
> >>>> In this way, when a reservation is made, the request is pushed onto a
> >>>> centralized queue, and some intermediate (i.e. vcld) process will shift
> >>>> messages off the queue and assign the reservations. If we used
> >>>> ActiveMQ,
> >>>> the php and perl code would communicate with the message broker over
> >>>> the
> >>>> STOMP protocol [1]. Redis would also work because it runs in a single
> >>>> thread and all operations are atomic -- requests can simply be pushed
> >>>> onto a FIFO-type list structure [2].
> >>>>
> >>>> 2. Use a NoSQL database such as CouchDB or Riak that uses a type of
> >>>> optimistic locking model for all writes.
> >>>>
> >>>> That is, if reservations are stored in such a way that the resource id
> >>>> (i.e. computer id) forms the database key, the assignment of a user to
> >>>> a
> >>>> particular compute resource requires sending the correct revision_id of
> >>>> that resource in order for the write to be successful. If successful,
> >>>> it
> >>>> returns an HTTP 200 status code and the client proceeds normally;
> >>>> otherwise, it sends a 409 header and it is up to the client to try
> >>>> again
> >>>> [3]. It would also be possible to use the existing MySQL database to
> >>>> emulate something like #2, but under a heavy load, I imagine that the
> >>>> row
> >>>> locking could really slow things down. Using Couch would really be much
> >>>> more scalable than trying to do everything in MySQL.
> >>>>
> >>>> Either of these approaches would then allow you to distribute the web
> >>>> front end across multiple machines.
> >>>>
> >>>> Cheers,
> >>>> Aaron Coburn
> >>>>
> >>>> [1] http://activemq.apache.org/stomp.html
> >>>> [2] http://redis.io/commands/lpush
> >>>> [3] http://wiki.apache.org/couchdb/HTTP_Document_API#PUT
> >>>>
> >>>> On Jul 17, 2013, at 1:06 PM, Aaron Peeler <aa...@ncsu.edu>
wrote:
> >>>>> This is definitely desired.
> >>>>>
> >>>>> An additional challenge with multiple web servers is to figure how to
> >>>>> sanely lock/unlock the resource to prevent it from getting assigned
> >>>>> two separate users that are making requests at the same instant.
> >>>>>
> >>>>> AP
> >>>>>
> >>>>> On Wed, Jul 17, 2013 at 12:57 PM, James O'Dell <jo...@fullerton.edu>
> >>
> >> wrote:
> >>>>>> I also tried load balancing the web. I didn't have any success.
> >>>>>>
> >>>>>> I also tried an SSL offload using the F5
> >>>>>>
> >>>>>> The SSLoffload didn't work because
> >>>>>> an internal VCL check noticed the header wasn't https
> >>>>>> and redirected to https. Basically it kept looping
> >>>>>>
> >>>>>> On 7/17/2013 9:36 AM, Dmitri Chebotarov wrote:
> >>>>>>
> >>>>>> Hi
> >>>>>>
> >>>>>> I would like to load balance multiple front-end VCL servers.
> >>>>>> It is F5 load balancer. The LB configuration allows to enable session
> >>>>>> persistency.
> >>>>>> Is this will be OK with VCL's front-end? I remember someone mentioned
> >>>>>> some
> >>>>>> issues with having multiple front-end servers, but don't remember
> >>>>>> details.
> >>>>>>
> >>>>>> Thanks.
> >>>>>
> >>>>> --
> >>>>> Aaron Peeler
> >>>>> Program Manager
> >>>>> Virtual Computing Lab
> >>>>> NC State University
> >>>>>
> >>>>> All electronic mail messages in connection with State business which
> >>>>> are sent to or received by this account are subject to the NC Public
> >>>>> Records Law and may be disclosed to third parties.
> >>
> >> - --
> >> - -------------------------------
> >> Josh Thompson
> >> VCL Developer
> >> North Carolina State University
> >>
> >> my GPG/PGP key can be found at pgp.mit.edu
> >>
> >> All electronic mail messages in connection with State business which
> >> are sent to or received by this account are subject to the NC Public
> >> Records Law and may be disclosed to third parties.
> >> -----BEGIN PGP SIGNATURE-----
> >> Version: GnuPG v2.0.19 (GNU/Linux)
> >>
> >> iEYEARECAAYFAlHtojYACgkQV/LQcNdtPQNNjwCfQ+PRt/yZXI4tf2YDiH2IZu2m
> >> coYAn2QbjICSa5MUR0cR9DIxniZVQFtK
> >> =/YZs
> >> -----END PGP SIGNATURE-----
> >
> > --
> > Thank you,
> >
> > Dmitri Chebotarov
> > VCL Sys Eng, Engineering & Architectural Support, TSD - Ent Servers &
> > Messaging 223 Aquia Building, Ffx, MSN: 1B5
> > Phone: (703) 993-6175 | Fax: (703) 993-3404
- --
- -------------------------------
Josh Thompson
VCL Developer
North Carolina State University
my GPG/PGP key can be found at pgp.mit.edu
All electronic mail messages in connection with State business which
are sent to or received by this account are subject to the NC Public
Records Law and may be disclosed to third parties.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
iEYEARECAAYFAlHu5/YACgkQV/LQcNdtPQNuGwCcDCHRphWiibzsLQEIfMRzxde8
B0MAmweorld+LI9w24PaecxA3cT7V0K9
=wrJ/
-----END PGP SIGNATURE-----
Re: VCL front-end and load balancer
Posted by Aaron Coburn <ac...@amherst.edu>.
The problem with shared locks on a distributed filesystem is that, if you have additional servers trying to share the same lock, you may actually achieve worse performance -- that is, there would, potentially, be an even higher level of contention over a single resource while also dealing with, potentially, inconsistent levels of latency across a network.
If the desire to support multiple web servers is motivated by a need to scale out the infrastructure, the best way to do that would be to avoid locks altogether.
Also, my concern with the JIRA issue related to using the XML-RPC interface, is that it introduces a single point of failure on the part of the web servers. That is, if you have ten web servers running, and all are pointing the XML-RPC calls to node 1, then what happens when node1 goes down?
In terms of what I had suggested re: NoSQL, this can be emulated in MySQL in a fairly straight forward manner. I still really like the Async Messaging approach because it would reduce the need to continuously poll the MySQL db and it could be used in a lot of other ways, too, but here is one way to do this without adding new dependencies to the current infrastructure:
First, you would need something like a 'pending_actions' table, structured like this:
`computer_id` INT NOT NULL,
`timestamp` timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP,
`revision` varchar(255) NOT NULL,
`data` text,
PRIMARY KEY (`computer_id`)
Then the action of "acquiring a semaphore" would happen on each computer (assume a lock is automatically released after 100 seconds -- that value is arbitrary):
SELECT computer_id, revision
FROM pending_actions
WHERE computer_id IN(X,Y,Z) AND NOW() - timestamp > {100};
In the PHP code, I would store a mapping from computer_id to 'revision' value, while also generating a new, random value for the new revision -- using a new revision value will cause any other concurrent attempts to grab this resource to fail.
... Loop over results ...
* generate new random revision value
UPDATE pending_actions
SET revision = {random, unique value}
WHERE computer_id = {X} AND revision = {Y}
* check if update succeeded via `mysql_affected_rows()`
* if so: put the new revision value in the computer_id -> revision mapping
otherwise: exclude the computer from the set of "locked" resources
... End of loop ...
In the PHP code, I would know that the update (i.e. "locking") operation succeeded on each resource if the `mysql_affected_rows()` function returns a positive integer. If it returns 0, then some other process had successfully "locked" a particular computer before I did. In that case, the code can either try to use another computer or tell the user to try again.
Now, for all the computers that were successfully "locked", the code has a period of time (i.e. 100 seconds) to do what it typically does between 'semLock()' and 'semUnlock()'
Then, in place of semUnlock(), the code could simply set the timestamp to 0 or sometime in the distant past:
UPDATE pending_actions
SET timestamp = 0
WHERE computer_id = {X} AND revision = {Y}
The key point is just that any UPDATE statements must use both the computer_id AND revision values.
If I am not mistaken, this type of code could be a simple drop-in replacement for semLock() / semUnlock(). There would clearly need to be a way to populate the pending_actions table in the first place, though INSERT IGNORE could be used for that.
-Aaron C
On Jul 23, 2013, at 9:26 AM, Dmitri Chebotarov <dc...@gmu.edu>
wrote:
>
> Is it possible to use shared filesystem to store/manage semaphore locks?
>
> On Jul 22, 2013, at 17:20 , Josh Thompson <jo...@ncsu.edu> wrote:
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> The problem with load balancing the front end is that there is a semaphore
>> lock (php's built in implementation) around the scheduling code. Without it,
>> 2 users coming in at the same time can get assigned the same computer. So,
>> having multiple frontends would allow the same thing to happen.
>>
>> I've considered just locking a table in the database as a semaphore. However,
>> in my testing of this, if the web server locks a table and then goes offline
>> before unlocking it, the table stays locked. So, done this way, if there were
>> multiple frontends and one went offline while in the middle of the lock, none
>> of them would be able to create reservations. I don't remember what had to be
>> done to release the lock.
>>
>> The option I'd like to use that I've not gotten around to implementing (I
>> believe there is a really old JIRA issue for it) is to add an XMLRPC API
>> function for calling the scheduling code. Then, have one frontend assigned as
>> the one that would handle all of the scheduling. The others would make calls
>> to that one for just the scheduling part, but would do everything else
>> normally. Optionally, there could be an election process so that a new
>> frontend could be selected to do the scheduling if the one that had been doing
>> it went down.
>>
>> Aaron C. listed some good ideas, but I think the above would be more
>> straightforward since it would not involve bringing in other technologies.
>>
>> Josh
>>
>> On Thursday, July 18, 2013 9:49:39 AM Aaron Peeler wrote:
>>> I like both of these approaches, especially 1, but I'm not sure of the
>>> effort it would take to convert.
>>> -AP
>>>
>>> On Wed, Jul 17, 2013 at 2:59 PM, Aaron Coburn <ac...@amherst.edu> wrote:
>>>> In my opinion, this issue has largely been solved by the NoSQL and
>>>> application messaging communities. But as long as the VCL web server(s)
>>>> use local, file-based semaphores to control resource allocation, this
>>>> will be a hard nut to crack.
>>>>
>>>> If I were going to tackle this problem, I would take one of two general
>>>> approaches:
>>>>
>>>> 1) Use an asynchronous messaging queue (i.e. Apache ActiveMQ or something
>>>> like Redis)
>>>>
>>>> In this way, when a reservation is made, the request is pushed onto a
>>>> centralized queue, and some intermediate (i.e. vcld) process will shift
>>>> messages off the queue and assign the reservations. If we used ActiveMQ,
>>>> the php and perl code would communicate with the message broker over the
>>>> STOMP protocol [1]. Redis would also work because it runs in a single
>>>> thread and all operations are atomic -- requests can simply be pushed
>>>> onto a FIFO-type list structure [2].
>>>>
>>>> 2. Use a NoSQL database such as CouchDB or Riak that uses a type of
>>>> optimistic locking model for all writes.
>>>>
>>>> That is, if reservations are stored in such a way that the resource id
>>>> (i.e. computer id) forms the database key, the assignment of a user to a
>>>> particular compute resource requires sending the correct revision_id of
>>>> that resource in order for the write to be successful. If successful, it
>>>> returns an HTTP 200 status code and the client proceeds normally;
>>>> otherwise, it sends a 409 header and it is up to the client to try again
>>>> [3]. It would also be possible to use the existing MySQL database to
>>>> emulate something like #2, but under a heavy load, I imagine that the row
>>>> locking could really slow things down. Using Couch would really be much
>>>> more scalable than trying to do everything in MySQL.
>>>>
>>>> Either of these approaches would then allow you to distribute the web
>>>> front end across multiple machines.
>>>>
>>>> Cheers,
>>>> Aaron Coburn
>>>>
>>>> [1] http://activemq.apache.org/stomp.html
>>>> [2] http://redis.io/commands/lpush
>>>> [3] http://wiki.apache.org/couchdb/HTTP_Document_API#PUT
>>>>
>>>> On Jul 17, 2013, at 1:06 PM, Aaron Peeler <aa...@ncsu.edu> wrote:
>>>>> This is definitely desired.
>>>>>
>>>>> An additional challenge with multiple web servers is to figure how to
>>>>> sanely lock/unlock the resource to prevent it from getting assigned
>>>>> two separate users that are making requests at the same instant.
>>>>>
>>>>> AP
>>>>>
>>>>> On Wed, Jul 17, 2013 at 12:57 PM, James O'Dell <jo...@fullerton.edu>
>> wrote:
>>>>>> I also tried load balancing the web. I didn't have any success.
>>>>>>
>>>>>> I also tried an SSL offload using the F5
>>>>>>
>>>>>> The SSLoffload didn't work because
>>>>>> an internal VCL check noticed the header wasn't https
>>>>>> and redirected to https. Basically it kept looping
>>>>>>
>>>>>> On 7/17/2013 9:36 AM, Dmitri Chebotarov wrote:
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>> I would like to load balance multiple front-end VCL servers.
>>>>>> It is F5 load balancer. The LB configuration allows to enable session
>>>>>> persistency.
>>>>>> Is this will be OK with VCL's front-end? I remember someone mentioned
>>>>>> some
>>>>>> issues with having multiple front-end servers, but don't remember
>>>>>> details.
>>>>>>
>>>>>> Thanks.
>>>>>
>>>>> --
>>>>> Aaron Peeler
>>>>> Program Manager
>>>>> Virtual Computing Lab
>>>>> NC State University
>>>>>
>>>>> All electronic mail messages in connection with State business which
>>>>> are sent to or received by this account are subject to the NC Public
>>>>> Records Law and may be disclosed to third parties.
>> - --
>> - -------------------------------
>> Josh Thompson
>> VCL Developer
>> North Carolina State University
>>
>> my GPG/PGP key can be found at pgp.mit.edu
>>
>> All electronic mail messages in connection with State business which
>> are sent to or received by this account are subject to the NC Public
>> Records Law and may be disclosed to third parties.
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v2.0.19 (GNU/Linux)
>>
>> iEYEARECAAYFAlHtojYACgkQV/LQcNdtPQNNjwCfQ+PRt/yZXI4tf2YDiH2IZu2m
>> coYAn2QbjICSa5MUR0cR9DIxniZVQFtK
>> =/YZs
>> -----END PGP SIGNATURE-----
>>
>>
>
>
>
> --
> Thank you,
>
> Dmitri Chebotarov
> VCL Sys Eng, Engineering & Architectural Support, TSD - Ent Servers & Messaging
> 223 Aquia Building, Ffx, MSN: 1B5
> Phone: (703) 993-6175 | Fax: (703) 993-3404
>
>
>
Re: VCL front-end and load balancer
Posted by Aaron Coburn <ac...@amherst.edu>.
The problem with shared locks on a distributed filesystem is that, if you have additional servers trying to share the same lock, you may actually achieve worse performance -- that is, there would, potentially, be an even higher level of contention over a single resource while also dealing with, potentially, inconsistent levels of latency across a network.
If the desire to support multiple web servers is motivated by a need to scale out the infrastructure, the best way to do that would be to avoid locks altogether.
Also, my concern with the JIRA issue related to using the XML-RPC interface, is that it introduces a single point of failure on the part of the web servers. That is, if you have ten web servers running, and all are pointing the XML-RPC calls to node 1, then what happens when node1 goes down?
In terms of what I had suggested re: NoSQL, this can be emulated in MySQL in a fairly straight forward manner. I still really like the Async Messaging approach because it would reduce the need to continuously poll the MySQL db and it could be used in a lot of other ways, too, but here is one way to do this without adding new dependencies to the current infrastructure:
First, you would need something like a 'pending_actions' table, structured like this:
`computer_id` INT NOT NULL,
`timestamp` timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP,
`revision` varchar(255) NOT NULL,
`data` text,
PRIMARY KEY (`computer_id`)
Then the action of "acquiring a semaphore" would happen on each computer (assume a lock is automatically released after 100 seconds -- that value is arbitrary):
SELECT computer_id, revision
FROM pending_actions
WHERE computer_id IN(X,Y,Z) AND NOW() - timestamp > {100};
In the PHP code, I would store a mapping from computer_id to 'revision' value, while also generating a new, random value for the new revision -- using a new revision value will cause any other concurrent attempts to grab this resource to fail.
... Loop over results ...
* generate new random revision value
UPDATE pending_actions
SET revision = {random, unique value}
WHERE computer_id = {X} AND revision = {Y}
* check if update succeeded via `mysql_affected_rows()`
* if so: put the new revision value in the computer_id -> revision mapping
otherwise: exclude the computer from the set of "locked" resources
... End of loop ...
In the PHP code, I would know that the update (i.e. "locking") operation succeeded on each resource if the `mysql_affected_rows()` function returns a positive integer. If it returns 0, then some other process had successfully "locked" a particular computer before I did. In that case, the code can either try to use another computer or tell the user to try again.
Now, for all the computers that were successfully "locked", the code has a period of time (i.e. 100 seconds) to do what it typically does between 'semLock()' and 'semUnlock()'
Then, in place of semUnlock(), the code could simply set the timestamp to 0 or sometime in the distant past:
UPDATE pending_actions
SET timestamp = 0
WHERE computer_id = {X} AND revision = {Y}
The key point is just that any UPDATE statements must use both the computer_id AND revision values.
If I am not mistaken, this type of code could be a simple drop-in replacement for semLock() / semUnlock(). There would clearly need to be a way to populate the pending_actions table in the first place, though INSERT IGNORE could be used for that.
-Aaron C
On Jul 23, 2013, at 9:26 AM, Dmitri Chebotarov <dc...@gmu.edu>
wrote:
>
> Is it possible to use shared filesystem to store/manage semaphore locks?
>
> On Jul 22, 2013, at 17:20 , Josh Thompson <jo...@ncsu.edu> wrote:
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> The problem with load balancing the front end is that there is a semaphore
>> lock (php's built in implementation) around the scheduling code. Without it,
>> 2 users coming in at the same time can get assigned the same computer. So,
>> having multiple frontends would allow the same thing to happen.
>>
>> I've considered just locking a table in the database as a semaphore. However,
>> in my testing of this, if the web server locks a table and then goes offline
>> before unlocking it, the table stays locked. So, done this way, if there were
>> multiple frontends and one went offline while in the middle of the lock, none
>> of them would be able to create reservations. I don't remember what had to be
>> done to release the lock.
>>
>> The option I'd like to use that I've not gotten around to implementing (I
>> believe there is a really old JIRA issue for it) is to add an XMLRPC API
>> function for calling the scheduling code. Then, have one frontend assigned as
>> the one that would handle all of the scheduling. The others would make calls
>> to that one for just the scheduling part, but would do everything else
>> normally. Optionally, there could be an election process so that a new
>> frontend could be selected to do the scheduling if the one that had been doing
>> it went down.
>>
>> Aaron C. listed some good ideas, but I think the above would be more
>> straightforward since it would not involve bringing in other technologies.
>>
>> Josh
>>
>> On Thursday, July 18, 2013 9:49:39 AM Aaron Peeler wrote:
>>> I like both of these approaches, especially 1, but I'm not sure of the
>>> effort it would take to convert.
>>> -AP
>>>
>>> On Wed, Jul 17, 2013 at 2:59 PM, Aaron Coburn <ac...@amherst.edu> wrote:
>>>> In my opinion, this issue has largely been solved by the NoSQL and
>>>> application messaging communities. But as long as the VCL web server(s)
>>>> use local, file-based semaphores to control resource allocation, this
>>>> will be a hard nut to crack.
>>>>
>>>> If I were going to tackle this problem, I would take one of two general
>>>> approaches:
>>>>
>>>> 1) Use an asynchronous messaging queue (i.e. Apache ActiveMQ or something
>>>> like Redis)
>>>>
>>>> In this way, when a reservation is made, the request is pushed onto a
>>>> centralized queue, and some intermediate (i.e. vcld) process will shift
>>>> messages off the queue and assign the reservations. If we used ActiveMQ,
>>>> the php and perl code would communicate with the message broker over the
>>>> STOMP protocol [1]. Redis would also work because it runs in a single
>>>> thread and all operations are atomic -- requests can simply be pushed
>>>> onto a FIFO-type list structure [2].
>>>>
>>>> 2. Use a NoSQL database such as CouchDB or Riak that uses a type of
>>>> optimistic locking model for all writes.
>>>>
>>>> That is, if reservations are stored in such a way that the resource id
>>>> (i.e. computer id) forms the database key, the assignment of a user to a
>>>> particular compute resource requires sending the correct revision_id of
>>>> that resource in order for the write to be successful. If successful, it
>>>> returns an HTTP 200 status code and the client proceeds normally;
>>>> otherwise, it sends a 409 header and it is up to the client to try again
>>>> [3]. It would also be possible to use the existing MySQL database to
>>>> emulate something like #2, but under a heavy load, I imagine that the row
>>>> locking could really slow things down. Using Couch would really be much
>>>> more scalable than trying to do everything in MySQL.
>>>>
>>>> Either of these approaches would then allow you to distribute the web
>>>> front end across multiple machines.
>>>>
>>>> Cheers,
>>>> Aaron Coburn
>>>>
>>>> [1] http://activemq.apache.org/stomp.html
>>>> [2] http://redis.io/commands/lpush
>>>> [3] http://wiki.apache.org/couchdb/HTTP_Document_API#PUT
>>>>
>>>> On Jul 17, 2013, at 1:06 PM, Aaron Peeler <aa...@ncsu.edu> wrote:
>>>>> This is definitely desired.
>>>>>
>>>>> An additional challenge with multiple web servers is to figure how to
>>>>> sanely lock/unlock the resource to prevent it from getting assigned
>>>>> two separate users that are making requests at the same instant.
>>>>>
>>>>> AP
>>>>>
>>>>> On Wed, Jul 17, 2013 at 12:57 PM, James O'Dell <jo...@fullerton.edu>
>> wrote:
>>>>>> I also tried load balancing the web. I didn't have any success.
>>>>>>
>>>>>> I also tried an SSL offload using the F5
>>>>>>
>>>>>> The SSLoffload didn't work because
>>>>>> an internal VCL check noticed the header wasn't https
>>>>>> and redirected to https. Basically it kept looping
>>>>>>
>>>>>> On 7/17/2013 9:36 AM, Dmitri Chebotarov wrote:
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>> I would like to load balance multiple front-end VCL servers.
>>>>>> It is F5 load balancer. The LB configuration allows to enable session
>>>>>> persistency.
>>>>>> Is this will be OK with VCL's front-end? I remember someone mentioned
>>>>>> some
>>>>>> issues with having multiple front-end servers, but don't remember
>>>>>> details.
>>>>>>
>>>>>> Thanks.
>>>>>
>>>>> --
>>>>> Aaron Peeler
>>>>> Program Manager
>>>>> Virtual Computing Lab
>>>>> NC State University
>>>>>
>>>>> All electronic mail messages in connection with State business which
>>>>> are sent to or received by this account are subject to the NC Public
>>>>> Records Law and may be disclosed to third parties.
>> - --
>> - -------------------------------
>> Josh Thompson
>> VCL Developer
>> North Carolina State University
>>
>> my GPG/PGP key can be found at pgp.mit.edu
>>
>> All electronic mail messages in connection with State business which
>> are sent to or received by this account are subject to the NC Public
>> Records Law and may be disclosed to third parties.
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v2.0.19 (GNU/Linux)
>>
>> iEYEARECAAYFAlHtojYACgkQV/LQcNdtPQNNjwCfQ+PRt/yZXI4tf2YDiH2IZu2m
>> coYAn2QbjICSa5MUR0cR9DIxniZVQFtK
>> =/YZs
>> -----END PGP SIGNATURE-----
>>
>>
>
>
>
> --
> Thank you,
>
> Dmitri Chebotarov
> VCL Sys Eng, Engineering & Architectural Support, TSD - Ent Servers & Messaging
> 223 Aquia Building, Ffx, MSN: 1B5
> Phone: (703) 993-6175 | Fax: (703) 993-3404
>
>
>
Re: VCL front-end and load balancer
Posted by Dmitri Chebotarov <dc...@gmu.edu>.
Is it possible to use shared filesystem to store/manage semaphore locks?
On Jul 22, 2013, at 17:20 , Josh Thompson <jo...@ncsu.edu> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> The problem with load balancing the front end is that there is a semaphore
> lock (php's built in implementation) around the scheduling code. Without it,
> 2 users coming in at the same time can get assigned the same computer. So,
> having multiple frontends would allow the same thing to happen.
>
> I've considered just locking a table in the database as a semaphore. However,
> in my testing of this, if the web server locks a table and then goes offline
> before unlocking it, the table stays locked. So, done this way, if there were
> multiple frontends and one went offline while in the middle of the lock, none
> of them would be able to create reservations. I don't remember what had to be
> done to release the lock.
>
> The option I'd like to use that I've not gotten around to implementing (I
> believe there is a really old JIRA issue for it) is to add an XMLRPC API
> function for calling the scheduling code. Then, have one frontend assigned as
> the one that would handle all of the scheduling. The others would make calls
> to that one for just the scheduling part, but would do everything else
> normally. Optionally, there could be an election process so that a new
> frontend could be selected to do the scheduling if the one that had been doing
> it went down.
>
> Aaron C. listed some good ideas, but I think the above would be more
> straightforward since it would not involve bringing in other technologies.
>
> Josh
>
> On Thursday, July 18, 2013 9:49:39 AM Aaron Peeler wrote:
>> I like both of these approaches, especially 1, but I'm not sure of the
>> effort it would take to convert.
>> -AP
>>
>> On Wed, Jul 17, 2013 at 2:59 PM, Aaron Coburn <ac...@amherst.edu> wrote:
>>> In my opinion, this issue has largely been solved by the NoSQL and
>>> application messaging communities. But as long as the VCL web server(s)
>>> use local, file-based semaphores to control resource allocation, this
>>> will be a hard nut to crack.
>>>
>>> If I were going to tackle this problem, I would take one of two general
>>> approaches:
>>>
>>> 1) Use an asynchronous messaging queue (i.e. Apache ActiveMQ or something
>>> like Redis)
>>>
>>> In this way, when a reservation is made, the request is pushed onto a
>>> centralized queue, and some intermediate (i.e. vcld) process will shift
>>> messages off the queue and assign the reservations. If we used ActiveMQ,
>>> the php and perl code would communicate with the message broker over the
>>> STOMP protocol [1]. Redis would also work because it runs in a single
>>> thread and all operations are atomic -- requests can simply be pushed
>>> onto a FIFO-type list structure [2].
>>>
>>> 2. Use a NoSQL database such as CouchDB or Riak that uses a type of
>>> optimistic locking model for all writes.
>>>
>>> That is, if reservations are stored in such a way that the resource id
>>> (i.e. computer id) forms the database key, the assignment of a user to a
>>> particular compute resource requires sending the correct revision_id of
>>> that resource in order for the write to be successful. If successful, it
>>> returns an HTTP 200 status code and the client proceeds normally;
>>> otherwise, it sends a 409 header and it is up to the client to try again
>>> [3]. It would also be possible to use the existing MySQL database to
>>> emulate something like #2, but under a heavy load, I imagine that the row
>>> locking could really slow things down. Using Couch would really be much
>>> more scalable than trying to do everything in MySQL.
>>>
>>> Either of these approaches would then allow you to distribute the web
>>> front end across multiple machines.
>>>
>>> Cheers,
>>> Aaron Coburn
>>>
>>> [1] http://activemq.apache.org/stomp.html
>>> [2] http://redis.io/commands/lpush
>>> [3] http://wiki.apache.org/couchdb/HTTP_Document_API#PUT
>>>
>>> On Jul 17, 2013, at 1:06 PM, Aaron Peeler <aa...@ncsu.edu> wrote:
>>>> This is definitely desired.
>>>>
>>>> An additional challenge with multiple web servers is to figure how to
>>>> sanely lock/unlock the resource to prevent it from getting assigned
>>>> two separate users that are making requests at the same instant.
>>>>
>>>> AP
>>>>
>>>> On Wed, Jul 17, 2013 at 12:57 PM, James O'Dell <jo...@fullerton.edu>
> wrote:
>>>>> I also tried load balancing the web. I didn't have any success.
>>>>>
>>>>> I also tried an SSL offload using the F5
>>>>>
>>>>> The SSLoffload didn't work because
>>>>> an internal VCL check noticed the header wasn't https
>>>>> and redirected to https. Basically it kept looping
>>>>>
>>>>> On 7/17/2013 9:36 AM, Dmitri Chebotarov wrote:
>>>>>
>>>>> Hi
>>>>>
>>>>> I would like to load balance multiple front-end VCL servers.
>>>>> It is F5 load balancer. The LB configuration allows to enable session
>>>>> persistency.
>>>>> Is this will be OK with VCL's front-end? I remember someone mentioned
>>>>> some
>>>>> issues with having multiple front-end servers, but don't remember
>>>>> details.
>>>>>
>>>>> Thanks.
>>>>
>>>> --
>>>> Aaron Peeler
>>>> Program Manager
>>>> Virtual Computing Lab
>>>> NC State University
>>>>
>>>> All electronic mail messages in connection with State business which
>>>> are sent to or received by this account are subject to the NC Public
>>>> Records Law and may be disclosed to third parties.
> - --
> - -------------------------------
> Josh Thompson
> VCL Developer
> North Carolina State University
>
> my GPG/PGP key can be found at pgp.mit.edu
>
> All electronic mail messages in connection with State business which
> are sent to or received by this account are subject to the NC Public
> Records Law and may be disclosed to third parties.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.19 (GNU/Linux)
>
> iEYEARECAAYFAlHtojYACgkQV/LQcNdtPQNNjwCfQ+PRt/yZXI4tf2YDiH2IZu2m
> coYAn2QbjICSa5MUR0cR9DIxniZVQFtK
> =/YZs
> -----END PGP SIGNATURE-----
>
>
--
Thank you,
Dmitri Chebotarov
VCL Sys Eng, Engineering & Architectural Support, TSD - Ent Servers & Messaging
223 Aquia Building, Ffx, MSN: 1B5
Phone: (703) 993-6175 | Fax: (703) 993-3404
Re: VCL front-end and load balancer
Posted by James O'Dell <jo...@fullerton.edu>.
Hey Josh,
I've been F5 load balancing, and SSL offloading, Moodle for the last
couple of years with out a problem.
Moodle just insists the database engine be ACID compliant
If the database engine is transaction safe (i.e. InnoDB) you shouldn't
have a problem.
It's really just a matter of handling the results (i.e. if someone else
grabs the reservation
what do you do)
Imho relying on the database seems like the simplest/cleanest approach
__Jim
On 7/22/2013 2:20 PM, Josh Thompson wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> The problem with load balancing the front end is that there is a semaphore
> lock (php's built in implementation) around the scheduling code. Without it,
> 2 users coming in at the same time can get assigned the same computer. So,
> having multiple frontends would allow the same thing to happen.
>
> I've considered just locking a table in the database as a semaphore. However,
> in my testing of this, if the web server locks a table and then goes offline
> before unlocking it, the table stays locked. So, done this way, if there were
> multiple frontends and one went offline while in the middle of the lock, none
> of them would be able to create reservations. I don't remember what had to be
> done to release the lock.
>
> The option I'd like to use that I've not gotten around to implementing (I
> believe there is a really old JIRA issue for it) is to add an XMLRPC API
> function for calling the scheduling code. Then, have one frontend assigned as
> the one that would handle all of the scheduling. The others would make calls
> to that one for just the scheduling part, but would do everything else
> normally. Optionally, there could be an election process so that a new
> frontend could be selected to do the scheduling if the one that had been doing
> it went down.
>
> Aaron C. listed some good ideas, but I think the above would be more
> straightforward since it would not involve bringing in other technologies.
>
> Josh
>
> On Thursday, July 18, 2013 9:49:39 AM Aaron Peeler wrote:
>> I like both of these approaches, especially 1, but I'm not sure of the
>> effort it would take to convert.
>> -AP
>>
>> On Wed, Jul 17, 2013 at 2:59 PM, Aaron Coburn <ac...@amherst.edu> wrote:
>>> In my opinion, this issue has largely been solved by the NoSQL and
>>> application messaging communities. But as long as the VCL web server(s)
>>> use local, file-based semaphores to control resource allocation, this
>>> will be a hard nut to crack.
>>>
>>> If I were going to tackle this problem, I would take one of two general
>>> approaches:
>>>
>>> 1) Use an asynchronous messaging queue (i.e. Apache ActiveMQ or something
>>> like Redis)
>>>
>>> In this way, when a reservation is made, the request is pushed onto a
>>> centralized queue, and some intermediate (i.e. vcld) process will shift
>>> messages off the queue and assign the reservations. If we used ActiveMQ,
>>> the php and perl code would communicate with the message broker over the
>>> STOMP protocol [1]. Redis would also work because it runs in a single
>>> thread and all operations are atomic -- requests can simply be pushed
>>> onto a FIFO-type list structure [2].
>>>
>>> 2. Use a NoSQL database such as CouchDB or Riak that uses a type of
>>> optimistic locking model for all writes.
>>>
>>> That is, if reservations are stored in such a way that the resource id
>>> (i.e. computer id) forms the database key, the assignment of a user to a
>>> particular compute resource requires sending the correct revision_id of
>>> that resource in order for the write to be successful. If successful, it
>>> returns an HTTP 200 status code and the client proceeds normally;
>>> otherwise, it sends a 409 header and it is up to the client to try again
>>> [3]. It would also be possible to use the existing MySQL database to
>>> emulate something like #2, but under a heavy load, I imagine that the row
>>> locking could really slow things down. Using Couch would really be much
>>> more scalable than trying to do everything in MySQL.
>>>
>>> Either of these approaches would then allow you to distribute the web
>>> front end across multiple machines.
>>>
>>> Cheers,
>>> Aaron Coburn
>>>
>>> [1] http://activemq.apache.org/stomp.html
>>> [2] http://redis.io/commands/lpush
>>> [3] http://wiki.apache.org/couchdb/HTTP_Document_API#PUT
>>>
>>> On Jul 17, 2013, at 1:06 PM, Aaron Peeler <aa...@ncsu.edu> wrote:
>>>> This is definitely desired.
>>>>
>>>> An additional challenge with multiple web servers is to figure how to
>>>> sanely lock/unlock the resource to prevent it from getting assigned
>>>> two separate users that are making requests at the same instant.
>>>>
>>>> AP
>>>>
>>>> On Wed, Jul 17, 2013 at 12:57 PM, James O'Dell <jo...@fullerton.edu>
> wrote:
>>>>> I also tried load balancing the web. I didn't have any success.
>>>>>
>>>>> I also tried an SSL offload using the F5
>>>>>
>>>>> The SSLoffload didn't work because
>>>>> an internal VCL check noticed the header wasn't https
>>>>> and redirected to https. Basically it kept looping
>>>>>
>>>>> On 7/17/2013 9:36 AM, Dmitri Chebotarov wrote:
>>>>>
>>>>> Hi
>>>>>
>>>>> I would like to load balance multiple front-end VCL servers.
>>>>> It is F5 load balancer. The LB configuration allows to enable session
>>>>> persistency.
>>>>> Is this will be OK with VCL's front-end? I remember someone mentioned
>>>>> some
>>>>> issues with having multiple front-end servers, but don't remember
>>>>> details.
>>>>>
>>>>> Thanks.
>>>> --
>>>> Aaron Peeler
>>>> Program Manager
>>>> Virtual Computing Lab
>>>> NC State University
>>>>
>>>> All electronic mail messages in connection with State business which
>>>> are sent to or received by this account are subject to the NC Public
>>>> Records Law and may be disclosed to third parties.
> - --
> - -------------------------------
> Josh Thompson
> VCL Developer
> North Carolina State University
>
> my GPG/PGP key can be found at pgp.mit.edu
>
> All electronic mail messages in connection with State business which
> are sent to or received by this account are subject to the NC Public
> Records Law and may be disclosed to third parties.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.19 (GNU/Linux)
>
> iEYEARECAAYFAlHtojYACgkQV/LQcNdtPQNNjwCfQ+PRt/yZXI4tf2YDiH2IZu2m
> coYAn2QbjICSa5MUR0cR9DIxniZVQFtK
> =/YZs
> -----END PGP SIGNATURE-----
>
Re: VCL front-end and load balancer
Posted by Josh Thompson <jo...@ncsu.edu>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
The problem with load balancing the front end is that there is a semaphore
lock (php's built in implementation) around the scheduling code. Without it,
2 users coming in at the same time can get assigned the same computer. So,
having multiple frontends would allow the same thing to happen.
I've considered just locking a table in the database as a semaphore. However,
in my testing of this, if the web server locks a table and then goes offline
before unlocking it, the table stays locked. So, done this way, if there were
multiple frontends and one went offline while in the middle of the lock, none
of them would be able to create reservations. I don't remember what had to be
done to release the lock.
The option I'd like to use that I've not gotten around to implementing (I
believe there is a really old JIRA issue for it) is to add an XMLRPC API
function for calling the scheduling code. Then, have one frontend assigned as
the one that would handle all of the scheduling. The others would make calls
to that one for just the scheduling part, but would do everything else
normally. Optionally, there could be an election process so that a new
frontend could be selected to do the scheduling if the one that had been doing
it went down.
Aaron C. listed some good ideas, but I think the above would be more
straightforward since it would not involve bringing in other technologies.
Josh
On Thursday, July 18, 2013 9:49:39 AM Aaron Peeler wrote:
> I like both of these approaches, especially 1, but I'm not sure of the
> effort it would take to convert.
> -AP
>
> On Wed, Jul 17, 2013 at 2:59 PM, Aaron Coburn <ac...@amherst.edu> wrote:
> > In my opinion, this issue has largely been solved by the NoSQL and
> > application messaging communities. But as long as the VCL web server(s)
> > use local, file-based semaphores to control resource allocation, this
> > will be a hard nut to crack.
> >
> > If I were going to tackle this problem, I would take one of two general
> > approaches:
> >
> > 1) Use an asynchronous messaging queue (i.e. Apache ActiveMQ or something
> > like Redis)
> >
> > In this way, when a reservation is made, the request is pushed onto a
> > centralized queue, and some intermediate (i.e. vcld) process will shift
> > messages off the queue and assign the reservations. If we used ActiveMQ,
> > the php and perl code would communicate with the message broker over the
> > STOMP protocol [1]. Redis would also work because it runs in a single
> > thread and all operations are atomic -- requests can simply be pushed
> > onto a FIFO-type list structure [2].
> >
> > 2. Use a NoSQL database such as CouchDB or Riak that uses a type of
> > optimistic locking model for all writes.
> >
> > That is, if reservations are stored in such a way that the resource id
> > (i.e. computer id) forms the database key, the assignment of a user to a
> > particular compute resource requires sending the correct revision_id of
> > that resource in order for the write to be successful. If successful, it
> > returns an HTTP 200 status code and the client proceeds normally;
> > otherwise, it sends a 409 header and it is up to the client to try again
> > [3]. It would also be possible to use the existing MySQL database to
> > emulate something like #2, but under a heavy load, I imagine that the row
> > locking could really slow things down. Using Couch would really be much
> > more scalable than trying to do everything in MySQL.
> >
> > Either of these approaches would then allow you to distribute the web
> > front end across multiple machines.
> >
> > Cheers,
> > Aaron Coburn
> >
> > [1] http://activemq.apache.org/stomp.html
> > [2] http://redis.io/commands/lpush
> > [3] http://wiki.apache.org/couchdb/HTTP_Document_API#PUT
> >
> > On Jul 17, 2013, at 1:06 PM, Aaron Peeler <aa...@ncsu.edu> wrote:
> >> This is definitely desired.
> >>
> >> An additional challenge with multiple web servers is to figure how to
> >> sanely lock/unlock the resource to prevent it from getting assigned
> >> two separate users that are making requests at the same instant.
> >>
> >> AP
> >>
> >> On Wed, Jul 17, 2013 at 12:57 PM, James O'Dell <jo...@fullerton.edu>
wrote:
> >>> I also tried load balancing the web. I didn't have any success.
> >>>
> >>> I also tried an SSL offload using the F5
> >>>
> >>> The SSLoffload didn't work because
> >>> an internal VCL check noticed the header wasn't https
> >>> and redirected to https. Basically it kept looping
> >>>
> >>> On 7/17/2013 9:36 AM, Dmitri Chebotarov wrote:
> >>>
> >>> Hi
> >>>
> >>> I would like to load balance multiple front-end VCL servers.
> >>> It is F5 load balancer. The LB configuration allows to enable session
> >>> persistency.
> >>> Is this will be OK with VCL's front-end? I remember someone mentioned
> >>> some
> >>> issues with having multiple front-end servers, but don't remember
> >>> details.
> >>>
> >>> Thanks.
> >>
> >> --
> >> Aaron Peeler
> >> Program Manager
> >> Virtual Computing Lab
> >> NC State University
> >>
> >> All electronic mail messages in connection with State business which
> >> are sent to or received by this account are subject to the NC Public
> >> Records Law and may be disclosed to third parties.
- --
- -------------------------------
Josh Thompson
VCL Developer
North Carolina State University
my GPG/PGP key can be found at pgp.mit.edu
All electronic mail messages in connection with State business which
are sent to or received by this account are subject to the NC Public
Records Law and may be disclosed to third parties.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
iEYEARECAAYFAlHtojYACgkQV/LQcNdtPQNNjwCfQ+PRt/yZXI4tf2YDiH2IZu2m
coYAn2QbjICSa5MUR0cR9DIxniZVQFtK
=/YZs
-----END PGP SIGNATURE-----
Re: VCL front-end and load balancer
Posted by Aaron Peeler <aa...@ncsu.edu>.
I like both of these approaches, especially 1, but I'm not sure of the
effort it would take to convert.
-AP
On Wed, Jul 17, 2013 at 2:59 PM, Aaron Coburn <ac...@amherst.edu> wrote:
> In my opinion, this issue has largely been solved by the NoSQL and application messaging communities. But as long as the VCL web server(s) use local, file-based semaphores to control resource allocation, this will be a hard nut to crack.
>
> If I were going to tackle this problem, I would take one of two general approaches:
>
> 1) Use an asynchronous messaging queue (i.e. Apache ActiveMQ or something like Redis)
>
> In this way, when a reservation is made, the request is pushed onto a centralized queue, and some intermediate (i.e. vcld) process will shift messages off the queue and assign the reservations. If we used ActiveMQ, the php and perl code would communicate with the message broker over the STOMP protocol [1]. Redis would also work because it runs in a single thread and all operations are atomic -- requests can simply be pushed onto a FIFO-type list structure [2].
>
> 2. Use a NoSQL database such as CouchDB or Riak that uses a type of optimistic locking model for all writes.
>
> That is, if reservations are stored in such a way that the resource id (i.e. computer id) forms the database key, the assignment of a user to a particular compute resource requires sending the correct revision_id of that resource in order for the write to be successful. If successful, it returns an HTTP 200 status code and the client proceeds normally; otherwise, it sends a 409 header and it is up to the client to try again [3]. It would also be possible to use the existing MySQL database to emulate something like #2, but under a heavy load, I imagine that the row locking could really slow things down. Using Couch would really be much more scalable than trying to do everything in MySQL.
>
> Either of these approaches would then allow you to distribute the web front end across multiple machines.
>
> Cheers,
> Aaron Coburn
>
> [1] http://activemq.apache.org/stomp.html
> [2] http://redis.io/commands/lpush
> [3] http://wiki.apache.org/couchdb/HTTP_Document_API#PUT
>
>
>
>
>
>
>
>
>
> On Jul 17, 2013, at 1:06 PM, Aaron Peeler <aa...@ncsu.edu> wrote:
>
>> This is definitely desired.
>>
>> An additional challenge with multiple web servers is to figure how to
>> sanely lock/unlock the resource to prevent it from getting assigned
>> two separate users that are making requests at the same instant.
>>
>> AP
>>
>> On Wed, Jul 17, 2013 at 12:57 PM, James O'Dell <jo...@fullerton.edu> wrote:
>>>
>>> I also tried load balancing the web. I didn't have any success.
>>>
>>> I also tried an SSL offload using the F5
>>>
>>> The SSLoffload didn't work because
>>> an internal VCL check noticed the header wasn't https
>>> and redirected to https. Basically it kept looping
>>>
>>> On 7/17/2013 9:36 AM, Dmitri Chebotarov wrote:
>>>
>>> Hi
>>>
>>> I would like to load balance multiple front-end VCL servers.
>>> It is F5 load balancer. The LB configuration allows to enable session
>>> persistency.
>>> Is this will be OK with VCL's front-end? I remember someone mentioned some
>>> issues with having multiple front-end servers, but don't remember details.
>>>
>>> Thanks.
>>>
>>>
>>
>>
>>
>> --
>> Aaron Peeler
>> Program Manager
>> Virtual Computing Lab
>> NC State University
>>
>> All electronic mail messages in connection with State business which
>> are sent to or received by this account are subject to the NC Public
>> Records Law and may be disclosed to third parties.
>
--
Aaron Peeler
Program Manager
Virtual Computing Lab
NC State University
All electronic mail messages in connection with State business which
are sent to or received by this account are subject to the NC Public
Records Law and may be disclosed to third parties.
Re: VCL front-end and load balancer
Posted by Aaron Coburn <ac...@amherst.edu>.
In my opinion, this issue has largely been solved by the NoSQL and application messaging communities. But as long as the VCL web server(s) use local, file-based semaphores to control resource allocation, this will be a hard nut to crack.
If I were going to tackle this problem, I would take one of two general approaches:
1) Use an asynchronous messaging queue (i.e. Apache ActiveMQ or something like Redis)
In this way, when a reservation is made, the request is pushed onto a centralized queue, and some intermediate (i.e. vcld) process will shift messages off the queue and assign the reservations. If we used ActiveMQ, the php and perl code would communicate with the message broker over the STOMP protocol [1]. Redis would also work because it runs in a single thread and all operations are atomic -- requests can simply be pushed onto a FIFO-type list structure [2].
2. Use a NoSQL database such as CouchDB or Riak that uses a type of optimistic locking model for all writes.
That is, if reservations are stored in such a way that the resource id (i.e. computer id) forms the database key, the assignment of a user to a particular compute resource requires sending the correct revision_id of that resource in order for the write to be successful. If successful, it returns an HTTP 200 status code and the client proceeds normally; otherwise, it sends a 409 header and it is up to the client to try again [3]. It would also be possible to use the existing MySQL database to emulate something like #2, but under a heavy load, I imagine that the row locking could really slow things down. Using Couch would really be much more scalable than trying to do everything in MySQL.
Either of these approaches would then allow you to distribute the web front end across multiple machines.
Cheers,
Aaron Coburn
[1] http://activemq.apache.org/stomp.html
[2] http://redis.io/commands/lpush
[3] http://wiki.apache.org/couchdb/HTTP_Document_API#PUT
On Jul 17, 2013, at 1:06 PM, Aaron Peeler <aa...@ncsu.edu> wrote:
> This is definitely desired.
>
> An additional challenge with multiple web servers is to figure how to
> sanely lock/unlock the resource to prevent it from getting assigned
> two separate users that are making requests at the same instant.
>
> AP
>
> On Wed, Jul 17, 2013 at 12:57 PM, James O'Dell <jo...@fullerton.edu> wrote:
>>
>> I also tried load balancing the web. I didn't have any success.
>>
>> I also tried an SSL offload using the F5
>>
>> The SSLoffload didn't work because
>> an internal VCL check noticed the header wasn't https
>> and redirected to https. Basically it kept looping
>>
>> On 7/17/2013 9:36 AM, Dmitri Chebotarov wrote:
>>
>> Hi
>>
>> I would like to load balance multiple front-end VCL servers.
>> It is F5 load balancer. The LB configuration allows to enable session
>> persistency.
>> Is this will be OK with VCL's front-end? I remember someone mentioned some
>> issues with having multiple front-end servers, but don't remember details.
>>
>> Thanks.
>>
>>
>
>
>
> --
> Aaron Peeler
> Program Manager
> Virtual Computing Lab
> NC State University
>
> All electronic mail messages in connection with State business which
> are sent to or received by this account are subject to the NC Public
> Records Law and may be disclosed to third parties.
Re: VCL front-end and load balancer
Posted by Aaron Peeler <aa...@ncsu.edu>.
This is definitely desired.
An additional challenge with multiple web servers is to figure how to
sanely lock/unlock the resource to prevent it from getting assigned
two separate users that are making requests at the same instant.
AP
On Wed, Jul 17, 2013 at 12:57 PM, James O'Dell <jo...@fullerton.edu> wrote:
>
> I also tried load balancing the web. I didn't have any success.
>
> I also tried an SSL offload using the F5
>
> The SSLoffload didn't work because
> an internal VCL check noticed the header wasn't https
> and redirected to https. Basically it kept looping
>
> On 7/17/2013 9:36 AM, Dmitri Chebotarov wrote:
>
> Hi
>
> I would like to load balance multiple front-end VCL servers.
> It is F5 load balancer. The LB configuration allows to enable session
> persistency.
> Is this will be OK with VCL's front-end? I remember someone mentioned some
> issues with having multiple front-end servers, but don't remember details.
>
> Thanks.
>
>
--
Aaron Peeler
Program Manager
Virtual Computing Lab
NC State University
All electronic mail messages in connection with State business which
are sent to or received by this account are subject to the NC Public
Records Law and may be disclosed to third parties.