Multithreaded Volume Attachment: mixed up when attaching two volumes in parallel

[Ambadas Ramanna Adam <Am...@infosys.com>]

Hi All,



I am using JClouds 1.7.3 and tried 1.8.0 also without success. I am running this against openstack on RHEL 6.5 (Setup done as per https://openstack.redhat.com/Quickstart).



Here is what we do.

1. Create virtual instance.

2. Create blank volume.

3. Create volume from a snapshot.

4. Attach blank volume to instance.

5. Attach volume from snapshot to instance.

6. Other IP address, security group stuff.



All these are done in separate threads but with dependencies added. Ex. 4 depends on 2 (4->2). 2->1. 5->3->1.

The problem is that it works sometimes and sometimes it doesn't.

1. One of the devices don't appear in the device list. Openstack shows both attached.

2. Both device don't appear in the device list. Openstack shows both attached.

3. Both appear but, disks at specified devices get switched! Ex. We attached 1GB disk on /dev/vdb and 10GB disk on /dev/vdc. But when we do fdisk -l /dev/vdc we get 1G and /dev/vdb shows 10G.

4. Works!!!



I am using JClouds API as shown below to attach volume in two separate threads.

VolumeAttachmentApi attachment = nova.getVolumeAttachmentExtensionForZone(CONSTANT_REGION).get();

VolumeAttachment attachment.attachVolumeToServerAsDevice(volumeIdExisting, instanceId, deviceDevXVDB);



I created a sample program to isolate the issue. This sample program does the following:



    /-> 2 -> 3 >\

1 -> -> 6 -> -> x

    \-> 4 -> 5 >/



Basically the main thread creates the instance and launches two threads. One thread creates blank volume (10GB) and attaches it. While the other creates a volume from snapshot (1GB) and attaches it. The main thread then creates IP and attaches it and waits for the two threads to complete.



The observation is rather strange. The attach volume response and what is observed inside of the instance OS is always mixed up. JClouds api calls respond saying the blank volume (10GB)got attached to /dev/vdc, and the volume from snapshot(1GB) attached to /dev/vdb.



But when I checked on the instance

fdisk -l /dev/vdb - returns 10GB and

fdisk -l /dev/vdc - returns 1GB



At this point I tried using direct ReST API calls (using Jersey Client) to the openstack installation. And they work perfectly well in separate threads in parallel.



Is JClouds supposed to be used in multiple threads? Can we attach two volumes onto a virtual server at the same time?



Please let me know how to fix this.

Any hints, pointers or references will be of help too.



Regards,

Adam A R

**************** CAUTION - Disclaimer *****************
This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely 
for the use of the addressee(s). If you are not the intended recipient, please 
notify the sender by e-mail and delete the original message. Further, you are not 
to copy, disclose, or distribute this e-mail or its contents to any other person and 
any such actions are unlawful. This e-mail may contain viruses. Infosys has taken 
every reasonable precaution to minimize this risk, but is not liable for any damage 
you may sustain as a result of any virus in this e-mail. You should carry out your 
own virus checks before opening the e-mail or attachment. Infosys reserves the 
right to monitor and review the content of all messages sent to or from this e-mail 
address. Messages sent to or from this e-mail address may be stored on the 
Infosys e-mail system.
***INFOSYS******** End of Disclaimer ********INFOSYS***