You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Indra Pramana <in...@sg.or.id> on 2013/07/10 11:45:25 UTC
Problem in adding Ceph RBD as primary storage for CloudStack 4.1.0
Dear all,
I am installing CloudStack 4.1.0 (upgraded from 4.0.2) and I also have a
Ceph cluster running. However, I am having issues in adding the RBD as
primary storage. Tried to follow the instruction here, but unable to make
it work:
http://ceph.com/docs/master/rbd/rbd-cloudstack/
I have setup a pool on the Ceph cluster. The status of the cluster is
healthy. Since I am using Ubuntu 12.04.2 LTS (Precise) for the hypervisors,
I also have compiled libvirt manually to ensure that the version 0.9.13 is
installed (previously it's 0.9.8).
indra@hv-kvm-01:~/rbd$ ceph
ceph> health
HEALTH_OK
indra@hv-kvm-01:~$ ceph osd lspools
0 data,1 metadata,2 rbd,3 sc1,
root@hv-kvm-01:/home/indra# libvirtd --version
libvirtd (libvirt) 0.9.13
I tried to add Primary Storage into the Cloudstack zone which I have
created:
Add Primary Storage:
Zone: my zone name
Pod: my pod name
Cluster: my cluster name
Name: ceph-rbd-pri-storage
Protocol: RBD
RADOS Monitor: my first Ceph monitor IP address
RADOS Pool: sc1 (the pool name on Ceph cluster)
RADOS User: client.admin
RADOS Secret: /etc/ceph/ceph.client.admin.keyring (keyring file location)
Storage Tags: rbd
This is the error message when I tried to add the primary storage by
clicking OK:
DB Exception on: com.mysql.jdbc.JDBC4PreparedStatement@4b2eb56: INSERT INTO
storage_pool (storage_pool.id, storage_pool.name, storage_pool.uuid,
storage_pool.pool_type, storage_pool.created, storage_pool.update_time,
storage_pool.data_center_id, storage_pool.pod_id,
storage_pool.available_bytes, storage_pool.capacity_bytes,
storage_pool.status, storage_pool.scope, storage_pool.storage_provider_id,
storage_pool.host_address, storage_pool.path, storage_pool.port,
storage_pool.user_info, storage_pool.cluster_id) VALUES (217,
_binary'ceph-rbd-pri-storage',
_binary'a226c9a1-da78-3f3a-b5ac-e18b925c9634', 'RBD', '2013-07-10
09:08:28', null, 2, 2, 0, 0, 'Up', null, null, null, _binary'ceph/
ceph.client.admin.keyring@10.237.11.2/sc1', 6789, null, 2)
On the management-server.log file:
2013-07-10 17:08:28,845 DEBUG [cloud.api.ApiServlet] (catalina-exec-2:null)
===START=== 192.168.0.100 -- GET
command=createStoragePool&zoneid=c116950e-e4ae-4f23-a7e7-
74a75c4ee638&podId=a748b063-3a83-4175-a0e9-de39118fe5ce&clusterid=1f87eb09-324d-4d49-83c2-88d84d7a15df&name=ceph-rbd-pri-storage&url=rbd%3A%2F%2Fclient.admin%3A_etc%2Fc
eph%2Fceph.client.admin.keyring%4010.237.11.2%2Fsc1&tags=rbd&response=json&sessionkey=rDRfWpqeKfQKbKZtHr398ULV%2F8k%3D&_=1373447307839
2013-07-10 17:08:28,862 DEBUG [cloud.storage.StorageManagerImpl]
(catalina-exec-2:null) createPool Params @ scheme - rbd storageHost - null
hostPath - /ceph/ceph.client
.admin.keyring@10.237.11.2/sc1 port - -1
2013-07-10 17:08:28,918 DEBUG [cloud.storage.StorageManagerImpl]
(catalina-exec-2:null) In createPool Setting poolId - 217 uuid -
a226c9a1-da78-3f3a-b5ac-e18b925c9634 z
oneId - 2 podId - 2 poolName - ceph-rbd-pri-storage
2013-07-10 17:08:28,921 DEBUG [db.Transaction.Transaction]
(catalina-exec-2:null) Rolling back the transaction: Time = 3 Name =
persist; called by -Transaction.rollbac
k:890-Transaction.removeUpTo:833-Transaction.close:657-TransactionContextBuilder.interceptException:63-ComponentInstantiationPostProcessor$InterceptorDispatcher.interce
pt:133-StorageManagerImpl.createPool:1378-StorageManagerImpl.createPool:147-CreateStoragePoolCmd.execute:123-ApiDispatcher.dispatch:162-ApiServer.queueCommand:505-ApiSe
rver.handleRequest:355-ApiServlet.processRequest:302
2013-07-10 17:08:28,923 ERROR [cloud.api.ApiServer] (catalina-exec-2:null)
unhandled exception executing api command: createStoragePool
com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
com.mysql.jdbc.JDBC4PreparedStatement@4b2eb56: INSERT INTO storage_pool (
storage_pool.id, storage_pool
.name, storage_pool.uuid, storage_pool.pool_type, storage_pool.created,
storage_pool.update_time, storage_pool.data_center_id, storage_pool.pod_id,
storage_pool.availab
le_bytes, storage_pool.capacity_bytes, storage_pool.status,
storage_pool.scope, storage_pool.storage_provider_id,
storage_pool.host_address, storage_pool.path, storage_
pool.port, storage_pool.user_info, storage_pool.cluster_id) VALUES (217,
_binary'ceph-rbd-pri-storage',
_binary'a226c9a1-da78-3f3a-b5ac-e18b925c9634', 'RBD', '2013-07-1
0 09:08:28', null, 2, 2, 0, 0, 'Up', null, null, null, _binary'ceph/
ceph.client.admin.keyring@10.237.11.2/sc1', 6789, null, 2)
at
com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1342)
at
com.cloud.storage.dao.StoragePoolDaoImpl.persist(StoragePoolDaoImpl.java:232)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:1378)
at
com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:147)
at
org.apache.cloudstack.api.command.admin.storage.CreateStoragePoolCmd.execute(CreateStoragePoolCmd.java:123)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:505)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:355)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:302)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:66)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:615)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at
org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
at
org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:744)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2282)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
Caused by:
com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
Column 'host_address' cannot be null
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.Util.getInstance(Util.java:386)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1040)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4074)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4006)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2468)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2629)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2719)
at
com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2155)
at
com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2450)
at
com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2371)
at
com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2355)
at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
at
com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1305)
... 32 more
2013-07-10 17:08:28,927 DEBUG [cloud.api.ApiServlet] (catalina-exec-2:null)
===END=== 192.168.0.100 -- GET
command=createStoragePool&zoneid=c116950e-e4ae-4f23-a7e7-74a75c4ee638&podId=a748b063-3a83-4175-a0e9-de39118fe5ce&clusterid=1f87eb09-324d-4d49-83c2-88d84d7a15df&name=ceph-rbd-pri-storage&url=rbd%3A%2F%2Fclient.admin%3A_etc%2Fceph%2Fceph.client.admin.keyring%4010.237.11.2%2Fsc1&tags=rbd&response=json&sessionkey=rDRfWpqeKfQKbKZtHr398ULV%2F8k%3D&_=1373447307839
Based on the error message, I tried to map the value of each field being
submitted, and it seems the problem is here:
Caused by:
com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
Column 'host_address' cannot be null
storage_pool.id, 215
storage_pool.name, _binary'sc1'
storage_pool.uuid, _binary'a226c9a1-da78-3f3a-b5ac-e18b925c9634'
storage_pool.pool_type, 'RBD'
storage_pool.created, '2013-07-10 08:27:03'
storage_pool.update_time, null
storage_pool.data_center_id, 2
storage_pool.pod_id, 2
storage_pool.available_bytes, 0
storage_pool.capacity_bytes, 0
storage_pool.status, 'Up'
storage_pool.scope, null
storage_pool.storage_provider_id, null
storage_pool.host_address, null <=======
storage_pool.path, _binary'ceph/ceph.client.admin.keyring@10.237.11.2/sc1
storage_pool.port, 6789
storage_pool.user_info, null
storage_pool.cluster_id, 2
It seems that it's expecting the IP address / hostname of the monitor
server on the "storage_pool.host_address" field. However, I noted that the
IP address is in the storage_pool.path field rather than the
storage_pool.host_address.
Anyone knows what can be done to resolve this problem?
Looking forward to your reply, thank you.
Cheers.
Re: Problem in adding Ceph RBD as primary storage for CloudStack
4.1.0
Posted by Prasanna Santhanam <ts...@apache.org>.
On Mon, Jul 15, 2013 at 06:44:12PM +0800, Indra Pramana wrote:
> > > Any other way on how to console in to the newly created VM other than
> > using
> > > the console proxy method? Unlike system VM, the instance VM doesn't give
> > > any link local address which I can SSH directly from the KVM host. Am I
> > > able to using virsh command to log in to the instance?
> >
> > Yup - virsh console to your domain would work. But is unsuitable for
> > users/tenants of the cloudstack deployment. I'll defer to someone with
> > more knowledge about the console-proxy issue.
> >
>
> I managed to find out the reason why the console-proxy didn't work. I set
> up Google's DNS servers (8.8.8.8 and 8.8.4.4) as Internal DNS 1 and 2
> during zone creation since we don't have our own internal DNS server.
>
> This caused a routing entry on the CPVM for 8.8.8.8 and 8.8.4.4 to the
> internal network interface of the VM, causing the CPVM not able to resolve
> domains within the VM, resulting to the issue. I did a "route del" of the
> two entries from the VM's routing table and it resolves the problem.
>
> /sbin/route del -host 8.8.4.4 gw 10.19.16.1
> /sbin/route del -host 8.8.8.8 gw 10.19.16.1
>
Thanks - so the CPVM talks to the vnc ports on your hosts on the
management network. I guess there should be a way to turn off
name-resolution in such cases as your deployment where you don't allow
any public traffic on your management network. Either that or we need
to note this in our documentation somewhere. Could you please point to
this thread and file a bug report?
You can sign up for our bugtracker here:
https://issues.apache.org/jira/secure/Signup!default.jspa
--
Prasanna.,
------------------------
Powered by BigRock.com
Re: Problem in adding Ceph RBD as primary storage for CloudStack 4.1.0
Posted by Indra Pramana <in...@sg.or.id>.
Hi Prasanna,
Good day to you, and thank you for your e-mail.
On Mon, Jul 15, 2013 at 5:57 PM, Prasanna Santhanam <ts...@apache.org> wrote:
> On Mon, Jul 15, 2013 at 04:45:34PM +0800, Indra Pramana wrote:
> > Hi Prasanna and all,
> >
> > I managed to work out on how to assign public IP to the VM instance using
> > static NAT.
> >
> > However, I am still stuck with the console proxy time-out issue. It seems
> > that my problem is exactly the one mentioned on this bug reported to
> JIRA:
> >
> >
> https://issues.apache.org/jira/browse/CLOUDSTACK-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel
> >
> > May I know whether there's any resolution to the bug above? The issue
> seems
> > to be identical to the one I am facing right now.
> >
> > Any other way on how to console in to the newly created VM other than
> using
> > the console proxy method? Unlike system VM, the instance VM doesn't give
> > any link local address which I can SSH directly from the KVM host. Am I
> > able to using virsh command to log in to the instance?
>
> Yup - virsh console to your domain would work. But is unsuitable for
> users/tenants of the cloudstack deployment. I'll defer to someone with
> more knowledge about the console-proxy issue.
>
I managed to find out the reason why the console-proxy didn't work. I set
up Google's DNS servers (8.8.8.8 and 8.8.4.4) as Internal DNS 1 and 2
during zone creation since we don't have our own internal DNS server.
This caused a routing entry on the CPVM for 8.8.8.8 and 8.8.4.4 to the
internal network interface of the VM, causing the CPVM not able to resolve
domains within the VM, resulting to the issue. I did a "route del" of the
two entries from the VM's routing table and it resolves the problem.
/sbin/route del -host 8.8.4.4 gw 10.19.16.1
/sbin/route del -host 8.8.8.8 gw 10.19.16.1
Thank you.
Re: Problem in adding Ceph RBD as primary storage for CloudStack
4.1.0
Posted by Prasanna Santhanam <ts...@apache.org>.
On Mon, Jul 15, 2013 at 04:45:34PM +0800, Indra Pramana wrote:
> Hi Prasanna and all,
>
> I managed to work out on how to assign public IP to the VM instance using
> static NAT.
>
> However, I am still stuck with the console proxy time-out issue. It seems
> that my problem is exactly the one mentioned on this bug reported to JIRA:
>
> https://issues.apache.org/jira/browse/CLOUDSTACK-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel
>
> May I know whether there's any resolution to the bug above? The issue seems
> to be identical to the one I am facing right now.
>
> Any other way on how to console in to the newly created VM other than using
> the console proxy method? Unlike system VM, the instance VM doesn't give
> any link local address which I can SSH directly from the KVM host. Am I
> able to using virsh command to log in to the instance?
Yup - virsh console to your domain would work. But is unsuitable for
users/tenants of the cloudstack deployment. I'll defer to someone with
more knowledge about the console-proxy issue.
--
Prasanna.,
------------------------
Powered by BigRock.com
Re: Problem in adding Ceph RBD as primary storage for CloudStack 4.1.0
Posted by Indra Pramana <in...@sg.or.id>.
Hi Prasanna and all,
I managed to work out on how to assign public IP to the VM instance using
static NAT.
However, I am still stuck with the console proxy time-out issue. It seems
that my problem is exactly the one mentioned on this bug reported to JIRA:
https://issues.apache.org/jira/browse/CLOUDSTACK-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel
May I know whether there's any resolution to the bug above? The issue seems
to be identical to the one I am facing right now.
Any other way on how to console in to the newly created VM other than using
the console proxy method? Unlike system VM, the instance VM doesn't give
any link local address which I can SSH directly from the KVM host. Am I
able to using virsh command to log in to the instance?
Looking forward to your reply, thank you.
Cheers.
On Mon, Jul 15, 2013 at 3:19 PM, Indra Pramana <in...@sg.or.id> wrote:
> Hi Prasanna and all,
>
> I managed to create a VM instance using the NFS primary storage (not using
> the RBD primary storage due to the bug we discussed on another thread).
>
> However, there is no public IP being assigned to the VM instance (unlike
> the system VMs which are getting public IPs). How do I assign public IP to
> the instance?
>
> ===
> NIC 1 (Default)
> Network Name Indra
> IP Address 10.19.8.186
> Type Isolated
> Gateway 10.19.0.1
> Netmask 255.255.240.0
> Is Default Yes
> ===
>
> The network was created under the
> "DefaultIsolatedNetworkOfferingWithSourceNatService" network offering. I am
> not able to create the network using different network offerings, despite
> the other offerings are available under Service Offerings > Network
> Offerings and the state of those network offerings are enabled. Is it
> related to the way how I configured the network bridges and configuration
> earlier during zone creation, anything that I might have missed out?
>
> I also tried to "view console" to the VM (via the web GUI) and not able to
> do so. Earlier I got the "Unable to start console session as connection is
> refused by the machine you are accessing" error message, which I have fixed
> by changing the /etc/libvirt/qemu.conf to enable this line:
>
> vnc_listen = "0.0.0.0"
>
> because initially the VNC only listens on localhost (127.0.0.1) by
> default. Now I am no longer getting the error message, but the console
> window still failed to load after some time, with Firefox browser error
> message:
>
> ===
> The connection was reset
>
> The connection to the server was reset while the page was loading.
> ===
>
> How can I further troubleshoot on this? I tried to SSH into the console
> proxy VM and can't find anything useful on the logs there.
>
>
> Looking forward to your reply, thank you.
>
> Cheers.
>
>
>
> On Mon, Jul 15, 2013 at 12:38 PM, Indra Pramana <in...@sg.or.id> wrote:
>
>> Hi Prasanna,
>>
>> On Mon, Jul 15, 2013 at 12:20 PM, Prasanna Santhanam <ts...@apache.org>wrote:
>>
>>> On Sat, Jul 13, 2013 at 02:25:35PM +0800, Indra Pramana wrote:
>>> > Hi Prasanna,
>>> >
>>> > Good day to you, and thank you for your e-mail.
>>> >
>>> Were you able to get beyond this error?
>>>
>>
>> Yes. I managed to get around the network bridges issue by setting up this
>> on my KVM host:
>>
>> ===
>> auto eth0.5
>> iface eth0.5 inet static
>> address X.X.X.22
>> netmask 255.255.255.240
>> network X.X.X.16
>> broadcast X.X.X.31
>> gateway X.X.X.17
>> # dns-* options are implemented by the resolvconf package, if
>> installed
>> dns-nameservers 8.8.8.8 8.8.4.4
>> dns-search xxx.com
>>
>> # Public network
>> auto cloudbr0
>> iface cloudbr0 inet manual
>> bridge_ports eth1.201
>> bridge_fd 5
>> bridge_stp off
>> bridge_maxwait 1
>>
>> # Guest network
>> auto cloudbr1
>> iface cloudbr1 inet manual
>> bridge_ports eth1.41
>> bridge_fd 5
>> bridge_stp off
>> bridge_maxwait 1
>>
>> # Management network
>> auto cloudbr2
>> iface cloudbr2 inet manual
>> bridge_ports eth0.6
>> bridge_fd 5
>> bridge_stp off
>> bridge_maxwait 1
>>
>> # The 10G network interface
>> auto eth2
>> iface eth2 inet static
>> address 10.237.11.22
>> netmask 255.255.255.0
>> network 10.237.11.0
>> broadcast 10.237.11.255
>> ===
>>
>> On the Cloudstack GUI, I created 3 physical networks:
>>
>> cloudbr0 - public network
>> cloudbr1 - guest network
>> cloudbr2 - management (and storage) network
>>
>> However, when I tried to setup the public network using VLAN 201, it
>> failed because CloudStack tried to create a new bridge (i.e. breth1-201)
>> instead of using the existing bridge I have set up for public network
>> (cloudbr mapped to eth1.201). So I set up the public network using a
>> different VLAN (VLAN 21) and change the settings on my router's gateway to
>> use VLAN 21 instead of VLAN 201.
>>
>> ===
>> root@hv-kvm-02:~# brctl show
>> bridge name bridge id STP enabled interfaces
>> breth1-21 8000.002590c29d7d no eth1.21
>> vnet2
>> vnet5
>> cloud0 8000.fe00a9fe0165 no vnet0
>> vnet3
>> cloudbr0 8000.002590c29d7d no eth1.201
>> cloudbr1 8000.002590c29d7d no eth1.41
>> cloudbr2 8000.002590c29d7c no eth0.6
>> vnet1
>> vnet4
>> vnet6
>> virbr0 8000.000000000000 yes
>> ===
>>
>> The above setup works, but I am not too sure whether this is the correct
>> way on how to do it. Can you advise?
>>
>> Looking forward to your reply, thank you.
>>
>> Cheers.
>>
>>
>>
>>
>>
>>> > Yesterday I troubleshooted further and found out that the mapping of
>>> the
>>> > bridge is incorrect. For example, eth1 of the SSVM, which is supposed
>>> to be
>>> > private network interface for communication with management server, is
>>> > being mapped to cloudbr0 bridge which is used for public. While eth2
>>> of the
>>> > SSVM, which is supposed to be public, is being mapped to cloudbr1
>>> bridge
>>> > which is supposed for guest traffic. I checked with brctl show on the
>>> KVM
>>> > host and traced the MAC address of the SSVM's eth interfaces.
>>> >
>>> > I have tried different type of configuration on the KVM hosts'
>>> > /etc/network/interfaces (I am using Ubuntu), different ways on how to
>>> > configure the advanced network zone (physical network interface and KVM
>>> > label for each traffic), but I am still not able to get it working.
>>> >
>>> > Can please advise what is the best way to set-up the network bridges?
>>>
>>> Use brctl to add bridges. That's how I've done it.
>>>
>>> >
>>> > My /etc/network/interfaces on KVM host:
>>> >
>>> > http://pastebin.com/nx8xJ1L2
>>> >
>>> > I used 3 physical NICs:
>>> >
>>> > eth0 --> for management and secondary storage traffic
>>> > eth1 --> for public and guest traffic
>>> > eth2 --> for primary storage traffic to the Ceph RBD (separate NIC not
>>> > configured by CloudStack)
>>> >
>>> > On the Cloudstack GUI, while creating the zone, I used 3 physical
>>> networks:
>>> >
>>> > eth0.6 --> Management traffic (KVM label: eth0.6)
>>> > cloudbr0 --> Public traffic (KVM label: cloudbr0)
>>> > cloudbr1 --> Guest traffic (KVM label: cloudbr1)
>>> >
>>> Is eth0.6 on a tagged VLAN?
>>>
>>> > I didn't specify storage traffic since I presume it will use the
>>> management
>>> > VLAN.
>>>
>>> That's correct.
>>>
>>> >
>>> > However, I always failed to add the KVM host using the above
>>> configuration.
>>> > I tried to run cloudstack-setup-agent manually on the KVM host and
>>> getting
>>> > this error message:
>>> >
>>> > root@hv-kvm-02:/etc/cloudstack/agent# cloudstack-setup-agent -m
>>> x.x.x.18 -z
>>> > 3 -p 3 -c 3 -g 00dd5dba-7419-3689-bc51-1671035c0d8f -a
>>> --pubNic=cloudbr0
>>> > --prvNic=eth0.6 --guestNic=cloudbr1
>>> > Starting to configure your system:
>>> > Configure Apparmor ... [OK]
>>> > Configure Network ... [Failed]
>>> > eth0.6 is not a bridge
>>> > Try to restore your system:
>>> > Restore Apparmor ... [OK]
>>> > Restore Network ... [OK]
>>> >
>>> > When I tried to create another bridge called cloudbr2 with
>>> bridge_ports to
>>> > eth0.6, the KVM host's network didn't work.
>>> >
>>> > Can advise what is the best practice on configuring the network
>>> bridges on
>>> > the advanced network zone setup?
>>> >
>>> > Looking forward to your reply, thank you.
>>> >
>>> > Cheers.
>>> >
>>> >
>>> >
>>> > On Sat, Jul 13, 2013 at 12:28 PM, Prasanna Santhanam <ts...@apache.org>
>>> wrote:
>>> >
>>> > > See Inline,
>>> > >
>>> > > On Fri, Jul 12, 2013 at 05:12:25PM +0800, Indra Pramana wrote:
>>> > > > Hi Wido,
>>> > > >
>>> > > > Noted, can't wait for 4.2 to be released. :)
>>> > > >
>>> > > > Dear Prasanna, Wido and all,
>>> > > >
>>> > > > I just realised that while the system VMs are running, they are
>>> still not
>>> > > > accessible through the public IPs assigned to them. I have been
>>> waiting
>>> > > for
>>> > > > the SSVM to download the default CentOS template and it doesn't
>>> appear on
>>> > > > the template list.
>>> > > >
>>> > > > I tried to SSH into the SSVM via the local link address from the
>>> KVM
>>> > > host,
>>> > > > and running the health check
>>> /usr/local/cloud/systemvm/ssvm-check.sh
>>> > > shows
>>> > > > that the VM cannot reach anywhere. It cannot reach the public DNS
>>> server
>>> > > (I
>>> > > > used Google's 8.8.8.8), cannot reach the management server, and
>>> cannot
>>> > > even
>>> > > > reach the public IP gateway.
>>> > >
>>> > > Ok - we're not there yet. But there's a few troubleshooting steps you
>>> > > can try:
>>> > >
>>> > > 1. Which of the checks fail apart from the DNS ping?
>>> > > 2. Are you able to have those tests pass from the KVM host itself?
>>> > > 3. Can you paste the output of $route -n on your SSVM and on your
>>> > > host?
>>> > >
>>> > > >
>>> > > > Is it due to misconfiguration of the KVM network bridges? How can
>>> I see
>>> > > the
>>> > > > mapping between the NIC interfaces of the SSVM (eth0, eth1, eth2
>>> and
>>> > > eth3)
>>> > > > and the actual physical NIC interfaces on the KVM hosts (eth0) and
>>> the
>>> > > > network bridges (cloudbr0, cloudbr1)? Any logs I can verify to
>>> ensure
>>> > > that
>>> > > > the VLAN and network bridging is working?
>>> > > >
>>> > >
>>> > > You can see this from the dumpxml.
>>> > >
>>> > > $virsh list
>>> > > (gives you all the domains. find the ssvm for example)
>>> > >
>>> > > $virsh dumpxml my-domain > mydomain.xml
>>> > >
>>> > > > Appreciate any advice.
>>> > > >
>>> > > > Thank you.
>>> > > >
>>> > > >
>>> > > >
>>> > > > On Fri, Jul 12, 2013 at 4:19 PM, Wido den Hollander <
>>> wido@widodh.nl>
>>> > > wrote:
>>> > > >
>>> > > > > On 07/12/2013 10:14 AM, Indra Pramana wrote:
>>> > > > >
>>> > > > >> Hi Prasanna,
>>> > > > >>
>>> > > > >> I managed to fix the problem, thanks for your advice to turn
>>> the agent
>>> > > > >> log level to debug:
>>> > > > >>
>>> > > > >> https://cwiki.apache.org/**confluence/display/CLOUDSTACK/**
>>> > > > >> KVM+agent+debug<
>>> > >
>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug>
>>> > > > >>
>>> > > > >> From the log, I found out that the agent on the KVM host tried
>>> to NFS
>>> > > > >> mount directly to 103.25.200.19:/mnt/vol1/sec-**
>>> > > > >> storage/template/tmpl/1/3,
>>> > > > >> which was not allowed by the NFS server due to its default
>>> > > configuration
>>> > > > >> to only allow to mount to /mnt/vol1/sec-storage (the root of
>>> the NFS
>>> > > > >> share).
>>> > > > >>
>>> > > > >>
>>> > > > > Ah, that's odd!
>>> > > > >
>>> > > > > Btw, in 4.2 you'll be able to deploy SSVMs on RBD as well, so
>>> that
>>> > > > > limitation will be gone.
>>> > > > >
>>> > > > > Wido
>>> > > > >
>>> > > > > After I changed the NFS server configuration to allow mount to
>>> all
>>> > > > >> sub-directories, re-export the NFS and voila, the system was
>>> able to
>>> > > > >> download the template and now both the system VMs (CPVM and
>>> SSVM) are
>>> > > > >> running!
>>> > > > >>
>>> > > > >> Many thanks for your help! :)
>>> > > > >>
>>> > > > >> Cheers.
>>> > > > >>
>>> > > > >>
>>> > > > >>
>>> > > > >> On Fri, Jul 12, 2013 at 3:31 PM, Indra Pramana <indra@sg.or.id
>>> > > > >> <ma...@sg.or.id>> wrote:
>>> > > > >>
>>> > > > >> Hi Prasanna,
>>> > > > >>
>>> > > > >> Good day to you, and thank you for your e-mail.
>>> > > > >>
>>> > > > >> Yes, the cloudstack-agent service is running on both the KVM
>>> > > hosts.
>>> > > > >> There is no "cloud" user being created though, when I
>>> installed
>>> > > the
>>> > > > >> agent. I installed the agent as root.
>>> > > > >>
>>> > > > >> root@hv-kvm-01:/home/indra# service cloudstack-agent status
>>> > > > >> * cloud-agent is running
>>> > > > >>
>>> > > > >> root@hv-kvm-01:/home/indra# su - cloud
>>> > > > >> Unknown id: cloud
>>> > > > >>
>>> > > > >> Please advise how can I resolve this problem, shall I
>>> create the
>>> > > > >> Unix "cloud" user manually? Basically I follow this
>>> instruction to
>>> > > > >> prepare the KVM host and install the CloudStack agent:
>>> > > > >>
>>> > > > >>
>>> http://cloudstack.apache.org/**docs/en-US/Apache_CloudStack/**
>>> > > > >>
>>> 4.1.0/html/Installation_Guide/**hypervisor-kvm-install-flow.**html<
>>> > >
>>> http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.1.0/html/Installation_Guide/hypervisor-kvm-install-flow.html
>>> > > >
>>> > > > >>
>>> > > > >> with this instruction from Wido on how to prepare libvirt
>>> with
>>> > > Ceph
>>> > > > >> RBD storage pool support:
>>> > > > >>
>>> > > > >> http://blog.widodh.nl/2013/06/**a-quick-note-on-running-**
>>> > > > >> cloudstack-with-rbd-on-ubuntu-**12-04/<
>>> > >
>>> http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/
>>> > > >
>>> > > > >>
>>> > > > >> I also have checked /var/log/cloud/agent/agent.log and I
>>> don't see
>>> > > > >> any error messages, except this error message which will
>>> show up
>>> > > > >> every time I restart the agent:
>>> > > > >>
>>> > > > >> 2013-07-12 15:22:47,454 ERROR
>>> > > [cloud.resource.**ServerResourceBase]
>>> > > > >> (main:null) Nics are not configured!
>>> > > > >> 2013-07-12 15:22:47,459 INFO
>>> > > [cloud.resource.**ServerResourceBase]
>>> > > > >> (main:null) Designating private to be nic eth0.5
>>> > > > >>
>>> > > > >> More logs can be found here: http://pastebin.com/yeNmCt7S
>>> > > > >>
>>> > > > >> I have configured the network bridges on the NIC interface
>>> as per
>>> > > > >> this instruction:
>>> > > > >>
>>> > > > >>
>>> http://cloudstack.apache.org/**docs/en-US/Apache_CloudStack/**
>>> > > > >> 4.1.0/html/Installation_Guide/**hypervisor-kvm-install-flow.**
>>> > > > >> html#hypervisor-host-install-**network<
>>> > >
>>> http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.1.0/html/Installation_Guide/hypervisor-kvm-install-flow.html#hypervisor-host-install-network
>>> > > >
>>> > > > >>
>>> > > > >> On the zone, I used advanced network configuration with
>>> just one
>>> > > > >> physical network for management, public and guest/private. I
>>> > > didn't
>>> > > > >> include storage, which by default the traffic will use the
>>> > > > >> management VLAN network.
>>> > > > >>
>>> > > > >> Please advise if there's anything else I might have been
>>> missing.
>>> > > > >>
>>> > > > >> Looking forward to your reply, thank you.
>>> > > > >>
>>> > > > >> Cheers.
>>> > > > >>
>>> > > > >>
>>> > > > >>
>>> > > > >> On Fri, Jul 12, 2013 at 2:56 PM, Prasanna Santhanam <
>>> > > tsp@apache.org
>>> > > > >> <ma...@apache.org>> wrote:
>>> > > > >>
>>> > > > >> Indeed, cloudstack will go through the allocation to
>>> startup
>>> > > the
>>> > > > >> system VMs too. So that process is failing to recognize
>>> the
>>> > > volume
>>> > > > >> (.qcow2) present on your NFS storage.
>>> > > > >>
>>> > > > >> Can you check if your cloudstack agent service is
>>> running with
>>> > > > >> the KVM
>>> > > > >> host? And it should've created the user cloud. $id
>>> cloud to
>>> > > check
>>> > > > >> if
>>> > > > >> there is a user.
>>> > > > >>
>>> > > > >> Did you see what's happening in the agent logs? These
>>> are
>>> > > under
>>> > > > >> /var/log/cloud/ on your host when the systemVMs are
>>> coming up.
>>> > > > >> If the
>>> > > > >> logs are not showing any useful information you can
>>> turn on
>>> > > debug
>>> > > > >> level for more verbosity.
>>> > > > >>
>>> > > > >> See here:
>>> https://cwiki.apache.org/**confluence/x/FgPMAQ<
>>> > > https://cwiki.apache.org/confluence/x/FgPMAQ>
>>> > > > >>
>>> > > > >> On Fri, Jul 12, 2013 at 02:40:35PM +0800, Indra Pramana
>>> wrote:
>>> > > > >> > Hi Prasanna,
>>> > > > >> >
>>> > > > >> > Good day to you, and thank you for your e-mail.
>>> > > > >> >
>>> > > > >> > Yes, when I export the NFS, I set the permission so
>>> that
>>> > > > >> normal user will
>>> > > > >> > be able to have read/write access to the files
>>> > > > >> (no_root_squash).
>>> > > > >> >
>>> > > > >> > I have tested and I can have read/write access from
>>> my KVM
>>> > > > >> hosts using
>>> > > > >> > normal user. BTW, there's no "cloud" user on the
>>> hosts, I
>>> > > > >> believe it's not
>>> > > > >> > created during cloudstack-agent installation?
>>> > > > >> >
>>> > > > >> > In any case, do you think the template issue and the
>>> > > storage
>>> > > > >> pool
>>> > > > >> > allocation issue might be related, or are they two
>>> > > different
>>> > > > >> problems
>>> > > > >> > altogether?
>>> > > > >> >
>>> > > > >> > Looking forward to your reply, thank you.
>>> > > > >> >
>>> > > > >> > Cheers.
>>> > > > >> >
>>> > > > >> >
>>> > > > >> >
>>> > > > >> > On Fri, Jul 12, 2013 at 2:26 PM, Prasanna Santhanam
>>> > > > >> <tsp@apache.org <ma...@apache.org>> wrote:
>>> > > > >> >
>>> > > > >> > > Can you access the file as user root? Or user
>>> cloud? The
>>> > > > >> cloudstack
>>> > > > >> > > agent on your KVM host runs as user cloud and the
>>> NFS
>>> > > > >> permissions
>>> > > > >> > > might be disallowing the volume (.qcow2) from being
>>> > > accessed.
>>> > > > >> > >
>>> > > > >> > > On Fri, Jul 12, 2013 at 02:16:41PM +0800, Indra
>>> Pramana
>>> > > > >> wrote:
>>> > > > >> > > > Hi Prasanna,
>>> > > > >> > > >
>>> > > > >> > > > Good day to you, and thank you for your e-mail.
>>> > > > >> > > >
>>> > > > >> > > > Yes, the file exists. I can access the file from
>>> the
>>> > > > >> management server
>>> > > > >> > > and
>>> > > > >> > > > the two hypervisors hosts if I mount manually.
>>> > > > >> > > >
>>> > > > >> > > > [root@cs-nas-01
>>> /mnt/vol1/sec-storage/**template/tmpl/1/3]#
>>> > > > >> ls
>>> > > > >> > > > -la
>>> > > > >> > > > total
>>> > > > >> > > > 1418787
>>> > > > >> > > > drwxr-xr-x 2 root wheel 4 Jul 11 20:21
>>> > > > >> > > > .
>>> > > > >> > > > drwxr-xr-x 3 root wheel 3 Jul 11 20:17
>>> > > > >> > > > ..
>>> > > > >> > > > -rw-r--r-- 1 root wheel 725811200 Jul 11 20:21
>>> > > > >> > > > 425b9e5a-fbc7-4637-a33a-**fe9d0ed4fa98.qcow2
>>> > > > >> > > > -rw-r--r-- 1 root wheel 295 Jul 11 20:21
>>> > > > >> > > > template.properties
>>> > > > >> > > > [root@cs-nas-01 /mnt/vol1/sec-storage/**
>>> > > > >> template/tmpl/1/3]#
>>> > > > >> > > > pwd
>>> > > > >> > > > /mnt/vol1/sec-storage/**template/tmpl/1/3
>>> > > > >> > > >
>>> > > > >> > > >
>>> > > > >> > > > Any advise?
>>> > > > >> > > >
>>> > > > >> > > > Looking forward to your reply, thank you.
>>> > > > >> > > >
>>> > > > >> > > > Cheers.
>>> > > > >> > > >
>>> > > > >> > > >
>>> > > > >> > > >
>>> > > > >> > > > On Fri, Jul 12, 2013 at 2:07 PM, Prasanna
>>> Santhanam
>>> > > > >> <tsp@apache.org <ma...@apache.org>>
>>> > > > >>
>>> > > > >> > > wrote:
>>> > > > >> > > >
>>> > > > >> > > > > Can you check whether there is a file at:
>>> > > > >> > > > >
>>> > > > >> nfs://
>>> 103.25.200.19/mnt/vol1/**sec-storage/template/tmpl/1/3/
>>> > > <http://103.25.200.19/mnt/vol1/sec-storage/template/tmpl/1/3/>
>>> > > > >> <
>>> > > http://103.25.200.19/mnt/**vol1/sec-storage/template/**tmpl/1/3/<
>>> > > http://103.25.200.19/mnt/vol1/sec-storage/template/tmpl/1/3/>
>>> > > > >> >
>>> > > > >>
>>> > > > >> > > > >
>>> > > > >> > > > > On Fri, Jul 12, 2013 at 01:59:34PM +0800, Indra
>>> > > Pramana
>>> > > > >> wrote:
>>> > > > >> > > > > > Hi Prasanna,
>>> > > > >> > > > > >
>>> > > > >> > > > > > Thanks for your e-mail.
>>> > > > >> > > > > >
>>> > > > >> > > > > > I have tried restarting the management
>>> server, and
>>> > > > >> the problem still
>>> > > > >> > > > > > persists. I even tried to re-do the
>>> installation
>>> > > and
>>> > > > >> configuration
>>> > > > >> > > again
>>> > > > >> > > > > > from scratch last night, but the problem
>>> still
>>> > > there.
>>> > > > >> > > > > >
>>> > > > >> > > > > > I also noted that on the beginning of the
>>> logs, I
>>> > > > >> found some error
>>> > > > >> > > > > messages
>>> > > > >> > > > > > saying that the template cannot be
>>> downloaded to
>>> > > the
>>> > > > >> pool. See this
>>> > > > >> > > logs:
>>> > > > >> > > > > >
>>> > > > >> > > > > > http://pastebin.com/BY1AVJ08
>>> > > > >> > > > > >
>>> > > > >> > > > > > It says it failed because cannot get volume
>>> from
>>> > > the
>>> > > > >> pool. Could it
>>> > > > >> > > be
>>> > > > >> > > > > > related, i.e. the absence of the template
>>> caused
>>> > > the
>>> > > > >> system VMs
>>> > > > >> > > cannot be
>>> > > > >> > > > > > created and started?
>>> > > > >> > > > > >
>>> > > > >> > > > > > I have ensured that I downloaded the system
>>> VM
>>> > > > >> template using
>>> > > > >> > > > > > cloud-install-sys-tmplt and verified that the
>>> > > > >> template is already
>>> > > > >> > > there
>>> > > > >> > > > > in
>>> > > > >> > > > > > the secondary storage server.
>>> > > > >> > > > > >
>>> > > > >> > > > > > Any advice is appreciated.
>>> > > > >> > > > > >
>>> > > > >> > > > > > Looking forward to your reply, thank you.
>>> > > > >> > > > > >
>>> > > > >> > > > > > Cheers.
>>> > > > >> > > > > >
>>> > > > >> > > > > >
>>> > > > >> > > > > >
>>> > > > >> > > > > > On Fri, Jul 12, 2013 at 1:21 PM, Prasanna
>>> Santhanam
>>> > > > >> <tsp@apache.org <ma...@apache.org>>
>>> > > > >>
>>> > > > >> > > > > wrote:
>>> > > > >> > > > > >
>>> > > > >> > > > > > > It looks like a previous attempt to start
>>> the
>>> > > > >> systemVMs has failed
>>> > > > >> > > > > > > putting the nfs storage in the avoid set.
>>> Did you
>>> > > > >> try restarting
>>> > > > >> > > your
>>> > > > >> > > > > > > management server?
>>> > > > >> > > > > > >
>>> > > > >> > > > > > > This line leads me to the above mentioned:
>>> > > > >> > > > > > > 2013-07-12 13:10:48,236 DEBUG
>>> > > > >> > > > > > >
>>> > > [storage.allocator.**AbstractStoragePoolAllocator]
>>> > > > >> > > (secstorage-1:null)
>>> > > > >> > > > > > > StoragePool is in avoid set, skipping this
>>> pool
>>> > > > >> > > > > > >
>>> > > > >> > > > > > >
>>> > > > >> > > > > > > On Fri, Jul 12, 2013 at 01:16:53PM +0800,
>>> Indra
>>> > > > >> Pramana wrote:
>>> > > > >> > > > > > > > Dear Wido and all,
>>> > > > >> > > > > > > >
>>> > > > >> > > > > > > > I have managed to get the hosts, primary
>>> and
>>> > > > >> secondary storage
>>> > > > >> > > > > running.
>>> > > > >> > > > > > > >
>>> > > > >> > > > > > > > - 2 KVM hypervisor hosts
>>> > > > >> > > > > > > > - One RBD primary storage
>>> > > > >> > > > > > > > - One NFS primary storage (for system
>>> VMs,
>>> > > since
>>> > > > >> I understand
>>> > > > >> > > that
>>> > > > >> > > > > system
>>> > > > >> > > > > > > > VMs cannot use RBD)
>>> > > > >> > > > > > > > - One NFS secondary storage
>>> > > > >> > > > > > > >
>>> > > > >> > > > > > > > However, now I am having problem with the
>>> > > system
>>> > > > >> VMs: CPVM and
>>> > > > >> > > SSVM,
>>> > > > >> > > > > > > unable
>>> > > > >> > > > > > > > to start.
>>> > > > >> > > > > > > >
>>> > > > >> > > > > > > > Excerpt from management-server.log file
>>> is
>>> > > here:
>>> > > > >> > > > > > > > http://pastebin.com/ENkpCALY
>>> > > > >> > > > > > > >
>>> > > > >> > > > > > > > It seems that the VMs were not able to be
>>> > > created
>>> > > > >> because unable
>>> > > > >> > > to
>>> > > > >> > > > > find
>>> > > > >> > > > > > > > suitable StoragePools.
>>> > > > >> > > > > > > >
>>> > > > >> > > > > > > > I understand that system VMs will be
>>> using the
>>> > > > >> NFS primary
>>> > > > >> > > storage
>>> > > > >> > > > > > > instead
>>> > > > >> > > > > > > > of RBD, so I have confirmed that I am
>>> able to
>>> > > > >> mount the primary
>>> > > > >> > > > > storage
>>> > > > >> > > > > > > via
>>> > > > >> > > > > > > > NFS and have read and write access, from
>>> both
>>> > > the
>>> > > > >> hypervisor and
>>> > > > >> > > the
>>> > > > >> > > > > > > > management server.
>>> > > > >> > > > > > > >
>>> > > > >> > > > > > > > Any advise how can I resolve the problem
>>> to
>>> > > make
>>> > > > >> both the system
>>> > > > >> > > VMs
>>> > > > >> > > > > > > > created and started?
>>> > > > >> > > > > > > >
>>> > > > >> > > > > > > > Looking forward to your reply, thank you.
>>> > > > >> > > > > > > >
>>> > > > >> > > > > > > > Cheers.
>>> > > > >> > > > > > > >
>>> > > > >> > > > > > > >
>>> > > > >> > > > > > > > On Fri, Jul 12, 2013 at 9:43 AM, Indra
>>> Pramana
>>> > > > >> <indra@sg.or.id <ma...@sg.or.id>>
>>> > > > >>
>>> > > > >> > > > > wrote:
>>> > > > >> > > > > > > >
>>> > > > >> > > > > > > > > Hi Wido,
>>> > > > >> > > > > > > > >
>>> > > > >> > > > > > > > > Thanks for the advice, I'm now able to
>>> add
>>> > > the
>>> > > > >> RBD pool as
>>> > > > >> > > primary
>>> > > > >> > > > > > > storage.
>>> > > > >> > > > > > > > >
>>> > > > >> > > > > > > > > Many thanks! :)
>>> > > > >> > > > > > > > >
>>> > > > >> > > > > > > > > Cheers.
>>> > > > >> > > > > > > > >
>>> > > > >> > > > > > > > >
>>> > > > >> > > > > > > > > On Thursday, July 11, 2013, Wido den
>>> > > Hollander
>>> > > > >> wrote:
>>> > > > >> > > > > > > > >
>>> > > > >> > > > > > > > >> Hi,
>>> > > > >> > > > > > > > >>
>>> > > > >> > > > > > > > >> On 07/10/2013 03:42 PM, Chip Childers
>>> wrote:
>>> > > > >> > > > > > > > >>
>>> > > > >> > > > > > > > >>> Cc'ing Wido, our resident Ceph
>>> expert. ;-)
>>> > > > >> > > > > > > > >>>
>>> > > > >> > > > > > > > >>>
>>> > > > >> > > > > > > > >> Hehe ;)
>>> > > > >> > > > > > > > >>
>>> > > > >> > > > > > > > >> On Wed, Jul 10, 2013 at 05:45:25PM
>>> +0800,
>>> > > > >> Indra Pramana
>>> > > > >> > > wrote:
>>> > > > >> > > > > > > > >>>
>>> > > > >> > > > > > > > >>>> Dear all,
>>> > > > >> > > > > > > > >>>>
>>> > > > >> > > > > > > > >>>> I am installing CloudStack 4.1.0
>>> (upgraded
>>> > > > >> from 4.0.2) and I
>>> > > > >> > > > > also
>>> > > > >> > > > > > > have a
>>> > > > >> > > > > > > > >>>> Ceph cluster running. However, I am
>>> having
>>> > > > >> issues in adding
>>> > > > >> > > the
>>> > > > >> > > > > RBD
>>> > > > >> > > > > > > as
>>> > > > >> > > > > > > > >>>> primary storage. Tried to follow the
>>> > > > >> instruction here, but
>>> > > > >> > > > > unable to
>>> > > > >> > > > > > > > >>>> make
>>> > > > >> > > > > > > > >>>> it work:
>>> > > > >> > > > > > > > >>>>
>>> > > > >> > > > > > > > >>>>
>>> > > > >> http://ceph.com/docs/master/****rbd/rbd-cloudstack/<
>>> > > http://ceph.com/docs/master/**rbd/rbd-cloudstack/>
>>> > > > >> <
>>> > > > >> > > > > > >
>>> > > http://ceph.com/docs/master/**rbd/rbd-cloudstack/<
>>> > > http://ceph.com/docs/master/rbd/rbd-cloudstack/>
>>> > > > >> >
>>> > > > >> > > > > > > > >>>>
>>> > > > >> > > > > > > > >>>> I have setup a pool on the Ceph
>>> cluster.
>>> > > The
>>> > > > >> status of the
>>> > > > >> > > > > cluster
>>> > > > >> > > > > > > is
>>> > > > >> > > > > > > > >>>> healthy. Since I am using Ubuntu
>>> 12.04.2
>>> > > LTS
>>> > > > >> (Precise) for
>>> > > > >> > > the
>>> > > > >> > > > > > > > >>>> hypervisors,
>>> > > > >> > > > > > > > >>>> I also have compiled libvirt
>>> manually to
>>> > > > >> ensure that the
>>> > > > >> > > version
>>> > > > >> > > > > > > 0.9.13
>>> > > > >> > > > > > > > >>>> is
>>> > > > >> > > > > > > > >>>> installed (previously it's 0.9.8).
>>> > > > >> > > > > > > > >>>>
>>> > > > >> > > > > > > > >>>>
>>> > > > >> > > > > > > > >> You can also use the Ubuntu Cloud
>>> Archive, I
>>> > > > >> still need to
>>> > > > >> > > get the
>>> > > > >> > > > > > > docs
>>> > > > >> > > > > > > > >> updated for that.
>>> > > > >> > > > > > > > >>
>>> > > > >> > > > > > > > >> I described the process in a blogpost:
>>> > > > >> > > > > > > http://blog.widodh.nl/2013/06/****<
>>> > > http://blog.widodh.nl/2013/06/**>
>>> > > > >> > > > > > > > >>
>>> > > > >> > >
>>> > > > >>
>>> a-quick-note-on-running-****cloudstack-with-rbd-on-ubuntu-**
>>> > > > >> **12-04/<
>>> > > > >> > > > > > >
>>> > > > >> > > > >
>>> > > > >> > >
>>> > > > >>
>>> http://blog.widodh.nl/2013/06/**a-quick-note-on-running-**
>>> > > > >> cloudstack-with-rbd-on-ubuntu-**12-04/<
>>> > >
>>> http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/
>>> > > >
>>> > > > >> > > > > > > >
>>> > > > >> > > > > > > > >>
>>> > > > >> > > > > > > > >> indra@hv-kvm-01:~/rbd$ ceph
>>> > > > >> > > > > > > > >>>> ceph> health
>>> > > > >> > > > > > > > >>>> HEALTH_OK
>>> > > > >> > > > > > > > >>>>
>>> > > > >> > > > > > > > >>>> indra@hv-kvm-01:~$ ceph osd lspools
>>> > > > >> > > > > > > > >>>> 0 data,1 metadata,2 rbd,3 sc1,
>>> > > > >> > > > > > > > >>>>
>>> > > > >> > > > > > > > >>>> root@hv-kvm-01:/home/indra#
>>> libvirtd
>>> > > > >> --version
>>> > > > >> > > > > > > > >>>> libvirtd (libvirt) 0.9.13
>>> > > > >> > > > > > > > >>>>
>>> > > > >> > > > > > > > >>>> I tried to add Primary Storage into
>>> the
>>> > > > >> Cloudstack zone
>>> > > > >> > > which I
>>> > > > >> > > > > have
>>> > > > >> > > > > > > > >>>> created:
>>> > > > >> > > > > > > > >>>>
>>> > > > >> > > > > > > > >>>> Add Primary Storage:
>>> > > > >> > > > > > > > >>>>
>>> > > > >> > > > > > > > >>>> Zone: my zone name
>>> > > > >> > > > > > > > >>>> Pod: my pod name
>>> > > > >> > > > > > > > >>>> Cluster: my cluster name
>>> > > > >> > > > > > > > >>>> Name: ceph-rbd-pri-storage
>>> > > > >> > > > > > > > >>>> Protocol: RBD
>>> > > > >> > > > > > > > >>>> RADOS Monitor: my first Ceph
>>> monitor IP
>>> > > > >> address
>>> > > > >> > > > > > > > >>>> RADOS Pool: sc1 (the pool name on
>>> Ceph
>>> > > > >> cluster)
>>> > > > >> > > > > > > > >>>> RADOS User: client.admin
>>> > > > >> > > > > > > > >>>> RADOS Secret:
>>> > > > >> /etc/ceph/ceph.client.admin.****keyring (keyring
>>> > > > >> > > > > file
>>> > > > >> > > > > > > > >>>> location)
>>> > > > >> > > > > > > > >>>>
>>> > > > >> > > > > > > > >>>
>>> > > > >> > > > > > > > >> This is your problem. That shouldn't
>>> be the
>>> > > > >> location of the
>>> > > > >> > > file,
>>> > > > >> > > > > but
>>> > > > >> > > > > > > it
>>> > > > >> > > > > > > > >> should be the secret, which is a
>>> base64
>>> > > > >> encoded string.
>>> > > > >> > > > > > > > >>
>>> > > > >> > > > > > > > >> $ ceph auth list
>>> > > > >> > > > > > > > >>
>>> > > > >> > > > > > > > >> That should tell you what the secret
>>> is.
>>> > > > >> > > > > > > > >>
>>> > > > >> > > > > > > > >> Storage Tags: rbd
>>> > > > >> > > > > > > > >>
>>> > > > >> > > > > > > > >> This is the error message when I
>>> tried to
>>> > > add
>>> > > > >> the primary
>>> > > > >> > > storage
>>> > > > >> > > > > by
>>> > > > >> > > > > > > > >> clicking OK:
>>> > > > >> > > > > > > > >>
>>> > > > >> > > > > > > > >> DB Exception on:
>>> > > > >> > > com.mysql.jdbc.****JDBC4PreparedStatement@4b2eb56
>>> > > > >> > > > > **:
>>> > > > >> > > > > > > > >> INSERT INTO
>>> > > > >> > > > > > > > >> storage_pool (storage_pool.id
>>> > > > >> <http://storage_pool.id>, storage_pool.name
>>> > > > >> <http://storage_pool.name>,
>>> > > > >>
>>> > > > >> > > > > storage_pool.uuid,
>>> > > > >> > > > > > > > >> storage_pool.pool_type,
>>> > > storage_pool.created,
>>> > > > >> > > > > > > storage_pool.update_time,
>>> > > > >> > > > > > > > >> storage_pool.data_center_id,
>>> > > > >> storage_pool.pod_id,
>>> > > > >> > > > > > > > >> storage_pool.available_bytes,
>>> > > > >> storage_pool.capacity_bytes,
>>> > > > >> > > > > > > > >> storage_pool.status,
>>> storage_pool.scope,
>>> > > > >> > > > > > > storage_pool.storage_provider_****
>>> > > > >> > > > > > > > >> id,
>>> > > > >> > > > > > > > >> storage_pool.host_address,
>>> > > storage_pool.path,
>>> > > > >> > > storage_pool.port,
>>> > > > >> > > > > > > > >> storage_pool.user_info,
>>> > > > >> storage_pool.cluster_id) VALUES (217,
>>> > > > >> > > > > > > > >> _binary'ceph-rbd-pri-storage',
>>> > > > >> > > > > > > > >>
>>> > > > >> _binary'a226c9a1-da78-3f3a-****b5ac-e18b925c9634',
>>> 'RBD',
>>> > > > >> > > > > '2013-07-10
>>> > > > >> > > > > > > > >> 09:08:28', null, 2, 2, 0, 0, 'Up',
>>> null,
>>> > > null,
>>> > > > >> null,
>>> > > > >> > > _binary'ceph/
>>> > > > >> > > > > > > > >> ceph.client.admin.keyring@10.
>>> > > ****237.11.2/sc1<
>>> > > > >> > > > > > > http://ceph.client.admin.**
>>> > > keyring@10.237.11.2/sc1<
>>> http://ceph.client.admin.keyring@10.237.11.2/sc1>
>>> > > > >> >',
>>> > > > >> > > > > > > > >> 6789, null, 2)
>>> > > > >> > > > > > > > >>
>>> > > > >> > > > > > > > >> On the management-server.log file:
>>> > > > >> > > > > > > > >>
>>> > > > >> > > > > > > > >> 2013-07-10 17:08:28,845 DEBUG
>>> > > > >> [cloud.api.ApiServlet]
>>> > > > >> > > > > > > > >> (catalina-exec-2:null)
>>> > > > >> > > > > > > > >> ===START=== 192.168.0.100 -- GET
>>> > > > >> > > > > > > > >>
>>> > > > >>
>>> > > command=createStoragePool&****zoneid=c116950e-e4ae-4f23-****a7e7-
>>> > > > >> > > > > > > > >>
>>> > > > >>
>>> 74a75c4ee638&podId=a748b063-****3a83-4175-a0e9-de39118fe5ce&**
>>> > > > >> > > > > > > > >>
>>> > > > >>
>>> > > clusterid=1f87eb09-324d-4d49-****83c2-88d84d7a15df&name=ceph-****
>>> > > > >> > > > > > > > >>
>>> > > > >>
>>> rbd-pri-storage&url=rbd%3A%2F%****2Fclient.admin%3A_etc%2Fc
>>> > > > >> > > > > > > > >>
>>> > > > >>
>>> eph%2Fceph.client.admin.****keyring%4010.237.11.2%2Fsc1&**
>>> > > > >> > > > > > > > >>
>>> > > > >> > > > >
>>> > > > >> tags=rbd&response=json&****sessionkey=****
>>> > > > >> rDRfWpqeKfQKbKZtHr398ULV%2F8k%****
>>> > > > >> > > > > > > > >> 3D&_=1373447307839
>>> > > > >> > > > > > > > >> 2013-07-10 17:08:28,862 DEBUG
>>> > > > >> > > [cloud.storage.****StorageManagerImpl]
>>> > > > >> > > > > > > > >> (catalina-exec-2:null) createPool
>>> Params @
>>> > > > >> scheme - rbd
>>> > > > >> > > > > storageHost -
>>> > > > >> > > > > > > null
>>> > > > >> > > > > > > > >> hostPath - /ceph/ceph.client
>>> > > > >> > > > > > > > >> .admin.keyring@10.237.11.2/sc1
>>> > > > >> <http://admin.keyring@10.237.**11.2/sc1<
>>> > > http://admin.keyring@10.237.11.2/sc1>>
>>> > > > >> port - -1
>>> > > > >>
>>> > > > >> > > > > > > > >> 2013-07-10 17:08:28,918 DEBUG
>>> > > > >> > > [cloud.storage.****StorageManagerImpl]
>>> > > > >> > > > > > > > >> (catalina-exec-2:null) In createPool
>>> Setting
>>> > > > >> poolId - 217
>>> > > > >> > > uuid -
>>> > > > >> > > > > > > > >>
>>> a226c9a1-da78-3f3a-b5ac-****e18b925c9634 z
>>> > > > >> > > > > > > > >> oneId - 2 podId - 2 poolName -
>>> > > > >> ceph-rbd-pri-storage
>>> > > > >> > > > > > > > >> 2013-07-10 17:08:28,921 DEBUG
>>> > > > >> [db.Transaction.Transaction]
>>> > > > >> > > > > > > > >> (catalina-exec-2:null) Rolling back
>>> the
>>> > > > >> transaction: Time = 3
>>> > > > >> > > > > Name =
>>> > > > >> > > > > > > > >> persist; called by
>>> -Transaction.rollbac
>>> > > > >> > > > > > > > >>
>>> > > > >>
>>> k:890-Transaction.removeUpTo:****833-Transaction.close:657-**
>>> > > > >> > > > > > > > >>
>>> > > > >> TransactionContextBuilder.****interceptException:63-**
>>> > > > >> > > > > > > > >>
>>> > > > >> > > > >
>>> > > > >>
>>> > > ComponentInstantiationPostProc****essor$InterceptorDispatcher.**
>>> > > > >> **interce
>>> > > > >> > > > > > > > >> pt:133-StorageManagerImpl.****
>>> > > > >> createPool:1378-**
>>> > > > >> > > > > > > > >>
>>> > > > >>
>>> StorageManagerImpl.createPool:****147-CreateStoragePoolCmd.**
>>> > > > >> > > > > > > > >>
>>> > > > >> execute:123-ApiDispatcher.****dispatch:162-ApiServer.**
>>> > > > >> > > > > > > > >> queueCommand:505-ApiSe
>>> > > > >> > > > > > > > >>
>>> > > > >> rver.handleRequest:355-****ApiServlet.processRequest:302
>>> > > > >> > > > > > > > >> 2013-07-10 17:08:28,923 ERROR
>>> > > > >> [cloud.api.ApiServer]
>>> > > > >> > > > > > > (catalina-exec-2:null)
>>> > > > >> > > > > > > > >> unhandled exception executing api
>>> command:
>>> > > > >> createStoragePool
>>> > > > >> > > > > > > > >>
>>> > > > >> com.cloud.utils.exception.****CloudRuntimeException: DB
>>> > > > >> > > Exception
>>> > > > >> > > > > on:
>>> > > > >> > > > > > > > >>
>>> > > > >> com.mysql.jdbc.****JDBC4PreparedStatement@4b2eb56****:
>>> INSERT
>>> > > > >> > > INTO
>>> > > > >> > > > > > > > >> storage_pool (
>>> > > > >> > > > > > > > >> storage_pool.id <
>>> http://storage_pool.id>,
>>> > > > >>
>>> > > > >> storage_pool
>>> > > > >> > > > > > > > >> .name, storage_pool.uuid,
>>> > > > >> storage_pool.pool_type,
>>> > > > >> > > > > > > storage_pool.created,
>>> > > > >> > > > > > > > >> storage_pool.update_time,
>>> > > > >> storage_pool.data_center_id,
>>> > > > >> > > > > > > > >> storage_pool.pod_id,
>>> > > > >> > > > > > > > >> storage_pool.availab
>>> > > > >> > > > > > > > >> le_bytes, storage_pool.capacity_bytes,
>>> > > > >> storage_pool.status,
>>> > > > >> > > > > > > > >> storage_pool.scope,
>>> > > > >> storage_pool.storage_provider_****id,
>>> > > > >> > > > > > > > >> storage_pool.host_address,
>>> > > storage_pool.path,
>>> > > > >> storage_
>>> > > > >> > > > > > > > >> pool.port, storage_pool.user_info,
>>> > > > >> storage_pool.cluster_id)
>>> > > > >> > > VALUES
>>> > > > >> > > > > > > (217,
>>> > > > >> > > > > > > > >> _binary'ceph-rbd-pri-storage',
>>> > > > >> > > > > > > > >>
>>> > > > >> _binary'a226c9a1-da78-3f3a-****b5ac-e18b925c9634',
>>> 'RBD',
>>> > > > >> > > '2013-07-1
>>> > > > >> > > > > > > > >> 0 09:08:28', null, 2, 2, 0, 0, 'Up',
>>> null,
>>> > > > >> null, null,
>>> > > > >> > > > > _binary'ceph/
>>> > > > >> > > > > > > > >> ceph.client.admin.keyring@10.
>>> > > ****237.11.2/sc1<
>>> > > > >> > > > > > > http://ceph.client.admin.**
>>> > > keyring@10.237.11.2/sc1<
>>> http://ceph.client.admin.keyring@10.237.11.2/sc1>
>>> > > > >> >',
>>> > > > >> > > > > > > > >> 6789, null, 2)
>>> > > > >> > > > > > > > >> at
>>> > > > >> > > > > > > > >>
>>> > > > >> > > > > > >
>>> > > > >> > >
>>> > > > >> com.cloud.utils.db.****GenericDaoBase.persist(****
>>> > > > >> GenericDaoBase.java:1342)
>>> > > > >> > > > > > > > >> at
>>> > > > >> > > > > > > > >>
>>> > > > >> com.cloud.storage.dao.****StoragePoolDaoImpl.persist(**
>>> > > > >> > > > > > > > >> StoragePoolDaoImpl.java:232)
>>> > > > >> > > > > > > > >> at
>>> > > > >> > > > > > > > >>
>>> > > > >>
>>> > > com.cloud.utils.component.****ComponentInstantiationPostProc****
>>> > > > >> > > > > > > > >>
>>> > > > >> > > > > > >
>>> > > > >> > > > >
>>> > > > >> > >
>>> > > > >> essor$InterceptorDispatcher.****intercept(****
>>> > > > >> ComponentInstantiationPostProc***
>>> > > > >> > > > > > > > >> *es
>>> > > > >> > > > > > > > >>
>>> > > > >> > > > > > > > >>
>>> > > > >> > > > > > >
>>> > > > >> > > > > > > --
>>> > > > >> > > > > > > Prasanna.,
>>> > > > >> > > > > > >
>>> > > > >> > > > > > > ------------------------
>>> > > > >> > > > > > > Powered by BigRock.com
>>> > > > >> > > > > > >
>>> > > > >> > > > > > >
>>> > > > >> > > > >
>>> > > > >> > > > > --
>>> > > > >> > > > > Prasanna.,
>>> > > > >> > > > >
>>> > > > >> > > > > ------------------------
>>> > > > >> > > > > Powered by BigRock.com
>>> > > > >> > > > >
>>> > > > >> > > > >
>>> > > > >> > >
>>> > > > >> > > --
>>> > > > >> > > Prasanna.,
>>> > > > >> > >
>>> > > > >> > > ------------------------
>>> > > > >> > > Powered by BigRock.com
>>> > > > >> > >
>>> > > > >> > >
>>> > > > >>
>>> > > > >> --
>>> > > > >> Prasanna.,
>>> > > > >>
>>> > > > >> ------------------------
>>> > > > >> Powered by BigRock.com
>>> > > > >>
>>> > > > >>
>>> > > > >>
>>> > > > >>
>>> > > > >
>>> > >
>>> > > --
>>> > > Prasanna.,
>>> > >
>>> > > ------------------------
>>> > > Powered by BigRock.com
>>> > >
>>> > >
>>>
>>> --
>>> Prasanna.,
>>>
>>> ------------------------
>>> Powered by BigRock.com
>>>
>>>
>>
>
Re: Problem in adding Ceph RBD as primary storage for CloudStack 4.1.0
Posted by Indra Pramana <in...@sg.or.id>.
Hi Prasanna and all,
I managed to create a VM instance using the NFS primary storage (not using
the RBD primary storage due to the bug we discussed on another thread).
However, there is no public IP being assigned to the VM instance (unlike
the system VMs which are getting public IPs). How do I assign public IP to
the instance?
===
NIC 1 (Default)
Network Name Indra
IP Address 10.19.8.186
Type Isolated
Gateway 10.19.0.1
Netmask 255.255.240.0
Is Default Yes
===
The network was created under the
"DefaultIsolatedNetworkOfferingWithSourceNatService" network offering. I am
not able to create the network using different network offerings, despite
the other offerings are available under Service Offerings > Network
Offerings and the state of those network offerings are enabled. Is it
related to the way how I configured the network bridges and configuration
earlier during zone creation, anything that I might have missed out?
I also tried to "view console" to the VM (via the web GUI) and not able to
do so. Earlier I got the "Unable to start console session as connection is
refused by the machine you are accessing" error message, which I have fixed
by changing the /etc/libvirt/qemu.conf to enable this line:
vnc_listen = "0.0.0.0"
because initially the VNC only listens on localhost (127.0.0.1) by default.
Now I am no longer getting the error message, but the console window still
failed to load after some time, with Firefox browser error message:
===
The connection was reset
The connection to the server was reset while the page was loading.
===
How can I further troubleshoot on this? I tried to SSH into the console
proxy VM and can't find anything useful on the logs there.
Looking forward to your reply, thank you.
Cheers.
On Mon, Jul 15, 2013 at 12:38 PM, Indra Pramana <in...@sg.or.id> wrote:
> Hi Prasanna,
>
> On Mon, Jul 15, 2013 at 12:20 PM, Prasanna Santhanam <ts...@apache.org>wrote:
>
>> On Sat, Jul 13, 2013 at 02:25:35PM +0800, Indra Pramana wrote:
>> > Hi Prasanna,
>> >
>> > Good day to you, and thank you for your e-mail.
>> >
>> Were you able to get beyond this error?
>>
>
> Yes. I managed to get around the network bridges issue by setting up this
> on my KVM host:
>
> ===
> auto eth0.5
> iface eth0.5 inet static
> address X.X.X.22
> netmask 255.255.255.240
> network X.X.X.16
> broadcast X.X.X.31
> gateway X.X.X.17
> # dns-* options are implemented by the resolvconf package, if
> installed
> dns-nameservers 8.8.8.8 8.8.4.4
> dns-search xxx.com
>
> # Public network
> auto cloudbr0
> iface cloudbr0 inet manual
> bridge_ports eth1.201
> bridge_fd 5
> bridge_stp off
> bridge_maxwait 1
>
> # Guest network
> auto cloudbr1
> iface cloudbr1 inet manual
> bridge_ports eth1.41
> bridge_fd 5
> bridge_stp off
> bridge_maxwait 1
>
> # Management network
> auto cloudbr2
> iface cloudbr2 inet manual
> bridge_ports eth0.6
> bridge_fd 5
> bridge_stp off
> bridge_maxwait 1
>
> # The 10G network interface
> auto eth2
> iface eth2 inet static
> address 10.237.11.22
> netmask 255.255.255.0
> network 10.237.11.0
> broadcast 10.237.11.255
> ===
>
> On the Cloudstack GUI, I created 3 physical networks:
>
> cloudbr0 - public network
> cloudbr1 - guest network
> cloudbr2 - management (and storage) network
>
> However, when I tried to setup the public network using VLAN 201, it
> failed because CloudStack tried to create a new bridge (i.e. breth1-201)
> instead of using the existing bridge I have set up for public network
> (cloudbr mapped to eth1.201). So I set up the public network using a
> different VLAN (VLAN 21) and change the settings on my router's gateway to
> use VLAN 21 instead of VLAN 201.
>
> ===
> root@hv-kvm-02:~# brctl show
> bridge name bridge id STP enabled interfaces
> breth1-21 8000.002590c29d7d no eth1.21
> vnet2
> vnet5
> cloud0 8000.fe00a9fe0165 no vnet0
> vnet3
> cloudbr0 8000.002590c29d7d no eth1.201
> cloudbr1 8000.002590c29d7d no eth1.41
> cloudbr2 8000.002590c29d7c no eth0.6
> vnet1
> vnet4
> vnet6
> virbr0 8000.000000000000 yes
> ===
>
> The above setup works, but I am not too sure whether this is the correct
> way on how to do it. Can you advise?
>
> Looking forward to your reply, thank you.
>
> Cheers.
>
>
>
>
>
>> > Yesterday I troubleshooted further and found out that the mapping of the
>> > bridge is incorrect. For example, eth1 of the SSVM, which is supposed
>> to be
>> > private network interface for communication with management server, is
>> > being mapped to cloudbr0 bridge which is used for public. While eth2 of
>> the
>> > SSVM, which is supposed to be public, is being mapped to cloudbr1 bridge
>> > which is supposed for guest traffic. I checked with brctl show on the
>> KVM
>> > host and traced the MAC address of the SSVM's eth interfaces.
>> >
>> > I have tried different type of configuration on the KVM hosts'
>> > /etc/network/interfaces (I am using Ubuntu), different ways on how to
>> > configure the advanced network zone (physical network interface and KVM
>> > label for each traffic), but I am still not able to get it working.
>> >
>> > Can please advise what is the best way to set-up the network bridges?
>>
>> Use brctl to add bridges. That's how I've done it.
>>
>> >
>> > My /etc/network/interfaces on KVM host:
>> >
>> > http://pastebin.com/nx8xJ1L2
>> >
>> > I used 3 physical NICs:
>> >
>> > eth0 --> for management and secondary storage traffic
>> > eth1 --> for public and guest traffic
>> > eth2 --> for primary storage traffic to the Ceph RBD (separate NIC not
>> > configured by CloudStack)
>> >
>> > On the Cloudstack GUI, while creating the zone, I used 3 physical
>> networks:
>> >
>> > eth0.6 --> Management traffic (KVM label: eth0.6)
>> > cloudbr0 --> Public traffic (KVM label: cloudbr0)
>> > cloudbr1 --> Guest traffic (KVM label: cloudbr1)
>> >
>> Is eth0.6 on a tagged VLAN?
>>
>> > I didn't specify storage traffic since I presume it will use the
>> management
>> > VLAN.
>>
>> That's correct.
>>
>> >
>> > However, I always failed to add the KVM host using the above
>> configuration.
>> > I tried to run cloudstack-setup-agent manually on the KVM host and
>> getting
>> > this error message:
>> >
>> > root@hv-kvm-02:/etc/cloudstack/agent# cloudstack-setup-agent -m
>> x.x.x.18 -z
>> > 3 -p 3 -c 3 -g 00dd5dba-7419-3689-bc51-1671035c0d8f -a --pubNic=cloudbr0
>> > --prvNic=eth0.6 --guestNic=cloudbr1
>> > Starting to configure your system:
>> > Configure Apparmor ... [OK]
>> > Configure Network ... [Failed]
>> > eth0.6 is not a bridge
>> > Try to restore your system:
>> > Restore Apparmor ... [OK]
>> > Restore Network ... [OK]
>> >
>> > When I tried to create another bridge called cloudbr2 with bridge_ports
>> to
>> > eth0.6, the KVM host's network didn't work.
>> >
>> > Can advise what is the best practice on configuring the network bridges
>> on
>> > the advanced network zone setup?
>> >
>> > Looking forward to your reply, thank you.
>> >
>> > Cheers.
>> >
>> >
>> >
>> > On Sat, Jul 13, 2013 at 12:28 PM, Prasanna Santhanam <ts...@apache.org>
>> wrote:
>> >
>> > > See Inline,
>> > >
>> > > On Fri, Jul 12, 2013 at 05:12:25PM +0800, Indra Pramana wrote:
>> > > > Hi Wido,
>> > > >
>> > > > Noted, can't wait for 4.2 to be released. :)
>> > > >
>> > > > Dear Prasanna, Wido and all,
>> > > >
>> > > > I just realised that while the system VMs are running, they are
>> still not
>> > > > accessible through the public IPs assigned to them. I have been
>> waiting
>> > > for
>> > > > the SSVM to download the default CentOS template and it doesn't
>> appear on
>> > > > the template list.
>> > > >
>> > > > I tried to SSH into the SSVM via the local link address from the KVM
>> > > host,
>> > > > and running the health check /usr/local/cloud/systemvm/ssvm-check.sh
>> > > shows
>> > > > that the VM cannot reach anywhere. It cannot reach the public DNS
>> server
>> > > (I
>> > > > used Google's 8.8.8.8), cannot reach the management server, and
>> cannot
>> > > even
>> > > > reach the public IP gateway.
>> > >
>> > > Ok - we're not there yet. But there's a few troubleshooting steps you
>> > > can try:
>> > >
>> > > 1. Which of the checks fail apart from the DNS ping?
>> > > 2. Are you able to have those tests pass from the KVM host itself?
>> > > 3. Can you paste the output of $route -n on your SSVM and on your
>> > > host?
>> > >
>> > > >
>> > > > Is it due to misconfiguration of the KVM network bridges? How can I
>> see
>> > > the
>> > > > mapping between the NIC interfaces of the SSVM (eth0, eth1, eth2 and
>> > > eth3)
>> > > > and the actual physical NIC interfaces on the KVM hosts (eth0) and
>> the
>> > > > network bridges (cloudbr0, cloudbr1)? Any logs I can verify to
>> ensure
>> > > that
>> > > > the VLAN and network bridging is working?
>> > > >
>> > >
>> > > You can see this from the dumpxml.
>> > >
>> > > $virsh list
>> > > (gives you all the domains. find the ssvm for example)
>> > >
>> > > $virsh dumpxml my-domain > mydomain.xml
>> > >
>> > > > Appreciate any advice.
>> > > >
>> > > > Thank you.
>> > > >
>> > > >
>> > > >
>> > > > On Fri, Jul 12, 2013 at 4:19 PM, Wido den Hollander <wido@widodh.nl
>> >
>> > > wrote:
>> > > >
>> > > > > On 07/12/2013 10:14 AM, Indra Pramana wrote:
>> > > > >
>> > > > >> Hi Prasanna,
>> > > > >>
>> > > > >> I managed to fix the problem, thanks for your advice to turn the
>> agent
>> > > > >> log level to debug:
>> > > > >>
>> > > > >> https://cwiki.apache.org/**confluence/display/CLOUDSTACK/**
>> > > > >> KVM+agent+debug<
>> > >
>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug>
>> > > > >>
>> > > > >> From the log, I found out that the agent on the KVM host tried
>> to NFS
>> > > > >> mount directly to 103.25.200.19:/mnt/vol1/sec-**
>> > > > >> storage/template/tmpl/1/3,
>> > > > >> which was not allowed by the NFS server due to its default
>> > > configuration
>> > > > >> to only allow to mount to /mnt/vol1/sec-storage (the root of the
>> NFS
>> > > > >> share).
>> > > > >>
>> > > > >>
>> > > > > Ah, that's odd!
>> > > > >
>> > > > > Btw, in 4.2 you'll be able to deploy SSVMs on RBD as well, so that
>> > > > > limitation will be gone.
>> > > > >
>> > > > > Wido
>> > > > >
>> > > > > After I changed the NFS server configuration to allow mount to
>> all
>> > > > >> sub-directories, re-export the NFS and voila, the system was
>> able to
>> > > > >> download the template and now both the system VMs (CPVM and
>> SSVM) are
>> > > > >> running!
>> > > > >>
>> > > > >> Many thanks for your help! :)
>> > > > >>
>> > > > >> Cheers.
>> > > > >>
>> > > > >>
>> > > > >>
>> > > > >> On Fri, Jul 12, 2013 at 3:31 PM, Indra Pramana <indra@sg.or.id
>> > > > >> <ma...@sg.or.id>> wrote:
>> > > > >>
>> > > > >> Hi Prasanna,
>> > > > >>
>> > > > >> Good day to you, and thank you for your e-mail.
>> > > > >>
>> > > > >> Yes, the cloudstack-agent service is running on both the KVM
>> > > hosts.
>> > > > >> There is no "cloud" user being created though, when I
>> installed
>> > > the
>> > > > >> agent. I installed the agent as root.
>> > > > >>
>> > > > >> root@hv-kvm-01:/home/indra# service cloudstack-agent status
>> > > > >> * cloud-agent is running
>> > > > >>
>> > > > >> root@hv-kvm-01:/home/indra# su - cloud
>> > > > >> Unknown id: cloud
>> > > > >>
>> > > > >> Please advise how can I resolve this problem, shall I create
>> the
>> > > > >> Unix "cloud" user manually? Basically I follow this
>> instruction to
>> > > > >> prepare the KVM host and install the CloudStack agent:
>> > > > >>
>> > > > >>
>> http://cloudstack.apache.org/**docs/en-US/Apache_CloudStack/**
>> > > > >>
>> 4.1.0/html/Installation_Guide/**hypervisor-kvm-install-flow.**html<
>> > >
>> http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.1.0/html/Installation_Guide/hypervisor-kvm-install-flow.html
>> > > >
>> > > > >>
>> > > > >> with this instruction from Wido on how to prepare libvirt
>> with
>> > > Ceph
>> > > > >> RBD storage pool support:
>> > > > >>
>> > > > >> http://blog.widodh.nl/2013/06/**a-quick-note-on-running-**
>> > > > >> cloudstack-with-rbd-on-ubuntu-**12-04/<
>> > >
>> http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/
>> > > >
>> > > > >>
>> > > > >> I also have checked /var/log/cloud/agent/agent.log and I
>> don't see
>> > > > >> any error messages, except this error message which will
>> show up
>> > > > >> every time I restart the agent:
>> > > > >>
>> > > > >> 2013-07-12 15:22:47,454 ERROR
>> > > [cloud.resource.**ServerResourceBase]
>> > > > >> (main:null) Nics are not configured!
>> > > > >> 2013-07-12 15:22:47,459 INFO
>> > > [cloud.resource.**ServerResourceBase]
>> > > > >> (main:null) Designating private to be nic eth0.5
>> > > > >>
>> > > > >> More logs can be found here: http://pastebin.com/yeNmCt7S
>> > > > >>
>> > > > >> I have configured the network bridges on the NIC interface
>> as per
>> > > > >> this instruction:
>> > > > >>
>> > > > >>
>> http://cloudstack.apache.org/**docs/en-US/Apache_CloudStack/**
>> > > > >> 4.1.0/html/Installation_Guide/**hypervisor-kvm-install-flow.**
>> > > > >> html#hypervisor-host-install-**network<
>> > >
>> http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.1.0/html/Installation_Guide/hypervisor-kvm-install-flow.html#hypervisor-host-install-network
>> > > >
>> > > > >>
>> > > > >> On the zone, I used advanced network configuration with just
>> one
>> > > > >> physical network for management, public and guest/private. I
>> > > didn't
>> > > > >> include storage, which by default the traffic will use the
>> > > > >> management VLAN network.
>> > > > >>
>> > > > >> Please advise if there's anything else I might have been
>> missing.
>> > > > >>
>> > > > >> Looking forward to your reply, thank you.
>> > > > >>
>> > > > >> Cheers.
>> > > > >>
>> > > > >>
>> > > > >>
>> > > > >> On Fri, Jul 12, 2013 at 2:56 PM, Prasanna Santhanam <
>> > > tsp@apache.org
>> > > > >> <ma...@apache.org>> wrote:
>> > > > >>
>> > > > >> Indeed, cloudstack will go through the allocation to
>> startup
>> > > the
>> > > > >> system VMs too. So that process is failing to recognize
>> the
>> > > volume
>> > > > >> (.qcow2) present on your NFS storage.
>> > > > >>
>> > > > >> Can you check if your cloudstack agent service is
>> running with
>> > > > >> the KVM
>> > > > >> host? And it should've created the user cloud. $id cloud
>> to
>> > > check
>> > > > >> if
>> > > > >> there is a user.
>> > > > >>
>> > > > >> Did you see what's happening in the agent logs? These are
>> > > under
>> > > > >> /var/log/cloud/ on your host when the systemVMs are
>> coming up.
>> > > > >> If the
>> > > > >> logs are not showing any useful information you can turn
>> on
>> > > debug
>> > > > >> level for more verbosity.
>> > > > >>
>> > > > >> See here: https://cwiki.apache.org/**confluence/x/FgPMAQ
>> <
>> > > https://cwiki.apache.org/confluence/x/FgPMAQ>
>> > > > >>
>> > > > >> On Fri, Jul 12, 2013 at 02:40:35PM +0800, Indra Pramana
>> wrote:
>> > > > >> > Hi Prasanna,
>> > > > >> >
>> > > > >> > Good day to you, and thank you for your e-mail.
>> > > > >> >
>> > > > >> > Yes, when I export the NFS, I set the permission so
>> that
>> > > > >> normal user will
>> > > > >> > be able to have read/write access to the files
>> > > > >> (no_root_squash).
>> > > > >> >
>> > > > >> > I have tested and I can have read/write access from
>> my KVM
>> > > > >> hosts using
>> > > > >> > normal user. BTW, there's no "cloud" user on the
>> hosts, I
>> > > > >> believe it's not
>> > > > >> > created during cloudstack-agent installation?
>> > > > >> >
>> > > > >> > In any case, do you think the template issue and the
>> > > storage
>> > > > >> pool
>> > > > >> > allocation issue might be related, or are they two
>> > > different
>> > > > >> problems
>> > > > >> > altogether?
>> > > > >> >
>> > > > >> > Looking forward to your reply, thank you.
>> > > > >> >
>> > > > >> > Cheers.
>> > > > >> >
>> > > > >> >
>> > > > >> >
>> > > > >> > On Fri, Jul 12, 2013 at 2:26 PM, Prasanna Santhanam
>> > > > >> <tsp@apache.org <ma...@apache.org>> wrote:
>> > > > >> >
>> > > > >> > > Can you access the file as user root? Or user
>> cloud? The
>> > > > >> cloudstack
>> > > > >> > > agent on your KVM host runs as user cloud and the
>> NFS
>> > > > >> permissions
>> > > > >> > > might be disallowing the volume (.qcow2) from being
>> > > accessed.
>> > > > >> > >
>> > > > >> > > On Fri, Jul 12, 2013 at 02:16:41PM +0800, Indra
>> Pramana
>> > > > >> wrote:
>> > > > >> > > > Hi Prasanna,
>> > > > >> > > >
>> > > > >> > > > Good day to you, and thank you for your e-mail.
>> > > > >> > > >
>> > > > >> > > > Yes, the file exists. I can access the file from
>> the
>> > > > >> management server
>> > > > >> > > and
>> > > > >> > > > the two hypervisors hosts if I mount manually.
>> > > > >> > > >
>> > > > >> > > > [root@cs-nas-01
>> /mnt/vol1/sec-storage/**template/tmpl/1/3]#
>> > > > >> ls
>> > > > >> > > > -la
>> > > > >> > > > total
>> > > > >> > > > 1418787
>> > > > >> > > > drwxr-xr-x 2 root wheel 4 Jul 11 20:21
>> > > > >> > > > .
>> > > > >> > > > drwxr-xr-x 3 root wheel 3 Jul 11 20:17
>> > > > >> > > > ..
>> > > > >> > > > -rw-r--r-- 1 root wheel 725811200 Jul 11 20:21
>> > > > >> > > > 425b9e5a-fbc7-4637-a33a-**fe9d0ed4fa98.qcow2
>> > > > >> > > > -rw-r--r-- 1 root wheel 295 Jul 11 20:21
>> > > > >> > > > template.properties
>> > > > >> > > > [root@cs-nas-01 /mnt/vol1/sec-storage/**
>> > > > >> template/tmpl/1/3]#
>> > > > >> > > > pwd
>> > > > >> > > > /mnt/vol1/sec-storage/**template/tmpl/1/3
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > > > Any advise?
>> > > > >> > > >
>> > > > >> > > > Looking forward to your reply, thank you.
>> > > > >> > > >
>> > > > >> > > > Cheers.
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > > > On Fri, Jul 12, 2013 at 2:07 PM, Prasanna
>> Santhanam
>> > > > >> <tsp@apache.org <ma...@apache.org>>
>> > > > >>
>> > > > >> > > wrote:
>> > > > >> > > >
>> > > > >> > > > > Can you check whether there is a file at:
>> > > > >> > > > >
>> > > > >> nfs://
>> 103.25.200.19/mnt/vol1/**sec-storage/template/tmpl/1/3/
>> > > <http://103.25.200.19/mnt/vol1/sec-storage/template/tmpl/1/3/>
>> > > > >> <
>> > > http://103.25.200.19/mnt/**vol1/sec-storage/template/**tmpl/1/3/<
>> > > http://103.25.200.19/mnt/vol1/sec-storage/template/tmpl/1/3/>
>> > > > >> >
>> > > > >>
>> > > > >> > > > >
>> > > > >> > > > > On Fri, Jul 12, 2013 at 01:59:34PM +0800, Indra
>> > > Pramana
>> > > > >> wrote:
>> > > > >> > > > > > Hi Prasanna,
>> > > > >> > > > > >
>> > > > >> > > > > > Thanks for your e-mail.
>> > > > >> > > > > >
>> > > > >> > > > > > I have tried restarting the management
>> server, and
>> > > > >> the problem still
>> > > > >> > > > > > persists. I even tried to re-do the
>> installation
>> > > and
>> > > > >> configuration
>> > > > >> > > again
>> > > > >> > > > > > from scratch last night, but the problem still
>> > > there.
>> > > > >> > > > > >
>> > > > >> > > > > > I also noted that on the beginning of the
>> logs, I
>> > > > >> found some error
>> > > > >> > > > > messages
>> > > > >> > > > > > saying that the template cannot be downloaded
>> to
>> > > the
>> > > > >> pool. See this
>> > > > >> > > logs:
>> > > > >> > > > > >
>> > > > >> > > > > > http://pastebin.com/BY1AVJ08
>> > > > >> > > > > >
>> > > > >> > > > > > It says it failed because cannot get volume
>> from
>> > > the
>> > > > >> pool. Could it
>> > > > >> > > be
>> > > > >> > > > > > related, i.e. the absence of the template
>> caused
>> > > the
>> > > > >> system VMs
>> > > > >> > > cannot be
>> > > > >> > > > > > created and started?
>> > > > >> > > > > >
>> > > > >> > > > > > I have ensured that I downloaded the system VM
>> > > > >> template using
>> > > > >> > > > > > cloud-install-sys-tmplt and verified that the
>> > > > >> template is already
>> > > > >> > > there
>> > > > >> > > > > in
>> > > > >> > > > > > the secondary storage server.
>> > > > >> > > > > >
>> > > > >> > > > > > Any advice is appreciated.
>> > > > >> > > > > >
>> > > > >> > > > > > Looking forward to your reply, thank you.
>> > > > >> > > > > >
>> > > > >> > > > > > Cheers.
>> > > > >> > > > > >
>> > > > >> > > > > >
>> > > > >> > > > > >
>> > > > >> > > > > > On Fri, Jul 12, 2013 at 1:21 PM, Prasanna
>> Santhanam
>> > > > >> <tsp@apache.org <ma...@apache.org>>
>> > > > >>
>> > > > >> > > > > wrote:
>> > > > >> > > > > >
>> > > > >> > > > > > > It looks like a previous attempt to start
>> the
>> > > > >> systemVMs has failed
>> > > > >> > > > > > > putting the nfs storage in the avoid set.
>> Did you
>> > > > >> try restarting
>> > > > >> > > your
>> > > > >> > > > > > > management server?
>> > > > >> > > > > > >
>> > > > >> > > > > > > This line leads me to the above mentioned:
>> > > > >> > > > > > > 2013-07-12 13:10:48,236 DEBUG
>> > > > >> > > > > > >
>> > > [storage.allocator.**AbstractStoragePoolAllocator]
>> > > > >> > > (secstorage-1:null)
>> > > > >> > > > > > > StoragePool is in avoid set, skipping this
>> pool
>> > > > >> > > > > > >
>> > > > >> > > > > > >
>> > > > >> > > > > > > On Fri, Jul 12, 2013 at 01:16:53PM +0800,
>> Indra
>> > > > >> Pramana wrote:
>> > > > >> > > > > > > > Dear Wido and all,
>> > > > >> > > > > > > >
>> > > > >> > > > > > > > I have managed to get the hosts, primary
>> and
>> > > > >> secondary storage
>> > > > >> > > > > running.
>> > > > >> > > > > > > >
>> > > > >> > > > > > > > - 2 KVM hypervisor hosts
>> > > > >> > > > > > > > - One RBD primary storage
>> > > > >> > > > > > > > - One NFS primary storage (for system VMs,
>> > > since
>> > > > >> I understand
>> > > > >> > > that
>> > > > >> > > > > system
>> > > > >> > > > > > > > VMs cannot use RBD)
>> > > > >> > > > > > > > - One NFS secondary storage
>> > > > >> > > > > > > >
>> > > > >> > > > > > > > However, now I am having problem with the
>> > > system
>> > > > >> VMs: CPVM and
>> > > > >> > > SSVM,
>> > > > >> > > > > > > unable
>> > > > >> > > > > > > > to start.
>> > > > >> > > > > > > >
>> > > > >> > > > > > > > Excerpt from management-server.log file is
>> > > here:
>> > > > >> > > > > > > > http://pastebin.com/ENkpCALY
>> > > > >> > > > > > > >
>> > > > >> > > > > > > > It seems that the VMs were not able to be
>> > > created
>> > > > >> because unable
>> > > > >> > > to
>> > > > >> > > > > find
>> > > > >> > > > > > > > suitable StoragePools.
>> > > > >> > > > > > > >
>> > > > >> > > > > > > > I understand that system VMs will be
>> using the
>> > > > >> NFS primary
>> > > > >> > > storage
>> > > > >> > > > > > > instead
>> > > > >> > > > > > > > of RBD, so I have confirmed that I am
>> able to
>> > > > >> mount the primary
>> > > > >> > > > > storage
>> > > > >> > > > > > > via
>> > > > >> > > > > > > > NFS and have read and write access, from
>> both
>> > > the
>> > > > >> hypervisor and
>> > > > >> > > the
>> > > > >> > > > > > > > management server.
>> > > > >> > > > > > > >
>> > > > >> > > > > > > > Any advise how can I resolve the problem
>> to
>> > > make
>> > > > >> both the system
>> > > > >> > > VMs
>> > > > >> > > > > > > > created and started?
>> > > > >> > > > > > > >
>> > > > >> > > > > > > > Looking forward to your reply, thank you.
>> > > > >> > > > > > > >
>> > > > >> > > > > > > > Cheers.
>> > > > >> > > > > > > >
>> > > > >> > > > > > > >
>> > > > >> > > > > > > > On Fri, Jul 12, 2013 at 9:43 AM, Indra
>> Pramana
>> > > > >> <indra@sg.or.id <ma...@sg.or.id>>
>> > > > >>
>> > > > >> > > > > wrote:
>> > > > >> > > > > > > >
>> > > > >> > > > > > > > > Hi Wido,
>> > > > >> > > > > > > > >
>> > > > >> > > > > > > > > Thanks for the advice, I'm now able to
>> add
>> > > the
>> > > > >> RBD pool as
>> > > > >> > > primary
>> > > > >> > > > > > > storage.
>> > > > >> > > > > > > > >
>> > > > >> > > > > > > > > Many thanks! :)
>> > > > >> > > > > > > > >
>> > > > >> > > > > > > > > Cheers.
>> > > > >> > > > > > > > >
>> > > > >> > > > > > > > >
>> > > > >> > > > > > > > > On Thursday, July 11, 2013, Wido den
>> > > Hollander
>> > > > >> wrote:
>> > > > >> > > > > > > > >
>> > > > >> > > > > > > > >> Hi,
>> > > > >> > > > > > > > >>
>> > > > >> > > > > > > > >> On 07/10/2013 03:42 PM, Chip Childers
>> wrote:
>> > > > >> > > > > > > > >>
>> > > > >> > > > > > > > >>> Cc'ing Wido, our resident Ceph
>> expert. ;-)
>> > > > >> > > > > > > > >>>
>> > > > >> > > > > > > > >>>
>> > > > >> > > > > > > > >> Hehe ;)
>> > > > >> > > > > > > > >>
>> > > > >> > > > > > > > >> On Wed, Jul 10, 2013 at 05:45:25PM
>> +0800,
>> > > > >> Indra Pramana
>> > > > >> > > wrote:
>> > > > >> > > > > > > > >>>
>> > > > >> > > > > > > > >>>> Dear all,
>> > > > >> > > > > > > > >>>>
>> > > > >> > > > > > > > >>>> I am installing CloudStack 4.1.0
>> (upgraded
>> > > > >> from 4.0.2) and I
>> > > > >> > > > > also
>> > > > >> > > > > > > have a
>> > > > >> > > > > > > > >>>> Ceph cluster running. However, I am
>> having
>> > > > >> issues in adding
>> > > > >> > > the
>> > > > >> > > > > RBD
>> > > > >> > > > > > > as
>> > > > >> > > > > > > > >>>> primary storage. Tried to follow the
>> > > > >> instruction here, but
>> > > > >> > > > > unable to
>> > > > >> > > > > > > > >>>> make
>> > > > >> > > > > > > > >>>> it work:
>> > > > >> > > > > > > > >>>>
>> > > > >> > > > > > > > >>>>
>> > > > >> http://ceph.com/docs/master/****rbd/rbd-cloudstack/<
>> > > http://ceph.com/docs/master/**rbd/rbd-cloudstack/>
>> > > > >> <
>> > > > >> > > > > > >
>> > > http://ceph.com/docs/master/**rbd/rbd-cloudstack/<
>> > > http://ceph.com/docs/master/rbd/rbd-cloudstack/>
>> > > > >> >
>> > > > >> > > > > > > > >>>>
>> > > > >> > > > > > > > >>>> I have setup a pool on the Ceph
>> cluster.
>> > > The
>> > > > >> status of the
>> > > > >> > > > > cluster
>> > > > >> > > > > > > is
>> > > > >> > > > > > > > >>>> healthy. Since I am using Ubuntu
>> 12.04.2
>> > > LTS
>> > > > >> (Precise) for
>> > > > >> > > the
>> > > > >> > > > > > > > >>>> hypervisors,
>> > > > >> > > > > > > > >>>> I also have compiled libvirt
>> manually to
>> > > > >> ensure that the
>> > > > >> > > version
>> > > > >> > > > > > > 0.9.13
>> > > > >> > > > > > > > >>>> is
>> > > > >> > > > > > > > >>>> installed (previously it's 0.9.8).
>> > > > >> > > > > > > > >>>>
>> > > > >> > > > > > > > >>>>
>> > > > >> > > > > > > > >> You can also use the Ubuntu Cloud
>> Archive, I
>> > > > >> still need to
>> > > > >> > > get the
>> > > > >> > > > > > > docs
>> > > > >> > > > > > > > >> updated for that.
>> > > > >> > > > > > > > >>
>> > > > >> > > > > > > > >> I described the process in a blogpost:
>> > > > >> > > > > > > http://blog.widodh.nl/2013/06/****<
>> > > http://blog.widodh.nl/2013/06/**>
>> > > > >> > > > > > > > >>
>> > > > >> > >
>> > > > >>
>> a-quick-note-on-running-****cloudstack-with-rbd-on-ubuntu-**
>> > > > >> **12-04/<
>> > > > >> > > > > > >
>> > > > >> > > > >
>> > > > >> > >
>> > > > >>
>> http://blog.widodh.nl/2013/06/**a-quick-note-on-running-**
>> > > > >> cloudstack-with-rbd-on-ubuntu-**12-04/<
>> > >
>> http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/
>> > > >
>> > > > >> > > > > > > >
>> > > > >> > > > > > > > >>
>> > > > >> > > > > > > > >> indra@hv-kvm-01:~/rbd$ ceph
>> > > > >> > > > > > > > >>>> ceph> health
>> > > > >> > > > > > > > >>>> HEALTH_OK
>> > > > >> > > > > > > > >>>>
>> > > > >> > > > > > > > >>>> indra@hv-kvm-01:~$ ceph osd lspools
>> > > > >> > > > > > > > >>>> 0 data,1 metadata,2 rbd,3 sc1,
>> > > > >> > > > > > > > >>>>
>> > > > >> > > > > > > > >>>> root@hv-kvm-01:/home/indra# libvirtd
>> > > > >> --version
>> > > > >> > > > > > > > >>>> libvirtd (libvirt) 0.9.13
>> > > > >> > > > > > > > >>>>
>> > > > >> > > > > > > > >>>> I tried to add Primary Storage into
>> the
>> > > > >> Cloudstack zone
>> > > > >> > > which I
>> > > > >> > > > > have
>> > > > >> > > > > > > > >>>> created:
>> > > > >> > > > > > > > >>>>
>> > > > >> > > > > > > > >>>> Add Primary Storage:
>> > > > >> > > > > > > > >>>>
>> > > > >> > > > > > > > >>>> Zone: my zone name
>> > > > >> > > > > > > > >>>> Pod: my pod name
>> > > > >> > > > > > > > >>>> Cluster: my cluster name
>> > > > >> > > > > > > > >>>> Name: ceph-rbd-pri-storage
>> > > > >> > > > > > > > >>>> Protocol: RBD
>> > > > >> > > > > > > > >>>> RADOS Monitor: my first Ceph monitor
>> IP
>> > > > >> address
>> > > > >> > > > > > > > >>>> RADOS Pool: sc1 (the pool name on
>> Ceph
>> > > > >> cluster)
>> > > > >> > > > > > > > >>>> RADOS User: client.admin
>> > > > >> > > > > > > > >>>> RADOS Secret:
>> > > > >> /etc/ceph/ceph.client.admin.****keyring (keyring
>> > > > >> > > > > file
>> > > > >> > > > > > > > >>>> location)
>> > > > >> > > > > > > > >>>>
>> > > > >> > > > > > > > >>>
>> > > > >> > > > > > > > >> This is your problem. That shouldn't
>> be the
>> > > > >> location of the
>> > > > >> > > file,
>> > > > >> > > > > but
>> > > > >> > > > > > > it
>> > > > >> > > > > > > > >> should be the secret, which is a base64
>> > > > >> encoded string.
>> > > > >> > > > > > > > >>
>> > > > >> > > > > > > > >> $ ceph auth list
>> > > > >> > > > > > > > >>
>> > > > >> > > > > > > > >> That should tell you what the secret
>> is.
>> > > > >> > > > > > > > >>
>> > > > >> > > > > > > > >> Storage Tags: rbd
>> > > > >> > > > > > > > >>
>> > > > >> > > > > > > > >> This is the error message when I tried
>> to
>> > > add
>> > > > >> the primary
>> > > > >> > > storage
>> > > > >> > > > > by
>> > > > >> > > > > > > > >> clicking OK:
>> > > > >> > > > > > > > >>
>> > > > >> > > > > > > > >> DB Exception on:
>> > > > >> > > com.mysql.jdbc.****JDBC4PreparedStatement@4b2eb56
>> > > > >> > > > > **:
>> > > > >> > > > > > > > >> INSERT INTO
>> > > > >> > > > > > > > >> storage_pool (storage_pool.id
>> > > > >> <http://storage_pool.id>, storage_pool.name
>> > > > >> <http://storage_pool.name>,
>> > > > >>
>> > > > >> > > > > storage_pool.uuid,
>> > > > >> > > > > > > > >> storage_pool.pool_type,
>> > > storage_pool.created,
>> > > > >> > > > > > > storage_pool.update_time,
>> > > > >> > > > > > > > >> storage_pool.data_center_id,
>> > > > >> storage_pool.pod_id,
>> > > > >> > > > > > > > >> storage_pool.available_bytes,
>> > > > >> storage_pool.capacity_bytes,
>> > > > >> > > > > > > > >> storage_pool.status,
>> storage_pool.scope,
>> > > > >> > > > > > > storage_pool.storage_provider_****
>> > > > >> > > > > > > > >> id,
>> > > > >> > > > > > > > >> storage_pool.host_address,
>> > > storage_pool.path,
>> > > > >> > > storage_pool.port,
>> > > > >> > > > > > > > >> storage_pool.user_info,
>> > > > >> storage_pool.cluster_id) VALUES (217,
>> > > > >> > > > > > > > >> _binary'ceph-rbd-pri-storage',
>> > > > >> > > > > > > > >>
>> > > > >> _binary'a226c9a1-da78-3f3a-****b5ac-e18b925c9634', 'RBD',
>> > > > >> > > > > '2013-07-10
>> > > > >> > > > > > > > >> 09:08:28', null, 2, 2, 0, 0, 'Up',
>> null,
>> > > null,
>> > > > >> null,
>> > > > >> > > _binary'ceph/
>> > > > >> > > > > > > > >> ceph.client.admin.keyring@10.
>> > > ****237.11.2/sc1<
>> > > > >> > > > > > > http://ceph.client.admin.**
>> > > keyring@10.237.11.2/sc1<
>> http://ceph.client.admin.keyring@10.237.11.2/sc1>
>> > > > >> >',
>> > > > >> > > > > > > > >> 6789, null, 2)
>> > > > >> > > > > > > > >>
>> > > > >> > > > > > > > >> On the management-server.log file:
>> > > > >> > > > > > > > >>
>> > > > >> > > > > > > > >> 2013-07-10 17:08:28,845 DEBUG
>> > > > >> [cloud.api.ApiServlet]
>> > > > >> > > > > > > > >> (catalina-exec-2:null)
>> > > > >> > > > > > > > >> ===START=== 192.168.0.100 -- GET
>> > > > >> > > > > > > > >>
>> > > > >>
>> > > command=createStoragePool&****zoneid=c116950e-e4ae-4f23-****a7e7-
>> > > > >> > > > > > > > >>
>> > > > >>
>> 74a75c4ee638&podId=a748b063-****3a83-4175-a0e9-de39118fe5ce&**
>> > > > >> > > > > > > > >>
>> > > > >>
>> > > clusterid=1f87eb09-324d-4d49-****83c2-88d84d7a15df&name=ceph-****
>> > > > >> > > > > > > > >>
>> > > > >>
>> rbd-pri-storage&url=rbd%3A%2F%****2Fclient.admin%3A_etc%2Fc
>> > > > >> > > > > > > > >>
>> > > > >>
>> eph%2Fceph.client.admin.****keyring%4010.237.11.2%2Fsc1&**
>> > > > >> > > > > > > > >>
>> > > > >> > > > >
>> > > > >> tags=rbd&response=json&****sessionkey=****
>> > > > >> rDRfWpqeKfQKbKZtHr398ULV%2F8k%****
>> > > > >> > > > > > > > >> 3D&_=1373447307839
>> > > > >> > > > > > > > >> 2013-07-10 17:08:28,862 DEBUG
>> > > > >> > > [cloud.storage.****StorageManagerImpl]
>> > > > >> > > > > > > > >> (catalina-exec-2:null) createPool
>> Params @
>> > > > >> scheme - rbd
>> > > > >> > > > > storageHost -
>> > > > >> > > > > > > null
>> > > > >> > > > > > > > >> hostPath - /ceph/ceph.client
>> > > > >> > > > > > > > >> .admin.keyring@10.237.11.2/sc1
>> > > > >> <http://admin.keyring@10.237.**11.2/sc1<
>> > > http://admin.keyring@10.237.11.2/sc1>>
>> > > > >> port - -1
>> > > > >>
>> > > > >> > > > > > > > >> 2013-07-10 17:08:28,918 DEBUG
>> > > > >> > > [cloud.storage.****StorageManagerImpl]
>> > > > >> > > > > > > > >> (catalina-exec-2:null) In createPool
>> Setting
>> > > > >> poolId - 217
>> > > > >> > > uuid -
>> > > > >> > > > > > > > >>
>> a226c9a1-da78-3f3a-b5ac-****e18b925c9634 z
>> > > > >> > > > > > > > >> oneId - 2 podId - 2 poolName -
>> > > > >> ceph-rbd-pri-storage
>> > > > >> > > > > > > > >> 2013-07-10 17:08:28,921 DEBUG
>> > > > >> [db.Transaction.Transaction]
>> > > > >> > > > > > > > >> (catalina-exec-2:null) Rolling back the
>> > > > >> transaction: Time = 3
>> > > > >> > > > > Name =
>> > > > >> > > > > > > > >> persist; called by -Transaction.rollbac
>> > > > >> > > > > > > > >>
>> > > > >>
>> k:890-Transaction.removeUpTo:****833-Transaction.close:657-**
>> > > > >> > > > > > > > >>
>> > > > >> TransactionContextBuilder.****interceptException:63-**
>> > > > >> > > > > > > > >>
>> > > > >> > > > >
>> > > > >>
>> > > ComponentInstantiationPostProc****essor$InterceptorDispatcher.**
>> > > > >> **interce
>> > > > >> > > > > > > > >> pt:133-StorageManagerImpl.****
>> > > > >> createPool:1378-**
>> > > > >> > > > > > > > >>
>> > > > >>
>> StorageManagerImpl.createPool:****147-CreateStoragePoolCmd.**
>> > > > >> > > > > > > > >>
>> > > > >> execute:123-ApiDispatcher.****dispatch:162-ApiServer.**
>> > > > >> > > > > > > > >> queueCommand:505-ApiSe
>> > > > >> > > > > > > > >>
>> > > > >> rver.handleRequest:355-****ApiServlet.processRequest:302
>> > > > >> > > > > > > > >> 2013-07-10 17:08:28,923 ERROR
>> > > > >> [cloud.api.ApiServer]
>> > > > >> > > > > > > (catalina-exec-2:null)
>> > > > >> > > > > > > > >> unhandled exception executing api
>> command:
>> > > > >> createStoragePool
>> > > > >> > > > > > > > >>
>> > > > >> com.cloud.utils.exception.****CloudRuntimeException: DB
>> > > > >> > > Exception
>> > > > >> > > > > on:
>> > > > >> > > > > > > > >>
>> > > > >> com.mysql.jdbc.****JDBC4PreparedStatement@4b2eb56****:
>> INSERT
>> > > > >> > > INTO
>> > > > >> > > > > > > > >> storage_pool (
>> > > > >> > > > > > > > >> storage_pool.id <
>> http://storage_pool.id>,
>> > > > >>
>> > > > >> storage_pool
>> > > > >> > > > > > > > >> .name, storage_pool.uuid,
>> > > > >> storage_pool.pool_type,
>> > > > >> > > > > > > storage_pool.created,
>> > > > >> > > > > > > > >> storage_pool.update_time,
>> > > > >> storage_pool.data_center_id,
>> > > > >> > > > > > > > >> storage_pool.pod_id,
>> > > > >> > > > > > > > >> storage_pool.availab
>> > > > >> > > > > > > > >> le_bytes, storage_pool.capacity_bytes,
>> > > > >> storage_pool.status,
>> > > > >> > > > > > > > >> storage_pool.scope,
>> > > > >> storage_pool.storage_provider_****id,
>> > > > >> > > > > > > > >> storage_pool.host_address,
>> > > storage_pool.path,
>> > > > >> storage_
>> > > > >> > > > > > > > >> pool.port, storage_pool.user_info,
>> > > > >> storage_pool.cluster_id)
>> > > > >> > > VALUES
>> > > > >> > > > > > > (217,
>> > > > >> > > > > > > > >> _binary'ceph-rbd-pri-storage',
>> > > > >> > > > > > > > >>
>> > > > >> _binary'a226c9a1-da78-3f3a-****b5ac-e18b925c9634', 'RBD',
>> > > > >> > > '2013-07-1
>> > > > >> > > > > > > > >> 0 09:08:28', null, 2, 2, 0, 0, 'Up',
>> null,
>> > > > >> null, null,
>> > > > >> > > > > _binary'ceph/
>> > > > >> > > > > > > > >> ceph.client.admin.keyring@10.
>> > > ****237.11.2/sc1<
>> > > > >> > > > > > > http://ceph.client.admin.**
>> > > keyring@10.237.11.2/sc1<
>> http://ceph.client.admin.keyring@10.237.11.2/sc1>
>> > > > >> >',
>> > > > >> > > > > > > > >> 6789, null, 2)
>> > > > >> > > > > > > > >> at
>> > > > >> > > > > > > > >>
>> > > > >> > > > > > >
>> > > > >> > >
>> > > > >> com.cloud.utils.db.****GenericDaoBase.persist(****
>> > > > >> GenericDaoBase.java:1342)
>> > > > >> > > > > > > > >> at
>> > > > >> > > > > > > > >>
>> > > > >> com.cloud.storage.dao.****StoragePoolDaoImpl.persist(**
>> > > > >> > > > > > > > >> StoragePoolDaoImpl.java:232)
>> > > > >> > > > > > > > >> at
>> > > > >> > > > > > > > >>
>> > > > >>
>> > > com.cloud.utils.component.****ComponentInstantiationPostProc****
>> > > > >> > > > > > > > >>
>> > > > >> > > > > > >
>> > > > >> > > > >
>> > > > >> > >
>> > > > >> essor$InterceptorDispatcher.****intercept(****
>> > > > >> ComponentInstantiationPostProc***
>> > > > >> > > > > > > > >> *es
>> > > > >> > > > > > > > >>
>> > > > >> > > > > > > > >>
>> > > > >> > > > > > >
>> > > > >> > > > > > > --
>> > > > >> > > > > > > Prasanna.,
>> > > > >> > > > > > >
>> > > > >> > > > > > > ------------------------
>> > > > >> > > > > > > Powered by BigRock.com
>> > > > >> > > > > > >
>> > > > >> > > > > > >
>> > > > >> > > > >
>> > > > >> > > > > --
>> > > > >> > > > > Prasanna.,
>> > > > >> > > > >
>> > > > >> > > > > ------------------------
>> > > > >> > > > > Powered by BigRock.com
>> > > > >> > > > >
>> > > > >> > > > >
>> > > > >> > >
>> > > > >> > > --
>> > > > >> > > Prasanna.,
>> > > > >> > >
>> > > > >> > > ------------------------
>> > > > >> > > Powered by BigRock.com
>> > > > >> > >
>> > > > >> > >
>> > > > >>
>> > > > >> --
>> > > > >> Prasanna.,
>> > > > >>
>> > > > >> ------------------------
>> > > > >> Powered by BigRock.com
>> > > > >>
>> > > > >>
>> > > > >>
>> > > > >>
>> > > > >
>> > >
>> > > --
>> > > Prasanna.,
>> > >
>> > > ------------------------
>> > > Powered by BigRock.com
>> > >
>> > >
>>
>> --
>> Prasanna.,
>>
>> ------------------------
>> Powered by BigRock.com
>>
>>
>
Re: Problem in adding Ceph RBD as primary storage for CloudStack 4.1.0
Posted by Indra Pramana <in...@sg.or.id>.
Hi Prasanna,
On Mon, Jul 15, 2013 at 12:20 PM, Prasanna Santhanam <ts...@apache.org> wrote:
> On Sat, Jul 13, 2013 at 02:25:35PM +0800, Indra Pramana wrote:
> > Hi Prasanna,
> >
> > Good day to you, and thank you for your e-mail.
> >
> Were you able to get beyond this error?
>
Yes. I managed to get around the network bridges issue by setting up this
on my KVM host:
===
auto eth0.5
iface eth0.5 inet static
address X.X.X.22
netmask 255.255.255.240
network X.X.X.16
broadcast X.X.X.31
gateway X.X.X.17
# dns-* options are implemented by the resolvconf package, if
installed
dns-nameservers 8.8.8.8 8.8.4.4
dns-search xxx.com
# Public network
auto cloudbr0
iface cloudbr0 inet manual
bridge_ports eth1.201
bridge_fd 5
bridge_stp off
bridge_maxwait 1
# Guest network
auto cloudbr1
iface cloudbr1 inet manual
bridge_ports eth1.41
bridge_fd 5
bridge_stp off
bridge_maxwait 1
# Management network
auto cloudbr2
iface cloudbr2 inet manual
bridge_ports eth0.6
bridge_fd 5
bridge_stp off
bridge_maxwait 1
# The 10G network interface
auto eth2
iface eth2 inet static
address 10.237.11.22
netmask 255.255.255.0
network 10.237.11.0
broadcast 10.237.11.255
===
On the Cloudstack GUI, I created 3 physical networks:
cloudbr0 - public network
cloudbr1 - guest network
cloudbr2 - management (and storage) network
However, when I tried to setup the public network using VLAN 201, it failed
because CloudStack tried to create a new bridge (i.e. breth1-201) instead
of using the existing bridge I have set up for public network (cloudbr
mapped to eth1.201). So I set up the public network using a different VLAN
(VLAN 21) and change the settings on my router's gateway to use VLAN 21
instead of VLAN 201.
===
root@hv-kvm-02:~# brctl show
bridge name bridge id STP enabled interfaces
breth1-21 8000.002590c29d7d no eth1.21
vnet2
vnet5
cloud0 8000.fe00a9fe0165 no vnet0
vnet3
cloudbr0 8000.002590c29d7d no eth1.201
cloudbr1 8000.002590c29d7d no eth1.41
cloudbr2 8000.002590c29d7c no eth0.6
vnet1
vnet4
vnet6
virbr0 8000.000000000000 yes
===
The above setup works, but I am not too sure whether this is the correct
way on how to do it. Can you advise?
Looking forward to your reply, thank you.
Cheers.
> > Yesterday I troubleshooted further and found out that the mapping of the
> > bridge is incorrect. For example, eth1 of the SSVM, which is supposed to
> be
> > private network interface for communication with management server, is
> > being mapped to cloudbr0 bridge which is used for public. While eth2 of
> the
> > SSVM, which is supposed to be public, is being mapped to cloudbr1 bridge
> > which is supposed for guest traffic. I checked with brctl show on the KVM
> > host and traced the MAC address of the SSVM's eth interfaces.
> >
> > I have tried different type of configuration on the KVM hosts'
> > /etc/network/interfaces (I am using Ubuntu), different ways on how to
> > configure the advanced network zone (physical network interface and KVM
> > label for each traffic), but I am still not able to get it working.
> >
> > Can please advise what is the best way to set-up the network bridges?
>
> Use brctl to add bridges. That's how I've done it.
>
> >
> > My /etc/network/interfaces on KVM host:
> >
> > http://pastebin.com/nx8xJ1L2
> >
> > I used 3 physical NICs:
> >
> > eth0 --> for management and secondary storage traffic
> > eth1 --> for public and guest traffic
> > eth2 --> for primary storage traffic to the Ceph RBD (separate NIC not
> > configured by CloudStack)
> >
> > On the Cloudstack GUI, while creating the zone, I used 3 physical
> networks:
> >
> > eth0.6 --> Management traffic (KVM label: eth0.6)
> > cloudbr0 --> Public traffic (KVM label: cloudbr0)
> > cloudbr1 --> Guest traffic (KVM label: cloudbr1)
> >
> Is eth0.6 on a tagged VLAN?
>
> > I didn't specify storage traffic since I presume it will use the
> management
> > VLAN.
>
> That's correct.
>
> >
> > However, I always failed to add the KVM host using the above
> configuration.
> > I tried to run cloudstack-setup-agent manually on the KVM host and
> getting
> > this error message:
> >
> > root@hv-kvm-02:/etc/cloudstack/agent# cloudstack-setup-agent -m
> x.x.x.18 -z
> > 3 -p 3 -c 3 -g 00dd5dba-7419-3689-bc51-1671035c0d8f -a --pubNic=cloudbr0
> > --prvNic=eth0.6 --guestNic=cloudbr1
> > Starting to configure your system:
> > Configure Apparmor ... [OK]
> > Configure Network ... [Failed]
> > eth0.6 is not a bridge
> > Try to restore your system:
> > Restore Apparmor ... [OK]
> > Restore Network ... [OK]
> >
> > When I tried to create another bridge called cloudbr2 with bridge_ports
> to
> > eth0.6, the KVM host's network didn't work.
> >
> > Can advise what is the best practice on configuring the network bridges
> on
> > the advanced network zone setup?
> >
> > Looking forward to your reply, thank you.
> >
> > Cheers.
> >
> >
> >
> > On Sat, Jul 13, 2013 at 12:28 PM, Prasanna Santhanam <ts...@apache.org>
> wrote:
> >
> > > See Inline,
> > >
> > > On Fri, Jul 12, 2013 at 05:12:25PM +0800, Indra Pramana wrote:
> > > > Hi Wido,
> > > >
> > > > Noted, can't wait for 4.2 to be released. :)
> > > >
> > > > Dear Prasanna, Wido and all,
> > > >
> > > > I just realised that while the system VMs are running, they are
> still not
> > > > accessible through the public IPs assigned to them. I have been
> waiting
> > > for
> > > > the SSVM to download the default CentOS template and it doesn't
> appear on
> > > > the template list.
> > > >
> > > > I tried to SSH into the SSVM via the local link address from the KVM
> > > host,
> > > > and running the health check /usr/local/cloud/systemvm/ssvm-check.sh
> > > shows
> > > > that the VM cannot reach anywhere. It cannot reach the public DNS
> server
> > > (I
> > > > used Google's 8.8.8.8), cannot reach the management server, and
> cannot
> > > even
> > > > reach the public IP gateway.
> > >
> > > Ok - we're not there yet. But there's a few troubleshooting steps you
> > > can try:
> > >
> > > 1. Which of the checks fail apart from the DNS ping?
> > > 2. Are you able to have those tests pass from the KVM host itself?
> > > 3. Can you paste the output of $route -n on your SSVM and on your
> > > host?
> > >
> > > >
> > > > Is it due to misconfiguration of the KVM network bridges? How can I
> see
> > > the
> > > > mapping between the NIC interfaces of the SSVM (eth0, eth1, eth2 and
> > > eth3)
> > > > and the actual physical NIC interfaces on the KVM hosts (eth0) and
> the
> > > > network bridges (cloudbr0, cloudbr1)? Any logs I can verify to ensure
> > > that
> > > > the VLAN and network bridging is working?
> > > >
> > >
> > > You can see this from the dumpxml.
> > >
> > > $virsh list
> > > (gives you all the domains. find the ssvm for example)
> > >
> > > $virsh dumpxml my-domain > mydomain.xml
> > >
> > > > Appreciate any advice.
> > > >
> > > > Thank you.
> > > >
> > > >
> > > >
> > > > On Fri, Jul 12, 2013 at 4:19 PM, Wido den Hollander <wi...@widodh.nl>
> > > wrote:
> > > >
> > > > > On 07/12/2013 10:14 AM, Indra Pramana wrote:
> > > > >
> > > > >> Hi Prasanna,
> > > > >>
> > > > >> I managed to fix the problem, thanks for your advice to turn the
> agent
> > > > >> log level to debug:
> > > > >>
> > > > >> https://cwiki.apache.org/**confluence/display/CLOUDSTACK/**
> > > > >> KVM+agent+debug<
> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug
> >
> > > > >>
> > > > >> From the log, I found out that the agent on the KVM host tried
> to NFS
> > > > >> mount directly to 103.25.200.19:/mnt/vol1/sec-**
> > > > >> storage/template/tmpl/1/3,
> > > > >> which was not allowed by the NFS server due to its default
> > > configuration
> > > > >> to only allow to mount to /mnt/vol1/sec-storage (the root of the
> NFS
> > > > >> share).
> > > > >>
> > > > >>
> > > > > Ah, that's odd!
> > > > >
> > > > > Btw, in 4.2 you'll be able to deploy SSVMs on RBD as well, so that
> > > > > limitation will be gone.
> > > > >
> > > > > Wido
> > > > >
> > > > > After I changed the NFS server configuration to allow mount to all
> > > > >> sub-directories, re-export the NFS and voila, the system was able
> to
> > > > >> download the template and now both the system VMs (CPVM and SSVM)
> are
> > > > >> running!
> > > > >>
> > > > >> Many thanks for your help! :)
> > > > >>
> > > > >> Cheers.
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Fri, Jul 12, 2013 at 3:31 PM, Indra Pramana <indra@sg.or.id
> > > > >> <ma...@sg.or.id>> wrote:
> > > > >>
> > > > >> Hi Prasanna,
> > > > >>
> > > > >> Good day to you, and thank you for your e-mail.
> > > > >>
> > > > >> Yes, the cloudstack-agent service is running on both the KVM
> > > hosts.
> > > > >> There is no "cloud" user being created though, when I
> installed
> > > the
> > > > >> agent. I installed the agent as root.
> > > > >>
> > > > >> root@hv-kvm-01:/home/indra# service cloudstack-agent status
> > > > >> * cloud-agent is running
> > > > >>
> > > > >> root@hv-kvm-01:/home/indra# su - cloud
> > > > >> Unknown id: cloud
> > > > >>
> > > > >> Please advise how can I resolve this problem, shall I create
> the
> > > > >> Unix "cloud" user manually? Basically I follow this
> instruction to
> > > > >> prepare the KVM host and install the CloudStack agent:
> > > > >>
> > > > >>
> http://cloudstack.apache.org/**docs/en-US/Apache_CloudStack/**
> > > > >>
> 4.1.0/html/Installation_Guide/**hypervisor-kvm-install-flow.**html<
> > >
> http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.1.0/html/Installation_Guide/hypervisor-kvm-install-flow.html
> > > >
> > > > >>
> > > > >> with this instruction from Wido on how to prepare libvirt with
> > > Ceph
> > > > >> RBD storage pool support:
> > > > >>
> > > > >> http://blog.widodh.nl/2013/06/**a-quick-note-on-running-**
> > > > >> cloudstack-with-rbd-on-ubuntu-**12-04/<
> > >
> http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/
> > > >
> > > > >>
> > > > >> I also have checked /var/log/cloud/agent/agent.log and I
> don't see
> > > > >> any error messages, except this error message which will show
> up
> > > > >> every time I restart the agent:
> > > > >>
> > > > >> 2013-07-12 15:22:47,454 ERROR
> > > [cloud.resource.**ServerResourceBase]
> > > > >> (main:null) Nics are not configured!
> > > > >> 2013-07-12 15:22:47,459 INFO
> > > [cloud.resource.**ServerResourceBase]
> > > > >> (main:null) Designating private to be nic eth0.5
> > > > >>
> > > > >> More logs can be found here: http://pastebin.com/yeNmCt7S
> > > > >>
> > > > >> I have configured the network bridges on the NIC interface as
> per
> > > > >> this instruction:
> > > > >>
> > > > >>
> http://cloudstack.apache.org/**docs/en-US/Apache_CloudStack/**
> > > > >> 4.1.0/html/Installation_Guide/**hypervisor-kvm-install-flow.**
> > > > >> html#hypervisor-host-install-**network<
> > >
> http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.1.0/html/Installation_Guide/hypervisor-kvm-install-flow.html#hypervisor-host-install-network
> > > >
> > > > >>
> > > > >> On the zone, I used advanced network configuration with just
> one
> > > > >> physical network for management, public and guest/private. I
> > > didn't
> > > > >> include storage, which by default the traffic will use the
> > > > >> management VLAN network.
> > > > >>
> > > > >> Please advise if there's anything else I might have been
> missing.
> > > > >>
> > > > >> Looking forward to your reply, thank you.
> > > > >>
> > > > >> Cheers.
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Fri, Jul 12, 2013 at 2:56 PM, Prasanna Santhanam <
> > > tsp@apache.org
> > > > >> <ma...@apache.org>> wrote:
> > > > >>
> > > > >> Indeed, cloudstack will go through the allocation to
> startup
> > > the
> > > > >> system VMs too. So that process is failing to recognize
> the
> > > volume
> > > > >> (.qcow2) present on your NFS storage.
> > > > >>
> > > > >> Can you check if your cloudstack agent service is running
> with
> > > > >> the KVM
> > > > >> host? And it should've created the user cloud. $id cloud
> to
> > > check
> > > > >> if
> > > > >> there is a user.
> > > > >>
> > > > >> Did you see what's happening in the agent logs? These are
> > > under
> > > > >> /var/log/cloud/ on your host when the systemVMs are
> coming up.
> > > > >> If the
> > > > >> logs are not showing any useful information you can turn
> on
> > > debug
> > > > >> level for more verbosity.
> > > > >>
> > > > >> See here: https://cwiki.apache.org/**confluence/x/FgPMAQ<
> > > https://cwiki.apache.org/confluence/x/FgPMAQ>
> > > > >>
> > > > >> On Fri, Jul 12, 2013 at 02:40:35PM +0800, Indra Pramana
> wrote:
> > > > >> > Hi Prasanna,
> > > > >> >
> > > > >> > Good day to you, and thank you for your e-mail.
> > > > >> >
> > > > >> > Yes, when I export the NFS, I set the permission so
> that
> > > > >> normal user will
> > > > >> > be able to have read/write access to the files
> > > > >> (no_root_squash).
> > > > >> >
> > > > >> > I have tested and I can have read/write access from my
> KVM
> > > > >> hosts using
> > > > >> > normal user. BTW, there's no "cloud" user on the
> hosts, I
> > > > >> believe it's not
> > > > >> > created during cloudstack-agent installation?
> > > > >> >
> > > > >> > In any case, do you think the template issue and the
> > > storage
> > > > >> pool
> > > > >> > allocation issue might be related, or are they two
> > > different
> > > > >> problems
> > > > >> > altogether?
> > > > >> >
> > > > >> > Looking forward to your reply, thank you.
> > > > >> >
> > > > >> > Cheers.
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Fri, Jul 12, 2013 at 2:26 PM, Prasanna Santhanam
> > > > >> <tsp@apache.org <ma...@apache.org>> wrote:
> > > > >> >
> > > > >> > > Can you access the file as user root? Or user cloud?
> The
> > > > >> cloudstack
> > > > >> > > agent on your KVM host runs as user cloud and the NFS
> > > > >> permissions
> > > > >> > > might be disallowing the volume (.qcow2) from being
> > > accessed.
> > > > >> > >
> > > > >> > > On Fri, Jul 12, 2013 at 02:16:41PM +0800, Indra
> Pramana
> > > > >> wrote:
> > > > >> > > > Hi Prasanna,
> > > > >> > > >
> > > > >> > > > Good day to you, and thank you for your e-mail.
> > > > >> > > >
> > > > >> > > > Yes, the file exists. I can access the file from
> the
> > > > >> management server
> > > > >> > > and
> > > > >> > > > the two hypervisors hosts if I mount manually.
> > > > >> > > >
> > > > >> > > > [root@cs-nas-01
> /mnt/vol1/sec-storage/**template/tmpl/1/3]#
> > > > >> ls
> > > > >> > > > -la
> > > > >> > > > total
> > > > >> > > > 1418787
> > > > >> > > > drwxr-xr-x 2 root wheel 4 Jul 11 20:21
> > > > >> > > > .
> > > > >> > > > drwxr-xr-x 3 root wheel 3 Jul 11 20:17
> > > > >> > > > ..
> > > > >> > > > -rw-r--r-- 1 root wheel 725811200 Jul 11 20:21
> > > > >> > > > 425b9e5a-fbc7-4637-a33a-**fe9d0ed4fa98.qcow2
> > > > >> > > > -rw-r--r-- 1 root wheel 295 Jul 11 20:21
> > > > >> > > > template.properties
> > > > >> > > > [root@cs-nas-01 /mnt/vol1/sec-storage/**
> > > > >> template/tmpl/1/3]#
> > > > >> > > > pwd
> > > > >> > > > /mnt/vol1/sec-storage/**template/tmpl/1/3
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > Any advise?
> > > > >> > > >
> > > > >> > > > Looking forward to your reply, thank you.
> > > > >> > > >
> > > > >> > > > Cheers.
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > On Fri, Jul 12, 2013 at 2:07 PM, Prasanna Santhanam
> > > > >> <tsp@apache.org <ma...@apache.org>>
> > > > >>
> > > > >> > > wrote:
> > > > >> > > >
> > > > >> > > > > Can you check whether there is a file at:
> > > > >> > > > >
> > > > >> nfs://
> 103.25.200.19/mnt/vol1/**sec-storage/template/tmpl/1/3/
> > > <http://103.25.200.19/mnt/vol1/sec-storage/template/tmpl/1/3/>
> > > > >> <
> > > http://103.25.200.19/mnt/**vol1/sec-storage/template/**tmpl/1/3/<
> > > http://103.25.200.19/mnt/vol1/sec-storage/template/tmpl/1/3/>
> > > > >> >
> > > > >>
> > > > >> > > > >
> > > > >> > > > > On Fri, Jul 12, 2013 at 01:59:34PM +0800, Indra
> > > Pramana
> > > > >> wrote:
> > > > >> > > > > > Hi Prasanna,
> > > > >> > > > > >
> > > > >> > > > > > Thanks for your e-mail.
> > > > >> > > > > >
> > > > >> > > > > > I have tried restarting the management server,
> and
> > > > >> the problem still
> > > > >> > > > > > persists. I even tried to re-do the
> installation
> > > and
> > > > >> configuration
> > > > >> > > again
> > > > >> > > > > > from scratch last night, but the problem still
> > > there.
> > > > >> > > > > >
> > > > >> > > > > > I also noted that on the beginning of the
> logs, I
> > > > >> found some error
> > > > >> > > > > messages
> > > > >> > > > > > saying that the template cannot be downloaded
> to
> > > the
> > > > >> pool. See this
> > > > >> > > logs:
> > > > >> > > > > >
> > > > >> > > > > > http://pastebin.com/BY1AVJ08
> > > > >> > > > > >
> > > > >> > > > > > It says it failed because cannot get volume
> from
> > > the
> > > > >> pool. Could it
> > > > >> > > be
> > > > >> > > > > > related, i.e. the absence of the template
> caused
> > > the
> > > > >> system VMs
> > > > >> > > cannot be
> > > > >> > > > > > created and started?
> > > > >> > > > > >
> > > > >> > > > > > I have ensured that I downloaded the system VM
> > > > >> template using
> > > > >> > > > > > cloud-install-sys-tmplt and verified that the
> > > > >> template is already
> > > > >> > > there
> > > > >> > > > > in
> > > > >> > > > > > the secondary storage server.
> > > > >> > > > > >
> > > > >> > > > > > Any advice is appreciated.
> > > > >> > > > > >
> > > > >> > > > > > Looking forward to your reply, thank you.
> > > > >> > > > > >
> > > > >> > > > > > Cheers.
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > On Fri, Jul 12, 2013 at 1:21 PM, Prasanna
> Santhanam
> > > > >> <tsp@apache.org <ma...@apache.org>>
> > > > >>
> > > > >> > > > > wrote:
> > > > >> > > > > >
> > > > >> > > > > > > It looks like a previous attempt to start the
> > > > >> systemVMs has failed
> > > > >> > > > > > > putting the nfs storage in the avoid set.
> Did you
> > > > >> try restarting
> > > > >> > > your
> > > > >> > > > > > > management server?
> > > > >> > > > > > >
> > > > >> > > > > > > This line leads me to the above mentioned:
> > > > >> > > > > > > 2013-07-12 13:10:48,236 DEBUG
> > > > >> > > > > > >
> > > [storage.allocator.**AbstractStoragePoolAllocator]
> > > > >> > > (secstorage-1:null)
> > > > >> > > > > > > StoragePool is in avoid set, skipping this
> pool
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > > On Fri, Jul 12, 2013 at 01:16:53PM +0800,
> Indra
> > > > >> Pramana wrote:
> > > > >> > > > > > > > Dear Wido and all,
> > > > >> > > > > > > >
> > > > >> > > > > > > > I have managed to get the hosts, primary
> and
> > > > >> secondary storage
> > > > >> > > > > running.
> > > > >> > > > > > > >
> > > > >> > > > > > > > - 2 KVM hypervisor hosts
> > > > >> > > > > > > > - One RBD primary storage
> > > > >> > > > > > > > - One NFS primary storage (for system VMs,
> > > since
> > > > >> I understand
> > > > >> > > that
> > > > >> > > > > system
> > > > >> > > > > > > > VMs cannot use RBD)
> > > > >> > > > > > > > - One NFS secondary storage
> > > > >> > > > > > > >
> > > > >> > > > > > > > However, now I am having problem with the
> > > system
> > > > >> VMs: CPVM and
> > > > >> > > SSVM,
> > > > >> > > > > > > unable
> > > > >> > > > > > > > to start.
> > > > >> > > > > > > >
> > > > >> > > > > > > > Excerpt from management-server.log file is
> > > here:
> > > > >> > > > > > > > http://pastebin.com/ENkpCALY
> > > > >> > > > > > > >
> > > > >> > > > > > > > It seems that the VMs were not able to be
> > > created
> > > > >> because unable
> > > > >> > > to
> > > > >> > > > > find
> > > > >> > > > > > > > suitable StoragePools.
> > > > >> > > > > > > >
> > > > >> > > > > > > > I understand that system VMs will be using
> the
> > > > >> NFS primary
> > > > >> > > storage
> > > > >> > > > > > > instead
> > > > >> > > > > > > > of RBD, so I have confirmed that I am able
> to
> > > > >> mount the primary
> > > > >> > > > > storage
> > > > >> > > > > > > via
> > > > >> > > > > > > > NFS and have read and write access, from
> both
> > > the
> > > > >> hypervisor and
> > > > >> > > the
> > > > >> > > > > > > > management server.
> > > > >> > > > > > > >
> > > > >> > > > > > > > Any advise how can I resolve the problem to
> > > make
> > > > >> both the system
> > > > >> > > VMs
> > > > >> > > > > > > > created and started?
> > > > >> > > > > > > >
> > > > >> > > > > > > > Looking forward to your reply, thank you.
> > > > >> > > > > > > >
> > > > >> > > > > > > > Cheers.
> > > > >> > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > > > On Fri, Jul 12, 2013 at 9:43 AM, Indra
> Pramana
> > > > >> <indra@sg.or.id <ma...@sg.or.id>>
> > > > >>
> > > > >> > > > > wrote:
> > > > >> > > > > > > >
> > > > >> > > > > > > > > Hi Wido,
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Thanks for the advice, I'm now able to
> add
> > > the
> > > > >> RBD pool as
> > > > >> > > primary
> > > > >> > > > > > > storage.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Many thanks! :)
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Cheers.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > On Thursday, July 11, 2013, Wido den
> > > Hollander
> > > > >> wrote:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >> Hi,
> > > > >> > > > > > > > >>
> > > > >> > > > > > > > >> On 07/10/2013 03:42 PM, Chip Childers
> wrote:
> > > > >> > > > > > > > >>
> > > > >> > > > > > > > >>> Cc'ing Wido, our resident Ceph expert.
> ;-)
> > > > >> > > > > > > > >>>
> > > > >> > > > > > > > >>>
> > > > >> > > > > > > > >> Hehe ;)
> > > > >> > > > > > > > >>
> > > > >> > > > > > > > >> On Wed, Jul 10, 2013 at 05:45:25PM
> +0800,
> > > > >> Indra Pramana
> > > > >> > > wrote:
> > > > >> > > > > > > > >>>
> > > > >> > > > > > > > >>>> Dear all,
> > > > >> > > > > > > > >>>>
> > > > >> > > > > > > > >>>> I am installing CloudStack 4.1.0
> (upgraded
> > > > >> from 4.0.2) and I
> > > > >> > > > > also
> > > > >> > > > > > > have a
> > > > >> > > > > > > > >>>> Ceph cluster running. However, I am
> having
> > > > >> issues in adding
> > > > >> > > the
> > > > >> > > > > RBD
> > > > >> > > > > > > as
> > > > >> > > > > > > > >>>> primary storage. Tried to follow the
> > > > >> instruction here, but
> > > > >> > > > > unable to
> > > > >> > > > > > > > >>>> make
> > > > >> > > > > > > > >>>> it work:
> > > > >> > > > > > > > >>>>
> > > > >> > > > > > > > >>>>
> > > > >> http://ceph.com/docs/master/****rbd/rbd-cloudstack/<
> > > http://ceph.com/docs/master/**rbd/rbd-cloudstack/>
> > > > >> <
> > > > >> > > > > > >
> > > http://ceph.com/docs/master/**rbd/rbd-cloudstack/<
> > > http://ceph.com/docs/master/rbd/rbd-cloudstack/>
> > > > >> >
> > > > >> > > > > > > > >>>>
> > > > >> > > > > > > > >>>> I have setup a pool on the Ceph
> cluster.
> > > The
> > > > >> status of the
> > > > >> > > > > cluster
> > > > >> > > > > > > is
> > > > >> > > > > > > > >>>> healthy. Since I am using Ubuntu
> 12.04.2
> > > LTS
> > > > >> (Precise) for
> > > > >> > > the
> > > > >> > > > > > > > >>>> hypervisors,
> > > > >> > > > > > > > >>>> I also have compiled libvirt manually
> to
> > > > >> ensure that the
> > > > >> > > version
> > > > >> > > > > > > 0.9.13
> > > > >> > > > > > > > >>>> is
> > > > >> > > > > > > > >>>> installed (previously it's 0.9.8).
> > > > >> > > > > > > > >>>>
> > > > >> > > > > > > > >>>>
> > > > >> > > > > > > > >> You can also use the Ubuntu Cloud
> Archive, I
> > > > >> still need to
> > > > >> > > get the
> > > > >> > > > > > > docs
> > > > >> > > > > > > > >> updated for that.
> > > > >> > > > > > > > >>
> > > > >> > > > > > > > >> I described the process in a blogpost:
> > > > >> > > > > > > http://blog.widodh.nl/2013/06/****<
> > > http://blog.widodh.nl/2013/06/**>
> > > > >> > > > > > > > >>
> > > > >> > >
> > > > >>
> a-quick-note-on-running-****cloudstack-with-rbd-on-ubuntu-**
> > > > >> **12-04/<
> > > > >> > > > > > >
> > > > >> > > > >
> > > > >> > >
> > > > >>
> http://blog.widodh.nl/2013/06/**a-quick-note-on-running-**
> > > > >> cloudstack-with-rbd-on-ubuntu-**12-04/<
> > >
> http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/
> > > >
> > > > >> > > > > > > >
> > > > >> > > > > > > > >>
> > > > >> > > > > > > > >> indra@hv-kvm-01:~/rbd$ ceph
> > > > >> > > > > > > > >>>> ceph> health
> > > > >> > > > > > > > >>>> HEALTH_OK
> > > > >> > > > > > > > >>>>
> > > > >> > > > > > > > >>>> indra@hv-kvm-01:~$ ceph osd lspools
> > > > >> > > > > > > > >>>> 0 data,1 metadata,2 rbd,3 sc1,
> > > > >> > > > > > > > >>>>
> > > > >> > > > > > > > >>>> root@hv-kvm-01:/home/indra# libvirtd
> > > > >> --version
> > > > >> > > > > > > > >>>> libvirtd (libvirt) 0.9.13
> > > > >> > > > > > > > >>>>
> > > > >> > > > > > > > >>>> I tried to add Primary Storage into
> the
> > > > >> Cloudstack zone
> > > > >> > > which I
> > > > >> > > > > have
> > > > >> > > > > > > > >>>> created:
> > > > >> > > > > > > > >>>>
> > > > >> > > > > > > > >>>> Add Primary Storage:
> > > > >> > > > > > > > >>>>
> > > > >> > > > > > > > >>>> Zone: my zone name
> > > > >> > > > > > > > >>>> Pod: my pod name
> > > > >> > > > > > > > >>>> Cluster: my cluster name
> > > > >> > > > > > > > >>>> Name: ceph-rbd-pri-storage
> > > > >> > > > > > > > >>>> Protocol: RBD
> > > > >> > > > > > > > >>>> RADOS Monitor: my first Ceph monitor
> IP
> > > > >> address
> > > > >> > > > > > > > >>>> RADOS Pool: sc1 (the pool name on Ceph
> > > > >> cluster)
> > > > >> > > > > > > > >>>> RADOS User: client.admin
> > > > >> > > > > > > > >>>> RADOS Secret:
> > > > >> /etc/ceph/ceph.client.admin.****keyring (keyring
> > > > >> > > > > file
> > > > >> > > > > > > > >>>> location)
> > > > >> > > > > > > > >>>>
> > > > >> > > > > > > > >>>
> > > > >> > > > > > > > >> This is your problem. That shouldn't be
> the
> > > > >> location of the
> > > > >> > > file,
> > > > >> > > > > but
> > > > >> > > > > > > it
> > > > >> > > > > > > > >> should be the secret, which is a base64
> > > > >> encoded string.
> > > > >> > > > > > > > >>
> > > > >> > > > > > > > >> $ ceph auth list
> > > > >> > > > > > > > >>
> > > > >> > > > > > > > >> That should tell you what the secret is.
> > > > >> > > > > > > > >>
> > > > >> > > > > > > > >> Storage Tags: rbd
> > > > >> > > > > > > > >>
> > > > >> > > > > > > > >> This is the error message when I tried
> to
> > > add
> > > > >> the primary
> > > > >> > > storage
> > > > >> > > > > by
> > > > >> > > > > > > > >> clicking OK:
> > > > >> > > > > > > > >>
> > > > >> > > > > > > > >> DB Exception on:
> > > > >> > > com.mysql.jdbc.****JDBC4PreparedStatement@4b2eb56
> > > > >> > > > > **:
> > > > >> > > > > > > > >> INSERT INTO
> > > > >> > > > > > > > >> storage_pool (storage_pool.id
> > > > >> <http://storage_pool.id>, storage_pool.name
> > > > >> <http://storage_pool.name>,
> > > > >>
> > > > >> > > > > storage_pool.uuid,
> > > > >> > > > > > > > >> storage_pool.pool_type,
> > > storage_pool.created,
> > > > >> > > > > > > storage_pool.update_time,
> > > > >> > > > > > > > >> storage_pool.data_center_id,
> > > > >> storage_pool.pod_id,
> > > > >> > > > > > > > >> storage_pool.available_bytes,
> > > > >> storage_pool.capacity_bytes,
> > > > >> > > > > > > > >> storage_pool.status, storage_pool.scope,
> > > > >> > > > > > > storage_pool.storage_provider_****
> > > > >> > > > > > > > >> id,
> > > > >> > > > > > > > >> storage_pool.host_address,
> > > storage_pool.path,
> > > > >> > > storage_pool.port,
> > > > >> > > > > > > > >> storage_pool.user_info,
> > > > >> storage_pool.cluster_id) VALUES (217,
> > > > >> > > > > > > > >> _binary'ceph-rbd-pri-storage',
> > > > >> > > > > > > > >>
> > > > >> _binary'a226c9a1-da78-3f3a-****b5ac-e18b925c9634', 'RBD',
> > > > >> > > > > '2013-07-10
> > > > >> > > > > > > > >> 09:08:28', null, 2, 2, 0, 0, 'Up', null,
> > > null,
> > > > >> null,
> > > > >> > > _binary'ceph/
> > > > >> > > > > > > > >> ceph.client.admin.keyring@10.
> > > ****237.11.2/sc1<
> > > > >> > > > > > > http://ceph.client.admin.**
> > > keyring@10.237.11.2/sc1<
> http://ceph.client.admin.keyring@10.237.11.2/sc1>
> > > > >> >',
> > > > >> > > > > > > > >> 6789, null, 2)
> > > > >> > > > > > > > >>
> > > > >> > > > > > > > >> On the management-server.log file:
> > > > >> > > > > > > > >>
> > > > >> > > > > > > > >> 2013-07-10 17:08:28,845 DEBUG
> > > > >> [cloud.api.ApiServlet]
> > > > >> > > > > > > > >> (catalina-exec-2:null)
> > > > >> > > > > > > > >> ===START=== 192.168.0.100 -- GET
> > > > >> > > > > > > > >>
> > > > >>
> > > command=createStoragePool&****zoneid=c116950e-e4ae-4f23-****a7e7-
> > > > >> > > > > > > > >>
> > > > >>
> 74a75c4ee638&podId=a748b063-****3a83-4175-a0e9-de39118fe5ce&**
> > > > >> > > > > > > > >>
> > > > >>
> > > clusterid=1f87eb09-324d-4d49-****83c2-88d84d7a15df&name=ceph-****
> > > > >> > > > > > > > >>
> > > > >>
> rbd-pri-storage&url=rbd%3A%2F%****2Fclient.admin%3A_etc%2Fc
> > > > >> > > > > > > > >>
> > > > >> eph%2Fceph.client.admin.****keyring%4010.237.11.2%2Fsc1&**
> > > > >> > > > > > > > >>
> > > > >> > > > >
> > > > >> tags=rbd&response=json&****sessionkey=****
> > > > >> rDRfWpqeKfQKbKZtHr398ULV%2F8k%****
> > > > >> > > > > > > > >> 3D&_=1373447307839
> > > > >> > > > > > > > >> 2013-07-10 17:08:28,862 DEBUG
> > > > >> > > [cloud.storage.****StorageManagerImpl]
> > > > >> > > > > > > > >> (catalina-exec-2:null) createPool
> Params @
> > > > >> scheme - rbd
> > > > >> > > > > storageHost -
> > > > >> > > > > > > null
> > > > >> > > > > > > > >> hostPath - /ceph/ceph.client
> > > > >> > > > > > > > >> .admin.keyring@10.237.11.2/sc1
> > > > >> <http://admin.keyring@10.237.**11.2/sc1<
> > > http://admin.keyring@10.237.11.2/sc1>>
> > > > >> port - -1
> > > > >>
> > > > >> > > > > > > > >> 2013-07-10 17:08:28,918 DEBUG
> > > > >> > > [cloud.storage.****StorageManagerImpl]
> > > > >> > > > > > > > >> (catalina-exec-2:null) In createPool
> Setting
> > > > >> poolId - 217
> > > > >> > > uuid -
> > > > >> > > > > > > > >>
> a226c9a1-da78-3f3a-b5ac-****e18b925c9634 z
> > > > >> > > > > > > > >> oneId - 2 podId - 2 poolName -
> > > > >> ceph-rbd-pri-storage
> > > > >> > > > > > > > >> 2013-07-10 17:08:28,921 DEBUG
> > > > >> [db.Transaction.Transaction]
> > > > >> > > > > > > > >> (catalina-exec-2:null) Rolling back the
> > > > >> transaction: Time = 3
> > > > >> > > > > Name =
> > > > >> > > > > > > > >> persist; called by -Transaction.rollbac
> > > > >> > > > > > > > >>
> > > > >>
> k:890-Transaction.removeUpTo:****833-Transaction.close:657-**
> > > > >> > > > > > > > >>
> > > > >> TransactionContextBuilder.****interceptException:63-**
> > > > >> > > > > > > > >>
> > > > >> > > > >
> > > > >>
> > > ComponentInstantiationPostProc****essor$InterceptorDispatcher.**
> > > > >> **interce
> > > > >> > > > > > > > >> pt:133-StorageManagerImpl.****
> > > > >> createPool:1378-**
> > > > >> > > > > > > > >>
> > > > >>
> StorageManagerImpl.createPool:****147-CreateStoragePoolCmd.**
> > > > >> > > > > > > > >>
> > > > >> execute:123-ApiDispatcher.****dispatch:162-ApiServer.**
> > > > >> > > > > > > > >> queueCommand:505-ApiSe
> > > > >> > > > > > > > >>
> > > > >> rver.handleRequest:355-****ApiServlet.processRequest:302
> > > > >> > > > > > > > >> 2013-07-10 17:08:28,923 ERROR
> > > > >> [cloud.api.ApiServer]
> > > > >> > > > > > > (catalina-exec-2:null)
> > > > >> > > > > > > > >> unhandled exception executing api
> command:
> > > > >> createStoragePool
> > > > >> > > > > > > > >>
> > > > >> com.cloud.utils.exception.****CloudRuntimeException: DB
> > > > >> > > Exception
> > > > >> > > > > on:
> > > > >> > > > > > > > >>
> > > > >> com.mysql.jdbc.****JDBC4PreparedStatement@4b2eb56****:
> INSERT
> > > > >> > > INTO
> > > > >> > > > > > > > >> storage_pool (
> > > > >> > > > > > > > >> storage_pool.id <http://storage_pool.id
> >,
> > > > >>
> > > > >> storage_pool
> > > > >> > > > > > > > >> .name, storage_pool.uuid,
> > > > >> storage_pool.pool_type,
> > > > >> > > > > > > storage_pool.created,
> > > > >> > > > > > > > >> storage_pool.update_time,
> > > > >> storage_pool.data_center_id,
> > > > >> > > > > > > > >> storage_pool.pod_id,
> > > > >> > > > > > > > >> storage_pool.availab
> > > > >> > > > > > > > >> le_bytes, storage_pool.capacity_bytes,
> > > > >> storage_pool.status,
> > > > >> > > > > > > > >> storage_pool.scope,
> > > > >> storage_pool.storage_provider_****id,
> > > > >> > > > > > > > >> storage_pool.host_address,
> > > storage_pool.path,
> > > > >> storage_
> > > > >> > > > > > > > >> pool.port, storage_pool.user_info,
> > > > >> storage_pool.cluster_id)
> > > > >> > > VALUES
> > > > >> > > > > > > (217,
> > > > >> > > > > > > > >> _binary'ceph-rbd-pri-storage',
> > > > >> > > > > > > > >>
> > > > >> _binary'a226c9a1-da78-3f3a-****b5ac-e18b925c9634', 'RBD',
> > > > >> > > '2013-07-1
> > > > >> > > > > > > > >> 0 09:08:28', null, 2, 2, 0, 0, 'Up',
> null,
> > > > >> null, null,
> > > > >> > > > > _binary'ceph/
> > > > >> > > > > > > > >> ceph.client.admin.keyring@10.
> > > ****237.11.2/sc1<
> > > > >> > > > > > > http://ceph.client.admin.**
> > > keyring@10.237.11.2/sc1<
> http://ceph.client.admin.keyring@10.237.11.2/sc1>
> > > > >> >',
> > > > >> > > > > > > > >> 6789, null, 2)
> > > > >> > > > > > > > >> at
> > > > >> > > > > > > > >>
> > > > >> > > > > > >
> > > > >> > >
> > > > >> com.cloud.utils.db.****GenericDaoBase.persist(****
> > > > >> GenericDaoBase.java:1342)
> > > > >> > > > > > > > >> at
> > > > >> > > > > > > > >>
> > > > >> com.cloud.storage.dao.****StoragePoolDaoImpl.persist(**
> > > > >> > > > > > > > >> StoragePoolDaoImpl.java:232)
> > > > >> > > > > > > > >> at
> > > > >> > > > > > > > >>
> > > > >>
> > > com.cloud.utils.component.****ComponentInstantiationPostProc****
> > > > >> > > > > > > > >>
> > > > >> > > > > > >
> > > > >> > > > >
> > > > >> > >
> > > > >> essor$InterceptorDispatcher.****intercept(****
> > > > >> ComponentInstantiationPostProc***
> > > > >> > > > > > > > >> *es
> > > > >> > > > > > > > >>
> > > > >> > > > > > > > >>
> > > > >> > > > > > >
> > > > >> > > > > > > --
> > > > >> > > > > > > Prasanna.,
> > > > >> > > > > > >
> > > > >> > > > > > > ------------------------
> > > > >> > > > > > > Powered by BigRock.com
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > >
> > > > >> > > > > --
> > > > >> > > > > Prasanna.,
> > > > >> > > > >
> > > > >> > > > > ------------------------
> > > > >> > > > > Powered by BigRock.com
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > >
> > > > >> > > --
> > > > >> > > Prasanna.,
> > > > >> > >
> > > > >> > > ------------------------
> > > > >> > > Powered by BigRock.com
> > > > >> > >
> > > > >> > >
> > > > >>
> > > > >> --
> > > > >> Prasanna.,
> > > > >>
> > > > >> ------------------------
> > > > >> Powered by BigRock.com
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >
> > >
> > > --
> > > Prasanna.,
> > >
> > > ------------------------
> > > Powered by BigRock.com
> > >
> > >
>
> --
> Prasanna.,
>
> ------------------------
> Powered by BigRock.com
>
>
Re: Problem in adding Ceph RBD as primary storage for CloudStack
4.1.0
Posted by Prasanna Santhanam <ts...@apache.org>.
On Sat, Jul 13, 2013 at 02:25:35PM +0800, Indra Pramana wrote:
> Hi Prasanna,
>
> Good day to you, and thank you for your e-mail.
>
Were you able to get beyond this error?
> Yesterday I troubleshooted further and found out that the mapping of the
> bridge is incorrect. For example, eth1 of the SSVM, which is supposed to be
> private network interface for communication with management server, is
> being mapped to cloudbr0 bridge which is used for public. While eth2 of the
> SSVM, which is supposed to be public, is being mapped to cloudbr1 bridge
> which is supposed for guest traffic. I checked with brctl show on the KVM
> host and traced the MAC address of the SSVM's eth interfaces.
>
> I have tried different type of configuration on the KVM hosts'
> /etc/network/interfaces (I am using Ubuntu), different ways on how to
> configure the advanced network zone (physical network interface and KVM
> label for each traffic), but I am still not able to get it working.
>
> Can please advise what is the best way to set-up the network bridges?
Use brctl to add bridges. That's how I've done it.
>
> My /etc/network/interfaces on KVM host:
>
> http://pastebin.com/nx8xJ1L2
>
> I used 3 physical NICs:
>
> eth0 --> for management and secondary storage traffic
> eth1 --> for public and guest traffic
> eth2 --> for primary storage traffic to the Ceph RBD (separate NIC not
> configured by CloudStack)
>
> On the Cloudstack GUI, while creating the zone, I used 3 physical networks:
>
> eth0.6 --> Management traffic (KVM label: eth0.6)
> cloudbr0 --> Public traffic (KVM label: cloudbr0)
> cloudbr1 --> Guest traffic (KVM label: cloudbr1)
>
Is eth0.6 on a tagged VLAN?
> I didn't specify storage traffic since I presume it will use the management
> VLAN.
That's correct.
>
> However, I always failed to add the KVM host using the above configuration.
> I tried to run cloudstack-setup-agent manually on the KVM host and getting
> this error message:
>
> root@hv-kvm-02:/etc/cloudstack/agent# cloudstack-setup-agent -m x.x.x.18 -z
> 3 -p 3 -c 3 -g 00dd5dba-7419-3689-bc51-1671035c0d8f -a --pubNic=cloudbr0
> --prvNic=eth0.6 --guestNic=cloudbr1
> Starting to configure your system:
> Configure Apparmor ... [OK]
> Configure Network ... [Failed]
> eth0.6 is not a bridge
> Try to restore your system:
> Restore Apparmor ... [OK]
> Restore Network ... [OK]
>
> When I tried to create another bridge called cloudbr2 with bridge_ports to
> eth0.6, the KVM host's network didn't work.
>
> Can advise what is the best practice on configuring the network bridges on
> the advanced network zone setup?
>
> Looking forward to your reply, thank you.
>
> Cheers.
>
>
>
> On Sat, Jul 13, 2013 at 12:28 PM, Prasanna Santhanam <ts...@apache.org> wrote:
>
> > See Inline,
> >
> > On Fri, Jul 12, 2013 at 05:12:25PM +0800, Indra Pramana wrote:
> > > Hi Wido,
> > >
> > > Noted, can't wait for 4.2 to be released. :)
> > >
> > > Dear Prasanna, Wido and all,
> > >
> > > I just realised that while the system VMs are running, they are still not
> > > accessible through the public IPs assigned to them. I have been waiting
> > for
> > > the SSVM to download the default CentOS template and it doesn't appear on
> > > the template list.
> > >
> > > I tried to SSH into the SSVM via the local link address from the KVM
> > host,
> > > and running the health check /usr/local/cloud/systemvm/ssvm-check.sh
> > shows
> > > that the VM cannot reach anywhere. It cannot reach the public DNS server
> > (I
> > > used Google's 8.8.8.8), cannot reach the management server, and cannot
> > even
> > > reach the public IP gateway.
> >
> > Ok - we're not there yet. But there's a few troubleshooting steps you
> > can try:
> >
> > 1. Which of the checks fail apart from the DNS ping?
> > 2. Are you able to have those tests pass from the KVM host itself?
> > 3. Can you paste the output of $route -n on your SSVM and on your
> > host?
> >
> > >
> > > Is it due to misconfiguration of the KVM network bridges? How can I see
> > the
> > > mapping between the NIC interfaces of the SSVM (eth0, eth1, eth2 and
> > eth3)
> > > and the actual physical NIC interfaces on the KVM hosts (eth0) and the
> > > network bridges (cloudbr0, cloudbr1)? Any logs I can verify to ensure
> > that
> > > the VLAN and network bridging is working?
> > >
> >
> > You can see this from the dumpxml.
> >
> > $virsh list
> > (gives you all the domains. find the ssvm for example)
> >
> > $virsh dumpxml my-domain > mydomain.xml
> >
> > > Appreciate any advice.
> > >
> > > Thank you.
> > >
> > >
> > >
> > > On Fri, Jul 12, 2013 at 4:19 PM, Wido den Hollander <wi...@widodh.nl>
> > wrote:
> > >
> > > > On 07/12/2013 10:14 AM, Indra Pramana wrote:
> > > >
> > > >> Hi Prasanna,
> > > >>
> > > >> I managed to fix the problem, thanks for your advice to turn the agent
> > > >> log level to debug:
> > > >>
> > > >> https://cwiki.apache.org/**confluence/display/CLOUDSTACK/**
> > > >> KVM+agent+debug<
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug>
> > > >>
> > > >> From the log, I found out that the agent on the KVM host tried to NFS
> > > >> mount directly to 103.25.200.19:/mnt/vol1/sec-**
> > > >> storage/template/tmpl/1/3,
> > > >> which was not allowed by the NFS server due to its default
> > configuration
> > > >> to only allow to mount to /mnt/vol1/sec-storage (the root of the NFS
> > > >> share).
> > > >>
> > > >>
> > > > Ah, that's odd!
> > > >
> > > > Btw, in 4.2 you'll be able to deploy SSVMs on RBD as well, so that
> > > > limitation will be gone.
> > > >
> > > > Wido
> > > >
> > > > After I changed the NFS server configuration to allow mount to all
> > > >> sub-directories, re-export the NFS and voila, the system was able to
> > > >> download the template and now both the system VMs (CPVM and SSVM) are
> > > >> running!
> > > >>
> > > >> Many thanks for your help! :)
> > > >>
> > > >> Cheers.
> > > >>
> > > >>
> > > >>
> > > >> On Fri, Jul 12, 2013 at 3:31 PM, Indra Pramana <indra@sg.or.id
> > > >> <ma...@sg.or.id>> wrote:
> > > >>
> > > >> Hi Prasanna,
> > > >>
> > > >> Good day to you, and thank you for your e-mail.
> > > >>
> > > >> Yes, the cloudstack-agent service is running on both the KVM
> > hosts.
> > > >> There is no "cloud" user being created though, when I installed
> > the
> > > >> agent. I installed the agent as root.
> > > >>
> > > >> root@hv-kvm-01:/home/indra# service cloudstack-agent status
> > > >> * cloud-agent is running
> > > >>
> > > >> root@hv-kvm-01:/home/indra# su - cloud
> > > >> Unknown id: cloud
> > > >>
> > > >> Please advise how can I resolve this problem, shall I create the
> > > >> Unix "cloud" user manually? Basically I follow this instruction to
> > > >> prepare the KVM host and install the CloudStack agent:
> > > >>
> > > >> http://cloudstack.apache.org/**docs/en-US/Apache_CloudStack/**
> > > >> 4.1.0/html/Installation_Guide/**hypervisor-kvm-install-flow.**html<
> > http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.1.0/html/Installation_Guide/hypervisor-kvm-install-flow.html
> > >
> > > >>
> > > >> with this instruction from Wido on how to prepare libvirt with
> > Ceph
> > > >> RBD storage pool support:
> > > >>
> > > >> http://blog.widodh.nl/2013/06/**a-quick-note-on-running-**
> > > >> cloudstack-with-rbd-on-ubuntu-**12-04/<
> > http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/
> > >
> > > >>
> > > >> I also have checked /var/log/cloud/agent/agent.log and I don't see
> > > >> any error messages, except this error message which will show up
> > > >> every time I restart the agent:
> > > >>
> > > >> 2013-07-12 15:22:47,454 ERROR
> > [cloud.resource.**ServerResourceBase]
> > > >> (main:null) Nics are not configured!
> > > >> 2013-07-12 15:22:47,459 INFO
> > [cloud.resource.**ServerResourceBase]
> > > >> (main:null) Designating private to be nic eth0.5
> > > >>
> > > >> More logs can be found here: http://pastebin.com/yeNmCt7S
> > > >>
> > > >> I have configured the network bridges on the NIC interface as per
> > > >> this instruction:
> > > >>
> > > >> http://cloudstack.apache.org/**docs/en-US/Apache_CloudStack/**
> > > >> 4.1.0/html/Installation_Guide/**hypervisor-kvm-install-flow.**
> > > >> html#hypervisor-host-install-**network<
> > http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.1.0/html/Installation_Guide/hypervisor-kvm-install-flow.html#hypervisor-host-install-network
> > >
> > > >>
> > > >> On the zone, I used advanced network configuration with just one
> > > >> physical network for management, public and guest/private. I
> > didn't
> > > >> include storage, which by default the traffic will use the
> > > >> management VLAN network.
> > > >>
> > > >> Please advise if there's anything else I might have been missing.
> > > >>
> > > >> Looking forward to your reply, thank you.
> > > >>
> > > >> Cheers.
> > > >>
> > > >>
> > > >>
> > > >> On Fri, Jul 12, 2013 at 2:56 PM, Prasanna Santhanam <
> > tsp@apache.org
> > > >> <ma...@apache.org>> wrote:
> > > >>
> > > >> Indeed, cloudstack will go through the allocation to startup
> > the
> > > >> system VMs too. So that process is failing to recognize the
> > volume
> > > >> (.qcow2) present on your NFS storage.
> > > >>
> > > >> Can you check if your cloudstack agent service is running with
> > > >> the KVM
> > > >> host? And it should've created the user cloud. $id cloud to
> > check
> > > >> if
> > > >> there is a user.
> > > >>
> > > >> Did you see what's happening in the agent logs? These are
> > under
> > > >> /var/log/cloud/ on your host when the systemVMs are coming up.
> > > >> If the
> > > >> logs are not showing any useful information you can turn on
> > debug
> > > >> level for more verbosity.
> > > >>
> > > >> See here: https://cwiki.apache.org/**confluence/x/FgPMAQ<
> > https://cwiki.apache.org/confluence/x/FgPMAQ>
> > > >>
> > > >> On Fri, Jul 12, 2013 at 02:40:35PM +0800, Indra Pramana wrote:
> > > >> > Hi Prasanna,
> > > >> >
> > > >> > Good day to you, and thank you for your e-mail.
> > > >> >
> > > >> > Yes, when I export the NFS, I set the permission so that
> > > >> normal user will
> > > >> > be able to have read/write access to the files
> > > >> (no_root_squash).
> > > >> >
> > > >> > I have tested and I can have read/write access from my KVM
> > > >> hosts using
> > > >> > normal user. BTW, there's no "cloud" user on the hosts, I
> > > >> believe it's not
> > > >> > created during cloudstack-agent installation?
> > > >> >
> > > >> > In any case, do you think the template issue and the
> > storage
> > > >> pool
> > > >> > allocation issue might be related, or are they two
> > different
> > > >> problems
> > > >> > altogether?
> > > >> >
> > > >> > Looking forward to your reply, thank you.
> > > >> >
> > > >> > Cheers.
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Fri, Jul 12, 2013 at 2:26 PM, Prasanna Santhanam
> > > >> <tsp@apache.org <ma...@apache.org>> wrote:
> > > >> >
> > > >> > > Can you access the file as user root? Or user cloud? The
> > > >> cloudstack
> > > >> > > agent on your KVM host runs as user cloud and the NFS
> > > >> permissions
> > > >> > > might be disallowing the volume (.qcow2) from being
> > accessed.
> > > >> > >
> > > >> > > On Fri, Jul 12, 2013 at 02:16:41PM +0800, Indra Pramana
> > > >> wrote:
> > > >> > > > Hi Prasanna,
> > > >> > > >
> > > >> > > > Good day to you, and thank you for your e-mail.
> > > >> > > >
> > > >> > > > Yes, the file exists. I can access the file from the
> > > >> management server
> > > >> > > and
> > > >> > > > the two hypervisors hosts if I mount manually.
> > > >> > > >
> > > >> > > > [root@cs-nas-01/mnt/vol1/sec-storage/**template/tmpl/1/3]#
> > > >> ls
> > > >> > > > -la
> > > >> > > > total
> > > >> > > > 1418787
> > > >> > > > drwxr-xr-x 2 root wheel 4 Jul 11 20:21
> > > >> > > > .
> > > >> > > > drwxr-xr-x 3 root wheel 3 Jul 11 20:17
> > > >> > > > ..
> > > >> > > > -rw-r--r-- 1 root wheel 725811200 Jul 11 20:21
> > > >> > > > 425b9e5a-fbc7-4637-a33a-**fe9d0ed4fa98.qcow2
> > > >> > > > -rw-r--r-- 1 root wheel 295 Jul 11 20:21
> > > >> > > > template.properties
> > > >> > > > [root@cs-nas-01 /mnt/vol1/sec-storage/**
> > > >> template/tmpl/1/3]#
> > > >> > > > pwd
> > > >> > > > /mnt/vol1/sec-storage/**template/tmpl/1/3
> > > >> > > >
> > > >> > > >
> > > >> > > > Any advise?
> > > >> > > >
> > > >> > > > Looking forward to your reply, thank you.
> > > >> > > >
> > > >> > > > Cheers.
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > > On Fri, Jul 12, 2013 at 2:07 PM, Prasanna Santhanam
> > > >> <tsp@apache.org <ma...@apache.org>>
> > > >>
> > > >> > > wrote:
> > > >> > > >
> > > >> > > > > Can you check whether there is a file at:
> > > >> > > > >
> > > >> nfs://103.25.200.19/mnt/vol1/**sec-storage/template/tmpl/1/3/
> > <http://103.25.200.19/mnt/vol1/sec-storage/template/tmpl/1/3/>
> > > >> <
> > http://103.25.200.19/mnt/**vol1/sec-storage/template/**tmpl/1/3/<
> > http://103.25.200.19/mnt/vol1/sec-storage/template/tmpl/1/3/>
> > > >> >
> > > >>
> > > >> > > > >
> > > >> > > > > On Fri, Jul 12, 2013 at 01:59:34PM +0800, Indra
> > Pramana
> > > >> wrote:
> > > >> > > > > > Hi Prasanna,
> > > >> > > > > >
> > > >> > > > > > Thanks for your e-mail.
> > > >> > > > > >
> > > >> > > > > > I have tried restarting the management server, and
> > > >> the problem still
> > > >> > > > > > persists. I even tried to re-do the installation
> > and
> > > >> configuration
> > > >> > > again
> > > >> > > > > > from scratch last night, but the problem still
> > there.
> > > >> > > > > >
> > > >> > > > > > I also noted that on the beginning of the logs, I
> > > >> found some error
> > > >> > > > > messages
> > > >> > > > > > saying that the template cannot be downloaded to
> > the
> > > >> pool. See this
> > > >> > > logs:
> > > >> > > > > >
> > > >> > > > > > http://pastebin.com/BY1AVJ08
> > > >> > > > > >
> > > >> > > > > > It says it failed because cannot get volume from
> > the
> > > >> pool. Could it
> > > >> > > be
> > > >> > > > > > related, i.e. the absence of the template caused
> > the
> > > >> system VMs
> > > >> > > cannot be
> > > >> > > > > > created and started?
> > > >> > > > > >
> > > >> > > > > > I have ensured that I downloaded the system VM
> > > >> template using
> > > >> > > > > > cloud-install-sys-tmplt and verified that the
> > > >> template is already
> > > >> > > there
> > > >> > > > > in
> > > >> > > > > > the secondary storage server.
> > > >> > > > > >
> > > >> > > > > > Any advice is appreciated.
> > > >> > > > > >
> > > >> > > > > > Looking forward to your reply, thank you.
> > > >> > > > > >
> > > >> > > > > > Cheers.
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > On Fri, Jul 12, 2013 at 1:21 PM, Prasanna Santhanam
> > > >> <tsp@apache.org <ma...@apache.org>>
> > > >>
> > > >> > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > It looks like a previous attempt to start the
> > > >> systemVMs has failed
> > > >> > > > > > > putting the nfs storage in the avoid set. Did you
> > > >> try restarting
> > > >> > > your
> > > >> > > > > > > management server?
> > > >> > > > > > >
> > > >> > > > > > > This line leads me to the above mentioned:
> > > >> > > > > > > 2013-07-12 13:10:48,236 DEBUG
> > > >> > > > > > >
> > [storage.allocator.**AbstractStoragePoolAllocator]
> > > >> > > (secstorage-1:null)
> > > >> > > > > > > StoragePool is in avoid set, skipping this pool
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > On Fri, Jul 12, 2013 at 01:16:53PM +0800, Indra
> > > >> Pramana wrote:
> > > >> > > > > > > > Dear Wido and all,
> > > >> > > > > > > >
> > > >> > > > > > > > I have managed to get the hosts, primary and
> > > >> secondary storage
> > > >> > > > > running.
> > > >> > > > > > > >
> > > >> > > > > > > > - 2 KVM hypervisor hosts
> > > >> > > > > > > > - One RBD primary storage
> > > >> > > > > > > > - One NFS primary storage (for system VMs,
> > since
> > > >> I understand
> > > >> > > that
> > > >> > > > > system
> > > >> > > > > > > > VMs cannot use RBD)
> > > >> > > > > > > > - One NFS secondary storage
> > > >> > > > > > > >
> > > >> > > > > > > > However, now I am having problem with the
> > system
> > > >> VMs: CPVM and
> > > >> > > SSVM,
> > > >> > > > > > > unable
> > > >> > > > > > > > to start.
> > > >> > > > > > > >
> > > >> > > > > > > > Excerpt from management-server.log file is
> > here:
> > > >> > > > > > > > http://pastebin.com/ENkpCALY
> > > >> > > > > > > >
> > > >> > > > > > > > It seems that the VMs were not able to be
> > created
> > > >> because unable
> > > >> > > to
> > > >> > > > > find
> > > >> > > > > > > > suitable StoragePools.
> > > >> > > > > > > >
> > > >> > > > > > > > I understand that system VMs will be using the
> > > >> NFS primary
> > > >> > > storage
> > > >> > > > > > > instead
> > > >> > > > > > > > of RBD, so I have confirmed that I am able to
> > > >> mount the primary
> > > >> > > > > storage
> > > >> > > > > > > via
> > > >> > > > > > > > NFS and have read and write access, from both
> > the
> > > >> hypervisor and
> > > >> > > the
> > > >> > > > > > > > management server.
> > > >> > > > > > > >
> > > >> > > > > > > > Any advise how can I resolve the problem to
> > make
> > > >> both the system
> > > >> > > VMs
> > > >> > > > > > > > created and started?
> > > >> > > > > > > >
> > > >> > > > > > > > Looking forward to your reply, thank you.
> > > >> > > > > > > >
> > > >> > > > > > > > Cheers.
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > On Fri, Jul 12, 2013 at 9:43 AM, Indra Pramana
> > > >> <indra@sg.or.id <ma...@sg.or.id>>
> > > >>
> > > >> > > > > wrote:
> > > >> > > > > > > >
> > > >> > > > > > > > > Hi Wido,
> > > >> > > > > > > > >
> > > >> > > > > > > > > Thanks for the advice, I'm now able to add
> > the
> > > >> RBD pool as
> > > >> > > primary
> > > >> > > > > > > storage.
> > > >> > > > > > > > >
> > > >> > > > > > > > > Many thanks! :)
> > > >> > > > > > > > >
> > > >> > > > > > > > > Cheers.
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > On Thursday, July 11, 2013, Wido den
> > Hollander
> > > >> wrote:
> > > >> > > > > > > > >
> > > >> > > > > > > > >> Hi,
> > > >> > > > > > > > >>
> > > >> > > > > > > > >> On 07/10/2013 03:42 PM, Chip Childers wrote:
> > > >> > > > > > > > >>
> > > >> > > > > > > > >>> Cc'ing Wido, our resident Ceph expert. ;-)
> > > >> > > > > > > > >>>
> > > >> > > > > > > > >>>
> > > >> > > > > > > > >> Hehe ;)
> > > >> > > > > > > > >>
> > > >> > > > > > > > >> On Wed, Jul 10, 2013 at 05:45:25PM +0800,
> > > >> Indra Pramana
> > > >> > > wrote:
> > > >> > > > > > > > >>>
> > > >> > > > > > > > >>>> Dear all,
> > > >> > > > > > > > >>>>
> > > >> > > > > > > > >>>> I am installing CloudStack 4.1.0 (upgraded
> > > >> from 4.0.2) and I
> > > >> > > > > also
> > > >> > > > > > > have a
> > > >> > > > > > > > >>>> Ceph cluster running. However, I am having
> > > >> issues in adding
> > > >> > > the
> > > >> > > > > RBD
> > > >> > > > > > > as
> > > >> > > > > > > > >>>> primary storage. Tried to follow the
> > > >> instruction here, but
> > > >> > > > > unable to
> > > >> > > > > > > > >>>> make
> > > >> > > > > > > > >>>> it work:
> > > >> > > > > > > > >>>>
> > > >> > > > > > > > >>>>
> > > >> http://ceph.com/docs/master/****rbd/rbd-cloudstack/<
> > http://ceph.com/docs/master/**rbd/rbd-cloudstack/>
> > > >> <
> > > >> > > > > > >
> > http://ceph.com/docs/master/**rbd/rbd-cloudstack/<
> > http://ceph.com/docs/master/rbd/rbd-cloudstack/>
> > > >> >
> > > >> > > > > > > > >>>>
> > > >> > > > > > > > >>>> I have setup a pool on the Ceph cluster.
> > The
> > > >> status of the
> > > >> > > > > cluster
> > > >> > > > > > > is
> > > >> > > > > > > > >>>> healthy. Since I am using Ubuntu 12.04.2
> > LTS
> > > >> (Precise) for
> > > >> > > the
> > > >> > > > > > > > >>>> hypervisors,
> > > >> > > > > > > > >>>> I also have compiled libvirt manually to
> > > >> ensure that the
> > > >> > > version
> > > >> > > > > > > 0.9.13
> > > >> > > > > > > > >>>> is
> > > >> > > > > > > > >>>> installed (previously it's 0.9.8).
> > > >> > > > > > > > >>>>
> > > >> > > > > > > > >>>>
> > > >> > > > > > > > >> You can also use the Ubuntu Cloud Archive, I
> > > >> still need to
> > > >> > > get the
> > > >> > > > > > > docs
> > > >> > > > > > > > >> updated for that.
> > > >> > > > > > > > >>
> > > >> > > > > > > > >> I described the process in a blogpost:
> > > >> > > > > > > http://blog.widodh.nl/2013/06/****<
> > http://blog.widodh.nl/2013/06/**>
> > > >> > > > > > > > >>
> > > >> > >
> > > >> a-quick-note-on-running-****cloudstack-with-rbd-on-ubuntu-**
> > > >> **12-04/<
> > > >> > > > > > >
> > > >> > > > >
> > > >> > >
> > > >> http://blog.widodh.nl/2013/06/**a-quick-note-on-running-**
> > > >> cloudstack-with-rbd-on-ubuntu-**12-04/<
> > http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/
> > >
> > > >> > > > > > > >
> > > >> > > > > > > > >>
> > > >> > > > > > > > >> indra@hv-kvm-01:~/rbd$ ceph
> > > >> > > > > > > > >>>> ceph> health
> > > >> > > > > > > > >>>> HEALTH_OK
> > > >> > > > > > > > >>>>
> > > >> > > > > > > > >>>> indra@hv-kvm-01:~$ ceph osd lspools
> > > >> > > > > > > > >>>> 0 data,1 metadata,2 rbd,3 sc1,
> > > >> > > > > > > > >>>>
> > > >> > > > > > > > >>>> root@hv-kvm-01:/home/indra# libvirtd
> > > >> --version
> > > >> > > > > > > > >>>> libvirtd (libvirt) 0.9.13
> > > >> > > > > > > > >>>>
> > > >> > > > > > > > >>>> I tried to add Primary Storage into the
> > > >> Cloudstack zone
> > > >> > > which I
> > > >> > > > > have
> > > >> > > > > > > > >>>> created:
> > > >> > > > > > > > >>>>
> > > >> > > > > > > > >>>> Add Primary Storage:
> > > >> > > > > > > > >>>>
> > > >> > > > > > > > >>>> Zone: my zone name
> > > >> > > > > > > > >>>> Pod: my pod name
> > > >> > > > > > > > >>>> Cluster: my cluster name
> > > >> > > > > > > > >>>> Name: ceph-rbd-pri-storage
> > > >> > > > > > > > >>>> Protocol: RBD
> > > >> > > > > > > > >>>> RADOS Monitor: my first Ceph monitor IP
> > > >> address
> > > >> > > > > > > > >>>> RADOS Pool: sc1 (the pool name on Ceph
> > > >> cluster)
> > > >> > > > > > > > >>>> RADOS User: client.admin
> > > >> > > > > > > > >>>> RADOS Secret:
> > > >> /etc/ceph/ceph.client.admin.****keyring (keyring
> > > >> > > > > file
> > > >> > > > > > > > >>>> location)
> > > >> > > > > > > > >>>>
> > > >> > > > > > > > >>>
> > > >> > > > > > > > >> This is your problem. That shouldn't be the
> > > >> location of the
> > > >> > > file,
> > > >> > > > > but
> > > >> > > > > > > it
> > > >> > > > > > > > >> should be the secret, which is a base64
> > > >> encoded string.
> > > >> > > > > > > > >>
> > > >> > > > > > > > >> $ ceph auth list
> > > >> > > > > > > > >>
> > > >> > > > > > > > >> That should tell you what the secret is.
> > > >> > > > > > > > >>
> > > >> > > > > > > > >> Storage Tags: rbd
> > > >> > > > > > > > >>
> > > >> > > > > > > > >> This is the error message when I tried to
> > add
> > > >> the primary
> > > >> > > storage
> > > >> > > > > by
> > > >> > > > > > > > >> clicking OK:
> > > >> > > > > > > > >>
> > > >> > > > > > > > >> DB Exception on:
> > > >> > > com.mysql.jdbc.****JDBC4PreparedStatement@4b2eb56
> > > >> > > > > **:
> > > >> > > > > > > > >> INSERT INTO
> > > >> > > > > > > > >> storage_pool (storage_pool.id
> > > >> <http://storage_pool.id>, storage_pool.name
> > > >> <http://storage_pool.name>,
> > > >>
> > > >> > > > > storage_pool.uuid,
> > > >> > > > > > > > >> storage_pool.pool_type,
> > storage_pool.created,
> > > >> > > > > > > storage_pool.update_time,
> > > >> > > > > > > > >> storage_pool.data_center_id,
> > > >> storage_pool.pod_id,
> > > >> > > > > > > > >> storage_pool.available_bytes,
> > > >> storage_pool.capacity_bytes,
> > > >> > > > > > > > >> storage_pool.status, storage_pool.scope,
> > > >> > > > > > > storage_pool.storage_provider_****
> > > >> > > > > > > > >> id,
> > > >> > > > > > > > >> storage_pool.host_address,
> > storage_pool.path,
> > > >> > > storage_pool.port,
> > > >> > > > > > > > >> storage_pool.user_info,
> > > >> storage_pool.cluster_id) VALUES (217,
> > > >> > > > > > > > >> _binary'ceph-rbd-pri-storage',
> > > >> > > > > > > > >>
> > > >> _binary'a226c9a1-da78-3f3a-****b5ac-e18b925c9634', 'RBD',
> > > >> > > > > '2013-07-10
> > > >> > > > > > > > >> 09:08:28', null, 2, 2, 0, 0, 'Up', null,
> > null,
> > > >> null,
> > > >> > > _binary'ceph/
> > > >> > > > > > > > >> ceph.client.admin.keyring@10.
> > ****237.11.2/sc1<
> > > >> > > > > > > http://ceph.client.admin.**
> > keyring@10.237.11.2/sc1<http://ceph.client.admin.keyring@10.237.11.2/sc1>
> > > >> >',
> > > >> > > > > > > > >> 6789, null, 2)
> > > >> > > > > > > > >>
> > > >> > > > > > > > >> On the management-server.log file:
> > > >> > > > > > > > >>
> > > >> > > > > > > > >> 2013-07-10 17:08:28,845 DEBUG
> > > >> [cloud.api.ApiServlet]
> > > >> > > > > > > > >> (catalina-exec-2:null)
> > > >> > > > > > > > >> ===START=== 192.168.0.100 -- GET
> > > >> > > > > > > > >>
> > > >>
> > command=createStoragePool&****zoneid=c116950e-e4ae-4f23-****a7e7-
> > > >> > > > > > > > >>
> > > >> 74a75c4ee638&podId=a748b063-****3a83-4175-a0e9-de39118fe5ce&**
> > > >> > > > > > > > >>
> > > >>
> > clusterid=1f87eb09-324d-4d49-****83c2-88d84d7a15df&name=ceph-****
> > > >> > > > > > > > >>
> > > >> rbd-pri-storage&url=rbd%3A%2F%****2Fclient.admin%3A_etc%2Fc
> > > >> > > > > > > > >>
> > > >> eph%2Fceph.client.admin.****keyring%4010.237.11.2%2Fsc1&**
> > > >> > > > > > > > >>
> > > >> > > > >
> > > >> tags=rbd&response=json&****sessionkey=****
> > > >> rDRfWpqeKfQKbKZtHr398ULV%2F8k%****
> > > >> > > > > > > > >> 3D&_=1373447307839
> > > >> > > > > > > > >> 2013-07-10 17:08:28,862 DEBUG
> > > >> > > [cloud.storage.****StorageManagerImpl]
> > > >> > > > > > > > >> (catalina-exec-2:null) createPool Params @
> > > >> scheme - rbd
> > > >> > > > > storageHost -
> > > >> > > > > > > null
> > > >> > > > > > > > >> hostPath - /ceph/ceph.client
> > > >> > > > > > > > >> .admin.keyring@10.237.11.2/sc1
> > > >> <http://admin.keyring@10.237.**11.2/sc1<
> > http://admin.keyring@10.237.11.2/sc1>>
> > > >> port - -1
> > > >>
> > > >> > > > > > > > >> 2013-07-10 17:08:28,918 DEBUG
> > > >> > > [cloud.storage.****StorageManagerImpl]
> > > >> > > > > > > > >> (catalina-exec-2:null) In createPool Setting
> > > >> poolId - 217
> > > >> > > uuid -
> > > >> > > > > > > > >> a226c9a1-da78-3f3a-b5ac-****e18b925c9634 z
> > > >> > > > > > > > >> oneId - 2 podId - 2 poolName -
> > > >> ceph-rbd-pri-storage
> > > >> > > > > > > > >> 2013-07-10 17:08:28,921 DEBUG
> > > >> [db.Transaction.Transaction]
> > > >> > > > > > > > >> (catalina-exec-2:null) Rolling back the
> > > >> transaction: Time = 3
> > > >> > > > > Name =
> > > >> > > > > > > > >> persist; called by -Transaction.rollbac
> > > >> > > > > > > > >>
> > > >> k:890-Transaction.removeUpTo:****833-Transaction.close:657-**
> > > >> > > > > > > > >>
> > > >> TransactionContextBuilder.****interceptException:63-**
> > > >> > > > > > > > >>
> > > >> > > > >
> > > >>
> > ComponentInstantiationPostProc****essor$InterceptorDispatcher.**
> > > >> **interce
> > > >> > > > > > > > >> pt:133-StorageManagerImpl.****
> > > >> createPool:1378-**
> > > >> > > > > > > > >>
> > > >> StorageManagerImpl.createPool:****147-CreateStoragePoolCmd.**
> > > >> > > > > > > > >>
> > > >> execute:123-ApiDispatcher.****dispatch:162-ApiServer.**
> > > >> > > > > > > > >> queueCommand:505-ApiSe
> > > >> > > > > > > > >>
> > > >> rver.handleRequest:355-****ApiServlet.processRequest:302
> > > >> > > > > > > > >> 2013-07-10 17:08:28,923 ERROR
> > > >> [cloud.api.ApiServer]
> > > >> > > > > > > (catalina-exec-2:null)
> > > >> > > > > > > > >> unhandled exception executing api command:
> > > >> createStoragePool
> > > >> > > > > > > > >>
> > > >> com.cloud.utils.exception.****CloudRuntimeException: DB
> > > >> > > Exception
> > > >> > > > > on:
> > > >> > > > > > > > >>
> > > >> com.mysql.jdbc.****JDBC4PreparedStatement@4b2eb56****: INSERT
> > > >> > > INTO
> > > >> > > > > > > > >> storage_pool (
> > > >> > > > > > > > >> storage_pool.id <http://storage_pool.id>,
> > > >>
> > > >> storage_pool
> > > >> > > > > > > > >> .name, storage_pool.uuid,
> > > >> storage_pool.pool_type,
> > > >> > > > > > > storage_pool.created,
> > > >> > > > > > > > >> storage_pool.update_time,
> > > >> storage_pool.data_center_id,
> > > >> > > > > > > > >> storage_pool.pod_id,
> > > >> > > > > > > > >> storage_pool.availab
> > > >> > > > > > > > >> le_bytes, storage_pool.capacity_bytes,
> > > >> storage_pool.status,
> > > >> > > > > > > > >> storage_pool.scope,
> > > >> storage_pool.storage_provider_****id,
> > > >> > > > > > > > >> storage_pool.host_address,
> > storage_pool.path,
> > > >> storage_
> > > >> > > > > > > > >> pool.port, storage_pool.user_info,
> > > >> storage_pool.cluster_id)
> > > >> > > VALUES
> > > >> > > > > > > (217,
> > > >> > > > > > > > >> _binary'ceph-rbd-pri-storage',
> > > >> > > > > > > > >>
> > > >> _binary'a226c9a1-da78-3f3a-****b5ac-e18b925c9634', 'RBD',
> > > >> > > '2013-07-1
> > > >> > > > > > > > >> 0 09:08:28', null, 2, 2, 0, 0, 'Up', null,
> > > >> null, null,
> > > >> > > > > _binary'ceph/
> > > >> > > > > > > > >> ceph.client.admin.keyring@10.
> > ****237.11.2/sc1<
> > > >> > > > > > > http://ceph.client.admin.**
> > keyring@10.237.11.2/sc1<http://ceph.client.admin.keyring@10.237.11.2/sc1>
> > > >> >',
> > > >> > > > > > > > >> 6789, null, 2)
> > > >> > > > > > > > >> at
> > > >> > > > > > > > >>
> > > >> > > > > > >
> > > >> > >
> > > >> com.cloud.utils.db.****GenericDaoBase.persist(****
> > > >> GenericDaoBase.java:1342)
> > > >> > > > > > > > >> at
> > > >> > > > > > > > >>
> > > >> com.cloud.storage.dao.****StoragePoolDaoImpl.persist(**
> > > >> > > > > > > > >> StoragePoolDaoImpl.java:232)
> > > >> > > > > > > > >> at
> > > >> > > > > > > > >>
> > > >>
> > com.cloud.utils.component.****ComponentInstantiationPostProc****
> > > >> > > > > > > > >>
> > > >> > > > > > >
> > > >> > > > >
> > > >> > >
> > > >> essor$InterceptorDispatcher.****intercept(****
> > > >> ComponentInstantiationPostProc***
> > > >> > > > > > > > >> *es
> > > >> > > > > > > > >>
> > > >> > > > > > > > >>
> > > >> > > > > > >
> > > >> > > > > > > --
> > > >> > > > > > > Prasanna.,
> > > >> > > > > > >
> > > >> > > > > > > ------------------------
> > > >> > > > > > > Powered by BigRock.com
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > >
> > > >> > > > > --
> > > >> > > > > Prasanna.,
> > > >> > > > >
> > > >> > > > > ------------------------
> > > >> > > > > Powered by BigRock.com
> > > >> > > > >
> > > >> > > > >
> > > >> > >
> > > >> > > --
> > > >> > > Prasanna.,
> > > >> > >
> > > >> > > ------------------------
> > > >> > > Powered by BigRock.com
> > > >> > >
> > > >> > >
> > > >>
> > > >> --
> > > >> Prasanna.,
> > > >>
> > > >> ------------------------
> > > >> Powered by BigRock.com
> > > >>
> > > >>
> > > >>
> > > >>
> > > >
> >
> > --
> > Prasanna.,
> >
> > ------------------------
> > Powered by BigRock.com
> >
> >
--
Prasanna.,
------------------------
Powered by BigRock.com
Re: Problem in adding Ceph RBD as primary storage for CloudStack 4.1.0
Posted by Indra Pramana <in...@sg.or.id>.
Hi Prasanna,
Good day to you, and thank you for your e-mail.
Yesterday I troubleshooted further and found out that the mapping of the
bridge is incorrect. For example, eth1 of the SSVM, which is supposed to be
private network interface for communication with management server, is
being mapped to cloudbr0 bridge which is used for public. While eth2 of the
SSVM, which is supposed to be public, is being mapped to cloudbr1 bridge
which is supposed for guest traffic. I checked with brctl show on the KVM
host and traced the MAC address of the SSVM's eth interfaces.
I have tried different type of configuration on the KVM hosts'
/etc/network/interfaces (I am using Ubuntu), different ways on how to
configure the advanced network zone (physical network interface and KVM
label for each traffic), but I am still not able to get it working.
Can please advise what is the best way to set-up the network bridges?
My /etc/network/interfaces on KVM host:
http://pastebin.com/nx8xJ1L2
I used 3 physical NICs:
eth0 --> for management and secondary storage traffic
eth1 --> for public and guest traffic
eth2 --> for primary storage traffic to the Ceph RBD (separate NIC not
configured by CloudStack)
On the Cloudstack GUI, while creating the zone, I used 3 physical networks:
eth0.6 --> Management traffic (KVM label: eth0.6)
cloudbr0 --> Public traffic (KVM label: cloudbr0)
cloudbr1 --> Guest traffic (KVM label: cloudbr1)
I didn't specify storage traffic since I presume it will use the management
VLAN.
However, I always failed to add the KVM host using the above configuration.
I tried to run cloudstack-setup-agent manually on the KVM host and getting
this error message:
root@hv-kvm-02:/etc/cloudstack/agent# cloudstack-setup-agent -m x.x.x.18 -z
3 -p 3 -c 3 -g 00dd5dba-7419-3689-bc51-1671035c0d8f -a --pubNic=cloudbr0
--prvNic=eth0.6 --guestNic=cloudbr1
Starting to configure your system:
Configure Apparmor ... [OK]
Configure Network ... [Failed]
eth0.6 is not a bridge
Try to restore your system:
Restore Apparmor ... [OK]
Restore Network ... [OK]
When I tried to create another bridge called cloudbr2 with bridge_ports to
eth0.6, the KVM host's network didn't work.
Can advise what is the best practice on configuring the network bridges on
the advanced network zone setup?
Looking forward to your reply, thank you.
Cheers.
On Sat, Jul 13, 2013 at 12:28 PM, Prasanna Santhanam <ts...@apache.org> wrote:
> See Inline,
>
> On Fri, Jul 12, 2013 at 05:12:25PM +0800, Indra Pramana wrote:
> > Hi Wido,
> >
> > Noted, can't wait for 4.2 to be released. :)
> >
> > Dear Prasanna, Wido and all,
> >
> > I just realised that while the system VMs are running, they are still not
> > accessible through the public IPs assigned to them. I have been waiting
> for
> > the SSVM to download the default CentOS template and it doesn't appear on
> > the template list.
> >
> > I tried to SSH into the SSVM via the local link address from the KVM
> host,
> > and running the health check /usr/local/cloud/systemvm/ssvm-check.sh
> shows
> > that the VM cannot reach anywhere. It cannot reach the public DNS server
> (I
> > used Google's 8.8.8.8), cannot reach the management server, and cannot
> even
> > reach the public IP gateway.
>
> Ok - we're not there yet. But there's a few troubleshooting steps you
> can try:
>
> 1. Which of the checks fail apart from the DNS ping?
> 2. Are you able to have those tests pass from the KVM host itself?
> 3. Can you paste the output of $route -n on your SSVM and on your
> host?
>
> >
> > Is it due to misconfiguration of the KVM network bridges? How can I see
> the
> > mapping between the NIC interfaces of the SSVM (eth0, eth1, eth2 and
> eth3)
> > and the actual physical NIC interfaces on the KVM hosts (eth0) and the
> > network bridges (cloudbr0, cloudbr1)? Any logs I can verify to ensure
> that
> > the VLAN and network bridging is working?
> >
>
> You can see this from the dumpxml.
>
> $virsh list
> (gives you all the domains. find the ssvm for example)
>
> $virsh dumpxml my-domain > mydomain.xml
>
> > Appreciate any advice.
> >
> > Thank you.
> >
> >
> >
> > On Fri, Jul 12, 2013 at 4:19 PM, Wido den Hollander <wi...@widodh.nl>
> wrote:
> >
> > > On 07/12/2013 10:14 AM, Indra Pramana wrote:
> > >
> > >> Hi Prasanna,
> > >>
> > >> I managed to fix the problem, thanks for your advice to turn the agent
> > >> log level to debug:
> > >>
> > >> https://cwiki.apache.org/**confluence/display/CLOUDSTACK/**
> > >> KVM+agent+debug<
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug>
> > >>
> > >> From the log, I found out that the agent on the KVM host tried to NFS
> > >> mount directly to 103.25.200.19:/mnt/vol1/sec-**
> > >> storage/template/tmpl/1/3,
> > >> which was not allowed by the NFS server due to its default
> configuration
> > >> to only allow to mount to /mnt/vol1/sec-storage (the root of the NFS
> > >> share).
> > >>
> > >>
> > > Ah, that's odd!
> > >
> > > Btw, in 4.2 you'll be able to deploy SSVMs on RBD as well, so that
> > > limitation will be gone.
> > >
> > > Wido
> > >
> > > After I changed the NFS server configuration to allow mount to all
> > >> sub-directories, re-export the NFS and voila, the system was able to
> > >> download the template and now both the system VMs (CPVM and SSVM) are
> > >> running!
> > >>
> > >> Many thanks for your help! :)
> > >>
> > >> Cheers.
> > >>
> > >>
> > >>
> > >> On Fri, Jul 12, 2013 at 3:31 PM, Indra Pramana <indra@sg.or.id
> > >> <ma...@sg.or.id>> wrote:
> > >>
> > >> Hi Prasanna,
> > >>
> > >> Good day to you, and thank you for your e-mail.
> > >>
> > >> Yes, the cloudstack-agent service is running on both the KVM
> hosts.
> > >> There is no "cloud" user being created though, when I installed
> the
> > >> agent. I installed the agent as root.
> > >>
> > >> root@hv-kvm-01:/home/indra# service cloudstack-agent status
> > >> * cloud-agent is running
> > >>
> > >> root@hv-kvm-01:/home/indra# su - cloud
> > >> Unknown id: cloud
> > >>
> > >> Please advise how can I resolve this problem, shall I create the
> > >> Unix "cloud" user manually? Basically I follow this instruction to
> > >> prepare the KVM host and install the CloudStack agent:
> > >>
> > >> http://cloudstack.apache.org/**docs/en-US/Apache_CloudStack/**
> > >> 4.1.0/html/Installation_Guide/**hypervisor-kvm-install-flow.**html<
> http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.1.0/html/Installation_Guide/hypervisor-kvm-install-flow.html
> >
> > >>
> > >> with this instruction from Wido on how to prepare libvirt with
> Ceph
> > >> RBD storage pool support:
> > >>
> > >> http://blog.widodh.nl/2013/06/**a-quick-note-on-running-**
> > >> cloudstack-with-rbd-on-ubuntu-**12-04/<
> http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/
> >
> > >>
> > >> I also have checked /var/log/cloud/agent/agent.log and I don't see
> > >> any error messages, except this error message which will show up
> > >> every time I restart the agent:
> > >>
> > >> 2013-07-12 15:22:47,454 ERROR
> [cloud.resource.**ServerResourceBase]
> > >> (main:null) Nics are not configured!
> > >> 2013-07-12 15:22:47,459 INFO
> [cloud.resource.**ServerResourceBase]
> > >> (main:null) Designating private to be nic eth0.5
> > >>
> > >> More logs can be found here: http://pastebin.com/yeNmCt7S
> > >>
> > >> I have configured the network bridges on the NIC interface as per
> > >> this instruction:
> > >>
> > >> http://cloudstack.apache.org/**docs/en-US/Apache_CloudStack/**
> > >> 4.1.0/html/Installation_Guide/**hypervisor-kvm-install-flow.**
> > >> html#hypervisor-host-install-**network<
> http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.1.0/html/Installation_Guide/hypervisor-kvm-install-flow.html#hypervisor-host-install-network
> >
> > >>
> > >> On the zone, I used advanced network configuration with just one
> > >> physical network for management, public and guest/private. I
> didn't
> > >> include storage, which by default the traffic will use the
> > >> management VLAN network.
> > >>
> > >> Please advise if there's anything else I might have been missing.
> > >>
> > >> Looking forward to your reply, thank you.
> > >>
> > >> Cheers.
> > >>
> > >>
> > >>
> > >> On Fri, Jul 12, 2013 at 2:56 PM, Prasanna Santhanam <
> tsp@apache.org
> > >> <ma...@apache.org>> wrote:
> > >>
> > >> Indeed, cloudstack will go through the allocation to startup
> the
> > >> system VMs too. So that process is failing to recognize the
> volume
> > >> (.qcow2) present on your NFS storage.
> > >>
> > >> Can you check if your cloudstack agent service is running with
> > >> the KVM
> > >> host? And it should've created the user cloud. $id cloud to
> check
> > >> if
> > >> there is a user.
> > >>
> > >> Did you see what's happening in the agent logs? These are
> under
> > >> /var/log/cloud/ on your host when the systemVMs are coming up.
> > >> If the
> > >> logs are not showing any useful information you can turn on
> debug
> > >> level for more verbosity.
> > >>
> > >> See here: https://cwiki.apache.org/**confluence/x/FgPMAQ<
> https://cwiki.apache.org/confluence/x/FgPMAQ>
> > >>
> > >> On Fri, Jul 12, 2013 at 02:40:35PM +0800, Indra Pramana wrote:
> > >> > Hi Prasanna,
> > >> >
> > >> > Good day to you, and thank you for your e-mail.
> > >> >
> > >> > Yes, when I export the NFS, I set the permission so that
> > >> normal user will
> > >> > be able to have read/write access to the files
> > >> (no_root_squash).
> > >> >
> > >> > I have tested and I can have read/write access from my KVM
> > >> hosts using
> > >> > normal user. BTW, there's no "cloud" user on the hosts, I
> > >> believe it's not
> > >> > created during cloudstack-agent installation?
> > >> >
> > >> > In any case, do you think the template issue and the
> storage
> > >> pool
> > >> > allocation issue might be related, or are they two
> different
> > >> problems
> > >> > altogether?
> > >> >
> > >> > Looking forward to your reply, thank you.
> > >> >
> > >> > Cheers.
> > >> >
> > >> >
> > >> >
> > >> > On Fri, Jul 12, 2013 at 2:26 PM, Prasanna Santhanam
> > >> <tsp@apache.org <ma...@apache.org>> wrote:
> > >> >
> > >> > > Can you access the file as user root? Or user cloud? The
> > >> cloudstack
> > >> > > agent on your KVM host runs as user cloud and the NFS
> > >> permissions
> > >> > > might be disallowing the volume (.qcow2) from being
> accessed.
> > >> > >
> > >> > > On Fri, Jul 12, 2013 at 02:16:41PM +0800, Indra Pramana
> > >> wrote:
> > >> > > > Hi Prasanna,
> > >> > > >
> > >> > > > Good day to you, and thank you for your e-mail.
> > >> > > >
> > >> > > > Yes, the file exists. I can access the file from the
> > >> management server
> > >> > > and
> > >> > > > the two hypervisors hosts if I mount manually.
> > >> > > >
> > >> > > > [root@cs-nas-01/mnt/vol1/sec-storage/**template/tmpl/1/3]#
> > >> ls
> > >> > > > -la
> > >> > > > total
> > >> > > > 1418787
> > >> > > > drwxr-xr-x 2 root wheel 4 Jul 11 20:21
> > >> > > > .
> > >> > > > drwxr-xr-x 3 root wheel 3 Jul 11 20:17
> > >> > > > ..
> > >> > > > -rw-r--r-- 1 root wheel 725811200 Jul 11 20:21
> > >> > > > 425b9e5a-fbc7-4637-a33a-**fe9d0ed4fa98.qcow2
> > >> > > > -rw-r--r-- 1 root wheel 295 Jul 11 20:21
> > >> > > > template.properties
> > >> > > > [root@cs-nas-01 /mnt/vol1/sec-storage/**
> > >> template/tmpl/1/3]#
> > >> > > > pwd
> > >> > > > /mnt/vol1/sec-storage/**template/tmpl/1/3
> > >> > > >
> > >> > > >
> > >> > > > Any advise?
> > >> > > >
> > >> > > > Looking forward to your reply, thank you.
> > >> > > >
> > >> > > > Cheers.
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > On Fri, Jul 12, 2013 at 2:07 PM, Prasanna Santhanam
> > >> <tsp@apache.org <ma...@apache.org>>
> > >>
> > >> > > wrote:
> > >> > > >
> > >> > > > > Can you check whether there is a file at:
> > >> > > > >
> > >> nfs://103.25.200.19/mnt/vol1/**sec-storage/template/tmpl/1/3/
> <http://103.25.200.19/mnt/vol1/sec-storage/template/tmpl/1/3/>
> > >> <
> http://103.25.200.19/mnt/**vol1/sec-storage/template/**tmpl/1/3/<
> http://103.25.200.19/mnt/vol1/sec-storage/template/tmpl/1/3/>
> > >> >
> > >>
> > >> > > > >
> > >> > > > > On Fri, Jul 12, 2013 at 01:59:34PM +0800, Indra
> Pramana
> > >> wrote:
> > >> > > > > > Hi Prasanna,
> > >> > > > > >
> > >> > > > > > Thanks for your e-mail.
> > >> > > > > >
> > >> > > > > > I have tried restarting the management server, and
> > >> the problem still
> > >> > > > > > persists. I even tried to re-do the installation
> and
> > >> configuration
> > >> > > again
> > >> > > > > > from scratch last night, but the problem still
> there.
> > >> > > > > >
> > >> > > > > > I also noted that on the beginning of the logs, I
> > >> found some error
> > >> > > > > messages
> > >> > > > > > saying that the template cannot be downloaded to
> the
> > >> pool. See this
> > >> > > logs:
> > >> > > > > >
> > >> > > > > > http://pastebin.com/BY1AVJ08
> > >> > > > > >
> > >> > > > > > It says it failed because cannot get volume from
> the
> > >> pool. Could it
> > >> > > be
> > >> > > > > > related, i.e. the absence of the template caused
> the
> > >> system VMs
> > >> > > cannot be
> > >> > > > > > created and started?
> > >> > > > > >
> > >> > > > > > I have ensured that I downloaded the system VM
> > >> template using
> > >> > > > > > cloud-install-sys-tmplt and verified that the
> > >> template is already
> > >> > > there
> > >> > > > > in
> > >> > > > > > the secondary storage server.
> > >> > > > > >
> > >> > > > > > Any advice is appreciated.
> > >> > > > > >
> > >> > > > > > Looking forward to your reply, thank you.
> > >> > > > > >
> > >> > > > > > Cheers.
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On Fri, Jul 12, 2013 at 1:21 PM, Prasanna Santhanam
> > >> <tsp@apache.org <ma...@apache.org>>
> > >>
> > >> > > > > wrote:
> > >> > > > > >
> > >> > > > > > > It looks like a previous attempt to start the
> > >> systemVMs has failed
> > >> > > > > > > putting the nfs storage in the avoid set. Did you
> > >> try restarting
> > >> > > your
> > >> > > > > > > management server?
> > >> > > > > > >
> > >> > > > > > > This line leads me to the above mentioned:
> > >> > > > > > > 2013-07-12 13:10:48,236 DEBUG
> > >> > > > > > >
> [storage.allocator.**AbstractStoragePoolAllocator]
> > >> > > (secstorage-1:null)
> > >> > > > > > > StoragePool is in avoid set, skipping this pool
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > On Fri, Jul 12, 2013 at 01:16:53PM +0800, Indra
> > >> Pramana wrote:
> > >> > > > > > > > Dear Wido and all,
> > >> > > > > > > >
> > >> > > > > > > > I have managed to get the hosts, primary and
> > >> secondary storage
> > >> > > > > running.
> > >> > > > > > > >
> > >> > > > > > > > - 2 KVM hypervisor hosts
> > >> > > > > > > > - One RBD primary storage
> > >> > > > > > > > - One NFS primary storage (for system VMs,
> since
> > >> I understand
> > >> > > that
> > >> > > > > system
> > >> > > > > > > > VMs cannot use RBD)
> > >> > > > > > > > - One NFS secondary storage
> > >> > > > > > > >
> > >> > > > > > > > However, now I am having problem with the
> system
> > >> VMs: CPVM and
> > >> > > SSVM,
> > >> > > > > > > unable
> > >> > > > > > > > to start.
> > >> > > > > > > >
> > >> > > > > > > > Excerpt from management-server.log file is
> here:
> > >> > > > > > > > http://pastebin.com/ENkpCALY
> > >> > > > > > > >
> > >> > > > > > > > It seems that the VMs were not able to be
> created
> > >> because unable
> > >> > > to
> > >> > > > > find
> > >> > > > > > > > suitable StoragePools.
> > >> > > > > > > >
> > >> > > > > > > > I understand that system VMs will be using the
> > >> NFS primary
> > >> > > storage
> > >> > > > > > > instead
> > >> > > > > > > > of RBD, so I have confirmed that I am able to
> > >> mount the primary
> > >> > > > > storage
> > >> > > > > > > via
> > >> > > > > > > > NFS and have read and write access, from both
> the
> > >> hypervisor and
> > >> > > the
> > >> > > > > > > > management server.
> > >> > > > > > > >
> > >> > > > > > > > Any advise how can I resolve the problem to
> make
> > >> both the system
> > >> > > VMs
> > >> > > > > > > > created and started?
> > >> > > > > > > >
> > >> > > > > > > > Looking forward to your reply, thank you.
> > >> > > > > > > >
> > >> > > > > > > > Cheers.
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > On Fri, Jul 12, 2013 at 9:43 AM, Indra Pramana
> > >> <indra@sg.or.id <ma...@sg.or.id>>
> > >>
> > >> > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > Hi Wido,
> > >> > > > > > > > >
> > >> > > > > > > > > Thanks for the advice, I'm now able to add
> the
> > >> RBD pool as
> > >> > > primary
> > >> > > > > > > storage.
> > >> > > > > > > > >
> > >> > > > > > > > > Many thanks! :)
> > >> > > > > > > > >
> > >> > > > > > > > > Cheers.
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > On Thursday, July 11, 2013, Wido den
> Hollander
> > >> wrote:
> > >> > > > > > > > >
> > >> > > > > > > > >> Hi,
> > >> > > > > > > > >>
> > >> > > > > > > > >> On 07/10/2013 03:42 PM, Chip Childers wrote:
> > >> > > > > > > > >>
> > >> > > > > > > > >>> Cc'ing Wido, our resident Ceph expert. ;-)
> > >> > > > > > > > >>>
> > >> > > > > > > > >>>
> > >> > > > > > > > >> Hehe ;)
> > >> > > > > > > > >>
> > >> > > > > > > > >> On Wed, Jul 10, 2013 at 05:45:25PM +0800,
> > >> Indra Pramana
> > >> > > wrote:
> > >> > > > > > > > >>>
> > >> > > > > > > > >>>> Dear all,
> > >> > > > > > > > >>>>
> > >> > > > > > > > >>>> I am installing CloudStack 4.1.0 (upgraded
> > >> from 4.0.2) and I
> > >> > > > > also
> > >> > > > > > > have a
> > >> > > > > > > > >>>> Ceph cluster running. However, I am having
> > >> issues in adding
> > >> > > the
> > >> > > > > RBD
> > >> > > > > > > as
> > >> > > > > > > > >>>> primary storage. Tried to follow the
> > >> instruction here, but
> > >> > > > > unable to
> > >> > > > > > > > >>>> make
> > >> > > > > > > > >>>> it work:
> > >> > > > > > > > >>>>
> > >> > > > > > > > >>>>
> > >> http://ceph.com/docs/master/****rbd/rbd-cloudstack/<
> http://ceph.com/docs/master/**rbd/rbd-cloudstack/>
> > >> <
> > >> > > > > > >
> http://ceph.com/docs/master/**rbd/rbd-cloudstack/<
> http://ceph.com/docs/master/rbd/rbd-cloudstack/>
> > >> >
> > >> > > > > > > > >>>>
> > >> > > > > > > > >>>> I have setup a pool on the Ceph cluster.
> The
> > >> status of the
> > >> > > > > cluster
> > >> > > > > > > is
> > >> > > > > > > > >>>> healthy. Since I am using Ubuntu 12.04.2
> LTS
> > >> (Precise) for
> > >> > > the
> > >> > > > > > > > >>>> hypervisors,
> > >> > > > > > > > >>>> I also have compiled libvirt manually to
> > >> ensure that the
> > >> > > version
> > >> > > > > > > 0.9.13
> > >> > > > > > > > >>>> is
> > >> > > > > > > > >>>> installed (previously it's 0.9.8).
> > >> > > > > > > > >>>>
> > >> > > > > > > > >>>>
> > >> > > > > > > > >> You can also use the Ubuntu Cloud Archive, I
> > >> still need to
> > >> > > get the
> > >> > > > > > > docs
> > >> > > > > > > > >> updated for that.
> > >> > > > > > > > >>
> > >> > > > > > > > >> I described the process in a blogpost:
> > >> > > > > > > http://blog.widodh.nl/2013/06/****<
> http://blog.widodh.nl/2013/06/**>
> > >> > > > > > > > >>
> > >> > >
> > >> a-quick-note-on-running-****cloudstack-with-rbd-on-ubuntu-**
> > >> **12-04/<
> > >> > > > > > >
> > >> > > > >
> > >> > >
> > >> http://blog.widodh.nl/2013/06/**a-quick-note-on-running-**
> > >> cloudstack-with-rbd-on-ubuntu-**12-04/<
> http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/
> >
> > >> > > > > > > >
> > >> > > > > > > > >>
> > >> > > > > > > > >> indra@hv-kvm-01:~/rbd$ ceph
> > >> > > > > > > > >>>> ceph> health
> > >> > > > > > > > >>>> HEALTH_OK
> > >> > > > > > > > >>>>
> > >> > > > > > > > >>>> indra@hv-kvm-01:~$ ceph osd lspools
> > >> > > > > > > > >>>> 0 data,1 metadata,2 rbd,3 sc1,
> > >> > > > > > > > >>>>
> > >> > > > > > > > >>>> root@hv-kvm-01:/home/indra# libvirtd
> > >> --version
> > >> > > > > > > > >>>> libvirtd (libvirt) 0.9.13
> > >> > > > > > > > >>>>
> > >> > > > > > > > >>>> I tried to add Primary Storage into the
> > >> Cloudstack zone
> > >> > > which I
> > >> > > > > have
> > >> > > > > > > > >>>> created:
> > >> > > > > > > > >>>>
> > >> > > > > > > > >>>> Add Primary Storage:
> > >> > > > > > > > >>>>
> > >> > > > > > > > >>>> Zone: my zone name
> > >> > > > > > > > >>>> Pod: my pod name
> > >> > > > > > > > >>>> Cluster: my cluster name
> > >> > > > > > > > >>>> Name: ceph-rbd-pri-storage
> > >> > > > > > > > >>>> Protocol: RBD
> > >> > > > > > > > >>>> RADOS Monitor: my first Ceph monitor IP
> > >> address
> > >> > > > > > > > >>>> RADOS Pool: sc1 (the pool name on Ceph
> > >> cluster)
> > >> > > > > > > > >>>> RADOS User: client.admin
> > >> > > > > > > > >>>> RADOS Secret:
> > >> /etc/ceph/ceph.client.admin.****keyring (keyring
> > >> > > > > file
> > >> > > > > > > > >>>> location)
> > >> > > > > > > > >>>>
> > >> > > > > > > > >>>
> > >> > > > > > > > >> This is your problem. That shouldn't be the
> > >> location of the
> > >> > > file,
> > >> > > > > but
> > >> > > > > > > it
> > >> > > > > > > > >> should be the secret, which is a base64
> > >> encoded string.
> > >> > > > > > > > >>
> > >> > > > > > > > >> $ ceph auth list
> > >> > > > > > > > >>
> > >> > > > > > > > >> That should tell you what the secret is.
> > >> > > > > > > > >>
> > >> > > > > > > > >> Storage Tags: rbd
> > >> > > > > > > > >>
> > >> > > > > > > > >> This is the error message when I tried to
> add
> > >> the primary
> > >> > > storage
> > >> > > > > by
> > >> > > > > > > > >> clicking OK:
> > >> > > > > > > > >>
> > >> > > > > > > > >> DB Exception on:
> > >> > > com.mysql.jdbc.****JDBC4PreparedStatement@4b2eb56
> > >> > > > > **:
> > >> > > > > > > > >> INSERT INTO
> > >> > > > > > > > >> storage_pool (storage_pool.id
> > >> <http://storage_pool.id>, storage_pool.name
> > >> <http://storage_pool.name>,
> > >>
> > >> > > > > storage_pool.uuid,
> > >> > > > > > > > >> storage_pool.pool_type,
> storage_pool.created,
> > >> > > > > > > storage_pool.update_time,
> > >> > > > > > > > >> storage_pool.data_center_id,
> > >> storage_pool.pod_id,
> > >> > > > > > > > >> storage_pool.available_bytes,
> > >> storage_pool.capacity_bytes,
> > >> > > > > > > > >> storage_pool.status, storage_pool.scope,
> > >> > > > > > > storage_pool.storage_provider_****
> > >> > > > > > > > >> id,
> > >> > > > > > > > >> storage_pool.host_address,
> storage_pool.path,
> > >> > > storage_pool.port,
> > >> > > > > > > > >> storage_pool.user_info,
> > >> storage_pool.cluster_id) VALUES (217,
> > >> > > > > > > > >> _binary'ceph-rbd-pri-storage',
> > >> > > > > > > > >>
> > >> _binary'a226c9a1-da78-3f3a-****b5ac-e18b925c9634', 'RBD',
> > >> > > > > '2013-07-10
> > >> > > > > > > > >> 09:08:28', null, 2, 2, 0, 0, 'Up', null,
> null,
> > >> null,
> > >> > > _binary'ceph/
> > >> > > > > > > > >> ceph.client.admin.keyring@10.
> ****237.11.2/sc1<
> > >> > > > > > > http://ceph.client.admin.**
> keyring@10.237.11.2/sc1<http://ceph.client.admin.keyring@10.237.11.2/sc1>
> > >> >',
> > >> > > > > > > > >> 6789, null, 2)
> > >> > > > > > > > >>
> > >> > > > > > > > >> On the management-server.log file:
> > >> > > > > > > > >>
> > >> > > > > > > > >> 2013-07-10 17:08:28,845 DEBUG
> > >> [cloud.api.ApiServlet]
> > >> > > > > > > > >> (catalina-exec-2:null)
> > >> > > > > > > > >> ===START=== 192.168.0.100 -- GET
> > >> > > > > > > > >>
> > >>
> command=createStoragePool&****zoneid=c116950e-e4ae-4f23-****a7e7-
> > >> > > > > > > > >>
> > >> 74a75c4ee638&podId=a748b063-****3a83-4175-a0e9-de39118fe5ce&**
> > >> > > > > > > > >>
> > >>
> clusterid=1f87eb09-324d-4d49-****83c2-88d84d7a15df&name=ceph-****
> > >> > > > > > > > >>
> > >> rbd-pri-storage&url=rbd%3A%2F%****2Fclient.admin%3A_etc%2Fc
> > >> > > > > > > > >>
> > >> eph%2Fceph.client.admin.****keyring%4010.237.11.2%2Fsc1&**
> > >> > > > > > > > >>
> > >> > > > >
> > >> tags=rbd&response=json&****sessionkey=****
> > >> rDRfWpqeKfQKbKZtHr398ULV%2F8k%****
> > >> > > > > > > > >> 3D&_=1373447307839
> > >> > > > > > > > >> 2013-07-10 17:08:28,862 DEBUG
> > >> > > [cloud.storage.****StorageManagerImpl]
> > >> > > > > > > > >> (catalina-exec-2:null) createPool Params @
> > >> scheme - rbd
> > >> > > > > storageHost -
> > >> > > > > > > null
> > >> > > > > > > > >> hostPath - /ceph/ceph.client
> > >> > > > > > > > >> .admin.keyring@10.237.11.2/sc1
> > >> <http://admin.keyring@10.237.**11.2/sc1<
> http://admin.keyring@10.237.11.2/sc1>>
> > >> port - -1
> > >>
> > >> > > > > > > > >> 2013-07-10 17:08:28,918 DEBUG
> > >> > > [cloud.storage.****StorageManagerImpl]
> > >> > > > > > > > >> (catalina-exec-2:null) In createPool Setting
> > >> poolId - 217
> > >> > > uuid -
> > >> > > > > > > > >> a226c9a1-da78-3f3a-b5ac-****e18b925c9634 z
> > >> > > > > > > > >> oneId - 2 podId - 2 poolName -
> > >> ceph-rbd-pri-storage
> > >> > > > > > > > >> 2013-07-10 17:08:28,921 DEBUG
> > >> [db.Transaction.Transaction]
> > >> > > > > > > > >> (catalina-exec-2:null) Rolling back the
> > >> transaction: Time = 3
> > >> > > > > Name =
> > >> > > > > > > > >> persist; called by -Transaction.rollbac
> > >> > > > > > > > >>
> > >> k:890-Transaction.removeUpTo:****833-Transaction.close:657-**
> > >> > > > > > > > >>
> > >> TransactionContextBuilder.****interceptException:63-**
> > >> > > > > > > > >>
> > >> > > > >
> > >>
> ComponentInstantiationPostProc****essor$InterceptorDispatcher.**
> > >> **interce
> > >> > > > > > > > >> pt:133-StorageManagerImpl.****
> > >> createPool:1378-**
> > >> > > > > > > > >>
> > >> StorageManagerImpl.createPool:****147-CreateStoragePoolCmd.**
> > >> > > > > > > > >>
> > >> execute:123-ApiDispatcher.****dispatch:162-ApiServer.**
> > >> > > > > > > > >> queueCommand:505-ApiSe
> > >> > > > > > > > >>
> > >> rver.handleRequest:355-****ApiServlet.processRequest:302
> > >> > > > > > > > >> 2013-07-10 17:08:28,923 ERROR
> > >> [cloud.api.ApiServer]
> > >> > > > > > > (catalina-exec-2:null)
> > >> > > > > > > > >> unhandled exception executing api command:
> > >> createStoragePool
> > >> > > > > > > > >>
> > >> com.cloud.utils.exception.****CloudRuntimeException: DB
> > >> > > Exception
> > >> > > > > on:
> > >> > > > > > > > >>
> > >> com.mysql.jdbc.****JDBC4PreparedStatement@4b2eb56****: INSERT
> > >> > > INTO
> > >> > > > > > > > >> storage_pool (
> > >> > > > > > > > >> storage_pool.id <http://storage_pool.id>,
> > >>
> > >> storage_pool
> > >> > > > > > > > >> .name, storage_pool.uuid,
> > >> storage_pool.pool_type,
> > >> > > > > > > storage_pool.created,
> > >> > > > > > > > >> storage_pool.update_time,
> > >> storage_pool.data_center_id,
> > >> > > > > > > > >> storage_pool.pod_id,
> > >> > > > > > > > >> storage_pool.availab
> > >> > > > > > > > >> le_bytes, storage_pool.capacity_bytes,
> > >> storage_pool.status,
> > >> > > > > > > > >> storage_pool.scope,
> > >> storage_pool.storage_provider_****id,
> > >> > > > > > > > >> storage_pool.host_address,
> storage_pool.path,
> > >> storage_
> > >> > > > > > > > >> pool.port, storage_pool.user_info,
> > >> storage_pool.cluster_id)
> > >> > > VALUES
> > >> > > > > > > (217,
> > >> > > > > > > > >> _binary'ceph-rbd-pri-storage',
> > >> > > > > > > > >>
> > >> _binary'a226c9a1-da78-3f3a-****b5ac-e18b925c9634', 'RBD',
> > >> > > '2013-07-1
> > >> > > > > > > > >> 0 09:08:28', null, 2, 2, 0, 0, 'Up', null,
> > >> null, null,
> > >> > > > > _binary'ceph/
> > >> > > > > > > > >> ceph.client.admin.keyring@10.
> ****237.11.2/sc1<
> > >> > > > > > > http://ceph.client.admin.**
> keyring@10.237.11.2/sc1<http://ceph.client.admin.keyring@10.237.11.2/sc1>
> > >> >',
> > >> > > > > > > > >> 6789, null, 2)
> > >> > > > > > > > >> at
> > >> > > > > > > > >>
> > >> > > > > > >
> > >> > >
> > >> com.cloud.utils.db.****GenericDaoBase.persist(****
> > >> GenericDaoBase.java:1342)
> > >> > > > > > > > >> at
> > >> > > > > > > > >>
> > >> com.cloud.storage.dao.****StoragePoolDaoImpl.persist(**
> > >> > > > > > > > >> StoragePoolDaoImpl.java:232)
> > >> > > > > > > > >> at
> > >> > > > > > > > >>
> > >>
> com.cloud.utils.component.****ComponentInstantiationPostProc****
> > >> > > > > > > > >>
> > >> > > > > > >
> > >> > > > >
> > >> > >
> > >> essor$InterceptorDispatcher.****intercept(****
> > >> ComponentInstantiationPostProc***
> > >> > > > > > > > >> *es
> > >> > > > > > > > >>
> > >> > > > > > > > >>
> > >> > > > > > >
> > >> > > > > > > --
> > >> > > > > > > Prasanna.,
> > >> > > > > > >
> > >> > > > > > > ------------------------
> > >> > > > > > > Powered by BigRock.com
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > >
> > >> > > > > --
> > >> > > > > Prasanna.,
> > >> > > > >
> > >> > > > > ------------------------
> > >> > > > > Powered by BigRock.com
> > >> > > > >
> > >> > > > >
> > >> > >
> > >> > > --
> > >> > > Prasanna.,
> > >> > >
> > >> > > ------------------------
> > >> > > Powered by BigRock.com
> > >> > >
> > >> > >
> > >>
> > >> --
> > >> Prasanna.,
> > >>
> > >> ------------------------
> > >> Powered by BigRock.com
> > >>
> > >>
> > >>
> > >>
> > >
>
> --
> Prasanna.,
>
> ------------------------
> Powered by BigRock.com
>
>
Re: Problem in adding Ceph RBD as primary storage for CloudStack
4.1.0
Posted by Prasanna Santhanam <ts...@apache.org>.
See Inline,
On Fri, Jul 12, 2013 at 05:12:25PM +0800, Indra Pramana wrote:
> Hi Wido,
>
> Noted, can't wait for 4.2 to be released. :)
>
> Dear Prasanna, Wido and all,
>
> I just realised that while the system VMs are running, they are still not
> accessible through the public IPs assigned to them. I have been waiting for
> the SSVM to download the default CentOS template and it doesn't appear on
> the template list.
>
> I tried to SSH into the SSVM via the local link address from the KVM host,
> and running the health check /usr/local/cloud/systemvm/ssvm-check.sh shows
> that the VM cannot reach anywhere. It cannot reach the public DNS server (I
> used Google's 8.8.8.8), cannot reach the management server, and cannot even
> reach the public IP gateway.
Ok - we're not there yet. But there's a few troubleshooting steps you
can try:
1. Which of the checks fail apart from the DNS ping?
2. Are you able to have those tests pass from the KVM host itself?
3. Can you paste the output of $route -n on your SSVM and on your
host?
>
> Is it due to misconfiguration of the KVM network bridges? How can I see the
> mapping between the NIC interfaces of the SSVM (eth0, eth1, eth2 and eth3)
> and the actual physical NIC interfaces on the KVM hosts (eth0) and the
> network bridges (cloudbr0, cloudbr1)? Any logs I can verify to ensure that
> the VLAN and network bridging is working?
>
You can see this from the dumpxml.
$virsh list
(gives you all the domains. find the ssvm for example)
$virsh dumpxml my-domain > mydomain.xml
> Appreciate any advice.
>
> Thank you.
>
>
>
> On Fri, Jul 12, 2013 at 4:19 PM, Wido den Hollander <wi...@widodh.nl> wrote:
>
> > On 07/12/2013 10:14 AM, Indra Pramana wrote:
> >
> >> Hi Prasanna,
> >>
> >> I managed to fix the problem, thanks for your advice to turn the agent
> >> log level to debug:
> >>
> >> https://cwiki.apache.org/**confluence/display/CLOUDSTACK/**
> >> KVM+agent+debug<https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug>
> >>
> >> From the log, I found out that the agent on the KVM host tried to NFS
> >> mount directly to 103.25.200.19:/mnt/vol1/sec-**
> >> storage/template/tmpl/1/3,
> >> which was not allowed by the NFS server due to its default configuration
> >> to only allow to mount to /mnt/vol1/sec-storage (the root of the NFS
> >> share).
> >>
> >>
> > Ah, that's odd!
> >
> > Btw, in 4.2 you'll be able to deploy SSVMs on RBD as well, so that
> > limitation will be gone.
> >
> > Wido
> >
> > After I changed the NFS server configuration to allow mount to all
> >> sub-directories, re-export the NFS and voila, the system was able to
> >> download the template and now both the system VMs (CPVM and SSVM) are
> >> running!
> >>
> >> Many thanks for your help! :)
> >>
> >> Cheers.
> >>
> >>
> >>
> >> On Fri, Jul 12, 2013 at 3:31 PM, Indra Pramana <indra@sg.or.id
> >> <ma...@sg.or.id>> wrote:
> >>
> >> Hi Prasanna,
> >>
> >> Good day to you, and thank you for your e-mail.
> >>
> >> Yes, the cloudstack-agent service is running on both the KVM hosts.
> >> There is no "cloud" user being created though, when I installed the
> >> agent. I installed the agent as root.
> >>
> >> root@hv-kvm-01:/home/indra# service cloudstack-agent status
> >> * cloud-agent is running
> >>
> >> root@hv-kvm-01:/home/indra# su - cloud
> >> Unknown id: cloud
> >>
> >> Please advise how can I resolve this problem, shall I create the
> >> Unix "cloud" user manually? Basically I follow this instruction to
> >> prepare the KVM host and install the CloudStack agent:
> >>
> >> http://cloudstack.apache.org/**docs/en-US/Apache_CloudStack/**
> >> 4.1.0/html/Installation_Guide/**hypervisor-kvm-install-flow.**html<http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.1.0/html/Installation_Guide/hypervisor-kvm-install-flow.html>
> >>
> >> with this instruction from Wido on how to prepare libvirt with Ceph
> >> RBD storage pool support:
> >>
> >> http://blog.widodh.nl/2013/06/**a-quick-note-on-running-**
> >> cloudstack-with-rbd-on-ubuntu-**12-04/<http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/>
> >>
> >> I also have checked /var/log/cloud/agent/agent.log and I don't see
> >> any error messages, except this error message which will show up
> >> every time I restart the agent:
> >>
> >> 2013-07-12 15:22:47,454 ERROR [cloud.resource.**ServerResourceBase]
> >> (main:null) Nics are not configured!
> >> 2013-07-12 15:22:47,459 INFO [cloud.resource.**ServerResourceBase]
> >> (main:null) Designating private to be nic eth0.5
> >>
> >> More logs can be found here: http://pastebin.com/yeNmCt7S
> >>
> >> I have configured the network bridges on the NIC interface as per
> >> this instruction:
> >>
> >> http://cloudstack.apache.org/**docs/en-US/Apache_CloudStack/**
> >> 4.1.0/html/Installation_Guide/**hypervisor-kvm-install-flow.**
> >> html#hypervisor-host-install-**network<http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.1.0/html/Installation_Guide/hypervisor-kvm-install-flow.html#hypervisor-host-install-network>
> >>
> >> On the zone, I used advanced network configuration with just one
> >> physical network for management, public and guest/private. I didn't
> >> include storage, which by default the traffic will use the
> >> management VLAN network.
> >>
> >> Please advise if there's anything else I might have been missing.
> >>
> >> Looking forward to your reply, thank you.
> >>
> >> Cheers.
> >>
> >>
> >>
> >> On Fri, Jul 12, 2013 at 2:56 PM, Prasanna Santhanam <tsp@apache.org
> >> <ma...@apache.org>> wrote:
> >>
> >> Indeed, cloudstack will go through the allocation to startup the
> >> system VMs too. So that process is failing to recognize the volume
> >> (.qcow2) present on your NFS storage.
> >>
> >> Can you check if your cloudstack agent service is running with
> >> the KVM
> >> host? And it should've created the user cloud. $id cloud to check
> >> if
> >> there is a user.
> >>
> >> Did you see what's happening in the agent logs? These are under
> >> /var/log/cloud/ on your host when the systemVMs are coming up.
> >> If the
> >> logs are not showing any useful information you can turn on debug
> >> level for more verbosity.
> >>
> >> See here: https://cwiki.apache.org/**confluence/x/FgPMAQ<https://cwiki.apache.org/confluence/x/FgPMAQ>
> >>
> >> On Fri, Jul 12, 2013 at 02:40:35PM +0800, Indra Pramana wrote:
> >> > Hi Prasanna,
> >> >
> >> > Good day to you, and thank you for your e-mail.
> >> >
> >> > Yes, when I export the NFS, I set the permission so that
> >> normal user will
> >> > be able to have read/write access to the files
> >> (no_root_squash).
> >> >
> >> > I have tested and I can have read/write access from my KVM
> >> hosts using
> >> > normal user. BTW, there's no "cloud" user on the hosts, I
> >> believe it's not
> >> > created during cloudstack-agent installation?
> >> >
> >> > In any case, do you think the template issue and the storage
> >> pool
> >> > allocation issue might be related, or are they two different
> >> problems
> >> > altogether?
> >> >
> >> > Looking forward to your reply, thank you.
> >> >
> >> > Cheers.
> >> >
> >> >
> >> >
> >> > On Fri, Jul 12, 2013 at 2:26 PM, Prasanna Santhanam
> >> <tsp@apache.org <ma...@apache.org>> wrote:
> >> >
> >> > > Can you access the file as user root? Or user cloud? The
> >> cloudstack
> >> > > agent on your KVM host runs as user cloud and the NFS
> >> permissions
> >> > > might be disallowing the volume (.qcow2) from being accessed.
> >> > >
> >> > > On Fri, Jul 12, 2013 at 02:16:41PM +0800, Indra Pramana
> >> wrote:
> >> > > > Hi Prasanna,
> >> > > >
> >> > > > Good day to you, and thank you for your e-mail.
> >> > > >
> >> > > > Yes, the file exists. I can access the file from the
> >> management server
> >> > > and
> >> > > > the two hypervisors hosts if I mount manually.
> >> > > >
> >> > > > [root@cs-nas-01 /mnt/vol1/sec-storage/**template/tmpl/1/3]#
> >> ls
> >> > > > -la
> >> > > > total
> >> > > > 1418787
> >> > > > drwxr-xr-x 2 root wheel 4 Jul 11 20:21
> >> > > > .
> >> > > > drwxr-xr-x 3 root wheel 3 Jul 11 20:17
> >> > > > ..
> >> > > > -rw-r--r-- 1 root wheel 725811200 Jul 11 20:21
> >> > > > 425b9e5a-fbc7-4637-a33a-**fe9d0ed4fa98.qcow2
> >> > > > -rw-r--r-- 1 root wheel 295 Jul 11 20:21
> >> > > > template.properties
> >> > > > [root@cs-nas-01 /mnt/vol1/sec-storage/**
> >> template/tmpl/1/3]#
> >> > > > pwd
> >> > > > /mnt/vol1/sec-storage/**template/tmpl/1/3
> >> > > >
> >> > > >
> >> > > > Any advise?
> >> > > >
> >> > > > Looking forward to your reply, thank you.
> >> > > >
> >> > > > Cheers.
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Fri, Jul 12, 2013 at 2:07 PM, Prasanna Santhanam
> >> <tsp@apache.org <ma...@apache.org>>
> >>
> >> > > wrote:
> >> > > >
> >> > > > > Can you check whether there is a file at:
> >> > > > >
> >> nfs://103.25.200.19/mnt/vol1/**sec-storage/template/tmpl/1/3/<http://103.25.200.19/mnt/vol1/sec-storage/template/tmpl/1/3/>
> >> <http://103.25.200.19/mnt/**vol1/sec-storage/template/**tmpl/1/3/<http://103.25.200.19/mnt/vol1/sec-storage/template/tmpl/1/3/>
> >> >
> >>
> >> > > > >
> >> > > > > On Fri, Jul 12, 2013 at 01:59:34PM +0800, Indra Pramana
> >> wrote:
> >> > > > > > Hi Prasanna,
> >> > > > > >
> >> > > > > > Thanks for your e-mail.
> >> > > > > >
> >> > > > > > I have tried restarting the management server, and
> >> the problem still
> >> > > > > > persists. I even tried to re-do the installation and
> >> configuration
> >> > > again
> >> > > > > > from scratch last night, but the problem still there.
> >> > > > > >
> >> > > > > > I also noted that on the beginning of the logs, I
> >> found some error
> >> > > > > messages
> >> > > > > > saying that the template cannot be downloaded to the
> >> pool. See this
> >> > > logs:
> >> > > > > >
> >> > > > > > http://pastebin.com/BY1AVJ08
> >> > > > > >
> >> > > > > > It says it failed because cannot get volume from the
> >> pool. Could it
> >> > > be
> >> > > > > > related, i.e. the absence of the template caused the
> >> system VMs
> >> > > cannot be
> >> > > > > > created and started?
> >> > > > > >
> >> > > > > > I have ensured that I downloaded the system VM
> >> template using
> >> > > > > > cloud-install-sys-tmplt and verified that the
> >> template is already
> >> > > there
> >> > > > > in
> >> > > > > > the secondary storage server.
> >> > > > > >
> >> > > > > > Any advice is appreciated.
> >> > > > > >
> >> > > > > > Looking forward to your reply, thank you.
> >> > > > > >
> >> > > > > > Cheers.
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > On Fri, Jul 12, 2013 at 1:21 PM, Prasanna Santhanam
> >> <tsp@apache.org <ma...@apache.org>>
> >>
> >> > > > > wrote:
> >> > > > > >
> >> > > > > > > It looks like a previous attempt to start the
> >> systemVMs has failed
> >> > > > > > > putting the nfs storage in the avoid set. Did you
> >> try restarting
> >> > > your
> >> > > > > > > management server?
> >> > > > > > >
> >> > > > > > > This line leads me to the above mentioned:
> >> > > > > > > 2013-07-12 13:10:48,236 DEBUG
> >> > > > > > > [storage.allocator.**AbstractStoragePoolAllocator]
> >> > > (secstorage-1:null)
> >> > > > > > > StoragePool is in avoid set, skipping this pool
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Fri, Jul 12, 2013 at 01:16:53PM +0800, Indra
> >> Pramana wrote:
> >> > > > > > > > Dear Wido and all,
> >> > > > > > > >
> >> > > > > > > > I have managed to get the hosts, primary and
> >> secondary storage
> >> > > > > running.
> >> > > > > > > >
> >> > > > > > > > - 2 KVM hypervisor hosts
> >> > > > > > > > - One RBD primary storage
> >> > > > > > > > - One NFS primary storage (for system VMs, since
> >> I understand
> >> > > that
> >> > > > > system
> >> > > > > > > > VMs cannot use RBD)
> >> > > > > > > > - One NFS secondary storage
> >> > > > > > > >
> >> > > > > > > > However, now I am having problem with the system
> >> VMs: CPVM and
> >> > > SSVM,
> >> > > > > > > unable
> >> > > > > > > > to start.
> >> > > > > > > >
> >> > > > > > > > Excerpt from management-server.log file is here:
> >> > > > > > > > http://pastebin.com/ENkpCALY
> >> > > > > > > >
> >> > > > > > > > It seems that the VMs were not able to be created
> >> because unable
> >> > > to
> >> > > > > find
> >> > > > > > > > suitable StoragePools.
> >> > > > > > > >
> >> > > > > > > > I understand that system VMs will be using the
> >> NFS primary
> >> > > storage
> >> > > > > > > instead
> >> > > > > > > > of RBD, so I have confirmed that I am able to
> >> mount the primary
> >> > > > > storage
> >> > > > > > > via
> >> > > > > > > > NFS and have read and write access, from both the
> >> hypervisor and
> >> > > the
> >> > > > > > > > management server.
> >> > > > > > > >
> >> > > > > > > > Any advise how can I resolve the problem to make
> >> both the system
> >> > > VMs
> >> > > > > > > > created and started?
> >> > > > > > > >
> >> > > > > > > > Looking forward to your reply, thank you.
> >> > > > > > > >
> >> > > > > > > > Cheers.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Fri, Jul 12, 2013 at 9:43 AM, Indra Pramana
> >> <indra@sg.or.id <ma...@sg.or.id>>
> >>
> >> > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Hi Wido,
> >> > > > > > > > >
> >> > > > > > > > > Thanks for the advice, I'm now able to add the
> >> RBD pool as
> >> > > primary
> >> > > > > > > storage.
> >> > > > > > > > >
> >> > > > > > > > > Many thanks! :)
> >> > > > > > > > >
> >> > > > > > > > > Cheers.
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > On Thursday, July 11, 2013, Wido den Hollander
> >> wrote:
> >> > > > > > > > >
> >> > > > > > > > >> Hi,
> >> > > > > > > > >>
> >> > > > > > > > >> On 07/10/2013 03:42 PM, Chip Childers wrote:
> >> > > > > > > > >>
> >> > > > > > > > >>> Cc'ing Wido, our resident Ceph expert. ;-)
> >> > > > > > > > >>>
> >> > > > > > > > >>>
> >> > > > > > > > >> Hehe ;)
> >> > > > > > > > >>
> >> > > > > > > > >> On Wed, Jul 10, 2013 at 05:45:25PM +0800,
> >> Indra Pramana
> >> > > wrote:
> >> > > > > > > > >>>
> >> > > > > > > > >>>> Dear all,
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> I am installing CloudStack 4.1.0 (upgraded
> >> from 4.0.2) and I
> >> > > > > also
> >> > > > > > > have a
> >> > > > > > > > >>>> Ceph cluster running. However, I am having
> >> issues in adding
> >> > > the
> >> > > > > RBD
> >> > > > > > > as
> >> > > > > > > > >>>> primary storage. Tried to follow the
> >> instruction here, but
> >> > > > > unable to
> >> > > > > > > > >>>> make
> >> > > > > > > > >>>> it work:
> >> > > > > > > > >>>>
> >> > > > > > > > >>>>
> >> http://ceph.com/docs/master/****rbd/rbd-cloudstack/<http://ceph.com/docs/master/**rbd/rbd-cloudstack/>
> >> <
> >> > > > > > > http://ceph.com/docs/master/**rbd/rbd-cloudstack/<http://ceph.com/docs/master/rbd/rbd-cloudstack/>
> >> >
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> I have setup a pool on the Ceph cluster. The
> >> status of the
> >> > > > > cluster
> >> > > > > > > is
> >> > > > > > > > >>>> healthy. Since I am using Ubuntu 12.04.2 LTS
> >> (Precise) for
> >> > > the
> >> > > > > > > > >>>> hypervisors,
> >> > > > > > > > >>>> I also have compiled libvirt manually to
> >> ensure that the
> >> > > version
> >> > > > > > > 0.9.13
> >> > > > > > > > >>>> is
> >> > > > > > > > >>>> installed (previously it's 0.9.8).
> >> > > > > > > > >>>>
> >> > > > > > > > >>>>
> >> > > > > > > > >> You can also use the Ubuntu Cloud Archive, I
> >> still need to
> >> > > get the
> >> > > > > > > docs
> >> > > > > > > > >> updated for that.
> >> > > > > > > > >>
> >> > > > > > > > >> I described the process in a blogpost:
> >> > > > > > > http://blog.widodh.nl/2013/06/****<http://blog.widodh.nl/2013/06/**>
> >> > > > > > > > >>
> >> > >
> >> a-quick-note-on-running-****cloudstack-with-rbd-on-ubuntu-**
> >> **12-04/<
> >> > > > > > >
> >> > > > >
> >> > >
> >> http://blog.widodh.nl/2013/06/**a-quick-note-on-running-**
> >> cloudstack-with-rbd-on-ubuntu-**12-04/<http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/>
> >> > > > > > > >
> >> > > > > > > > >>
> >> > > > > > > > >> indra@hv-kvm-01:~/rbd$ ceph
> >> > > > > > > > >>>> ceph> health
> >> > > > > > > > >>>> HEALTH_OK
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> indra@hv-kvm-01:~$ ceph osd lspools
> >> > > > > > > > >>>> 0 data,1 metadata,2 rbd,3 sc1,
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> root@hv-kvm-01:/home/indra# libvirtd
> >> --version
> >> > > > > > > > >>>> libvirtd (libvirt) 0.9.13
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> I tried to add Primary Storage into the
> >> Cloudstack zone
> >> > > which I
> >> > > > > have
> >> > > > > > > > >>>> created:
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> Add Primary Storage:
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> Zone: my zone name
> >> > > > > > > > >>>> Pod: my pod name
> >> > > > > > > > >>>> Cluster: my cluster name
> >> > > > > > > > >>>> Name: ceph-rbd-pri-storage
> >> > > > > > > > >>>> Protocol: RBD
> >> > > > > > > > >>>> RADOS Monitor: my first Ceph monitor IP
> >> address
> >> > > > > > > > >>>> RADOS Pool: sc1 (the pool name on Ceph
> >> cluster)
> >> > > > > > > > >>>> RADOS User: client.admin
> >> > > > > > > > >>>> RADOS Secret:
> >> /etc/ceph/ceph.client.admin.****keyring (keyring
> >> > > > > file
> >> > > > > > > > >>>> location)
> >> > > > > > > > >>>>
> >> > > > > > > > >>>
> >> > > > > > > > >> This is your problem. That shouldn't be the
> >> location of the
> >> > > file,
> >> > > > > but
> >> > > > > > > it
> >> > > > > > > > >> should be the secret, which is a base64
> >> encoded string.
> >> > > > > > > > >>
> >> > > > > > > > >> $ ceph auth list
> >> > > > > > > > >>
> >> > > > > > > > >> That should tell you what the secret is.
> >> > > > > > > > >>
> >> > > > > > > > >> Storage Tags: rbd
> >> > > > > > > > >>
> >> > > > > > > > >> This is the error message when I tried to add
> >> the primary
> >> > > storage
> >> > > > > by
> >> > > > > > > > >> clicking OK:
> >> > > > > > > > >>
> >> > > > > > > > >> DB Exception on:
> >> > > com.mysql.jdbc.****JDBC4PreparedStatement@4b2eb56
> >> > > > > **:
> >> > > > > > > > >> INSERT INTO
> >> > > > > > > > >> storage_pool (storage_pool.id
> >> <http://storage_pool.id>, storage_pool.name
> >> <http://storage_pool.name>,
> >>
> >> > > > > storage_pool.uuid,
> >> > > > > > > > >> storage_pool.pool_type, storage_pool.created,
> >> > > > > > > storage_pool.update_time,
> >> > > > > > > > >> storage_pool.data_center_id,
> >> storage_pool.pod_id,
> >> > > > > > > > >> storage_pool.available_bytes,
> >> storage_pool.capacity_bytes,
> >> > > > > > > > >> storage_pool.status, storage_pool.scope,
> >> > > > > > > storage_pool.storage_provider_****
> >> > > > > > > > >> id,
> >> > > > > > > > >> storage_pool.host_address, storage_pool.path,
> >> > > storage_pool.port,
> >> > > > > > > > >> storage_pool.user_info,
> >> storage_pool.cluster_id) VALUES (217,
> >> > > > > > > > >> _binary'ceph-rbd-pri-storage',
> >> > > > > > > > >>
> >> _binary'a226c9a1-da78-3f3a-****b5ac-e18b925c9634', 'RBD',
> >> > > > > '2013-07-10
> >> > > > > > > > >> 09:08:28', null, 2, 2, 0, 0, 'Up', null, null,
> >> null,
> >> > > _binary'ceph/
> >> > > > > > > > >> ceph.client.admin.keyring@10.****237.11.2/sc1<
> >> > > > > > > http://ceph.client.admin.**keyring@10.237.11.2/sc1<http://ceph.client.admin.keyring@10.237.11.2/sc1>
> >> >',
> >> > > > > > > > >> 6789, null, 2)
> >> > > > > > > > >>
> >> > > > > > > > >> On the management-server.log file:
> >> > > > > > > > >>
> >> > > > > > > > >> 2013-07-10 17:08:28,845 DEBUG
> >> [cloud.api.ApiServlet]
> >> > > > > > > > >> (catalina-exec-2:null)
> >> > > > > > > > >> ===START=== 192.168.0.100 -- GET
> >> > > > > > > > >>
> >> command=createStoragePool&****zoneid=c116950e-e4ae-4f23-****a7e7-
> >> > > > > > > > >>
> >> 74a75c4ee638&podId=a748b063-****3a83-4175-a0e9-de39118fe5ce&**
> >> > > > > > > > >>
> >> clusterid=1f87eb09-324d-4d49-****83c2-88d84d7a15df&name=ceph-****
> >> > > > > > > > >>
> >> rbd-pri-storage&url=rbd%3A%2F%****2Fclient.admin%3A_etc%2Fc
> >> > > > > > > > >>
> >> eph%2Fceph.client.admin.****keyring%4010.237.11.2%2Fsc1&**
> >> > > > > > > > >>
> >> > > > >
> >> tags=rbd&response=json&****sessionkey=****
> >> rDRfWpqeKfQKbKZtHr398ULV%2F8k%****
> >> > > > > > > > >> 3D&_=1373447307839
> >> > > > > > > > >> 2013-07-10 17:08:28,862 DEBUG
> >> > > [cloud.storage.****StorageManagerImpl]
> >> > > > > > > > >> (catalina-exec-2:null) createPool Params @
> >> scheme - rbd
> >> > > > > storageHost -
> >> > > > > > > null
> >> > > > > > > > >> hostPath - /ceph/ceph.client
> >> > > > > > > > >> .admin.keyring@10.237.11.2/sc1
> >> <http://admin.keyring@10.237.**11.2/sc1<http://admin.keyring@10.237.11.2/sc1>>
> >> port - -1
> >>
> >> > > > > > > > >> 2013-07-10 17:08:28,918 DEBUG
> >> > > [cloud.storage.****StorageManagerImpl]
> >> > > > > > > > >> (catalina-exec-2:null) In createPool Setting
> >> poolId - 217
> >> > > uuid -
> >> > > > > > > > >> a226c9a1-da78-3f3a-b5ac-****e18b925c9634 z
> >> > > > > > > > >> oneId - 2 podId - 2 poolName -
> >> ceph-rbd-pri-storage
> >> > > > > > > > >> 2013-07-10 17:08:28,921 DEBUG
> >> [db.Transaction.Transaction]
> >> > > > > > > > >> (catalina-exec-2:null) Rolling back the
> >> transaction: Time = 3
> >> > > > > Name =
> >> > > > > > > > >> persist; called by -Transaction.rollbac
> >> > > > > > > > >>
> >> k:890-Transaction.removeUpTo:****833-Transaction.close:657-**
> >> > > > > > > > >>
> >> TransactionContextBuilder.****interceptException:63-**
> >> > > > > > > > >>
> >> > > > >
> >> ComponentInstantiationPostProc****essor$InterceptorDispatcher.**
> >> **interce
> >> > > > > > > > >> pt:133-StorageManagerImpl.****
> >> createPool:1378-**
> >> > > > > > > > >>
> >> StorageManagerImpl.createPool:****147-CreateStoragePoolCmd.**
> >> > > > > > > > >>
> >> execute:123-ApiDispatcher.****dispatch:162-ApiServer.**
> >> > > > > > > > >> queueCommand:505-ApiSe
> >> > > > > > > > >>
> >> rver.handleRequest:355-****ApiServlet.processRequest:302
> >> > > > > > > > >> 2013-07-10 17:08:28,923 ERROR
> >> [cloud.api.ApiServer]
> >> > > > > > > (catalina-exec-2:null)
> >> > > > > > > > >> unhandled exception executing api command:
> >> createStoragePool
> >> > > > > > > > >>
> >> com.cloud.utils.exception.****CloudRuntimeException: DB
> >> > > Exception
> >> > > > > on:
> >> > > > > > > > >>
> >> com.mysql.jdbc.****JDBC4PreparedStatement@4b2eb56****: INSERT
> >> > > INTO
> >> > > > > > > > >> storage_pool (
> >> > > > > > > > >> storage_pool.id <http://storage_pool.id>,
> >>
> >> storage_pool
> >> > > > > > > > >> .name, storage_pool.uuid,
> >> storage_pool.pool_type,
> >> > > > > > > storage_pool.created,
> >> > > > > > > > >> storage_pool.update_time,
> >> storage_pool.data_center_id,
> >> > > > > > > > >> storage_pool.pod_id,
> >> > > > > > > > >> storage_pool.availab
> >> > > > > > > > >> le_bytes, storage_pool.capacity_bytes,
> >> storage_pool.status,
> >> > > > > > > > >> storage_pool.scope,
> >> storage_pool.storage_provider_****id,
> >> > > > > > > > >> storage_pool.host_address, storage_pool.path,
> >> storage_
> >> > > > > > > > >> pool.port, storage_pool.user_info,
> >> storage_pool.cluster_id)
> >> > > VALUES
> >> > > > > > > (217,
> >> > > > > > > > >> _binary'ceph-rbd-pri-storage',
> >> > > > > > > > >>
> >> _binary'a226c9a1-da78-3f3a-****b5ac-e18b925c9634', 'RBD',
> >> > > '2013-07-1
> >> > > > > > > > >> 0 09:08:28', null, 2, 2, 0, 0, 'Up', null,
> >> null, null,
> >> > > > > _binary'ceph/
> >> > > > > > > > >> ceph.client.admin.keyring@10.****237.11.2/sc1<
> >> > > > > > > http://ceph.client.admin.**keyring@10.237.11.2/sc1<http://ceph.client.admin.keyring@10.237.11.2/sc1>
> >> >',
> >> > > > > > > > >> 6789, null, 2)
> >> > > > > > > > >> at
> >> > > > > > > > >>
> >> > > > > > >
> >> > >
> >> com.cloud.utils.db.****GenericDaoBase.persist(****
> >> GenericDaoBase.java:1342)
> >> > > > > > > > >> at
> >> > > > > > > > >>
> >> com.cloud.storage.dao.****StoragePoolDaoImpl.persist(**
> >> > > > > > > > >> StoragePoolDaoImpl.java:232)
> >> > > > > > > > >> at
> >> > > > > > > > >>
> >> com.cloud.utils.component.****ComponentInstantiationPostProc****
> >> > > > > > > > >>
> >> > > > > > >
> >> > > > >
> >> > >
> >> essor$InterceptorDispatcher.****intercept(****
> >> ComponentInstantiationPostProc***
> >> > > > > > > > >> *es
> >> > > > > > > > >>
> >> > > > > > > > >>
> >> > > > > > >
> >> > > > > > > --
> >> > > > > > > Prasanna.,
> >> > > > > > >
> >> > > > > > > ------------------------
> >> > > > > > > Powered by BigRock.com
> >> > > > > > >
> >> > > > > > >
> >> > > > >
> >> > > > > --
> >> > > > > Prasanna.,
> >> > > > >
> >> > > > > ------------------------
> >> > > > > Powered by BigRock.com
> >> > > > >
> >> > > > >
> >> > >
> >> > > --
> >> > > Prasanna.,
> >> > >
> >> > > ------------------------
> >> > > Powered by BigRock.com
> >> > >
> >> > >
> >>
> >> --
> >> Prasanna.,
> >>
> >> ------------------------
> >> Powered by BigRock.com
> >>
> >>
> >>
> >>
> >
--
Prasanna.,
------------------------
Powered by BigRock.com
RE: Problem in adding Ceph RBD as primary storage for CloudStack
4.1.0
Posted by Sudha Ponnaganti <su...@citrix.com>.
Hi Indra,
We are trailing behind you by couple of week to use Ceph and we will be using it with CloudStack 4.2.0. Your lead has been helpful. If you run in to any issues, can you log defects and those would help us to get head start with 4.2.0 validation.
Thanks
/sudha
-----Original Message-----
From: Indra Pramana [mailto:indra@sg.or.id]
Sent: Friday, July 12, 2013 2:12 AM
To: Wido den Hollander
Cc: users@cloudstack.apache.org; Chip Childers
Subject: Re: Problem in adding Ceph RBD as primary storage for CloudStack 4.1.0
Hi Wido,
Noted, can't wait for 4.2 to be released. :)
Dear Prasanna, Wido and all,
I just realised that while the system VMs are running, they are still not
accessible through the public IPs assigned to them. I have been waiting for
the SSVM to download the default CentOS template and it doesn't appear on
the template list.
I tried to SSH into the SSVM via the local link address from the KVM host,
and running the health check /usr/local/cloud/systemvm/ssvm-check.sh shows
that the VM cannot reach anywhere. It cannot reach the public DNS server (I
used Google's 8.8.8.8), cannot reach the management server, and cannot even
reach the public IP gateway.
Is it due to misconfiguration of the KVM network bridges? How can I see the
mapping between the NIC interfaces of the SSVM (eth0, eth1, eth2 and eth3)
and the actual physical NIC interfaces on the KVM hosts (eth0) and the
network bridges (cloudbr0, cloudbr1)? Any logs I can verify to ensure that
the VLAN and network bridging is working?
Appreciate any advice.
Thank you.
On Fri, Jul 12, 2013 at 4:19 PM, Wido den Hollander <wi...@widodh.nl> wrote:
> On 07/12/2013 10:14 AM, Indra Pramana wrote:
>
>> Hi Prasanna,
>>
>> I managed to fix the problem, thanks for your advice to turn the agent
>> log level to debug:
>>
>> https://cwiki.apache.org/**confluence/display/CLOUDSTACK/**
>> KVM+agent+debug<https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug>
>>
>> From the log, I found out that the agent on the KVM host tried to NFS
>> mount directly to 103.25.200.19:/mnt/vol1/sec-**
>> storage/template/tmpl/1/3,
>> which was not allowed by the NFS server due to its default configuration
>> to only allow to mount to /mnt/vol1/sec-storage (the root of the NFS
>> share).
>>
>>
> Ah, that's odd!
>
> Btw, in 4.2 you'll be able to deploy SSVMs on RBD as well, so that
> limitation will be gone.
>
> Wido
>
> After I changed the NFS server configuration to allow mount to all
>> sub-directories, re-export the NFS and voila, the system was able to
>> download the template and now both the system VMs (CPVM and SSVM) are
>> running!
>>
>> Many thanks for your help! :)
>>
>> Cheers.
>>
>>
>>
>> On Fri, Jul 12, 2013 at 3:31 PM, Indra Pramana <indra@sg.or.id
>> <ma...@sg.or.id>> wrote:
>>
>> Hi Prasanna,
>>
>> Good day to you, and thank you for your e-mail.
>>
>> Yes, the cloudstack-agent service is running on both the KVM hosts.
>> There is no "cloud" user being created though, when I installed the
>> agent. I installed the agent as root.
>>
>> root@hv-kvm-01:/home/indra# service cloudstack-agent status
>> * cloud-agent is running
>>
>> root@hv-kvm-01:/home/indra# su - cloud
>> Unknown id: cloud
>>
>> Please advise how can I resolve this problem, shall I create the
>> Unix "cloud" user manually? Basically I follow this instruction to
>> prepare the KVM host and install the CloudStack agent:
>>
>> http://cloudstack.apache.org/**docs/en-US/Apache_CloudStack/**
>> 4.1.0/html/Installation_Guide/**hypervisor-kvm-install-flow.**html<http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.1.0/html/Installation_Guide/hypervisor-kvm-install-flow.html>
>>
>> with this instruction from Wido on how to prepare libvirt with Ceph
>> RBD storage pool support:
>>
>> http://blog.widodh.nl/2013/06/**a-quick-note-on-running-**
>> cloudstack-with-rbd-on-ubuntu-**12-04/<http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/>
>>
>> I also have checked /var/log/cloud/agent/agent.log and I don't see
>> any error messages, except this error message which will show up
>> every time I restart the agent:
>>
>> 2013-07-12 15:22:47,454 ERROR [cloud.resource.**ServerResourceBase]
>> (main:null) Nics are not configured!
>> 2013-07-12 15:22:47,459 INFO [cloud.resource.**ServerResourceBase]
>> (main:null) Designating private to be nic eth0.5
>>
>> More logs can be found here: http://pastebin.com/yeNmCt7S
>>
>> I have configured the network bridges on the NIC interface as per
>> this instruction:
>>
>> http://cloudstack.apache.org/**docs/en-US/Apache_CloudStack/**
>> 4.1.0/html/Installation_Guide/**hypervisor-kvm-install-flow.**
>> html#hypervisor-host-install-**network<http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.1.0/html/Installation_Guide/hypervisor-kvm-install-flow.html#hypervisor-host-install-network>
>>
>> On the zone, I used advanced network configuration with just one
>> physical network for management, public and guest/private. I didn't
>> include storage, which by default the traffic will use the
>> management VLAN network.
>>
>> Please advise if there's anything else I might have been missing.
>>
>> Looking forward to your reply, thank you.
>>
>> Cheers.
>>
>>
>>
>> On Fri, Jul 12, 2013 at 2:56 PM, Prasanna Santhanam <tsp@apache.org
>> <ma...@apache.org>> wrote:
>>
>> Indeed, cloudstack will go through the allocation to startup the
>> system VMs too. So that process is failing to recognize the volume
>> (.qcow2) present on your NFS storage.
>>
>> Can you check if your cloudstack agent service is running with
>> the KVM
>> host? And it should've created the user cloud. $id cloud to check
>> if
>> there is a user.
>>
>> Did you see what's happening in the agent logs? These are under
>> /var/log/cloud/ on your host when the systemVMs are coming up.
>> If the
>> logs are not showing any useful information you can turn on debug
>> level for more verbosity.
>>
>> See here: https://cwiki.apache.org/**confluence/x/FgPMAQ<https://cwiki.apache.org/confluence/x/FgPMAQ>
>>
>> On Fri, Jul 12, 2013 at 02:40:35PM +0800, Indra Pramana wrote:
>> > Hi Prasanna,
>> >
>> > Good day to you, and thank you for your e-mail.
>> >
>> > Yes, when I export the NFS, I set the permission so that
>> normal user will
>> > be able to have read/write access to the files
>> (no_root_squash).
>> >
>> > I have tested and I can have read/write access from my KVM
>> hosts using
>> > normal user. BTW, there's no "cloud" user on the hosts, I
>> believe it's not
>> > created during cloudstack-agent installation?
>> >
>> > In any case, do you think the template issue and the storage
>> pool
>> > allocation issue might be related, or are they two different
>> problems
>> > altogether?
>> >
>> > Looking forward to your reply, thank you.
>> >
>> > Cheers.
>> >
>> >
>> >
>> > On Fri, Jul 12, 2013 at 2:26 PM, Prasanna Santhanam
>> <tsp@apache.org <ma...@apache.org>> wrote:
>> >
>> > > Can you access the file as user root? Or user cloud? The
>> cloudstack
>> > > agent on your KVM host runs as user cloud and the NFS
>> permissions
>> > > might be disallowing the volume (.qcow2) from being accessed.
>> > >
>> > > On Fri, Jul 12, 2013 at 02:16:41PM +0800, Indra Pramana
>> wrote:
>> > > > Hi Prasanna,
>> > > >
>> > > > Good day to you, and thank you for your e-mail.
>> > > >
>> > > > Yes, the file exists. I can access the file from the
>> management server
>> > > and
>> > > > the two hypervisors hosts if I mount manually.
>> > > >
>> > > > [root@cs-nas-01 /mnt/vol1/sec-storage/**template/tmpl/1/3]#
>> ls
>> > > > -la
>> > > > total
>> > > > 1418787
>> > > > drwxr-xr-x 2 root wheel 4 Jul 11 20:21
>> > > > .
>> > > > drwxr-xr-x 3 root wheel 3 Jul 11 20:17
>> > > > ..
>> > > > -rw-r--r-- 1 root wheel 725811200 Jul 11 20:21
>> > > > 425b9e5a-fbc7-4637-a33a-**fe9d0ed4fa98.qcow2
>> > > > -rw-r--r-- 1 root wheel 295 Jul 11 20:21
>> > > > template.properties
>> > > > [root@cs-nas-01 /mnt/vol1/sec-storage/**
>> template/tmpl/1/3]#
>> > > > pwd
>> > > > /mnt/vol1/sec-storage/**template/tmpl/1/3
>> > > >
>> > > >
>> > > > Any advise?
>> > > >
>> > > > Looking forward to your reply, thank you.
>> > > >
>> > > > Cheers.
>> > > >
>> > > >
>> > > >
>> > > > On Fri, Jul 12, 2013 at 2:07 PM, Prasanna Santhanam
>> <tsp@apache.org <ma...@apache.org>>
>>
>> > > wrote:
>> > > >
>> > > > > Can you check whether there is a file at:
>> > > > >
>> nfs://103.25.200.19/mnt/vol1/**sec-storage/template/tmpl/1/3/<http://103.25.200.19/mnt/vol1/sec-storage/template/tmpl/1/3/>
>> <http://103.25.200.19/mnt/**vol1/sec-storage/template/**tmpl/1/3/<http://103.25.200.19/mnt/vol1/sec-storage/template/tmpl/1/3/>
>> >
>>
>> > > > >
>> > > > > On Fri, Jul 12, 2013 at 01:59:34PM +0800, Indra Pramana
>> wrote:
>> > > > > > Hi Prasanna,
>> > > > > >
>> > > > > > Thanks for your e-mail.
>> > > > > >
>> > > > > > I have tried restarting the management server, and
>> the problem still
>> > > > > > persists. I even tried to re-do the installation and
>> configuration
>> > > again
>> > > > > > from scratch last night, but the problem still there.
>> > > > > >
>> > > > > > I also noted that on the beginning of the logs, I
>> found some error
>> > > > > messages
>> > > > > > saying that the template cannot be downloaded to the
>> pool. See this
>> > > logs:
>> > > > > >
>> > > > > > http://pastebin.com/BY1AVJ08
>> > > > > >
>> > > > > > It says it failed because cannot get volume from the
>> pool. Could it
>> > > be
>> > > > > > related, i.e. the absence of the template caused the
>> system VMs
>> > > cannot be
>> > > > > > created and started?
>> > > > > >
>> > > > > > I have ensured that I downloaded the system VM
>> template using
>> > > > > > cloud-install-sys-tmplt and verified that the
>> template is already
>> > > there
>> > > > > in
>> > > > > > the secondary storage server.
>> > > > > >
>> > > > > > Any advice is appreciated.
>> > > > > >
>> > > > > > Looking forward to your reply, thank you.
>> > > > > >
>> > > > > > Cheers.
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On Fri, Jul 12, 2013 at 1:21 PM, Prasanna Santhanam
>> <tsp@apache.org <ma...@apache.org>>
>>
>> > > > > wrote:
>> > > > > >
>> > > > > > > It looks like a previous attempt to start the
>> systemVMs has failed
>> > > > > > > putting the nfs storage in the avoid set. Did you
>> try restarting
>> > > your
>> > > > > > > management server?
>> > > > > > >
>> > > > > > > This line leads me to the above mentioned:
>> > > > > > > 2013-07-12 13:10:48,236 DEBUG
>> > > > > > > [storage.allocator.**AbstractStoragePoolAllocator]
>> > > (secstorage-1:null)
>> > > > > > > StoragePool is in avoid set, skipping this pool
>> > > > > > >
>> > > > > > >
>> > > > > > > On Fri, Jul 12, 2013 at 01:16:53PM +0800, Indra
>> Pramana wrote:
>> > > > > > > > Dear Wido and all,
>> > > > > > > >
>> > > > > > > > I have managed to get the hosts, primary and
>> secondary storage
>> > > > > running.
>> > > > > > > >
>> > > > > > > > - 2 KVM hypervisor hosts
>> > > > > > > > - One RBD primary storage
>> > > > > > > > - One NFS primary storage (for system VMs, since
>> I understand
>> > > that
>> > > > > system
>> > > > > > > > VMs cannot use RBD)
>> > > > > > > > - One NFS secondary storage
>> > > > > > > >
>> > > > > > > > However, now I am having problem with the system
>> VMs: CPVM and
>> > > SSVM,
>> > > > > > > unable
>> > > > > > > > to start.
>> > > > > > > >
>> > > > > > > > Excerpt from management-server.log file is here:
>> > > > > > > > http://pastebin.com/ENkpCALY
>> > > > > > > >
>> > > > > > > > It seems that the VMs were not able to be created
>> because unable
>> > > to
>> > > > > find
>> > > > > > > > suitable StoragePools.
>> > > > > > > >
>> > > > > > > > I understand that system VMs will be using the
>> NFS primary
>> > > storage
>> > > > > > > instead
>> > > > > > > > of RBD, so I have confirmed that I am able to
>> mount the primary
>> > > > > storage
>> > > > > > > via
>> > > > > > > > NFS and have read and write access, from both the
>> hypervisor and
>> > > the
>> > > > > > > > management server.
>> > > > > > > >
>> > > > > > > > Any advise how can I resolve the problem to make
>> both the system
>> > > VMs
>> > > > > > > > created and started?
>> > > > > > > >
>> > > > > > > > Looking forward to your reply, thank you.
>> > > > > > > >
>> > > > > > > > Cheers.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Fri, Jul 12, 2013 at 9:43 AM, Indra Pramana
>> <indra@sg.or.id <ma...@sg.or.id>>
>>
>> > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Hi Wido,
>> > > > > > > > >
>> > > > > > > > > Thanks for the advice, I'm now able to add the
>> RBD pool as
>> > > primary
>> > > > > > > storage.
>> > > > > > > > >
>> > > > > > > > > Many thanks! :)
>> > > > > > > > >
>> > > > > > > > > Cheers.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Thursday, July 11, 2013, Wido den Hollander
>> wrote:
>> > > > > > > > >
>> > > > > > > > >> Hi,
>> > > > > > > > >>
>> > > > > > > > >> On 07/10/2013 03:42 PM, Chip Childers wrote:
>> > > > > > > > >>
>> > > > > > > > >>> Cc'ing Wido, our resident Ceph expert. ;-)
>> > > > > > > > >>>
>> > > > > > > > >>>
>> > > > > > > > >> Hehe ;)
>> > > > > > > > >>
>> > > > > > > > >> On Wed, Jul 10, 2013 at 05:45:25PM +0800,
>> Indra Pramana
>> > > wrote:
>> > > > > > > > >>>
>> > > > > > > > >>>> Dear all,
>> > > > > > > > >>>>
>> > > > > > > > >>>> I am installing CloudStack 4.1.0 (upgraded
>> from 4.0.2) and I
>> > > > > also
>> > > > > > > have a
>> > > > > > > > >>>> Ceph cluster running. However, I am having
>> issues in adding
>> > > the
>> > > > > RBD
>> > > > > > > as
>> > > > > > > > >>>> primary storage. Tried to follow the
>> instruction here, but
>> > > > > unable to
>> > > > > > > > >>>> make
>> > > > > > > > >>>> it work:
>> > > > > > > > >>>>
>> > > > > > > > >>>>
>> http://ceph.com/docs/master/****rbd/rbd-cloudstack/<http://ceph.com/docs/master/**rbd/rbd-cloudstack/>
>> <
>> > > > > > > http://ceph.com/docs/master/**rbd/rbd-cloudstack/<http://ceph.com/docs/master/rbd/rbd-cloudstack/>
>> >
>> > > > > > > > >>>>
>> > > > > > > > >>>> I have setup a pool on the Ceph cluster. The
>> status of the
>> > > > > cluster
>> > > > > > > is
>> > > > > > > > >>>> healthy. Since I am using Ubuntu 12.04.2 LTS
>> (Precise) for
>> > > the
>> > > > > > > > >>>> hypervisors,
>> > > > > > > > >>>> I also have compiled libvirt manually to
>> ensure that the
>> > > version
>> > > > > > > 0.9.13
>> > > > > > > > >>>> is
>> > > > > > > > >>>> installed (previously it's 0.9.8).
>> > > > > > > > >>>>
>> > > > > > > > >>>>
>> > > > > > > > >> You can also use the Ubuntu Cloud Archive, I
>> still need to
>> > > get the
>> > > > > > > docs
>> > > > > > > > >> updated for that.
>> > > > > > > > >>
>> > > > > > > > >> I described the process in a blogpost:
>> > > > > > > http://blog.widodh.nl/2013/06/****<http://blog.widodh.nl/2013/06/**>
>> > > > > > > > >>
>> > >
>> a-quick-note-on-running-****cloudstack-with-rbd-on-ubuntu-**
>> **12-04/<
>> > > > > > >
>> > > > >
>> > >
>> http://blog.widodh.nl/2013/06/**a-quick-note-on-running-**
>> cloudstack-with-rbd-on-ubuntu-**12-04/<http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/>
>> > > > > > > >
>> > > > > > > > >>
>> > > > > > > > >> indra@hv-kvm-01:~/rbd$ ceph
>> > > > > > > > >>>> ceph> health
>> > > > > > > > >>>> HEALTH_OK
>> > > > > > > > >>>>
>> > > > > > > > >>>> indra@hv-kvm-01:~$ ceph osd lspools
>> > > > > > > > >>>> 0 data,1 metadata,2 rbd,3 sc1,
>> > > > > > > > >>>>
>> > > > > > > > >>>> root@hv-kvm-01:/home/indra# libvirtd
>> --version
>> > > > > > > > >>>> libvirtd (libvirt) 0.9.13
>> > > > > > > > >>>>
>> > > > > > > > >>>> I tried to add Primary Storage into the
>> Cloudstack zone
>> > > which I
>> > > > > have
>> > > > > > > > >>>> created:
>> > > > > > > > >>>>
>> > > > > > > > >>>> Add Primary Storage:
>> > > > > > > > >>>>
>> > > > > > > > >>>> Zone: my zone name
>> > > > > > > > >>>> Pod: my pod name
>> > > > > > > > >>>> Cluster: my cluster name
>> > > > > > > > >>>> Name: ceph-rbd-pri-storage
>> > > > > > > > >>>> Protocol: RBD
>> > > > > > > > >>>> RADOS Monitor: my first Ceph monitor IP
>> address
>> > > > > > > > >>>> RADOS Pool: sc1 (the pool name on Ceph
>> cluster)
>> > > > > > > > >>>> RADOS User: client.admin
>> > > > > > > > >>>> RADOS Secret:
>> /etc/ceph/ceph.client.admin.****keyring (keyring
>> > > > > file
>> > > > > > > > >>>> location)
>> > > > > > > > >>>>
>> > > > > > > > >>>
>> > > > > > > > >> This is your problem. That shouldn't be the
>> location of the
>> > > file,
>> > > > > but
>> > > > > > > it
>> > > > > > > > >> should be the secret, which is a base64
>> encoded string.
>> > > > > > > > >>
>> > > > > > > > >> $ ceph auth list
>> > > > > > > > >>
>> > > > > > > > >> That should tell you what the secret is.
>> > > > > > > > >>
>> > > > > > > > >> Storage Tags: rbd
>> > > > > > > > >>
>> > > > > > > > >> This is the error message when I tried to add
>> the primary
>> > > storage
>> > > > > by
>> > > > > > > > >> clicking OK:
>> > > > > > > > >>
>> > > > > > > > >> DB Exception on:
>> > > com.mysql.jdbc.****JDBC4PreparedStatement@4b2eb56
>> > > > > **:
>> > > > > > > > >> INSERT INTO
>> > > > > > > > >> storage_pool (storage_pool.id
>> <http://storage_pool.id>, storage_pool.name
>> <http://storage_pool.name>,
>>
>> > > > > storage_pool.uuid,
>> > > > > > > > >> storage_pool.pool_type, storage_pool.created,
>> > > > > > > storage_pool.update_time,
>> > > > > > > > >> storage_pool.data_center_id,
>> storage_pool.pod_id,
>> > > > > > > > >> storage_pool.available_bytes,
>> storage_pool.capacity_bytes,
>> > > > > > > > >> storage_pool.status, storage_pool.scope,
>> > > > > > > storage_pool.storage_provider_****
>> > > > > > > > >> id,
>> > > > > > > > >> storage_pool.host_address, storage_pool.path,
>> > > storage_pool.port,
>> > > > > > > > >> storage_pool.user_info,
>> storage_pool.cluster_id) VALUES (217,
>> > > > > > > > >> _binary'ceph-rbd-pri-storage',
>> > > > > > > > >>
>> _binary'a226c9a1-da78-3f3a-****b5ac-e18b925c9634', 'RBD',
>> > > > > '2013-07-10
>> > > > > > > > >> 09:08:28', null, 2, 2, 0, 0, 'Up', null, null,
>> null,
>> > > _binary'ceph/
>> > > > > > > > >> ceph.client.admin.keyring@10.****237.11.2/sc1<
>> > > > > > > http://ceph.client.admin.**keyring@10.237.11.2/sc1<http://ceph.client.admin.keyring@10.237.11.2/sc1>
>> >',
>> > > > > > > > >> 6789, null, 2)
>> > > > > > > > >>
>> > > > > > > > >> On the management-server.log file:
>> > > > > > > > >>
>> > > > > > > > >> 2013-07-10 17:08:28,845 DEBUG
>> [cloud.api.ApiServlet]
>> > > > > > > > >> (catalina-exec-2:null)
>> > > > > > > > >> ===START=== 192.168.0.100 -- GET
>> > > > > > > > >>
>> command=createStoragePool&****zoneid=c116950e-e4ae-4f23-****a7e7-
>> > > > > > > > >>
>> 74a75c4ee638&podId=a748b063-****3a83-4175-a0e9-de39118fe5ce&**
>> > > > > > > > >>
>> clusterid=1f87eb09-324d-4d49-****83c2-88d84d7a15df&name=ceph-****
>> > > > > > > > >>
>> rbd-pri-storage&url=rbd%3A%2F%****2Fclient.admin%3A_etc%2Fc
>> > > > > > > > >>
>> eph%2Fceph.client.admin.****keyring%4010.237.11.2%2Fsc1&**
>> > > > > > > > >>
>> > > > >
>> tags=rbd&response=json&****sessionkey=****
>> rDRfWpqeKfQKbKZtHr398ULV%2F8k%****
>> > > > > > > > >> 3D&_=1373447307839
>> > > > > > > > >> 2013-07-10 17:08:28,862 DEBUG
>> > > [cloud.storage.****StorageManagerImpl]
>> > > > > > > > >> (catalina-exec-2:null) createPool Params @
>> scheme - rbd
>> > > > > storageHost -
>> > > > > > > null
>> > > > > > > > >> hostPath - /ceph/ceph.client
>> > > > > > > > >> .admin.keyring@10.237.11.2/sc1
>> <http://admin.keyring@10.237.**11.2/sc1<http://admin.keyring@10.237.11.2/sc1>>
>> port - -1
>>
>> > > > > > > > >> 2013-07-10 17:08:28,918 DEBUG
>> > > [cloud.storage.****StorageManagerImpl]
>> > > > > > > > >> (catalina-exec-2:null) In createPool Setting
>> poolId - 217
>> > > uuid -
>> > > > > > > > >> a226c9a1-da78-3f3a-b5ac-****e18b925c9634 z
>> > > > > > > > >> oneId - 2 podId - 2 poolName -
>> ceph-rbd-pri-storage
>> > > > > > > > >> 2013-07-10 17:08:28,921 DEBUG
>> [db.Transaction.Transaction]
>> > > > > > > > >> (catalina-exec-2:null) Rolling back the
>> transaction: Time = 3
>> > > > > Name =
>> > > > > > > > >> persist; called by -Transaction.rollbac
>> > > > > > > > >>
>> k:890-Transaction.removeUpTo:****833-Transaction.close:657-**
>> > > > > > > > >>
>> TransactionContextBuilder.****interceptException:63-**
>> > > > > > > > >>
>> > > > >
>> ComponentInstantiationPostProc****essor$InterceptorDispatcher.**
>> **interce
>> > > > > > > > >> pt:133-StorageManagerImpl.****
>> createPool:1378-**
>> > > > > > > > >>
>> StorageManagerImpl.createPool:****147-CreateStoragePoolCmd.**
>> > > > > > > > >>
>> execute:123-ApiDispatcher.****dispatch:162-ApiServer.**
>> > > > > > > > >> queueCommand:505-ApiSe
>> > > > > > > > >>
>> rver.handleRequest:355-****ApiServlet.processRequest:302
>> > > > > > > > >> 2013-07-10 17:08:28,923 ERROR
>> [cloud.api.ApiServer]
>> > > > > > > (catalina-exec-2:null)
>> > > > > > > > >> unhandled exception executing api command:
>> createStoragePool
>> > > > > > > > >>
>> com.cloud.utils.exception.****CloudRuntimeException: DB
>> > > Exception
>> > > > > on:
>> > > > > > > > >>
>> com.mysql.jdbc.****JDBC4PreparedStatement@4b2eb56****: INSERT
>> > > INTO
>> > > > > > > > >> storage_pool (
>> > > > > > > > >> storage_pool.id <http://storage_pool.id>,
>>
>> storage_pool
>> > > > > > > > >> .name, storage_pool.uuid,
>> storage_pool.pool_type,
>> > > > > > > storage_pool.created,
>> > > > > > > > >> storage_pool.update_time,
>> storage_pool.data_center_id,
>> > > > > > > > >> storage_pool.pod_id,
>> > > > > > > > >> storage_pool.availab
>> > > > > > > > >> le_bytes, storage_pool.capacity_bytes,
>> storage_pool.status,
>> > > > > > > > >> storage_pool.scope,
>> storage_pool.storage_provider_****id,
>> > > > > > > > >> storage_pool.host_address, storage_pool.path,
>> storage_
>> > > > > > > > >> pool.port, storage_pool.user_info,
>> storage_pool.cluster_id)
>> > > VALUES
>> > > > > > > (217,
>> > > > > > > > >> _binary'ceph-rbd-pri-storage',
>> > > > > > > > >>
>> _binary'a226c9a1-da78-3f3a-****b5ac-e18b925c9634', 'RBD',
>> > > '2013-07-1
>> > > > > > > > >> 0 09:08:28', null, 2, 2, 0, 0, 'Up', null,
>> null, null,
>> > > > > _binary'ceph/
>> > > > > > > > >> ceph.client.admin.keyring@10.****237.11.2/sc1<
>> > > > > > > http://ceph.client.admin.**keyring@10.237.11.2/sc1<http://ceph.client.admin.keyring@10.237.11.2/sc1>
>> >',
>> > > > > > > > >> 6789, null, 2)
>> > > > > > > > >> at
>> > > > > > > > >>
>> > > > > > >
>> > >
>> com.cloud.utils.db.****GenericDaoBase.persist(****
>> GenericDaoBase.java:1342)
>> > > > > > > > >> at
>> > > > > > > > >>
>> com.cloud.storage.dao.****StoragePoolDaoImpl.persist(**
>> > > > > > > > >> StoragePoolDaoImpl.java:232)
>> > > > > > > > >> at
>> > > > > > > > >>
>> com.cloud.utils.component.****ComponentInstantiationPostProc****
>> > > > > > > > >>
>> > > > > > >
>> > > > >
>> > >
>> essor$InterceptorDispatcher.****intercept(****
>> ComponentInstantiationPostProc***
>> > > > > > > > >> *es
>> > > > > > > > >>
>> > > > > > > > >>
>> > > > > > >
>> > > > > > > --
>> > > > > > > Prasanna.,
>> > > > > > >
>> > > > > > > ------------------------
>> > > > > > > Powered by BigRock.com
>> > > > > > >
>> > > > > > >
>> > > > >
>> > > > > --
>> > > > > Prasanna.,
>> > > > >
>> > > > > ------------------------
>> > > > > Powered by BigRock.com
>> > > > >
>> > > > >
>> > >
>> > > --
>> > > Prasanna.,
>> > >
>> > > ------------------------
>> > > Powered by BigRock.com
>> > >
>> > >
>>
>> --
>> Prasanna.,
>>
>> ------------------------
>> Powered by BigRock.com
>>
>>
>>
>>
>
Re: Problem in adding Ceph RBD as primary storage for CloudStack 4.1.0
Posted by Indra Pramana <in...@sg.or.id>.
Hi Wido,
Noted, can't wait for 4.2 to be released. :)
Dear Prasanna, Wido and all,
I just realised that while the system VMs are running, they are still not
accessible through the public IPs assigned to them. I have been waiting for
the SSVM to download the default CentOS template and it doesn't appear on
the template list.
I tried to SSH into the SSVM via the local link address from the KVM host,
and running the health check /usr/local/cloud/systemvm/ssvm-check.sh shows
that the VM cannot reach anywhere. It cannot reach the public DNS server (I
used Google's 8.8.8.8), cannot reach the management server, and cannot even
reach the public IP gateway.
Is it due to misconfiguration of the KVM network bridges? How can I see the
mapping between the NIC interfaces of the SSVM (eth0, eth1, eth2 and eth3)
and the actual physical NIC interfaces on the KVM hosts (eth0) and the
network bridges (cloudbr0, cloudbr1)? Any logs I can verify to ensure that
the VLAN and network bridging is working?
Appreciate any advice.
Thank you.
On Fri, Jul 12, 2013 at 4:19 PM, Wido den Hollander <wi...@widodh.nl> wrote:
> On 07/12/2013 10:14 AM, Indra Pramana wrote:
>
>> Hi Prasanna,
>>
>> I managed to fix the problem, thanks for your advice to turn the agent
>> log level to debug:
>>
>> https://cwiki.apache.org/**confluence/display/CLOUDSTACK/**
>> KVM+agent+debug<https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug>
>>
>> From the log, I found out that the agent on the KVM host tried to NFS
>> mount directly to 103.25.200.19:/mnt/vol1/sec-**
>> storage/template/tmpl/1/3,
>> which was not allowed by the NFS server due to its default configuration
>> to only allow to mount to /mnt/vol1/sec-storage (the root of the NFS
>> share).
>>
>>
> Ah, that's odd!
>
> Btw, in 4.2 you'll be able to deploy SSVMs on RBD as well, so that
> limitation will be gone.
>
> Wido
>
> After I changed the NFS server configuration to allow mount to all
>> sub-directories, re-export the NFS and voila, the system was able to
>> download the template and now both the system VMs (CPVM and SSVM) are
>> running!
>>
>> Many thanks for your help! :)
>>
>> Cheers.
>>
>>
>>
>> On Fri, Jul 12, 2013 at 3:31 PM, Indra Pramana <indra@sg.or.id
>> <ma...@sg.or.id>> wrote:
>>
>> Hi Prasanna,
>>
>> Good day to you, and thank you for your e-mail.
>>
>> Yes, the cloudstack-agent service is running on both the KVM hosts.
>> There is no "cloud" user being created though, when I installed the
>> agent. I installed the agent as root.
>>
>> root@hv-kvm-01:/home/indra# service cloudstack-agent status
>> * cloud-agent is running
>>
>> root@hv-kvm-01:/home/indra# su - cloud
>> Unknown id: cloud
>>
>> Please advise how can I resolve this problem, shall I create the
>> Unix "cloud" user manually? Basically I follow this instruction to
>> prepare the KVM host and install the CloudStack agent:
>>
>> http://cloudstack.apache.org/**docs/en-US/Apache_CloudStack/**
>> 4.1.0/html/Installation_Guide/**hypervisor-kvm-install-flow.**html<http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.1.0/html/Installation_Guide/hypervisor-kvm-install-flow.html>
>>
>> with this instruction from Wido on how to prepare libvirt with Ceph
>> RBD storage pool support:
>>
>> http://blog.widodh.nl/2013/06/**a-quick-note-on-running-**
>> cloudstack-with-rbd-on-ubuntu-**12-04/<http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/>
>>
>> I also have checked /var/log/cloud/agent/agent.log and I don't see
>> any error messages, except this error message which will show up
>> every time I restart the agent:
>>
>> 2013-07-12 15:22:47,454 ERROR [cloud.resource.**ServerResourceBase]
>> (main:null) Nics are not configured!
>> 2013-07-12 15:22:47,459 INFO [cloud.resource.**ServerResourceBase]
>> (main:null) Designating private to be nic eth0.5
>>
>> More logs can be found here: http://pastebin.com/yeNmCt7S
>>
>> I have configured the network bridges on the NIC interface as per
>> this instruction:
>>
>> http://cloudstack.apache.org/**docs/en-US/Apache_CloudStack/**
>> 4.1.0/html/Installation_Guide/**hypervisor-kvm-install-flow.**
>> html#hypervisor-host-install-**network<http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.1.0/html/Installation_Guide/hypervisor-kvm-install-flow.html#hypervisor-host-install-network>
>>
>> On the zone, I used advanced network configuration with just one
>> physical network for management, public and guest/private. I didn't
>> include storage, which by default the traffic will use the
>> management VLAN network.
>>
>> Please advise if there's anything else I might have been missing.
>>
>> Looking forward to your reply, thank you.
>>
>> Cheers.
>>
>>
>>
>> On Fri, Jul 12, 2013 at 2:56 PM, Prasanna Santhanam <tsp@apache.org
>> <ma...@apache.org>> wrote:
>>
>> Indeed, cloudstack will go through the allocation to startup the
>> system VMs too. So that process is failing to recognize the volume
>> (.qcow2) present on your NFS storage.
>>
>> Can you check if your cloudstack agent service is running with
>> the KVM
>> host? And it should've created the user cloud. $id cloud to check
>> if
>> there is a user.
>>
>> Did you see what's happening in the agent logs? These are under
>> /var/log/cloud/ on your host when the systemVMs are coming up.
>> If the
>> logs are not showing any useful information you can turn on debug
>> level for more verbosity.
>>
>> See here: https://cwiki.apache.org/**confluence/x/FgPMAQ<https://cwiki.apache.org/confluence/x/FgPMAQ>
>>
>> On Fri, Jul 12, 2013 at 02:40:35PM +0800, Indra Pramana wrote:
>> > Hi Prasanna,
>> >
>> > Good day to you, and thank you for your e-mail.
>> >
>> > Yes, when I export the NFS, I set the permission so that
>> normal user will
>> > be able to have read/write access to the files
>> (no_root_squash).
>> >
>> > I have tested and I can have read/write access from my KVM
>> hosts using
>> > normal user. BTW, there's no "cloud" user on the hosts, I
>> believe it's not
>> > created during cloudstack-agent installation?
>> >
>> > In any case, do you think the template issue and the storage
>> pool
>> > allocation issue might be related, or are they two different
>> problems
>> > altogether?
>> >
>> > Looking forward to your reply, thank you.
>> >
>> > Cheers.
>> >
>> >
>> >
>> > On Fri, Jul 12, 2013 at 2:26 PM, Prasanna Santhanam
>> <tsp@apache.org <ma...@apache.org>> wrote:
>> >
>> > > Can you access the file as user root? Or user cloud? The
>> cloudstack
>> > > agent on your KVM host runs as user cloud and the NFS
>> permissions
>> > > might be disallowing the volume (.qcow2) from being accessed.
>> > >
>> > > On Fri, Jul 12, 2013 at 02:16:41PM +0800, Indra Pramana
>> wrote:
>> > > > Hi Prasanna,
>> > > >
>> > > > Good day to you, and thank you for your e-mail.
>> > > >
>> > > > Yes, the file exists. I can access the file from the
>> management server
>> > > and
>> > > > the two hypervisors hosts if I mount manually.
>> > > >
>> > > > [root@cs-nas-01 /mnt/vol1/sec-storage/**template/tmpl/1/3]#
>> ls
>> > > > -la
>> > > > total
>> > > > 1418787
>> > > > drwxr-xr-x 2 root wheel 4 Jul 11 20:21
>> > > > .
>> > > > drwxr-xr-x 3 root wheel 3 Jul 11 20:17
>> > > > ..
>> > > > -rw-r--r-- 1 root wheel 725811200 Jul 11 20:21
>> > > > 425b9e5a-fbc7-4637-a33a-**fe9d0ed4fa98.qcow2
>> > > > -rw-r--r-- 1 root wheel 295 Jul 11 20:21
>> > > > template.properties
>> > > > [root@cs-nas-01 /mnt/vol1/sec-storage/**
>> template/tmpl/1/3]#
>> > > > pwd
>> > > > /mnt/vol1/sec-storage/**template/tmpl/1/3
>> > > >
>> > > >
>> > > > Any advise?
>> > > >
>> > > > Looking forward to your reply, thank you.
>> > > >
>> > > > Cheers.
>> > > >
>> > > >
>> > > >
>> > > > On Fri, Jul 12, 2013 at 2:07 PM, Prasanna Santhanam
>> <tsp@apache.org <ma...@apache.org>>
>>
>> > > wrote:
>> > > >
>> > > > > Can you check whether there is a file at:
>> > > > >
>> nfs://103.25.200.19/mnt/vol1/**sec-storage/template/tmpl/1/3/<http://103.25.200.19/mnt/vol1/sec-storage/template/tmpl/1/3/>
>> <http://103.25.200.19/mnt/**vol1/sec-storage/template/**tmpl/1/3/<http://103.25.200.19/mnt/vol1/sec-storage/template/tmpl/1/3/>
>> >
>>
>> > > > >
>> > > > > On Fri, Jul 12, 2013 at 01:59:34PM +0800, Indra Pramana
>> wrote:
>> > > > > > Hi Prasanna,
>> > > > > >
>> > > > > > Thanks for your e-mail.
>> > > > > >
>> > > > > > I have tried restarting the management server, and
>> the problem still
>> > > > > > persists. I even tried to re-do the installation and
>> configuration
>> > > again
>> > > > > > from scratch last night, but the problem still there.
>> > > > > >
>> > > > > > I also noted that on the beginning of the logs, I
>> found some error
>> > > > > messages
>> > > > > > saying that the template cannot be downloaded to the
>> pool. See this
>> > > logs:
>> > > > > >
>> > > > > > http://pastebin.com/BY1AVJ08
>> > > > > >
>> > > > > > It says it failed because cannot get volume from the
>> pool. Could it
>> > > be
>> > > > > > related, i.e. the absence of the template caused the
>> system VMs
>> > > cannot be
>> > > > > > created and started?
>> > > > > >
>> > > > > > I have ensured that I downloaded the system VM
>> template using
>> > > > > > cloud-install-sys-tmplt and verified that the
>> template is already
>> > > there
>> > > > > in
>> > > > > > the secondary storage server.
>> > > > > >
>> > > > > > Any advice is appreciated.
>> > > > > >
>> > > > > > Looking forward to your reply, thank you.
>> > > > > >
>> > > > > > Cheers.
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On Fri, Jul 12, 2013 at 1:21 PM, Prasanna Santhanam
>> <tsp@apache.org <ma...@apache.org>>
>>
>> > > > > wrote:
>> > > > > >
>> > > > > > > It looks like a previous attempt to start the
>> systemVMs has failed
>> > > > > > > putting the nfs storage in the avoid set. Did you
>> try restarting
>> > > your
>> > > > > > > management server?
>> > > > > > >
>> > > > > > > This line leads me to the above mentioned:
>> > > > > > > 2013-07-12 13:10:48,236 DEBUG
>> > > > > > > [storage.allocator.**AbstractStoragePoolAllocator]
>> > > (secstorage-1:null)
>> > > > > > > StoragePool is in avoid set, skipping this pool
>> > > > > > >
>> > > > > > >
>> > > > > > > On Fri, Jul 12, 2013 at 01:16:53PM +0800, Indra
>> Pramana wrote:
>> > > > > > > > Dear Wido and all,
>> > > > > > > >
>> > > > > > > > I have managed to get the hosts, primary and
>> secondary storage
>> > > > > running.
>> > > > > > > >
>> > > > > > > > - 2 KVM hypervisor hosts
>> > > > > > > > - One RBD primary storage
>> > > > > > > > - One NFS primary storage (for system VMs, since
>> I understand
>> > > that
>> > > > > system
>> > > > > > > > VMs cannot use RBD)
>> > > > > > > > - One NFS secondary storage
>> > > > > > > >
>> > > > > > > > However, now I am having problem with the system
>> VMs: CPVM and
>> > > SSVM,
>> > > > > > > unable
>> > > > > > > > to start.
>> > > > > > > >
>> > > > > > > > Excerpt from management-server.log file is here:
>> > > > > > > > http://pastebin.com/ENkpCALY
>> > > > > > > >
>> > > > > > > > It seems that the VMs were not able to be created
>> because unable
>> > > to
>> > > > > find
>> > > > > > > > suitable StoragePools.
>> > > > > > > >
>> > > > > > > > I understand that system VMs will be using the
>> NFS primary
>> > > storage
>> > > > > > > instead
>> > > > > > > > of RBD, so I have confirmed that I am able to
>> mount the primary
>> > > > > storage
>> > > > > > > via
>> > > > > > > > NFS and have read and write access, from both the
>> hypervisor and
>> > > the
>> > > > > > > > management server.
>> > > > > > > >
>> > > > > > > > Any advise how can I resolve the problem to make
>> both the system
>> > > VMs
>> > > > > > > > created and started?
>> > > > > > > >
>> > > > > > > > Looking forward to your reply, thank you.
>> > > > > > > >
>> > > > > > > > Cheers.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Fri, Jul 12, 2013 at 9:43 AM, Indra Pramana
>> <indra@sg.or.id <ma...@sg.or.id>>
>>
>> > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Hi Wido,
>> > > > > > > > >
>> > > > > > > > > Thanks for the advice, I'm now able to add the
>> RBD pool as
>> > > primary
>> > > > > > > storage.
>> > > > > > > > >
>> > > > > > > > > Many thanks! :)
>> > > > > > > > >
>> > > > > > > > > Cheers.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Thursday, July 11, 2013, Wido den Hollander
>> wrote:
>> > > > > > > > >
>> > > > > > > > >> Hi,
>> > > > > > > > >>
>> > > > > > > > >> On 07/10/2013 03:42 PM, Chip Childers wrote:
>> > > > > > > > >>
>> > > > > > > > >>> Cc'ing Wido, our resident Ceph expert. ;-)
>> > > > > > > > >>>
>> > > > > > > > >>>
>> > > > > > > > >> Hehe ;)
>> > > > > > > > >>
>> > > > > > > > >> On Wed, Jul 10, 2013 at 05:45:25PM +0800,
>> Indra Pramana
>> > > wrote:
>> > > > > > > > >>>
>> > > > > > > > >>>> Dear all,
>> > > > > > > > >>>>
>> > > > > > > > >>>> I am installing CloudStack 4.1.0 (upgraded
>> from 4.0.2) and I
>> > > > > also
>> > > > > > > have a
>> > > > > > > > >>>> Ceph cluster running. However, I am having
>> issues in adding
>> > > the
>> > > > > RBD
>> > > > > > > as
>> > > > > > > > >>>> primary storage. Tried to follow the
>> instruction here, but
>> > > > > unable to
>> > > > > > > > >>>> make
>> > > > > > > > >>>> it work:
>> > > > > > > > >>>>
>> > > > > > > > >>>>
>> http://ceph.com/docs/master/****rbd/rbd-cloudstack/<http://ceph.com/docs/master/**rbd/rbd-cloudstack/>
>> <
>> > > > > > > http://ceph.com/docs/master/**rbd/rbd-cloudstack/<http://ceph.com/docs/master/rbd/rbd-cloudstack/>
>> >
>> > > > > > > > >>>>
>> > > > > > > > >>>> I have setup a pool on the Ceph cluster. The
>> status of the
>> > > > > cluster
>> > > > > > > is
>> > > > > > > > >>>> healthy. Since I am using Ubuntu 12.04.2 LTS
>> (Precise) for
>> > > the
>> > > > > > > > >>>> hypervisors,
>> > > > > > > > >>>> I also have compiled libvirt manually to
>> ensure that the
>> > > version
>> > > > > > > 0.9.13
>> > > > > > > > >>>> is
>> > > > > > > > >>>> installed (previously it's 0.9.8).
>> > > > > > > > >>>>
>> > > > > > > > >>>>
>> > > > > > > > >> You can also use the Ubuntu Cloud Archive, I
>> still need to
>> > > get the
>> > > > > > > docs
>> > > > > > > > >> updated for that.
>> > > > > > > > >>
>> > > > > > > > >> I described the process in a blogpost:
>> > > > > > > http://blog.widodh.nl/2013/06/****<http://blog.widodh.nl/2013/06/**>
>> > > > > > > > >>
>> > >
>> a-quick-note-on-running-****cloudstack-with-rbd-on-ubuntu-**
>> **12-04/<
>> > > > > > >
>> > > > >
>> > >
>> http://blog.widodh.nl/2013/06/**a-quick-note-on-running-**
>> cloudstack-with-rbd-on-ubuntu-**12-04/<http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/>
>> > > > > > > >
>> > > > > > > > >>
>> > > > > > > > >> indra@hv-kvm-01:~/rbd$ ceph
>> > > > > > > > >>>> ceph> health
>> > > > > > > > >>>> HEALTH_OK
>> > > > > > > > >>>>
>> > > > > > > > >>>> indra@hv-kvm-01:~$ ceph osd lspools
>> > > > > > > > >>>> 0 data,1 metadata,2 rbd,3 sc1,
>> > > > > > > > >>>>
>> > > > > > > > >>>> root@hv-kvm-01:/home/indra# libvirtd
>> --version
>> > > > > > > > >>>> libvirtd (libvirt) 0.9.13
>> > > > > > > > >>>>
>> > > > > > > > >>>> I tried to add Primary Storage into the
>> Cloudstack zone
>> > > which I
>> > > > > have
>> > > > > > > > >>>> created:
>> > > > > > > > >>>>
>> > > > > > > > >>>> Add Primary Storage:
>> > > > > > > > >>>>
>> > > > > > > > >>>> Zone: my zone name
>> > > > > > > > >>>> Pod: my pod name
>> > > > > > > > >>>> Cluster: my cluster name
>> > > > > > > > >>>> Name: ceph-rbd-pri-storage
>> > > > > > > > >>>> Protocol: RBD
>> > > > > > > > >>>> RADOS Monitor: my first Ceph monitor IP
>> address
>> > > > > > > > >>>> RADOS Pool: sc1 (the pool name on Ceph
>> cluster)
>> > > > > > > > >>>> RADOS User: client.admin
>> > > > > > > > >>>> RADOS Secret:
>> /etc/ceph/ceph.client.admin.****keyring (keyring
>> > > > > file
>> > > > > > > > >>>> location)
>> > > > > > > > >>>>
>> > > > > > > > >>>
>> > > > > > > > >> This is your problem. That shouldn't be the
>> location of the
>> > > file,
>> > > > > but
>> > > > > > > it
>> > > > > > > > >> should be the secret, which is a base64
>> encoded string.
>> > > > > > > > >>
>> > > > > > > > >> $ ceph auth list
>> > > > > > > > >>
>> > > > > > > > >> That should tell you what the secret is.
>> > > > > > > > >>
>> > > > > > > > >> Storage Tags: rbd
>> > > > > > > > >>
>> > > > > > > > >> This is the error message when I tried to add
>> the primary
>> > > storage
>> > > > > by
>> > > > > > > > >> clicking OK:
>> > > > > > > > >>
>> > > > > > > > >> DB Exception on:
>> > > com.mysql.jdbc.****JDBC4PreparedStatement@4b2eb56
>> > > > > **:
>> > > > > > > > >> INSERT INTO
>> > > > > > > > >> storage_pool (storage_pool.id
>> <http://storage_pool.id>, storage_pool.name
>> <http://storage_pool.name>,
>>
>> > > > > storage_pool.uuid,
>> > > > > > > > >> storage_pool.pool_type, storage_pool.created,
>> > > > > > > storage_pool.update_time,
>> > > > > > > > >> storage_pool.data_center_id,
>> storage_pool.pod_id,
>> > > > > > > > >> storage_pool.available_bytes,
>> storage_pool.capacity_bytes,
>> > > > > > > > >> storage_pool.status, storage_pool.scope,
>> > > > > > > storage_pool.storage_provider_****
>> > > > > > > > >> id,
>> > > > > > > > >> storage_pool.host_address, storage_pool.path,
>> > > storage_pool.port,
>> > > > > > > > >> storage_pool.user_info,
>> storage_pool.cluster_id) VALUES (217,
>> > > > > > > > >> _binary'ceph-rbd-pri-storage',
>> > > > > > > > >>
>> _binary'a226c9a1-da78-3f3a-****b5ac-e18b925c9634', 'RBD',
>> > > > > '2013-07-10
>> > > > > > > > >> 09:08:28', null, 2, 2, 0, 0, 'Up', null, null,
>> null,
>> > > _binary'ceph/
>> > > > > > > > >> ceph.client.admin.keyring@10.****237.11.2/sc1<
>> > > > > > > http://ceph.client.admin.**keyring@10.237.11.2/sc1<http://ceph.client.admin.keyring@10.237.11.2/sc1>
>> >',
>> > > > > > > > >> 6789, null, 2)
>> > > > > > > > >>
>> > > > > > > > >> On the management-server.log file:
>> > > > > > > > >>
>> > > > > > > > >> 2013-07-10 17:08:28,845 DEBUG
>> [cloud.api.ApiServlet]
>> > > > > > > > >> (catalina-exec-2:null)
>> > > > > > > > >> ===START=== 192.168.0.100 -- GET
>> > > > > > > > >>
>> command=createStoragePool&****zoneid=c116950e-e4ae-4f23-****a7e7-
>> > > > > > > > >>
>> 74a75c4ee638&podId=a748b063-****3a83-4175-a0e9-de39118fe5ce&**
>> > > > > > > > >>
>> clusterid=1f87eb09-324d-4d49-****83c2-88d84d7a15df&name=ceph-****
>> > > > > > > > >>
>> rbd-pri-storage&url=rbd%3A%2F%****2Fclient.admin%3A_etc%2Fc
>> > > > > > > > >>
>> eph%2Fceph.client.admin.****keyring%4010.237.11.2%2Fsc1&**
>> > > > > > > > >>
>> > > > >
>> tags=rbd&response=json&****sessionkey=****
>> rDRfWpqeKfQKbKZtHr398ULV%2F8k%****
>> > > > > > > > >> 3D&_=1373447307839
>> > > > > > > > >> 2013-07-10 17:08:28,862 DEBUG
>> > > [cloud.storage.****StorageManagerImpl]
>> > > > > > > > >> (catalina-exec-2:null) createPool Params @
>> scheme - rbd
>> > > > > storageHost -
>> > > > > > > null
>> > > > > > > > >> hostPath - /ceph/ceph.client
>> > > > > > > > >> .admin.keyring@10.237.11.2/sc1
>> <http://admin.keyring@10.237.**11.2/sc1<http://admin.keyring@10.237.11.2/sc1>>
>> port - -1
>>
>> > > > > > > > >> 2013-07-10 17:08:28,918 DEBUG
>> > > [cloud.storage.****StorageManagerImpl]
>> > > > > > > > >> (catalina-exec-2:null) In createPool Setting
>> poolId - 217
>> > > uuid -
>> > > > > > > > >> a226c9a1-da78-3f3a-b5ac-****e18b925c9634 z
>> > > > > > > > >> oneId - 2 podId - 2 poolName -
>> ceph-rbd-pri-storage
>> > > > > > > > >> 2013-07-10 17:08:28,921 DEBUG
>> [db.Transaction.Transaction]
>> > > > > > > > >> (catalina-exec-2:null) Rolling back the
>> transaction: Time = 3
>> > > > > Name =
>> > > > > > > > >> persist; called by -Transaction.rollbac
>> > > > > > > > >>
>> k:890-Transaction.removeUpTo:****833-Transaction.close:657-**
>> > > > > > > > >>
>> TransactionContextBuilder.****interceptException:63-**
>> > > > > > > > >>
>> > > > >
>> ComponentInstantiationPostProc****essor$InterceptorDispatcher.**
>> **interce
>> > > > > > > > >> pt:133-StorageManagerImpl.****
>> createPool:1378-**
>> > > > > > > > >>
>> StorageManagerImpl.createPool:****147-CreateStoragePoolCmd.**
>> > > > > > > > >>
>> execute:123-ApiDispatcher.****dispatch:162-ApiServer.**
>> > > > > > > > >> queueCommand:505-ApiSe
>> > > > > > > > >>
>> rver.handleRequest:355-****ApiServlet.processRequest:302
>> > > > > > > > >> 2013-07-10 17:08:28,923 ERROR
>> [cloud.api.ApiServer]
>> > > > > > > (catalina-exec-2:null)
>> > > > > > > > >> unhandled exception executing api command:
>> createStoragePool
>> > > > > > > > >>
>> com.cloud.utils.exception.****CloudRuntimeException: DB
>> > > Exception
>> > > > > on:
>> > > > > > > > >>
>> com.mysql.jdbc.****JDBC4PreparedStatement@4b2eb56****: INSERT
>> > > INTO
>> > > > > > > > >> storage_pool (
>> > > > > > > > >> storage_pool.id <http://storage_pool.id>,
>>
>> storage_pool
>> > > > > > > > >> .name, storage_pool.uuid,
>> storage_pool.pool_type,
>> > > > > > > storage_pool.created,
>> > > > > > > > >> storage_pool.update_time,
>> storage_pool.data_center_id,
>> > > > > > > > >> storage_pool.pod_id,
>> > > > > > > > >> storage_pool.availab
>> > > > > > > > >> le_bytes, storage_pool.capacity_bytes,
>> storage_pool.status,
>> > > > > > > > >> storage_pool.scope,
>> storage_pool.storage_provider_****id,
>> > > > > > > > >> storage_pool.host_address, storage_pool.path,
>> storage_
>> > > > > > > > >> pool.port, storage_pool.user_info,
>> storage_pool.cluster_id)
>> > > VALUES
>> > > > > > > (217,
>> > > > > > > > >> _binary'ceph-rbd-pri-storage',
>> > > > > > > > >>
>> _binary'a226c9a1-da78-3f3a-****b5ac-e18b925c9634', 'RBD',
>> > > '2013-07-1
>> > > > > > > > >> 0 09:08:28', null, 2, 2, 0, 0, 'Up', null,
>> null, null,
>> > > > > _binary'ceph/
>> > > > > > > > >> ceph.client.admin.keyring@10.****237.11.2/sc1<
>> > > > > > > http://ceph.client.admin.**keyring@10.237.11.2/sc1<http://ceph.client.admin.keyring@10.237.11.2/sc1>
>> >',
>> > > > > > > > >> 6789, null, 2)
>> > > > > > > > >> at
>> > > > > > > > >>
>> > > > > > >
>> > >
>> com.cloud.utils.db.****GenericDaoBase.persist(****
>> GenericDaoBase.java:1342)
>> > > > > > > > >> at
>> > > > > > > > >>
>> com.cloud.storage.dao.****StoragePoolDaoImpl.persist(**
>> > > > > > > > >> StoragePoolDaoImpl.java:232)
>> > > > > > > > >> at
>> > > > > > > > >>
>> com.cloud.utils.component.****ComponentInstantiationPostProc****
>> > > > > > > > >>
>> > > > > > >
>> > > > >
>> > >
>> essor$InterceptorDispatcher.****intercept(****
>> ComponentInstantiationPostProc***
>> > > > > > > > >> *es
>> > > > > > > > >>
>> > > > > > > > >>
>> > > > > > >
>> > > > > > > --
>> > > > > > > Prasanna.,
>> > > > > > >
>> > > > > > > ------------------------
>> > > > > > > Powered by BigRock.com
>> > > > > > >
>> > > > > > >
>> > > > >
>> > > > > --
>> > > > > Prasanna.,
>> > > > >
>> > > > > ------------------------
>> > > > > Powered by BigRock.com
>> > > > >
>> > > > >
>> > >
>> > > --
>> > > Prasanna.,
>> > >
>> > > ------------------------
>> > > Powered by BigRock.com
>> > >
>> > >
>>
>> --
>> Prasanna.,
>>
>> ------------------------
>> Powered by BigRock.com
>>
>>
>>
>>
>
Re: Problem in adding Ceph RBD as primary storage for CloudStack
4.1.0
Posted by Wido den Hollander <wi...@widodh.nl>.
On 07/12/2013 10:14 AM, Indra Pramana wrote:
> Hi Prasanna,
>
> I managed to fix the problem, thanks for your advice to turn the agent
> log level to debug:
>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug
>
> From the log, I found out that the agent on the KVM host tried to NFS
> mount directly to 103.25.200.19:/mnt/vol1/sec-storage/template/tmpl/1/3,
> which was not allowed by the NFS server due to its default configuration
> to only allow to mount to /mnt/vol1/sec-storage (the root of the NFS share).
>
Ah, that's odd!
Btw, in 4.2 you'll be able to deploy SSVMs on RBD as well, so that
limitation will be gone.
Wido
> After I changed the NFS server configuration to allow mount to all
> sub-directories, re-export the NFS and voila, the system was able to
> download the template and now both the system VMs (CPVM and SSVM) are
> running!
>
> Many thanks for your help! :)
>
> Cheers.
>
>
>
> On Fri, Jul 12, 2013 at 3:31 PM, Indra Pramana <indra@sg.or.id
> <ma...@sg.or.id>> wrote:
>
> Hi Prasanna,
>
> Good day to you, and thank you for your e-mail.
>
> Yes, the cloudstack-agent service is running on both the KVM hosts.
> There is no "cloud" user being created though, when I installed the
> agent. I installed the agent as root.
>
> root@hv-kvm-01:/home/indra# service cloudstack-agent status
> * cloud-agent is running
>
> root@hv-kvm-01:/home/indra# su - cloud
> Unknown id: cloud
>
> Please advise how can I resolve this problem, shall I create the
> Unix "cloud" user manually? Basically I follow this instruction to
> prepare the KVM host and install the CloudStack agent:
>
> http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.1.0/html/Installation_Guide/hypervisor-kvm-install-flow.html
>
> with this instruction from Wido on how to prepare libvirt with Ceph
> RBD storage pool support:
>
> http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/
>
> I also have checked /var/log/cloud/agent/agent.log and I don't see
> any error messages, except this error message which will show up
> every time I restart the agent:
>
> 2013-07-12 15:22:47,454 ERROR [cloud.resource.ServerResourceBase]
> (main:null) Nics are not configured!
> 2013-07-12 15:22:47,459 INFO [cloud.resource.ServerResourceBase]
> (main:null) Designating private to be nic eth0.5
>
> More logs can be found here: http://pastebin.com/yeNmCt7S
>
> I have configured the network bridges on the NIC interface as per
> this instruction:
>
> http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.1.0/html/Installation_Guide/hypervisor-kvm-install-flow.html#hypervisor-host-install-network
>
> On the zone, I used advanced network configuration with just one
> physical network for management, public and guest/private. I didn't
> include storage, which by default the traffic will use the
> management VLAN network.
>
> Please advise if there's anything else I might have been missing.
>
> Looking forward to your reply, thank you.
>
> Cheers.
>
>
>
> On Fri, Jul 12, 2013 at 2:56 PM, Prasanna Santhanam <tsp@apache.org
> <ma...@apache.org>> wrote:
>
> Indeed, cloudstack will go through the allocation to startup the
> system VMs too. So that process is failing to recognize the volume
> (.qcow2) present on your NFS storage.
>
> Can you check if your cloudstack agent service is running with
> the KVM
> host? And it should've created the user cloud. $id cloud to check if
> there is a user.
>
> Did you see what's happening in the agent logs? These are under
> /var/log/cloud/ on your host when the systemVMs are coming up.
> If the
> logs are not showing any useful information you can turn on debug
> level for more verbosity.
>
> See here: https://cwiki.apache.org/confluence/x/FgPMAQ
>
> On Fri, Jul 12, 2013 at 02:40:35PM +0800, Indra Pramana wrote:
> > Hi Prasanna,
> >
> > Good day to you, and thank you for your e-mail.
> >
> > Yes, when I export the NFS, I set the permission so that
> normal user will
> > be able to have read/write access to the files (no_root_squash).
> >
> > I have tested and I can have read/write access from my KVM
> hosts using
> > normal user. BTW, there's no "cloud" user on the hosts, I
> believe it's not
> > created during cloudstack-agent installation?
> >
> > In any case, do you think the template issue and the storage pool
> > allocation issue might be related, or are they two different
> problems
> > altogether?
> >
> > Looking forward to your reply, thank you.
> >
> > Cheers.
> >
> >
> >
> > On Fri, Jul 12, 2013 at 2:26 PM, Prasanna Santhanam
> <tsp@apache.org <ma...@apache.org>> wrote:
> >
> > > Can you access the file as user root? Or user cloud? The
> cloudstack
> > > agent on your KVM host runs as user cloud and the NFS
> permissions
> > > might be disallowing the volume (.qcow2) from being accessed.
> > >
> > > On Fri, Jul 12, 2013 at 02:16:41PM +0800, Indra Pramana wrote:
> > > > Hi Prasanna,
> > > >
> > > > Good day to you, and thank you for your e-mail.
> > > >
> > > > Yes, the file exists. I can access the file from the
> management server
> > > and
> > > > the two hypervisors hosts if I mount manually.
> > > >
> > > > [root@cs-nas-01 /mnt/vol1/sec-storage/template/tmpl/1/3]# ls
> > > > -la
> > > > total
> > > > 1418787
> > > > drwxr-xr-x 2 root wheel 4 Jul 11 20:21
> > > > .
> > > > drwxr-xr-x 3 root wheel 3 Jul 11 20:17
> > > > ..
> > > > -rw-r--r-- 1 root wheel 725811200 Jul 11 20:21
> > > > 425b9e5a-fbc7-4637-a33a-fe9d0ed4fa98.qcow2
> > > > -rw-r--r-- 1 root wheel 295 Jul 11 20:21
> > > > template.properties
> > > > [root@cs-nas-01 /mnt/vol1/sec-storage/template/tmpl/1/3]#
> > > > pwd
> > > > /mnt/vol1/sec-storage/template/tmpl/1/3
> > > >
> > > >
> > > > Any advise?
> > > >
> > > > Looking forward to your reply, thank you.
> > > >
> > > > Cheers.
> > > >
> > > >
> > > >
> > > > On Fri, Jul 12, 2013 at 2:07 PM, Prasanna Santhanam
> <tsp@apache.org <ma...@apache.org>>
> > > wrote:
> > > >
> > > > > Can you check whether there is a file at:
> > > > >
> nfs://103.25.200.19/mnt/vol1/sec-storage/template/tmpl/1/3/
> <http://103.25.200.19/mnt/vol1/sec-storage/template/tmpl/1/3/>
> > > > >
> > > > > On Fri, Jul 12, 2013 at 01:59:34PM +0800, Indra Pramana
> wrote:
> > > > > > Hi Prasanna,
> > > > > >
> > > > > > Thanks for your e-mail.
> > > > > >
> > > > > > I have tried restarting the management server, and
> the problem still
> > > > > > persists. I even tried to re-do the installation and
> configuration
> > > again
> > > > > > from scratch last night, but the problem still there.
> > > > > >
> > > > > > I also noted that on the beginning of the logs, I
> found some error
> > > > > messages
> > > > > > saying that the template cannot be downloaded to the
> pool. See this
> > > logs:
> > > > > >
> > > > > > http://pastebin.com/BY1AVJ08
> > > > > >
> > > > > > It says it failed because cannot get volume from the
> pool. Could it
> > > be
> > > > > > related, i.e. the absence of the template caused the
> system VMs
> > > cannot be
> > > > > > created and started?
> > > > > >
> > > > > > I have ensured that I downloaded the system VM
> template using
> > > > > > cloud-install-sys-tmplt and verified that the
> template is already
> > > there
> > > > > in
> > > > > > the secondary storage server.
> > > > > >
> > > > > > Any advice is appreciated.
> > > > > >
> > > > > > Looking forward to your reply, thank you.
> > > > > >
> > > > > > Cheers.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Jul 12, 2013 at 1:21 PM, Prasanna Santhanam
> <tsp@apache.org <ma...@apache.org>>
> > > > > wrote:
> > > > > >
> > > > > > > It looks like a previous attempt to start the
> systemVMs has failed
> > > > > > > putting the nfs storage in the avoid set. Did you
> try restarting
> > > your
> > > > > > > management server?
> > > > > > >
> > > > > > > This line leads me to the above mentioned:
> > > > > > > 2013-07-12 13:10:48,236 DEBUG
> > > > > > > [storage.allocator.AbstractStoragePoolAllocator]
> > > (secstorage-1:null)
> > > > > > > StoragePool is in avoid set, skipping this pool
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Jul 12, 2013 at 01:16:53PM +0800, Indra
> Pramana wrote:
> > > > > > > > Dear Wido and all,
> > > > > > > >
> > > > > > > > I have managed to get the hosts, primary and
> secondary storage
> > > > > running.
> > > > > > > >
> > > > > > > > - 2 KVM hypervisor hosts
> > > > > > > > - One RBD primary storage
> > > > > > > > - One NFS primary storage (for system VMs, since
> I understand
> > > that
> > > > > system
> > > > > > > > VMs cannot use RBD)
> > > > > > > > - One NFS secondary storage
> > > > > > > >
> > > > > > > > However, now I am having problem with the system
> VMs: CPVM and
> > > SSVM,
> > > > > > > unable
> > > > > > > > to start.
> > > > > > > >
> > > > > > > > Excerpt from management-server.log file is here:
> > > > > > > > http://pastebin.com/ENkpCALY
> > > > > > > >
> > > > > > > > It seems that the VMs were not able to be created
> because unable
> > > to
> > > > > find
> > > > > > > > suitable StoragePools.
> > > > > > > >
> > > > > > > > I understand that system VMs will be using the
> NFS primary
> > > storage
> > > > > > > instead
> > > > > > > > of RBD, so I have confirmed that I am able to
> mount the primary
> > > > > storage
> > > > > > > via
> > > > > > > > NFS and have read and write access, from both the
> hypervisor and
> > > the
> > > > > > > > management server.
> > > > > > > >
> > > > > > > > Any advise how can I resolve the problem to make
> both the system
> > > VMs
> > > > > > > > created and started?
> > > > > > > >
> > > > > > > > Looking forward to your reply, thank you.
> > > > > > > >
> > > > > > > > Cheers.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Jul 12, 2013 at 9:43 AM, Indra Pramana
> <indra@sg.or.id <ma...@sg.or.id>>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Wido,
> > > > > > > > >
> > > > > > > > > Thanks for the advice, I'm now able to add the
> RBD pool as
> > > primary
> > > > > > > storage.
> > > > > > > > >
> > > > > > > > > Many thanks! :)
> > > > > > > > >
> > > > > > > > > Cheers.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thursday, July 11, 2013, Wido den Hollander
> wrote:
> > > > > > > > >
> > > > > > > > >> Hi,
> > > > > > > > >>
> > > > > > > > >> On 07/10/2013 03:42 PM, Chip Childers wrote:
> > > > > > > > >>
> > > > > > > > >>> Cc'ing Wido, our resident Ceph expert. ;-)
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >> Hehe ;)
> > > > > > > > >>
> > > > > > > > >> On Wed, Jul 10, 2013 at 05:45:25PM +0800,
> Indra Pramana
> > > wrote:
> > > > > > > > >>>
> > > > > > > > >>>> Dear all,
> > > > > > > > >>>>
> > > > > > > > >>>> I am installing CloudStack 4.1.0 (upgraded
> from 4.0.2) and I
> > > > > also
> > > > > > > have a
> > > > > > > > >>>> Ceph cluster running. However, I am having
> issues in adding
> > > the
> > > > > RBD
> > > > > > > as
> > > > > > > > >>>> primary storage. Tried to follow the
> instruction here, but
> > > > > unable to
> > > > > > > > >>>> make
> > > > > > > > >>>> it work:
> > > > > > > > >>>>
> > > > > > > > >>>>
> http://ceph.com/docs/master/**rbd/rbd-cloudstack/<
> > > > > > > http://ceph.com/docs/master/rbd/rbd-cloudstack/>
> > > > > > > > >>>>
> > > > > > > > >>>> I have setup a pool on the Ceph cluster. The
> status of the
> > > > > cluster
> > > > > > > is
> > > > > > > > >>>> healthy. Since I am using Ubuntu 12.04.2 LTS
> (Precise) for
> > > the
> > > > > > > > >>>> hypervisors,
> > > > > > > > >>>> I also have compiled libvirt manually to
> ensure that the
> > > version
> > > > > > > 0.9.13
> > > > > > > > >>>> is
> > > > > > > > >>>> installed (previously it's 0.9.8).
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >> You can also use the Ubuntu Cloud Archive, I
> still need to
> > > get the
> > > > > > > docs
> > > > > > > > >> updated for that.
> > > > > > > > >>
> > > > > > > > >> I described the process in a blogpost:
> > > > > > > http://blog.widodh.nl/2013/06/**
> > > > > > > > >>
> > >
> a-quick-note-on-running-**cloudstack-with-rbd-on-ubuntu-**12-04/<
> > > > > > >
> > > > >
> > >
> http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/
> > > > > > > >
> > > > > > > > >>
> > > > > > > > >> indra@hv-kvm-01:~/rbd$ ceph
> > > > > > > > >>>> ceph> health
> > > > > > > > >>>> HEALTH_OK
> > > > > > > > >>>>
> > > > > > > > >>>> indra@hv-kvm-01:~$ ceph osd lspools
> > > > > > > > >>>> 0 data,1 metadata,2 rbd,3 sc1,
> > > > > > > > >>>>
> > > > > > > > >>>> root@hv-kvm-01:/home/indra# libvirtd --version
> > > > > > > > >>>> libvirtd (libvirt) 0.9.13
> > > > > > > > >>>>
> > > > > > > > >>>> I tried to add Primary Storage into the
> Cloudstack zone
> > > which I
> > > > > have
> > > > > > > > >>>> created:
> > > > > > > > >>>>
> > > > > > > > >>>> Add Primary Storage:
> > > > > > > > >>>>
> > > > > > > > >>>> Zone: my zone name
> > > > > > > > >>>> Pod: my pod name
> > > > > > > > >>>> Cluster: my cluster name
> > > > > > > > >>>> Name: ceph-rbd-pri-storage
> > > > > > > > >>>> Protocol: RBD
> > > > > > > > >>>> RADOS Monitor: my first Ceph monitor IP address
> > > > > > > > >>>> RADOS Pool: sc1 (the pool name on Ceph cluster)
> > > > > > > > >>>> RADOS User: client.admin
> > > > > > > > >>>> RADOS Secret:
> /etc/ceph/ceph.client.admin.**keyring (keyring
> > > > > file
> > > > > > > > >>>> location)
> > > > > > > > >>>>
> > > > > > > > >>>
> > > > > > > > >> This is your problem. That shouldn't be the
> location of the
> > > file,
> > > > > but
> > > > > > > it
> > > > > > > > >> should be the secret, which is a base64
> encoded string.
> > > > > > > > >>
> > > > > > > > >> $ ceph auth list
> > > > > > > > >>
> > > > > > > > >> That should tell you what the secret is.
> > > > > > > > >>
> > > > > > > > >> Storage Tags: rbd
> > > > > > > > >>
> > > > > > > > >> This is the error message when I tried to add
> the primary
> > > storage
> > > > > by
> > > > > > > > >> clicking OK:
> > > > > > > > >>
> > > > > > > > >> DB Exception on:
> > > com.mysql.jdbc.**JDBC4PreparedStatement@4b2eb56
> > > > > **:
> > > > > > > > >> INSERT INTO
> > > > > > > > >> storage_pool (storage_pool.id
> <http://storage_pool.id>, storage_pool.name
> <http://storage_pool.name>,
> > > > > storage_pool.uuid,
> > > > > > > > >> storage_pool.pool_type, storage_pool.created,
> > > > > > > storage_pool.update_time,
> > > > > > > > >> storage_pool.data_center_id, storage_pool.pod_id,
> > > > > > > > >> storage_pool.available_bytes,
> storage_pool.capacity_bytes,
> > > > > > > > >> storage_pool.status, storage_pool.scope,
> > > > > > > storage_pool.storage_provider_**
> > > > > > > > >> id,
> > > > > > > > >> storage_pool.host_address, storage_pool.path,
> > > storage_pool.port,
> > > > > > > > >> storage_pool.user_info,
> storage_pool.cluster_id) VALUES (217,
> > > > > > > > >> _binary'ceph-rbd-pri-storage',
> > > > > > > > >>
> _binary'a226c9a1-da78-3f3a-**b5ac-e18b925c9634', 'RBD',
> > > > > '2013-07-10
> > > > > > > > >> 09:08:28', null, 2, 2, 0, 0, 'Up', null, null,
> null,
> > > _binary'ceph/
> > > > > > > > >> ceph.client.admin.keyring@10.**237.11.2/sc1<
> > > > > > > http://ceph.client.admin.keyring@10.237.11.2/sc1>',
> > > > > > > > >> 6789, null, 2)
> > > > > > > > >>
> > > > > > > > >> On the management-server.log file:
> > > > > > > > >>
> > > > > > > > >> 2013-07-10 17:08:28,845 DEBUG
> [cloud.api.ApiServlet]
> > > > > > > > >> (catalina-exec-2:null)
> > > > > > > > >> ===START=== 192.168.0.100 -- GET
> > > > > > > > >>
> command=createStoragePool&**zoneid=c116950e-e4ae-4f23-**a7e7-
> > > > > > > > >>
> 74a75c4ee638&podId=a748b063-**3a83-4175-a0e9-de39118fe5ce&**
> > > > > > > > >>
> clusterid=1f87eb09-324d-4d49-**83c2-88d84d7a15df&name=ceph-**
> > > > > > > > >>
> rbd-pri-storage&url=rbd%3A%2F%**2Fclient.admin%3A_etc%2Fc
> > > > > > > > >>
> eph%2Fceph.client.admin.**keyring%4010.237.11.2%2Fsc1&**
> > > > > > > > >>
> > > > >
> tags=rbd&response=json&**sessionkey=**rDRfWpqeKfQKbKZtHr398ULV%2F8k%**
> > > > > > > > >> 3D&_=1373447307839
> > > > > > > > >> 2013-07-10 17:08:28,862 DEBUG
> > > [cloud.storage.**StorageManagerImpl]
> > > > > > > > >> (catalina-exec-2:null) createPool Params @
> scheme - rbd
> > > > > storageHost -
> > > > > > > null
> > > > > > > > >> hostPath - /ceph/ceph.client
> > > > > > > > >> .admin.keyring@10.237.11.2/sc1
> <http://admin.keyring@10.237.11.2/sc1> port - -1
> > > > > > > > >> 2013-07-10 17:08:28,918 DEBUG
> > > [cloud.storage.**StorageManagerImpl]
> > > > > > > > >> (catalina-exec-2:null) In createPool Setting
> poolId - 217
> > > uuid -
> > > > > > > > >> a226c9a1-da78-3f3a-b5ac-**e18b925c9634 z
> > > > > > > > >> oneId - 2 podId - 2 poolName -
> ceph-rbd-pri-storage
> > > > > > > > >> 2013-07-10 17:08:28,921 DEBUG
> [db.Transaction.Transaction]
> > > > > > > > >> (catalina-exec-2:null) Rolling back the
> transaction: Time = 3
> > > > > Name =
> > > > > > > > >> persist; called by -Transaction.rollbac
> > > > > > > > >>
> k:890-Transaction.removeUpTo:**833-Transaction.close:657-**
> > > > > > > > >>
> TransactionContextBuilder.**interceptException:63-**
> > > > > > > > >>
> > > > >
> ComponentInstantiationPostProc**essor$InterceptorDispatcher.**interce
> > > > > > > > >> pt:133-StorageManagerImpl.**createPool:1378-**
> > > > > > > > >>
> StorageManagerImpl.createPool:**147-CreateStoragePoolCmd.**
> > > > > > > > >>
> execute:123-ApiDispatcher.**dispatch:162-ApiServer.**
> > > > > > > > >> queueCommand:505-ApiSe
> > > > > > > > >>
> rver.handleRequest:355-**ApiServlet.processRequest:302
> > > > > > > > >> 2013-07-10 17:08:28,923 ERROR
> [cloud.api.ApiServer]
> > > > > > > (catalina-exec-2:null)
> > > > > > > > >> unhandled exception executing api command:
> createStoragePool
> > > > > > > > >>
> com.cloud.utils.exception.**CloudRuntimeException: DB
> > > Exception
> > > > > on:
> > > > > > > > >>
> com.mysql.jdbc.**JDBC4PreparedStatement@4b2eb56**: INSERT
> > > INTO
> > > > > > > > >> storage_pool (
> > > > > > > > >> storage_pool.id <http://storage_pool.id>,
> storage_pool
> > > > > > > > >> .name, storage_pool.uuid, storage_pool.pool_type,
> > > > > > > storage_pool.created,
> > > > > > > > >> storage_pool.update_time,
> storage_pool.data_center_id,
> > > > > > > > >> storage_pool.pod_id,
> > > > > > > > >> storage_pool.availab
> > > > > > > > >> le_bytes, storage_pool.capacity_bytes,
> storage_pool.status,
> > > > > > > > >> storage_pool.scope,
> storage_pool.storage_provider_**id,
> > > > > > > > >> storage_pool.host_address, storage_pool.path,
> storage_
> > > > > > > > >> pool.port, storage_pool.user_info,
> storage_pool.cluster_id)
> > > VALUES
> > > > > > > (217,
> > > > > > > > >> _binary'ceph-rbd-pri-storage',
> > > > > > > > >>
> _binary'a226c9a1-da78-3f3a-**b5ac-e18b925c9634', 'RBD',
> > > '2013-07-1
> > > > > > > > >> 0 09:08:28', null, 2, 2, 0, 0, 'Up', null,
> null, null,
> > > > > _binary'ceph/
> > > > > > > > >> ceph.client.admin.keyring@10.**237.11.2/sc1<
> > > > > > > http://ceph.client.admin.keyring@10.237.11.2/sc1>',
> > > > > > > > >> 6789, null, 2)
> > > > > > > > >> at
> > > > > > > > >>
> > > > > > >
> > >
> com.cloud.utils.db.**GenericDaoBase.persist(**GenericDaoBase.java:1342)
> > > > > > > > >> at
> > > > > > > > >>
> com.cloud.storage.dao.**StoragePoolDaoImpl.persist(**
> > > > > > > > >> StoragePoolDaoImpl.java:232)
> > > > > > > > >> at
> > > > > > > > >>
> com.cloud.utils.component.**ComponentInstantiationPostProc**
> > > > > > > > >>
> > > > > > >
> > > > >
> > >
> essor$InterceptorDispatcher.**intercept(**ComponentInstantiationPostProc*
> > > > > > > > >> *es
> > > > > > > > >>
> > > > > > > > >>
> > > > > > >
> > > > > > > --
> > > > > > > Prasanna.,
> > > > > > >
> > > > > > > ------------------------
> > > > > > > Powered by BigRock.com
> > > > > > >
> > > > > > >
> > > > >
> > > > > --
> > > > > Prasanna.,
> > > > >
> > > > > ------------------------
> > > > > Powered by BigRock.com
> > > > >
> > > > >
> > >
> > > --
> > > Prasanna.,
> > >
> > > ------------------------
> > > Powered by BigRock.com
> > >
> > >
>
> --
> Prasanna.,
>
> ------------------------
> Powered by BigRock.com
>
>
>
Re: Problem in adding Ceph RBD as primary storage for CloudStack 4.1.0
Posted by Indra Pramana <in...@sg.or.id>.
Hi Prasanna,
I managed to fix the problem, thanks for your advice to turn the agent log
level to debug:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug
>From the log, I found out that the agent on the KVM host tried to NFS mount
directly to 103.25.200.19:/mnt/vol1/sec-storage/template/tmpl/1/3, which
was not allowed by the NFS server due to its default configuration to only
allow to mount to /mnt/vol1/sec-storage (the root of the NFS share).
After I changed the NFS server configuration to allow mount to all
sub-directories, re-export the NFS and voila, the system was able to
download the template and now both the system VMs (CPVM and SSVM) are
running!
Many thanks for your help! :)
Cheers.
On Fri, Jul 12, 2013 at 3:31 PM, Indra Pramana <in...@sg.or.id> wrote:
> Hi Prasanna,
>
> Good day to you, and thank you for your e-mail.
>
> Yes, the cloudstack-agent service is running on both the KVM hosts. There
> is no "cloud" user being created though, when I installed the agent. I
> installed the agent as root.
>
> root@hv-kvm-01:/home/indra# service cloudstack-agent status
> * cloud-agent is running
>
> root@hv-kvm-01:/home/indra# su - cloud
> Unknown id: cloud
>
> Please advise how can I resolve this problem, shall I create the Unix
> "cloud" user manually? Basically I follow this instruction to prepare the
> KVM host and install the CloudStack agent:
>
>
> http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.1.0/html/Installation_Guide/hypervisor-kvm-install-flow.html
>
> with this instruction from Wido on how to prepare libvirt with Ceph RBD
> storage pool support:
>
>
> http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/
>
> I also have checked /var/log/cloud/agent/agent.log and I don't see any
> error messages, except this error message which will show up every time I
> restart the agent:
>
> 2013-07-12 15:22:47,454 ERROR [cloud.resource.ServerResourceBase]
> (main:null) Nics are not configured!
> 2013-07-12 15:22:47,459 INFO [cloud.resource.ServerResourceBase]
> (main:null) Designating private to be nic eth0.5
>
> More logs can be found here: http://pastebin.com/yeNmCt7S
>
> I have configured the network bridges on the NIC interface as per this
> instruction:
>
>
> http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.1.0/html/Installation_Guide/hypervisor-kvm-install-flow.html#hypervisor-host-install-network
>
> On the zone, I used advanced network configuration with just one physical
> network for management, public and guest/private. I didn't include storage,
> which by default the traffic will use the management VLAN network.
>
> Please advise if there's anything else I might have been missing.
>
> Looking forward to your reply, thank you.
>
> Cheers.
>
>
>
> On Fri, Jul 12, 2013 at 2:56 PM, Prasanna Santhanam <ts...@apache.org>wrote:
>
>> Indeed, cloudstack will go through the allocation to startup the
>> system VMs too. So that process is failing to recognize the volume
>> (.qcow2) present on your NFS storage.
>>
>> Can you check if your cloudstack agent service is running with the KVM
>> host? And it should've created the user cloud. $id cloud to check if
>> there is a user.
>>
>> Did you see what's happening in the agent logs? These are under
>> /var/log/cloud/ on your host when the systemVMs are coming up. If the
>> logs are not showing any useful information you can turn on debug
>> level for more verbosity.
>>
>> See here: https://cwiki.apache.org/confluence/x/FgPMAQ
>>
>> On Fri, Jul 12, 2013 at 02:40:35PM +0800, Indra Pramana wrote:
>> > Hi Prasanna,
>> >
>> > Good day to you, and thank you for your e-mail.
>> >
>> > Yes, when I export the NFS, I set the permission so that normal user
>> will
>> > be able to have read/write access to the files (no_root_squash).
>> >
>> > I have tested and I can have read/write access from my KVM hosts using
>> > normal user. BTW, there's no "cloud" user on the hosts, I believe it's
>> not
>> > created during cloudstack-agent installation?
>> >
>> > In any case, do you think the template issue and the storage pool
>> > allocation issue might be related, or are they two different problems
>> > altogether?
>> >
>> > Looking forward to your reply, thank you.
>> >
>> > Cheers.
>> >
>> >
>> >
>> > On Fri, Jul 12, 2013 at 2:26 PM, Prasanna Santhanam <ts...@apache.org>
>> wrote:
>> >
>> > > Can you access the file as user root? Or user cloud? The cloudstack
>> > > agent on your KVM host runs as user cloud and the NFS permissions
>> > > might be disallowing the volume (.qcow2) from being accessed.
>> > >
>> > > On Fri, Jul 12, 2013 at 02:16:41PM +0800, Indra Pramana wrote:
>> > > > Hi Prasanna,
>> > > >
>> > > > Good day to you, and thank you for your e-mail.
>> > > >
>> > > > Yes, the file exists. I can access the file from the management
>> server
>> > > and
>> > > > the two hypervisors hosts if I mount manually.
>> > > >
>> > > > [root@cs-nas-01 /mnt/vol1/sec-storage/template/tmpl/1/3]# ls
>> > > > -la
>> > > > total
>> > > > 1418787
>> > > > drwxr-xr-x 2 root wheel 4 Jul 11 20:21
>> > > > .
>> > > > drwxr-xr-x 3 root wheel 3 Jul 11 20:17
>> > > > ..
>> > > > -rw-r--r-- 1 root wheel 725811200 Jul 11 20:21
>> > > > 425b9e5a-fbc7-4637-a33a-fe9d0ed4fa98.qcow2
>> > > > -rw-r--r-- 1 root wheel 295 Jul 11 20:21
>> > > > template.properties
>> > > > [root@cs-nas-01 /mnt/vol1/sec-storage/template/tmpl/1/3]#
>> > > > pwd
>> > > > /mnt/vol1/sec-storage/template/tmpl/1/3
>> > > >
>> > > >
>> > > > Any advise?
>> > > >
>> > > > Looking forward to your reply, thank you.
>> > > >
>> > > > Cheers.
>> > > >
>> > > >
>> > > >
>> > > > On Fri, Jul 12, 2013 at 2:07 PM, Prasanna Santhanam <tsp@apache.org
>> >
>> > > wrote:
>> > > >
>> > > > > Can you check whether there is a file at:
>> > > > > nfs://103.25.200.19/mnt/vol1/sec-storage/template/tmpl/1/3/
>> > > > >
>> > > > > On Fri, Jul 12, 2013 at 01:59:34PM +0800, Indra Pramana wrote:
>> > > > > > Hi Prasanna,
>> > > > > >
>> > > > > > Thanks for your e-mail.
>> > > > > >
>> > > > > > I have tried restarting the management server, and the problem
>> still
>> > > > > > persists. I even tried to re-do the installation and
>> configuration
>> > > again
>> > > > > > from scratch last night, but the problem still there.
>> > > > > >
>> > > > > > I also noted that on the beginning of the logs, I found some
>> error
>> > > > > messages
>> > > > > > saying that the template cannot be downloaded to the pool. See
>> this
>> > > logs:
>> > > > > >
>> > > > > > http://pastebin.com/BY1AVJ08
>> > > > > >
>> > > > > > It says it failed because cannot get volume from the pool.
>> Could it
>> > > be
>> > > > > > related, i.e. the absence of the template caused the system VMs
>> > > cannot be
>> > > > > > created and started?
>> > > > > >
>> > > > > > I have ensured that I downloaded the system VM template using
>> > > > > > cloud-install-sys-tmplt and verified that the template is
>> already
>> > > there
>> > > > > in
>> > > > > > the secondary storage server.
>> > > > > >
>> > > > > > Any advice is appreciated.
>> > > > > >
>> > > > > > Looking forward to your reply, thank you.
>> > > > > >
>> > > > > > Cheers.
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On Fri, Jul 12, 2013 at 1:21 PM, Prasanna Santhanam <
>> tsp@apache.org>
>> > > > > wrote:
>> > > > > >
>> > > > > > > It looks like a previous attempt to start the systemVMs has
>> failed
>> > > > > > > putting the nfs storage in the avoid set. Did you try
>> restarting
>> > > your
>> > > > > > > management server?
>> > > > > > >
>> > > > > > > This line leads me to the above mentioned:
>> > > > > > > 2013-07-12 13:10:48,236 DEBUG
>> > > > > > > [storage.allocator.AbstractStoragePoolAllocator]
>> > > (secstorage-1:null)
>> > > > > > > StoragePool is in avoid set, skipping this pool
>> > > > > > >
>> > > > > > >
>> > > > > > > On Fri, Jul 12, 2013 at 01:16:53PM +0800, Indra Pramana wrote:
>> > > > > > > > Dear Wido and all,
>> > > > > > > >
>> > > > > > > > I have managed to get the hosts, primary and secondary
>> storage
>> > > > > running.
>> > > > > > > >
>> > > > > > > > - 2 KVM hypervisor hosts
>> > > > > > > > - One RBD primary storage
>> > > > > > > > - One NFS primary storage (for system VMs, since I
>> understand
>> > > that
>> > > > > system
>> > > > > > > > VMs cannot use RBD)
>> > > > > > > > - One NFS secondary storage
>> > > > > > > >
>> > > > > > > > However, now I am having problem with the system VMs: CPVM
>> and
>> > > SSVM,
>> > > > > > > unable
>> > > > > > > > to start.
>> > > > > > > >
>> > > > > > > > Excerpt from management-server.log file is here:
>> > > > > > > > http://pastebin.com/ENkpCALY
>> > > > > > > >
>> > > > > > > > It seems that the VMs were not able to be created because
>> unable
>> > > to
>> > > > > find
>> > > > > > > > suitable StoragePools.
>> > > > > > > >
>> > > > > > > > I understand that system VMs will be using the NFS primary
>> > > storage
>> > > > > > > instead
>> > > > > > > > of RBD, so I have confirmed that I am able to mount the
>> primary
>> > > > > storage
>> > > > > > > via
>> > > > > > > > NFS and have read and write access, from both the
>> hypervisor and
>> > > the
>> > > > > > > > management server.
>> > > > > > > >
>> > > > > > > > Any advise how can I resolve the problem to make both the
>> system
>> > > VMs
>> > > > > > > > created and started?
>> > > > > > > >
>> > > > > > > > Looking forward to your reply, thank you.
>> > > > > > > >
>> > > > > > > > Cheers.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Fri, Jul 12, 2013 at 9:43 AM, Indra Pramana <
>> indra@sg.or.id>
>> > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Hi Wido,
>> > > > > > > > >
>> > > > > > > > > Thanks for the advice, I'm now able to add the RBD pool as
>> > > primary
>> > > > > > > storage.
>> > > > > > > > >
>> > > > > > > > > Many thanks! :)
>> > > > > > > > >
>> > > > > > > > > Cheers.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Thursday, July 11, 2013, Wido den Hollander wrote:
>> > > > > > > > >
>> > > > > > > > >> Hi,
>> > > > > > > > >>
>> > > > > > > > >> On 07/10/2013 03:42 PM, Chip Childers wrote:
>> > > > > > > > >>
>> > > > > > > > >>> Cc'ing Wido, our resident Ceph expert. ;-)
>> > > > > > > > >>>
>> > > > > > > > >>>
>> > > > > > > > >> Hehe ;)
>> > > > > > > > >>
>> > > > > > > > >> On Wed, Jul 10, 2013 at 05:45:25PM +0800, Indra Pramana
>> > > wrote:
>> > > > > > > > >>>
>> > > > > > > > >>>> Dear all,
>> > > > > > > > >>>>
>> > > > > > > > >>>> I am installing CloudStack 4.1.0 (upgraded from 4.0.2)
>> and I
>> > > > > also
>> > > > > > > have a
>> > > > > > > > >>>> Ceph cluster running. However, I am having issues in
>> adding
>> > > the
>> > > > > RBD
>> > > > > > > as
>> > > > > > > > >>>> primary storage. Tried to follow the instruction here,
>> but
>> > > > > unable to
>> > > > > > > > >>>> make
>> > > > > > > > >>>> it work:
>> > > > > > > > >>>>
>> > > > > > > > >>>> http://ceph.com/docs/master/**rbd/rbd-cloudstack/<
>> > > > > > > http://ceph.com/docs/master/rbd/rbd-cloudstack/>
>> > > > > > > > >>>>
>> > > > > > > > >>>> I have setup a pool on the Ceph cluster. The status of
>> the
>> > > > > cluster
>> > > > > > > is
>> > > > > > > > >>>> healthy. Since I am using Ubuntu 12.04.2 LTS (Precise)
>> for
>> > > the
>> > > > > > > > >>>> hypervisors,
>> > > > > > > > >>>> I also have compiled libvirt manually to ensure that
>> the
>> > > version
>> > > > > > > 0.9.13
>> > > > > > > > >>>> is
>> > > > > > > > >>>> installed (previously it's 0.9.8).
>> > > > > > > > >>>>
>> > > > > > > > >>>>
>> > > > > > > > >> You can also use the Ubuntu Cloud Archive, I still need
>> to
>> > > get the
>> > > > > > > docs
>> > > > > > > > >> updated for that.
>> > > > > > > > >>
>> > > > > > > > >> I described the process in a blogpost:
>> > > > > > > http://blog.widodh.nl/2013/06/**
>> > > > > > > > >>
>> > > a-quick-note-on-running-**cloudstack-with-rbd-on-ubuntu-**12-04/<
>> > > > > > >
>> > > > >
>> > >
>> http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/
>> > > > > > > >
>> > > > > > > > >>
>> > > > > > > > >> indra@hv-kvm-01:~/rbd$ ceph
>> > > > > > > > >>>> ceph> health
>> > > > > > > > >>>> HEALTH_OK
>> > > > > > > > >>>>
>> > > > > > > > >>>> indra@hv-kvm-01:~$ ceph osd lspools
>> > > > > > > > >>>> 0 data,1 metadata,2 rbd,3 sc1,
>> > > > > > > > >>>>
>> > > > > > > > >>>> root@hv-kvm-01:/home/indra# libvirtd --version
>> > > > > > > > >>>> libvirtd (libvirt) 0.9.13
>> > > > > > > > >>>>
>> > > > > > > > >>>> I tried to add Primary Storage into the Cloudstack zone
>> > > which I
>> > > > > have
>> > > > > > > > >>>> created:
>> > > > > > > > >>>>
>> > > > > > > > >>>> Add Primary Storage:
>> > > > > > > > >>>>
>> > > > > > > > >>>> Zone: my zone name
>> > > > > > > > >>>> Pod: my pod name
>> > > > > > > > >>>> Cluster: my cluster name
>> > > > > > > > >>>> Name: ceph-rbd-pri-storage
>> > > > > > > > >>>> Protocol: RBD
>> > > > > > > > >>>> RADOS Monitor: my first Ceph monitor IP address
>> > > > > > > > >>>> RADOS Pool: sc1 (the pool name on Ceph cluster)
>> > > > > > > > >>>> RADOS User: client.admin
>> > > > > > > > >>>> RADOS Secret: /etc/ceph/ceph.client.admin.**keyring
>> (keyring
>> > > > > file
>> > > > > > > > >>>> location)
>> > > > > > > > >>>>
>> > > > > > > > >>>
>> > > > > > > > >> This is your problem. That shouldn't be the location of
>> the
>> > > file,
>> > > > > but
>> > > > > > > it
>> > > > > > > > >> should be the secret, which is a base64 encoded string.
>> > > > > > > > >>
>> > > > > > > > >> $ ceph auth list
>> > > > > > > > >>
>> > > > > > > > >> That should tell you what the secret is.
>> > > > > > > > >>
>> > > > > > > > >> Storage Tags: rbd
>> > > > > > > > >>
>> > > > > > > > >> This is the error message when I tried to add the primary
>> > > storage
>> > > > > by
>> > > > > > > > >> clicking OK:
>> > > > > > > > >>
>> > > > > > > > >> DB Exception on:
>> > > com.mysql.jdbc.**JDBC4PreparedStatement@4b2eb56
>> > > > > **:
>> > > > > > > > >> INSERT INTO
>> > > > > > > > >> storage_pool (storage_pool.id, storage_pool.name,
>> > > > > storage_pool.uuid,
>> > > > > > > > >> storage_pool.pool_type, storage_pool.created,
>> > > > > > > storage_pool.update_time,
>> > > > > > > > >> storage_pool.data_center_id, storage_pool.pod_id,
>> > > > > > > > >> storage_pool.available_bytes,
>> storage_pool.capacity_bytes,
>> > > > > > > > >> storage_pool.status, storage_pool.scope,
>> > > > > > > storage_pool.storage_provider_**
>> > > > > > > > >> id,
>> > > > > > > > >> storage_pool.host_address, storage_pool.path,
>> > > storage_pool.port,
>> > > > > > > > >> storage_pool.user_info, storage_pool.cluster_id) VALUES
>> (217,
>> > > > > > > > >> _binary'ceph-rbd-pri-storage',
>> > > > > > > > >> _binary'a226c9a1-da78-3f3a-**b5ac-e18b925c9634', 'RBD',
>> > > > > '2013-07-10
>> > > > > > > > >> 09:08:28', null, 2, 2, 0, 0, 'Up', null, null, null,
>> > > _binary'ceph/
>> > > > > > > > >> ceph.client.admin.keyring@10.**237.11.2/sc1<
>> > > > > > > http://ceph.client.admin.keyring@10.237.11.2/sc1>',
>> > > > > > > > >> 6789, null, 2)
>> > > > > > > > >>
>> > > > > > > > >> On the management-server.log file:
>> > > > > > > > >>
>> > > > > > > > >> 2013-07-10 17:08:28,845 DEBUG [cloud.api.ApiServlet]
>> > > > > > > > >> (catalina-exec-2:null)
>> > > > > > > > >> ===START=== 192.168.0.100 -- GET
>> > > > > > > > >>
>> command=createStoragePool&**zoneid=c116950e-e4ae-4f23-**a7e7-
>> > > > > > > > >>
>> 74a75c4ee638&podId=a748b063-**3a83-4175-a0e9-de39118fe5ce&**
>> > > > > > > > >>
>> clusterid=1f87eb09-324d-4d49-**83c2-88d84d7a15df&name=ceph-**
>> > > > > > > > >> rbd-pri-storage&url=rbd%3A%2F%**2Fclient.admin%3A_etc%2Fc
>> > > > > > > > >> eph%2Fceph.client.admin.**keyring%4010.237.11.2%2Fsc1&**
>> > > > > > > > >>
>> > > > >
>> tags=rbd&response=json&**sessionkey=**rDRfWpqeKfQKbKZtHr398ULV%2F8k%**
>> > > > > > > > >> 3D&_=1373447307839
>> > > > > > > > >> 2013-07-10 17:08:28,862 DEBUG
>> > > [cloud.storage.**StorageManagerImpl]
>> > > > > > > > >> (catalina-exec-2:null) createPool Params @ scheme - rbd
>> > > > > storageHost -
>> > > > > > > null
>> > > > > > > > >> hostPath - /ceph/ceph.client
>> > > > > > > > >> .admin.keyring@10.237.11.2/sc1 port - -1
>> > > > > > > > >> 2013-07-10 17:08:28,918 DEBUG
>> > > [cloud.storage.**StorageManagerImpl]
>> > > > > > > > >> (catalina-exec-2:null) In createPool Setting poolId - 217
>> > > uuid -
>> > > > > > > > >> a226c9a1-da78-3f3a-b5ac-**e18b925c9634 z
>> > > > > > > > >> oneId - 2 podId - 2 poolName - ceph-rbd-pri-storage
>> > > > > > > > >> 2013-07-10 17:08:28,921 DEBUG
>> [db.Transaction.Transaction]
>> > > > > > > > >> (catalina-exec-2:null) Rolling back the transaction:
>> Time = 3
>> > > > > Name =
>> > > > > > > > >> persist; called by -Transaction.rollbac
>> > > > > > > > >>
>> k:890-Transaction.removeUpTo:**833-Transaction.close:657-**
>> > > > > > > > >> TransactionContextBuilder.**interceptException:63-**
>> > > > > > > > >>
>> > > > >
>> ComponentInstantiationPostProc**essor$InterceptorDispatcher.**interce
>> > > > > > > > >> pt:133-StorageManagerImpl.**createPool:1378-**
>> > > > > > > > >>
>> StorageManagerImpl.createPool:**147-CreateStoragePoolCmd.**
>> > > > > > > > >> execute:123-ApiDispatcher.**dispatch:162-ApiServer.**
>> > > > > > > > >> queueCommand:505-ApiSe
>> > > > > > > > >> rver.handleRequest:355-**ApiServlet.processRequest:302
>> > > > > > > > >> 2013-07-10 17:08:28,923 ERROR [cloud.api.ApiServer]
>> > > > > > > (catalina-exec-2:null)
>> > > > > > > > >> unhandled exception executing api command:
>> createStoragePool
>> > > > > > > > >> com.cloud.utils.exception.**CloudRuntimeException: DB
>> > > Exception
>> > > > > on:
>> > > > > > > > >> com.mysql.jdbc.**JDBC4PreparedStatement@4b2eb56**:
>> INSERT
>> > > INTO
>> > > > > > > > >> storage_pool (
>> > > > > > > > >> storage_pool.id, storage_pool
>> > > > > > > > >> .name, storage_pool.uuid, storage_pool.pool_type,
>> > > > > > > storage_pool.created,
>> > > > > > > > >> storage_pool.update_time, storage_pool.data_center_id,
>> > > > > > > > >> storage_pool.pod_id,
>> > > > > > > > >> storage_pool.availab
>> > > > > > > > >> le_bytes, storage_pool.capacity_bytes,
>> storage_pool.status,
>> > > > > > > > >> storage_pool.scope, storage_pool.storage_provider_**id,
>> > > > > > > > >> storage_pool.host_address, storage_pool.path, storage_
>> > > > > > > > >> pool.port, storage_pool.user_info,
>> storage_pool.cluster_id)
>> > > VALUES
>> > > > > > > (217,
>> > > > > > > > >> _binary'ceph-rbd-pri-storage',
>> > > > > > > > >> _binary'a226c9a1-da78-3f3a-**b5ac-e18b925c9634', 'RBD',
>> > > '2013-07-1
>> > > > > > > > >> 0 09:08:28', null, 2, 2, 0, 0, 'Up', null, null, null,
>> > > > > _binary'ceph/
>> > > > > > > > >> ceph.client.admin.keyring@10.**237.11.2/sc1<
>> > > > > > > http://ceph.client.admin.keyring@10.237.11.2/sc1>',
>> > > > > > > > >> 6789, null, 2)
>> > > > > > > > >> at
>> > > > > > > > >>
>> > > > > > >
>> > >
>> com.cloud.utils.db.**GenericDaoBase.persist(**GenericDaoBase.java:1342)
>> > > > > > > > >> at
>> > > > > > > > >> com.cloud.storage.dao.**StoragePoolDaoImpl.persist(**
>> > > > > > > > >> StoragePoolDaoImpl.java:232)
>> > > > > > > > >> at
>> > > > > > > > >>
>> com.cloud.utils.component.**ComponentInstantiationPostProc**
>> > > > > > > > >>
>> > > > > > >
>> > > > >
>> > >
>> essor$InterceptorDispatcher.**intercept(**ComponentInstantiationPostProc*
>> > > > > > > > >> *es
>> > > > > > > > >>
>> > > > > > > > >>
>> > > > > > >
>> > > > > > > --
>> > > > > > > Prasanna.,
>> > > > > > >
>> > > > > > > ------------------------
>> > > > > > > Powered by BigRock.com
>> > > > > > >
>> > > > > > >
>> > > > >
>> > > > > --
>> > > > > Prasanna.,
>> > > > >
>> > > > > ------------------------
>> > > > > Powered by BigRock.com
>> > > > >
>> > > > >
>> > >
>> > > --
>> > > Prasanna.,
>> > >
>> > > ------------------------
>> > > Powered by BigRock.com
>> > >
>> > >
>>
>> --
>> Prasanna.,
>>
>> ------------------------
>> Powered by BigRock.com
>>
>>
>
Re: Problem in adding Ceph RBD as primary storage for CloudStack 4.1.0
Posted by Indra Pramana <in...@sg.or.id>.
Hi Prasanna,
Good day to you, and thank you for your e-mail.
Yes, the cloudstack-agent service is running on both the KVM hosts. There
is no "cloud" user being created though, when I installed the agent. I
installed the agent as root.
root@hv-kvm-01:/home/indra# service cloudstack-agent status
* cloud-agent is running
root@hv-kvm-01:/home/indra# su - cloud
Unknown id: cloud
Please advise how can I resolve this problem, shall I create the Unix
"cloud" user manually? Basically I follow this instruction to prepare the
KVM host and install the CloudStack agent:
http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.1.0/html/Installation_Guide/hypervisor-kvm-install-flow.html
with this instruction from Wido on how to prepare libvirt with Ceph RBD
storage pool support:
http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/
I also have checked /var/log/cloud/agent/agent.log and I don't see any
error messages, except this error message which will show up every time I
restart the agent:
2013-07-12 15:22:47,454 ERROR [cloud.resource.ServerResourceBase]
(main:null) Nics are not configured!
2013-07-12 15:22:47,459 INFO [cloud.resource.ServerResourceBase]
(main:null) Designating private to be nic eth0.5
More logs can be found here: http://pastebin.com/yeNmCt7S
I have configured the network bridges on the NIC interface as per this
instruction:
http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.1.0/html/Installation_Guide/hypervisor-kvm-install-flow.html#hypervisor-host-install-network
On the zone, I used advanced network configuration with just one physical
network for management, public and guest/private. I didn't include storage,
which by default the traffic will use the management VLAN network.
Please advise if there's anything else I might have been missing.
Looking forward to your reply, thank you.
Cheers.
On Fri, Jul 12, 2013 at 2:56 PM, Prasanna Santhanam <ts...@apache.org> wrote:
> Indeed, cloudstack will go through the allocation to startup the
> system VMs too. So that process is failing to recognize the volume
> (.qcow2) present on your NFS storage.
>
> Can you check if your cloudstack agent service is running with the KVM
> host? And it should've created the user cloud. $id cloud to check if
> there is a user.
>
> Did you see what's happening in the agent logs? These are under
> /var/log/cloud/ on your host when the systemVMs are coming up. If the
> logs are not showing any useful information you can turn on debug
> level for more verbosity.
>
> See here: https://cwiki.apache.org/confluence/x/FgPMAQ
>
> On Fri, Jul 12, 2013 at 02:40:35PM +0800, Indra Pramana wrote:
> > Hi Prasanna,
> >
> > Good day to you, and thank you for your e-mail.
> >
> > Yes, when I export the NFS, I set the permission so that normal user will
> > be able to have read/write access to the files (no_root_squash).
> >
> > I have tested and I can have read/write access from my KVM hosts using
> > normal user. BTW, there's no "cloud" user on the hosts, I believe it's
> not
> > created during cloudstack-agent installation?
> >
> > In any case, do you think the template issue and the storage pool
> > allocation issue might be related, or are they two different problems
> > altogether?
> >
> > Looking forward to your reply, thank you.
> >
> > Cheers.
> >
> >
> >
> > On Fri, Jul 12, 2013 at 2:26 PM, Prasanna Santhanam <ts...@apache.org>
> wrote:
> >
> > > Can you access the file as user root? Or user cloud? The cloudstack
> > > agent on your KVM host runs as user cloud and the NFS permissions
> > > might be disallowing the volume (.qcow2) from being accessed.
> > >
> > > On Fri, Jul 12, 2013 at 02:16:41PM +0800, Indra Pramana wrote:
> > > > Hi Prasanna,
> > > >
> > > > Good day to you, and thank you for your e-mail.
> > > >
> > > > Yes, the file exists. I can access the file from the management
> server
> > > and
> > > > the two hypervisors hosts if I mount manually.
> > > >
> > > > [root@cs-nas-01 /mnt/vol1/sec-storage/template/tmpl/1/3]# ls
> > > > -la
> > > > total
> > > > 1418787
> > > > drwxr-xr-x 2 root wheel 4 Jul 11 20:21
> > > > .
> > > > drwxr-xr-x 3 root wheel 3 Jul 11 20:17
> > > > ..
> > > > -rw-r--r-- 1 root wheel 725811200 Jul 11 20:21
> > > > 425b9e5a-fbc7-4637-a33a-fe9d0ed4fa98.qcow2
> > > > -rw-r--r-- 1 root wheel 295 Jul 11 20:21
> > > > template.properties
> > > > [root@cs-nas-01 /mnt/vol1/sec-storage/template/tmpl/1/3]#
> > > > pwd
> > > > /mnt/vol1/sec-storage/template/tmpl/1/3
> > > >
> > > >
> > > > Any advise?
> > > >
> > > > Looking forward to your reply, thank you.
> > > >
> > > > Cheers.
> > > >
> > > >
> > > >
> > > > On Fri, Jul 12, 2013 at 2:07 PM, Prasanna Santhanam <ts...@apache.org>
> > > wrote:
> > > >
> > > > > Can you check whether there is a file at:
> > > > > nfs://103.25.200.19/mnt/vol1/sec-storage/template/tmpl/1/3/
> > > > >
> > > > > On Fri, Jul 12, 2013 at 01:59:34PM +0800, Indra Pramana wrote:
> > > > > > Hi Prasanna,
> > > > > >
> > > > > > Thanks for your e-mail.
> > > > > >
> > > > > > I have tried restarting the management server, and the problem
> still
> > > > > > persists. I even tried to re-do the installation and
> configuration
> > > again
> > > > > > from scratch last night, but the problem still there.
> > > > > >
> > > > > > I also noted that on the beginning of the logs, I found some
> error
> > > > > messages
> > > > > > saying that the template cannot be downloaded to the pool. See
> this
> > > logs:
> > > > > >
> > > > > > http://pastebin.com/BY1AVJ08
> > > > > >
> > > > > > It says it failed because cannot get volume from the pool. Could
> it
> > > be
> > > > > > related, i.e. the absence of the template caused the system VMs
> > > cannot be
> > > > > > created and started?
> > > > > >
> > > > > > I have ensured that I downloaded the system VM template using
> > > > > > cloud-install-sys-tmplt and verified that the template is already
> > > there
> > > > > in
> > > > > > the secondary storage server.
> > > > > >
> > > > > > Any advice is appreciated.
> > > > > >
> > > > > > Looking forward to your reply, thank you.
> > > > > >
> > > > > > Cheers.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Jul 12, 2013 at 1:21 PM, Prasanna Santhanam <
> tsp@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > > It looks like a previous attempt to start the systemVMs has
> failed
> > > > > > > putting the nfs storage in the avoid set. Did you try
> restarting
> > > your
> > > > > > > management server?
> > > > > > >
> > > > > > > This line leads me to the above mentioned:
> > > > > > > 2013-07-12 13:10:48,236 DEBUG
> > > > > > > [storage.allocator.AbstractStoragePoolAllocator]
> > > (secstorage-1:null)
> > > > > > > StoragePool is in avoid set, skipping this pool
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Jul 12, 2013 at 01:16:53PM +0800, Indra Pramana wrote:
> > > > > > > > Dear Wido and all,
> > > > > > > >
> > > > > > > > I have managed to get the hosts, primary and secondary
> storage
> > > > > running.
> > > > > > > >
> > > > > > > > - 2 KVM hypervisor hosts
> > > > > > > > - One RBD primary storage
> > > > > > > > - One NFS primary storage (for system VMs, since I understand
> > > that
> > > > > system
> > > > > > > > VMs cannot use RBD)
> > > > > > > > - One NFS secondary storage
> > > > > > > >
> > > > > > > > However, now I am having problem with the system VMs: CPVM
> and
> > > SSVM,
> > > > > > > unable
> > > > > > > > to start.
> > > > > > > >
> > > > > > > > Excerpt from management-server.log file is here:
> > > > > > > > http://pastebin.com/ENkpCALY
> > > > > > > >
> > > > > > > > It seems that the VMs were not able to be created because
> unable
> > > to
> > > > > find
> > > > > > > > suitable StoragePools.
> > > > > > > >
> > > > > > > > I understand that system VMs will be using the NFS primary
> > > storage
> > > > > > > instead
> > > > > > > > of RBD, so I have confirmed that I am able to mount the
> primary
> > > > > storage
> > > > > > > via
> > > > > > > > NFS and have read and write access, from both the hypervisor
> and
> > > the
> > > > > > > > management server.
> > > > > > > >
> > > > > > > > Any advise how can I resolve the problem to make both the
> system
> > > VMs
> > > > > > > > created and started?
> > > > > > > >
> > > > > > > > Looking forward to your reply, thank you.
> > > > > > > >
> > > > > > > > Cheers.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Jul 12, 2013 at 9:43 AM, Indra Pramana <
> indra@sg.or.id>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Wido,
> > > > > > > > >
> > > > > > > > > Thanks for the advice, I'm now able to add the RBD pool as
> > > primary
> > > > > > > storage.
> > > > > > > > >
> > > > > > > > > Many thanks! :)
> > > > > > > > >
> > > > > > > > > Cheers.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thursday, July 11, 2013, Wido den Hollander wrote:
> > > > > > > > >
> > > > > > > > >> Hi,
> > > > > > > > >>
> > > > > > > > >> On 07/10/2013 03:42 PM, Chip Childers wrote:
> > > > > > > > >>
> > > > > > > > >>> Cc'ing Wido, our resident Ceph expert. ;-)
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >> Hehe ;)
> > > > > > > > >>
> > > > > > > > >> On Wed, Jul 10, 2013 at 05:45:25PM +0800, Indra Pramana
> > > wrote:
> > > > > > > > >>>
> > > > > > > > >>>> Dear all,
> > > > > > > > >>>>
> > > > > > > > >>>> I am installing CloudStack 4.1.0 (upgraded from 4.0.2)
> and I
> > > > > also
> > > > > > > have a
> > > > > > > > >>>> Ceph cluster running. However, I am having issues in
> adding
> > > the
> > > > > RBD
> > > > > > > as
> > > > > > > > >>>> primary storage. Tried to follow the instruction here,
> but
> > > > > unable to
> > > > > > > > >>>> make
> > > > > > > > >>>> it work:
> > > > > > > > >>>>
> > > > > > > > >>>> http://ceph.com/docs/master/**rbd/rbd-cloudstack/<
> > > > > > > http://ceph.com/docs/master/rbd/rbd-cloudstack/>
> > > > > > > > >>>>
> > > > > > > > >>>> I have setup a pool on the Ceph cluster. The status of
> the
> > > > > cluster
> > > > > > > is
> > > > > > > > >>>> healthy. Since I am using Ubuntu 12.04.2 LTS (Precise)
> for
> > > the
> > > > > > > > >>>> hypervisors,
> > > > > > > > >>>> I also have compiled libvirt manually to ensure that the
> > > version
> > > > > > > 0.9.13
> > > > > > > > >>>> is
> > > > > > > > >>>> installed (previously it's 0.9.8).
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >> You can also use the Ubuntu Cloud Archive, I still need to
> > > get the
> > > > > > > docs
> > > > > > > > >> updated for that.
> > > > > > > > >>
> > > > > > > > >> I described the process in a blogpost:
> > > > > > > http://blog.widodh.nl/2013/06/**
> > > > > > > > >>
> > > a-quick-note-on-running-**cloudstack-with-rbd-on-ubuntu-**12-04/<
> > > > > > >
> > > > >
> > >
> http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/
> > > > > > > >
> > > > > > > > >>
> > > > > > > > >> indra@hv-kvm-01:~/rbd$ ceph
> > > > > > > > >>>> ceph> health
> > > > > > > > >>>> HEALTH_OK
> > > > > > > > >>>>
> > > > > > > > >>>> indra@hv-kvm-01:~$ ceph osd lspools
> > > > > > > > >>>> 0 data,1 metadata,2 rbd,3 sc1,
> > > > > > > > >>>>
> > > > > > > > >>>> root@hv-kvm-01:/home/indra# libvirtd --version
> > > > > > > > >>>> libvirtd (libvirt) 0.9.13
> > > > > > > > >>>>
> > > > > > > > >>>> I tried to add Primary Storage into the Cloudstack zone
> > > which I
> > > > > have
> > > > > > > > >>>> created:
> > > > > > > > >>>>
> > > > > > > > >>>> Add Primary Storage:
> > > > > > > > >>>>
> > > > > > > > >>>> Zone: my zone name
> > > > > > > > >>>> Pod: my pod name
> > > > > > > > >>>> Cluster: my cluster name
> > > > > > > > >>>> Name: ceph-rbd-pri-storage
> > > > > > > > >>>> Protocol: RBD
> > > > > > > > >>>> RADOS Monitor: my first Ceph monitor IP address
> > > > > > > > >>>> RADOS Pool: sc1 (the pool name on Ceph cluster)
> > > > > > > > >>>> RADOS User: client.admin
> > > > > > > > >>>> RADOS Secret: /etc/ceph/ceph.client.admin.**keyring
> (keyring
> > > > > file
> > > > > > > > >>>> location)
> > > > > > > > >>>>
> > > > > > > > >>>
> > > > > > > > >> This is your problem. That shouldn't be the location of
> the
> > > file,
> > > > > but
> > > > > > > it
> > > > > > > > >> should be the secret, which is a base64 encoded string.
> > > > > > > > >>
> > > > > > > > >> $ ceph auth list
> > > > > > > > >>
> > > > > > > > >> That should tell you what the secret is.
> > > > > > > > >>
> > > > > > > > >> Storage Tags: rbd
> > > > > > > > >>
> > > > > > > > >> This is the error message when I tried to add the primary
> > > storage
> > > > > by
> > > > > > > > >> clicking OK:
> > > > > > > > >>
> > > > > > > > >> DB Exception on:
> > > com.mysql.jdbc.**JDBC4PreparedStatement@4b2eb56
> > > > > **:
> > > > > > > > >> INSERT INTO
> > > > > > > > >> storage_pool (storage_pool.id, storage_pool.name,
> > > > > storage_pool.uuid,
> > > > > > > > >> storage_pool.pool_type, storage_pool.created,
> > > > > > > storage_pool.update_time,
> > > > > > > > >> storage_pool.data_center_id, storage_pool.pod_id,
> > > > > > > > >> storage_pool.available_bytes, storage_pool.capacity_bytes,
> > > > > > > > >> storage_pool.status, storage_pool.scope,
> > > > > > > storage_pool.storage_provider_**
> > > > > > > > >> id,
> > > > > > > > >> storage_pool.host_address, storage_pool.path,
> > > storage_pool.port,
> > > > > > > > >> storage_pool.user_info, storage_pool.cluster_id) VALUES
> (217,
> > > > > > > > >> _binary'ceph-rbd-pri-storage',
> > > > > > > > >> _binary'a226c9a1-da78-3f3a-**b5ac-e18b925c9634', 'RBD',
> > > > > '2013-07-10
> > > > > > > > >> 09:08:28', null, 2, 2, 0, 0, 'Up', null, null, null,
> > > _binary'ceph/
> > > > > > > > >> ceph.client.admin.keyring@10.**237.11.2/sc1<
> > > > > > > http://ceph.client.admin.keyring@10.237.11.2/sc1>',
> > > > > > > > >> 6789, null, 2)
> > > > > > > > >>
> > > > > > > > >> On the management-server.log file:
> > > > > > > > >>
> > > > > > > > >> 2013-07-10 17:08:28,845 DEBUG [cloud.api.ApiServlet]
> > > > > > > > >> (catalina-exec-2:null)
> > > > > > > > >> ===START=== 192.168.0.100 -- GET
> > > > > > > > >>
> command=createStoragePool&**zoneid=c116950e-e4ae-4f23-**a7e7-
> > > > > > > > >>
> 74a75c4ee638&podId=a748b063-**3a83-4175-a0e9-de39118fe5ce&**
> > > > > > > > >>
> clusterid=1f87eb09-324d-4d49-**83c2-88d84d7a15df&name=ceph-**
> > > > > > > > >> rbd-pri-storage&url=rbd%3A%2F%**2Fclient.admin%3A_etc%2Fc
> > > > > > > > >> eph%2Fceph.client.admin.**keyring%4010.237.11.2%2Fsc1&**
> > > > > > > > >>
> > > > >
> tags=rbd&response=json&**sessionkey=**rDRfWpqeKfQKbKZtHr398ULV%2F8k%**
> > > > > > > > >> 3D&_=1373447307839
> > > > > > > > >> 2013-07-10 17:08:28,862 DEBUG
> > > [cloud.storage.**StorageManagerImpl]
> > > > > > > > >> (catalina-exec-2:null) createPool Params @ scheme - rbd
> > > > > storageHost -
> > > > > > > null
> > > > > > > > >> hostPath - /ceph/ceph.client
> > > > > > > > >> .admin.keyring@10.237.11.2/sc1 port - -1
> > > > > > > > >> 2013-07-10 17:08:28,918 DEBUG
> > > [cloud.storage.**StorageManagerImpl]
> > > > > > > > >> (catalina-exec-2:null) In createPool Setting poolId - 217
> > > uuid -
> > > > > > > > >> a226c9a1-da78-3f3a-b5ac-**e18b925c9634 z
> > > > > > > > >> oneId - 2 podId - 2 poolName - ceph-rbd-pri-storage
> > > > > > > > >> 2013-07-10 17:08:28,921 DEBUG [db.Transaction.Transaction]
> > > > > > > > >> (catalina-exec-2:null) Rolling back the transaction: Time
> = 3
> > > > > Name =
> > > > > > > > >> persist; called by -Transaction.rollbac
> > > > > > > > >>
> k:890-Transaction.removeUpTo:**833-Transaction.close:657-**
> > > > > > > > >> TransactionContextBuilder.**interceptException:63-**
> > > > > > > > >>
> > > > >
> ComponentInstantiationPostProc**essor$InterceptorDispatcher.**interce
> > > > > > > > >> pt:133-StorageManagerImpl.**createPool:1378-**
> > > > > > > > >>
> StorageManagerImpl.createPool:**147-CreateStoragePoolCmd.**
> > > > > > > > >> execute:123-ApiDispatcher.**dispatch:162-ApiServer.**
> > > > > > > > >> queueCommand:505-ApiSe
> > > > > > > > >> rver.handleRequest:355-**ApiServlet.processRequest:302
> > > > > > > > >> 2013-07-10 17:08:28,923 ERROR [cloud.api.ApiServer]
> > > > > > > (catalina-exec-2:null)
> > > > > > > > >> unhandled exception executing api command:
> createStoragePool
> > > > > > > > >> com.cloud.utils.exception.**CloudRuntimeException: DB
> > > Exception
> > > > > on:
> > > > > > > > >> com.mysql.jdbc.**JDBC4PreparedStatement@4b2eb56**: INSERT
> > > INTO
> > > > > > > > >> storage_pool (
> > > > > > > > >> storage_pool.id, storage_pool
> > > > > > > > >> .name, storage_pool.uuid, storage_pool.pool_type,
> > > > > > > storage_pool.created,
> > > > > > > > >> storage_pool.update_time, storage_pool.data_center_id,
> > > > > > > > >> storage_pool.pod_id,
> > > > > > > > >> storage_pool.availab
> > > > > > > > >> le_bytes, storage_pool.capacity_bytes,
> storage_pool.status,
> > > > > > > > >> storage_pool.scope, storage_pool.storage_provider_**id,
> > > > > > > > >> storage_pool.host_address, storage_pool.path, storage_
> > > > > > > > >> pool.port, storage_pool.user_info,
> storage_pool.cluster_id)
> > > VALUES
> > > > > > > (217,
> > > > > > > > >> _binary'ceph-rbd-pri-storage',
> > > > > > > > >> _binary'a226c9a1-da78-3f3a-**b5ac-e18b925c9634', 'RBD',
> > > '2013-07-1
> > > > > > > > >> 0 09:08:28', null, 2, 2, 0, 0, 'Up', null, null, null,
> > > > > _binary'ceph/
> > > > > > > > >> ceph.client.admin.keyring@10.**237.11.2/sc1<
> > > > > > > http://ceph.client.admin.keyring@10.237.11.2/sc1>',
> > > > > > > > >> 6789, null, 2)
> > > > > > > > >> at
> > > > > > > > >>
> > > > > > >
> > > com.cloud.utils.db.**GenericDaoBase.persist(**GenericDaoBase.java:1342)
> > > > > > > > >> at
> > > > > > > > >> com.cloud.storage.dao.**StoragePoolDaoImpl.persist(**
> > > > > > > > >> StoragePoolDaoImpl.java:232)
> > > > > > > > >> at
> > > > > > > > >>
> com.cloud.utils.component.**ComponentInstantiationPostProc**
> > > > > > > > >>
> > > > > > >
> > > > >
> > >
> essor$InterceptorDispatcher.**intercept(**ComponentInstantiationPostProc*
> > > > > > > > >> *es
> > > > > > > > >>
> > > > > > > > >>
> > > > > > >
> > > > > > > --
> > > > > > > Prasanna.,
> > > > > > >
> > > > > > > ------------------------
> > > > > > > Powered by BigRock.com
> > > > > > >
> > > > > > >
> > > > >
> > > > > --
> > > > > Prasanna.,
> > > > >
> > > > > ------------------------
> > > > > Powered by BigRock.com
> > > > >
> > > > >
> > >
> > > --
> > > Prasanna.,
> > >
> > > ------------------------
> > > Powered by BigRock.com
> > >
> > >
>
> --
> Prasanna.,
>
> ------------------------
> Powered by BigRock.com
>
>
Re: Problem in adding Ceph RBD as primary storage for CloudStack
4.1.0
Posted by Prasanna Santhanam <ts...@apache.org>.
Indeed, cloudstack will go through the allocation to startup the
system VMs too. So that process is failing to recognize the volume
(.qcow2) present on your NFS storage.
Can you check if your cloudstack agent service is running with the KVM
host? And it should've created the user cloud. $id cloud to check if
there is a user.
Did you see what's happening in the agent logs? These are under
/var/log/cloud/ on your host when the systemVMs are coming up. If the
logs are not showing any useful information you can turn on debug
level for more verbosity.
See here: https://cwiki.apache.org/confluence/x/FgPMAQ
On Fri, Jul 12, 2013 at 02:40:35PM +0800, Indra Pramana wrote:
> Hi Prasanna,
>
> Good day to you, and thank you for your e-mail.
>
> Yes, when I export the NFS, I set the permission so that normal user will
> be able to have read/write access to the files (no_root_squash).
>
> I have tested and I can have read/write access from my KVM hosts using
> normal user. BTW, there's no "cloud" user on the hosts, I believe it's not
> created during cloudstack-agent installation?
>
> In any case, do you think the template issue and the storage pool
> allocation issue might be related, or are they two different problems
> altogether?
>
> Looking forward to your reply, thank you.
>
> Cheers.
>
>
>
> On Fri, Jul 12, 2013 at 2:26 PM, Prasanna Santhanam <ts...@apache.org> wrote:
>
> > Can you access the file as user root? Or user cloud? The cloudstack
> > agent on your KVM host runs as user cloud and the NFS permissions
> > might be disallowing the volume (.qcow2) from being accessed.
> >
> > On Fri, Jul 12, 2013 at 02:16:41PM +0800, Indra Pramana wrote:
> > > Hi Prasanna,
> > >
> > > Good day to you, and thank you for your e-mail.
> > >
> > > Yes, the file exists. I can access the file from the management server
> > and
> > > the two hypervisors hosts if I mount manually.
> > >
> > > [root@cs-nas-01 /mnt/vol1/sec-storage/template/tmpl/1/3]# ls
> > > -la
> > > total
> > > 1418787
> > > drwxr-xr-x 2 root wheel 4 Jul 11 20:21
> > > .
> > > drwxr-xr-x 3 root wheel 3 Jul 11 20:17
> > > ..
> > > -rw-r--r-- 1 root wheel 725811200 Jul 11 20:21
> > > 425b9e5a-fbc7-4637-a33a-fe9d0ed4fa98.qcow2
> > > -rw-r--r-- 1 root wheel 295 Jul 11 20:21
> > > template.properties
> > > [root@cs-nas-01 /mnt/vol1/sec-storage/template/tmpl/1/3]#
> > > pwd
> > > /mnt/vol1/sec-storage/template/tmpl/1/3
> > >
> > >
> > > Any advise?
> > >
> > > Looking forward to your reply, thank you.
> > >
> > > Cheers.
> > >
> > >
> > >
> > > On Fri, Jul 12, 2013 at 2:07 PM, Prasanna Santhanam <ts...@apache.org>
> > wrote:
> > >
> > > > Can you check whether there is a file at:
> > > > nfs://103.25.200.19/mnt/vol1/sec-storage/template/tmpl/1/3/
> > > >
> > > > On Fri, Jul 12, 2013 at 01:59:34PM +0800, Indra Pramana wrote:
> > > > > Hi Prasanna,
> > > > >
> > > > > Thanks for your e-mail.
> > > > >
> > > > > I have tried restarting the management server, and the problem still
> > > > > persists. I even tried to re-do the installation and configuration
> > again
> > > > > from scratch last night, but the problem still there.
> > > > >
> > > > > I also noted that on the beginning of the logs, I found some error
> > > > messages
> > > > > saying that the template cannot be downloaded to the pool. See this
> > logs:
> > > > >
> > > > > http://pastebin.com/BY1AVJ08
> > > > >
> > > > > It says it failed because cannot get volume from the pool. Could it
> > be
> > > > > related, i.e. the absence of the template caused the system VMs
> > cannot be
> > > > > created and started?
> > > > >
> > > > > I have ensured that I downloaded the system VM template using
> > > > > cloud-install-sys-tmplt and verified that the template is already
> > there
> > > > in
> > > > > the secondary storage server.
> > > > >
> > > > > Any advice is appreciated.
> > > > >
> > > > > Looking forward to your reply, thank you.
> > > > >
> > > > > Cheers.
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Jul 12, 2013 at 1:21 PM, Prasanna Santhanam <ts...@apache.org>
> > > > wrote:
> > > > >
> > > > > > It looks like a previous attempt to start the systemVMs has failed
> > > > > > putting the nfs storage in the avoid set. Did you try restarting
> > your
> > > > > > management server?
> > > > > >
> > > > > > This line leads me to the above mentioned:
> > > > > > 2013-07-12 13:10:48,236 DEBUG
> > > > > > [storage.allocator.AbstractStoragePoolAllocator]
> > (secstorage-1:null)
> > > > > > StoragePool is in avoid set, skipping this pool
> > > > > >
> > > > > >
> > > > > > On Fri, Jul 12, 2013 at 01:16:53PM +0800, Indra Pramana wrote:
> > > > > > > Dear Wido and all,
> > > > > > >
> > > > > > > I have managed to get the hosts, primary and secondary storage
> > > > running.
> > > > > > >
> > > > > > > - 2 KVM hypervisor hosts
> > > > > > > - One RBD primary storage
> > > > > > > - One NFS primary storage (for system VMs, since I understand
> > that
> > > > system
> > > > > > > VMs cannot use RBD)
> > > > > > > - One NFS secondary storage
> > > > > > >
> > > > > > > However, now I am having problem with the system VMs: CPVM and
> > SSVM,
> > > > > > unable
> > > > > > > to start.
> > > > > > >
> > > > > > > Excerpt from management-server.log file is here:
> > > > > > > http://pastebin.com/ENkpCALY
> > > > > > >
> > > > > > > It seems that the VMs were not able to be created because unable
> > to
> > > > find
> > > > > > > suitable StoragePools.
> > > > > > >
> > > > > > > I understand that system VMs will be using the NFS primary
> > storage
> > > > > > instead
> > > > > > > of RBD, so I have confirmed that I am able to mount the primary
> > > > storage
> > > > > > via
> > > > > > > NFS and have read and write access, from both the hypervisor and
> > the
> > > > > > > management server.
> > > > > > >
> > > > > > > Any advise how can I resolve the problem to make both the system
> > VMs
> > > > > > > created and started?
> > > > > > >
> > > > > > > Looking forward to your reply, thank you.
> > > > > > >
> > > > > > > Cheers.
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Jul 12, 2013 at 9:43 AM, Indra Pramana <in...@sg.or.id>
> > > > wrote:
> > > > > > >
> > > > > > > > Hi Wido,
> > > > > > > >
> > > > > > > > Thanks for the advice, I'm now able to add the RBD pool as
> > primary
> > > > > > storage.
> > > > > > > >
> > > > > > > > Many thanks! :)
> > > > > > > >
> > > > > > > > Cheers.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thursday, July 11, 2013, Wido den Hollander wrote:
> > > > > > > >
> > > > > > > >> Hi,
> > > > > > > >>
> > > > > > > >> On 07/10/2013 03:42 PM, Chip Childers wrote:
> > > > > > > >>
> > > > > > > >>> Cc'ing Wido, our resident Ceph expert. ;-)
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >> Hehe ;)
> > > > > > > >>
> > > > > > > >> On Wed, Jul 10, 2013 at 05:45:25PM +0800, Indra Pramana
> > wrote:
> > > > > > > >>>
> > > > > > > >>>> Dear all,
> > > > > > > >>>>
> > > > > > > >>>> I am installing CloudStack 4.1.0 (upgraded from 4.0.2) and I
> > > > also
> > > > > > have a
> > > > > > > >>>> Ceph cluster running. However, I am having issues in adding
> > the
> > > > RBD
> > > > > > as
> > > > > > > >>>> primary storage. Tried to follow the instruction here, but
> > > > unable to
> > > > > > > >>>> make
> > > > > > > >>>> it work:
> > > > > > > >>>>
> > > > > > > >>>> http://ceph.com/docs/master/**rbd/rbd-cloudstack/<
> > > > > > http://ceph.com/docs/master/rbd/rbd-cloudstack/>
> > > > > > > >>>>
> > > > > > > >>>> I have setup a pool on the Ceph cluster. The status of the
> > > > cluster
> > > > > > is
> > > > > > > >>>> healthy. Since I am using Ubuntu 12.04.2 LTS (Precise) for
> > the
> > > > > > > >>>> hypervisors,
> > > > > > > >>>> I also have compiled libvirt manually to ensure that the
> > version
> > > > > > 0.9.13
> > > > > > > >>>> is
> > > > > > > >>>> installed (previously it's 0.9.8).
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >> You can also use the Ubuntu Cloud Archive, I still need to
> > get the
> > > > > > docs
> > > > > > > >> updated for that.
> > > > > > > >>
> > > > > > > >> I described the process in a blogpost:
> > > > > > http://blog.widodh.nl/2013/06/**
> > > > > > > >>
> > a-quick-note-on-running-**cloudstack-with-rbd-on-ubuntu-**12-04/<
> > > > > >
> > > >
> > http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/
> > > > > > >
> > > > > > > >>
> > > > > > > >> indra@hv-kvm-01:~/rbd$ ceph
> > > > > > > >>>> ceph> health
> > > > > > > >>>> HEALTH_OK
> > > > > > > >>>>
> > > > > > > >>>> indra@hv-kvm-01:~$ ceph osd lspools
> > > > > > > >>>> 0 data,1 metadata,2 rbd,3 sc1,
> > > > > > > >>>>
> > > > > > > >>>> root@hv-kvm-01:/home/indra# libvirtd --version
> > > > > > > >>>> libvirtd (libvirt) 0.9.13
> > > > > > > >>>>
> > > > > > > >>>> I tried to add Primary Storage into the Cloudstack zone
> > which I
> > > > have
> > > > > > > >>>> created:
> > > > > > > >>>>
> > > > > > > >>>> Add Primary Storage:
> > > > > > > >>>>
> > > > > > > >>>> Zone: my zone name
> > > > > > > >>>> Pod: my pod name
> > > > > > > >>>> Cluster: my cluster name
> > > > > > > >>>> Name: ceph-rbd-pri-storage
> > > > > > > >>>> Protocol: RBD
> > > > > > > >>>> RADOS Monitor: my first Ceph monitor IP address
> > > > > > > >>>> RADOS Pool: sc1 (the pool name on Ceph cluster)
> > > > > > > >>>> RADOS User: client.admin
> > > > > > > >>>> RADOS Secret: /etc/ceph/ceph.client.admin.**keyring (keyring
> > > > file
> > > > > > > >>>> location)
> > > > > > > >>>>
> > > > > > > >>>
> > > > > > > >> This is your problem. That shouldn't be the location of the
> > file,
> > > > but
> > > > > > it
> > > > > > > >> should be the secret, which is a base64 encoded string.
> > > > > > > >>
> > > > > > > >> $ ceph auth list
> > > > > > > >>
> > > > > > > >> That should tell you what the secret is.
> > > > > > > >>
> > > > > > > >> Storage Tags: rbd
> > > > > > > >>
> > > > > > > >> This is the error message when I tried to add the primary
> > storage
> > > > by
> > > > > > > >> clicking OK:
> > > > > > > >>
> > > > > > > >> DB Exception on:
> > com.mysql.jdbc.**JDBC4PreparedStatement@4b2eb56
> > > > **:
> > > > > > > >> INSERT INTO
> > > > > > > >> storage_pool (storage_pool.id, storage_pool.name,
> > > > storage_pool.uuid,
> > > > > > > >> storage_pool.pool_type, storage_pool.created,
> > > > > > storage_pool.update_time,
> > > > > > > >> storage_pool.data_center_id, storage_pool.pod_id,
> > > > > > > >> storage_pool.available_bytes, storage_pool.capacity_bytes,
> > > > > > > >> storage_pool.status, storage_pool.scope,
> > > > > > storage_pool.storage_provider_**
> > > > > > > >> id,
> > > > > > > >> storage_pool.host_address, storage_pool.path,
> > storage_pool.port,
> > > > > > > >> storage_pool.user_info, storage_pool.cluster_id) VALUES (217,
> > > > > > > >> _binary'ceph-rbd-pri-storage',
> > > > > > > >> _binary'a226c9a1-da78-3f3a-**b5ac-e18b925c9634', 'RBD',
> > > > '2013-07-10
> > > > > > > >> 09:08:28', null, 2, 2, 0, 0, 'Up', null, null, null,
> > _binary'ceph/
> > > > > > > >> ceph.client.admin.keyring@10.**237.11.2/sc1<
> > > > > > http://ceph.client.admin.keyring@10.237.11.2/sc1>',
> > > > > > > >> 6789, null, 2)
> > > > > > > >>
> > > > > > > >> On the management-server.log file:
> > > > > > > >>
> > > > > > > >> 2013-07-10 17:08:28,845 DEBUG [cloud.api.ApiServlet]
> > > > > > > >> (catalina-exec-2:null)
> > > > > > > >> ===START=== 192.168.0.100 -- GET
> > > > > > > >> command=createStoragePool&**zoneid=c116950e-e4ae-4f23-**a7e7-
> > > > > > > >> 74a75c4ee638&podId=a748b063-**3a83-4175-a0e9-de39118fe5ce&**
> > > > > > > >> clusterid=1f87eb09-324d-4d49-**83c2-88d84d7a15df&name=ceph-**
> > > > > > > >> rbd-pri-storage&url=rbd%3A%2F%**2Fclient.admin%3A_etc%2Fc
> > > > > > > >> eph%2Fceph.client.admin.**keyring%4010.237.11.2%2Fsc1&**
> > > > > > > >>
> > > > tags=rbd&response=json&**sessionkey=**rDRfWpqeKfQKbKZtHr398ULV%2F8k%**
> > > > > > > >> 3D&_=1373447307839
> > > > > > > >> 2013-07-10 17:08:28,862 DEBUG
> > [cloud.storage.**StorageManagerImpl]
> > > > > > > >> (catalina-exec-2:null) createPool Params @ scheme - rbd
> > > > storageHost -
> > > > > > null
> > > > > > > >> hostPath - /ceph/ceph.client
> > > > > > > >> .admin.keyring@10.237.11.2/sc1 port - -1
> > > > > > > >> 2013-07-10 17:08:28,918 DEBUG
> > [cloud.storage.**StorageManagerImpl]
> > > > > > > >> (catalina-exec-2:null) In createPool Setting poolId - 217
> > uuid -
> > > > > > > >> a226c9a1-da78-3f3a-b5ac-**e18b925c9634 z
> > > > > > > >> oneId - 2 podId - 2 poolName - ceph-rbd-pri-storage
> > > > > > > >> 2013-07-10 17:08:28,921 DEBUG [db.Transaction.Transaction]
> > > > > > > >> (catalina-exec-2:null) Rolling back the transaction: Time = 3
> > > > Name =
> > > > > > > >> persist; called by -Transaction.rollbac
> > > > > > > >> k:890-Transaction.removeUpTo:**833-Transaction.close:657-**
> > > > > > > >> TransactionContextBuilder.**interceptException:63-**
> > > > > > > >>
> > > > ComponentInstantiationPostProc**essor$InterceptorDispatcher.**interce
> > > > > > > >> pt:133-StorageManagerImpl.**createPool:1378-**
> > > > > > > >> StorageManagerImpl.createPool:**147-CreateStoragePoolCmd.**
> > > > > > > >> execute:123-ApiDispatcher.**dispatch:162-ApiServer.**
> > > > > > > >> queueCommand:505-ApiSe
> > > > > > > >> rver.handleRequest:355-**ApiServlet.processRequest:302
> > > > > > > >> 2013-07-10 17:08:28,923 ERROR [cloud.api.ApiServer]
> > > > > > (catalina-exec-2:null)
> > > > > > > >> unhandled exception executing api command: createStoragePool
> > > > > > > >> com.cloud.utils.exception.**CloudRuntimeException: DB
> > Exception
> > > > on:
> > > > > > > >> com.mysql.jdbc.**JDBC4PreparedStatement@4b2eb56**: INSERT
> > INTO
> > > > > > > >> storage_pool (
> > > > > > > >> storage_pool.id, storage_pool
> > > > > > > >> .name, storage_pool.uuid, storage_pool.pool_type,
> > > > > > storage_pool.created,
> > > > > > > >> storage_pool.update_time, storage_pool.data_center_id,
> > > > > > > >> storage_pool.pod_id,
> > > > > > > >> storage_pool.availab
> > > > > > > >> le_bytes, storage_pool.capacity_bytes, storage_pool.status,
> > > > > > > >> storage_pool.scope, storage_pool.storage_provider_**id,
> > > > > > > >> storage_pool.host_address, storage_pool.path, storage_
> > > > > > > >> pool.port, storage_pool.user_info, storage_pool.cluster_id)
> > VALUES
> > > > > > (217,
> > > > > > > >> _binary'ceph-rbd-pri-storage',
> > > > > > > >> _binary'a226c9a1-da78-3f3a-**b5ac-e18b925c9634', 'RBD',
> > '2013-07-1
> > > > > > > >> 0 09:08:28', null, 2, 2, 0, 0, 'Up', null, null, null,
> > > > _binary'ceph/
> > > > > > > >> ceph.client.admin.keyring@10.**237.11.2/sc1<
> > > > > > http://ceph.client.admin.keyring@10.237.11.2/sc1>',
> > > > > > > >> 6789, null, 2)
> > > > > > > >> at
> > > > > > > >>
> > > > > >
> > com.cloud.utils.db.**GenericDaoBase.persist(**GenericDaoBase.java:1342)
> > > > > > > >> at
> > > > > > > >> com.cloud.storage.dao.**StoragePoolDaoImpl.persist(**
> > > > > > > >> StoragePoolDaoImpl.java:232)
> > > > > > > >> at
> > > > > > > >> com.cloud.utils.component.**ComponentInstantiationPostProc**
> > > > > > > >>
> > > > > >
> > > >
> > essor$InterceptorDispatcher.**intercept(**ComponentInstantiationPostProc*
> > > > > > > >> *es
> > > > > > > >>
> > > > > > > >>
> > > > > >
> > > > > > --
> > > > > > Prasanna.,
> > > > > >
> > > > > > ------------------------
> > > > > > Powered by BigRock.com
> > > > > >
> > > > > >
> > > >
> > > > --
> > > > Prasanna.,
> > > >
> > > > ------------------------
> > > > Powered by BigRock.com
> > > >
> > > >
> >
> > --
> > Prasanna.,
> >
> > ------------------------
> > Powered by BigRock.com
> >
> >
--
Prasanna.,
------------------------
Powered by BigRock.com
Re: Problem in adding Ceph RBD as primary storage for CloudStack 4.1.0
Posted by Indra Pramana <in...@sg.or.id>.
Hi Prasanna,
Good day to you, and thank you for your e-mail.
Yes, when I export the NFS, I set the permission so that normal user will
be able to have read/write access to the files (no_root_squash).
I have tested and I can have read/write access from my KVM hosts using
normal user. BTW, there's no "cloud" user on the hosts, I believe it's not
created during cloudstack-agent installation?
In any case, do you think the template issue and the storage pool
allocation issue might be related, or are they two different problems
altogether?
Looking forward to your reply, thank you.
Cheers.
On Fri, Jul 12, 2013 at 2:26 PM, Prasanna Santhanam <ts...@apache.org> wrote:
> Can you access the file as user root? Or user cloud? The cloudstack
> agent on your KVM host runs as user cloud and the NFS permissions
> might be disallowing the volume (.qcow2) from being accessed.
>
> On Fri, Jul 12, 2013 at 02:16:41PM +0800, Indra Pramana wrote:
> > Hi Prasanna,
> >
> > Good day to you, and thank you for your e-mail.
> >
> > Yes, the file exists. I can access the file from the management server
> and
> > the two hypervisors hosts if I mount manually.
> >
> > [root@cs-nas-01 /mnt/vol1/sec-storage/template/tmpl/1/3]# ls
> > -la
> > total
> > 1418787
> > drwxr-xr-x 2 root wheel 4 Jul 11 20:21
> > .
> > drwxr-xr-x 3 root wheel 3 Jul 11 20:17
> > ..
> > -rw-r--r-- 1 root wheel 725811200 Jul 11 20:21
> > 425b9e5a-fbc7-4637-a33a-fe9d0ed4fa98.qcow2
> > -rw-r--r-- 1 root wheel 295 Jul 11 20:21
> > template.properties
> > [root@cs-nas-01 /mnt/vol1/sec-storage/template/tmpl/1/3]#
> > pwd
> > /mnt/vol1/sec-storage/template/tmpl/1/3
> >
> >
> > Any advise?
> >
> > Looking forward to your reply, thank you.
> >
> > Cheers.
> >
> >
> >
> > On Fri, Jul 12, 2013 at 2:07 PM, Prasanna Santhanam <ts...@apache.org>
> wrote:
> >
> > > Can you check whether there is a file at:
> > > nfs://103.25.200.19/mnt/vol1/sec-storage/template/tmpl/1/3/
> > >
> > > On Fri, Jul 12, 2013 at 01:59:34PM +0800, Indra Pramana wrote:
> > > > Hi Prasanna,
> > > >
> > > > Thanks for your e-mail.
> > > >
> > > > I have tried restarting the management server, and the problem still
> > > > persists. I even tried to re-do the installation and configuration
> again
> > > > from scratch last night, but the problem still there.
> > > >
> > > > I also noted that on the beginning of the logs, I found some error
> > > messages
> > > > saying that the template cannot be downloaded to the pool. See this
> logs:
> > > >
> > > > http://pastebin.com/BY1AVJ08
> > > >
> > > > It says it failed because cannot get volume from the pool. Could it
> be
> > > > related, i.e. the absence of the template caused the system VMs
> cannot be
> > > > created and started?
> > > >
> > > > I have ensured that I downloaded the system VM template using
> > > > cloud-install-sys-tmplt and verified that the template is already
> there
> > > in
> > > > the secondary storage server.
> > > >
> > > > Any advice is appreciated.
> > > >
> > > > Looking forward to your reply, thank you.
> > > >
> > > > Cheers.
> > > >
> > > >
> > > >
> > > > On Fri, Jul 12, 2013 at 1:21 PM, Prasanna Santhanam <ts...@apache.org>
> > > wrote:
> > > >
> > > > > It looks like a previous attempt to start the systemVMs has failed
> > > > > putting the nfs storage in the avoid set. Did you try restarting
> your
> > > > > management server?
> > > > >
> > > > > This line leads me to the above mentioned:
> > > > > 2013-07-12 13:10:48,236 DEBUG
> > > > > [storage.allocator.AbstractStoragePoolAllocator]
> (secstorage-1:null)
> > > > > StoragePool is in avoid set, skipping this pool
> > > > >
> > > > >
> > > > > On Fri, Jul 12, 2013 at 01:16:53PM +0800, Indra Pramana wrote:
> > > > > > Dear Wido and all,
> > > > > >
> > > > > > I have managed to get the hosts, primary and secondary storage
> > > running.
> > > > > >
> > > > > > - 2 KVM hypervisor hosts
> > > > > > - One RBD primary storage
> > > > > > - One NFS primary storage (for system VMs, since I understand
> that
> > > system
> > > > > > VMs cannot use RBD)
> > > > > > - One NFS secondary storage
> > > > > >
> > > > > > However, now I am having problem with the system VMs: CPVM and
> SSVM,
> > > > > unable
> > > > > > to start.
> > > > > >
> > > > > > Excerpt from management-server.log file is here:
> > > > > > http://pastebin.com/ENkpCALY
> > > > > >
> > > > > > It seems that the VMs were not able to be created because unable
> to
> > > find
> > > > > > suitable StoragePools.
> > > > > >
> > > > > > I understand that system VMs will be using the NFS primary
> storage
> > > > > instead
> > > > > > of RBD, so I have confirmed that I am able to mount the primary
> > > storage
> > > > > via
> > > > > > NFS and have read and write access, from both the hypervisor and
> the
> > > > > > management server.
> > > > > >
> > > > > > Any advise how can I resolve the problem to make both the system
> VMs
> > > > > > created and started?
> > > > > >
> > > > > > Looking forward to your reply, thank you.
> > > > > >
> > > > > > Cheers.
> > > > > >
> > > > > >
> > > > > > On Fri, Jul 12, 2013 at 9:43 AM, Indra Pramana <in...@sg.or.id>
> > > wrote:
> > > > > >
> > > > > > > Hi Wido,
> > > > > > >
> > > > > > > Thanks for the advice, I'm now able to add the RBD pool as
> primary
> > > > > storage.
> > > > > > >
> > > > > > > Many thanks! :)
> > > > > > >
> > > > > > > Cheers.
> > > > > > >
> > > > > > >
> > > > > > > On Thursday, July 11, 2013, Wido den Hollander wrote:
> > > > > > >
> > > > > > >> Hi,
> > > > > > >>
> > > > > > >> On 07/10/2013 03:42 PM, Chip Childers wrote:
> > > > > > >>
> > > > > > >>> Cc'ing Wido, our resident Ceph expert. ;-)
> > > > > > >>>
> > > > > > >>>
> > > > > > >> Hehe ;)
> > > > > > >>
> > > > > > >> On Wed, Jul 10, 2013 at 05:45:25PM +0800, Indra Pramana
> wrote:
> > > > > > >>>
> > > > > > >>>> Dear all,
> > > > > > >>>>
> > > > > > >>>> I am installing CloudStack 4.1.0 (upgraded from 4.0.2) and I
> > > also
> > > > > have a
> > > > > > >>>> Ceph cluster running. However, I am having issues in adding
> the
> > > RBD
> > > > > as
> > > > > > >>>> primary storage. Tried to follow the instruction here, but
> > > unable to
> > > > > > >>>> make
> > > > > > >>>> it work:
> > > > > > >>>>
> > > > > > >>>> http://ceph.com/docs/master/**rbd/rbd-cloudstack/<
> > > > > http://ceph.com/docs/master/rbd/rbd-cloudstack/>
> > > > > > >>>>
> > > > > > >>>> I have setup a pool on the Ceph cluster. The status of the
> > > cluster
> > > > > is
> > > > > > >>>> healthy. Since I am using Ubuntu 12.04.2 LTS (Precise) for
> the
> > > > > > >>>> hypervisors,
> > > > > > >>>> I also have compiled libvirt manually to ensure that the
> version
> > > > > 0.9.13
> > > > > > >>>> is
> > > > > > >>>> installed (previously it's 0.9.8).
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >> You can also use the Ubuntu Cloud Archive, I still need to
> get the
> > > > > docs
> > > > > > >> updated for that.
> > > > > > >>
> > > > > > >> I described the process in a blogpost:
> > > > > http://blog.widodh.nl/2013/06/**
> > > > > > >>
> a-quick-note-on-running-**cloudstack-with-rbd-on-ubuntu-**12-04/<
> > > > >
> > >
> http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/
> > > > > >
> > > > > > >>
> > > > > > >> indra@hv-kvm-01:~/rbd$ ceph
> > > > > > >>>> ceph> health
> > > > > > >>>> HEALTH_OK
> > > > > > >>>>
> > > > > > >>>> indra@hv-kvm-01:~$ ceph osd lspools
> > > > > > >>>> 0 data,1 metadata,2 rbd,3 sc1,
> > > > > > >>>>
> > > > > > >>>> root@hv-kvm-01:/home/indra# libvirtd --version
> > > > > > >>>> libvirtd (libvirt) 0.9.13
> > > > > > >>>>
> > > > > > >>>> I tried to add Primary Storage into the Cloudstack zone
> which I
> > > have
> > > > > > >>>> created:
> > > > > > >>>>
> > > > > > >>>> Add Primary Storage:
> > > > > > >>>>
> > > > > > >>>> Zone: my zone name
> > > > > > >>>> Pod: my pod name
> > > > > > >>>> Cluster: my cluster name
> > > > > > >>>> Name: ceph-rbd-pri-storage
> > > > > > >>>> Protocol: RBD
> > > > > > >>>> RADOS Monitor: my first Ceph monitor IP address
> > > > > > >>>> RADOS Pool: sc1 (the pool name on Ceph cluster)
> > > > > > >>>> RADOS User: client.admin
> > > > > > >>>> RADOS Secret: /etc/ceph/ceph.client.admin.**keyring (keyring
> > > file
> > > > > > >>>> location)
> > > > > > >>>>
> > > > > > >>>
> > > > > > >> This is your problem. That shouldn't be the location of the
> file,
> > > but
> > > > > it
> > > > > > >> should be the secret, which is a base64 encoded string.
> > > > > > >>
> > > > > > >> $ ceph auth list
> > > > > > >>
> > > > > > >> That should tell you what the secret is.
> > > > > > >>
> > > > > > >> Storage Tags: rbd
> > > > > > >>
> > > > > > >> This is the error message when I tried to add the primary
> storage
> > > by
> > > > > > >> clicking OK:
> > > > > > >>
> > > > > > >> DB Exception on:
> com.mysql.jdbc.**JDBC4PreparedStatement@4b2eb56
> > > **:
> > > > > > >> INSERT INTO
> > > > > > >> storage_pool (storage_pool.id, storage_pool.name,
> > > storage_pool.uuid,
> > > > > > >> storage_pool.pool_type, storage_pool.created,
> > > > > storage_pool.update_time,
> > > > > > >> storage_pool.data_center_id, storage_pool.pod_id,
> > > > > > >> storage_pool.available_bytes, storage_pool.capacity_bytes,
> > > > > > >> storage_pool.status, storage_pool.scope,
> > > > > storage_pool.storage_provider_**
> > > > > > >> id,
> > > > > > >> storage_pool.host_address, storage_pool.path,
> storage_pool.port,
> > > > > > >> storage_pool.user_info, storage_pool.cluster_id) VALUES (217,
> > > > > > >> _binary'ceph-rbd-pri-storage',
> > > > > > >> _binary'a226c9a1-da78-3f3a-**b5ac-e18b925c9634', 'RBD',
> > > '2013-07-10
> > > > > > >> 09:08:28', null, 2, 2, 0, 0, 'Up', null, null, null,
> _binary'ceph/
> > > > > > >> ceph.client.admin.keyring@10.**237.11.2/sc1<
> > > > > http://ceph.client.admin.keyring@10.237.11.2/sc1>',
> > > > > > >> 6789, null, 2)
> > > > > > >>
> > > > > > >> On the management-server.log file:
> > > > > > >>
> > > > > > >> 2013-07-10 17:08:28,845 DEBUG [cloud.api.ApiServlet]
> > > > > > >> (catalina-exec-2:null)
> > > > > > >> ===START=== 192.168.0.100 -- GET
> > > > > > >> command=createStoragePool&**zoneid=c116950e-e4ae-4f23-**a7e7-
> > > > > > >> 74a75c4ee638&podId=a748b063-**3a83-4175-a0e9-de39118fe5ce&**
> > > > > > >> clusterid=1f87eb09-324d-4d49-**83c2-88d84d7a15df&name=ceph-**
> > > > > > >> rbd-pri-storage&url=rbd%3A%2F%**2Fclient.admin%3A_etc%2Fc
> > > > > > >> eph%2Fceph.client.admin.**keyring%4010.237.11.2%2Fsc1&**
> > > > > > >>
> > > tags=rbd&response=json&**sessionkey=**rDRfWpqeKfQKbKZtHr398ULV%2F8k%**
> > > > > > >> 3D&_=1373447307839
> > > > > > >> 2013-07-10 17:08:28,862 DEBUG
> [cloud.storage.**StorageManagerImpl]
> > > > > > >> (catalina-exec-2:null) createPool Params @ scheme - rbd
> > > storageHost -
> > > > > null
> > > > > > >> hostPath - /ceph/ceph.client
> > > > > > >> .admin.keyring@10.237.11.2/sc1 port - -1
> > > > > > >> 2013-07-10 17:08:28,918 DEBUG
> [cloud.storage.**StorageManagerImpl]
> > > > > > >> (catalina-exec-2:null) In createPool Setting poolId - 217
> uuid -
> > > > > > >> a226c9a1-da78-3f3a-b5ac-**e18b925c9634 z
> > > > > > >> oneId - 2 podId - 2 poolName - ceph-rbd-pri-storage
> > > > > > >> 2013-07-10 17:08:28,921 DEBUG [db.Transaction.Transaction]
> > > > > > >> (catalina-exec-2:null) Rolling back the transaction: Time = 3
> > > Name =
> > > > > > >> persist; called by -Transaction.rollbac
> > > > > > >> k:890-Transaction.removeUpTo:**833-Transaction.close:657-**
> > > > > > >> TransactionContextBuilder.**interceptException:63-**
> > > > > > >>
> > > ComponentInstantiationPostProc**essor$InterceptorDispatcher.**interce
> > > > > > >> pt:133-StorageManagerImpl.**createPool:1378-**
> > > > > > >> StorageManagerImpl.createPool:**147-CreateStoragePoolCmd.**
> > > > > > >> execute:123-ApiDispatcher.**dispatch:162-ApiServer.**
> > > > > > >> queueCommand:505-ApiSe
> > > > > > >> rver.handleRequest:355-**ApiServlet.processRequest:302
> > > > > > >> 2013-07-10 17:08:28,923 ERROR [cloud.api.ApiServer]
> > > > > (catalina-exec-2:null)
> > > > > > >> unhandled exception executing api command: createStoragePool
> > > > > > >> com.cloud.utils.exception.**CloudRuntimeException: DB
> Exception
> > > on:
> > > > > > >> com.mysql.jdbc.**JDBC4PreparedStatement@4b2eb56**: INSERT
> INTO
> > > > > > >> storage_pool (
> > > > > > >> storage_pool.id, storage_pool
> > > > > > >> .name, storage_pool.uuid, storage_pool.pool_type,
> > > > > storage_pool.created,
> > > > > > >> storage_pool.update_time, storage_pool.data_center_id,
> > > > > > >> storage_pool.pod_id,
> > > > > > >> storage_pool.availab
> > > > > > >> le_bytes, storage_pool.capacity_bytes, storage_pool.status,
> > > > > > >> storage_pool.scope, storage_pool.storage_provider_**id,
> > > > > > >> storage_pool.host_address, storage_pool.path, storage_
> > > > > > >> pool.port, storage_pool.user_info, storage_pool.cluster_id)
> VALUES
> > > > > (217,
> > > > > > >> _binary'ceph-rbd-pri-storage',
> > > > > > >> _binary'a226c9a1-da78-3f3a-**b5ac-e18b925c9634', 'RBD',
> '2013-07-1
> > > > > > >> 0 09:08:28', null, 2, 2, 0, 0, 'Up', null, null, null,
> > > _binary'ceph/
> > > > > > >> ceph.client.admin.keyring@10.**237.11.2/sc1<
> > > > > http://ceph.client.admin.keyring@10.237.11.2/sc1>',
> > > > > > >> 6789, null, 2)
> > > > > > >> at
> > > > > > >>
> > > > >
> com.cloud.utils.db.**GenericDaoBase.persist(**GenericDaoBase.java:1342)
> > > > > > >> at
> > > > > > >> com.cloud.storage.dao.**StoragePoolDaoImpl.persist(**
> > > > > > >> StoragePoolDaoImpl.java:232)
> > > > > > >> at
> > > > > > >> com.cloud.utils.component.**ComponentInstantiationPostProc**
> > > > > > >>
> > > > >
> > >
> essor$InterceptorDispatcher.**intercept(**ComponentInstantiationPostProc*
> > > > > > >> *es
> > > > > > >>
> > > > > > >>
> > > > >
> > > > > --
> > > > > Prasanna.,
> > > > >
> > > > > ------------------------
> > > > > Powered by BigRock.com
> > > > >
> > > > >
> > >
> > > --
> > > Prasanna.,
> > >
> > > ------------------------
> > > Powered by BigRock.com
> > >
> > >
>
> --
> Prasanna.,
>
> ------------------------
> Powered by BigRock.com
>
>
Re: Problem in adding Ceph RBD as primary storage for CloudStack
4.1.0
Posted by Prasanna Santhanam <ts...@apache.org>.
Can you access the file as user root? Or user cloud? The cloudstack
agent on your KVM host runs as user cloud and the NFS permissions
might be disallowing the volume (.qcow2) from being accessed.
On Fri, Jul 12, 2013 at 02:16:41PM +0800, Indra Pramana wrote:
> Hi Prasanna,
>
> Good day to you, and thank you for your e-mail.
>
> Yes, the file exists. I can access the file from the management server and
> the two hypervisors hosts if I mount manually.
>
> [root@cs-nas-01 /mnt/vol1/sec-storage/template/tmpl/1/3]# ls
> -la
> total
> 1418787
> drwxr-xr-x 2 root wheel 4 Jul 11 20:21
> .
> drwxr-xr-x 3 root wheel 3 Jul 11 20:17
> ..
> -rw-r--r-- 1 root wheel 725811200 Jul 11 20:21
> 425b9e5a-fbc7-4637-a33a-fe9d0ed4fa98.qcow2
> -rw-r--r-- 1 root wheel 295 Jul 11 20:21
> template.properties
> [root@cs-nas-01 /mnt/vol1/sec-storage/template/tmpl/1/3]#
> pwd
> /mnt/vol1/sec-storage/template/tmpl/1/3
>
>
> Any advise?
>
> Looking forward to your reply, thank you.
>
> Cheers.
>
>
>
> On Fri, Jul 12, 2013 at 2:07 PM, Prasanna Santhanam <ts...@apache.org> wrote:
>
> > Can you check whether there is a file at:
> > nfs://103.25.200.19/mnt/vol1/sec-storage/template/tmpl/1/3/
> >
> > On Fri, Jul 12, 2013 at 01:59:34PM +0800, Indra Pramana wrote:
> > > Hi Prasanna,
> > >
> > > Thanks for your e-mail.
> > >
> > > I have tried restarting the management server, and the problem still
> > > persists. I even tried to re-do the installation and configuration again
> > > from scratch last night, but the problem still there.
> > >
> > > I also noted that on the beginning of the logs, I found some error
> > messages
> > > saying that the template cannot be downloaded to the pool. See this logs:
> > >
> > > http://pastebin.com/BY1AVJ08
> > >
> > > It says it failed because cannot get volume from the pool. Could it be
> > > related, i.e. the absence of the template caused the system VMs cannot be
> > > created and started?
> > >
> > > I have ensured that I downloaded the system VM template using
> > > cloud-install-sys-tmplt and verified that the template is already there
> > in
> > > the secondary storage server.
> > >
> > > Any advice is appreciated.
> > >
> > > Looking forward to your reply, thank you.
> > >
> > > Cheers.
> > >
> > >
> > >
> > > On Fri, Jul 12, 2013 at 1:21 PM, Prasanna Santhanam <ts...@apache.org>
> > wrote:
> > >
> > > > It looks like a previous attempt to start the systemVMs has failed
> > > > putting the nfs storage in the avoid set. Did you try restarting your
> > > > management server?
> > > >
> > > > This line leads me to the above mentioned:
> > > > 2013-07-12 13:10:48,236 DEBUG
> > > > [storage.allocator.AbstractStoragePoolAllocator] (secstorage-1:null)
> > > > StoragePool is in avoid set, skipping this pool
> > > >
> > > >
> > > > On Fri, Jul 12, 2013 at 01:16:53PM +0800, Indra Pramana wrote:
> > > > > Dear Wido and all,
> > > > >
> > > > > I have managed to get the hosts, primary and secondary storage
> > running.
> > > > >
> > > > > - 2 KVM hypervisor hosts
> > > > > - One RBD primary storage
> > > > > - One NFS primary storage (for system VMs, since I understand that
> > system
> > > > > VMs cannot use RBD)
> > > > > - One NFS secondary storage
> > > > >
> > > > > However, now I am having problem with the system VMs: CPVM and SSVM,
> > > > unable
> > > > > to start.
> > > > >
> > > > > Excerpt from management-server.log file is here:
> > > > > http://pastebin.com/ENkpCALY
> > > > >
> > > > > It seems that the VMs were not able to be created because unable to
> > find
> > > > > suitable StoragePools.
> > > > >
> > > > > I understand that system VMs will be using the NFS primary storage
> > > > instead
> > > > > of RBD, so I have confirmed that I am able to mount the primary
> > storage
> > > > via
> > > > > NFS and have read and write access, from both the hypervisor and the
> > > > > management server.
> > > > >
> > > > > Any advise how can I resolve the problem to make both the system VMs
> > > > > created and started?
> > > > >
> > > > > Looking forward to your reply, thank you.
> > > > >
> > > > > Cheers.
> > > > >
> > > > >
> > > > > On Fri, Jul 12, 2013 at 9:43 AM, Indra Pramana <in...@sg.or.id>
> > wrote:
> > > > >
> > > > > > Hi Wido,
> > > > > >
> > > > > > Thanks for the advice, I'm now able to add the RBD pool as primary
> > > > storage.
> > > > > >
> > > > > > Many thanks! :)
> > > > > >
> > > > > > Cheers.
> > > > > >
> > > > > >
> > > > > > On Thursday, July 11, 2013, Wido den Hollander wrote:
> > > > > >
> > > > > >> Hi,
> > > > > >>
> > > > > >> On 07/10/2013 03:42 PM, Chip Childers wrote:
> > > > > >>
> > > > > >>> Cc'ing Wido, our resident Ceph expert. ;-)
> > > > > >>>
> > > > > >>>
> > > > > >> Hehe ;)
> > > > > >>
> > > > > >> On Wed, Jul 10, 2013 at 05:45:25PM +0800, Indra Pramana wrote:
> > > > > >>>
> > > > > >>>> Dear all,
> > > > > >>>>
> > > > > >>>> I am installing CloudStack 4.1.0 (upgraded from 4.0.2) and I
> > also
> > > > have a
> > > > > >>>> Ceph cluster running. However, I am having issues in adding the
> > RBD
> > > > as
> > > > > >>>> primary storage. Tried to follow the instruction here, but
> > unable to
> > > > > >>>> make
> > > > > >>>> it work:
> > > > > >>>>
> > > > > >>>> http://ceph.com/docs/master/**rbd/rbd-cloudstack/<
> > > > http://ceph.com/docs/master/rbd/rbd-cloudstack/>
> > > > > >>>>
> > > > > >>>> I have setup a pool on the Ceph cluster. The status of the
> > cluster
> > > > is
> > > > > >>>> healthy. Since I am using Ubuntu 12.04.2 LTS (Precise) for the
> > > > > >>>> hypervisors,
> > > > > >>>> I also have compiled libvirt manually to ensure that the version
> > > > 0.9.13
> > > > > >>>> is
> > > > > >>>> installed (previously it's 0.9.8).
> > > > > >>>>
> > > > > >>>>
> > > > > >> You can also use the Ubuntu Cloud Archive, I still need to get the
> > > > docs
> > > > > >> updated for that.
> > > > > >>
> > > > > >> I described the process in a blogpost:
> > > > http://blog.widodh.nl/2013/06/**
> > > > > >> a-quick-note-on-running-**cloudstack-with-rbd-on-ubuntu-**12-04/<
> > > >
> > http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/
> > > > >
> > > > > >>
> > > > > >> indra@hv-kvm-01:~/rbd$ ceph
> > > > > >>>> ceph> health
> > > > > >>>> HEALTH_OK
> > > > > >>>>
> > > > > >>>> indra@hv-kvm-01:~$ ceph osd lspools
> > > > > >>>> 0 data,1 metadata,2 rbd,3 sc1,
> > > > > >>>>
> > > > > >>>> root@hv-kvm-01:/home/indra# libvirtd --version
> > > > > >>>> libvirtd (libvirt) 0.9.13
> > > > > >>>>
> > > > > >>>> I tried to add Primary Storage into the Cloudstack zone which I
> > have
> > > > > >>>> created:
> > > > > >>>>
> > > > > >>>> Add Primary Storage:
> > > > > >>>>
> > > > > >>>> Zone: my zone name
> > > > > >>>> Pod: my pod name
> > > > > >>>> Cluster: my cluster name
> > > > > >>>> Name: ceph-rbd-pri-storage
> > > > > >>>> Protocol: RBD
> > > > > >>>> RADOS Monitor: my first Ceph monitor IP address
> > > > > >>>> RADOS Pool: sc1 (the pool name on Ceph cluster)
> > > > > >>>> RADOS User: client.admin
> > > > > >>>> RADOS Secret: /etc/ceph/ceph.client.admin.**keyring (keyring
> > file
> > > > > >>>> location)
> > > > > >>>>
> > > > > >>>
> > > > > >> This is your problem. That shouldn't be the location of the file,
> > but
> > > > it
> > > > > >> should be the secret, which is a base64 encoded string.
> > > > > >>
> > > > > >> $ ceph auth list
> > > > > >>
> > > > > >> That should tell you what the secret is.
> > > > > >>
> > > > > >> Storage Tags: rbd
> > > > > >>
> > > > > >> This is the error message when I tried to add the primary storage
> > by
> > > > > >> clicking OK:
> > > > > >>
> > > > > >> DB Exception on: com.mysql.jdbc.**JDBC4PreparedStatement@4b2eb56
> > **:
> > > > > >> INSERT INTO
> > > > > >> storage_pool (storage_pool.id, storage_pool.name,
> > storage_pool.uuid,
> > > > > >> storage_pool.pool_type, storage_pool.created,
> > > > storage_pool.update_time,
> > > > > >> storage_pool.data_center_id, storage_pool.pod_id,
> > > > > >> storage_pool.available_bytes, storage_pool.capacity_bytes,
> > > > > >> storage_pool.status, storage_pool.scope,
> > > > storage_pool.storage_provider_**
> > > > > >> id,
> > > > > >> storage_pool.host_address, storage_pool.path, storage_pool.port,
> > > > > >> storage_pool.user_info, storage_pool.cluster_id) VALUES (217,
> > > > > >> _binary'ceph-rbd-pri-storage',
> > > > > >> _binary'a226c9a1-da78-3f3a-**b5ac-e18b925c9634', 'RBD',
> > '2013-07-10
> > > > > >> 09:08:28', null, 2, 2, 0, 0, 'Up', null, null, null, _binary'ceph/
> > > > > >> ceph.client.admin.keyring@10.**237.11.2/sc1<
> > > > http://ceph.client.admin.keyring@10.237.11.2/sc1>',
> > > > > >> 6789, null, 2)
> > > > > >>
> > > > > >> On the management-server.log file:
> > > > > >>
> > > > > >> 2013-07-10 17:08:28,845 DEBUG [cloud.api.ApiServlet]
> > > > > >> (catalina-exec-2:null)
> > > > > >> ===START=== 192.168.0.100 -- GET
> > > > > >> command=createStoragePool&**zoneid=c116950e-e4ae-4f23-**a7e7-
> > > > > >> 74a75c4ee638&podId=a748b063-**3a83-4175-a0e9-de39118fe5ce&**
> > > > > >> clusterid=1f87eb09-324d-4d49-**83c2-88d84d7a15df&name=ceph-**
> > > > > >> rbd-pri-storage&url=rbd%3A%2F%**2Fclient.admin%3A_etc%2Fc
> > > > > >> eph%2Fceph.client.admin.**keyring%4010.237.11.2%2Fsc1&**
> > > > > >>
> > tags=rbd&response=json&**sessionkey=**rDRfWpqeKfQKbKZtHr398ULV%2F8k%**
> > > > > >> 3D&_=1373447307839
> > > > > >> 2013-07-10 17:08:28,862 DEBUG [cloud.storage.**StorageManagerImpl]
> > > > > >> (catalina-exec-2:null) createPool Params @ scheme - rbd
> > storageHost -
> > > > null
> > > > > >> hostPath - /ceph/ceph.client
> > > > > >> .admin.keyring@10.237.11.2/sc1 port - -1
> > > > > >> 2013-07-10 17:08:28,918 DEBUG [cloud.storage.**StorageManagerImpl]
> > > > > >> (catalina-exec-2:null) In createPool Setting poolId - 217 uuid -
> > > > > >> a226c9a1-da78-3f3a-b5ac-**e18b925c9634 z
> > > > > >> oneId - 2 podId - 2 poolName - ceph-rbd-pri-storage
> > > > > >> 2013-07-10 17:08:28,921 DEBUG [db.Transaction.Transaction]
> > > > > >> (catalina-exec-2:null) Rolling back the transaction: Time = 3
> > Name =
> > > > > >> persist; called by -Transaction.rollbac
> > > > > >> k:890-Transaction.removeUpTo:**833-Transaction.close:657-**
> > > > > >> TransactionContextBuilder.**interceptException:63-**
> > > > > >>
> > ComponentInstantiationPostProc**essor$InterceptorDispatcher.**interce
> > > > > >> pt:133-StorageManagerImpl.**createPool:1378-**
> > > > > >> StorageManagerImpl.createPool:**147-CreateStoragePoolCmd.**
> > > > > >> execute:123-ApiDispatcher.**dispatch:162-ApiServer.**
> > > > > >> queueCommand:505-ApiSe
> > > > > >> rver.handleRequest:355-**ApiServlet.processRequest:302
> > > > > >> 2013-07-10 17:08:28,923 ERROR [cloud.api.ApiServer]
> > > > (catalina-exec-2:null)
> > > > > >> unhandled exception executing api command: createStoragePool
> > > > > >> com.cloud.utils.exception.**CloudRuntimeException: DB Exception
> > on:
> > > > > >> com.mysql.jdbc.**JDBC4PreparedStatement@4b2eb56**: INSERT INTO
> > > > > >> storage_pool (
> > > > > >> storage_pool.id, storage_pool
> > > > > >> .name, storage_pool.uuid, storage_pool.pool_type,
> > > > storage_pool.created,
> > > > > >> storage_pool.update_time, storage_pool.data_center_id,
> > > > > >> storage_pool.pod_id,
> > > > > >> storage_pool.availab
> > > > > >> le_bytes, storage_pool.capacity_bytes, storage_pool.status,
> > > > > >> storage_pool.scope, storage_pool.storage_provider_**id,
> > > > > >> storage_pool.host_address, storage_pool.path, storage_
> > > > > >> pool.port, storage_pool.user_info, storage_pool.cluster_id) VALUES
> > > > (217,
> > > > > >> _binary'ceph-rbd-pri-storage',
> > > > > >> _binary'a226c9a1-da78-3f3a-**b5ac-e18b925c9634', 'RBD', '2013-07-1
> > > > > >> 0 09:08:28', null, 2, 2, 0, 0, 'Up', null, null, null,
> > _binary'ceph/
> > > > > >> ceph.client.admin.keyring@10.**237.11.2/sc1<
> > > > http://ceph.client.admin.keyring@10.237.11.2/sc1>',
> > > > > >> 6789, null, 2)
> > > > > >> at
> > > > > >>
> > > > com.cloud.utils.db.**GenericDaoBase.persist(**GenericDaoBase.java:1342)
> > > > > >> at
> > > > > >> com.cloud.storage.dao.**StoragePoolDaoImpl.persist(**
> > > > > >> StoragePoolDaoImpl.java:232)
> > > > > >> at
> > > > > >> com.cloud.utils.component.**ComponentInstantiationPostProc**
> > > > > >>
> > > >
> > essor$InterceptorDispatcher.**intercept(**ComponentInstantiationPostProc*
> > > > > >> *es
> > > > > >>
> > > > > >>
> > > >
> > > > --
> > > > Prasanna.,
> > > >
> > > > ------------------------
> > > > Powered by BigRock.com
> > > >
> > > >
> >
> > --
> > Prasanna.,
> >
> > ------------------------
> > Powered by BigRock.com
> >
> >
--
Prasanna.,
------------------------
Powered by BigRock.com
Re: Problem in adding Ceph RBD as primary storage for CloudStack 4.1.0
Posted by Indra Pramana <in...@sg.or.id>.
Hi Prasanna,
Good day to you, and thank you for your e-mail.
Yes, the file exists. I can access the file from the management server and
the two hypervisors hosts if I mount manually.
[root@cs-nas-01 /mnt/vol1/sec-storage/template/tmpl/1/3]# ls
-la
total
1418787
drwxr-xr-x 2 root wheel 4 Jul 11 20:21
.
drwxr-xr-x 3 root wheel 3 Jul 11 20:17
..
-rw-r--r-- 1 root wheel 725811200 Jul 11 20:21
425b9e5a-fbc7-4637-a33a-fe9d0ed4fa98.qcow2
-rw-r--r-- 1 root wheel 295 Jul 11 20:21
template.properties
[root@cs-nas-01 /mnt/vol1/sec-storage/template/tmpl/1/3]#
pwd
/mnt/vol1/sec-storage/template/tmpl/1/3
Any advise?
Looking forward to your reply, thank you.
Cheers.
On Fri, Jul 12, 2013 at 2:07 PM, Prasanna Santhanam <ts...@apache.org> wrote:
> Can you check whether there is a file at:
> nfs://103.25.200.19/mnt/vol1/sec-storage/template/tmpl/1/3/
>
> On Fri, Jul 12, 2013 at 01:59:34PM +0800, Indra Pramana wrote:
> > Hi Prasanna,
> >
> > Thanks for your e-mail.
> >
> > I have tried restarting the management server, and the problem still
> > persists. I even tried to re-do the installation and configuration again
> > from scratch last night, but the problem still there.
> >
> > I also noted that on the beginning of the logs, I found some error
> messages
> > saying that the template cannot be downloaded to the pool. See this logs:
> >
> > http://pastebin.com/BY1AVJ08
> >
> > It says it failed because cannot get volume from the pool. Could it be
> > related, i.e. the absence of the template caused the system VMs cannot be
> > created and started?
> >
> > I have ensured that I downloaded the system VM template using
> > cloud-install-sys-tmplt and verified that the template is already there
> in
> > the secondary storage server.
> >
> > Any advice is appreciated.
> >
> > Looking forward to your reply, thank you.
> >
> > Cheers.
> >
> >
> >
> > On Fri, Jul 12, 2013 at 1:21 PM, Prasanna Santhanam <ts...@apache.org>
> wrote:
> >
> > > It looks like a previous attempt to start the systemVMs has failed
> > > putting the nfs storage in the avoid set. Did you try restarting your
> > > management server?
> > >
> > > This line leads me to the above mentioned:
> > > 2013-07-12 13:10:48,236 DEBUG
> > > [storage.allocator.AbstractStoragePoolAllocator] (secstorage-1:null)
> > > StoragePool is in avoid set, skipping this pool
> > >
> > >
> > > On Fri, Jul 12, 2013 at 01:16:53PM +0800, Indra Pramana wrote:
> > > > Dear Wido and all,
> > > >
> > > > I have managed to get the hosts, primary and secondary storage
> running.
> > > >
> > > > - 2 KVM hypervisor hosts
> > > > - One RBD primary storage
> > > > - One NFS primary storage (for system VMs, since I understand that
> system
> > > > VMs cannot use RBD)
> > > > - One NFS secondary storage
> > > >
> > > > However, now I am having problem with the system VMs: CPVM and SSVM,
> > > unable
> > > > to start.
> > > >
> > > > Excerpt from management-server.log file is here:
> > > > http://pastebin.com/ENkpCALY
> > > >
> > > > It seems that the VMs were not able to be created because unable to
> find
> > > > suitable StoragePools.
> > > >
> > > > I understand that system VMs will be using the NFS primary storage
> > > instead
> > > > of RBD, so I have confirmed that I am able to mount the primary
> storage
> > > via
> > > > NFS and have read and write access, from both the hypervisor and the
> > > > management server.
> > > >
> > > > Any advise how can I resolve the problem to make both the system VMs
> > > > created and started?
> > > >
> > > > Looking forward to your reply, thank you.
> > > >
> > > > Cheers.
> > > >
> > > >
> > > > On Fri, Jul 12, 2013 at 9:43 AM, Indra Pramana <in...@sg.or.id>
> wrote:
> > > >
> > > > > Hi Wido,
> > > > >
> > > > > Thanks for the advice, I'm now able to add the RBD pool as primary
> > > storage.
> > > > >
> > > > > Many thanks! :)
> > > > >
> > > > > Cheers.
> > > > >
> > > > >
> > > > > On Thursday, July 11, 2013, Wido den Hollander wrote:
> > > > >
> > > > >> Hi,
> > > > >>
> > > > >> On 07/10/2013 03:42 PM, Chip Childers wrote:
> > > > >>
> > > > >>> Cc'ing Wido, our resident Ceph expert. ;-)
> > > > >>>
> > > > >>>
> > > > >> Hehe ;)
> > > > >>
> > > > >> On Wed, Jul 10, 2013 at 05:45:25PM +0800, Indra Pramana wrote:
> > > > >>>
> > > > >>>> Dear all,
> > > > >>>>
> > > > >>>> I am installing CloudStack 4.1.0 (upgraded from 4.0.2) and I
> also
> > > have a
> > > > >>>> Ceph cluster running. However, I am having issues in adding the
> RBD
> > > as
> > > > >>>> primary storage. Tried to follow the instruction here, but
> unable to
> > > > >>>> make
> > > > >>>> it work:
> > > > >>>>
> > > > >>>> http://ceph.com/docs/master/**rbd/rbd-cloudstack/<
> > > http://ceph.com/docs/master/rbd/rbd-cloudstack/>
> > > > >>>>
> > > > >>>> I have setup a pool on the Ceph cluster. The status of the
> cluster
> > > is
> > > > >>>> healthy. Since I am using Ubuntu 12.04.2 LTS (Precise) for the
> > > > >>>> hypervisors,
> > > > >>>> I also have compiled libvirt manually to ensure that the version
> > > 0.9.13
> > > > >>>> is
> > > > >>>> installed (previously it's 0.9.8).
> > > > >>>>
> > > > >>>>
> > > > >> You can also use the Ubuntu Cloud Archive, I still need to get the
> > > docs
> > > > >> updated for that.
> > > > >>
> > > > >> I described the process in a blogpost:
> > > http://blog.widodh.nl/2013/06/**
> > > > >> a-quick-note-on-running-**cloudstack-with-rbd-on-ubuntu-**12-04/<
> > >
> http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/
> > > >
> > > > >>
> > > > >> indra@hv-kvm-01:~/rbd$ ceph
> > > > >>>> ceph> health
> > > > >>>> HEALTH_OK
> > > > >>>>
> > > > >>>> indra@hv-kvm-01:~$ ceph osd lspools
> > > > >>>> 0 data,1 metadata,2 rbd,3 sc1,
> > > > >>>>
> > > > >>>> root@hv-kvm-01:/home/indra# libvirtd --version
> > > > >>>> libvirtd (libvirt) 0.9.13
> > > > >>>>
> > > > >>>> I tried to add Primary Storage into the Cloudstack zone which I
> have
> > > > >>>> created:
> > > > >>>>
> > > > >>>> Add Primary Storage:
> > > > >>>>
> > > > >>>> Zone: my zone name
> > > > >>>> Pod: my pod name
> > > > >>>> Cluster: my cluster name
> > > > >>>> Name: ceph-rbd-pri-storage
> > > > >>>> Protocol: RBD
> > > > >>>> RADOS Monitor: my first Ceph monitor IP address
> > > > >>>> RADOS Pool: sc1 (the pool name on Ceph cluster)
> > > > >>>> RADOS User: client.admin
> > > > >>>> RADOS Secret: /etc/ceph/ceph.client.admin.**keyring (keyring
> file
> > > > >>>> location)
> > > > >>>>
> > > > >>>
> > > > >> This is your problem. That shouldn't be the location of the file,
> but
> > > it
> > > > >> should be the secret, which is a base64 encoded string.
> > > > >>
> > > > >> $ ceph auth list
> > > > >>
> > > > >> That should tell you what the secret is.
> > > > >>
> > > > >> Storage Tags: rbd
> > > > >>
> > > > >> This is the error message when I tried to add the primary storage
> by
> > > > >> clicking OK:
> > > > >>
> > > > >> DB Exception on: com.mysql.jdbc.**JDBC4PreparedStatement@4b2eb56
> **:
> > > > >> INSERT INTO
> > > > >> storage_pool (storage_pool.id, storage_pool.name,
> storage_pool.uuid,
> > > > >> storage_pool.pool_type, storage_pool.created,
> > > storage_pool.update_time,
> > > > >> storage_pool.data_center_id, storage_pool.pod_id,
> > > > >> storage_pool.available_bytes, storage_pool.capacity_bytes,
> > > > >> storage_pool.status, storage_pool.scope,
> > > storage_pool.storage_provider_**
> > > > >> id,
> > > > >> storage_pool.host_address, storage_pool.path, storage_pool.port,
> > > > >> storage_pool.user_info, storage_pool.cluster_id) VALUES (217,
> > > > >> _binary'ceph-rbd-pri-storage',
> > > > >> _binary'a226c9a1-da78-3f3a-**b5ac-e18b925c9634', 'RBD',
> '2013-07-10
> > > > >> 09:08:28', null, 2, 2, 0, 0, 'Up', null, null, null, _binary'ceph/
> > > > >> ceph.client.admin.keyring@10.**237.11.2/sc1<
> > > http://ceph.client.admin.keyring@10.237.11.2/sc1>',
> > > > >> 6789, null, 2)
> > > > >>
> > > > >> On the management-server.log file:
> > > > >>
> > > > >> 2013-07-10 17:08:28,845 DEBUG [cloud.api.ApiServlet]
> > > > >> (catalina-exec-2:null)
> > > > >> ===START=== 192.168.0.100 -- GET
> > > > >> command=createStoragePool&**zoneid=c116950e-e4ae-4f23-**a7e7-
> > > > >> 74a75c4ee638&podId=a748b063-**3a83-4175-a0e9-de39118fe5ce&**
> > > > >> clusterid=1f87eb09-324d-4d49-**83c2-88d84d7a15df&name=ceph-**
> > > > >> rbd-pri-storage&url=rbd%3A%2F%**2Fclient.admin%3A_etc%2Fc
> > > > >> eph%2Fceph.client.admin.**keyring%4010.237.11.2%2Fsc1&**
> > > > >>
> tags=rbd&response=json&**sessionkey=**rDRfWpqeKfQKbKZtHr398ULV%2F8k%**
> > > > >> 3D&_=1373447307839
> > > > >> 2013-07-10 17:08:28,862 DEBUG [cloud.storage.**StorageManagerImpl]
> > > > >> (catalina-exec-2:null) createPool Params @ scheme - rbd
> storageHost -
> > > null
> > > > >> hostPath - /ceph/ceph.client
> > > > >> .admin.keyring@10.237.11.2/sc1 port - -1
> > > > >> 2013-07-10 17:08:28,918 DEBUG [cloud.storage.**StorageManagerImpl]
> > > > >> (catalina-exec-2:null) In createPool Setting poolId - 217 uuid -
> > > > >> a226c9a1-da78-3f3a-b5ac-**e18b925c9634 z
> > > > >> oneId - 2 podId - 2 poolName - ceph-rbd-pri-storage
> > > > >> 2013-07-10 17:08:28,921 DEBUG [db.Transaction.Transaction]
> > > > >> (catalina-exec-2:null) Rolling back the transaction: Time = 3
> Name =
> > > > >> persist; called by -Transaction.rollbac
> > > > >> k:890-Transaction.removeUpTo:**833-Transaction.close:657-**
> > > > >> TransactionContextBuilder.**interceptException:63-**
> > > > >>
> ComponentInstantiationPostProc**essor$InterceptorDispatcher.**interce
> > > > >> pt:133-StorageManagerImpl.**createPool:1378-**
> > > > >> StorageManagerImpl.createPool:**147-CreateStoragePoolCmd.**
> > > > >> execute:123-ApiDispatcher.**dispatch:162-ApiServer.**
> > > > >> queueCommand:505-ApiSe
> > > > >> rver.handleRequest:355-**ApiServlet.processRequest:302
> > > > >> 2013-07-10 17:08:28,923 ERROR [cloud.api.ApiServer]
> > > (catalina-exec-2:null)
> > > > >> unhandled exception executing api command: createStoragePool
> > > > >> com.cloud.utils.exception.**CloudRuntimeException: DB Exception
> on:
> > > > >> com.mysql.jdbc.**JDBC4PreparedStatement@4b2eb56**: INSERT INTO
> > > > >> storage_pool (
> > > > >> storage_pool.id, storage_pool
> > > > >> .name, storage_pool.uuid, storage_pool.pool_type,
> > > storage_pool.created,
> > > > >> storage_pool.update_time, storage_pool.data_center_id,
> > > > >> storage_pool.pod_id,
> > > > >> storage_pool.availab
> > > > >> le_bytes, storage_pool.capacity_bytes, storage_pool.status,
> > > > >> storage_pool.scope, storage_pool.storage_provider_**id,
> > > > >> storage_pool.host_address, storage_pool.path, storage_
> > > > >> pool.port, storage_pool.user_info, storage_pool.cluster_id) VALUES
> > > (217,
> > > > >> _binary'ceph-rbd-pri-storage',
> > > > >> _binary'a226c9a1-da78-3f3a-**b5ac-e18b925c9634', 'RBD', '2013-07-1
> > > > >> 0 09:08:28', null, 2, 2, 0, 0, 'Up', null, null, null,
> _binary'ceph/
> > > > >> ceph.client.admin.keyring@10.**237.11.2/sc1<
> > > http://ceph.client.admin.keyring@10.237.11.2/sc1>',
> > > > >> 6789, null, 2)
> > > > >> at
> > > > >>
> > > com.cloud.utils.db.**GenericDaoBase.persist(**GenericDaoBase.java:1342)
> > > > >> at
> > > > >> com.cloud.storage.dao.**StoragePoolDaoImpl.persist(**
> > > > >> StoragePoolDaoImpl.java:232)
> > > > >> at
> > > > >> com.cloud.utils.component.**ComponentInstantiationPostProc**
> > > > >>
> > >
> essor$InterceptorDispatcher.**intercept(**ComponentInstantiationPostProc*
> > > > >> *es
> > > > >>
> > > > >>
> > >
> > > --
> > > Prasanna.,
> > >
> > > ------------------------
> > > Powered by BigRock.com
> > >
> > >
>
> --
> Prasanna.,
>
> ------------------------
> Powered by BigRock.com
>
>
Re: Problem in adding Ceph RBD as primary storage for CloudStack
4.1.0
Posted by Prasanna Santhanam <ts...@apache.org>.
Can you check whether there is a file at:
nfs://103.25.200.19/mnt/vol1/sec-storage/template/tmpl/1/3/
On Fri, Jul 12, 2013 at 01:59:34PM +0800, Indra Pramana wrote:
> Hi Prasanna,
>
> Thanks for your e-mail.
>
> I have tried restarting the management server, and the problem still
> persists. I even tried to re-do the installation and configuration again
> from scratch last night, but the problem still there.
>
> I also noted that on the beginning of the logs, I found some error messages
> saying that the template cannot be downloaded to the pool. See this logs:
>
> http://pastebin.com/BY1AVJ08
>
> It says it failed because cannot get volume from the pool. Could it be
> related, i.e. the absence of the template caused the system VMs cannot be
> created and started?
>
> I have ensured that I downloaded the system VM template using
> cloud-install-sys-tmplt and verified that the template is already there in
> the secondary storage server.
>
> Any advice is appreciated.
>
> Looking forward to your reply, thank you.
>
> Cheers.
>
>
>
> On Fri, Jul 12, 2013 at 1:21 PM, Prasanna Santhanam <ts...@apache.org> wrote:
>
> > It looks like a previous attempt to start the systemVMs has failed
> > putting the nfs storage in the avoid set. Did you try restarting your
> > management server?
> >
> > This line leads me to the above mentioned:
> > 2013-07-12 13:10:48,236 DEBUG
> > [storage.allocator.AbstractStoragePoolAllocator] (secstorage-1:null)
> > StoragePool is in avoid set, skipping this pool
> >
> >
> > On Fri, Jul 12, 2013 at 01:16:53PM +0800, Indra Pramana wrote:
> > > Dear Wido and all,
> > >
> > > I have managed to get the hosts, primary and secondary storage running.
> > >
> > > - 2 KVM hypervisor hosts
> > > - One RBD primary storage
> > > - One NFS primary storage (for system VMs, since I understand that system
> > > VMs cannot use RBD)
> > > - One NFS secondary storage
> > >
> > > However, now I am having problem with the system VMs: CPVM and SSVM,
> > unable
> > > to start.
> > >
> > > Excerpt from management-server.log file is here:
> > > http://pastebin.com/ENkpCALY
> > >
> > > It seems that the VMs were not able to be created because unable to find
> > > suitable StoragePools.
> > >
> > > I understand that system VMs will be using the NFS primary storage
> > instead
> > > of RBD, so I have confirmed that I am able to mount the primary storage
> > via
> > > NFS and have read and write access, from both the hypervisor and the
> > > management server.
> > >
> > > Any advise how can I resolve the problem to make both the system VMs
> > > created and started?
> > >
> > > Looking forward to your reply, thank you.
> > >
> > > Cheers.
> > >
> > >
> > > On Fri, Jul 12, 2013 at 9:43 AM, Indra Pramana <in...@sg.or.id> wrote:
> > >
> > > > Hi Wido,
> > > >
> > > > Thanks for the advice, I'm now able to add the RBD pool as primary
> > storage.
> > > >
> > > > Many thanks! :)
> > > >
> > > > Cheers.
> > > >
> > > >
> > > > On Thursday, July 11, 2013, Wido den Hollander wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> On 07/10/2013 03:42 PM, Chip Childers wrote:
> > > >>
> > > >>> Cc'ing Wido, our resident Ceph expert. ;-)
> > > >>>
> > > >>>
> > > >> Hehe ;)
> > > >>
> > > >> On Wed, Jul 10, 2013 at 05:45:25PM +0800, Indra Pramana wrote:
> > > >>>
> > > >>>> Dear all,
> > > >>>>
> > > >>>> I am installing CloudStack 4.1.0 (upgraded from 4.0.2) and I also
> > have a
> > > >>>> Ceph cluster running. However, I am having issues in adding the RBD
> > as
> > > >>>> primary storage. Tried to follow the instruction here, but unable to
> > > >>>> make
> > > >>>> it work:
> > > >>>>
> > > >>>> http://ceph.com/docs/master/**rbd/rbd-cloudstack/<
> > http://ceph.com/docs/master/rbd/rbd-cloudstack/>
> > > >>>>
> > > >>>> I have setup a pool on the Ceph cluster. The status of the cluster
> > is
> > > >>>> healthy. Since I am using Ubuntu 12.04.2 LTS (Precise) for the
> > > >>>> hypervisors,
> > > >>>> I also have compiled libvirt manually to ensure that the version
> > 0.9.13
> > > >>>> is
> > > >>>> installed (previously it's 0.9.8).
> > > >>>>
> > > >>>>
> > > >> You can also use the Ubuntu Cloud Archive, I still need to get the
> > docs
> > > >> updated for that.
> > > >>
> > > >> I described the process in a blogpost:
> > http://blog.widodh.nl/2013/06/**
> > > >> a-quick-note-on-running-**cloudstack-with-rbd-on-ubuntu-**12-04/<
> > http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/
> > >
> > > >>
> > > >> indra@hv-kvm-01:~/rbd$ ceph
> > > >>>> ceph> health
> > > >>>> HEALTH_OK
> > > >>>>
> > > >>>> indra@hv-kvm-01:~$ ceph osd lspools
> > > >>>> 0 data,1 metadata,2 rbd,3 sc1,
> > > >>>>
> > > >>>> root@hv-kvm-01:/home/indra# libvirtd --version
> > > >>>> libvirtd (libvirt) 0.9.13
> > > >>>>
> > > >>>> I tried to add Primary Storage into the Cloudstack zone which I have
> > > >>>> created:
> > > >>>>
> > > >>>> Add Primary Storage:
> > > >>>>
> > > >>>> Zone: my zone name
> > > >>>> Pod: my pod name
> > > >>>> Cluster: my cluster name
> > > >>>> Name: ceph-rbd-pri-storage
> > > >>>> Protocol: RBD
> > > >>>> RADOS Monitor: my first Ceph monitor IP address
> > > >>>> RADOS Pool: sc1 (the pool name on Ceph cluster)
> > > >>>> RADOS User: client.admin
> > > >>>> RADOS Secret: /etc/ceph/ceph.client.admin.**keyring (keyring file
> > > >>>> location)
> > > >>>>
> > > >>>
> > > >> This is your problem. That shouldn't be the location of the file, but
> > it
> > > >> should be the secret, which is a base64 encoded string.
> > > >>
> > > >> $ ceph auth list
> > > >>
> > > >> That should tell you what the secret is.
> > > >>
> > > >> Storage Tags: rbd
> > > >>
> > > >> This is the error message when I tried to add the primary storage by
> > > >> clicking OK:
> > > >>
> > > >> DB Exception on: com.mysql.jdbc.**JDBC4PreparedStatement@4b2eb56**:
> > > >> INSERT INTO
> > > >> storage_pool (storage_pool.id, storage_pool.name, storage_pool.uuid,
> > > >> storage_pool.pool_type, storage_pool.created,
> > storage_pool.update_time,
> > > >> storage_pool.data_center_id, storage_pool.pod_id,
> > > >> storage_pool.available_bytes, storage_pool.capacity_bytes,
> > > >> storage_pool.status, storage_pool.scope,
> > storage_pool.storage_provider_**
> > > >> id,
> > > >> storage_pool.host_address, storage_pool.path, storage_pool.port,
> > > >> storage_pool.user_info, storage_pool.cluster_id) VALUES (217,
> > > >> _binary'ceph-rbd-pri-storage',
> > > >> _binary'a226c9a1-da78-3f3a-**b5ac-e18b925c9634', 'RBD', '2013-07-10
> > > >> 09:08:28', null, 2, 2, 0, 0, 'Up', null, null, null, _binary'ceph/
> > > >> ceph.client.admin.keyring@10.**237.11.2/sc1<
> > http://ceph.client.admin.keyring@10.237.11.2/sc1>',
> > > >> 6789, null, 2)
> > > >>
> > > >> On the management-server.log file:
> > > >>
> > > >> 2013-07-10 17:08:28,845 DEBUG [cloud.api.ApiServlet]
> > > >> (catalina-exec-2:null)
> > > >> ===START=== 192.168.0.100 -- GET
> > > >> command=createStoragePool&**zoneid=c116950e-e4ae-4f23-**a7e7-
> > > >> 74a75c4ee638&podId=a748b063-**3a83-4175-a0e9-de39118fe5ce&**
> > > >> clusterid=1f87eb09-324d-4d49-**83c2-88d84d7a15df&name=ceph-**
> > > >> rbd-pri-storage&url=rbd%3A%2F%**2Fclient.admin%3A_etc%2Fc
> > > >> eph%2Fceph.client.admin.**keyring%4010.237.11.2%2Fsc1&**
> > > >> tags=rbd&response=json&**sessionkey=**rDRfWpqeKfQKbKZtHr398ULV%2F8k%**
> > > >> 3D&_=1373447307839
> > > >> 2013-07-10 17:08:28,862 DEBUG [cloud.storage.**StorageManagerImpl]
> > > >> (catalina-exec-2:null) createPool Params @ scheme - rbd storageHost -
> > null
> > > >> hostPath - /ceph/ceph.client
> > > >> .admin.keyring@10.237.11.2/sc1 port - -1
> > > >> 2013-07-10 17:08:28,918 DEBUG [cloud.storage.**StorageManagerImpl]
> > > >> (catalina-exec-2:null) In createPool Setting poolId - 217 uuid -
> > > >> a226c9a1-da78-3f3a-b5ac-**e18b925c9634 z
> > > >> oneId - 2 podId - 2 poolName - ceph-rbd-pri-storage
> > > >> 2013-07-10 17:08:28,921 DEBUG [db.Transaction.Transaction]
> > > >> (catalina-exec-2:null) Rolling back the transaction: Time = 3 Name =
> > > >> persist; called by -Transaction.rollbac
> > > >> k:890-Transaction.removeUpTo:**833-Transaction.close:657-**
> > > >> TransactionContextBuilder.**interceptException:63-**
> > > >> ComponentInstantiationPostProc**essor$InterceptorDispatcher.**interce
> > > >> pt:133-StorageManagerImpl.**createPool:1378-**
> > > >> StorageManagerImpl.createPool:**147-CreateStoragePoolCmd.**
> > > >> execute:123-ApiDispatcher.**dispatch:162-ApiServer.**
> > > >> queueCommand:505-ApiSe
> > > >> rver.handleRequest:355-**ApiServlet.processRequest:302
> > > >> 2013-07-10 17:08:28,923 ERROR [cloud.api.ApiServer]
> > (catalina-exec-2:null)
> > > >> unhandled exception executing api command: createStoragePool
> > > >> com.cloud.utils.exception.**CloudRuntimeException: DB Exception on:
> > > >> com.mysql.jdbc.**JDBC4PreparedStatement@4b2eb56**: INSERT INTO
> > > >> storage_pool (
> > > >> storage_pool.id, storage_pool
> > > >> .name, storage_pool.uuid, storage_pool.pool_type,
> > storage_pool.created,
> > > >> storage_pool.update_time, storage_pool.data_center_id,
> > > >> storage_pool.pod_id,
> > > >> storage_pool.availab
> > > >> le_bytes, storage_pool.capacity_bytes, storage_pool.status,
> > > >> storage_pool.scope, storage_pool.storage_provider_**id,
> > > >> storage_pool.host_address, storage_pool.path, storage_
> > > >> pool.port, storage_pool.user_info, storage_pool.cluster_id) VALUES
> > (217,
> > > >> _binary'ceph-rbd-pri-storage',
> > > >> _binary'a226c9a1-da78-3f3a-**b5ac-e18b925c9634', 'RBD', '2013-07-1
> > > >> 0 09:08:28', null, 2, 2, 0, 0, 'Up', null, null, null, _binary'ceph/
> > > >> ceph.client.admin.keyring@10.**237.11.2/sc1<
> > http://ceph.client.admin.keyring@10.237.11.2/sc1>',
> > > >> 6789, null, 2)
> > > >> at
> > > >>
> > com.cloud.utils.db.**GenericDaoBase.persist(**GenericDaoBase.java:1342)
> > > >> at
> > > >> com.cloud.storage.dao.**StoragePoolDaoImpl.persist(**
> > > >> StoragePoolDaoImpl.java:232)
> > > >> at
> > > >> com.cloud.utils.component.**ComponentInstantiationPostProc**
> > > >>
> > essor$InterceptorDispatcher.**intercept(**ComponentInstantiationPostProc*
> > > >> *es
> > > >>
> > > >>
> >
> > --
> > Prasanna.,
> >
> > ------------------------
> > Powered by BigRock.com
> >
> >
--
Prasanna.,
------------------------
Powered by BigRock.com
Re: Problem in adding Ceph RBD as primary storage for CloudStack 4.1.0
Posted by Indra Pramana <in...@sg.or.id>.
Hi Prasanna,
Thanks for your e-mail.
I have tried restarting the management server, and the problem still
persists. I even tried to re-do the installation and configuration again
from scratch last night, but the problem still there.
I also noted that on the beginning of the logs, I found some error messages
saying that the template cannot be downloaded to the pool. See this logs:
http://pastebin.com/BY1AVJ08
It says it failed because cannot get volume from the pool. Could it be
related, i.e. the absence of the template caused the system VMs cannot be
created and started?
I have ensured that I downloaded the system VM template using
cloud-install-sys-tmplt and verified that the template is already there in
the secondary storage server.
Any advice is appreciated.
Looking forward to your reply, thank you.
Cheers.
On Fri, Jul 12, 2013 at 1:21 PM, Prasanna Santhanam <ts...@apache.org> wrote:
> It looks like a previous attempt to start the systemVMs has failed
> putting the nfs storage in the avoid set. Did you try restarting your
> management server?
>
> This line leads me to the above mentioned:
> 2013-07-12 13:10:48,236 DEBUG
> [storage.allocator.AbstractStoragePoolAllocator] (secstorage-1:null)
> StoragePool is in avoid set, skipping this pool
>
>
> On Fri, Jul 12, 2013 at 01:16:53PM +0800, Indra Pramana wrote:
> > Dear Wido and all,
> >
> > I have managed to get the hosts, primary and secondary storage running.
> >
> > - 2 KVM hypervisor hosts
> > - One RBD primary storage
> > - One NFS primary storage (for system VMs, since I understand that system
> > VMs cannot use RBD)
> > - One NFS secondary storage
> >
> > However, now I am having problem with the system VMs: CPVM and SSVM,
> unable
> > to start.
> >
> > Excerpt from management-server.log file is here:
> > http://pastebin.com/ENkpCALY
> >
> > It seems that the VMs were not able to be created because unable to find
> > suitable StoragePools.
> >
> > I understand that system VMs will be using the NFS primary storage
> instead
> > of RBD, so I have confirmed that I am able to mount the primary storage
> via
> > NFS and have read and write access, from both the hypervisor and the
> > management server.
> >
> > Any advise how can I resolve the problem to make both the system VMs
> > created and started?
> >
> > Looking forward to your reply, thank you.
> >
> > Cheers.
> >
> >
> > On Fri, Jul 12, 2013 at 9:43 AM, Indra Pramana <in...@sg.or.id> wrote:
> >
> > > Hi Wido,
> > >
> > > Thanks for the advice, I'm now able to add the RBD pool as primary
> storage.
> > >
> > > Many thanks! :)
> > >
> > > Cheers.
> > >
> > >
> > > On Thursday, July 11, 2013, Wido den Hollander wrote:
> > >
> > >> Hi,
> > >>
> > >> On 07/10/2013 03:42 PM, Chip Childers wrote:
> > >>
> > >>> Cc'ing Wido, our resident Ceph expert. ;-)
> > >>>
> > >>>
> > >> Hehe ;)
> > >>
> > >> On Wed, Jul 10, 2013 at 05:45:25PM +0800, Indra Pramana wrote:
> > >>>
> > >>>> Dear all,
> > >>>>
> > >>>> I am installing CloudStack 4.1.0 (upgraded from 4.0.2) and I also
> have a
> > >>>> Ceph cluster running. However, I am having issues in adding the RBD
> as
> > >>>> primary storage. Tried to follow the instruction here, but unable to
> > >>>> make
> > >>>> it work:
> > >>>>
> > >>>> http://ceph.com/docs/master/**rbd/rbd-cloudstack/<
> http://ceph.com/docs/master/rbd/rbd-cloudstack/>
> > >>>>
> > >>>> I have setup a pool on the Ceph cluster. The status of the cluster
> is
> > >>>> healthy. Since I am using Ubuntu 12.04.2 LTS (Precise) for the
> > >>>> hypervisors,
> > >>>> I also have compiled libvirt manually to ensure that the version
> 0.9.13
> > >>>> is
> > >>>> installed (previously it's 0.9.8).
> > >>>>
> > >>>>
> > >> You can also use the Ubuntu Cloud Archive, I still need to get the
> docs
> > >> updated for that.
> > >>
> > >> I described the process in a blogpost:
> http://blog.widodh.nl/2013/06/**
> > >> a-quick-note-on-running-**cloudstack-with-rbd-on-ubuntu-**12-04/<
> http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/
> >
> > >>
> > >> indra@hv-kvm-01:~/rbd$ ceph
> > >>>> ceph> health
> > >>>> HEALTH_OK
> > >>>>
> > >>>> indra@hv-kvm-01:~$ ceph osd lspools
> > >>>> 0 data,1 metadata,2 rbd,3 sc1,
> > >>>>
> > >>>> root@hv-kvm-01:/home/indra# libvirtd --version
> > >>>> libvirtd (libvirt) 0.9.13
> > >>>>
> > >>>> I tried to add Primary Storage into the Cloudstack zone which I have
> > >>>> created:
> > >>>>
> > >>>> Add Primary Storage:
> > >>>>
> > >>>> Zone: my zone name
> > >>>> Pod: my pod name
> > >>>> Cluster: my cluster name
> > >>>> Name: ceph-rbd-pri-storage
> > >>>> Protocol: RBD
> > >>>> RADOS Monitor: my first Ceph monitor IP address
> > >>>> RADOS Pool: sc1 (the pool name on Ceph cluster)
> > >>>> RADOS User: client.admin
> > >>>> RADOS Secret: /etc/ceph/ceph.client.admin.**keyring (keyring file
> > >>>> location)
> > >>>>
> > >>>
> > >> This is your problem. That shouldn't be the location of the file, but
> it
> > >> should be the secret, which is a base64 encoded string.
> > >>
> > >> $ ceph auth list
> > >>
> > >> That should tell you what the secret is.
> > >>
> > >> Storage Tags: rbd
> > >>
> > >> This is the error message when I tried to add the primary storage by
> > >> clicking OK:
> > >>
> > >> DB Exception on: com.mysql.jdbc.**JDBC4PreparedStatement@4b2eb56**:
> > >> INSERT INTO
> > >> storage_pool (storage_pool.id, storage_pool.name, storage_pool.uuid,
> > >> storage_pool.pool_type, storage_pool.created,
> storage_pool.update_time,
> > >> storage_pool.data_center_id, storage_pool.pod_id,
> > >> storage_pool.available_bytes, storage_pool.capacity_bytes,
> > >> storage_pool.status, storage_pool.scope,
> storage_pool.storage_provider_**
> > >> id,
> > >> storage_pool.host_address, storage_pool.path, storage_pool.port,
> > >> storage_pool.user_info, storage_pool.cluster_id) VALUES (217,
> > >> _binary'ceph-rbd-pri-storage',
> > >> _binary'a226c9a1-da78-3f3a-**b5ac-e18b925c9634', 'RBD', '2013-07-10
> > >> 09:08:28', null, 2, 2, 0, 0, 'Up', null, null, null, _binary'ceph/
> > >> ceph.client.admin.keyring@10.**237.11.2/sc1<
> http://ceph.client.admin.keyring@10.237.11.2/sc1>',
> > >> 6789, null, 2)
> > >>
> > >> On the management-server.log file:
> > >>
> > >> 2013-07-10 17:08:28,845 DEBUG [cloud.api.ApiServlet]
> > >> (catalina-exec-2:null)
> > >> ===START=== 192.168.0.100 -- GET
> > >> command=createStoragePool&**zoneid=c116950e-e4ae-4f23-**a7e7-
> > >> 74a75c4ee638&podId=a748b063-**3a83-4175-a0e9-de39118fe5ce&**
> > >> clusterid=1f87eb09-324d-4d49-**83c2-88d84d7a15df&name=ceph-**
> > >> rbd-pri-storage&url=rbd%3A%2F%**2Fclient.admin%3A_etc%2Fc
> > >> eph%2Fceph.client.admin.**keyring%4010.237.11.2%2Fsc1&**
> > >> tags=rbd&response=json&**sessionkey=**rDRfWpqeKfQKbKZtHr398ULV%2F8k%**
> > >> 3D&_=1373447307839
> > >> 2013-07-10 17:08:28,862 DEBUG [cloud.storage.**StorageManagerImpl]
> > >> (catalina-exec-2:null) createPool Params @ scheme - rbd storageHost -
> null
> > >> hostPath - /ceph/ceph.client
> > >> .admin.keyring@10.237.11.2/sc1 port - -1
> > >> 2013-07-10 17:08:28,918 DEBUG [cloud.storage.**StorageManagerImpl]
> > >> (catalina-exec-2:null) In createPool Setting poolId - 217 uuid -
> > >> a226c9a1-da78-3f3a-b5ac-**e18b925c9634 z
> > >> oneId - 2 podId - 2 poolName - ceph-rbd-pri-storage
> > >> 2013-07-10 17:08:28,921 DEBUG [db.Transaction.Transaction]
> > >> (catalina-exec-2:null) Rolling back the transaction: Time = 3 Name =
> > >> persist; called by -Transaction.rollbac
> > >> k:890-Transaction.removeUpTo:**833-Transaction.close:657-**
> > >> TransactionContextBuilder.**interceptException:63-**
> > >> ComponentInstantiationPostProc**essor$InterceptorDispatcher.**interce
> > >> pt:133-StorageManagerImpl.**createPool:1378-**
> > >> StorageManagerImpl.createPool:**147-CreateStoragePoolCmd.**
> > >> execute:123-ApiDispatcher.**dispatch:162-ApiServer.**
> > >> queueCommand:505-ApiSe
> > >> rver.handleRequest:355-**ApiServlet.processRequest:302
> > >> 2013-07-10 17:08:28,923 ERROR [cloud.api.ApiServer]
> (catalina-exec-2:null)
> > >> unhandled exception executing api command: createStoragePool
> > >> com.cloud.utils.exception.**CloudRuntimeException: DB Exception on:
> > >> com.mysql.jdbc.**JDBC4PreparedStatement@4b2eb56**: INSERT INTO
> > >> storage_pool (
> > >> storage_pool.id, storage_pool
> > >> .name, storage_pool.uuid, storage_pool.pool_type,
> storage_pool.created,
> > >> storage_pool.update_time, storage_pool.data_center_id,
> > >> storage_pool.pod_id,
> > >> storage_pool.availab
> > >> le_bytes, storage_pool.capacity_bytes, storage_pool.status,
> > >> storage_pool.scope, storage_pool.storage_provider_**id,
> > >> storage_pool.host_address, storage_pool.path, storage_
> > >> pool.port, storage_pool.user_info, storage_pool.cluster_id) VALUES
> (217,
> > >> _binary'ceph-rbd-pri-storage',
> > >> _binary'a226c9a1-da78-3f3a-**b5ac-e18b925c9634', 'RBD', '2013-07-1
> > >> 0 09:08:28', null, 2, 2, 0, 0, 'Up', null, null, null, _binary'ceph/
> > >> ceph.client.admin.keyring@10.**237.11.2/sc1<
> http://ceph.client.admin.keyring@10.237.11.2/sc1>',
> > >> 6789, null, 2)
> > >> at
> > >>
> com.cloud.utils.db.**GenericDaoBase.persist(**GenericDaoBase.java:1342)
> > >> at
> > >> com.cloud.storage.dao.**StoragePoolDaoImpl.persist(**
> > >> StoragePoolDaoImpl.java:232)
> > >> at
> > >> com.cloud.utils.component.**ComponentInstantiationPostProc**
> > >>
> essor$InterceptorDispatcher.**intercept(**ComponentInstantiationPostProc*
> > >> *es
> > >>
> > >>
>
> --
> Prasanna.,
>
> ------------------------
> Powered by BigRock.com
>
>
Re: Problem in adding Ceph RBD as primary storage for CloudStack
4.1.0
Posted by Prasanna Santhanam <ts...@apache.org>.
It looks like a previous attempt to start the systemVMs has failed
putting the nfs storage in the avoid set. Did you try restarting your
management server?
This line leads me to the above mentioned:
2013-07-12 13:10:48,236 DEBUG [storage.allocator.AbstractStoragePoolAllocator] (secstorage-1:null) StoragePool is in avoid set, skipping this pool
On Fri, Jul 12, 2013 at 01:16:53PM +0800, Indra Pramana wrote:
> Dear Wido and all,
>
> I have managed to get the hosts, primary and secondary storage running.
>
> - 2 KVM hypervisor hosts
> - One RBD primary storage
> - One NFS primary storage (for system VMs, since I understand that system
> VMs cannot use RBD)
> - One NFS secondary storage
>
> However, now I am having problem with the system VMs: CPVM and SSVM, unable
> to start.
>
> Excerpt from management-server.log file is here:
> http://pastebin.com/ENkpCALY
>
> It seems that the VMs were not able to be created because unable to find
> suitable StoragePools.
>
> I understand that system VMs will be using the NFS primary storage instead
> of RBD, so I have confirmed that I am able to mount the primary storage via
> NFS and have read and write access, from both the hypervisor and the
> management server.
>
> Any advise how can I resolve the problem to make both the system VMs
> created and started?
>
> Looking forward to your reply, thank you.
>
> Cheers.
>
>
> On Fri, Jul 12, 2013 at 9:43 AM, Indra Pramana <in...@sg.or.id> wrote:
>
> > Hi Wido,
> >
> > Thanks for the advice, I'm now able to add the RBD pool as primary storage.
> >
> > Many thanks! :)
> >
> > Cheers.
> >
> >
> > On Thursday, July 11, 2013, Wido den Hollander wrote:
> >
> >> Hi,
> >>
> >> On 07/10/2013 03:42 PM, Chip Childers wrote:
> >>
> >>> Cc'ing Wido, our resident Ceph expert. ;-)
> >>>
> >>>
> >> Hehe ;)
> >>
> >> On Wed, Jul 10, 2013 at 05:45:25PM +0800, Indra Pramana wrote:
> >>>
> >>>> Dear all,
> >>>>
> >>>> I am installing CloudStack 4.1.0 (upgraded from 4.0.2) and I also have a
> >>>> Ceph cluster running. However, I am having issues in adding the RBD as
> >>>> primary storage. Tried to follow the instruction here, but unable to
> >>>> make
> >>>> it work:
> >>>>
> >>>> http://ceph.com/docs/master/**rbd/rbd-cloudstack/<http://ceph.com/docs/master/rbd/rbd-cloudstack/>
> >>>>
> >>>> I have setup a pool on the Ceph cluster. The status of the cluster is
> >>>> healthy. Since I am using Ubuntu 12.04.2 LTS (Precise) for the
> >>>> hypervisors,
> >>>> I also have compiled libvirt manually to ensure that the version 0.9.13
> >>>> is
> >>>> installed (previously it's 0.9.8).
> >>>>
> >>>>
> >> You can also use the Ubuntu Cloud Archive, I still need to get the docs
> >> updated for that.
> >>
> >> I described the process in a blogpost: http://blog.widodh.nl/2013/06/**
> >> a-quick-note-on-running-**cloudstack-with-rbd-on-ubuntu-**12-04/<http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/>
> >>
> >> indra@hv-kvm-01:~/rbd$ ceph
> >>>> ceph> health
> >>>> HEALTH_OK
> >>>>
> >>>> indra@hv-kvm-01:~$ ceph osd lspools
> >>>> 0 data,1 metadata,2 rbd,3 sc1,
> >>>>
> >>>> root@hv-kvm-01:/home/indra# libvirtd --version
> >>>> libvirtd (libvirt) 0.9.13
> >>>>
> >>>> I tried to add Primary Storage into the Cloudstack zone which I have
> >>>> created:
> >>>>
> >>>> Add Primary Storage:
> >>>>
> >>>> Zone: my zone name
> >>>> Pod: my pod name
> >>>> Cluster: my cluster name
> >>>> Name: ceph-rbd-pri-storage
> >>>> Protocol: RBD
> >>>> RADOS Monitor: my first Ceph monitor IP address
> >>>> RADOS Pool: sc1 (the pool name on Ceph cluster)
> >>>> RADOS User: client.admin
> >>>> RADOS Secret: /etc/ceph/ceph.client.admin.**keyring (keyring file
> >>>> location)
> >>>>
> >>>
> >> This is your problem. That shouldn't be the location of the file, but it
> >> should be the secret, which is a base64 encoded string.
> >>
> >> $ ceph auth list
> >>
> >> That should tell you what the secret is.
> >>
> >> Storage Tags: rbd
> >>
> >> This is the error message when I tried to add the primary storage by
> >> clicking OK:
> >>
> >> DB Exception on: com.mysql.jdbc.**JDBC4PreparedStatement@4b2eb56**:
> >> INSERT INTO
> >> storage_pool (storage_pool.id, storage_pool.name, storage_pool.uuid,
> >> storage_pool.pool_type, storage_pool.created, storage_pool.update_time,
> >> storage_pool.data_center_id, storage_pool.pod_id,
> >> storage_pool.available_bytes, storage_pool.capacity_bytes,
> >> storage_pool.status, storage_pool.scope, storage_pool.storage_provider_**
> >> id,
> >> storage_pool.host_address, storage_pool.path, storage_pool.port,
> >> storage_pool.user_info, storage_pool.cluster_id) VALUES (217,
> >> _binary'ceph-rbd-pri-storage',
> >> _binary'a226c9a1-da78-3f3a-**b5ac-e18b925c9634', 'RBD', '2013-07-10
> >> 09:08:28', null, 2, 2, 0, 0, 'Up', null, null, null, _binary'ceph/
> >> ceph.client.admin.keyring@10.**237.11.2/sc1<http://ceph.client.admin.keyring@10.237.11.2/sc1>',
> >> 6789, null, 2)
> >>
> >> On the management-server.log file:
> >>
> >> 2013-07-10 17:08:28,845 DEBUG [cloud.api.ApiServlet]
> >> (catalina-exec-2:null)
> >> ===START=== 192.168.0.100 -- GET
> >> command=createStoragePool&**zoneid=c116950e-e4ae-4f23-**a7e7-
> >> 74a75c4ee638&podId=a748b063-**3a83-4175-a0e9-de39118fe5ce&**
> >> clusterid=1f87eb09-324d-4d49-**83c2-88d84d7a15df&name=ceph-**
> >> rbd-pri-storage&url=rbd%3A%2F%**2Fclient.admin%3A_etc%2Fc
> >> eph%2Fceph.client.admin.**keyring%4010.237.11.2%2Fsc1&**
> >> tags=rbd&response=json&**sessionkey=**rDRfWpqeKfQKbKZtHr398ULV%2F8k%**
> >> 3D&_=1373447307839
> >> 2013-07-10 17:08:28,862 DEBUG [cloud.storage.**StorageManagerImpl]
> >> (catalina-exec-2:null) createPool Params @ scheme - rbd storageHost - null
> >> hostPath - /ceph/ceph.client
> >> .admin.keyring@10.237.11.2/sc1 port - -1
> >> 2013-07-10 17:08:28,918 DEBUG [cloud.storage.**StorageManagerImpl]
> >> (catalina-exec-2:null) In createPool Setting poolId - 217 uuid -
> >> a226c9a1-da78-3f3a-b5ac-**e18b925c9634 z
> >> oneId - 2 podId - 2 poolName - ceph-rbd-pri-storage
> >> 2013-07-10 17:08:28,921 DEBUG [db.Transaction.Transaction]
> >> (catalina-exec-2:null) Rolling back the transaction: Time = 3 Name =
> >> persist; called by -Transaction.rollbac
> >> k:890-Transaction.removeUpTo:**833-Transaction.close:657-**
> >> TransactionContextBuilder.**interceptException:63-**
> >> ComponentInstantiationPostProc**essor$InterceptorDispatcher.**interce
> >> pt:133-StorageManagerImpl.**createPool:1378-**
> >> StorageManagerImpl.createPool:**147-CreateStoragePoolCmd.**
> >> execute:123-ApiDispatcher.**dispatch:162-ApiServer.**
> >> queueCommand:505-ApiSe
> >> rver.handleRequest:355-**ApiServlet.processRequest:302
> >> 2013-07-10 17:08:28,923 ERROR [cloud.api.ApiServer] (catalina-exec-2:null)
> >> unhandled exception executing api command: createStoragePool
> >> com.cloud.utils.exception.**CloudRuntimeException: DB Exception on:
> >> com.mysql.jdbc.**JDBC4PreparedStatement@4b2eb56**: INSERT INTO
> >> storage_pool (
> >> storage_pool.id, storage_pool
> >> .name, storage_pool.uuid, storage_pool.pool_type, storage_pool.created,
> >> storage_pool.update_time, storage_pool.data_center_id,
> >> storage_pool.pod_id,
> >> storage_pool.availab
> >> le_bytes, storage_pool.capacity_bytes, storage_pool.status,
> >> storage_pool.scope, storage_pool.storage_provider_**id,
> >> storage_pool.host_address, storage_pool.path, storage_
> >> pool.port, storage_pool.user_info, storage_pool.cluster_id) VALUES (217,
> >> _binary'ceph-rbd-pri-storage',
> >> _binary'a226c9a1-da78-3f3a-**b5ac-e18b925c9634', 'RBD', '2013-07-1
> >> 0 09:08:28', null, 2, 2, 0, 0, 'Up', null, null, null, _binary'ceph/
> >> ceph.client.admin.keyring@10.**237.11.2/sc1<http://ceph.client.admin.keyring@10.237.11.2/sc1>',
> >> 6789, null, 2)
> >> at
> >> com.cloud.utils.db.**GenericDaoBase.persist(**GenericDaoBase.java:1342)
> >> at
> >> com.cloud.storage.dao.**StoragePoolDaoImpl.persist(**
> >> StoragePoolDaoImpl.java:232)
> >> at
> >> com.cloud.utils.component.**ComponentInstantiationPostProc**
> >> essor$InterceptorDispatcher.**intercept(**ComponentInstantiationPostProc*
> >> *es
> >>
> >>
--
Prasanna.,
------------------------
Powered by BigRock.com
Re: Problem in adding Ceph RBD as primary storage for CloudStack 4.1.0
Posted by Indra Pramana <in...@sg.or.id>.
Dear Wido and all,
I have managed to get the hosts, primary and secondary storage running.
- 2 KVM hypervisor hosts
- One RBD primary storage
- One NFS primary storage (for system VMs, since I understand that system
VMs cannot use RBD)
- One NFS secondary storage
However, now I am having problem with the system VMs: CPVM and SSVM, unable
to start.
Excerpt from management-server.log file is here:
http://pastebin.com/ENkpCALY
It seems that the VMs were not able to be created because unable to find
suitable StoragePools.
I understand that system VMs will be using the NFS primary storage instead
of RBD, so I have confirmed that I am able to mount the primary storage via
NFS and have read and write access, from both the hypervisor and the
management server.
Any advise how can I resolve the problem to make both the system VMs
created and started?
Looking forward to your reply, thank you.
Cheers.
On Fri, Jul 12, 2013 at 9:43 AM, Indra Pramana <in...@sg.or.id> wrote:
> Hi Wido,
>
> Thanks for the advice, I'm now able to add the RBD pool as primary storage.
>
> Many thanks! :)
>
> Cheers.
>
>
> On Thursday, July 11, 2013, Wido den Hollander wrote:
>
>> Hi,
>>
>> On 07/10/2013 03:42 PM, Chip Childers wrote:
>>
>>> Cc'ing Wido, our resident Ceph expert. ;-)
>>>
>>>
>> Hehe ;)
>>
>> On Wed, Jul 10, 2013 at 05:45:25PM +0800, Indra Pramana wrote:
>>>
>>>> Dear all,
>>>>
>>>> I am installing CloudStack 4.1.0 (upgraded from 4.0.2) and I also have a
>>>> Ceph cluster running. However, I am having issues in adding the RBD as
>>>> primary storage. Tried to follow the instruction here, but unable to
>>>> make
>>>> it work:
>>>>
>>>> http://ceph.com/docs/master/**rbd/rbd-cloudstack/<http://ceph.com/docs/master/rbd/rbd-cloudstack/>
>>>>
>>>> I have setup a pool on the Ceph cluster. The status of the cluster is
>>>> healthy. Since I am using Ubuntu 12.04.2 LTS (Precise) for the
>>>> hypervisors,
>>>> I also have compiled libvirt manually to ensure that the version 0.9.13
>>>> is
>>>> installed (previously it's 0.9.8).
>>>>
>>>>
>> You can also use the Ubuntu Cloud Archive, I still need to get the docs
>> updated for that.
>>
>> I described the process in a blogpost: http://blog.widodh.nl/2013/06/**
>> a-quick-note-on-running-**cloudstack-with-rbd-on-ubuntu-**12-04/<http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/>
>>
>> indra@hv-kvm-01:~/rbd$ ceph
>>>> ceph> health
>>>> HEALTH_OK
>>>>
>>>> indra@hv-kvm-01:~$ ceph osd lspools
>>>> 0 data,1 metadata,2 rbd,3 sc1,
>>>>
>>>> root@hv-kvm-01:/home/indra# libvirtd --version
>>>> libvirtd (libvirt) 0.9.13
>>>>
>>>> I tried to add Primary Storage into the Cloudstack zone which I have
>>>> created:
>>>>
>>>> Add Primary Storage:
>>>>
>>>> Zone: my zone name
>>>> Pod: my pod name
>>>> Cluster: my cluster name
>>>> Name: ceph-rbd-pri-storage
>>>> Protocol: RBD
>>>> RADOS Monitor: my first Ceph monitor IP address
>>>> RADOS Pool: sc1 (the pool name on Ceph cluster)
>>>> RADOS User: client.admin
>>>> RADOS Secret: /etc/ceph/ceph.client.admin.**keyring (keyring file
>>>> location)
>>>>
>>>
>> This is your problem. That shouldn't be the location of the file, but it
>> should be the secret, which is a base64 encoded string.
>>
>> $ ceph auth list
>>
>> That should tell you what the secret is.
>>
>> Storage Tags: rbd
>>
>> This is the error message when I tried to add the primary storage by
>> clicking OK:
>>
>> DB Exception on: com.mysql.jdbc.**JDBC4PreparedStatement@4b2eb56**:
>> INSERT INTO
>> storage_pool (storage_pool.id, storage_pool.name, storage_pool.uuid,
>> storage_pool.pool_type, storage_pool.created, storage_pool.update_time,
>> storage_pool.data_center_id, storage_pool.pod_id,
>> storage_pool.available_bytes, storage_pool.capacity_bytes,
>> storage_pool.status, storage_pool.scope, storage_pool.storage_provider_**
>> id,
>> storage_pool.host_address, storage_pool.path, storage_pool.port,
>> storage_pool.user_info, storage_pool.cluster_id) VALUES (217,
>> _binary'ceph-rbd-pri-storage',
>> _binary'a226c9a1-da78-3f3a-**b5ac-e18b925c9634', 'RBD', '2013-07-10
>> 09:08:28', null, 2, 2, 0, 0, 'Up', null, null, null, _binary'ceph/
>> ceph.client.admin.keyring@10.**237.11.2/sc1<http://ceph.client.admin.keyring@10.237.11.2/sc1>',
>> 6789, null, 2)
>>
>> On the management-server.log file:
>>
>> 2013-07-10 17:08:28,845 DEBUG [cloud.api.ApiServlet]
>> (catalina-exec-2:null)
>> ===START=== 192.168.0.100 -- GET
>> command=createStoragePool&**zoneid=c116950e-e4ae-4f23-**a7e7-
>> 74a75c4ee638&podId=a748b063-**3a83-4175-a0e9-de39118fe5ce&**
>> clusterid=1f87eb09-324d-4d49-**83c2-88d84d7a15df&name=ceph-**
>> rbd-pri-storage&url=rbd%3A%2F%**2Fclient.admin%3A_etc%2Fc
>> eph%2Fceph.client.admin.**keyring%4010.237.11.2%2Fsc1&**
>> tags=rbd&response=json&**sessionkey=**rDRfWpqeKfQKbKZtHr398ULV%2F8k%**
>> 3D&_=1373447307839
>> 2013-07-10 17:08:28,862 DEBUG [cloud.storage.**StorageManagerImpl]
>> (catalina-exec-2:null) createPool Params @ scheme - rbd storageHost - null
>> hostPath - /ceph/ceph.client
>> .admin.keyring@10.237.11.2/sc1 port - -1
>> 2013-07-10 17:08:28,918 DEBUG [cloud.storage.**StorageManagerImpl]
>> (catalina-exec-2:null) In createPool Setting poolId - 217 uuid -
>> a226c9a1-da78-3f3a-b5ac-**e18b925c9634 z
>> oneId - 2 podId - 2 poolName - ceph-rbd-pri-storage
>> 2013-07-10 17:08:28,921 DEBUG [db.Transaction.Transaction]
>> (catalina-exec-2:null) Rolling back the transaction: Time = 3 Name =
>> persist; called by -Transaction.rollbac
>> k:890-Transaction.removeUpTo:**833-Transaction.close:657-**
>> TransactionContextBuilder.**interceptException:63-**
>> ComponentInstantiationPostProc**essor$InterceptorDispatcher.**interce
>> pt:133-StorageManagerImpl.**createPool:1378-**
>> StorageManagerImpl.createPool:**147-CreateStoragePoolCmd.**
>> execute:123-ApiDispatcher.**dispatch:162-ApiServer.**
>> queueCommand:505-ApiSe
>> rver.handleRequest:355-**ApiServlet.processRequest:302
>> 2013-07-10 17:08:28,923 ERROR [cloud.api.ApiServer] (catalina-exec-2:null)
>> unhandled exception executing api command: createStoragePool
>> com.cloud.utils.exception.**CloudRuntimeException: DB Exception on:
>> com.mysql.jdbc.**JDBC4PreparedStatement@4b2eb56**: INSERT INTO
>> storage_pool (
>> storage_pool.id, storage_pool
>> .name, storage_pool.uuid, storage_pool.pool_type, storage_pool.created,
>> storage_pool.update_time, storage_pool.data_center_id,
>> storage_pool.pod_id,
>> storage_pool.availab
>> le_bytes, storage_pool.capacity_bytes, storage_pool.status,
>> storage_pool.scope, storage_pool.storage_provider_**id,
>> storage_pool.host_address, storage_pool.path, storage_
>> pool.port, storage_pool.user_info, storage_pool.cluster_id) VALUES (217,
>> _binary'ceph-rbd-pri-storage',
>> _binary'a226c9a1-da78-3f3a-**b5ac-e18b925c9634', 'RBD', '2013-07-1
>> 0 09:08:28', null, 2, 2, 0, 0, 'Up', null, null, null, _binary'ceph/
>> ceph.client.admin.keyring@10.**237.11.2/sc1<http://ceph.client.admin.keyring@10.237.11.2/sc1>',
>> 6789, null, 2)
>> at
>> com.cloud.utils.db.**GenericDaoBase.persist(**GenericDaoBase.java:1342)
>> at
>> com.cloud.storage.dao.**StoragePoolDaoImpl.persist(**
>> StoragePoolDaoImpl.java:232)
>> at
>> com.cloud.utils.component.**ComponentInstantiationPostProc**
>> essor$InterceptorDispatcher.**intercept(**ComponentInstantiationPostProc*
>> *es
>>
>>
Re: Problem in adding Ceph RBD as primary storage for CloudStack 4.1.0
Posted by Indra Pramana <in...@sg.or.id>.
Hi Wido,
Thanks for the advice, I'm now able to add the RBD pool as primary storage.
Many thanks! :)
Cheers.
On Thursday, July 11, 2013, Wido den Hollander wrote:
> Hi,
>
> On 07/10/2013 03:42 PM, Chip Childers wrote:
>
>> Cc'ing Wido, our resident Ceph expert. ;-)
>>
>>
> Hehe ;)
>
> On Wed, Jul 10, 2013 at 05:45:25PM +0800, Indra Pramana wrote:
>>
>>> Dear all,
>>>
>>> I am installing CloudStack 4.1.0 (upgraded from 4.0.2) and I also have a
>>> Ceph cluster running. However, I am having issues in adding the RBD as
>>> primary storage. Tried to follow the instruction here, but unable to make
>>> it work:
>>>
>>> http://ceph.com/docs/master/**rbd/rbd-cloudstack/<http://ceph.com/docs/master/rbd/rbd-cloudstack/>
>>>
>>> I have setup a pool on the Ceph cluster. The status of the cluster is
>>> healthy. Since I am using Ubuntu 12.04.2 LTS (Precise) for the
>>> hypervisors,
>>> I also have compiled libvirt manually to ensure that the version 0.9.13
>>> is
>>> installed (previously it's 0.9.8).
>>>
>>>
> You can also use the Ubuntu Cloud Archive, I still need to get the docs
> updated for that.
>
> I described the process in a blogpost: http://blog.widodh.nl/2013/06/**
> a-quick-note-on-running-**cloudstack-with-rbd-on-ubuntu-**12-04/<http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/>
>
> indra@hv-kvm-01:~/rbd$ ceph
>>> ceph> health
>>> HEALTH_OK
>>>
>>> indra@hv-kvm-01:~$ ceph osd lspools
>>> 0 data,1 metadata,2 rbd,3 sc1,
>>>
>>> root@hv-kvm-01:/home/indra# libvirtd --version
>>> libvirtd (libvirt) 0.9.13
>>>
>>> I tried to add Primary Storage into the Cloudstack zone which I have
>>> created:
>>>
>>> Add Primary Storage:
>>>
>>> Zone: my zone name
>>> Pod: my pod name
>>> Cluster: my cluster name
>>> Name: ceph-rbd-pri-storage
>>> Protocol: RBD
>>> RADOS Monitor: my first Ceph monitor IP address
>>> RADOS Pool: sc1 (the pool name on Ceph cluster)
>>> RADOS User: client.admin
>>> RADOS Secret: /etc/ceph/ceph.client.admin.**keyring (keyring file
>>> location)
>>>
>>
> This is your problem. That shouldn't be the location of the file, but it
> should be the secret, which is a base64 encoded string.
>
> $ ceph auth list
>
> That should tell you what the secret is.
>
> Storage Tags: rbd
>
> This is the error message when I tried to add the primary storage by
> clicking OK:
>
> DB Exception on: com.mysql.jdbc.**JDBC4PreparedStatement@4b2eb56**:
> INSERT INTO
> storage_pool (storage_pool.id, storage_pool.name, storage_pool.uuid,
> storage_pool.pool_type, storage_pool.created, storage_pool.update_time,
> storage_pool.data_center_id, storage_pool.pod_id,
> storage_pool.available_bytes, storage_pool.capacity_bytes,
> storage_pool.status, storage_pool.scope, storage_pool.storage_provider_**
> id,
> storage_pool.host_address, storage_pool.path, storage_pool.port,
> storage_pool.user_info, storage_pool.cluster_id) VALUES (217,
> _binary'ceph-rbd-pri-storage',
> _binary'a226c9a1-da78-3f3a-**b5ac-e18b925c9634', 'RBD', '2013-07-10
> 09:08:28', null, 2, 2, 0, 0, 'Up', null, null, null, _binary'ceph/
> ceph.client.admin.keyring@10.**237.11.2/sc1<http://ceph.client.admin.keyring@10.237.11.2/sc1>',
> 6789, null, 2)
>
> On the management-server.log file:
>
> 2013-07-10 17:08:28,845 DEBUG [cloud.api.ApiServlet] (catalina-exec-2:null)
> ===START=== 192.168.0.100 -- GET
> command=createStoragePool&**zoneid=c116950e-e4ae-4f23-**a7e7-
> 74a75c4ee638&podId=a748b063-**3a83-4175-a0e9-de39118fe5ce&**
> clusterid=1f87eb09-324d-4d49-**83c2-88d84d7a15df&name=ceph-**
> rbd-pri-storage&url=rbd%3A%2F%**2Fclient.admin%3A_etc%2Fc
> eph%2Fceph.client.admin.**keyring%4010.237.11.2%2Fsc1&**
> tags=rbd&response=json&**sessionkey=**rDRfWpqeKfQKbKZtHr398ULV%2F8k%**
> 3D&_=1373447307839
> 2013-07-10 17:08:28,862 DEBUG [cloud.storage.**StorageManagerImpl]
> (catalina-exec-2:null) createPool Params @ scheme - rbd storageHost - null
> hostPath - /ceph/ceph.client
> .admin.keyring@10.237.11.2/sc1 port - -1
> 2013-07-10 17:08:28,918 DEBUG [cloud.storage.**StorageManagerImpl]
> (catalina-exec-2:null) In createPool Setting poolId - 217 uuid -
> a226c9a1-da78-3f3a-b5ac-**e18b925c9634 z
> oneId - 2 podId - 2 poolName - ceph-rbd-pri-storage
> 2013-07-10 17:08:28,921 DEBUG [db.Transaction.Transaction]
> (catalina-exec-2:null) Rolling back the transaction: Time = 3 Name =
> persist; called by -Transaction.rollbac
> k:890-Transaction.removeUpTo:**833-Transaction.close:657-**
> TransactionContextBuilder.**interceptException:63-**
> ComponentInstantiationPostProc**essor$InterceptorDispatcher.**interce
> pt:133-StorageManagerImpl.**createPool:1378-**
> StorageManagerImpl.createPool:**147-CreateStoragePoolCmd.**
> execute:123-ApiDispatcher.**dispatch:162-ApiServer.**
> queueCommand:505-ApiSe
> rver.handleRequest:355-**ApiServlet.processRequest:302
> 2013-07-10 17:08:28,923 ERROR [cloud.api.ApiServer] (catalina-exec-2:null)
> unhandled exception executing api command: createStoragePool
> com.cloud.utils.exception.**CloudRuntimeException: DB Exception on:
> com.mysql.jdbc.**JDBC4PreparedStatement@4b2eb56**: INSERT INTO
> storage_pool (
> storage_pool.id, storage_pool
> .name, storage_pool.uuid, storage_pool.pool_type, storage_pool.created,
> storage_pool.update_time, storage_pool.data_center_id, storage_pool.pod_id,
> storage_pool.availab
> le_bytes, storage_pool.capacity_bytes, storage_pool.status,
> storage_pool.scope, storage_pool.storage_provider_**id,
> storage_pool.host_address, storage_pool.path, storage_
> pool.port, storage_pool.user_info, storage_pool.cluster_id) VALUES (217,
> _binary'ceph-rbd-pri-storage',
> _binary'a226c9a1-da78-3f3a-**b5ac-e18b925c9634', 'RBD', '2013-07-1
> 0 09:08:28', null, 2, 2, 0, 0, 'Up', null, null, null, _binary'ceph/
> ceph.client.admin.keyring@10.**237.11.2/sc1<http://ceph.client.admin.keyring@10.237.11.2/sc1>',
> 6789, null, 2)
> at
> com.cloud.utils.db.**GenericDaoBase.persist(**GenericDaoBase.java:1342)
> at
> com.cloud.storage.dao.**StoragePoolDaoImpl.persist(**
> StoragePoolDaoImpl.java:232)
> at
> com.cloud.utils.component.**ComponentInstantiationPostProc**
> essor$InterceptorDispatcher.**intercept(**ComponentInstantiationPostProc**
> es
>
>
Re: Problem in adding Ceph RBD as primary storage for CloudStack
4.1.0
Posted by Wido den Hollander <wi...@widodh.nl>.
Hi,
On 07/10/2013 03:42 PM, Chip Childers wrote:
> Cc'ing Wido, our resident Ceph expert. ;-)
>
Hehe ;)
> On Wed, Jul 10, 2013 at 05:45:25PM +0800, Indra Pramana wrote:
>> Dear all,
>>
>> I am installing CloudStack 4.1.0 (upgraded from 4.0.2) and I also have a
>> Ceph cluster running. However, I am having issues in adding the RBD as
>> primary storage. Tried to follow the instruction here, but unable to make
>> it work:
>>
>> http://ceph.com/docs/master/rbd/rbd-cloudstack/
>>
>> I have setup a pool on the Ceph cluster. The status of the cluster is
>> healthy. Since I am using Ubuntu 12.04.2 LTS (Precise) for the hypervisors,
>> I also have compiled libvirt manually to ensure that the version 0.9.13 is
>> installed (previously it's 0.9.8).
>>
You can also use the Ubuntu Cloud Archive, I still need to get the docs
updated for that.
I described the process in a blogpost:
http://blog.widodh.nl/2013/06/a-quick-note-on-running-cloudstack-with-rbd-on-ubuntu-12-04/
>> indra@hv-kvm-01:~/rbd$ ceph
>> ceph> health
>> HEALTH_OK
>>
>> indra@hv-kvm-01:~$ ceph osd lspools
>> 0 data,1 metadata,2 rbd,3 sc1,
>>
>> root@hv-kvm-01:/home/indra# libvirtd --version
>> libvirtd (libvirt) 0.9.13
>>
>> I tried to add Primary Storage into the Cloudstack zone which I have
>> created:
>>
>> Add Primary Storage:
>>
>> Zone: my zone name
>> Pod: my pod name
>> Cluster: my cluster name
>> Name: ceph-rbd-pri-storage
>> Protocol: RBD
>> RADOS Monitor: my first Ceph monitor IP address
>> RADOS Pool: sc1 (the pool name on Ceph cluster)
>> RADOS User: client.admin
>> RADOS Secret: /etc/ceph/ceph.client.admin.keyring (keyring file location)
This is your problem. That shouldn't be the location of the file, but it
should be the secret, which is a base64 encoded string.
$ ceph auth list
That should tell you what the secret is.
>> Storage Tags: rbd
>>
>> This is the error message when I tried to add the primary storage by
>> clicking OK:
>>
>> DB Exception on: com.mysql.jdbc.JDBC4PreparedStatement@4b2eb56: INSERT INTO
>> storage_pool (storage_pool.id, storage_pool.name, storage_pool.uuid,
>> storage_pool.pool_type, storage_pool.created, storage_pool.update_time,
>> storage_pool.data_center_id, storage_pool.pod_id,
>> storage_pool.available_bytes, storage_pool.capacity_bytes,
>> storage_pool.status, storage_pool.scope, storage_pool.storage_provider_id,
>> storage_pool.host_address, storage_pool.path, storage_pool.port,
>> storage_pool.user_info, storage_pool.cluster_id) VALUES (217,
>> _binary'ceph-rbd-pri-storage',
>> _binary'a226c9a1-da78-3f3a-b5ac-e18b925c9634', 'RBD', '2013-07-10
>> 09:08:28', null, 2, 2, 0, 0, 'Up', null, null, null, _binary'ceph/
>> ceph.client.admin.keyring@10.237.11.2/sc1', 6789, null, 2)
>>
>> On the management-server.log file:
>>
>> 2013-07-10 17:08:28,845 DEBUG [cloud.api.ApiServlet] (catalina-exec-2:null)
>> ===START=== 192.168.0.100 -- GET
>> command=createStoragePool&zoneid=c116950e-e4ae-4f23-a7e7-
>> 74a75c4ee638&podId=a748b063-3a83-4175-a0e9-de39118fe5ce&clusterid=1f87eb09-324d-4d49-83c2-88d84d7a15df&name=ceph-rbd-pri-storage&url=rbd%3A%2F%2Fclient.admin%3A_etc%2Fc
>> eph%2Fceph.client.admin.keyring%4010.237.11.2%2Fsc1&tags=rbd&response=json&sessionkey=rDRfWpqeKfQKbKZtHr398ULV%2F8k%3D&_=1373447307839
>> 2013-07-10 17:08:28,862 DEBUG [cloud.storage.StorageManagerImpl]
>> (catalina-exec-2:null) createPool Params @ scheme - rbd storageHost - null
>> hostPath - /ceph/ceph.client
>> .admin.keyring@10.237.11.2/sc1 port - -1
>> 2013-07-10 17:08:28,918 DEBUG [cloud.storage.StorageManagerImpl]
>> (catalina-exec-2:null) In createPool Setting poolId - 217 uuid -
>> a226c9a1-da78-3f3a-b5ac-e18b925c9634 z
>> oneId - 2 podId - 2 poolName - ceph-rbd-pri-storage
>> 2013-07-10 17:08:28,921 DEBUG [db.Transaction.Transaction]
>> (catalina-exec-2:null) Rolling back the transaction: Time = 3 Name =
>> persist; called by -Transaction.rollbac
>> k:890-Transaction.removeUpTo:833-Transaction.close:657-TransactionContextBuilder.interceptException:63-ComponentInstantiationPostProcessor$InterceptorDispatcher.interce
>> pt:133-StorageManagerImpl.createPool:1378-StorageManagerImpl.createPool:147-CreateStoragePoolCmd.execute:123-ApiDispatcher.dispatch:162-ApiServer.queueCommand:505-ApiSe
>> rver.handleRequest:355-ApiServlet.processRequest:302
>> 2013-07-10 17:08:28,923 ERROR [cloud.api.ApiServer] (catalina-exec-2:null)
>> unhandled exception executing api command: createStoragePool
>> com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
>> com.mysql.jdbc.JDBC4PreparedStatement@4b2eb56: INSERT INTO storage_pool (
>> storage_pool.id, storage_pool
>> .name, storage_pool.uuid, storage_pool.pool_type, storage_pool.created,
>> storage_pool.update_time, storage_pool.data_center_id, storage_pool.pod_id,
>> storage_pool.availab
>> le_bytes, storage_pool.capacity_bytes, storage_pool.status,
>> storage_pool.scope, storage_pool.storage_provider_id,
>> storage_pool.host_address, storage_pool.path, storage_
>> pool.port, storage_pool.user_info, storage_pool.cluster_id) VALUES (217,
>> _binary'ceph-rbd-pri-storage',
>> _binary'a226c9a1-da78-3f3a-b5ac-e18b925c9634', 'RBD', '2013-07-1
>> 0 09:08:28', null, 2, 2, 0, 0, 'Up', null, null, null, _binary'ceph/
>> ceph.client.admin.keyring@10.237.11.2/sc1', 6789, null, 2)
>> at
>> com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1342)
>> at
>> com.cloud.storage.dao.StoragePoolDaoImpl.persist(StoragePoolDaoImpl.java:232)
>> at
>> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>> at
>> com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:1378)
>> at
>> com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:147)
>> at
>> org.apache.cloudstack.api.command.admin.storage.CreateStoragePoolCmd.execute(CreateStoragePoolCmd.java:123)
>> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162)
>> at com.cloud.api.ApiServer.queueCommand(ApiServer.java:505)
>> at com.cloud.api.ApiServer.handleRequest(ApiServer.java:355)
>> at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:302)
>> at com.cloud.api.ApiServlet.doGet(ApiServlet.java:66)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>> at
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>> at
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>> at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>> at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>> at
>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:615)
>> at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>> at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
>> at
>> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
>> at
>> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:744)
>> at
>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2282)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:679)
>> Caused by:
>> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
>> Column 'host_address' cannot be null
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> Method)
>> at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>> at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>> at com.mysql.jdbc.Util.getInstance(Util.java:386)
>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1040)
>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4074)
>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4006)
>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2468)
>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2629)
>> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2719)
>> at
>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2155)
>> at
>> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2450)
>> at
>> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2371)
>> at
>> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2355)
>> at
>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
>> at
>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
>> at
>> com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1305)
>> ... 32 more
>> 2013-07-10 17:08:28,927 DEBUG [cloud.api.ApiServlet] (catalina-exec-2:null)
>> ===END=== 192.168.0.100 -- GET
>> command=createStoragePool&zoneid=c116950e-e4ae-4f23-a7e7-74a75c4ee638&podId=a748b063-3a83-4175-a0e9-de39118fe5ce&clusterid=1f87eb09-324d-4d49-83c2-88d84d7a15df&name=ceph-rbd-pri-storage&url=rbd%3A%2F%2Fclient.admin%3A_etc%2Fceph%2Fceph.client.admin.keyring%4010.237.11.2%2Fsc1&tags=rbd&response=json&sessionkey=rDRfWpqeKfQKbKZtHr398ULV%2F8k%3D&_=1373447307839
>>
>> Based on the error message, I tried to map the value of each field being
>> submitted, and it seems the problem is here:
>>
>> Caused by:
>> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
>> Column 'host_address' cannot be null
>>
>> storage_pool.id, 215
>> storage_pool.name, _binary'sc1'
>> storage_pool.uuid, _binary'a226c9a1-da78-3f3a-b5ac-e18b925c9634'
>> storage_pool.pool_type, 'RBD'
>> storage_pool.created, '2013-07-10 08:27:03'
>> storage_pool.update_time, null
>> storage_pool.data_center_id, 2
>> storage_pool.pod_id, 2
>> storage_pool.available_bytes, 0
>> storage_pool.capacity_bytes, 0
>> storage_pool.status, 'Up'
>> storage_pool.scope, null
>> storage_pool.storage_provider_id, null
>> storage_pool.host_address, null <=======
>> storage_pool.path, _binary'ceph/ceph.client.admin.keyring@10.237.11.2/sc1
>> storage_pool.port, 6789
>> storage_pool.user_info, null
>> storage_pool.cluster_id, 2
>>
>> It seems that it's expecting the IP address / hostname of the monitor
>> server on the "storage_pool.host_address" field. However, I noted that the
>> IP address is in the storage_pool.path field rather than the
>> storage_pool.host_address.
>>
>> Anyone knows what can be done to resolve this problem?
>>
>> Looking forward to your reply, thank you.
>>
>> Cheers.
Re: Problem in adding Ceph RBD as primary storage for CloudStack 4.1.0
Posted by Indra Pramana <in...@sg.or.id>.
Dear all,
Any advise? I have tried to uninstall and reinstall cloudstack-management
on the management server, as well as the cloudstack-agent on the
hypervisors. The problem still persists.
When I try to add primary storage from the Cloudstack GUI, is the adding
process performed by the management server or by the hypervisors?
Looking forward to your reply, thank you.
Cheers.
On Wed, Jul 10, 2013 at 9:42 PM, Chip Childers <ch...@sungard.com>wrote:
> Cc'ing Wido, our resident Ceph expert. ;-)
>
> On Wed, Jul 10, 2013 at 05:45:25PM +0800, Indra Pramana wrote:
> > Dear all,
> >
> > I am installing CloudStack 4.1.0 (upgraded from 4.0.2) and I also have a
> > Ceph cluster running. However, I am having issues in adding the RBD as
> > primary storage. Tried to follow the instruction here, but unable to make
> > it work:
> >
> > http://ceph.com/docs/master/rbd/rbd-cloudstack/
> >
> > I have setup a pool on the Ceph cluster. The status of the cluster is
> > healthy. Since I am using Ubuntu 12.04.2 LTS (Precise) for the
> hypervisors,
> > I also have compiled libvirt manually to ensure that the version 0.9.13
> is
> > installed (previously it's 0.9.8).
> >
> > indra@hv-kvm-01:~/rbd$ ceph
> > ceph> health
> > HEALTH_OK
> >
> > indra@hv-kvm-01:~$ ceph osd lspools
> > 0 data,1 metadata,2 rbd,3 sc1,
> >
> > root@hv-kvm-01:/home/indra# libvirtd --version
> > libvirtd (libvirt) 0.9.13
> >
> > I tried to add Primary Storage into the Cloudstack zone which I have
> > created:
> >
> > Add Primary Storage:
> >
> > Zone: my zone name
> > Pod: my pod name
> > Cluster: my cluster name
> > Name: ceph-rbd-pri-storage
> > Protocol: RBD
> > RADOS Monitor: my first Ceph monitor IP address
> > RADOS Pool: sc1 (the pool name on Ceph cluster)
> > RADOS User: client.admin
> > RADOS Secret: /etc/ceph/ceph.client.admin.keyring (keyring file location)
> > Storage Tags: rbd
> >
> > This is the error message when I tried to add the primary storage by
> > clicking OK:
> >
> > DB Exception on: com.mysql.jdbc.JDBC4PreparedStatement@4b2eb56: INSERT
> INTO
> > storage_pool (storage_pool.id, storage_pool.name, storage_pool.uuid,
> > storage_pool.pool_type, storage_pool.created, storage_pool.update_time,
> > storage_pool.data_center_id, storage_pool.pod_id,
> > storage_pool.available_bytes, storage_pool.capacity_bytes,
> > storage_pool.status, storage_pool.scope,
> storage_pool.storage_provider_id,
> > storage_pool.host_address, storage_pool.path, storage_pool.port,
> > storage_pool.user_info, storage_pool.cluster_id) VALUES (217,
> > _binary'ceph-rbd-pri-storage',
> > _binary'a226c9a1-da78-3f3a-b5ac-e18b925c9634', 'RBD', '2013-07-10
> > 09:08:28', null, 2, 2, 0, 0, 'Up', null, null, null, _binary'ceph/
> > ceph.client.admin.keyring@10.237.11.2/sc1', 6789, null, 2)
> >
> > On the management-server.log file:
> >
> > 2013-07-10 17:08:28,845 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-2:null)
> > ===START=== 192.168.0.100 -- GET
> > command=createStoragePool&zoneid=c116950e-e4ae-4f23-a7e7-
> >
> 74a75c4ee638&podId=a748b063-3a83-4175-a0e9-de39118fe5ce&clusterid=1f87eb09-324d-4d49-83c2-88d84d7a15df&name=ceph-rbd-pri-storage&url=rbd%3A%2F%2Fclient.admin%3A_etc%2Fc
> >
> eph%2Fceph.client.admin.keyring%4010.237.11.2%2Fsc1&tags=rbd&response=json&sessionkey=rDRfWpqeKfQKbKZtHr398ULV%2F8k%3D&_=1373447307839
> > 2013-07-10 17:08:28,862 DEBUG [cloud.storage.StorageManagerImpl]
> > (catalina-exec-2:null) createPool Params @ scheme - rbd storageHost -
> null
> > hostPath - /ceph/ceph.client
> > .admin.keyring@10.237.11.2/sc1 port - -1
> > 2013-07-10 17:08:28,918 DEBUG [cloud.storage.StorageManagerImpl]
> > (catalina-exec-2:null) In createPool Setting poolId - 217 uuid -
> > a226c9a1-da78-3f3a-b5ac-e18b925c9634 z
> > oneId - 2 podId - 2 poolName - ceph-rbd-pri-storage
> > 2013-07-10 17:08:28,921 DEBUG [db.Transaction.Transaction]
> > (catalina-exec-2:null) Rolling back the transaction: Time = 3 Name =
> > persist; called by -Transaction.rollbac
> >
> k:890-Transaction.removeUpTo:833-Transaction.close:657-TransactionContextBuilder.interceptException:63-ComponentInstantiationPostProcessor$InterceptorDispatcher.interce
> >
> pt:133-StorageManagerImpl.createPool:1378-StorageManagerImpl.createPool:147-CreateStoragePoolCmd.execute:123-ApiDispatcher.dispatch:162-ApiServer.queueCommand:505-ApiSe
> > rver.handleRequest:355-ApiServlet.processRequest:302
> > 2013-07-10 17:08:28,923 ERROR [cloud.api.ApiServer]
> (catalina-exec-2:null)
> > unhandled exception executing api command: createStoragePool
> > com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
> > com.mysql.jdbc.JDBC4PreparedStatement@4b2eb56: INSERT INTO storage_pool
> (
> > storage_pool.id, storage_pool
> > .name, storage_pool.uuid, storage_pool.pool_type, storage_pool.created,
> > storage_pool.update_time, storage_pool.data_center_id,
> storage_pool.pod_id,
> > storage_pool.availab
> > le_bytes, storage_pool.capacity_bytes, storage_pool.status,
> > storage_pool.scope, storage_pool.storage_provider_id,
> > storage_pool.host_address, storage_pool.path, storage_
> > pool.port, storage_pool.user_info, storage_pool.cluster_id) VALUES (217,
> > _binary'ceph-rbd-pri-storage',
> > _binary'a226c9a1-da78-3f3a-b5ac-e18b925c9634', 'RBD', '2013-07-1
> > 0 09:08:28', null, 2, 2, 0, 0, 'Up', null, null, null, _binary'ceph/
> > ceph.client.admin.keyring@10.237.11.2/sc1', 6789, null, 2)
> > at
> > com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1342)
> > at
> >
> com.cloud.storage.dao.StoragePoolDaoImpl.persist(StoragePoolDaoImpl.java:232)
> > at
> >
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> > at
> >
> com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:1378)
> > at
> >
> com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:147)
> > at
> >
> org.apache.cloudstack.api.command.admin.storage.CreateStoragePoolCmd.execute(CreateStoragePoolCmd.java:123)
> > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162)
> > at com.cloud.api.ApiServer.queueCommand(ApiServer.java:505)
> > at com.cloud.api.ApiServer.handleRequest(ApiServer.java:355)
> > at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:302)
> > at com.cloud.api.ApiServlet.doGet(ApiServlet.java:66)
> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
> > at
> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> > at
> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> > at
> >
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> > at
> >
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> > at
> >
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> > at
> >
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> > at
> > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:615)
> > at
> >
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> > at
> >
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
> > at
> >
> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
> > at
> >
> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:744)
> > at
> >
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2282)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:679)
> > Caused by:
> >
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> > Column 'host_address' cannot be null
> > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > Method)
> > at
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> > at
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> > at
> java.lang.reflect.Constructor.newInstance(Constructor.java:532)
> > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> > at com.mysql.jdbc.Util.getInstance(Util.java:386)
> > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1040)
> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4074)
> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4006)
> > at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2468)
> > at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2629)
> > at
> com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2719)
> > at
> >
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2155)
> > at
> >
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2450)
> > at
> >
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2371)
> > at
> >
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2355)
> > at
> >
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
> > at
> >
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
> > at
> > com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1305)
> > ... 32 more
> > 2013-07-10 17:08:28,927 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-2:null)
> > ===END=== 192.168.0.100 -- GET
> >
> command=createStoragePool&zoneid=c116950e-e4ae-4f23-a7e7-74a75c4ee638&podId=a748b063-3a83-4175-a0e9-de39118fe5ce&clusterid=1f87eb09-324d-4d49-83c2-88d84d7a15df&name=ceph-rbd-pri-storage&url=rbd%3A%2F%2Fclient.admin%3A_etc%2Fceph%2Fceph.client.admin.keyring%4010.237.11.2%2Fsc1&tags=rbd&response=json&sessionkey=rDRfWpqeKfQKbKZtHr398ULV%2F8k%3D&_=1373447307839
> >
> > Based on the error message, I tried to map the value of each field being
> > submitted, and it seems the problem is here:
> >
> > Caused by:
> >
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> > Column 'host_address' cannot be null
> >
> > storage_pool.id, 215
> > storage_pool.name, _binary'sc1'
> > storage_pool.uuid, _binary'a226c9a1-da78-3f3a-b5ac-e18b925c9634'
> > storage_pool.pool_type, 'RBD'
> > storage_pool.created, '2013-07-10 08:27:03'
> > storage_pool.update_time, null
> > storage_pool.data_center_id, 2
> > storage_pool.pod_id, 2
> > storage_pool.available_bytes, 0
> > storage_pool.capacity_bytes, 0
> > storage_pool.status, 'Up'
> > storage_pool.scope, null
> > storage_pool.storage_provider_id, null
> > storage_pool.host_address, null <=======
> > storage_pool.path, _binary'ceph/
> ceph.client.admin.keyring@10.237.11.2/sc1
> > storage_pool.port, 6789
> > storage_pool.user_info, null
> > storage_pool.cluster_id, 2
> >
> > It seems that it's expecting the IP address / hostname of the monitor
> > server on the "storage_pool.host_address" field. However, I noted that
> the
> > IP address is in the storage_pool.path field rather than the
> > storage_pool.host_address.
> >
> > Anyone knows what can be done to resolve this problem?
> >
> > Looking forward to your reply, thank you.
> >
> > Cheers.
>
Re: Problem in adding Ceph RBD as primary storage for CloudStack
4.1.0
Posted by Chip Childers <ch...@sungard.com>.
Cc'ing Wido, our resident Ceph expert. ;-)
On Wed, Jul 10, 2013 at 05:45:25PM +0800, Indra Pramana wrote:
> Dear all,
>
> I am installing CloudStack 4.1.0 (upgraded from 4.0.2) and I also have a
> Ceph cluster running. However, I am having issues in adding the RBD as
> primary storage. Tried to follow the instruction here, but unable to make
> it work:
>
> http://ceph.com/docs/master/rbd/rbd-cloudstack/
>
> I have setup a pool on the Ceph cluster. The status of the cluster is
> healthy. Since I am using Ubuntu 12.04.2 LTS (Precise) for the hypervisors,
> I also have compiled libvirt manually to ensure that the version 0.9.13 is
> installed (previously it's 0.9.8).
>
> indra@hv-kvm-01:~/rbd$ ceph
> ceph> health
> HEALTH_OK
>
> indra@hv-kvm-01:~$ ceph osd lspools
> 0 data,1 metadata,2 rbd,3 sc1,
>
> root@hv-kvm-01:/home/indra# libvirtd --version
> libvirtd (libvirt) 0.9.13
>
> I tried to add Primary Storage into the Cloudstack zone which I have
> created:
>
> Add Primary Storage:
>
> Zone: my zone name
> Pod: my pod name
> Cluster: my cluster name
> Name: ceph-rbd-pri-storage
> Protocol: RBD
> RADOS Monitor: my first Ceph monitor IP address
> RADOS Pool: sc1 (the pool name on Ceph cluster)
> RADOS User: client.admin
> RADOS Secret: /etc/ceph/ceph.client.admin.keyring (keyring file location)
> Storage Tags: rbd
>
> This is the error message when I tried to add the primary storage by
> clicking OK:
>
> DB Exception on: com.mysql.jdbc.JDBC4PreparedStatement@4b2eb56: INSERT INTO
> storage_pool (storage_pool.id, storage_pool.name, storage_pool.uuid,
> storage_pool.pool_type, storage_pool.created, storage_pool.update_time,
> storage_pool.data_center_id, storage_pool.pod_id,
> storage_pool.available_bytes, storage_pool.capacity_bytes,
> storage_pool.status, storage_pool.scope, storage_pool.storage_provider_id,
> storage_pool.host_address, storage_pool.path, storage_pool.port,
> storage_pool.user_info, storage_pool.cluster_id) VALUES (217,
> _binary'ceph-rbd-pri-storage',
> _binary'a226c9a1-da78-3f3a-b5ac-e18b925c9634', 'RBD', '2013-07-10
> 09:08:28', null, 2, 2, 0, 0, 'Up', null, null, null, _binary'ceph/
> ceph.client.admin.keyring@10.237.11.2/sc1', 6789, null, 2)
>
> On the management-server.log file:
>
> 2013-07-10 17:08:28,845 DEBUG [cloud.api.ApiServlet] (catalina-exec-2:null)
> ===START=== 192.168.0.100 -- GET
> command=createStoragePool&zoneid=c116950e-e4ae-4f23-a7e7-
> 74a75c4ee638&podId=a748b063-3a83-4175-a0e9-de39118fe5ce&clusterid=1f87eb09-324d-4d49-83c2-88d84d7a15df&name=ceph-rbd-pri-storage&url=rbd%3A%2F%2Fclient.admin%3A_etc%2Fc
> eph%2Fceph.client.admin.keyring%4010.237.11.2%2Fsc1&tags=rbd&response=json&sessionkey=rDRfWpqeKfQKbKZtHr398ULV%2F8k%3D&_=1373447307839
> 2013-07-10 17:08:28,862 DEBUG [cloud.storage.StorageManagerImpl]
> (catalina-exec-2:null) createPool Params @ scheme - rbd storageHost - null
> hostPath - /ceph/ceph.client
> .admin.keyring@10.237.11.2/sc1 port - -1
> 2013-07-10 17:08:28,918 DEBUG [cloud.storage.StorageManagerImpl]
> (catalina-exec-2:null) In createPool Setting poolId - 217 uuid -
> a226c9a1-da78-3f3a-b5ac-e18b925c9634 z
> oneId - 2 podId - 2 poolName - ceph-rbd-pri-storage
> 2013-07-10 17:08:28,921 DEBUG [db.Transaction.Transaction]
> (catalina-exec-2:null) Rolling back the transaction: Time = 3 Name =
> persist; called by -Transaction.rollbac
> k:890-Transaction.removeUpTo:833-Transaction.close:657-TransactionContextBuilder.interceptException:63-ComponentInstantiationPostProcessor$InterceptorDispatcher.interce
> pt:133-StorageManagerImpl.createPool:1378-StorageManagerImpl.createPool:147-CreateStoragePoolCmd.execute:123-ApiDispatcher.dispatch:162-ApiServer.queueCommand:505-ApiSe
> rver.handleRequest:355-ApiServlet.processRequest:302
> 2013-07-10 17:08:28,923 ERROR [cloud.api.ApiServer] (catalina-exec-2:null)
> unhandled exception executing api command: createStoragePool
> com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
> com.mysql.jdbc.JDBC4PreparedStatement@4b2eb56: INSERT INTO storage_pool (
> storage_pool.id, storage_pool
> .name, storage_pool.uuid, storage_pool.pool_type, storage_pool.created,
> storage_pool.update_time, storage_pool.data_center_id, storage_pool.pod_id,
> storage_pool.availab
> le_bytes, storage_pool.capacity_bytes, storage_pool.status,
> storage_pool.scope, storage_pool.storage_provider_id,
> storage_pool.host_address, storage_pool.path, storage_
> pool.port, storage_pool.user_info, storage_pool.cluster_id) VALUES (217,
> _binary'ceph-rbd-pri-storage',
> _binary'a226c9a1-da78-3f3a-b5ac-e18b925c9634', 'RBD', '2013-07-1
> 0 09:08:28', null, 2, 2, 0, 0, 'Up', null, null, null, _binary'ceph/
> ceph.client.admin.keyring@10.237.11.2/sc1', 6789, null, 2)
> at
> com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1342)
> at
> com.cloud.storage.dao.StoragePoolDaoImpl.persist(StoragePoolDaoImpl.java:232)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:1378)
> at
> com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:147)
> at
> org.apache.cloudstack.api.command.admin.storage.CreateStoragePoolCmd.execute(CreateStoragePoolCmd.java:123)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162)
> at com.cloud.api.ApiServer.queueCommand(ApiServer.java:505)
> at com.cloud.api.ApiServer.handleRequest(ApiServer.java:355)
> at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:302)
> at com.cloud.api.ApiServlet.doGet(ApiServlet.java:66)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:615)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
> at
> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
> at
> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:744)
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2282)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:679)
> Caused by:
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> Column 'host_address' cannot be null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> at com.mysql.jdbc.Util.getInstance(Util.java:386)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1040)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4074)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4006)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2468)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2629)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2719)
> at
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2155)
> at
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2450)
> at
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2371)
> at
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2355)
> at
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
> at
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
> at
> com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1305)
> ... 32 more
> 2013-07-10 17:08:28,927 DEBUG [cloud.api.ApiServlet] (catalina-exec-2:null)
> ===END=== 192.168.0.100 -- GET
> command=createStoragePool&zoneid=c116950e-e4ae-4f23-a7e7-74a75c4ee638&podId=a748b063-3a83-4175-a0e9-de39118fe5ce&clusterid=1f87eb09-324d-4d49-83c2-88d84d7a15df&name=ceph-rbd-pri-storage&url=rbd%3A%2F%2Fclient.admin%3A_etc%2Fceph%2Fceph.client.admin.keyring%4010.237.11.2%2Fsc1&tags=rbd&response=json&sessionkey=rDRfWpqeKfQKbKZtHr398ULV%2F8k%3D&_=1373447307839
>
> Based on the error message, I tried to map the value of each field being
> submitted, and it seems the problem is here:
>
> Caused by:
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> Column 'host_address' cannot be null
>
> storage_pool.id, 215
> storage_pool.name, _binary'sc1'
> storage_pool.uuid, _binary'a226c9a1-da78-3f3a-b5ac-e18b925c9634'
> storage_pool.pool_type, 'RBD'
> storage_pool.created, '2013-07-10 08:27:03'
> storage_pool.update_time, null
> storage_pool.data_center_id, 2
> storage_pool.pod_id, 2
> storage_pool.available_bytes, 0
> storage_pool.capacity_bytes, 0
> storage_pool.status, 'Up'
> storage_pool.scope, null
> storage_pool.storage_provider_id, null
> storage_pool.host_address, null <=======
> storage_pool.path, _binary'ceph/ceph.client.admin.keyring@10.237.11.2/sc1
> storage_pool.port, 6789
> storage_pool.user_info, null
> storage_pool.cluster_id, 2
>
> It seems that it's expecting the IP address / hostname of the monitor
> server on the "storage_pool.host_address" field. However, I noted that the
> IP address is in the storage_pool.path field rather than the
> storage_pool.host_address.
>
> Anyone knows what can be done to resolve this problem?
>
> Looking forward to your reply, thank you.
>
> Cheers.