You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by "mwaag (via GitHub)" <gi...@apache.org> on 2025/06/13 10:03:29 UTC

[D] Directory /var/cloudstack not created (cloudstack 4.18.2.4 / Ubuntu) [cloudstack]

GitHub user mwaag created a discussion: Directory /var/cloudstack not created (cloudstack 4.18.2.4 / Ubuntu)

Hi,
iam trying to rotate our old management-server, based on outdated os and replace them with ubuntu-based management-hosts.

Here is what i do so far (using https://docs.cloudstack.apache.org/en/4.18.2.4/installguide/management-server/index.html)
- Prepare OS
- Configure Package Repo https://download.cloudstack.org for ubuntu jammy (because i still need 4.18 atm)
- Install packages cloudstack-management, -usage, -common (Version 4.18.2.4)
- copy db.properties of one of the other Servers and replace (instead of using cloudstack-setup-databases) the generic one
- execute cloudstack-setup-management

cloudstack-management then starts fine and the web-interface is reachable, but there are some errors in /var/log/cloudstack/management/management-server.log:

```
2025-06-13 11:17:54,634 ERROR [o.a.c.s.NfsMountManager] (BackgroundTaskPollManager-6:ctx-74c87b2f) (logid:2752ce59) Unable to create mount: /var/cloudstack/mnt/345049466209.7b38fd2
2025-06-13 11:17:54,635 ERROR [o.a.c.s.NfsMountManager] (BackgroundTaskPollManager-6:ctx-74c87b2f) (logid:2752ce59) Unable to create mount: /var/cloudstack/mnt/345049466209.6650a7d
2025-06-13 11:17:54,635 ERROR [o.a.c.s.NfsMountManager] (BackgroundTaskPollManager-6:ctx-74c87b2f) (logid:2752ce59) Unable to create mount: /var/cloudstack/mnt/345049466209.74a4a218
2025-06-13 11:17:54,635 ERROR [o.a.c.s.NfsMountManager] (BackgroundTaskPollManager-6:ctx-74c87b2f) (logid:2752ce59) Unable to create mount: /var/cloudstack/mnt/345049466209.5e6a7e1a
2025-06-13 11:17:54,635 ERROR [o.a.c.s.NfsMountManager] (BackgroundTaskPollManager-6:ctx-74c87b2f) (logid:2752ce59) Unable to create mount: /var/cloudstack/mnt/345049466209.5e0936f5
2025-06-13 11:17:54,636 ERROR [o.a.c.s.NfsMountManager] (BackgroundTaskPollManager-6:ctx-74c87b2f) (logid:2752ce59) Unable to create mount: /var/cloudstack/mnt/345049466209.3effc941
2025-06-13 11:17:54,636 ERROR [o.a.c.s.NfsMountManager] (BackgroundTaskPollManager-6:ctx-74c87b2f) (logid:2752ce59) Unable to create mount: /var/cloudstack/mnt/345049466209.a93d98b
2025-06-13 11:17:54,636 ERROR [o.a.c.s.NfsMountManager] (BackgroundTaskPollManager-6:ctx-74c87b2f) (logid:2752ce59) Unable to create mount: /var/cloudstack/mnt/345049466209.72ec9776
2025-06-13 11:17:54,636 ERROR [o.a.c.s.NfsMountManager] (BackgroundTaskPollManager-6:ctx-74c87b2f) (logid:2752ce59) Unable to create mount: /var/cloudstack/mnt/345049466209.2cdd8c2d
2025-06-13 11:17:54,636 ERROR [o.a.c.s.NfsMountManager] (BackgroundTaskPollManager-6:ctx-74c87b2f) (logid:2752ce59) Unable to create mount: /var/cloudstack/mnt/345049466209.19f01fe3
2025-06-13 11:17:54,636 ERROR [o.a.c.s.NfsMountManager] (BackgroundTaskPollManager-6:ctx-74c87b2f) (logid:2752ce59) Unable to create mount point for nfs://172.27.249.130/bi_nfs_sec_01
```

When i look at the given path, there is no /var/cloudstack directory at all. Is this relevant for clean functionality?

We use a separate nfs-server for secondary Storage which is reachable and the export is mountable, just to be sure i can test it, but following the log messages there is a problem with creating the mount-directories not with reaching the nfs-export.
```
ping xxx.xxx.xxx.xxx
PING xxx.xxx.xxx.xxx (xxx.xxx.xxx.xxx) 56(84) bytes of data.
64 bytes from xxx.xxx.xxx.xxx: icmp_seq=1 ttl=63 time=0.331 ms
64 bytes from xxx.xxx.xxx.xxx: icmp_seq=2 ttl=63 time=0.455 ms
64 bytes from xxx.xxx.xxx.xxx: icmp_seq=3 ttl=63 time=0.264 ms
```

```
mount -t nfs xxx.xxx.xxx.xxx:/bi_nfs_sec_01 /mnt/
ls /mnt/
2501-311-a4ff37ec-efdd-3ed5-b03a-927c705974d6  snapshots  systemvm  template  volumes
```

Checking on the other two hosts i see following directory-structure / permissions:
```
ls -lah /var/cloudstack/
total 4.0K
drwxr-xr-x.  4 root  root    35 Apr  4  2022 .
drwxr-xr-x. 20 root  root   285 Apr  4  2022 ..
drwxrwx---.  4 cloud cloud   38 Oct 10  2024 management
drwxrwx---. 25 root  cloud 4.0K Jun 10 15:00 mnt
```

The errors disappear when i manually create these directories as well but i expect the package cloudstack-management to create those dirs.
Checking the package on the old hosts indicates this:
```
rpm -ql cloudstack-management | grep var
/var/cache/cloudstack/management
/var/cloudstack/management
/var/cloudstack/mnt
/var/log/cloudstack/ipallocator
/var/log/cloudstack/management
/var/run/cloudstack-management.pid
```

The cloudstack-java-process is running as user 'cloud' which is expected, because it's the same on the old management-hosts. 
Additionally i cannot remember needing to create those directories manually in the past.

What could be wrong here?

Thanks in advance
Markus

GitHub link: https://github.com/apache/cloudstack/discussions/11023

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] Directory /var/cloudstack not created (cloudstack 4.18.2.4 / Ubuntu) [cloudstack]

Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
GitHub user DaanHoogland added a comment to the discussion: Directory /var/cloudstack not created (cloudstack 4.18.2.4 / Ubuntu)

thanks for this trouble shooting work. If you are sure this works it would be good to add a PR to change the default vaule in case the value is not explicitely set by the user.

GitHub link: https://github.com/apache/cloudstack/discussions/11023#discussioncomment-13460303

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] Directory /var/cloudstack not created (cloudstack 4.18.2.4 / Ubuntu) [cloudstack]

Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
GitHub user DaanHoogland added a comment to the discussion: Directory /var/cloudstack not created (cloudstack 4.18.2.4 / Ubuntu)

would that mean that there is an exception handler that can see that the mount point is not available, check the configuration to see if it should and then do the java equivalent of `mkdir -p <the configuration value>` . The downside of this is that the operator can make a mistake and cloudstack will forgive them instead of pointing them to their error. Is there maybe a way to make the "Unable to create mount:” message more clear instead?

GitHub link: https://github.com/apache/cloudstack/discussions/11023#discussioncomment-13484820

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] Directory /var/cloudstack not created (cloudstack 4.18.2.4 / Ubuntu) [cloudstack]

Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
GitHub user DaanHoogland added a comment to the discussion: Directory /var/cloudstack not created (cloudstack 4.18.2.4 / Ubuntu)

@mwaag , as you put it now it still seems like a bug in the upgrade procedure. Would be nice to fix it for future potential victims. Do you still have an old DB dump to find out whether you had changed it before?

GitHub link: https://github.com/apache/cloudstack/discussions/11023#discussioncomment-13461749

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] Directory /var/cloudstack not created (cloudstack 4.18.2.4 / Ubuntu) [cloudstack]

Posted by "mwaag (via GitHub)" <gi...@apache.org>.
GitHub user mwaag closed a discussion: Directory /var/cloudstack not created (cloudstack 4.18.2.4 / Ubuntu)

Hi,
iam trying to rotate our old management-server, based on outdated os and replace them with ubuntu-based management-hosts.

Here is what i do so far (using https://docs.cloudstack.apache.org/en/4.18.2.4/installguide/management-server/index.html)
- Prepare OS
- Configure Package Repo https://download.cloudstack.org for ubuntu jammy (because i still need 4.18 atm)
- Install packages cloudstack-management, -usage, -common (Version 4.18.2.4)
- copy db.properties of one of the other Servers and replace (instead of using cloudstack-setup-databases) the generic one
- execute cloudstack-setup-management

cloudstack-management then starts fine and the web-interface is reachable, but there are some errors in /var/log/cloudstack/management/management-server.log:

```
2025-06-13 11:17:54,634 ERROR [o.a.c.s.NfsMountManager] (BackgroundTaskPollManager-6:ctx-74c87b2f) (logid:2752ce59) Unable to create mount: /var/cloudstack/mnt/345049466209.7b38fd2
2025-06-13 11:17:54,635 ERROR [o.a.c.s.NfsMountManager] (BackgroundTaskPollManager-6:ctx-74c87b2f) (logid:2752ce59) Unable to create mount: /var/cloudstack/mnt/345049466209.6650a7d
2025-06-13 11:17:54,635 ERROR [o.a.c.s.NfsMountManager] (BackgroundTaskPollManager-6:ctx-74c87b2f) (logid:2752ce59) Unable to create mount: /var/cloudstack/mnt/345049466209.74a4a218
2025-06-13 11:17:54,635 ERROR [o.a.c.s.NfsMountManager] (BackgroundTaskPollManager-6:ctx-74c87b2f) (logid:2752ce59) Unable to create mount: /var/cloudstack/mnt/345049466209.5e6a7e1a
2025-06-13 11:17:54,635 ERROR [o.a.c.s.NfsMountManager] (BackgroundTaskPollManager-6:ctx-74c87b2f) (logid:2752ce59) Unable to create mount: /var/cloudstack/mnt/345049466209.5e0936f5
2025-06-13 11:17:54,636 ERROR [o.a.c.s.NfsMountManager] (BackgroundTaskPollManager-6:ctx-74c87b2f) (logid:2752ce59) Unable to create mount: /var/cloudstack/mnt/345049466209.3effc941
2025-06-13 11:17:54,636 ERROR [o.a.c.s.NfsMountManager] (BackgroundTaskPollManager-6:ctx-74c87b2f) (logid:2752ce59) Unable to create mount: /var/cloudstack/mnt/345049466209.a93d98b
2025-06-13 11:17:54,636 ERROR [o.a.c.s.NfsMountManager] (BackgroundTaskPollManager-6:ctx-74c87b2f) (logid:2752ce59) Unable to create mount: /var/cloudstack/mnt/345049466209.72ec9776
2025-06-13 11:17:54,636 ERROR [o.a.c.s.NfsMountManager] (BackgroundTaskPollManager-6:ctx-74c87b2f) (logid:2752ce59) Unable to create mount: /var/cloudstack/mnt/345049466209.2cdd8c2d
2025-06-13 11:17:54,636 ERROR [o.a.c.s.NfsMountManager] (BackgroundTaskPollManager-6:ctx-74c87b2f) (logid:2752ce59) Unable to create mount: /var/cloudstack/mnt/345049466209.19f01fe3
2025-06-13 11:17:54,636 ERROR [o.a.c.s.NfsMountManager] (BackgroundTaskPollManager-6:ctx-74c87b2f) (logid:2752ce59) Unable to create mount point for nfs://172.27.249.130/bi_nfs_sec_01
```

When i look at the given path, there is no /var/cloudstack directory at all. Is this relevant for clean functionality?

We use a separate nfs-server for secondary Storage which is reachable and the export is mountable, just to be sure i can test it, but following the log messages there is a problem with creating the mount-directories not with reaching the nfs-export.
```
ping xxx.xxx.xxx.xxx
PING xxx.xxx.xxx.xxx (xxx.xxx.xxx.xxx) 56(84) bytes of data.
64 bytes from xxx.xxx.xxx.xxx: icmp_seq=1 ttl=63 time=0.331 ms
64 bytes from xxx.xxx.xxx.xxx: icmp_seq=2 ttl=63 time=0.455 ms
64 bytes from xxx.xxx.xxx.xxx: icmp_seq=3 ttl=63 time=0.264 ms
```

```
mount -t nfs xxx.xxx.xxx.xxx:/bi_nfs_sec_01 /mnt/
ls /mnt/
2501-311-a4ff37ec-efdd-3ed5-b03a-927c705974d6  snapshots  systemvm  template  volumes
```

Checking on the other two hosts i see following directory-structure / permissions:
```
ls -lah /var/cloudstack/
total 4.0K
drwxr-xr-x.  4 root  root    35 Apr  4  2022 .
drwxr-xr-x. 20 root  root   285 Apr  4  2022 ..
drwxrwx---.  4 cloud cloud   38 Oct 10  2024 management
drwxrwx---. 25 root  cloud 4.0K Jun 10 15:00 mnt
```

The errors disappear when i manually create these directories as well but i expect the package cloudstack-management to create those dirs.
Checking the package on the old hosts indicates this:
```
rpm -ql cloudstack-management | grep var
/var/cache/cloudstack/management
/var/cloudstack/management
/var/cloudstack/mnt
/var/log/cloudstack/ipallocator
/var/log/cloudstack/management
/var/run/cloudstack-management.pid
```

The cloudstack-java-process is running as user 'cloud' which is expected, because it's the same on the old management-hosts. 
Additionally i cannot remember needing to create those directories manually in the past.

What could be wrong here?

Thanks in advance
Markus

GitHub link: https://github.com/apache/cloudstack/discussions/11023

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] Directory /var/cloudstack not created (cloudstack 4.18.2.4 / Ubuntu) [cloudstack]

Posted by "mwaag (via GitHub)" <gi...@apache.org>.
GitHub user mwaag added a comment to the discussion: Directory /var/cloudstack not created (cloudstack 4.18.2.4 / Ubuntu)

Yeah, i already tested it in our lab and it seems to work as expected. And i think the default value of /var/lib/cloudstack/mnt is fine. I just had this problem because we never replaced our management-server before, only upgraded them. Either /var/cloudstack/mnt was the old default or we explicitly set it then. Shouldn't be a problem with whole fresh installations.

GitHub link: https://github.com/apache/cloudstack/discussions/11023#discussioncomment-13460613

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] Directory /var/cloudstack not created (cloudstack 4.18.2.4 / Ubuntu) [cloudstack]

Posted by "mwaag (via GitHub)" <gi...@apache.org>.
GitHub user mwaag added a comment to the discussion: Directory /var/cloudstack not created (cloudstack 4.18.2.4 / Ubuntu)

@DaanHoogland 
Not from that particular database, but i have very old ones from an old productive instance.
It seems your default never has changed but we edited it at some point. Attached a screenshot of an old 4.11-database from around 2022 compared to a fresh 4.18.2.4-database from 2025.
![cs411-db](https://github.com/user-attachments/assets/98fe5246-82d8-4a8a-b725-45ad34b3e25c)
![cs418-db](https://github.com/user-attachments/assets/f8dc9e92-046b-4c21-9166-14ccb9cc411f)

Plus this beeing the special case where one want to add a freshly installed management-host to existing ones (which may have lived a long time already). As i understand, the directory /var/cloudstack will be created at installation time, there is no knowledge about the information from a potential existing database at this stage.

Ofc it would be nice, if cloudstack fix it by itself through creating the customized path (after a service-restart for example).


GitHub link: https://github.com/apache/cloudstack/discussions/11023#discussioncomment-13484532

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] Directory /var/cloudstack not created (cloudstack 4.18.2.4 / Ubuntu) [cloudstack]

Posted by "mwaag (via GitHub)" <gi...@apache.org>.
GitHub user mwaag added a comment to the discussion: Directory /var/cloudstack not created (cloudstack 4.18.2.4 / Ubuntu)

Ok, i think i found the culprit already. I think its because of the global setting mount.parent, which is /mnt/cloudstack/mnt but the new package brings /var/lib/cloudstack/mnt. /var/lib/cloudstack is succesfully created by cloudstack-packages on the new server.

I assume it would be preferred to use the new path? Will it work to change the path in global settings of the existing installation and restart the management-servers to apply? I may have to make the new directory first on the old servers.


GitHub link: https://github.com/apache/cloudstack/discussions/11023#discussioncomment-13459997

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] Directory /var/cloudstack not created (cloudstack 4.18.2.4 / Ubuntu) [cloudstack]

Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
GitHub user DaanHoogland added a comment to the discussion: Directory /var/cloudstack not created (cloudstack 4.18.2.4 / Ubuntu)

@mwaag , do you want to create an issue for this or shall we close?

GitHub link: https://github.com/apache/cloudstack/discussions/11023#discussioncomment-13494695

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] Directory /var/cloudstack not created (cloudstack 4.18.2.4 / Ubuntu) [cloudstack]

Posted by "mwaag (via GitHub)" <gi...@apache.org>.
GitHub user mwaag added a comment to the discussion: Directory /var/cloudstack not created (cloudstack 4.18.2.4 / Ubuntu)

No. Pointing me to the misconfiguration was fine for me at the end. You can close.
Thanks!

GitHub link: https://github.com/apache/cloudstack/discussions/11023#discussioncomment-13512520

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org