You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by "mosys0815 (via GitHub)" <gi...@apache.org> on 2025/07/07 07:21:11 UTC

[D] RootCAProvider error on management server with L3-network config [cloudstack]

GitHub user mosys0815 created a discussion: RootCAProvider error on management server with L3-network config

I am currently preparing a cluster and installed a new (the 1st) management server.
That server's network is a fully routed L3-network.
Starting the cloudstack-management service does not create the server certificate from root-ca with the following error:

```
2025-06-10 08:28:57,082 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from RootCAProvider
2025-06-10 08:28:58,589 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [RootCAProvider] in [Ca Providers Registry]
2025-06-10 08:28:58,589 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.ca.provider.RootCAProvider@611f82a8
2025-06-10 08:28:58,589 INFO  [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components
2025-06-10 08:28:58,589 INFO  [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean RootCAProvider.
2025-06-10 08:28:59,037 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Executing command [/bin/bash -c ip route show default 0.0.0.0/0 | head -1 | awk '{print $5}' ].
2025-06-10 08:28:59,043 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Successfully executed process [105870] for command [/bin/bash -c ip route show default 0.0.0.0/0 | head -1 | awk '{print $5}' ].
2025-06-10 08:28:59,127 ERROR [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Error on configuring bean RootCAProvider - Cannot invoke "java.net.NetworkInterface.getInterfaceAddresses()" because "nic" is null java.lang.NullPointerException: Cannot invoke "java.net.NetworkInterface.getInterfaceAddresses()" because "nic" is null
```

* these are the active network-interfaces, "hostip" is the bgp-announced ip on a virtual interface for networking, eth1* are the ethernet interfaces
```
~# ip -4 -br a | egrep '(eth1|hostip)'
eth1a            UP             10.72.44.198/30
eth1b            UP             10.72.45.198/30
hostip           UNKNOWN        10.72.44.3/32
```

* CS did retrieve the correct ip from hostip interface
```
~# ip route show default 0.0.0.0/0 | head -1 | awk '{print $5}'
10.72.44.3
```

* certificate check
```
~# openssl s_client -connect 10.72.44.3:9090 </dev/null 2>/dev/null | openssl x509 -noout -ext subjectAltNam
Could not read certificate from <stdin>
4087895B58700000:error:1608010C:STORE routines:ossl_store_handle_load_result:unsupported:../crypto/store/store_result.c:151:
Unable to load certificate
```

For testing i installed 2 virtual machines with cloudstack-management and a new mysql-database on one of these. Both instances are connected on a layer 2 network.
Here the management servers (one after another) came once with fully functional certificates, both servers see each other as peers in the cs-ui management server overview.

* certificates SAN on one of the test-instances:
```
~# openssl s_client -connect 10.65.254.48:9090 </dev/null 2>/dev/null | openssl x509 -noout -ext subjectAltName
X509v3 Subject Alternative Name:
    IP Address:FE80:0:0:0:1C00:B1FF:FE00:164, IP Address:10.65.254.48, DNS:<redacted> DNS:cloudstack.internal
```

I then connected the former server with a fresh installation  to the database of my test-setup.
I got the same error as mentioned above.

I suspect some issue with the L3-network and retrieving some networking-stuff by cloudstack-management-server.

Any idea how to proceed here?


GitHub link: https://github.com/apache/cloudstack/discussions/11152

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] RootCAProvider error on management server with L3-network config [cloudstack]

Posted by "mosys0815 (via GitHub)" <gi...@apache.org>.
GitHub user mosys0815 added a comment to the discussion: RootCAProvider error on management server with L3-network config

@DaanHoogland in my build it is fixed, any chance to get that into upstream?

GitHub link: https://github.com/apache/cloudstack/discussions/11152#discussioncomment-14612673

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] RootCAProvider error on management server with L3-network config [cloudstack]

Posted by "mosys0815 (via GitHub)" <gi...@apache.org>.
GitHub user mosys0815 added a comment to the discussion: RootCAProvider error on management server with L3-network config

Sry for the delay, meanwhile other topics got higher priority.

I managed to build cloudstack from source (fork), even though setting up a build-server was quite a hassle, the documentation is outdated and not working for current distributions like ubuntu 24.04 (noble).

I made a slight change to the interface detection command.
It now works on our L3 network and on a L2-Network inside a virtual machine.
Unfortunately we do not have any hardware servers in common L2 networks anymore, so I am not able to test this with for example bond-setups.




GitHub link: https://github.com/apache/cloudstack/discussions/11152#discussioncomment-13749325

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] RootCAProvider error on management server with L3-network config [cloudstack]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
GitHub user boring-cyborg[bot] added a comment to the discussion: RootCAProvider error on management server with L3-network config

Thanks for opening your first issue here! Be sure to follow the issue template!


GitHub link: https://github.com/apache/cloudstack/discussions/11152#discussioncomment-13680349

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] RootCAProvider error on management server with L3-network config [cloudstack]

Posted by "mosys0815 (via GitHub)" <gi...@apache.org>.
GitHub user mosys0815 added a comment to the discussion: RootCAProvider error on management server with L3-network config

yes, I thought so, a one-liner using structured data could like this:

```
ip -j a | jq -r '.[] | .addr_info | map(select(.local == "'"$(ip -j r s default | jq -r '.[0] | .prefsrc')"'")) | .[].label'
```

output on my L2-network test virtual machine:
```
~# ip -j a | jq -r '.[] | .addr_info | map(select(.local == "'"$(ip -j r s default | jq -r '.[0] | .prefsrc')"'")) | .[].label'
ens3
```

output on L3-network server:
```
~# ip -j a | jq -r '.[] | .addr_info | map(select(.local == "'"$(ip -j r s default | jq -r '.[0] | .prefsrc')"'")) | .[].label'
hostip
```

Feel free to use and adapt :)

But for now, is there anything I can do to get the certificate, even manually?


GitHub link: https://github.com/apache/cloudstack/discussions/11152#discussioncomment-13680357

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] RootCAProvider error on management server with L3-network config [cloudstack]

Posted by "mosys0815 (via GitHub)" <gi...@apache.org>.
GitHub user mosys0815 added a comment to the discussion: RootCAProvider error on management server with L3-network config

AH, thx for clarification :)

Yes, i' like to expect the ip "10.72.44.3" added to the certificate, in fact with your current ip route command it already gets the correct ip.

The server is up and running, but https://10.72.44.3:9090/clusterservice is not available due to a missing certificate.
Therefore a lot of certificate error messages appear in the logs and the both test virtual management servers are not able to communicate with this one server.

GitHub link: https://github.com/apache/cloudstack/discussions/11152#discussioncomment-13680355

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] RootCAProvider error on management server with L3-network config [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
GitHub user weizhouapache added a comment to the discussion: RootCAProvider error on management server with L3-network config

> What exactly is the device used for? In our setup we explicitly do _not_ want to use a fixed interface, since these are supposed to "failover" each other.

@mosys0815 
the command gets the default device, and certificate is for all IPs of the default device

what IP would you like to use primarily ? 10.72.44.3 ?

btw: is the management server running , or stopped after `Error on configuring bean RootCAProvider` ?

GitHub link: https://github.com/apache/cloudstack/discussions/11152#discussioncomment-13680354

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] RootCAProvider error on management server with L3-network config [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
GitHub user weizhouapache added a comment to the discussion: RootCAProvider error on management server with L3-network config

what's the output of command `ip route show default 0.0.0.0/0` ?



GitHub link: https://github.com/apache/cloudstack/discussions/11152#discussioncomment-13680350

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] RootCAProvider error on management server with L3-network config [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
GitHub user weizhouapache added a comment to the discussion: RootCAProvider error on management server with L3-network config

@mosys0815 
you can create a pull request

GitHub link: https://github.com/apache/cloudstack/discussions/11152#discussioncomment-14612915

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] RootCAProvider error on management server with L3-network config [cloudstack]

Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
GitHub user DaanHoogland added a comment to the discussion: RootCAProvider error on management server with L3-network config

can you refer your PR here and close the discussion @mosys0815 ?

GitHub link: https://github.com/apache/cloudstack/discussions/11152#discussioncomment-15135422

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] RootCAProvider error on management server with L3-network config [cloudstack]

Posted by "mosys0815 (via GitHub)" <gi...@apache.org>.
GitHub user mosys0815 added a comment to the discussion: RootCAProvider error on management server with L3-network config

```
~# ip route show default 0.0.0.0/0
default proto bird src 10.72.44.3 metric 32
	nexthop via inet6 fe80::429e:a4ff:fe79:f50f dev eth1b weight 1
	nexthop via inet6 fe80::429e:a4ff:fe7b:8f0f dev eth1a weight 1
default via 10.72.44.197 dev eth1a proto static src 10.72.44.198 metric 1024 onlink
default via 10.72.45.197 dev eth1b proto static src 10.72.45.198 metric 1024 onlink
```

GitHub link: https://github.com/apache/cloudstack/discussions/11152#discussioncomment-13680351

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] RootCAProvider error on management server with L3-network config [cloudstack]

Posted by "mosys0815 (via GitHub)" <gi...@apache.org>.
GitHub user mosys0815 added a comment to the discussion: RootCAProvider error on management server with L3-network config

ok, thx, I will have a look into the source and try to get it running :)


GitHub link: https://github.com/apache/cloudstack/discussions/11152#discussioncomment-13680359

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] RootCAProvider error on management server with L3-network config [cloudstack]

Posted by "mosys0815 (via GitHub)" <gi...@apache.org>.
GitHub user mosys0815 closed a discussion: RootCAProvider error on management server with L3-network config

I am currently preparing a cluster and installed a new (the 1st) management server.
That server's network is a fully routed L3-network.
Starting the cloudstack-management service does not create the server certificate from root-ca with the following error:

```
2025-06-10 08:28:57,082 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from RootCAProvider
2025-06-10 08:28:58,589 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [RootCAProvider] in [Ca Providers Registry]
2025-06-10 08:28:58,589 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.ca.provider.RootCAProvider@611f82a8
2025-06-10 08:28:58,589 INFO  [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components
2025-06-10 08:28:58,589 INFO  [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean RootCAProvider.
2025-06-10 08:28:59,037 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Executing command [/bin/bash -c ip route show default 0.0.0.0/0 | head -1 | awk '{print $5}' ].
2025-06-10 08:28:59,043 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Successfully executed process [105870] for command [/bin/bash -c ip route show default 0.0.0.0/0 | head -1 | awk '{print $5}' ].
2025-06-10 08:28:59,127 ERROR [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Error on configuring bean RootCAProvider - Cannot invoke "java.net.NetworkInterface.getInterfaceAddresses()" because "nic" is null java.lang.NullPointerException: Cannot invoke "java.net.NetworkInterface.getInterfaceAddresses()" because "nic" is null
```

* these are the active network-interfaces, "hostip" is the bgp-announced ip on a virtual interface for networking, eth1* are the ethernet interfaces
```
~# ip -4 -br a | egrep '(eth1|hostip)'
eth1a            UP             10.72.44.198/30
eth1b            UP             10.72.45.198/30
hostip           UNKNOWN        10.72.44.3/32
```

* CS did retrieve the correct ip from hostip interface
```
~# ip route show default 0.0.0.0/0 | head -1 | awk '{print $5}'
10.72.44.3
```

* certificate check
```
~# openssl s_client -connect 10.72.44.3:9090 </dev/null 2>/dev/null | openssl x509 -noout -ext subjectAltNam
Could not read certificate from <stdin>
4087895B58700000:error:1608010C:STORE routines:ossl_store_handle_load_result:unsupported:../crypto/store/store_result.c:151:
Unable to load certificate
```

For testing i installed 2 virtual machines with cloudstack-management and a new mysql-database on one of these. Both instances are connected on a layer 2 network.
Here the management servers (one after another) came once with fully functional certificates, both servers see each other as peers in the cs-ui management server overview.

* certificates SAN on one of the test-instances:
```
~# openssl s_client -connect 10.65.254.48:9090 </dev/null 2>/dev/null | openssl x509 -noout -ext subjectAltName
X509v3 Subject Alternative Name:
    IP Address:FE80:0:0:0:1C00:B1FF:FE00:164, IP Address:10.65.254.48, DNS:<redacted> DNS:cloudstack.internal
```

I then connected the former server with a fresh installation  to the database of my test-setup.
I got the same error as mentioned above.

I suspect some issue with the L3-network and retrieving some networking-stuff by cloudstack-management-server.

Any idea how to proceed here?


GitHub link: https://github.com/apache/cloudstack/discussions/11152

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] RootCAProvider error on management server with L3-network config [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
GitHub user weizhouapache added a comment to the discussion: RootCAProvider error on management server with L3-network config

> Yes, i' like to expect the ip "10.72.44.3" added to the certificate, in fact with your current ip route command it already gets the correct ip.

well, the command intends to get the default device name, but it returns the IP ... so it is unexpected

to support this special user case, some simple code changes are required I think

GitHub link: https://github.com/apache/cloudstack/discussions/11152#discussioncomment-13680356

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] RootCAProvider error on management server with L3-network config [cloudstack]

Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
GitHub user DaanHoogland added a comment to the discussion: RootCAProvider error on management server with L3-network config

@mosys0815 does this mean your issue is fixed?

GitHub link: https://github.com/apache/cloudstack/discussions/11152#discussioncomment-14612564

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] RootCAProvider error on management server with L3-network config [cloudstack]

Posted by "mosys0815 (via GitHub)" <gi...@apache.org>.
GitHub user mosys0815 added a comment to the discussion: RootCAProvider error on management server with L3-network config

here the output from management server start:
```
2025-07-14 07:32:55,999 INFO  [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean RootCAProvider.
2025-07-14 07:32:56,076 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Executing command [/bin/bash -c /bin/ip -j a | /bin/jq -r '.[] | .addr_info | map(select(.local == "'`/bin/ip -j r s default | /bin/jq -r '.[0] | .prefsrc'`'")) | .[].label' ].
2025-07-14 07:32:56,086 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Successfully executed process [3384384] for command [/bin/bash -c /bin/ip -j a | /bin/jq -r '.[] | .addr_info | map(select(.local == "'`/bin/ip -j r s default | /bin/jq -r '.[0] | .prefsrc'`'")) | .[].label' ].
2025-07-14 07:32:56,087 DEBUG [o.a.c.c.p.RootCAProvider] (main:[]) (logid:) Trying to find management IP in CIDR range [10.65.252.0/22].
2025-07-14 07:32:56,521 INFO  [o.a.c.c.p.RootCAProvider] (main:[]) (logid:) Creating new management server certificate and keystore
```

and the certificate
```
openssl s_client -connect 10.72.44.3:9090 </dev/null 2>/dev/null | openssl x509 -noout -ext subjectAltName
X509v3 Subject Alternative Name:
    IP Address:2A13:B240:2010:1302:0:0:0:1, IP Address:FE80:0:0:0:4002:62FF:FEBB:8E8E, IP Address:10.72.44.3, DNS:<redacted>, DNS:cloudstack.internal
```


GitHub link: https://github.com/apache/cloudstack/discussions/11152#discussioncomment-13749414

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] RootCAProvider error on management server with L3-network config [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
GitHub user weizhouapache added a comment to the discussion: RootCAProvider error on management server with L3-network config

> ```
> ~# ip route show default 0.0.0.0/0
> default proto bird src 10.72.44.3 metric 32
> 	nexthop via inet6 fe80::429e:a4ff:fe79:f50f dev eth1b weight 1
> 	nexthop via inet6 fe80::429e:a4ff:fe7b:8f0f dev eth1a weight 1
> default via 10.72.44.197 dev eth1a proto static src 10.72.44.198 metric 1024 onlink
> default via 10.72.45.197 dev eth1b proto static src 10.72.45.198 metric 1024 onlink
> ```

maybe we should add a `default via` (or at least ` via `) to make sure the command gets the default device, like

```
ip route show default 0.0.0.0/0 | grep ' via ' | head -1 | awk '{print $5}'
```

GitHub link: https://github.com/apache/cloudstack/discussions/11152#discussioncomment-13680352

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] RootCAProvider error on management server with L3-network config [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
GitHub user weizhouapache added a comment to the discussion: RootCAProvider error on management server with L3-network config

> But for now, is there anything I can do to get the certificate, even manually?

I do not think so, unless the command returns results like
```
$ ip route show default 0.0.0.0/0
default via 10.72.44.3 dev hostip
```

GitHub link: https://github.com/apache/cloudstack/discussions/11152#discussioncomment-13680358

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] RootCAProvider error on management server with L3-network config [cloudstack]

Posted by "mosys0815 (via GitHub)" <gi...@apache.org>.
GitHub user mosys0815 added a comment to the discussion: RootCAProvider error on management server with L3-network config

What exactly is the device used for?
In our setup we explicitly do _not_ want to use a fixed interface, since these are supposed to "failover" each other.

GitHub link: https://github.com/apache/cloudstack/discussions/11152#discussioncomment-13680353

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] RootCAProvider error on management server with L3-network config [cloudstack]

Posted by "mosys0815 (via GitHub)" <gi...@apache.org>.
GitHub user mosys0815 added a comment to the discussion: RootCAProvider error on management server with L3-network config

PR: https://github.com/apache/cloudstack/pull/11800

GitHub link: https://github.com/apache/cloudstack/discussions/11152#discussioncomment-15159113

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org