You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Andrei Mikhailovsky <an...@arhont.com.INVALID> on 2018/10/22 12:36:07 UTC
ACS 4.11.1.0 - agent.properties file became empty on a KVM host
Hi
I have an issue with one of the host servers. This issue is rather strange. Perhaps someone can help me with understanding how this happened and how to fix it.
About 3 days ago one of the KVM host servers ran out of disk space on its root partition. I have fixed the issue and reconnected the agent by running 'service cloudstack-agent restart'.
I've noticed that the host server is still showing Disconnected status in the web gui. The agent log files repeat the following 3 lines every 10 or so seconds:
2018-10-22 12:50:54,339 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2018-10-22 12:50:54,343 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.11.1.0
2018-10-22 12:50:54,345 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
looking further revealed that the file is 0 bytes:
-rw------- 1 root root 0 Oct 20 06:39 agent.properties
-rwxr-xr-x 1 root root 8890 Jul 6 14:01 agent.properties.dpkg-dist
Something has replaced the original agent.properties file. The creation/modification dates of the agent.properties file on other KVM host servers are all different (times and dates). As I always upgrade the host servers at the same time, this led me to believe that agent.properties file is automatically generated or modified by some script or service that is running on the host server or perhaps the modification is pushed from the management server to the agent.
As the server is in the Disconnected state I can't migrate servers and virtual routers from that host server and I can't set it to Maintenance either.
How do I manually force the creation / update of the agent.properties file on that host server? The challenge is that vms /vrs which are running on that host server are production servers and they should keep running without shutting down.
Thanks for any tips/help.
Andrei
Re: ACS 4.11.1.0 - agent.properties file became empty on a KVM host
Posted by Andrija Panic <an...@gmail.com>.
Hi Andrei,
upgrading packages (it should defaults to keep your current configuration
file ?) will usually save a .rpmsave file. or dpkg-old (like you already
have it - I assume during last upgrade...)
Anyway, so far, whenever you restart agent, it will get overwritten again
with just plain configuration lines in some order (i.e. you can create
comments and so on, but it will completely get overwritten) - I assume
something broke here (no use of assumptions obviously...) and config file
was not populated again.
Did you try to remove host from CloudStack and re-add it again from zero -
perhaps that would solve the problem?
Cheers
On Mon, 22 Oct 2018 at 15:42, Andrei Mikhailovsky <an...@arhont.com.invalid>
wrote:
> Hi Gabriel,
>
> thanks for your reply. What you've suggested will create a default
> agent.properties file, which is no good. The agent will not connect to the
> server with the default agent.properties file for many reasons.
>
> I think I can recreate most of the file content by looking at the other
> agent.properties files. However, one thing that I am missing is the:
>
> keystore.passphrase=
>
> Where do I get the passphrase for the keystore file? is it stored
> somewhere in db of the management server or on the KVM host itself?
>
> Thanks
>
> ----- Original Message -----
> > From: "Gabriel Beims Bräscher" <ga...@gmail.com>
> > To: "users" <us...@cloudstack.apache.org>
> > Sent: Monday, 22 October, 2018 13:54:32
> > Subject: Re: ACS 4.11.1.0 - agent.properties file became empty on a KVM
> host
>
> > Hi Andrei,
> >
> > When upgrading the CloudStack agent you can accept or refuse to change
> the
> > agent.properties. The default operation is to not change configuration
> > files.
> >
> > The agent service does not impact directly on the running VMs; thus, I
> > would suggest you remove the /etc/cloudstack/agent/agent.properties file,
> > uninstall the CloudStack agent service and reinstall it. I would also
> > suggest you keep a copy of /etc/cloudstack/ just to have a saved state of
> > your agent configurations before reinstalling and compare them if needed.
> >
> > Em seg, 22 de out de 2018 às 09:36, Andrei Mikhailovsky
> > <an...@arhont.com.invalid> escreveu:
> >
> >> Hi
> >>
> >> I have an issue with one of the host servers. This issue is rather
> >> strange. Perhaps someone can help me with understanding how this
> happened
> >> and how to fix it.
> >>
> >> About 3 days ago one of the KVM host servers ran out of disk space on
> its
> >> root partition. I have fixed the issue and reconnected the agent by
> running
> >> 'service cloudstack-agent restart'.
> >>
> >> I've noticed that the host server is still showing Disconnected status
> in
> >> the web gui. The agent log files repeat the following 3 lines every 10
> or
> >> so seconds:
> >>
> >> 2018-10-22 12:50:54,339 INFO [cloud.agent.AgentShell] (main:null)
> (logid:)
> >> Agent started
> >> 2018-10-22 12:50:54,343 INFO [cloud.agent.AgentShell] (main:null)
> (logid:)
> >> Implementation Version is 4.11.1.0
> >> 2018-10-22 12:50:54,345 INFO [cloud.agent.AgentShell] (main:null)
> (logid:)
> >> agent.properties found at /etc/cloudstack/agent/agent.properties
> >>
> >>
> >> looking further revealed that the file is 0 bytes:
> >>
> >> -rw------- 1 root root 0 Oct 20 06:39 agent.properties
> >> -rwxr-xr-x 1 root root 8890 Jul 6 14:01 agent.properties.dpkg-dist
> >>
> >> Something has replaced the original agent.properties file. The
> >> creation/modification dates of the agent.properties file on other KVM
> host
> >> servers are all different (times and dates). As I always upgrade the
> host
> >> servers at the same time, this led me to believe that agent.properties
> file
> >> is automatically generated or modified by some script or service that is
> >> running on the host server or perhaps the modification is pushed from
> the
> >> management server to the agent.
> >>
> >> As the server is in the Disconnected state I can't migrate servers and
> >> virtual routers from that host server and I can't set it to Maintenance
> >> either.
> >>
> >> How do I manually force the creation / update of the agent.properties
> file
> >> on that host server? The challenge is that vms /vrs which are running on
> >> that host server are production servers and they should keep running
> >> without shutting down.
> >>
> >> Thanks for any tips/help.
> >>
> >> Andrei
>
--
Andrija Panić
Re: ACS 4.11.1.0 - agent.properties file became empty on a KVM host
Posted by Andrei Mikhailovsky <an...@arhont.com.INVALID>.
Right, I've managed to fix the issue. Here is how I've done it.
1. Copied the contents of the agent.properties file from another KVM host.
2. Removed the values in the fields like: guid and keystore.passphrase
3. Set the ca.plugin.root.auth.strictness to false in Global Settings. Restarted the management server
4. Started the agent. This created a new host entry with Unsecure status. The old host entry was still showing Disconnecrted.
5. Provisioned the new keys
6. Removed the old host entry with the Force option ticked
7. Reverted the setting in 3 above
8. Restarted the management server
9. Job done
Cheers
P.S. I think I've figured out what caused the agent.properties file to be modified. After I've done point 3 above, I've noticed that the agent.properties file on ALL my host servers have been modified (at least according to the modified time stamp). Now, it seems that the workflow for performing this action is broken - very silly indeed!!! Will send a message to the mailing list. Perhaps the responsible developers for this code will fix the issue and create a few more sanity checks.
----- Original Message -----
> From: "Andrei Mikhailovsky" <an...@arhont.com.INVALID>
> To: "users" <us...@cloudstack.apache.org>
> Sent: Monday, 22 October, 2018 14:42:12
> Subject: Re: ACS 4.11.1.0 - agent.properties file became empty on a KVM host
> Hi Gabriel,
>
> thanks for your reply. What you've suggested will create a default
> agent.properties file, which is no good. The agent will not connect to the
> server with the default agent.properties file for many reasons.
>
> I think I can recreate most of the file content by looking at the other
> agent.properties files. However, one thing that I am missing is the:
>
> keystore.passphrase=
>
> Where do I get the passphrase for the keystore file? is it stored somewhere in
> db of the management server or on the KVM host itself?
>
> Thanks
>
> ----- Original Message -----
>> From: "Gabriel Beims Bräscher" <ga...@gmail.com>
>> To: "users" <us...@cloudstack.apache.org>
>> Sent: Monday, 22 October, 2018 13:54:32
>> Subject: Re: ACS 4.11.1.0 - agent.properties file became empty on a KVM host
>
>> Hi Andrei,
>>
>> When upgrading the CloudStack agent you can accept or refuse to change the
>> agent.properties. The default operation is to not change configuration
>> files.
>>
>> The agent service does not impact directly on the running VMs; thus, I
>> would suggest you remove the /etc/cloudstack/agent/agent.properties file,
>> uninstall the CloudStack agent service and reinstall it. I would also
>> suggest you keep a copy of /etc/cloudstack/ just to have a saved state of
>> your agent configurations before reinstalling and compare them if needed.
>>
>> Em seg, 22 de out de 2018 às 09:36, Andrei Mikhailovsky
>> <an...@arhont.com.invalid> escreveu:
>>
>>> Hi
>>>
>>> I have an issue with one of the host servers. This issue is rather
>>> strange. Perhaps someone can help me with understanding how this happened
>>> and how to fix it.
>>>
>>> About 3 days ago one of the KVM host servers ran out of disk space on its
>>> root partition. I have fixed the issue and reconnected the agent by running
>>> 'service cloudstack-agent restart'.
>>>
>>> I've noticed that the host server is still showing Disconnected status in
>>> the web gui. The agent log files repeat the following 3 lines every 10 or
>>> so seconds:
>>>
>>> 2018-10-22 12:50:54,339 INFO [cloud.agent.AgentShell] (main:null) (logid:)
>>> Agent started
>>> 2018-10-22 12:50:54,343 INFO [cloud.agent.AgentShell] (main:null) (logid:)
>>> Implementation Version is 4.11.1.0
>>> 2018-10-22 12:50:54,345 INFO [cloud.agent.AgentShell] (main:null) (logid:)
>>> agent.properties found at /etc/cloudstack/agent/agent.properties
>>>
>>>
>>> looking further revealed that the file is 0 bytes:
>>>
>>> -rw------- 1 root root 0 Oct 20 06:39 agent.properties
>>> -rwxr-xr-x 1 root root 8890 Jul 6 14:01 agent.properties.dpkg-dist
>>>
>>> Something has replaced the original agent.properties file. The
>>> creation/modification dates of the agent.properties file on other KVM host
>>> servers are all different (times and dates). As I always upgrade the host
>>> servers at the same time, this led me to believe that agent.properties file
>>> is automatically generated or modified by some script or service that is
>>> running on the host server or perhaps the modification is pushed from the
>>> management server to the agent.
>>>
>>> As the server is in the Disconnected state I can't migrate servers and
>>> virtual routers from that host server and I can't set it to Maintenance
>>> either.
>>>
>>> How do I manually force the creation / update of the agent.properties file
>>> on that host server? The challenge is that vms /vrs which are running on
>>> that host server are production servers and they should keep running
>>> without shutting down.
>>>
>>> Thanks for any tips/help.
>>>
> >> Andrei
Re: ACS 4.11.1.0 - agent.properties file became empty
on a KVM host
Posted by Rohit Yadav <ro...@shapeblue.com>.
Hi Andrei,
The keystore.passphrase is a randomly generated passphrase that only the agent knows about. If you've lost it, simply re-add the KVM host.
- Rohit
<https://cloudstack.apache.org>
________________________________
From: Andrei Mikhailovsky <an...@arhont.com.INVALID>
Sent: Monday, October 22, 2018 7:12:12 PM
To: users
Subject: Re: ACS 4.11.1.0 - agent.properties file became empty on a KVM host
Hi Gabriel,
thanks for your reply. What you've suggested will create a default agent.properties file, which is no good. The agent will not connect to the server with the default agent.properties file for many reasons.
I think I can recreate most of the file content by looking at the other agent.properties files. However, one thing that I am missing is the:
keystore.passphrase=
Where do I get the passphrase for the keystore file? is it stored somewhere in db of the management server or on the KVM host itself?
Thanks
rohit.yadav@shapeblue.com
www.shapeblue.com
Amadeus House, Floral Street, London WC2E 9DPUK
@shapeblue
----- Original Message -----
> From: "Gabriel Beims Bräscher" <ga...@gmail.com>
> To: "users" <us...@cloudstack.apache.org>
> Sent: Monday, 22 October, 2018 13:54:32
> Subject: Re: ACS 4.11.1.0 - agent.properties file became empty on a KVM host
> Hi Andrei,
>
> When upgrading the CloudStack agent you can accept or refuse to change the
> agent.properties. The default operation is to not change configuration
> files.
>
> The agent service does not impact directly on the running VMs; thus, I
> would suggest you remove the /etc/cloudstack/agent/agent.properties file,
> uninstall the CloudStack agent service and reinstall it. I would also
> suggest you keep a copy of /etc/cloudstack/ just to have a saved state of
> your agent configurations before reinstalling and compare them if needed.
>
> Em seg, 22 de out de 2018 às 09:36, Andrei Mikhailovsky
> <an...@arhont.com.invalid> escreveu:
>
>> Hi
>>
>> I have an issue with one of the host servers. This issue is rather
>> strange. Perhaps someone can help me with understanding how this happened
>> and how to fix it.
>>
>> About 3 days ago one of the KVM host servers ran out of disk space on its
>> root partition. I have fixed the issue and reconnected the agent by running
>> 'service cloudstack-agent restart'.
>>
>> I've noticed that the host server is still showing Disconnected status in
>> the web gui. The agent log files repeat the following 3 lines every 10 or
>> so seconds:
>>
>> 2018-10-22 12:50:54,339 INFO [cloud.agent.AgentShell] (main:null) (logid:)
>> Agent started
>> 2018-10-22 12:50:54,343 INFO [cloud.agent.AgentShell] (main:null) (logid:)
>> Implementation Version is 4.11.1.0
>> 2018-10-22 12:50:54,345 INFO [cloud.agent.AgentShell] (main:null) (logid:)
>> agent.properties found at /etc/cloudstack/agent/agent.properties
>>
>>
>> looking further revealed that the file is 0 bytes:
>>
>> -rw------- 1 root root 0 Oct 20 06:39 agent.properties
>> -rwxr-xr-x 1 root root 8890 Jul 6 14:01 agent.properties.dpkg-dist
>>
>> Something has replaced the original agent.properties file. The
>> creation/modification dates of the agent.properties file on other KVM host
>> servers are all different (times and dates). As I always upgrade the host
>> servers at the same time, this led me to believe that agent.properties file
>> is automatically generated or modified by some script or service that is
>> running on the host server or perhaps the modification is pushed from the
>> management server to the agent.
>>
>> As the server is in the Disconnected state I can't migrate servers and
>> virtual routers from that host server and I can't set it to Maintenance
>> either.
>>
>> How do I manually force the creation / update of the agent.properties file
>> on that host server? The challenge is that vms /vrs which are running on
>> that host server are production servers and they should keep running
>> without shutting down.
>>
>> Thanks for any tips/help.
>>
>> Andrei
Re: ACS 4.11.1.0 - agent.properties file became empty on a KVM host
Posted by Andrei Mikhailovsky <an...@arhont.com.INVALID>.
Hi Gabriel,
thanks for your reply. What you've suggested will create a default agent.properties file, which is no good. The agent will not connect to the server with the default agent.properties file for many reasons.
I think I can recreate most of the file content by looking at the other agent.properties files. However, one thing that I am missing is the:
keystore.passphrase=
Where do I get the passphrase for the keystore file? is it stored somewhere in db of the management server or on the KVM host itself?
Thanks
----- Original Message -----
> From: "Gabriel Beims Bräscher" <ga...@gmail.com>
> To: "users" <us...@cloudstack.apache.org>
> Sent: Monday, 22 October, 2018 13:54:32
> Subject: Re: ACS 4.11.1.0 - agent.properties file became empty on a KVM host
> Hi Andrei,
>
> When upgrading the CloudStack agent you can accept or refuse to change the
> agent.properties. The default operation is to not change configuration
> files.
>
> The agent service does not impact directly on the running VMs; thus, I
> would suggest you remove the /etc/cloudstack/agent/agent.properties file,
> uninstall the CloudStack agent service and reinstall it. I would also
> suggest you keep a copy of /etc/cloudstack/ just to have a saved state of
> your agent configurations before reinstalling and compare them if needed.
>
> Em seg, 22 de out de 2018 às 09:36, Andrei Mikhailovsky
> <an...@arhont.com.invalid> escreveu:
>
>> Hi
>>
>> I have an issue with one of the host servers. This issue is rather
>> strange. Perhaps someone can help me with understanding how this happened
>> and how to fix it.
>>
>> About 3 days ago one of the KVM host servers ran out of disk space on its
>> root partition. I have fixed the issue and reconnected the agent by running
>> 'service cloudstack-agent restart'.
>>
>> I've noticed that the host server is still showing Disconnected status in
>> the web gui. The agent log files repeat the following 3 lines every 10 or
>> so seconds:
>>
>> 2018-10-22 12:50:54,339 INFO [cloud.agent.AgentShell] (main:null) (logid:)
>> Agent started
>> 2018-10-22 12:50:54,343 INFO [cloud.agent.AgentShell] (main:null) (logid:)
>> Implementation Version is 4.11.1.0
>> 2018-10-22 12:50:54,345 INFO [cloud.agent.AgentShell] (main:null) (logid:)
>> agent.properties found at /etc/cloudstack/agent/agent.properties
>>
>>
>> looking further revealed that the file is 0 bytes:
>>
>> -rw------- 1 root root 0 Oct 20 06:39 agent.properties
>> -rwxr-xr-x 1 root root 8890 Jul 6 14:01 agent.properties.dpkg-dist
>>
>> Something has replaced the original agent.properties file. The
>> creation/modification dates of the agent.properties file on other KVM host
>> servers are all different (times and dates). As I always upgrade the host
>> servers at the same time, this led me to believe that agent.properties file
>> is automatically generated or modified by some script or service that is
>> running on the host server or perhaps the modification is pushed from the
>> management server to the agent.
>>
>> As the server is in the Disconnected state I can't migrate servers and
>> virtual routers from that host server and I can't set it to Maintenance
>> either.
>>
>> How do I manually force the creation / update of the agent.properties file
>> on that host server? The challenge is that vms /vrs which are running on
>> that host server are production servers and they should keep running
>> without shutting down.
>>
>> Thanks for any tips/help.
>>
>> Andrei
Re: ACS 4.11.1.0 - agent.properties file became empty on a KVM host
Posted by Gabriel Beims Bräscher <ga...@gmail.com>.
Hi Andrei,
When upgrading the CloudStack agent you can accept or refuse to change the
agent.properties. The default operation is to not change configuration
files.
The agent service does not impact directly on the running VMs; thus, I
would suggest you remove the /etc/cloudstack/agent/agent.properties file,
uninstall the CloudStack agent service and reinstall it. I would also
suggest you keep a copy of /etc/cloudstack/ just to have a saved state of
your agent configurations before reinstalling and compare them if needed.
Em seg, 22 de out de 2018 às 09:36, Andrei Mikhailovsky
<an...@arhont.com.invalid> escreveu:
> Hi
>
> I have an issue with one of the host servers. This issue is rather
> strange. Perhaps someone can help me with understanding how this happened
> and how to fix it.
>
> About 3 days ago one of the KVM host servers ran out of disk space on its
> root partition. I have fixed the issue and reconnected the agent by running
> 'service cloudstack-agent restart'.
>
> I've noticed that the host server is still showing Disconnected status in
> the web gui. The agent log files repeat the following 3 lines every 10 or
> so seconds:
>
> 2018-10-22 12:50:54,339 INFO [cloud.agent.AgentShell] (main:null) (logid:)
> Agent started
> 2018-10-22 12:50:54,343 INFO [cloud.agent.AgentShell] (main:null) (logid:)
> Implementation Version is 4.11.1.0
> 2018-10-22 12:50:54,345 INFO [cloud.agent.AgentShell] (main:null) (logid:)
> agent.properties found at /etc/cloudstack/agent/agent.properties
>
>
> looking further revealed that the file is 0 bytes:
>
> -rw------- 1 root root 0 Oct 20 06:39 agent.properties
> -rwxr-xr-x 1 root root 8890 Jul 6 14:01 agent.properties.dpkg-dist
>
> Something has replaced the original agent.properties file. The
> creation/modification dates of the agent.properties file on other KVM host
> servers are all different (times and dates). As I always upgrade the host
> servers at the same time, this led me to believe that agent.properties file
> is automatically generated or modified by some script or service that is
> running on the host server or perhaps the modification is pushed from the
> management server to the agent.
>
> As the server is in the Disconnected state I can't migrate servers and
> virtual routers from that host server and I can't set it to Maintenance
> either.
>
> How do I manually force the creation / update of the agent.properties file
> on that host server? The challenge is that vms /vrs which are running on
> that host server are production servers and they should keep running
> without shutting down.
>
> Thanks for any tips/help.
>
> Andrei
>