You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Antoine Boucher <an...@haltondc.com> on 2022/05/11 15:41:08 UTC
NPE while listing Snapshots
Hello Suresh,
That primary storage was part of the local storage of a host that was removed.
SELECT name, uuid, pool_type, scope, created, removed FROM cloud.storage_pool WHERE id = 5
+----------------+------+------------+-------+---------------------+---------------------+
| name | uuid | pool_type | scope | created | removed |
+----------------+------+------------+-------+---------------------+---------------------+
| cs-kvm03-local | NULL | Filesystem | HOST | 2022-02-12 15:01:45 | 2022-04-19 02:11:51 |
+----------------+------+------------+-------+---------------------+——————————+
SELECT volume_id, store_id, snapshot_id, store_role, install_path, size, state FROM cloud.snapshot_store_ref WHERE store_id = '5';
+-----------+----------+-------------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
| volume_id | store_id | snapshot_id | store_role | install_path | size | state |
+-----------+----------+-------------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
| 155 | 5 | 109 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/66210b31-7445-4586-89ea-52a37f5b1ab5 | 85899345920 | Ready |
| 155 | 5 | 117 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/a2c4fbbb-12b2-4c43-9647-375f6e334a46 | 85899345920 | Ready |
| 155 | 5 | 127 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/794d6dff-ba45-4a56-8377-1e48034bf014 | 85899345920 | Ready |
| 155 | 5 | 138 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/38648d51-bb19-49a2-b01f-12c4e6ed4529 | 85899345920 | Ready |
+-----------+----------+-------------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
All 155 snapshots have the same account_id and domain_id.
SELECT id, uuid, name, status, path, created, removed, location_type FROM cloud.snapshots WHERE volume_id = '155';
+-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------------------+---------------+
| id | uuid | name | status | path | created | removed | location_type |
+-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------------------+---------------+
| 101 | 13870bbd-d7d7-4664-a22f-f09c80a56d41 | TS01_ROOT-110_20220414054855 | Destroyed | NULL | 2022-04-14 05:48:55 | NULL | NULL |
| 109 | 77d84bf6-f601-44a6-a239-821f735a7f48 | TS01_ROOT-110_20220415054855 | BackedUp | NULL | 2022-04-15 05:48:55 | NULL | NULL |
| 117 | 7db02f43-a9d8-4c73-905a-1524d90f570a | TS01_ROOT-110_20220416054855 | BackedUp | NULL | 2022-04-16 05:48:55 | NULL | NULL |
| 127 | 47288ab9-2da2-40f2-98e3-9ced3f8e87df | TS01_ROOT-110_20220417054855 | BackedUp | NULL | 2022-04-17 05:48:55 | NULL | NULL |
| 138 | e9639f4a-fa09-4698-849b-572011045993 | TS01_ROOT-110_20220418054855 | BackedUp | NULL | 2022-04-18 05:48:55 | NULL | NULL |
| 146 | 875d0298-6ed7-43fc-a468-da6042388ac0 | TS01_ROOT-110_20220419054747 | BackedUp | NULL | 2022-04-19 05:47:47 | NULL | NULL |
| 154 | 0122c9b3-a9f8-473f-a8d7-6e69ba77265b | TS01_ROOT-110_20220420054748 | Destroyed | NULL | 2022-04-20 05:47:48 | NULL | NULL |
| 157 | fe2248c6-aa80-4e9f-b539-7f495f1ff39f | Manual - Apr 20 | Destroyed | NULL | 2022-04-21 01:50:16 | NULL | NULL |
| 163 | 32923ad7-cbe3-4708-95d8-23eca921d711 | TS01_ROOT-110_20220421054817 | Destroyed | NULL | 2022-04-21 05:48:17 | NULL | NULL |
| 172 | 6e03ea90-e8b0-4405-aed6-7bd17a40de57 | TS01_ROOT-110_20220422054933 | Destroyed | NULL | 2022-04-22 05:49:33 | NULL | NULL |
| 181 | 9059a26a-0e4e-4d46-95f2-b0a9236ba90a | TS01_ROOT-110_20220423054933 | Destroyed | NULL | 2022-04-23 05:49:33 | NULL | NULL |
| 193 | c9828cbc-f282-4dfe-b4cf-9df792c522c2 | TS01_ROOT-110_20220424054934 | BackedUp | NULL | 2022-04-24 05:49:34 | NULL | NULL |
| 207 | 2e5dfd48-c4de-45ff-bbe1-4cf0d90bb698 | TS01_ROOT-110_20220425054934 | Destroyed | NULL | 2022-04-25 05:49:34 | NULL | NULL |
| 217 | 99d9d9ff-44ae-41a9-9761-f722b370c01c | TS01_ROOT-110_20220426054934 | Destroyed | NULL | 2022-04-26 05:49:34 | NULL | NULL |
| 227 | 1df14a7b-ba56-4d26-aeb4-c79d4de978fc | TS01_ROOT-110_20220427054934 | BackedUp | NULL | 2022-04-27 05:49:34 | NULL | NULL |
| 237 | 657479e0-3587-4a72-a8c8-bd7dff54a5e4 | TS01_ROOT-110_20220428054934 | BackedUp | NULL | 2022-04-28 05:49:34 | NULL | NULL |
| 248 | 2ef0b539-be01-4b0c-9fa5-5fc9c60a6a96 | TS01_ROOT-110_20220429054935 | BackedUp | NULL | 2022-04-29 05:49:35 | NULL | NULL |
| 257 | 8ab75270-8e3d-471e-8e4a-8d5f3cbc6bc1 | TS01_ROOT-110_20220430054935 | BackedUp | NULL | 2022-04-30 05:49:35 | NULL | NULL |
| 259 | a1e77bec-8fc7-457f-b0a5-961c339487f7 | TS01_ROOT-110_20220430060935 | Destroyed | NULL | 2022-04-30 06:09:35 | 2022-05-09 20:02:38 | NULL |
| 313 | 8ad0c04a-1d74-4d51-8223-a6aa3a82d62c | TS01_ROOT-110_20220509015742 | BackedUp | NULL | 2022-05-09 01:57:42 | NULL | NULL |
| 327 | 0f5e9366-747f-4a34-bec4-1fd31c461ac3 | TS01_ROOT-110_20220509170242 | BackedUp | NULL | 2022-05-09 17:02:42 | NULL | NULL |
+-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------------------+---------------+
How would you suggest I resolve the issue moving forward?
I tried the following with error…
(nimbus) 🐱 > delete snapshot id=109
{
"accountid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838",
"cmd": "org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd",
"completed": "2022-05-11T11:32:08-0400",
"created": "2022-05-11T11:32:08-0400",
"jobid": "29661b2b-46c5-42bd-9974-d1a226f3fe68",
"jobinstanceid": "77d84bf6-f601-44a6-a239-821f735a7f48",
"jobinstancetype": "Snapshot",
"jobprocstatus": 0,
"jobresult": {
"errorcode": 530,
"errortext": "Command failed due to Internal Server Error"
},
"jobresultcode": 530,
"jobresulttype": "object",
"jobstatus": 2,
"userid": "2cefb708-8bc4-11ec-9c43-001e67fd4838"
}
🙈 Error: async API failed for job 29661b2b-46c5-42bd-9974-d1a226f3fe68
Deleting some table rows in the right sequence?
Same issue with id=117 to id=138, but I was able to delete id=146
Regards,
Antoine
> On Apr 27, 2022, at 9:00 AM, Suresh Anaparti <Su...@shapeblue.com> wrote:
>
> Hi Antoine,
>
> The NPE error seems to be due to the primary storage with id '5', when checking for snapshot revertible or not during list snapshots cmd. Is that primary storage removed? Can you check with sql below.
>
> SELECT name, uuid, pool_type, scope, created, removed FROM cloud.storage_pool WHERE id = 5
>
>
> I think, the account/user might not have the permissions to list all snapshots. You can check the account / domain details in snapshots table.
>
>
> Regards,
> Suresh
>
> On 21/04/22, 6:50 PM, "Antoine Boucher" <an...@haltondc.com> wrote:
>
> Thank you Suresh, you will find my answers below,
>
>
>
>
>
>> On Apr 21, 2022, at 1:02 AM, Suresh Anaparti <Su...@shapeblue.com> wrote:
>>
>> Hi Antoine,
>>
>> What is the CloudStack version, where you see this issue.
>
> Latest - 4.16.1
>
>
>> When you noticed this issue, is it while listing snapshots for a particular volume, or all volumes? Try list snapshots from API/cmk as well.
>
> This is a customer VM, the account may have hit its snapshot limit during a snapshot.
>
> cmk list snapshots > only lists 26 snapshots out of 62
>
> cmk list snapshots volumeid= > Works for some volume ids, provides null result while the WebUI lists the correct snapshots for others and provides null results while show undefined error for one VM mentioned below
>
>
>>
>> Also, Check the snapshots details for the volume (with issue) in the db, using the following sql queries.
>>
>> 1. SELECT id, uuid, state, pool_id, path, size, removed FROM cloud.volumes WHERE name LIKE '<volume_name>'
>>
>
> mysql> SELECT id, uuid, state, pool_id, path, size, removed FROM cloud.volumes WHERE name LIKE 'ROOT-110';
> +-----+--------------------------------------+----------+---------+--------------------------------------+-------------+---------------------+
> | id | uuid | state | pool_id | path | size | removed |
> +-----+--------------------------------------+----------+---------+--------------------------------------+-------------+---------------------+
> | 111 | NULL | Expunged | 5 | da585fd4-b10f-480b-9287-45ecb28b6355 | 85899345920 | 2022-04-19 01:29:08 |
> | 138 | da585fd4-b10f-480b-9287-45ecb28b6355 | Ready | 6 | 04bd5d13-72ca-4240-93ce-ed1b015d3437 | 85899345920 | NULL |
> +-----+--------------------------------------+----------+---------+--------------------------------------+-------------+---------------------+
>
>
>> 2. SELECT id, uuid, name, status, path, created, removed, location_type FROM cloud.snapshots WHERE volume_id = <volume_id_from_step_1>;
>
> mysql> SELECT id, uuid, name, status, path, created, removed, location_type FROM cloud.snapshots WHERE volume_id = '138';
> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------+---------------+
> | id | uuid | name | status | path | created | removed | location_type |
> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------+---------------+
> | 101 | 13870bbd-d7d7-4664-a22f-f09c80a56d41 | TS01_ROOT-110_20220414054855 | Destroyed | NULL | 2022-04-14 05:48:55 | NULL | NULL |
> | 109 | 77d84bf6-f601-44a6-a239-821f735a7f48 | TS01_ROOT-110_20220415054855 | BackedUp | NULL | 2022-04-15 05:48:55 | NULL | NULL |
> | 117 | 7db02f43-a9d8-4c73-905a-1524d90f570a | TS01_ROOT-110_20220416054855 | BackedUp | NULL | 2022-04-16 05:48:55 | NULL | NULL |
> | 127 | 47288ab9-2da2-40f2-98e3-9ced3f8e87df | TS01_ROOT-110_20220417054855 | BackedUp | NULL | 2022-04-17 05:48:55 | NULL | NULL |
> | 138 | e9639f4a-fa09-4698-849b-572011045993 | TS01_ROOT-110_20220418054855 | BackedUp | NULL | 2022-04-18 05:48:55 | NULL | NULL |
> | 146 | 875d0298-6ed7-43fc-a468-da6042388ac0 | TS01_ROOT-110_20220419054747 | BackedUp | NULL | 2022-04-19 05:47:47 | NULL | NULL |
> | 154 | 0122c9b3-a9f8-473f-a8d7-6e69ba77265b | TS01_ROOT-110_20220420054748 | BackedUp | NULL | 2022-04-20 05:47:48 | NULL | NULL |
> | 157 | fe2248c6-aa80-4e9f-b539-7f495f1ff39f | Manual - Apr 20 | BackedUp | NULL | 2022-04-21 01:50:16 | NULL | NULL |
> | 163 | 32923ad7-cbe3-4708-95d8-23eca921d711 | TS01_ROOT-110_20220421054817 | BackedUp | NULL | 2022-04-21 05:48:17 | NULL | NULL |
> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------+---------------+
>
>
>>
>> 3. SELECT volume_id, store_id, store_role, install_path, size, state FROM cloud.snapshot_store_ref WHERE snapshot_id = <snapshot_id_from_step_2>
>>
>
> mysql> SELECT volume_id, store_id, store_role, install_path, size, state FROM cloud.snapshot_store_ref WHERE snapshot_id = '101';
> +-----------+----------+------------+-------------------------------------------------------+-------------+------------+
> | volume_id | store_id | store_role | install_path | size | state |
> +-----------+----------+------------+-------------------------------------------------------+-------------+------------+
> | 138 | 1 | Image | snapshots/16/111/a6b1bead-ca9a-464b-82c1-d631611ad790 | 85899345920 | Destroying |
> +-----------+----------+------------+-------------------------------------------------------+-------------+------------+
> 1 row in set (0.00 sec)
>
> mysql> SELECT volume_id, store_id, store_role, install_path, size, state FROM cloud.snapshot_store_ref WHERE snapshot_id = '109';
> +-----------+----------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
> | volume_id | store_id | store_role | install_path | size | state |
> +-----------+----------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
> | 138 | 5 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/66210b31-7445-4586-89ea-52a37f5b1ab5 | 85899345920 | Ready |
> | 138 | 1 | Image | snapshots/16/111/66210b31-7445-4586-89ea-52a37f5b1ab5 | 85899345920 | Ready |
> +-----------+----------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
> 2 rows in set (0.00 sec)
>
> Regards,
> Antoine
>
>> Regards,
>> Suresh
>>
>> On 21/04/22, 8:45 AM, "Antoine Boucher" <an...@haltondc.com> wrote:
>>
>> Unfortunately I spoke too fast. The list is back but listing the snapshots of the volume with issue still show a blank list with error “undefined”.
>>
>>
>>
>>
>> On Apr 20, 2022, at 23:11, Antoine Boucher <an...@haltondc.com> wrote:
>>
>> I was able to recover from the situation by forcing a snapshot on the volume with issue.
>>
>> On Apr 20, 2022, at 22:00, Antoine Boucher <an...@haltondc.com> wrote:
>>
>> I pin-pointed the volume with the issue by going through all volumes and clicking the “view snapshot"
>>
>>
>>> On Apr 20, 2022, at 9:35 PM, Antoine Boucher <an...@haltondc.com> wrote:
>>>
>>> I’m no longer able to see my list of snapshots (storage > snapshots), instead I see a small popup window with the word “undefined" on an empty list snapshot list.
>>>
>>> Has anyone seen this issue? I restart and rebooted the Management server but no resolution.
>>>
>>> Here are my logs:
>>>
>>> 2022-04-20 21:17:02,428 DEBUG [c.c.a.ApiServlet] (qtp515715487-292:ctx-a220f915) (logid:624389ae) ===START=== 10.101.254.1 -- GET listall=true&page=1&pagesize=20&command=listSnapshots&response=json
>>> 2022-04-20 21:17:02,439 DEBUG [c.c.a.ApiServer] (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) CIDRs from which account 'Acct[2cee75f9-8bc4-11ec-9c43-001e67fd4838-admin] -- Account {"id": 2, "name": "admin", "uuid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838"}' is allowed to perform API calls: 0.0.0.0/0,::/0
>>> 2022-04-20 21:17:02,675 ERROR [c.c.a.ApiServer] (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) unhandled exception executing api command: [Ljava.lang.String;@42fa8f7d
>>> java.lang.NullPointerException
>>> at org.apache.cloudstack.storage.snapshot.StorageSystemSnapshotStrategy.canHandle(StorageSystemSnapshotStrategy.java:985)
>>> at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:72)
>>> at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:69)
>>> at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.bestMatch(StorageStrategyFactoryImpl.java:95)
>>> at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.getSnapshotStrategy(StorageStrategyFactoryImpl.java:69)
>>> at org.apache.cloudstack.storage.snapshot.SnapshotObject.isRevertable(SnapshotObject.java:156)
>>> at com.cloud.api.ApiResponseHelper.createSnapshotResponse(ApiResponseHelper.java:591)
>>> at org.apache.cloudstack.api.command.user.snapshot.ListSnapshotsCmd.execute(ListSnapshotsCmd.java:117)
>>> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
>>> at com.cloud.api.ApiServer.queueCommand(ApiServer.java:772)
>>> at com.cloud.api.ApiServer.handleRequest(ApiServer.java:596)
>>> at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:321)
>>> at com.cloud.api.ApiServlet$1.run(ApiServlet.java:134)
>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>> at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:131)
>>> at com.cloud.api.ApiServlet.doGet(ApiServlet.java:93)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
>>> at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
>>> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
>>> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
>>> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>>> at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
>>> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
>>> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
>>> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
>>> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
>>> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
>>> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
>>> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
>>> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
>>> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
>>> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
>>> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>>> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
>>> at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
>>> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
>>> at org.eclipse.jetty.server.Server.handle(Server.java:516)
>>> at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400)
>>> at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645)
>>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392)
>>> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
>>> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
>>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
>>> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
>>> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
>>> at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
>>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
>>> at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
>>> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
>>> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
>>> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
>>> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
>>> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
>>> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
>>> at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>> 2022-04-20 21:17:02,677 DEBUG [c.c.a.ApiServlet] (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) ===END=== 10.101.254.1 -- GET listall=true&page=1&pagesize=20&command=listSnapshots&response=json
>>>
>>> Antoine Boucher
>>>
>>>
>>> Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential, and may be privileged. If you are not the intended recipient, you are hereby notified that any review, retransmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, and delete this message and any attachments from your system.
>>>
>>
>>
>
>
Re: NPE while listing Snapshots
Posted by Suresh Anaparti <Su...@shapeblue.com>.
Hi Antoine,
Thanks for the update, good to know that no more issues with snapshots listing.
The query result confirms why the snapshot location for the volume ids 110 and 112 exists, these volumes are not yet removed.
Regards,
Suresh
From: Antoine Boucher <an...@haltondc.com>
Reply to: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>
Date: Friday, 27 May 2022 at 10:09 PM
To: users <us...@cloudstack.apache.org>
Subject: Re: NPE while listing Snapshots
Hello Suresh,
I did your 2 commands on all 4 ids and I have no more issues on the listing the Snapshot from the webUI.
Here is the result of the query you requested:
mysql> SELECT id, uuid, name, account_id, pool_id, path, size, volume_type, state, removed FROM cloud.volumes WHERE id IN (110, 111, 112, 155);
+-----+--------------------------------------+----------+------------+---------+--------------------------------------+--------------+-------------+----------+---------------------+
| id | uuid | name | account_id | pool_id | path | size | volume_type | state | removed |
+-----+--------------------------------------+----------+------------+---------+--------------------------------------+--------------+-------------+----------+---------------------+
| 110 | 390198b4-45d4-425e-87c3-d0f11ef2ca11 | ROOT-109 | 16 | 4 | 390198b4-45d4-425e-87c3-d0f11ef2ca11 | 42949672960 | ROOT | Ready | NULL |
| 111 | NULL | ROOT-110 | 16 | 5 | da585fd4-b10f-480b-9287-45ecb28b6355 | 85899345920 | DATADISK | Expunged | 2022-04-19 01:29:08 |
| 112 | 285140e4-555e-4cac-a2ec-afb4bd6921a8 | ROOT-111 | 16 | 3 | 285140e4-555e-4cac-a2ec-afb4bd6921a8 | 53687091200 | ROOT | Ready | NULL |
| 155 | da585fd4-b10f-480b-9287-45ecb28b6355 | ROOT-110 | 16 | 17 | 4f29b0d2-a1e2-45bc-932a-31d67fd3c572 | 128849018880 | ROOT | Ready | NULL |
+-----+--------------------------------------+----------+------------+---------+--------------------------------------+--------------+-------------+----------+---------------------+
4 rows in set (0.00 sec)
Thank you for all your help.
Regards,
Antoine
Antoine Boucher
AntoineB@haltondc.com<ma...@haltondc.com>
[o] +1-226-505-9734
www.haltondc.com<http://www.haltondc.com>
“Data security made simple and affordable”
[cid:image001.png@01D875B3.631C6270]
Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential, and may be privileged. If you are not the intended recipient, you are hereby notified that any review, retransmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, and delete this message and any attachments from your system.
On May 20, 2022, at 4:56 AM, Suresh Anaparti <Su...@shapeblue.com>> wrote:
Hi Antoine,
Thanks for sharing these details. The delete snapshot cmd failed with NPE, as the underlying primary storage doesn't exists.
Usually, when a snapshot is deleted, the backed up snapshot file is removed from the secondary storage, the status in the cloud.snapshots table for that snaphshot is marked as Destroyed, and the state in the cloud.snapshot_store_ref table marked as Destroyed for the entries of that snapshot (both, for store_role Primary and Image). Later, the storage garbage collector (when enabled), sets the removed date in cloud.snapshots table and removes the entries in cloud.snapshot_store_ref table.
You can manually update the DB with the same behavior, first try with one snapshot (117) and if no issues, apply it to others (109,127,138).
Sample sqls for snapshot id=117 below.
- UPDATE cloud.snapshots SET status = 'Destroyed', removed = now() WHERE id = 117;
- DELETE FROM cloud.snapshot_store_ref WHERE snapshot_id = 117;
'111' corresponds to volume id, in the snapshot location 'snapshots/16/111/66210b31-7445-4586-89ea-52a37f5b1ab5'. I see the record with volume id 111 is expunged (migrated to new id) from the email thread, not sure about volume ids 110 and 112. What's the output of the sql below.
- SELECT id, uuid, name, account_id, pool_id, path, size, volume_type, state, removed FROM cloud.volumes WHERE id IN (110, 111, 112, 155);
Regards,
Suresh
On 17/05/22, 4:36 AM, "Antoine Boucher" <an...@haltondc.com>> wrote:
Hi Suresh,
Here ids the logs of Cmk delete snapshot id=117 and similar for id=109,127,138
2022-05-16 18:18:49,667 DEBUG [c.c.a.ApiServer] (qtp515715487-147449:ctx-9470efda ctx-7b642401) (logid:dc836839) CIDRs from which account 'Acct[2cee75f9-8bc4-11ec-9c43-001e67fd4838-admin] -- Account {"id": 2, "name": "admin", "uuid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838"}' is allowed to perform API calls: 0.0.0.0/0,::/0
2022-05-16 18:18:49,689 DEBUG [c.c.a.ApiServlet] (qtp515715487-147449:ctx-9470efda ctx-7b642401) (logid:dc836839) ===END=== 0:0:0:0:0:0:0:1 -- GET command=deleteSnapshot&id=117&response=json&sessionkey=zLfGQdrCq6kxldkzrnAxYZ8HWQE
2022-05-16 18:18:49,689 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-204:ctx-1a4cbdc0 job-2964) (logid:161004ff) Add job-2964 into job monitoring
2022-05-16 18:18:49,704 DEBUG [c.c.a.ApiServlet] (qtp515715487-290:ctx-b653031c) (logid:4c211dfd) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobid=e6172481-968c-4ee4-993e-a9851af91c29&response=json&sessionkey=zLfGQdrCq6kxldkzrnAxYZ8HWQE
2022-05-16 18:18:49,715 DEBUG [c.c.a.ApiServer] (qtp515715487-290:ctx-b653031c ctx-61ed5689) (logid:4c211dfd) CIDRs from which account 'Acct[2cee75f9-8bc4-11ec-9c43-001e67fd4838-admin] -- Account {"id": 2, "name": "admin", "uuid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838"}' is allowed to perform API calls: 0.0.0.0/0,::/0
2022-05-16 18:18:49,736 DEBUG [c.c.a.ApiServlet] (qtp515715487-290:ctx-b653031c ctx-61ed5689) (logid:4c211dfd) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobid=e6172481-968c-4ee4-993e-a9851af91c29&response=json&sessionkey=zLfGQdrCq6kxldkzrnAxYZ8HWQE
2022-05-16 18:18:49,741 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-204:ctx-1a4cbdc0 job-2964) (logid:e6172481) Unexpected exception while executing org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd
java.lang.NullPointerException
at com.cloud.storage.snapshot.SnapshotManagerImpl.getDataStoreRole(SnapshotManagerImpl.java:1354)
at com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshot(SnapshotManagerImpl.java:591)
at jdk.internal.reflect.GeneratedMethodAccessor4669.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy218.deleteSnapshot(Unknown Source)
at org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd.execute(DeleteSnapshotCmd.java:103)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-05-16 18:18:49,748 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-204:ctx-1a4cbdc0 job-2964) (logid:e6172481) Remove job-2964 from job monitoring
2022-05-16 18:18:50,692 DEBUG [c.c.a.ApiServlet] (qtp515715487-147221:ctx-4c1cd6c0) (logid:3652b6cf) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobid=e6172481-968c-4ee4-993e-a9851af91c29&response=json&sessionkey=zLfGQdrCq6kxldkzrnAxYZ8HWQE
2022-05-16 18:18:50,706 DEBUG [c.c.a.ApiServer] (qtp515715487-147221:ctx-4c1cd6c0 ctx-9cbca8dc) (logid:3652b6cf) CIDRs from which account 'Acct[2cee75f9-8bc4-11ec-9c43-001e67fd4838-admin] -- Account {"id": 2, "name": "admin", "uuid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838"}' is allowed to perform API calls: 0.0.0.0/0,::/0
2022-05-16 18:18:50,723 DEBUG [c.c.a.ApiServlet] (qtp515715487-147221:ctx-4c1cd6c0 ctx-9cbca8dc) (logid:3652b6cf) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobid=e6172481-968c-4ee4-993e-a9851af91c29&response=json&sessionkey=zLfGQdrCq6kxldkzrnAxYZ8HWQE
My snapshot.backup.to.secondary is configured to ‘true’
The snapshot location 'snapshots/16/111/66210b31-7445-4586-89ea-52a37f5b1ab5’ for id=117 no longer exist. All of the concerned snapshots reference 'snapshots/16/111/...' for location but but directory 111 no longer exist. 110 and 112 exists.
Regards,
Antoine
Antoine Boucher
AntoineB@haltondc.com<ma...@haltondc.com>
[o] +1-226-505-9734
www.haltondc.com<http://www.haltondc.com>
On May 13, 2022, at 9:18 AM, Suresh Anaparti <Su...@shapeblue.com>> wrote:
Hi Antoine,
What is the error in the management server logs while deleting the ‘BackedUp’ snapshots? Can you share the log if possible.
Are these snapshots backed up to secondary storage as well (and what’s the config ‘snapshot.backup.to.secondary’ value when taking snapshot )? If yes, can you find the snapshot files at the install_path there? You can check the install_path details in cloud.snapshot_store_ref table with volume_id: 155 and store_role: Image.
Regards,
Suresh
From: Antoine Boucher <an...@haltondc.com> <ma...@haltondc.com>>
Reply to: "users@cloudstack.apache.org<ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org>" <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org>>
Date: Wednesday, 11 May 2022 at 9:15 PM
To: users <us...@cloudstack.apache.org>>
Subject: Re: NPE while listing Snapshots
Too fast id=146 also failed
{
"accountid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838",
"cmd": "org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd",
"completed": "2022-05-11T11:43:01-0400",
"created": "2022-05-11T11:38:00-0400",
"jobid": "0bd30e5c-2e61-4988-8152-36835b791769",
"jobinstanceid": "875d0298-6ed7-43fc-a468-da6042388ac0",
"jobinstancetype": "Snapshot",
"jobprocstatus": 0,
"jobresult": {
"errorcode": 530,
"errortext": "Failed to delete snapshot:com.cloud.utils.exception.CloudRuntimeException: Failed to remove snapshot "
},
"jobresultcode": 530,
"jobresulttype": "object",
"jobstatus": 2,
"userid": "2cefb708-8bc4-11ec-9c43-001e67fd4838"
}
🙈 Error: async API failed for job 0bd30e5c-2e61-4988-8152-36835b791769
Antoine Boucher
AntoineB@haltondc.com<ma...@haltondc.com> <ma...@haltondc.com>
[o] +1-226-505-9734
www.haltondc.com<http://www.haltondc.com/> <http://www.haltondc.com/>
“Data security made simple and affordable”
Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential, and may be privileged. If you are not the intended recipient, you are hereby notified that any review, retransmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, and delete this message and any attachments from your system.
On May 11, 2022, at 11:41 AM, Antoine Boucher <an...@haltondc.com> <ma...@haltondc.com>> wrote:
Hello Suresh,
That primary storage was part of the local storage of a host that was removed.
SELECT name, uuid, pool_type, scope, created, removed FROM cloud.storage_pool WHERE id = 5
+----------------+------+------------+-------+---------------------+---------------------+
| name | uuid | pool_type | scope | created | removed |
+----------------+------+------------+-------+---------------------+---------------------+
| cs-kvm03-local | NULL | Filesystem | HOST | 2022-02-12 15:01:45 | 2022-04-19 02:11:51 |
+----------------+------+------------+-------+---------------------+——————————+
SELECT volume_id, store_id, snapshot_id, store_role, install_path, size, state FROM cloud.snapshot_store_ref WHERE store_id = '5';
+-----------+----------+-------------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
| volume_id | store_id | snapshot_id | store_role | install_path | size | state |
+-----------+----------+-------------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
| 155 | 5 | 109 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/66210b31-7445-4586-89ea-52a37f5b1ab5 | 85899345920 | Ready |
| 155 | 5 | 117 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/a2c4fbbb-12b2-4c43-9647-375f6e334a46 | 85899345920 | Ready |
| 155 | 5 | 127 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/794d6dff-ba45-4a56-8377-1e48034bf014 | 85899345920 | Ready |
| 155 | 5 | 138 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/38648d51-bb19-49a2-b01f-12c4e6ed4529 | 85899345920 | Ready |
+-----------+----------+-------------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
All 155 snapshots have the same account_id and domain_id.
SELECT id, uuid, name, status, path, created, removed, location_type FROM cloud.snapshots WHERE volume_id = '155';
+-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------------------+---------------+
| id | uuid | name | status | path | created | removed | location_type |
+-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------------------+---------------+
| 101 | 13870bbd-d7d7-4664-a22f-f09c80a56d41 | TS01_ROOT-110_20220414054855 | Destroyed | NULL | 2022-04-14 05:48:55 | NULL | NULL |
| 109 | 77d84bf6-f601-44a6-a239-821f735a7f48 | TS01_ROOT-110_20220415054855 | BackedUp | NULL | 2022-04-15 05:48:55 | NULL | NULL |
| 117 | 7db02f43-a9d8-4c73-905a-1524d90f570a | TS01_ROOT-110_20220416054855 | BackedUp | NULL | 2022-04-16 05:48:55 | NULL | NULL |
| 127 | 47288ab9-2da2-40f2-98e3-9ced3f8e87df | TS01_ROOT-110_20220417054855 | BackedUp | NULL | 2022-04-17 05:48:55 | NULL | NULL |
| 138 | e9639f4a-fa09-4698-849b-572011045993 | TS01_ROOT-110_20220418054855 | BackedUp | NULL | 2022-04-18 05:48:55 | NULL | NULL |
| 146 | 875d0298-6ed7-43fc-a468-da6042388ac0 | TS01_ROOT-110_20220419054747 | BackedUp | NULL | 2022-04-19 05:47:47 | NULL | NULL |
| 154 | 0122c9b3-a9f8-473f-a8d7-6e69ba77265b | TS01_ROOT-110_20220420054748 | Destroyed | NULL | 2022-04-20 05:47:48 | NULL | NULL |
| 157 | fe2248c6-aa80-4e9f-b539-7f495f1ff39f | Manual - Apr 20 | Destroyed | NULL | 2022-04-21 01:50:16 | NULL | NULL |
| 163 | 32923ad7-cbe3-4708-95d8-23eca921d711 | TS01_ROOT-110_20220421054817 | Destroyed | NULL | 2022-04-21 05:48:17 | NULL | NULL |
| 172 | 6e03ea90-e8b0-4405-aed6-7bd17a40de57 | TS01_ROOT-110_20220422054933 | Destroyed | NULL | 2022-04-22 05:49:33 | NULL | NULL |
| 181 | 9059a26a-0e4e-4d46-95f2-b0a9236ba90a | TS01_ROOT-110_20220423054933 | Destroyed | NULL | 2022-04-23 05:49:33 | NULL | NULL |
| 193 | c9828cbc-f282-4dfe-b4cf-9df792c522c2 | TS01_ROOT-110_20220424054934 | BackedUp | NULL | 2022-04-24 05:49:34 | NULL | NULL |
| 207 | 2e5dfd48-c4de-45ff-bbe1-4cf0d90bb698 | TS01_ROOT-110_20220425054934 | Destroyed | NULL | 2022-04-25 05:49:34 | NULL | NULL |
| 217 | 99d9d9ff-44ae-41a9-9761-f722b370c01c | TS01_ROOT-110_20220426054934 | Destroyed | NULL | 2022-04-26 05:49:34 | NULL | NULL |
| 227 | 1df14a7b-ba56-4d26-aeb4-c79d4de978fc | TS01_ROOT-110_20220427054934 | BackedUp | NULL | 2022-04-27 05:49:34 | NULL | NULL |
| 237 | 657479e0-3587-4a72-a8c8-bd7dff54a5e4 | TS01_ROOT-110_20220428054934 | BackedUp | NULL | 2022-04-28 05:49:34 | NULL | NULL |
| 248 | 2ef0b539-be01-4b0c-9fa5-5fc9c60a6a96 | TS01_ROOT-110_20220429054935 | BackedUp | NULL | 2022-04-29 05:49:35 | NULL | NULL |
| 257 | 8ab75270-8e3d-471e-8e4a-8d5f3cbc6bc1 | TS01_ROOT-110_20220430054935 | BackedUp | NULL | 2022-04-30 05:49:35 | NULL | NULL |
| 259 | a1e77bec-8fc7-457f-b0a5-961c339487f7 | TS01_ROOT-110_20220430060935 | Destroyed | NULL | 2022-04-30 06:09:35 | 2022-05-09 20:02:38 | NULL |
| 313 | 8ad0c04a-1d74-4d51-8223-a6aa3a82d62c | TS01_ROOT-110_20220509015742 | BackedUp | NULL | 2022-05-09 01:57:42 | NULL | NULL |
| 327 | 0f5e9366-747f-4a34-bec4-1fd31c461ac3 | TS01_ROOT-110_20220509170242 | BackedUp | NULL | 2022-05-09 17:02:42 | NULL | NULL |
+-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------------------+---------------+
How would you suggest I resolve the issue moving forward?
I tried the following with error…
(nimbus) 🐱 > delete snapshot id=109
{
"accountid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838",
"cmd": "org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd",
"completed": "2022-05-11T11:32:08-0400",
"created": "2022-05-11T11:32:08-0400",
"jobid": "29661b2b-46c5-42bd-9974-d1a226f3fe68",
"jobinstanceid": "77d84bf6-f601-44a6-a239-821f735a7f48",
"jobinstancetype": "Snapshot",
"jobprocstatus": 0,
"jobresult": {
"errorcode": 530,
"errortext": "Command failed due to Internal Server Error"
},
"jobresultcode": 530,
"jobresulttype": "object",
"jobstatus": 2,
"userid": "2cefb708-8bc4-11ec-9c43-001e67fd4838"
}
🙈 Error: async API failed for job 29661b2b-46c5-42bd-9974-d1a226f3fe68
Deleting some table rows in the right sequence?
Same issue with id=117 to id=138, but I was able to delete id=146
Regards,
Antoine
On Apr 27, 2022, at 9:00 AM, Suresh Anaparti <Su...@shapeblue.com> <ma...@shapeblue.com>> wrote:
Hi Antoine,
The NPE error seems to be due to the primary storage with id '5', when checking for snapshot revertible or not during list snapshots cmd. Is that primary storage removed? Can you check with sql below.
SELECT name, uuid, pool_type, scope, created, removed FROM cloud.storage_pool WHERE id = 5
I think, the account/user might not have the permissions to list all snapshots. You can check the account / domain details in snapshots table.
Regards,
Suresh
On 21/04/22, 6:50 PM, "Antoine Boucher" <an...@haltondc.com> <ma...@haltondc.com>> wrote:
Thank you Suresh, you will find my answers below,
On Apr 21, 2022, at 1:02 AM, Suresh Anaparti <Su...@shapeblue.com> <ma...@shapeblue.com>> wrote:
Hi Antoine,
What is the CloudStack version, where you see this issue.
Latest - 4.16.1
When you noticed this issue, is it while listing snapshots for a particular volume, or all volumes? Try list snapshots from API/cmk as well.
This is a customer VM, the account may have hit its snapshot limit during a snapshot.
cmk list snapshots > only lists 26 snapshots out of 62
cmk list snapshots volumeid= > Works for some volume ids, provides null result while the WebUI lists the correct snapshots for others and provides null results while show undefined error for one VM mentioned below
Also, Check the snapshots details for the volume (with issue) in the db, using the following sql queries.
1. SELECT id, uuid, state, pool_id, path, size, removed FROM cloud.volumes WHERE name LIKE '<volume_name>'
mysql> SELECT id, uuid, state, pool_id, path, size, removed FROM cloud.volumes WHERE name LIKE 'ROOT-110';
+-----+--------------------------------------+----------+---------+--------------------------------------+-------------+---------------------+
| id | uuid | state | pool_id | path | size | removed |
+-----+--------------------------------------+----------+---------+--------------------------------------+-------------+---------------------+
| 111 | NULL | Expunged | 5 | da585fd4-b10f-480b-9287-45ecb28b6355 | 85899345920 | 2022-04-19 01:29:08 |
| 138 | da585fd4-b10f-480b-9287-45ecb28b6355 | Ready | 6 | 04bd5d13-72ca-4240-93ce-ed1b015d3437 | 85899345920 | NULL |
+-----+--------------------------------------+----------+---------+--------------------------------------+-------------+---------------------+
2. SELECT id, uuid, name, status, path, created, removed, location_type FROM cloud.snapshots WHERE volume_id = <volume_id_from_step_1>;
mysql> SELECT id, uuid, name, status, path, created, removed, location_type FROM cloud.snapshots WHERE volume_id = '138';
+-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------+---------------+
| id | uuid | name | status | path | created | removed | location_type |
+-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------+---------------+
| 101 | 13870bbd-d7d7-4664-a22f-f09c80a56d41 | TS01_ROOT-110_20220414054855 | Destroyed | NULL | 2022-04-14 05:48:55 | NULL | NULL |
| 109 | 77d84bf6-f601-44a6-a239-821f735a7f48 | TS01_ROOT-110_20220415054855 | BackedUp | NULL | 2022-04-15 05:48:55 | NULL | NULL |
| 117 | 7db02f43-a9d8-4c73-905a-1524d90f570a | TS01_ROOT-110_20220416054855 | BackedUp | NULL | 2022-04-16 05:48:55 | NULL | NULL |
| 127 | 47288ab9-2da2-40f2-98e3-9ced3f8e87df | TS01_ROOT-110_20220417054855 | BackedUp | NULL | 2022-04-17 05:48:55 | NULL | NULL |
| 138 | e9639f4a-fa09-4698-849b-572011045993 | TS01_ROOT-110_20220418054855 | BackedUp | NULL | 2022-04-18 05:48:55 | NULL | NULL |
| 146 | 875d0298-6ed7-43fc-a468-da6042388ac0 | TS01_ROOT-110_20220419054747 | BackedUp | NULL | 2022-04-19 05:47:47 | NULL | NULL |
| 154 | 0122c9b3-a9f8-473f-a8d7-6e69ba77265b | TS01_ROOT-110_20220420054748 | BackedUp | NULL | 2022-04-20 05:47:48 | NULL | NULL |
| 157 | fe2248c6-aa80-4e9f-b539-7f495f1ff39f | Manual - Apr 20 | BackedUp | NULL | 2022-04-21 01:50:16 | NULL | NULL |
| 163 | 32923ad7-cbe3-4708-95d8-23eca921d711 | TS01_ROOT-110_20220421054817 | BackedUp | NULL | 2022-04-21 05:48:17 | NULL | NULL |
+-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------+---------------+
3. SELECT volume_id, store_id, store_role, install_path, size, state FROM cloud.snapshot_store_ref WHERE snapshot_id = <snapshot_id_from_step_2>
mysql> SELECT volume_id, store_id, store_role, install_path, size, state FROM cloud.snapshot_store_ref WHERE snapshot_id = '101';
+-----------+----------+------------+-------------------------------------------------------+-------------+------------+
| volume_id | store_id | store_role | install_path | size | state |
+-----------+----------+------------+-------------------------------------------------------+-------------+------------+
| 138 | 1 | Image | snapshots/16/111/a6b1bead-ca9a-464b-82c1-d631611ad790 | 85899345920 | Destroying |
+-----------+----------+------------+-------------------------------------------------------+-------------+------------+
1 row in set (0.00 sec)
mysql> SELECT volume_id, store_id, store_role, install_path, size, state FROM cloud.snapshot_store_ref WHERE snapshot_id = '109';
+-----------+----------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
| volume_id | store_id | store_role | install_path | size | state |
+-----------+----------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
| 138 | 5 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/66210b31-7445-4586-89ea-52a37f5b1ab5 | 85899345920 | Ready |
| 138 | 1 | Image | snapshots/16/111/66210b31-7445-4586-89ea-52a37f5b1ab5 | 85899345920 | Ready |
+-----------+----------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
2 rows in set (0.00 sec)
Regards,
Antoine
Regards,
Suresh
On 21/04/22, 8:45 AM, "Antoine Boucher" <an...@haltondc.com> <ma...@haltondc.com>> wrote:
Unfortunately I spoke too fast. The list is back but listing the snapshots of the volume with issue still show a blank list with error “undefined”.
On Apr 20, 2022, at 23:11, Antoine Boucher <an...@haltondc.com> <ma...@haltondc.com>> wrote:
I was able to recover from the situation by forcing a snapshot on the volume with issue.
On Apr 20, 2022, at 22:00, Antoine Boucher <an...@haltondc.com> <ma...@haltondc.com>> wrote:
I pin-pointed the volume with the issue by going through all volumes and clicking the “view snapshot"
On Apr 20, 2022, at 9:35 PM, Antoine Boucher <an...@haltondc.com> <ma...@haltondc.com>> wrote:
I’m no longer able to see my list of snapshots (storage > snapshots), instead I see a small popup window with the word “undefined" on an empty list snapshot list.
Has anyone seen this issue? I restart and rebooted the Management server but no resolution.
Here are my logs:
2022-04-20 21:17:02,428 DEBUG [c.c.a.ApiServlet] (qtp515715487-292:ctx-a220f915) (logid:624389ae) ===START=== 10.101.254.1 -- GET listall=true&page=1&pagesize=20&command=listSnapshots&response=json
2022-04-20 21:17:02,439 DEBUG [c.c.a.ApiServer] (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) CIDRs from which account 'Acct[2cee75f9-8bc4-11ec-9c43-001e67fd4838-admin] -- Account {"id": 2, "name": "admin", "uuid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838"}' is allowed to perform API calls: 0.0.0.0/0,::/0
2022-04-20 21:17:02,675 ERROR [c.c.a.ApiServer] (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) unhandled exception executing api command: [Ljava.lang.String;@42fa8f7d
java.lang.NullPointerException
at org.apache.cloudstack.storage.snapshot.StorageSystemSnapshotStrategy.canHandle(StorageSystemSnapshotStrategy.java:985)
at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:72)
at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:69)
at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.bestMatch(StorageStrategyFactoryImpl.java:95)
at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.getSnapshotStrategy(StorageStrategyFactoryImpl.java:69)
at org.apache.cloudstack.storage.snapshot.SnapshotObject.isRevertable(SnapshotObject.java:156)
at com.cloud.api.ApiResponseHelper.createSnapshotResponse(ApiResponseHelper.java:591)
at org.apache.cloudstack.api.command.user.snapshot.ListSnapshotsCmd.execute(ListSnapshotsCmd.java:117)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:772)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:596)
at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:321)
at com.cloud.api.ApiServlet$1.run(ApiServlet.java:134)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:131)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:93)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-04-20 21:17:02,677 DEBUG [c.c.a.ApiServlet] (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) ===END=== 10.101.254.1 -- GET listall=true&page=1&pagesize=20&command=listSnapshots&response=json
Antoine Boucher
Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential, and may be privileged. If you are not the intended recipient, you are hereby notified that any review, retransmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, and delete this message and any attachments from your system.
Re: NPE while listing Snapshots
Posted by Antoine Boucher <an...@haltondc.com>.
Hello Suresh,
I did your 2 commands on all 4 ids and I have no more issues on the listing the Snapshot from the webUI.
Here is the result of the query you requested:
mysql> SELECT id, uuid, name, account_id, pool_id, path, size, volume_type, state, removed FROM cloud.volumes WHERE id IN (110, 111, 112, 155);
+-----+--------------------------------------+----------+------------+---------+--------------------------------------+--------------+-------------+----------+---------------------+
| id | uuid | name | account_id | pool_id | path | size | volume_type | state | removed |
+-----+--------------------------------------+----------+------------+---------+--------------------------------------+--------------+-------------+----------+---------------------+
| 110 | 390198b4-45d4-425e-87c3-d0f11ef2ca11 | ROOT-109 | 16 | 4 | 390198b4-45d4-425e-87c3-d0f11ef2ca11 | 42949672960 | ROOT | Ready | NULL |
| 111 | NULL | ROOT-110 | 16 | 5 | da585fd4-b10f-480b-9287-45ecb28b6355 | 85899345920 | DATADISK | Expunged | 2022-04-19 01:29:08 |
| 112 | 285140e4-555e-4cac-a2ec-afb4bd6921a8 | ROOT-111 | 16 | 3 | 285140e4-555e-4cac-a2ec-afb4bd6921a8 | 53687091200 | ROOT | Ready | NULL |
| 155 | da585fd4-b10f-480b-9287-45ecb28b6355 | ROOT-110 | 16 | 17 | 4f29b0d2-a1e2-45bc-932a-31d67fd3c572 | 128849018880 | ROOT | Ready | NULL |
+-----+--------------------------------------+----------+------------+---------+--------------------------------------+--------------+-------------+----------+---------------------+
4 rows in set (0.00 sec)
Thank you for all your help.
Regards,
Antoine
Antoine Boucher
AntoineB@haltondc.com
[o] +1-226-505-9734
www.haltondc.com
“Data security made simple and affordable”
Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential, and may be privileged. If you are not the intended recipient, you are hereby notified that any review, retransmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, and delete this message and any attachments from your system.
> On May 20, 2022, at 4:56 AM, Suresh Anaparti <Su...@shapeblue.com> wrote:
>
> Hi Antoine,
>
> Thanks for sharing these details. The delete snapshot cmd failed with NPE, as the underlying primary storage doesn't exists.
>
> Usually, when a snapshot is deleted, the backed up snapshot file is removed from the secondary storage, the status in the cloud.snapshots table for that snaphshot is marked as Destroyed, and the state in the cloud.snapshot_store_ref table marked as Destroyed for the entries of that snapshot (both, for store_role Primary and Image). Later, the storage garbage collector (when enabled), sets the removed date in cloud.snapshots table and removes the entries in cloud.snapshot_store_ref table.
>
> You can manually update the DB with the same behavior, first try with one snapshot (117) and if no issues, apply it to others (109,127,138).
>
> Sample sqls for snapshot id=117 below.
>
> - UPDATE cloud.snapshots SET status = 'Destroyed', removed = now() WHERE id = 117;
>
> - DELETE FROM cloud.snapshot_store_ref WHERE snapshot_id = 117;
>
>
> '111' corresponds to volume id, in the snapshot location 'snapshots/16/111/66210b31-7445-4586-89ea-52a37f5b1ab5'. I see the record with volume id 111 is expunged (migrated to new id) from the email thread, not sure about volume ids 110 and 112. What's the output of the sql below.
>
> - SELECT id, uuid, name, account_id, pool_id, path, size, volume_type, state, removed FROM cloud.volumes WHERE id IN (110, 111, 112, 155);
>
>
> Regards,
> Suresh
>
> On 17/05/22, 4:36 AM, "Antoine Boucher" <antoineb@haltondc.com <ma...@haltondc.com>> wrote:
>
> Hi Suresh,
>
> Here ids the logs of Cmk delete snapshot id=117 and similar for id=109,127,138
>
>
> 2022-05-16 18:18:49,667 DEBUG [c.c.a.ApiServer] (qtp515715487-147449:ctx-9470efda ctx-7b642401) (logid:dc836839) CIDRs from which account 'Acct[2cee75f9-8bc4-11ec-9c43-001e67fd4838-admin] -- Account {"id": 2, "name": "admin", "uuid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838"}' is allowed to perform API calls: 0.0.0.0/0,::/0
> 2022-05-16 18:18:49,689 DEBUG [c.c.a.ApiServlet] (qtp515715487-147449:ctx-9470efda ctx-7b642401) (logid:dc836839) ===END=== 0:0:0:0:0:0:0:1 -- GET command=deleteSnapshot&id=117&response=json&sessionkey=zLfGQdrCq6kxldkzrnAxYZ8HWQE
> 2022-05-16 18:18:49,689 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-204:ctx-1a4cbdc0 job-2964) (logid:161004ff) Add job-2964 into job monitoring
> 2022-05-16 18:18:49,704 DEBUG [c.c.a.ApiServlet] (qtp515715487-290:ctx-b653031c) (logid:4c211dfd) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobid=e6172481-968c-4ee4-993e-a9851af91c29&response=json&sessionkey=zLfGQdrCq6kxldkzrnAxYZ8HWQE
> 2022-05-16 18:18:49,715 DEBUG [c.c.a.ApiServer] (qtp515715487-290:ctx-b653031c ctx-61ed5689) (logid:4c211dfd) CIDRs from which account 'Acct[2cee75f9-8bc4-11ec-9c43-001e67fd4838-admin] -- Account {"id": 2, "name": "admin", "uuid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838"}' is allowed to perform API calls: 0.0.0.0/0,::/0
> 2022-05-16 18:18:49,736 DEBUG [c.c.a.ApiServlet] (qtp515715487-290:ctx-b653031c ctx-61ed5689) (logid:4c211dfd) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobid=e6172481-968c-4ee4-993e-a9851af91c29&response=json&sessionkey=zLfGQdrCq6kxldkzrnAxYZ8HWQE
> 2022-05-16 18:18:49,741 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-204:ctx-1a4cbdc0 job-2964) (logid:e6172481) Unexpected exception while executing org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd
> java.lang.NullPointerException
> at com.cloud.storage.snapshot.SnapshotManagerImpl.getDataStoreRole(SnapshotManagerImpl.java:1354)
> at com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshot(SnapshotManagerImpl.java:591)
> at jdk.internal.reflect.GeneratedMethodAccessor4669.invoke(Unknown Source)
> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
> at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
> at com.sun.proxy.$Proxy218.deleteSnapshot(Unknown Source)
> at org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd.execute(DeleteSnapshotCmd.java:103)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
> at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> 2022-05-16 18:18:49,748 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-204:ctx-1a4cbdc0 job-2964) (logid:e6172481) Remove job-2964 from job monitoring
> 2022-05-16 18:18:50,692 DEBUG [c.c.a.ApiServlet] (qtp515715487-147221:ctx-4c1cd6c0) (logid:3652b6cf) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobid=e6172481-968c-4ee4-993e-a9851af91c29&response=json&sessionkey=zLfGQdrCq6kxldkzrnAxYZ8HWQE
> 2022-05-16 18:18:50,706 DEBUG [c.c.a.ApiServer] (qtp515715487-147221:ctx-4c1cd6c0 ctx-9cbca8dc) (logid:3652b6cf) CIDRs from which account 'Acct[2cee75f9-8bc4-11ec-9c43-001e67fd4838-admin] -- Account {"id": 2, "name": "admin", "uuid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838"}' is allowed to perform API calls: 0.0.0.0/0,::/0
> 2022-05-16 18:18:50,723 DEBUG [c.c.a.ApiServlet] (qtp515715487-147221:ctx-4c1cd6c0 ctx-9cbca8dc) (logid:3652b6cf) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobid=e6172481-968c-4ee4-993e-a9851af91c29&response=json&sessionkey=zLfGQdrCq6kxldkzrnAxYZ8HWQE
>
> My snapshot.backup.to.secondary is configured to ‘true’
>
> The snapshot location 'snapshots/16/111/66210b31-7445-4586-89ea-52a37f5b1ab5’ for id=117 no longer exist. All of the concerned snapshots reference 'snapshots/16/111/...' for location but but directory 111 no longer exist. 110 and 112 exists.
>
> Regards,
> Antoine
>
>
>
> Antoine Boucher
> AntoineB@haltondc.com
> [o] +1-226-505-9734
> www.haltondc.com
>
>
>
>
>
>
>> On May 13, 2022, at 9:18 AM, Suresh Anaparti <Su...@shapeblue.com> wrote:
>>
>> Hi Antoine,
>>
>> What is the error in the management server logs while deleting the ‘BackedUp’ snapshots? Can you share the log if possible.
>>
>> Are these snapshots backed up to secondary storage as well (and what’s the config ‘snapshot.backup.to.secondary’ value when taking snapshot )? If yes, can you find the snapshot files at the install_path there? You can check the install_path details in cloud.snapshot_store_ref table with volume_id: 155 and store_role: Image.
>>
>>
>> Regards,
>> Suresh
>>
>>
>>
>>
>> From: Antoine Boucher <antoineb@haltondc.com <ma...@haltondc.com> <mailto:antoineb@haltondc.com <ma...@haltondc.com>>>
>> Reply to: "users@cloudstack.apache.org <ma...@cloudstack.apache.org> <mailto:users@cloudstack.apache.org <ma...@cloudstack.apache.org>>" <users@cloudstack.apache.org <ma...@cloudstack.apache.org> <mailto:users@cloudstack.apache.org <ma...@cloudstack.apache.org>>>
>> Date: Wednesday, 11 May 2022 at 9:15 PM
>> To: users <users@cloudstack.apache.org <ma...@cloudstack.apache.org>>
>> Subject: Re: NPE while listing Snapshots
>>
>> Too fast id=146 also failed
>>
>> {
>> "accountid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838",
>> "cmd": "org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd",
>> "completed": "2022-05-11T11:43:01-0400",
>> "created": "2022-05-11T11:38:00-0400",
>> "jobid": "0bd30e5c-2e61-4988-8152-36835b791769",
>> "jobinstanceid": "875d0298-6ed7-43fc-a468-da6042388ac0",
>> "jobinstancetype": "Snapshot",
>> "jobprocstatus": 0,
>> "jobresult": {
>> "errorcode": 530,
>> "errortext": "Failed to delete snapshot:com.cloud.utils.exception.CloudRuntimeException: Failed to remove snapshot "
>> },
>> "jobresultcode": 530,
>> "jobresulttype": "object",
>> "jobstatus": 2,
>> "userid": "2cefb708-8bc4-11ec-9c43-001e67fd4838"
>> }
>> 🙈 Error: async API failed for job 0bd30e5c-2e61-4988-8152-36835b791769
>>
>>
>>
>>
>>
>> Antoine Boucher
>> AntoineB@haltondc.com <ma...@haltondc.com> <mailto:AntoineB@haltondc.com <ma...@haltondc.com>>
>> [o] +1-226-505-9734
>> www.haltondc.com <http://www.haltondc.com/> <http://www.haltondc.com/ <http://www.haltondc.com/>>
>>
>> “Data security made simple and affordable”
>>
>>
>>
>>
>>
>>
>>
>> Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential, and may be privileged. If you are not the intended recipient, you are hereby notified that any review, retransmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, and delete this message and any attachments from your system.
>>
>>
>>
>>
>>> On May 11, 2022, at 11:41 AM, Antoine Boucher <antoineb@haltondc.com <ma...@haltondc.com> <mailto:antoineb@haltondc.com <ma...@haltondc.com>>> wrote:
>>>
>>> Hello Suresh,
>>>
>>> That primary storage was part of the local storage of a host that was removed.
>>>
>>> SELECT name, uuid, pool_type, scope, created, removed FROM cloud.storage_pool WHERE id = 5
>>> +----------------+------+------------+-------+---------------------+---------------------+
>>> | name | uuid | pool_type | scope | created | removed |
>>> +----------------+------+------------+-------+---------------------+---------------------+
>>> | cs-kvm03-local | NULL | Filesystem | HOST | 2022-02-12 15:01:45 | 2022-04-19 02:11:51 |
>>> +----------------+------+------------+-------+---------------------+——————————+
>>>
>>>
>>>
>>>
>>> SELECT volume_id, store_id, snapshot_id, store_role, install_path, size, state FROM cloud.snapshot_store_ref WHERE store_id = '5';
>>> +-----------+----------+-------------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
>>> | volume_id | store_id | snapshot_id | store_role | install_path | size | state |
>>> +-----------+----------+-------------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
>>> | 155 | 5 | 109 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/66210b31-7445-4586-89ea-52a37f5b1ab5 | 85899345920 | Ready |
>>> | 155 | 5 | 117 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/a2c4fbbb-12b2-4c43-9647-375f6e334a46 | 85899345920 | Ready |
>>> | 155 | 5 | 127 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/794d6dff-ba45-4a56-8377-1e48034bf014 | 85899345920 | Ready |
>>> | 155 | 5 | 138 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/38648d51-bb19-49a2-b01f-12c4e6ed4529 | 85899345920 | Ready |
>>> +-----------+----------+-------------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
>>>
>>>
>>>
>>>
>>> All 155 snapshots have the same account_id and domain_id.
>>>
>>>
>>>
>>>
>>> SELECT id, uuid, name, status, path, created, removed, location_type FROM cloud.snapshots WHERE volume_id = '155';
>>> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------------------+---------------+
>>> | id | uuid | name | status | path | created | removed | location_type |
>>> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------------------+---------------+
>>> | 101 | 13870bbd-d7d7-4664-a22f-f09c80a56d41 | TS01_ROOT-110_20220414054855 | Destroyed | NULL | 2022-04-14 05:48:55 | NULL | NULL |
>>> | 109 | 77d84bf6-f601-44a6-a239-821f735a7f48 | TS01_ROOT-110_20220415054855 | BackedUp | NULL | 2022-04-15 05:48:55 | NULL | NULL |
>>> | 117 | 7db02f43-a9d8-4c73-905a-1524d90f570a | TS01_ROOT-110_20220416054855 | BackedUp | NULL | 2022-04-16 05:48:55 | NULL | NULL |
>>> | 127 | 47288ab9-2da2-40f2-98e3-9ced3f8e87df | TS01_ROOT-110_20220417054855 | BackedUp | NULL | 2022-04-17 05:48:55 | NULL | NULL |
>>> | 138 | e9639f4a-fa09-4698-849b-572011045993 | TS01_ROOT-110_20220418054855 | BackedUp | NULL | 2022-04-18 05:48:55 | NULL | NULL |
>>> | 146 | 875d0298-6ed7-43fc-a468-da6042388ac0 | TS01_ROOT-110_20220419054747 | BackedUp | NULL | 2022-04-19 05:47:47 | NULL | NULL |
>>> | 154 | 0122c9b3-a9f8-473f-a8d7-6e69ba77265b | TS01_ROOT-110_20220420054748 | Destroyed | NULL | 2022-04-20 05:47:48 | NULL | NULL |
>>> | 157 | fe2248c6-aa80-4e9f-b539-7f495f1ff39f | Manual - Apr 20 | Destroyed | NULL | 2022-04-21 01:50:16 | NULL | NULL |
>>> | 163 | 32923ad7-cbe3-4708-95d8-23eca921d711 | TS01_ROOT-110_20220421054817 | Destroyed | NULL | 2022-04-21 05:48:17 | NULL | NULL |
>>> | 172 | 6e03ea90-e8b0-4405-aed6-7bd17a40de57 | TS01_ROOT-110_20220422054933 | Destroyed | NULL | 2022-04-22 05:49:33 | NULL | NULL |
>>> | 181 | 9059a26a-0e4e-4d46-95f2-b0a9236ba90a | TS01_ROOT-110_20220423054933 | Destroyed | NULL | 2022-04-23 05:49:33 | NULL | NULL |
>>> | 193 | c9828cbc-f282-4dfe-b4cf-9df792c522c2 | TS01_ROOT-110_20220424054934 | BackedUp | NULL | 2022-04-24 05:49:34 | NULL | NULL |
>>> | 207 | 2e5dfd48-c4de-45ff-bbe1-4cf0d90bb698 | TS01_ROOT-110_20220425054934 | Destroyed | NULL | 2022-04-25 05:49:34 | NULL | NULL |
>>> | 217 | 99d9d9ff-44ae-41a9-9761-f722b370c01c | TS01_ROOT-110_20220426054934 | Destroyed | NULL | 2022-04-26 05:49:34 | NULL | NULL |
>>> | 227 | 1df14a7b-ba56-4d26-aeb4-c79d4de978fc | TS01_ROOT-110_20220427054934 | BackedUp | NULL | 2022-04-27 05:49:34 | NULL | NULL |
>>> | 237 | 657479e0-3587-4a72-a8c8-bd7dff54a5e4 | TS01_ROOT-110_20220428054934 | BackedUp | NULL | 2022-04-28 05:49:34 | NULL | NULL |
>>> | 248 | 2ef0b539-be01-4b0c-9fa5-5fc9c60a6a96 | TS01_ROOT-110_20220429054935 | BackedUp | NULL | 2022-04-29 05:49:35 | NULL | NULL |
>>> | 257 | 8ab75270-8e3d-471e-8e4a-8d5f3cbc6bc1 | TS01_ROOT-110_20220430054935 | BackedUp | NULL | 2022-04-30 05:49:35 | NULL | NULL |
>>> | 259 | a1e77bec-8fc7-457f-b0a5-961c339487f7 | TS01_ROOT-110_20220430060935 | Destroyed | NULL | 2022-04-30 06:09:35 | 2022-05-09 20:02:38 | NULL |
>>> | 313 | 8ad0c04a-1d74-4d51-8223-a6aa3a82d62c | TS01_ROOT-110_20220509015742 | BackedUp | NULL | 2022-05-09 01:57:42 | NULL | NULL |
>>> | 327 | 0f5e9366-747f-4a34-bec4-1fd31c461ac3 | TS01_ROOT-110_20220509170242 | BackedUp | NULL | 2022-05-09 17:02:42 | NULL | NULL |
>>> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------------------+---------------+
>>>
>>> How would you suggest I resolve the issue moving forward?
>>>
>>> I tried the following with error…
>>>
>>> (nimbus) 🐱 > delete snapshot id=109
>>> {
>>> "accountid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838",
>>> "cmd": "org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd",
>>> "completed": "2022-05-11T11:32:08-0400",
>>> "created": "2022-05-11T11:32:08-0400",
>>> "jobid": "29661b2b-46c5-42bd-9974-d1a226f3fe68",
>>> "jobinstanceid": "77d84bf6-f601-44a6-a239-821f735a7f48",
>>> "jobinstancetype": "Snapshot",
>>> "jobprocstatus": 0,
>>> "jobresult": {
>>> "errorcode": 530,
>>> "errortext": "Command failed due to Internal Server Error"
>>> },
>>> "jobresultcode": 530,
>>> "jobresulttype": "object",
>>> "jobstatus": 2,
>>> "userid": "2cefb708-8bc4-11ec-9c43-001e67fd4838"
>>> }
>>> 🙈 Error: async API failed for job 29661b2b-46c5-42bd-9974-d1a226f3fe68
>>>
>>>
>>> Deleting some table rows in the right sequence?
>>>
>>> Same issue with id=117 to id=138, but I was able to delete id=146
>>>
>>> Regards,
>>> Antoine
>>>
>>>
>>>
>>>
>>>> On Apr 27, 2022, at 9:00 AM, Suresh Anaparti <Suresh.Anaparti@shapeblue.com <ma...@shapeblue.com> <mailto:Suresh.Anaparti@shapeblue.com <ma...@shapeblue.com>>> wrote:
>>>>
>>>> Hi Antoine,
>>>>
>>>> The NPE error seems to be due to the primary storage with id '5', when checking for snapshot revertible or not during list snapshots cmd. Is that primary storage removed? Can you check with sql below.
>>>>
>>>> SELECT name, uuid, pool_type, scope, created, removed FROM cloud.storage_pool WHERE id = 5
>>>>
>>>>
>>>> I think, the account/user might not have the permissions to list all snapshots. You can check the account / domain details in snapshots table.
>>>>
>>>>
>>>> Regards,
>>>> Suresh
>>>>
>>>> On 21/04/22, 6:50 PM, "Antoine Boucher" <antoineb@haltondc.com <ma...@haltondc.com> <mailto:antoineb@haltondc.com <ma...@haltondc.com>>> wrote:
>>>>
>>>> Thank you Suresh, you will find my answers below,
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> On Apr 21, 2022, at 1:02 AM, Suresh Anaparti <Suresh.Anaparti@shapeblue.com <ma...@shapeblue.com> <mailto:Suresh.Anaparti@shapeblue.com <ma...@shapeblue.com>>> wrote:
>>>>>
>>>>> Hi Antoine,
>>>>>
>>>>> What is the CloudStack version, where you see this issue.
>>>>
>>>> Latest - 4.16.1
>>>>
>>>>
>>>>
>>>>> When you noticed this issue, is it while listing snapshots for a particular volume, or all volumes? Try list snapshots from API/cmk as well.
>>>>
>>>> This is a customer VM, the account may have hit its snapshot limit during a snapshot.
>>>>
>>>> cmk list snapshots > only lists 26 snapshots out of 62
>>>>
>>>> cmk list snapshots volumeid= > Works for some volume ids, provides null result while the WebUI lists the correct snapshots for others and provides null results while show undefined error for one VM mentioned below
>>>>
>>>>
>>>>
>>>>>
>>>>> Also, Check the snapshots details for the volume (with issue) in the db, using the following sql queries.
>>>>>
>>>>> 1. SELECT id, uuid, state, pool_id, path, size, removed FROM cloud.volumes WHERE name LIKE '<volume_name>'
>>>>>
>>>>
>>>> mysql> SELECT id, uuid, state, pool_id, path, size, removed FROM cloud.volumes WHERE name LIKE 'ROOT-110';
>>>> +-----+--------------------------------------+----------+---------+--------------------------------------+-------------+---------------------+
>>>> | id | uuid | state | pool_id | path | size | removed |
>>>> +-----+--------------------------------------+----------+---------+--------------------------------------+-------------+---------------------+
>>>> | 111 | NULL | Expunged | 5 | da585fd4-b10f-480b-9287-45ecb28b6355 | 85899345920 | 2022-04-19 01:29:08 |
>>>> | 138 | da585fd4-b10f-480b-9287-45ecb28b6355 | Ready | 6 | 04bd5d13-72ca-4240-93ce-ed1b015d3437 | 85899345920 | NULL |
>>>> +-----+--------------------------------------+----------+---------+--------------------------------------+-------------+---------------------+
>>>>
>>>>
>>>>
>>>>> 2. SELECT id, uuid, name, status, path, created, removed, location_type FROM cloud.snapshots WHERE volume_id = <volume_id_from_step_1>;
>>>>
>>>> mysql> SELECT id, uuid, name, status, path, created, removed, location_type FROM cloud.snapshots WHERE volume_id = '138';
>>>> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------+---------------+
>>>> | id | uuid | name | status | path | created | removed | location_type |
>>>> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------+---------------+
>>>> | 101 | 13870bbd-d7d7-4664-a22f-f09c80a56d41 | TS01_ROOT-110_20220414054855 | Destroyed | NULL | 2022-04-14 05:48:55 | NULL | NULL |
>>>> | 109 | 77d84bf6-f601-44a6-a239-821f735a7f48 | TS01_ROOT-110_20220415054855 | BackedUp | NULL | 2022-04-15 05:48:55 | NULL | NULL |
>>>> | 117 | 7db02f43-a9d8-4c73-905a-1524d90f570a | TS01_ROOT-110_20220416054855 | BackedUp | NULL | 2022-04-16 05:48:55 | NULL | NULL |
>>>> | 127 | 47288ab9-2da2-40f2-98e3-9ced3f8e87df | TS01_ROOT-110_20220417054855 | BackedUp | NULL | 2022-04-17 05:48:55 | NULL | NULL |
>>>> | 138 | e9639f4a-fa09-4698-849b-572011045993 | TS01_ROOT-110_20220418054855 | BackedUp | NULL | 2022-04-18 05:48:55 | NULL | NULL |
>>>> | 146 | 875d0298-6ed7-43fc-a468-da6042388ac0 | TS01_ROOT-110_20220419054747 | BackedUp | NULL | 2022-04-19 05:47:47 | NULL | NULL |
>>>> | 154 | 0122c9b3-a9f8-473f-a8d7-6e69ba77265b | TS01_ROOT-110_20220420054748 | BackedUp | NULL | 2022-04-20 05:47:48 | NULL | NULL |
>>>> | 157 | fe2248c6-aa80-4e9f-b539-7f495f1ff39f | Manual - Apr 20 | BackedUp | NULL | 2022-04-21 01:50:16 | NULL | NULL |
>>>> | 163 | 32923ad7-cbe3-4708-95d8-23eca921d711 | TS01_ROOT-110_20220421054817 | BackedUp | NULL | 2022-04-21 05:48:17 | NULL | NULL |
>>>> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------+---------------+
>>>>
>>>>
>>>>
>>>>>
>>>>> 3. SELECT volume_id, store_id, store_role, install_path, size, state FROM cloud.snapshot_store_ref WHERE snapshot_id = <snapshot_id_from_step_2>
>>>>>
>>>>
>>>> mysql> SELECT volume_id, store_id, store_role, install_path, size, state FROM cloud.snapshot_store_ref WHERE snapshot_id = '101';
>>>> +-----------+----------+------------+-------------------------------------------------------+-------------+------------+
>>>> | volume_id | store_id | store_role | install_path | size | state |
>>>> +-----------+----------+------------+-------------------------------------------------------+-------------+------------+
>>>> | 138 | 1 | Image | snapshots/16/111/a6b1bead-ca9a-464b-82c1-d631611ad790 | 85899345920 | Destroying |
>>>> +-----------+----------+------------+-------------------------------------------------------+-------------+------------+
>>>> 1 row in set (0.00 sec)
>>>>
>>>> mysql> SELECT volume_id, store_id, store_role, install_path, size, state FROM cloud.snapshot_store_ref WHERE snapshot_id = '109';
>>>> +-----------+----------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
>>>> | volume_id | store_id | store_role | install_path | size | state |
>>>> +-----------+----------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
>>>> | 138 | 5 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/66210b31-7445-4586-89ea-52a37f5b1ab5 | 85899345920 | Ready |
>>>> | 138 | 1 | Image | snapshots/16/111/66210b31-7445-4586-89ea-52a37f5b1ab5 | 85899345920 | Ready |
>>>> +-----------+----------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
>>>> 2 rows in set (0.00 sec)
>>>>
>>>> Regards,
>>>> Antoine
>>>>
>>>>
>>>>> Regards,
>>>>> Suresh
>>>>>
>>>>> On 21/04/22, 8:45 AM, "Antoine Boucher" <antoineb@haltondc.com <ma...@haltondc.com> <mailto:antoineb@haltondc.com <ma...@haltondc.com>>> wrote:
>>>>>
>>>>> Unfortunately I spoke too fast. The list is back but listing the snapshots of the volume with issue still show a blank list with error “undefined”.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Apr 20, 2022, at 23:11, Antoine Boucher <antoineb@haltondc.com <ma...@haltondc.com> <mailto:antoineb@haltondc.com <ma...@haltondc.com>>> wrote:
>>>>>
>>>>> I was able to recover from the situation by forcing a snapshot on the volume with issue.
>>>>>
>>>>> On Apr 20, 2022, at 22:00, Antoine Boucher <antoineb@haltondc.com <ma...@haltondc.com> <mailto:antoineb@haltondc.com <ma...@haltondc.com>>> wrote:
>>>>>
>>>>> I pin-pointed the volume with the issue by going through all volumes and clicking the “view snapshot"
>>>>>
>>>>>
>>>>>
>>>>>> On Apr 20, 2022, at 9:35 PM, Antoine Boucher <antoineb@haltondc.com <ma...@haltondc.com> <mailto:antoineb@haltondc.com <ma...@haltondc.com>>> wrote:
>>>>>>
>>>>>> I’m no longer able to see my list of snapshots (storage > snapshots), instead I see a small popup window with the word “undefined" on an empty list snapshot list.
>>>>>>
>>>>>> Has anyone seen this issue? I restart and rebooted the Management server but no resolution.
>>>>>>
>>>>>> Here are my logs:
>>>>>>
>>>>>> 2022-04-20 21:17:02,428 DEBUG [c.c.a.ApiServlet] (qtp515715487-292:ctx-a220f915) (logid:624389ae) ===START=== 10.101.254.1 -- GET listall=true&page=1&pagesize=20&command=listSnapshots&response=json
>>>>>> 2022-04-20 21:17:02,439 DEBUG [c.c.a.ApiServer] (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) CIDRs from which account 'Acct[2cee75f9-8bc4-11ec-9c43-001e67fd4838-admin] -- Account {"id": 2, "name": "admin", "uuid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838"}' is allowed to perform API calls: 0.0.0.0/0,::/0
>>>>>> 2022-04-20 21:17:02,675 ERROR [c.c.a.ApiServer] (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) unhandled exception executing api command: [Ljava.lang.String;@42fa8f7d
>>>>>> java.lang.NullPointerException
>>>>>> at org.apache.cloudstack.storage.snapshot.StorageSystemSnapshotStrategy.canHandle(StorageSystemSnapshotStrategy.java:985)
>>>>>> at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:72)
>>>>>> at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:69)
>>>>>> at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.bestMatch(StorageStrategyFactoryImpl.java:95)
>>>>>> at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.getSnapshotStrategy(StorageStrategyFactoryImpl.java:69)
>>>>>> at org.apache.cloudstack.storage.snapshot.SnapshotObject.isRevertable(SnapshotObject.java:156)
>>>>>> at com.cloud.api.ApiResponseHelper.createSnapshotResponse(ApiResponseHelper.java:591)
>>>>>> at org.apache.cloudstack.api.command.user.snapshot.ListSnapshotsCmd.execute(ListSnapshotsCmd.java:117)
>>>>>> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
>>>>>> at com.cloud.api.ApiServer.queueCommand(ApiServer.java:772)
>>>>>> at com.cloud.api.ApiServer.handleRequest(ApiServer.java:596)
>>>>>> at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:321)
>>>>>> at com.cloud.api.ApiServlet$1.run(ApiServlet.java:134)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>>>>> at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:131)
>>>>>> at com.cloud.api.ApiServlet.doGet(ApiServlet.java:93)
>>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
>>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
>>>>>> at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
>>>>>> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
>>>>>> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
>>>>>> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>>>>>> at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
>>>>>> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
>>>>>> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
>>>>>> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
>>>>>> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
>>>>>> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
>>>>>> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
>>>>>> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
>>>>>> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
>>>>>> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
>>>>>> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
>>>>>> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>>>>>> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
>>>>>> at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
>>>>>> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
>>>>>> at org.eclipse.jetty.server.Server.handle(Server.java:516)
>>>>>> at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400)
>>>>>> at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645)
>>>>>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392)
>>>>>> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
>>>>>> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
>>>>>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
>>>>>> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
>>>>>> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
>>>>>> at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
>>>>>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
>>>>>> at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
>>>>>> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
>>>>>> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
>>>>>> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
>>>>>> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
>>>>>> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
>>>>>> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
>>>>>> at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
>>>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>>>> 2022-04-20 21:17:02,677 DEBUG [c.c.a.ApiServlet] (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) ===END=== 10.101.254.1 -- GET listall=true&page=1&pagesize=20&command=listSnapshots&response=json
>>>>>>
>>>>>> Antoine Boucher
>>>>>>
>>>>>>
>>>>>> Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential, and may be privileged. If you are not the intended recipient, you are hereby notified that any review, retransmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, and delete this message and any attachments from your system.
Re: NPE while listing Snapshots
Posted by Suresh Anaparti <Su...@shapeblue.com>.
Hi Antoine,
Thanks for sharing these details. The delete snapshot cmd failed with NPE, as the underlying primary storage doesn't exists.
Usually, when a snapshot is deleted, the backed up snapshot file is removed from the secondary storage, the status in the cloud.snapshots table for that snaphshot is marked as Destroyed, and the state in the cloud.snapshot_store_ref table marked as Destroyed for the entries of that snapshot (both, for store_role Primary and Image). Later, the storage garbage collector (when enabled), sets the removed date in cloud.snapshots table and removes the entries in cloud.snapshot_store_ref table.
You can manually update the DB with the same behavior, first try with one snapshot (117) and if no issues, apply it to others (109,127,138).
Sample sqls for snapshot id=117 below.
- UPDATE cloud.snapshots SET status = 'Destroyed', removed = now() WHERE id = 117;
- DELETE FROM cloud.snapshot_store_ref WHERE snapshot_id = 117;
'111' corresponds to volume id, in the snapshot location 'snapshots/16/111/66210b31-7445-4586-89ea-52a37f5b1ab5'. I see the record with volume id 111 is expunged (migrated to new id) from the email thread, not sure about volume ids 110 and 112. What's the output of the sql below.
- SELECT id, uuid, name, account_id, pool_id, path, size, volume_type, state, removed FROM cloud.volumes WHERE id IN (110, 111, 112, 155);
Regards,
Suresh
On 17/05/22, 4:36 AM, "Antoine Boucher" <an...@haltondc.com> wrote:
Hi Suresh,
Here ids the logs of Cmk delete snapshot id=117 and similar for id=109,127,138
2022-05-16 18:18:49,667 DEBUG [c.c.a.ApiServer] (qtp515715487-147449:ctx-9470efda ctx-7b642401) (logid:dc836839) CIDRs from which account 'Acct[2cee75f9-8bc4-11ec-9c43-001e67fd4838-admin] -- Account {"id": 2, "name": "admin", "uuid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838"}' is allowed to perform API calls: 0.0.0.0/0,::/0
2022-05-16 18:18:49,689 DEBUG [c.c.a.ApiServlet] (qtp515715487-147449:ctx-9470efda ctx-7b642401) (logid:dc836839) ===END=== 0:0:0:0:0:0:0:1 -- GET command=deleteSnapshot&id=117&response=json&sessionkey=zLfGQdrCq6kxldkzrnAxYZ8HWQE
2022-05-16 18:18:49,689 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-204:ctx-1a4cbdc0 job-2964) (logid:161004ff) Add job-2964 into job monitoring
2022-05-16 18:18:49,704 DEBUG [c.c.a.ApiServlet] (qtp515715487-290:ctx-b653031c) (logid:4c211dfd) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobid=e6172481-968c-4ee4-993e-a9851af91c29&response=json&sessionkey=zLfGQdrCq6kxldkzrnAxYZ8HWQE
2022-05-16 18:18:49,715 DEBUG [c.c.a.ApiServer] (qtp515715487-290:ctx-b653031c ctx-61ed5689) (logid:4c211dfd) CIDRs from which account 'Acct[2cee75f9-8bc4-11ec-9c43-001e67fd4838-admin] -- Account {"id": 2, "name": "admin", "uuid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838"}' is allowed to perform API calls: 0.0.0.0/0,::/0
2022-05-16 18:18:49,736 DEBUG [c.c.a.ApiServlet] (qtp515715487-290:ctx-b653031c ctx-61ed5689) (logid:4c211dfd) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobid=e6172481-968c-4ee4-993e-a9851af91c29&response=json&sessionkey=zLfGQdrCq6kxldkzrnAxYZ8HWQE
2022-05-16 18:18:49,741 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-204:ctx-1a4cbdc0 job-2964) (logid:e6172481) Unexpected exception while executing org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd
java.lang.NullPointerException
at com.cloud.storage.snapshot.SnapshotManagerImpl.getDataStoreRole(SnapshotManagerImpl.java:1354)
at com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshot(SnapshotManagerImpl.java:591)
at jdk.internal.reflect.GeneratedMethodAccessor4669.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy218.deleteSnapshot(Unknown Source)
at org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd.execute(DeleteSnapshotCmd.java:103)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-05-16 18:18:49,748 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-204:ctx-1a4cbdc0 job-2964) (logid:e6172481) Remove job-2964 from job monitoring
2022-05-16 18:18:50,692 DEBUG [c.c.a.ApiServlet] (qtp515715487-147221:ctx-4c1cd6c0) (logid:3652b6cf) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobid=e6172481-968c-4ee4-993e-a9851af91c29&response=json&sessionkey=zLfGQdrCq6kxldkzrnAxYZ8HWQE
2022-05-16 18:18:50,706 DEBUG [c.c.a.ApiServer] (qtp515715487-147221:ctx-4c1cd6c0 ctx-9cbca8dc) (logid:3652b6cf) CIDRs from which account 'Acct[2cee75f9-8bc4-11ec-9c43-001e67fd4838-admin] -- Account {"id": 2, "name": "admin", "uuid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838"}' is allowed to perform API calls: 0.0.0.0/0,::/0
2022-05-16 18:18:50,723 DEBUG [c.c.a.ApiServlet] (qtp515715487-147221:ctx-4c1cd6c0 ctx-9cbca8dc) (logid:3652b6cf) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobid=e6172481-968c-4ee4-993e-a9851af91c29&response=json&sessionkey=zLfGQdrCq6kxldkzrnAxYZ8HWQE
My snapshot.backup.to.secondary is configured to ‘true’
The snapshot location 'snapshots/16/111/66210b31-7445-4586-89ea-52a37f5b1ab5’ for id=117 no longer exist. All of the concerned snapshots reference 'snapshots/16/111/...' for location but but directory 111 no longer exist. 110 and 112 exists.
Regards,
Antoine
Antoine Boucher
AntoineB@haltondc.com
[o] +1-226-505-9734
www.haltondc.com
> On May 13, 2022, at 9:18 AM, Suresh Anaparti <Su...@shapeblue.com> wrote:
>
> Hi Antoine,
>
> What is the error in the management server logs while deleting the ‘BackedUp’ snapshots? Can you share the log if possible.
>
> Are these snapshots backed up to secondary storage as well (and what’s the config ‘snapshot.backup.to.secondary’ value when taking snapshot )? If yes, can you find the snapshot files at the install_path there? You can check the install_path details in cloud.snapshot_store_ref table with volume_id: 155 and store_role: Image.
>
>
> Regards,
> Suresh
>
>
>
>
> From: Antoine Boucher <antoineb@haltondc.com <ma...@haltondc.com>>
> Reply to: "users@cloudstack.apache.org <ma...@cloudstack.apache.org>" <users@cloudstack.apache.org <ma...@cloudstack.apache.org>>
> Date: Wednesday, 11 May 2022 at 9:15 PM
> To: users <us...@cloudstack.apache.org>
> Subject: Re: NPE while listing Snapshots
>
> Too fast id=146 also failed
>
> {
> "accountid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838",
> "cmd": "org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd",
> "completed": "2022-05-11T11:43:01-0400",
> "created": "2022-05-11T11:38:00-0400",
> "jobid": "0bd30e5c-2e61-4988-8152-36835b791769",
> "jobinstanceid": "875d0298-6ed7-43fc-a468-da6042388ac0",
> "jobinstancetype": "Snapshot",
> "jobprocstatus": 0,
> "jobresult": {
> "errorcode": 530,
> "errortext": "Failed to delete snapshot:com.cloud.utils.exception.CloudRuntimeException: Failed to remove snapshot "
> },
> "jobresultcode": 530,
> "jobresulttype": "object",
> "jobstatus": 2,
> "userid": "2cefb708-8bc4-11ec-9c43-001e67fd4838"
> }
> 🙈 Error: async API failed for job 0bd30e5c-2e61-4988-8152-36835b791769
>
>
>
>
>
> Antoine Boucher
> AntoineB@haltondc.com <ma...@haltondc.com>
> [o] +1-226-505-9734
> www.haltondc.com <http://www.haltondc.com/>
>
> “Data security made simple and affordable”
>
>
>
>
>
>
>
> Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential, and may be privileged. If you are not the intended recipient, you are hereby notified that any review, retransmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, and delete this message and any attachments from your system.
>
>
>
>
>> On May 11, 2022, at 11:41 AM, Antoine Boucher <antoineb@haltondc.com <ma...@haltondc.com>> wrote:
>>
>> Hello Suresh,
>>
>> That primary storage was part of the local storage of a host that was removed.
>>
>> SELECT name, uuid, pool_type, scope, created, removed FROM cloud.storage_pool WHERE id = 5
>> +----------------+------+------------+-------+---------------------+---------------------+
>> | name | uuid | pool_type | scope | created | removed |
>> +----------------+------+------------+-------+---------------------+---------------------+
>> | cs-kvm03-local | NULL | Filesystem | HOST | 2022-02-12 15:01:45 | 2022-04-19 02:11:51 |
>> +----------------+------+------------+-------+---------------------+——————————+
>>
>>
>>
>>
>> SELECT volume_id, store_id, snapshot_id, store_role, install_path, size, state FROM cloud.snapshot_store_ref WHERE store_id = '5';
>> +-----------+----------+-------------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
>> | volume_id | store_id | snapshot_id | store_role | install_path | size | state |
>> +-----------+----------+-------------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
>> | 155 | 5 | 109 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/66210b31-7445-4586-89ea-52a37f5b1ab5 | 85899345920 | Ready |
>> | 155 | 5 | 117 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/a2c4fbbb-12b2-4c43-9647-375f6e334a46 | 85899345920 | Ready |
>> | 155 | 5 | 127 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/794d6dff-ba45-4a56-8377-1e48034bf014 | 85899345920 | Ready |
>> | 155 | 5 | 138 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/38648d51-bb19-49a2-b01f-12c4e6ed4529 | 85899345920 | Ready |
>> +-----------+----------+-------------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
>>
>>
>>
>>
>> All 155 snapshots have the same account_id and domain_id.
>>
>>
>>
>>
>> SELECT id, uuid, name, status, path, created, removed, location_type FROM cloud.snapshots WHERE volume_id = '155';
>> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------------------+---------------+
>> | id | uuid | name | status | path | created | removed | location_type |
>> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------------------+---------------+
>> | 101 | 13870bbd-d7d7-4664-a22f-f09c80a56d41 | TS01_ROOT-110_20220414054855 | Destroyed | NULL | 2022-04-14 05:48:55 | NULL | NULL |
>> | 109 | 77d84bf6-f601-44a6-a239-821f735a7f48 | TS01_ROOT-110_20220415054855 | BackedUp | NULL | 2022-04-15 05:48:55 | NULL | NULL |
>> | 117 | 7db02f43-a9d8-4c73-905a-1524d90f570a | TS01_ROOT-110_20220416054855 | BackedUp | NULL | 2022-04-16 05:48:55 | NULL | NULL |
>> | 127 | 47288ab9-2da2-40f2-98e3-9ced3f8e87df | TS01_ROOT-110_20220417054855 | BackedUp | NULL | 2022-04-17 05:48:55 | NULL | NULL |
>> | 138 | e9639f4a-fa09-4698-849b-572011045993 | TS01_ROOT-110_20220418054855 | BackedUp | NULL | 2022-04-18 05:48:55 | NULL | NULL |
>> | 146 | 875d0298-6ed7-43fc-a468-da6042388ac0 | TS01_ROOT-110_20220419054747 | BackedUp | NULL | 2022-04-19 05:47:47 | NULL | NULL |
>> | 154 | 0122c9b3-a9f8-473f-a8d7-6e69ba77265b | TS01_ROOT-110_20220420054748 | Destroyed | NULL | 2022-04-20 05:47:48 | NULL | NULL |
>> | 157 | fe2248c6-aa80-4e9f-b539-7f495f1ff39f | Manual - Apr 20 | Destroyed | NULL | 2022-04-21 01:50:16 | NULL | NULL |
>> | 163 | 32923ad7-cbe3-4708-95d8-23eca921d711 | TS01_ROOT-110_20220421054817 | Destroyed | NULL | 2022-04-21 05:48:17 | NULL | NULL |
>> | 172 | 6e03ea90-e8b0-4405-aed6-7bd17a40de57 | TS01_ROOT-110_20220422054933 | Destroyed | NULL | 2022-04-22 05:49:33 | NULL | NULL |
>> | 181 | 9059a26a-0e4e-4d46-95f2-b0a9236ba90a | TS01_ROOT-110_20220423054933 | Destroyed | NULL | 2022-04-23 05:49:33 | NULL | NULL |
>> | 193 | c9828cbc-f282-4dfe-b4cf-9df792c522c2 | TS01_ROOT-110_20220424054934 | BackedUp | NULL | 2022-04-24 05:49:34 | NULL | NULL |
>> | 207 | 2e5dfd48-c4de-45ff-bbe1-4cf0d90bb698 | TS01_ROOT-110_20220425054934 | Destroyed | NULL | 2022-04-25 05:49:34 | NULL | NULL |
>> | 217 | 99d9d9ff-44ae-41a9-9761-f722b370c01c | TS01_ROOT-110_20220426054934 | Destroyed | NULL | 2022-04-26 05:49:34 | NULL | NULL |
>> | 227 | 1df14a7b-ba56-4d26-aeb4-c79d4de978fc | TS01_ROOT-110_20220427054934 | BackedUp | NULL | 2022-04-27 05:49:34 | NULL | NULL |
>> | 237 | 657479e0-3587-4a72-a8c8-bd7dff54a5e4 | TS01_ROOT-110_20220428054934 | BackedUp | NULL | 2022-04-28 05:49:34 | NULL | NULL |
>> | 248 | 2ef0b539-be01-4b0c-9fa5-5fc9c60a6a96 | TS01_ROOT-110_20220429054935 | BackedUp | NULL | 2022-04-29 05:49:35 | NULL | NULL |
>> | 257 | 8ab75270-8e3d-471e-8e4a-8d5f3cbc6bc1 | TS01_ROOT-110_20220430054935 | BackedUp | NULL | 2022-04-30 05:49:35 | NULL | NULL |
>> | 259 | a1e77bec-8fc7-457f-b0a5-961c339487f7 | TS01_ROOT-110_20220430060935 | Destroyed | NULL | 2022-04-30 06:09:35 | 2022-05-09 20:02:38 | NULL |
>> | 313 | 8ad0c04a-1d74-4d51-8223-a6aa3a82d62c | TS01_ROOT-110_20220509015742 | BackedUp | NULL | 2022-05-09 01:57:42 | NULL | NULL |
>> | 327 | 0f5e9366-747f-4a34-bec4-1fd31c461ac3 | TS01_ROOT-110_20220509170242 | BackedUp | NULL | 2022-05-09 17:02:42 | NULL | NULL |
>> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------------------+---------------+
>>
>> How would you suggest I resolve the issue moving forward?
>>
>> I tried the following with error…
>>
>> (nimbus) 🐱 > delete snapshot id=109
>> {
>> "accountid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838",
>> "cmd": "org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd",
>> "completed": "2022-05-11T11:32:08-0400",
>> "created": "2022-05-11T11:32:08-0400",
>> "jobid": "29661b2b-46c5-42bd-9974-d1a226f3fe68",
>> "jobinstanceid": "77d84bf6-f601-44a6-a239-821f735a7f48",
>> "jobinstancetype": "Snapshot",
>> "jobprocstatus": 0,
>> "jobresult": {
>> "errorcode": 530,
>> "errortext": "Command failed due to Internal Server Error"
>> },
>> "jobresultcode": 530,
>> "jobresulttype": "object",
>> "jobstatus": 2,
>> "userid": "2cefb708-8bc4-11ec-9c43-001e67fd4838"
>> }
>> 🙈 Error: async API failed for job 29661b2b-46c5-42bd-9974-d1a226f3fe68
>>
>>
>> Deleting some table rows in the right sequence?
>>
>> Same issue with id=117 to id=138, but I was able to delete id=146
>>
>> Regards,
>> Antoine
>>
>>
>>
>>
>>> On Apr 27, 2022, at 9:00 AM, Suresh Anaparti <Suresh.Anaparti@shapeblue.com <ma...@shapeblue.com>> wrote:
>>>
>>> Hi Antoine,
>>>
>>> The NPE error seems to be due to the primary storage with id '5', when checking for snapshot revertible or not during list snapshots cmd. Is that primary storage removed? Can you check with sql below.
>>>
>>> SELECT name, uuid, pool_type, scope, created, removed FROM cloud.storage_pool WHERE id = 5
>>>
>>>
>>> I think, the account/user might not have the permissions to list all snapshots. You can check the account / domain details in snapshots table.
>>>
>>>
>>> Regards,
>>> Suresh
>>>
>>> On 21/04/22, 6:50 PM, "Antoine Boucher" <antoineb@haltondc.com <ma...@haltondc.com>> wrote:
>>>
>>> Thank you Suresh, you will find my answers below,
>>>
>>>
>>>
>>>
>>>
>>>
>>>> On Apr 21, 2022, at 1:02 AM, Suresh Anaparti <Suresh.Anaparti@shapeblue.com <ma...@shapeblue.com>> wrote:
>>>>
>>>> Hi Antoine,
>>>>
>>>> What is the CloudStack version, where you see this issue.
>>>
>>> Latest - 4.16.1
>>>
>>>
>>>
>>>> When you noticed this issue, is it while listing snapshots for a particular volume, or all volumes? Try list snapshots from API/cmk as well.
>>>
>>> This is a customer VM, the account may have hit its snapshot limit during a snapshot.
>>>
>>> cmk list snapshots > only lists 26 snapshots out of 62
>>>
>>> cmk list snapshots volumeid= > Works for some volume ids, provides null result while the WebUI lists the correct snapshots for others and provides null results while show undefined error for one VM mentioned below
>>>
>>>
>>>
>>>>
>>>> Also, Check the snapshots details for the volume (with issue) in the db, using the following sql queries.
>>>>
>>>> 1. SELECT id, uuid, state, pool_id, path, size, removed FROM cloud.volumes WHERE name LIKE '<volume_name>'
>>>>
>>>
>>> mysql> SELECT id, uuid, state, pool_id, path, size, removed FROM cloud.volumes WHERE name LIKE 'ROOT-110';
>>> +-----+--------------------------------------+----------+---------+--------------------------------------+-------------+---------------------+
>>> | id | uuid | state | pool_id | path | size | removed |
>>> +-----+--------------------------------------+----------+---------+--------------------------------------+-------------+---------------------+
>>> | 111 | NULL | Expunged | 5 | da585fd4-b10f-480b-9287-45ecb28b6355 | 85899345920 | 2022-04-19 01:29:08 |
>>> | 138 | da585fd4-b10f-480b-9287-45ecb28b6355 | Ready | 6 | 04bd5d13-72ca-4240-93ce-ed1b015d3437 | 85899345920 | NULL |
>>> +-----+--------------------------------------+----------+---------+--------------------------------------+-------------+---------------------+
>>>
>>>
>>>
>>>> 2. SELECT id, uuid, name, status, path, created, removed, location_type FROM cloud.snapshots WHERE volume_id = <volume_id_from_step_1>;
>>>
>>> mysql> SELECT id, uuid, name, status, path, created, removed, location_type FROM cloud.snapshots WHERE volume_id = '138';
>>> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------+---------------+
>>> | id | uuid | name | status | path | created | removed | location_type |
>>> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------+---------------+
>>> | 101 | 13870bbd-d7d7-4664-a22f-f09c80a56d41 | TS01_ROOT-110_20220414054855 | Destroyed | NULL | 2022-04-14 05:48:55 | NULL | NULL |
>>> | 109 | 77d84bf6-f601-44a6-a239-821f735a7f48 | TS01_ROOT-110_20220415054855 | BackedUp | NULL | 2022-04-15 05:48:55 | NULL | NULL |
>>> | 117 | 7db02f43-a9d8-4c73-905a-1524d90f570a | TS01_ROOT-110_20220416054855 | BackedUp | NULL | 2022-04-16 05:48:55 | NULL | NULL |
>>> | 127 | 47288ab9-2da2-40f2-98e3-9ced3f8e87df | TS01_ROOT-110_20220417054855 | BackedUp | NULL | 2022-04-17 05:48:55 | NULL | NULL |
>>> | 138 | e9639f4a-fa09-4698-849b-572011045993 | TS01_ROOT-110_20220418054855 | BackedUp | NULL | 2022-04-18 05:48:55 | NULL | NULL |
>>> | 146 | 875d0298-6ed7-43fc-a468-da6042388ac0 | TS01_ROOT-110_20220419054747 | BackedUp | NULL | 2022-04-19 05:47:47 | NULL | NULL |
>>> | 154 | 0122c9b3-a9f8-473f-a8d7-6e69ba77265b | TS01_ROOT-110_20220420054748 | BackedUp | NULL | 2022-04-20 05:47:48 | NULL | NULL |
>>> | 157 | fe2248c6-aa80-4e9f-b539-7f495f1ff39f | Manual - Apr 20 | BackedUp | NULL | 2022-04-21 01:50:16 | NULL | NULL |
>>> | 163 | 32923ad7-cbe3-4708-95d8-23eca921d711 | TS01_ROOT-110_20220421054817 | BackedUp | NULL | 2022-04-21 05:48:17 | NULL | NULL |
>>> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------+---------------+
>>>
>>>
>>>
>>>>
>>>> 3. SELECT volume_id, store_id, store_role, install_path, size, state FROM cloud.snapshot_store_ref WHERE snapshot_id = <snapshot_id_from_step_2>
>>>>
>>>
>>> mysql> SELECT volume_id, store_id, store_role, install_path, size, state FROM cloud.snapshot_store_ref WHERE snapshot_id = '101';
>>> +-----------+----------+------------+-------------------------------------------------------+-------------+------------+
>>> | volume_id | store_id | store_role | install_path | size | state |
>>> +-----------+----------+------------+-------------------------------------------------------+-------------+------------+
>>> | 138 | 1 | Image | snapshots/16/111/a6b1bead-ca9a-464b-82c1-d631611ad790 | 85899345920 | Destroying |
>>> +-----------+----------+------------+-------------------------------------------------------+-------------+------------+
>>> 1 row in set (0.00 sec)
>>>
>>> mysql> SELECT volume_id, store_id, store_role, install_path, size, state FROM cloud.snapshot_store_ref WHERE snapshot_id = '109';
>>> +-----------+----------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
>>> | volume_id | store_id | store_role | install_path | size | state |
>>> +-----------+----------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
>>> | 138 | 5 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/66210b31-7445-4586-89ea-52a37f5b1ab5 | 85899345920 | Ready |
>>> | 138 | 1 | Image | snapshots/16/111/66210b31-7445-4586-89ea-52a37f5b1ab5 | 85899345920 | Ready |
>>> +-----------+----------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
>>> 2 rows in set (0.00 sec)
>>>
>>> Regards,
>>> Antoine
>>>
>>>
>>>> Regards,
>>>> Suresh
>>>>
>>>> On 21/04/22, 8:45 AM, "Antoine Boucher" <antoineb@haltondc.com <ma...@haltondc.com>> wrote:
>>>>
>>>> Unfortunately I spoke too fast. The list is back but listing the snapshots of the volume with issue still show a blank list with error “undefined”.
>>>>
>>>>
>>>>
>>>>
>>>> On Apr 20, 2022, at 23:11, Antoine Boucher <antoineb@haltondc.com <ma...@haltondc.com>> wrote:
>>>>
>>>> I was able to recover from the situation by forcing a snapshot on the volume with issue.
>>>>
>>>> On Apr 20, 2022, at 22:00, Antoine Boucher <antoineb@haltondc.com <ma...@haltondc.com>> wrote:
>>>>
>>>> I pin-pointed the volume with the issue by going through all volumes and clicking the “view snapshot"
>>>>
>>>>
>>>>
>>>>> On Apr 20, 2022, at 9:35 PM, Antoine Boucher <antoineb@haltondc.com <ma...@haltondc.com>> wrote:
>>>>>
>>>>> I’m no longer able to see my list of snapshots (storage > snapshots), instead I see a small popup window with the word “undefined" on an empty list snapshot list.
>>>>>
>>>>> Has anyone seen this issue? I restart and rebooted the Management server but no resolution.
>>>>>
>>>>> Here are my logs:
>>>>>
>>>>> 2022-04-20 21:17:02,428 DEBUG [c.c.a.ApiServlet] (qtp515715487-292:ctx-a220f915) (logid:624389ae) ===START=== 10.101.254.1 -- GET listall=true&page=1&pagesize=20&command=listSnapshots&response=json
>>>>> 2022-04-20 21:17:02,439 DEBUG [c.c.a.ApiServer] (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) CIDRs from which account 'Acct[2cee75f9-8bc4-11ec-9c43-001e67fd4838-admin] -- Account {"id": 2, "name": "admin", "uuid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838"}' is allowed to perform API calls: 0.0.0.0/0,::/0
>>>>> 2022-04-20 21:17:02,675 ERROR [c.c.a.ApiServer] (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) unhandled exception executing api command: [Ljava.lang.String;@42fa8f7d
>>>>> java.lang.NullPointerException
>>>>> at org.apache.cloudstack.storage.snapshot.StorageSystemSnapshotStrategy.canHandle(StorageSystemSnapshotStrategy.java:985)
>>>>> at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:72)
>>>>> at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:69)
>>>>> at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.bestMatch(StorageStrategyFactoryImpl.java:95)
>>>>> at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.getSnapshotStrategy(StorageStrategyFactoryImpl.java:69)
>>>>> at org.apache.cloudstack.storage.snapshot.SnapshotObject.isRevertable(SnapshotObject.java:156)
>>>>> at com.cloud.api.ApiResponseHelper.createSnapshotResponse(ApiResponseHelper.java:591)
>>>>> at org.apache.cloudstack.api.command.user.snapshot.ListSnapshotsCmd.execute(ListSnapshotsCmd.java:117)
>>>>> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
>>>>> at com.cloud.api.ApiServer.queueCommand(ApiServer.java:772)
>>>>> at com.cloud.api.ApiServer.handleRequest(ApiServer.java:596)
>>>>> at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:321)
>>>>> at com.cloud.api.ApiServlet$1.run(ApiServlet.java:134)
>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>>>> at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:131)
>>>>> at com.cloud.api.ApiServlet.doGet(ApiServlet.java:93)
>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
>>>>> at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
>>>>> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
>>>>> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
>>>>> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>>>>> at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
>>>>> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
>>>>> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
>>>>> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
>>>>> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
>>>>> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
>>>>> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
>>>>> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
>>>>> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
>>>>> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
>>>>> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
>>>>> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>>>>> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
>>>>> at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
>>>>> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
>>>>> at org.eclipse.jetty.server.Server.handle(Server.java:516)
>>>>> at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400)
>>>>> at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645)
>>>>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392)
>>>>> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
>>>>> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
>>>>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
>>>>> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
>>>>> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
>>>>> at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
>>>>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
>>>>> at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
>>>>> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
>>>>> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
>>>>> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
>>>>> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
>>>>> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
>>>>> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
>>>>> at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
>>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>>> 2022-04-20 21:17:02,677 DEBUG [c.c.a.ApiServlet] (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) ===END=== 10.101.254.1 -- GET listall=true&page=1&pagesize=20&command=listSnapshots&response=json
>>>>>
>>>>> Antoine Boucher
>>>>>
>>>>>
>>>>> Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential, and may be privileged. If you are not the intended recipient, you are hereby notified that any review, retransmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, and delete this message and any attachments from your system.
>>>>>
Re: NPE while listing Snapshots
Posted by Antoine Boucher <an...@haltondc.com>.
Hi Suresh,
Here ids the logs of Cmk delete snapshot id=117 and similar for id=109,127,138
2022-05-16 18:18:49,667 DEBUG [c.c.a.ApiServer] (qtp515715487-147449:ctx-9470efda ctx-7b642401) (logid:dc836839) CIDRs from which account 'Acct[2cee75f9-8bc4-11ec-9c43-001e67fd4838-admin] -- Account {"id": 2, "name": "admin", "uuid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838"}' is allowed to perform API calls: 0.0.0.0/0,::/0
2022-05-16 18:18:49,689 DEBUG [c.c.a.ApiServlet] (qtp515715487-147449:ctx-9470efda ctx-7b642401) (logid:dc836839) ===END=== 0:0:0:0:0:0:0:1 -- GET command=deleteSnapshot&id=117&response=json&sessionkey=zLfGQdrCq6kxldkzrnAxYZ8HWQE
2022-05-16 18:18:49,689 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-204:ctx-1a4cbdc0 job-2964) (logid:161004ff) Add job-2964 into job monitoring
2022-05-16 18:18:49,704 DEBUG [c.c.a.ApiServlet] (qtp515715487-290:ctx-b653031c) (logid:4c211dfd) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobid=e6172481-968c-4ee4-993e-a9851af91c29&response=json&sessionkey=zLfGQdrCq6kxldkzrnAxYZ8HWQE
2022-05-16 18:18:49,715 DEBUG [c.c.a.ApiServer] (qtp515715487-290:ctx-b653031c ctx-61ed5689) (logid:4c211dfd) CIDRs from which account 'Acct[2cee75f9-8bc4-11ec-9c43-001e67fd4838-admin] -- Account {"id": 2, "name": "admin", "uuid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838"}' is allowed to perform API calls: 0.0.0.0/0,::/0
2022-05-16 18:18:49,736 DEBUG [c.c.a.ApiServlet] (qtp515715487-290:ctx-b653031c ctx-61ed5689) (logid:4c211dfd) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobid=e6172481-968c-4ee4-993e-a9851af91c29&response=json&sessionkey=zLfGQdrCq6kxldkzrnAxYZ8HWQE
2022-05-16 18:18:49,741 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-204:ctx-1a4cbdc0 job-2964) (logid:e6172481) Unexpected exception while executing org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd
java.lang.NullPointerException
at com.cloud.storage.snapshot.SnapshotManagerImpl.getDataStoreRole(SnapshotManagerImpl.java:1354)
at com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshot(SnapshotManagerImpl.java:591)
at jdk.internal.reflect.GeneratedMethodAccessor4669.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy218.deleteSnapshot(Unknown Source)
at org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd.execute(DeleteSnapshotCmd.java:103)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-05-16 18:18:49,748 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-204:ctx-1a4cbdc0 job-2964) (logid:e6172481) Remove job-2964 from job monitoring
2022-05-16 18:18:50,692 DEBUG [c.c.a.ApiServlet] (qtp515715487-147221:ctx-4c1cd6c0) (logid:3652b6cf) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobid=e6172481-968c-4ee4-993e-a9851af91c29&response=json&sessionkey=zLfGQdrCq6kxldkzrnAxYZ8HWQE
2022-05-16 18:18:50,706 DEBUG [c.c.a.ApiServer] (qtp515715487-147221:ctx-4c1cd6c0 ctx-9cbca8dc) (logid:3652b6cf) CIDRs from which account 'Acct[2cee75f9-8bc4-11ec-9c43-001e67fd4838-admin] -- Account {"id": 2, "name": "admin", "uuid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838"}' is allowed to perform API calls: 0.0.0.0/0,::/0
2022-05-16 18:18:50,723 DEBUG [c.c.a.ApiServlet] (qtp515715487-147221:ctx-4c1cd6c0 ctx-9cbca8dc) (logid:3652b6cf) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobid=e6172481-968c-4ee4-993e-a9851af91c29&response=json&sessionkey=zLfGQdrCq6kxldkzrnAxYZ8HWQE
My snapshot.backup.to.secondary is configured to ‘true’
The snapshot location 'snapshots/16/111/66210b31-7445-4586-89ea-52a37f5b1ab5’ for id=117 no longer exist. All of the concerned snapshots reference 'snapshots/16/111/...' for location but but directory 111 no longer exist. 110 and 112 exists.
Regards,
Antoine
Antoine Boucher
AntoineB@haltondc.com
[o] +1-226-505-9734
www.haltondc.com
> On May 13, 2022, at 9:18 AM, Suresh Anaparti <Su...@shapeblue.com> wrote:
>
> Hi Antoine,
>
> What is the error in the management server logs while deleting the ‘BackedUp’ snapshots? Can you share the log if possible.
>
> Are these snapshots backed up to secondary storage as well (and what’s the config ‘snapshot.backup.to.secondary’ value when taking snapshot )? If yes, can you find the snapshot files at the install_path there? You can check the install_path details in cloud.snapshot_store_ref table with volume_id: 155 and store_role: Image.
>
>
> Regards,
> Suresh
>
>
>
>
> From: Antoine Boucher <antoineb@haltondc.com <ma...@haltondc.com>>
> Reply to: "users@cloudstack.apache.org <ma...@cloudstack.apache.org>" <users@cloudstack.apache.org <ma...@cloudstack.apache.org>>
> Date: Wednesday, 11 May 2022 at 9:15 PM
> To: users <us...@cloudstack.apache.org>
> Subject: Re: NPE while listing Snapshots
>
> Too fast id=146 also failed
>
> {
> "accountid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838",
> "cmd": "org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd",
> "completed": "2022-05-11T11:43:01-0400",
> "created": "2022-05-11T11:38:00-0400",
> "jobid": "0bd30e5c-2e61-4988-8152-36835b791769",
> "jobinstanceid": "875d0298-6ed7-43fc-a468-da6042388ac0",
> "jobinstancetype": "Snapshot",
> "jobprocstatus": 0,
> "jobresult": {
> "errorcode": 530,
> "errortext": "Failed to delete snapshot:com.cloud.utils.exception.CloudRuntimeException: Failed to remove snapshot "
> },
> "jobresultcode": 530,
> "jobresulttype": "object",
> "jobstatus": 2,
> "userid": "2cefb708-8bc4-11ec-9c43-001e67fd4838"
> }
> 🙈 Error: async API failed for job 0bd30e5c-2e61-4988-8152-36835b791769
>
>
>
>
>
> Antoine Boucher
> AntoineB@haltondc.com <ma...@haltondc.com>
> [o] +1-226-505-9734
> www.haltondc.com <http://www.haltondc.com/>
>
> “Data security made simple and affordable”
>
>
>
>
>
>
>
> Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential, and may be privileged. If you are not the intended recipient, you are hereby notified that any review, retransmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, and delete this message and any attachments from your system.
>
>
>
>
>> On May 11, 2022, at 11:41 AM, Antoine Boucher <antoineb@haltondc.com <ma...@haltondc.com>> wrote:
>>
>> Hello Suresh,
>>
>> That primary storage was part of the local storage of a host that was removed.
>>
>> SELECT name, uuid, pool_type, scope, created, removed FROM cloud.storage_pool WHERE id = 5
>> +----------------+------+------------+-------+---------------------+---------------------+
>> | name | uuid | pool_type | scope | created | removed |
>> +----------------+------+------------+-------+---------------------+---------------------+
>> | cs-kvm03-local | NULL | Filesystem | HOST | 2022-02-12 15:01:45 | 2022-04-19 02:11:51 |
>> +----------------+------+------------+-------+---------------------+——————————+
>>
>>
>>
>>
>> SELECT volume_id, store_id, snapshot_id, store_role, install_path, size, state FROM cloud.snapshot_store_ref WHERE store_id = '5';
>> +-----------+----------+-------------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
>> | volume_id | store_id | snapshot_id | store_role | install_path | size | state |
>> +-----------+----------+-------------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
>> | 155 | 5 | 109 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/66210b31-7445-4586-89ea-52a37f5b1ab5 | 85899345920 | Ready |
>> | 155 | 5 | 117 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/a2c4fbbb-12b2-4c43-9647-375f6e334a46 | 85899345920 | Ready |
>> | 155 | 5 | 127 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/794d6dff-ba45-4a56-8377-1e48034bf014 | 85899345920 | Ready |
>> | 155 | 5 | 138 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/38648d51-bb19-49a2-b01f-12c4e6ed4529 | 85899345920 | Ready |
>> +-----------+----------+-------------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
>>
>>
>>
>>
>> All 155 snapshots have the same account_id and domain_id.
>>
>>
>>
>>
>> SELECT id, uuid, name, status, path, created, removed, location_type FROM cloud.snapshots WHERE volume_id = '155';
>> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------------------+---------------+
>> | id | uuid | name | status | path | created | removed | location_type |
>> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------------------+---------------+
>> | 101 | 13870bbd-d7d7-4664-a22f-f09c80a56d41 | TS01_ROOT-110_20220414054855 | Destroyed | NULL | 2022-04-14 05:48:55 | NULL | NULL |
>> | 109 | 77d84bf6-f601-44a6-a239-821f735a7f48 | TS01_ROOT-110_20220415054855 | BackedUp | NULL | 2022-04-15 05:48:55 | NULL | NULL |
>> | 117 | 7db02f43-a9d8-4c73-905a-1524d90f570a | TS01_ROOT-110_20220416054855 | BackedUp | NULL | 2022-04-16 05:48:55 | NULL | NULL |
>> | 127 | 47288ab9-2da2-40f2-98e3-9ced3f8e87df | TS01_ROOT-110_20220417054855 | BackedUp | NULL | 2022-04-17 05:48:55 | NULL | NULL |
>> | 138 | e9639f4a-fa09-4698-849b-572011045993 | TS01_ROOT-110_20220418054855 | BackedUp | NULL | 2022-04-18 05:48:55 | NULL | NULL |
>> | 146 | 875d0298-6ed7-43fc-a468-da6042388ac0 | TS01_ROOT-110_20220419054747 | BackedUp | NULL | 2022-04-19 05:47:47 | NULL | NULL |
>> | 154 | 0122c9b3-a9f8-473f-a8d7-6e69ba77265b | TS01_ROOT-110_20220420054748 | Destroyed | NULL | 2022-04-20 05:47:48 | NULL | NULL |
>> | 157 | fe2248c6-aa80-4e9f-b539-7f495f1ff39f | Manual - Apr 20 | Destroyed | NULL | 2022-04-21 01:50:16 | NULL | NULL |
>> | 163 | 32923ad7-cbe3-4708-95d8-23eca921d711 | TS01_ROOT-110_20220421054817 | Destroyed | NULL | 2022-04-21 05:48:17 | NULL | NULL |
>> | 172 | 6e03ea90-e8b0-4405-aed6-7bd17a40de57 | TS01_ROOT-110_20220422054933 | Destroyed | NULL | 2022-04-22 05:49:33 | NULL | NULL |
>> | 181 | 9059a26a-0e4e-4d46-95f2-b0a9236ba90a | TS01_ROOT-110_20220423054933 | Destroyed | NULL | 2022-04-23 05:49:33 | NULL | NULL |
>> | 193 | c9828cbc-f282-4dfe-b4cf-9df792c522c2 | TS01_ROOT-110_20220424054934 | BackedUp | NULL | 2022-04-24 05:49:34 | NULL | NULL |
>> | 207 | 2e5dfd48-c4de-45ff-bbe1-4cf0d90bb698 | TS01_ROOT-110_20220425054934 | Destroyed | NULL | 2022-04-25 05:49:34 | NULL | NULL |
>> | 217 | 99d9d9ff-44ae-41a9-9761-f722b370c01c | TS01_ROOT-110_20220426054934 | Destroyed | NULL | 2022-04-26 05:49:34 | NULL | NULL |
>> | 227 | 1df14a7b-ba56-4d26-aeb4-c79d4de978fc | TS01_ROOT-110_20220427054934 | BackedUp | NULL | 2022-04-27 05:49:34 | NULL | NULL |
>> | 237 | 657479e0-3587-4a72-a8c8-bd7dff54a5e4 | TS01_ROOT-110_20220428054934 | BackedUp | NULL | 2022-04-28 05:49:34 | NULL | NULL |
>> | 248 | 2ef0b539-be01-4b0c-9fa5-5fc9c60a6a96 | TS01_ROOT-110_20220429054935 | BackedUp | NULL | 2022-04-29 05:49:35 | NULL | NULL |
>> | 257 | 8ab75270-8e3d-471e-8e4a-8d5f3cbc6bc1 | TS01_ROOT-110_20220430054935 | BackedUp | NULL | 2022-04-30 05:49:35 | NULL | NULL |
>> | 259 | a1e77bec-8fc7-457f-b0a5-961c339487f7 | TS01_ROOT-110_20220430060935 | Destroyed | NULL | 2022-04-30 06:09:35 | 2022-05-09 20:02:38 | NULL |
>> | 313 | 8ad0c04a-1d74-4d51-8223-a6aa3a82d62c | TS01_ROOT-110_20220509015742 | BackedUp | NULL | 2022-05-09 01:57:42 | NULL | NULL |
>> | 327 | 0f5e9366-747f-4a34-bec4-1fd31c461ac3 | TS01_ROOT-110_20220509170242 | BackedUp | NULL | 2022-05-09 17:02:42 | NULL | NULL |
>> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------------------+---------------+
>>
>> How would you suggest I resolve the issue moving forward?
>>
>> I tried the following with error…
>>
>> (nimbus) 🐱 > delete snapshot id=109
>> {
>> "accountid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838",
>> "cmd": "org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd",
>> "completed": "2022-05-11T11:32:08-0400",
>> "created": "2022-05-11T11:32:08-0400",
>> "jobid": "29661b2b-46c5-42bd-9974-d1a226f3fe68",
>> "jobinstanceid": "77d84bf6-f601-44a6-a239-821f735a7f48",
>> "jobinstancetype": "Snapshot",
>> "jobprocstatus": 0,
>> "jobresult": {
>> "errorcode": 530,
>> "errortext": "Command failed due to Internal Server Error"
>> },
>> "jobresultcode": 530,
>> "jobresulttype": "object",
>> "jobstatus": 2,
>> "userid": "2cefb708-8bc4-11ec-9c43-001e67fd4838"
>> }
>> 🙈 Error: async API failed for job 29661b2b-46c5-42bd-9974-d1a226f3fe68
>>
>>
>> Deleting some table rows in the right sequence?
>>
>> Same issue with id=117 to id=138, but I was able to delete id=146
>>
>> Regards,
>> Antoine
>>
>>
>>
>>
>>> On Apr 27, 2022, at 9:00 AM, Suresh Anaparti <Suresh.Anaparti@shapeblue.com <ma...@shapeblue.com>> wrote:
>>>
>>> Hi Antoine,
>>>
>>> The NPE error seems to be due to the primary storage with id '5', when checking for snapshot revertible or not during list snapshots cmd. Is that primary storage removed? Can you check with sql below.
>>>
>>> SELECT name, uuid, pool_type, scope, created, removed FROM cloud.storage_pool WHERE id = 5
>>>
>>>
>>> I think, the account/user might not have the permissions to list all snapshots. You can check the account / domain details in snapshots table.
>>>
>>>
>>> Regards,
>>> Suresh
>>>
>>> On 21/04/22, 6:50 PM, "Antoine Boucher" <antoineb@haltondc.com <ma...@haltondc.com>> wrote:
>>>
>>> Thank you Suresh, you will find my answers below,
>>>
>>>
>>>
>>>
>>>
>>>
>>>> On Apr 21, 2022, at 1:02 AM, Suresh Anaparti <Suresh.Anaparti@shapeblue.com <ma...@shapeblue.com>> wrote:
>>>>
>>>> Hi Antoine,
>>>>
>>>> What is the CloudStack version, where you see this issue.
>>>
>>> Latest - 4.16.1
>>>
>>>
>>>
>>>> When you noticed this issue, is it while listing snapshots for a particular volume, or all volumes? Try list snapshots from API/cmk as well.
>>>
>>> This is a customer VM, the account may have hit its snapshot limit during a snapshot.
>>>
>>> cmk list snapshots > only lists 26 snapshots out of 62
>>>
>>> cmk list snapshots volumeid= > Works for some volume ids, provides null result while the WebUI lists the correct snapshots for others and provides null results while show undefined error for one VM mentioned below
>>>
>>>
>>>
>>>>
>>>> Also, Check the snapshots details for the volume (with issue) in the db, using the following sql queries.
>>>>
>>>> 1. SELECT id, uuid, state, pool_id, path, size, removed FROM cloud.volumes WHERE name LIKE '<volume_name>'
>>>>
>>>
>>> mysql> SELECT id, uuid, state, pool_id, path, size, removed FROM cloud.volumes WHERE name LIKE 'ROOT-110';
>>> +-----+--------------------------------------+----------+---------+--------------------------------------+-------------+---------------------+
>>> | id | uuid | state | pool_id | path | size | removed |
>>> +-----+--------------------------------------+----------+---------+--------------------------------------+-------------+---------------------+
>>> | 111 | NULL | Expunged | 5 | da585fd4-b10f-480b-9287-45ecb28b6355 | 85899345920 | 2022-04-19 01:29:08 |
>>> | 138 | da585fd4-b10f-480b-9287-45ecb28b6355 | Ready | 6 | 04bd5d13-72ca-4240-93ce-ed1b015d3437 | 85899345920 | NULL |
>>> +-----+--------------------------------------+----------+---------+--------------------------------------+-------------+---------------------+
>>>
>>>
>>>
>>>> 2. SELECT id, uuid, name, status, path, created, removed, location_type FROM cloud.snapshots WHERE volume_id = <volume_id_from_step_1>;
>>>
>>> mysql> SELECT id, uuid, name, status, path, created, removed, location_type FROM cloud.snapshots WHERE volume_id = '138';
>>> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------+---------------+
>>> | id | uuid | name | status | path | created | removed | location_type |
>>> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------+---------------+
>>> | 101 | 13870bbd-d7d7-4664-a22f-f09c80a56d41 | TS01_ROOT-110_20220414054855 | Destroyed | NULL | 2022-04-14 05:48:55 | NULL | NULL |
>>> | 109 | 77d84bf6-f601-44a6-a239-821f735a7f48 | TS01_ROOT-110_20220415054855 | BackedUp | NULL | 2022-04-15 05:48:55 | NULL | NULL |
>>> | 117 | 7db02f43-a9d8-4c73-905a-1524d90f570a | TS01_ROOT-110_20220416054855 | BackedUp | NULL | 2022-04-16 05:48:55 | NULL | NULL |
>>> | 127 | 47288ab9-2da2-40f2-98e3-9ced3f8e87df | TS01_ROOT-110_20220417054855 | BackedUp | NULL | 2022-04-17 05:48:55 | NULL | NULL |
>>> | 138 | e9639f4a-fa09-4698-849b-572011045993 | TS01_ROOT-110_20220418054855 | BackedUp | NULL | 2022-04-18 05:48:55 | NULL | NULL |
>>> | 146 | 875d0298-6ed7-43fc-a468-da6042388ac0 | TS01_ROOT-110_20220419054747 | BackedUp | NULL | 2022-04-19 05:47:47 | NULL | NULL |
>>> | 154 | 0122c9b3-a9f8-473f-a8d7-6e69ba77265b | TS01_ROOT-110_20220420054748 | BackedUp | NULL | 2022-04-20 05:47:48 | NULL | NULL |
>>> | 157 | fe2248c6-aa80-4e9f-b539-7f495f1ff39f | Manual - Apr 20 | BackedUp | NULL | 2022-04-21 01:50:16 | NULL | NULL |
>>> | 163 | 32923ad7-cbe3-4708-95d8-23eca921d711 | TS01_ROOT-110_20220421054817 | BackedUp | NULL | 2022-04-21 05:48:17 | NULL | NULL |
>>> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------+---------------+
>>>
>>>
>>>
>>>>
>>>> 3. SELECT volume_id, store_id, store_role, install_path, size, state FROM cloud.snapshot_store_ref WHERE snapshot_id = <snapshot_id_from_step_2>
>>>>
>>>
>>> mysql> SELECT volume_id, store_id, store_role, install_path, size, state FROM cloud.snapshot_store_ref WHERE snapshot_id = '101';
>>> +-----------+----------+------------+-------------------------------------------------------+-------------+------------+
>>> | volume_id | store_id | store_role | install_path | size | state |
>>> +-----------+----------+------------+-------------------------------------------------------+-------------+------------+
>>> | 138 | 1 | Image | snapshots/16/111/a6b1bead-ca9a-464b-82c1-d631611ad790 | 85899345920 | Destroying |
>>> +-----------+----------+------------+-------------------------------------------------------+-------------+------------+
>>> 1 row in set (0.00 sec)
>>>
>>> mysql> SELECT volume_id, store_id, store_role, install_path, size, state FROM cloud.snapshot_store_ref WHERE snapshot_id = '109';
>>> +-----------+----------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
>>> | volume_id | store_id | store_role | install_path | size | state |
>>> +-----------+----------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
>>> | 138 | 5 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/66210b31-7445-4586-89ea-52a37f5b1ab5 | 85899345920 | Ready |
>>> | 138 | 1 | Image | snapshots/16/111/66210b31-7445-4586-89ea-52a37f5b1ab5 | 85899345920 | Ready |
>>> +-----------+----------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
>>> 2 rows in set (0.00 sec)
>>>
>>> Regards,
>>> Antoine
>>>
>>>
>>>> Regards,
>>>> Suresh
>>>>
>>>> On 21/04/22, 8:45 AM, "Antoine Boucher" <antoineb@haltondc.com <ma...@haltondc.com>> wrote:
>>>>
>>>> Unfortunately I spoke too fast. The list is back but listing the snapshots of the volume with issue still show a blank list with error “undefined”.
>>>>
>>>>
>>>>
>>>>
>>>> On Apr 20, 2022, at 23:11, Antoine Boucher <antoineb@haltondc.com <ma...@haltondc.com>> wrote:
>>>>
>>>> I was able to recover from the situation by forcing a snapshot on the volume with issue.
>>>>
>>>> On Apr 20, 2022, at 22:00, Antoine Boucher <antoineb@haltondc.com <ma...@haltondc.com>> wrote:
>>>>
>>>> I pin-pointed the volume with the issue by going through all volumes and clicking the “view snapshot"
>>>>
>>>>
>>>>
>>>>> On Apr 20, 2022, at 9:35 PM, Antoine Boucher <antoineb@haltondc.com <ma...@haltondc.com>> wrote:
>>>>>
>>>>> I’m no longer able to see my list of snapshots (storage > snapshots), instead I see a small popup window with the word “undefined" on an empty list snapshot list.
>>>>>
>>>>> Has anyone seen this issue? I restart and rebooted the Management server but no resolution.
>>>>>
>>>>> Here are my logs:
>>>>>
>>>>> 2022-04-20 21:17:02,428 DEBUG [c.c.a.ApiServlet] (qtp515715487-292:ctx-a220f915) (logid:624389ae) ===START=== 10.101.254.1 -- GET listall=true&page=1&pagesize=20&command=listSnapshots&response=json
>>>>> 2022-04-20 21:17:02,439 DEBUG [c.c.a.ApiServer] (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) CIDRs from which account 'Acct[2cee75f9-8bc4-11ec-9c43-001e67fd4838-admin] -- Account {"id": 2, "name": "admin", "uuid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838"}' is allowed to perform API calls: 0.0.0.0/0,::/0
>>>>> 2022-04-20 21:17:02,675 ERROR [c.c.a.ApiServer] (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) unhandled exception executing api command: [Ljava.lang.String;@42fa8f7d
>>>>> java.lang.NullPointerException
>>>>> at org.apache.cloudstack.storage.snapshot.StorageSystemSnapshotStrategy.canHandle(StorageSystemSnapshotStrategy.java:985)
>>>>> at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:72)
>>>>> at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:69)
>>>>> at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.bestMatch(StorageStrategyFactoryImpl.java:95)
>>>>> at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.getSnapshotStrategy(StorageStrategyFactoryImpl.java:69)
>>>>> at org.apache.cloudstack.storage.snapshot.SnapshotObject.isRevertable(SnapshotObject.java:156)
>>>>> at com.cloud.api.ApiResponseHelper.createSnapshotResponse(ApiResponseHelper.java:591)
>>>>> at org.apache.cloudstack.api.command.user.snapshot.ListSnapshotsCmd.execute(ListSnapshotsCmd.java:117)
>>>>> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
>>>>> at com.cloud.api.ApiServer.queueCommand(ApiServer.java:772)
>>>>> at com.cloud.api.ApiServer.handleRequest(ApiServer.java:596)
>>>>> at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:321)
>>>>> at com.cloud.api.ApiServlet$1.run(ApiServlet.java:134)
>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>>>> at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:131)
>>>>> at com.cloud.api.ApiServlet.doGet(ApiServlet.java:93)
>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
>>>>> at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
>>>>> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
>>>>> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
>>>>> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>>>>> at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
>>>>> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
>>>>> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
>>>>> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
>>>>> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
>>>>> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
>>>>> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
>>>>> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
>>>>> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
>>>>> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
>>>>> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
>>>>> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>>>>> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
>>>>> at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
>>>>> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
>>>>> at org.eclipse.jetty.server.Server.handle(Server.java:516)
>>>>> at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400)
>>>>> at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645)
>>>>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392)
>>>>> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
>>>>> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
>>>>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
>>>>> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
>>>>> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
>>>>> at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
>>>>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
>>>>> at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
>>>>> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
>>>>> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
>>>>> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
>>>>> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
>>>>> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
>>>>> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
>>>>> at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
>>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>>> 2022-04-20 21:17:02,677 DEBUG [c.c.a.ApiServlet] (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) ===END=== 10.101.254.1 -- GET listall=true&page=1&pagesize=20&command=listSnapshots&response=json
>>>>>
>>>>> Antoine Boucher
>>>>>
>>>>>
>>>>> Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential, and may be privileged. If you are not the intended recipient, you are hereby notified that any review, retransmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, and delete this message and any attachments from your system.
>>>>>
Re: NPE while listing Snapshots
Posted by Suresh Anaparti <Su...@shapeblue.com>.
Hi Antoine,
What is the error in the management server logs while deleting the ‘BackedUp’ snapshots? Can you share the log if possible.
Are these snapshots backed up to secondary storage as well (and what’s the config ‘snapshot.backup.to.secondary’ value when taking snapshot )? If yes, can you find the snapshot files at the install_path there? You can check the install_path details in cloud.snapshot_store_ref table with volume_id: 155 and store_role: Image.
Regards,
Suresh
From: Antoine Boucher <an...@haltondc.com>
Reply to: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>
Date: Wednesday, 11 May 2022 at 9:15 PM
To: users <us...@cloudstack.apache.org>
Subject: Re: NPE while listing Snapshots
Too fast id=146 also failed
{
"accountid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838",
"cmd": "org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd",
"completed": "2022-05-11T11:43:01-0400",
"created": "2022-05-11T11:38:00-0400",
"jobid": "0bd30e5c-2e61-4988-8152-36835b791769",
"jobinstanceid": "875d0298-6ed7-43fc-a468-da6042388ac0",
"jobinstancetype": "Snapshot",
"jobprocstatus": 0,
"jobresult": {
"errorcode": 530,
"errortext": "Failed to delete snapshot:com.cloud.utils.exception.CloudRuntimeException: Failed to remove snapshot "
},
"jobresultcode": 530,
"jobresulttype": "object",
"jobstatus": 2,
"userid": "2cefb708-8bc4-11ec-9c43-001e67fd4838"
}
🙈 Error: async API failed for job 0bd30e5c-2e61-4988-8152-36835b791769
Antoine Boucher
AntoineB@haltondc.com<ma...@haltondc.com>
[o] +1-226-505-9734
www.haltondc.com<http://www.haltondc.com>
“Data security made simple and affordable”
[cid:image001.png@01D866F9.FD673940]
Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential, and may be privileged. If you are not the intended recipient, you are hereby notified that any review, retransmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, and delete this message and any attachments from your system.
On May 11, 2022, at 11:41 AM, Antoine Boucher <an...@haltondc.com>> wrote:
Hello Suresh,
That primary storage was part of the local storage of a host that was removed.
SELECT name, uuid, pool_type, scope, created, removed FROM cloud.storage_pool WHERE id = 5
+----------------+------+------------+-------+---------------------+---------------------+
| name | uuid | pool_type | scope | created | removed |
+----------------+------+------------+-------+---------------------+---------------------+
| cs-kvm03-local | NULL | Filesystem | HOST | 2022-02-12 15:01:45 | 2022-04-19 02:11:51 |
+----------------+------+------------+-------+---------------------+——————————+
SELECT volume_id, store_id, snapshot_id, store_role, install_path, size, state FROM cloud.snapshot_store_ref WHERE store_id = '5';
+-----------+----------+-------------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
| volume_id | store_id | snapshot_id | store_role | install_path | size | state |
+-----------+----------+-------------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
| 155 | 5 | 109 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/66210b31-7445-4586-89ea-52a37f5b1ab5 | 85899345920 | Ready |
| 155 | 5 | 117 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/a2c4fbbb-12b2-4c43-9647-375f6e334a46 | 85899345920 | Ready |
| 155 | 5 | 127 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/794d6dff-ba45-4a56-8377-1e48034bf014 | 85899345920 | Ready |
| 155 | 5 | 138 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/38648d51-bb19-49a2-b01f-12c4e6ed4529 | 85899345920 | Ready |
+-----------+----------+-------------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
All 155 snapshots have the same account_id and domain_id.
SELECT id, uuid, name, status, path, created, removed, location_type FROM cloud.snapshots WHERE volume_id = '155';
+-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------------------+---------------+
| id | uuid | name | status | path | created | removed | location_type |
+-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------------------+---------------+
| 101 | 13870bbd-d7d7-4664-a22f-f09c80a56d41 | TS01_ROOT-110_20220414054855 | Destroyed | NULL | 2022-04-14 05:48:55 | NULL | NULL |
| 109 | 77d84bf6-f601-44a6-a239-821f735a7f48 | TS01_ROOT-110_20220415054855 | BackedUp | NULL | 2022-04-15 05:48:55 | NULL | NULL |
| 117 | 7db02f43-a9d8-4c73-905a-1524d90f570a | TS01_ROOT-110_20220416054855 | BackedUp | NULL | 2022-04-16 05:48:55 | NULL | NULL |
| 127 | 47288ab9-2da2-40f2-98e3-9ced3f8e87df | TS01_ROOT-110_20220417054855 | BackedUp | NULL | 2022-04-17 05:48:55 | NULL | NULL |
| 138 | e9639f4a-fa09-4698-849b-572011045993 | TS01_ROOT-110_20220418054855 | BackedUp | NULL | 2022-04-18 05:48:55 | NULL | NULL |
| 146 | 875d0298-6ed7-43fc-a468-da6042388ac0 | TS01_ROOT-110_20220419054747 | BackedUp | NULL | 2022-04-19 05:47:47 | NULL | NULL |
| 154 | 0122c9b3-a9f8-473f-a8d7-6e69ba77265b | TS01_ROOT-110_20220420054748 | Destroyed | NULL | 2022-04-20 05:47:48 | NULL | NULL |
| 157 | fe2248c6-aa80-4e9f-b539-7f495f1ff39f | Manual - Apr 20 | Destroyed | NULL | 2022-04-21 01:50:16 | NULL | NULL |
| 163 | 32923ad7-cbe3-4708-95d8-23eca921d711 | TS01_ROOT-110_20220421054817 | Destroyed | NULL | 2022-04-21 05:48:17 | NULL | NULL |
| 172 | 6e03ea90-e8b0-4405-aed6-7bd17a40de57 | TS01_ROOT-110_20220422054933 | Destroyed | NULL | 2022-04-22 05:49:33 | NULL | NULL |
| 181 | 9059a26a-0e4e-4d46-95f2-b0a9236ba90a | TS01_ROOT-110_20220423054933 | Destroyed | NULL | 2022-04-23 05:49:33 | NULL | NULL |
| 193 | c9828cbc-f282-4dfe-b4cf-9df792c522c2 | TS01_ROOT-110_20220424054934 | BackedUp | NULL | 2022-04-24 05:49:34 | NULL | NULL |
| 207 | 2e5dfd48-c4de-45ff-bbe1-4cf0d90bb698 | TS01_ROOT-110_20220425054934 | Destroyed | NULL | 2022-04-25 05:49:34 | NULL | NULL |
| 217 | 99d9d9ff-44ae-41a9-9761-f722b370c01c | TS01_ROOT-110_20220426054934 | Destroyed | NULL | 2022-04-26 05:49:34 | NULL | NULL |
| 227 | 1df14a7b-ba56-4d26-aeb4-c79d4de978fc | TS01_ROOT-110_20220427054934 | BackedUp | NULL | 2022-04-27 05:49:34 | NULL | NULL |
| 237 | 657479e0-3587-4a72-a8c8-bd7dff54a5e4 | TS01_ROOT-110_20220428054934 | BackedUp | NULL | 2022-04-28 05:49:34 | NULL | NULL |
| 248 | 2ef0b539-be01-4b0c-9fa5-5fc9c60a6a96 | TS01_ROOT-110_20220429054935 | BackedUp | NULL | 2022-04-29 05:49:35 | NULL | NULL |
| 257 | 8ab75270-8e3d-471e-8e4a-8d5f3cbc6bc1 | TS01_ROOT-110_20220430054935 | BackedUp | NULL | 2022-04-30 05:49:35 | NULL | NULL |
| 259 | a1e77bec-8fc7-457f-b0a5-961c339487f7 | TS01_ROOT-110_20220430060935 | Destroyed | NULL | 2022-04-30 06:09:35 | 2022-05-09 20:02:38 | NULL |
| 313 | 8ad0c04a-1d74-4d51-8223-a6aa3a82d62c | TS01_ROOT-110_20220509015742 | BackedUp | NULL | 2022-05-09 01:57:42 | NULL | NULL |
| 327 | 0f5e9366-747f-4a34-bec4-1fd31c461ac3 | TS01_ROOT-110_20220509170242 | BackedUp | NULL | 2022-05-09 17:02:42 | NULL | NULL |
+-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------------------+---------------+
How would you suggest I resolve the issue moving forward?
I tried the following with error…
(nimbus) 🐱 > delete snapshot id=109
{
"accountid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838",
"cmd": "org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd",
"completed": "2022-05-11T11:32:08-0400",
"created": "2022-05-11T11:32:08-0400",
"jobid": "29661b2b-46c5-42bd-9974-d1a226f3fe68",
"jobinstanceid": "77d84bf6-f601-44a6-a239-821f735a7f48",
"jobinstancetype": "Snapshot",
"jobprocstatus": 0,
"jobresult": {
"errorcode": 530,
"errortext": "Command failed due to Internal Server Error"
},
"jobresultcode": 530,
"jobresulttype": "object",
"jobstatus": 2,
"userid": "2cefb708-8bc4-11ec-9c43-001e67fd4838"
}
🙈 Error: async API failed for job 29661b2b-46c5-42bd-9974-d1a226f3fe68
Deleting some table rows in the right sequence?
Same issue with id=117 to id=138, but I was able to delete id=146
Regards,
Antoine
On Apr 27, 2022, at 9:00 AM, Suresh Anaparti <Su...@shapeblue.com>> wrote:
Hi Antoine,
The NPE error seems to be due to the primary storage with id '5', when checking for snapshot revertible or not during list snapshots cmd. Is that primary storage removed? Can you check with sql below.
SELECT name, uuid, pool_type, scope, created, removed FROM cloud.storage_pool WHERE id = 5
I think, the account/user might not have the permissions to list all snapshots. You can check the account / domain details in snapshots table.
Regards,
Suresh
On 21/04/22, 6:50 PM, "Antoine Boucher" <an...@haltondc.com>> wrote:
Thank you Suresh, you will find my answers below,
On Apr 21, 2022, at 1:02 AM, Suresh Anaparti <Su...@shapeblue.com>> wrote:
Hi Antoine,
What is the CloudStack version, where you see this issue.
Latest - 4.16.1
When you noticed this issue, is it while listing snapshots for a particular volume, or all volumes? Try list snapshots from API/cmk as well.
This is a customer VM, the account may have hit its snapshot limit during a snapshot.
cmk list snapshots > only lists 26 snapshots out of 62
cmk list snapshots volumeid= > Works for some volume ids, provides null result while the WebUI lists the correct snapshots for others and provides null results while show undefined error for one VM mentioned below
Also, Check the snapshots details for the volume (with issue) in the db, using the following sql queries.
1. SELECT id, uuid, state, pool_id, path, size, removed FROM cloud.volumes WHERE name LIKE '<volume_name>'
mysql> SELECT id, uuid, state, pool_id, path, size, removed FROM cloud.volumes WHERE name LIKE 'ROOT-110';
+-----+--------------------------------------+----------+---------+--------------------------------------+-------------+---------------------+
| id | uuid | state | pool_id | path | size | removed |
+-----+--------------------------------------+----------+---------+--------------------------------------+-------------+---------------------+
| 111 | NULL | Expunged | 5 | da585fd4-b10f-480b-9287-45ecb28b6355 | 85899345920 | 2022-04-19 01:29:08 |
| 138 | da585fd4-b10f-480b-9287-45ecb28b6355 | Ready | 6 | 04bd5d13-72ca-4240-93ce-ed1b015d3437 | 85899345920 | NULL |
+-----+--------------------------------------+----------+---------+--------------------------------------+-------------+---------------------+
2. SELECT id, uuid, name, status, path, created, removed, location_type FROM cloud.snapshots WHERE volume_id = <volume_id_from_step_1>;
mysql> SELECT id, uuid, name, status, path, created, removed, location_type FROM cloud.snapshots WHERE volume_id = '138';
+-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------+---------------+
| id | uuid | name | status | path | created | removed | location_type |
+-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------+---------------+
| 101 | 13870bbd-d7d7-4664-a22f-f09c80a56d41 | TS01_ROOT-110_20220414054855 | Destroyed | NULL | 2022-04-14 05:48:55 | NULL | NULL |
| 109 | 77d84bf6-f601-44a6-a239-821f735a7f48 | TS01_ROOT-110_20220415054855 | BackedUp | NULL | 2022-04-15 05:48:55 | NULL | NULL |
| 117 | 7db02f43-a9d8-4c73-905a-1524d90f570a | TS01_ROOT-110_20220416054855 | BackedUp | NULL | 2022-04-16 05:48:55 | NULL | NULL |
| 127 | 47288ab9-2da2-40f2-98e3-9ced3f8e87df | TS01_ROOT-110_20220417054855 | BackedUp | NULL | 2022-04-17 05:48:55 | NULL | NULL |
| 138 | e9639f4a-fa09-4698-849b-572011045993 | TS01_ROOT-110_20220418054855 | BackedUp | NULL | 2022-04-18 05:48:55 | NULL | NULL |
| 146 | 875d0298-6ed7-43fc-a468-da6042388ac0 | TS01_ROOT-110_20220419054747 | BackedUp | NULL | 2022-04-19 05:47:47 | NULL | NULL |
| 154 | 0122c9b3-a9f8-473f-a8d7-6e69ba77265b | TS01_ROOT-110_20220420054748 | BackedUp | NULL | 2022-04-20 05:47:48 | NULL | NULL |
| 157 | fe2248c6-aa80-4e9f-b539-7f495f1ff39f | Manual - Apr 20 | BackedUp | NULL | 2022-04-21 01:50:16 | NULL | NULL |
| 163 | 32923ad7-cbe3-4708-95d8-23eca921d711 | TS01_ROOT-110_20220421054817 | BackedUp | NULL | 2022-04-21 05:48:17 | NULL | NULL |
+-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------+---------------+
3. SELECT volume_id, store_id, store_role, install_path, size, state FROM cloud.snapshot_store_ref WHERE snapshot_id = <snapshot_id_from_step_2>
mysql> SELECT volume_id, store_id, store_role, install_path, size, state FROM cloud.snapshot_store_ref WHERE snapshot_id = '101';
+-----------+----------+------------+-------------------------------------------------------+-------------+------------+
| volume_id | store_id | store_role | install_path | size | state |
+-----------+----------+------------+-------------------------------------------------------+-------------+------------+
| 138 | 1 | Image | snapshots/16/111/a6b1bead-ca9a-464b-82c1-d631611ad790 | 85899345920 | Destroying |
+-----------+----------+------------+-------------------------------------------------------+-------------+------------+
1 row in set (0.00 sec)
mysql> SELECT volume_id, store_id, store_role, install_path, size, state FROM cloud.snapshot_store_ref WHERE snapshot_id = '109';
+-----------+----------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
| volume_id | store_id | store_role | install_path | size | state |
+-----------+----------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
| 138 | 5 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/66210b31-7445-4586-89ea-52a37f5b1ab5 | 85899345920 | Ready |
| 138 | 1 | Image | snapshots/16/111/66210b31-7445-4586-89ea-52a37f5b1ab5 | 85899345920 | Ready |
+-----------+----------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
2 rows in set (0.00 sec)
Regards,
Antoine
Regards,
Suresh
On 21/04/22, 8:45 AM, "Antoine Boucher" <an...@haltondc.com>> wrote:
Unfortunately I spoke too fast. The list is back but listing the snapshots of the volume with issue still show a blank list with error “undefined”.
On Apr 20, 2022, at 23:11, Antoine Boucher <an...@haltondc.com>> wrote:
I was able to recover from the situation by forcing a snapshot on the volume with issue.
On Apr 20, 2022, at 22:00, Antoine Boucher <an...@haltondc.com>> wrote:
I pin-pointed the volume with the issue by going through all volumes and clicking the “view snapshot"
On Apr 20, 2022, at 9:35 PM, Antoine Boucher <an...@haltondc.com>> wrote:
I’m no longer able to see my list of snapshots (storage > snapshots), instead I see a small popup window with the word “undefined" on an empty list snapshot list.
Has anyone seen this issue? I restart and rebooted the Management server but no resolution.
Here are my logs:
2022-04-20 21:17:02,428 DEBUG [c.c.a.ApiServlet] (qtp515715487-292:ctx-a220f915) (logid:624389ae) ===START=== 10.101.254.1 -- GET listall=true&page=1&pagesize=20&command=listSnapshots&response=json
2022-04-20 21:17:02,439 DEBUG [c.c.a.ApiServer] (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) CIDRs from which account 'Acct[2cee75f9-8bc4-11ec-9c43-001e67fd4838-admin] -- Account {"id": 2, "name": "admin", "uuid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838"}' is allowed to perform API calls: 0.0.0.0/0,::/0
2022-04-20 21:17:02,675 ERROR [c.c.a.ApiServer] (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) unhandled exception executing api command: [Ljava.lang.String;@42fa8f7d
java.lang.NullPointerException
at org.apache.cloudstack.storage.snapshot.StorageSystemSnapshotStrategy.canHandle(StorageSystemSnapshotStrategy.java:985)
at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:72)
at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:69)
at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.bestMatch(StorageStrategyFactoryImpl.java:95)
at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.getSnapshotStrategy(StorageStrategyFactoryImpl.java:69)
at org.apache.cloudstack.storage.snapshot.SnapshotObject.isRevertable(SnapshotObject.java:156)
at com.cloud.api.ApiResponseHelper.createSnapshotResponse(ApiResponseHelper.java:591)
at org.apache.cloudstack.api.command.user.snapshot.ListSnapshotsCmd.execute(ListSnapshotsCmd.java:117)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:772)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:596)
at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:321)
at com.cloud.api.ApiServlet$1.run(ApiServlet.java:134)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:131)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:93)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-04-20 21:17:02,677 DEBUG [c.c.a.ApiServlet] (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) ===END=== 10.101.254.1 -- GET listall=true&page=1&pagesize=20&command=listSnapshots&response=json
Antoine Boucher
Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential, and may be privileged. If you are not the intended recipient, you are hereby notified that any review, retransmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, and delete this message and any attachments from your system.
Re: NPE while listing Snapshots
Posted by Antoine Boucher <an...@haltondc.com>.
Too fast id=146 also failed
{
"accountid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838",
"cmd": "org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd",
"completed": "2022-05-11T11:43:01-0400",
"created": "2022-05-11T11:38:00-0400",
"jobid": "0bd30e5c-2e61-4988-8152-36835b791769",
"jobinstanceid": "875d0298-6ed7-43fc-a468-da6042388ac0",
"jobinstancetype": "Snapshot",
"jobprocstatus": 0,
"jobresult": {
"errorcode": 530,
"errortext": "Failed to delete snapshot:com.cloud.utils.exception.CloudRuntimeException: Failed to remove snapshot "
},
"jobresultcode": 530,
"jobresulttype": "object",
"jobstatus": 2,
"userid": "2cefb708-8bc4-11ec-9c43-001e67fd4838"
}
🙈 Error: async API failed for job 0bd30e5c-2e61-4988-8152-36835b791769
Antoine Boucher
AntoineB@haltondc.com
[o] +1-226-505-9734
www.haltondc.com
“Data security made simple and affordable”
Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential, and may be privileged. If you are not the intended recipient, you are hereby notified that any review, retransmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, and delete this message and any attachments from your system.
> On May 11, 2022, at 11:41 AM, Antoine Boucher <an...@haltondc.com> wrote:
>
> Hello Suresh,
>
> That primary storage was part of the local storage of a host that was removed.
>
> SELECT name, uuid, pool_type, scope, created, removed FROM cloud.storage_pool WHERE id = 5
> +----------------+------+------------+-------+---------------------+---------------------+
> | name | uuid | pool_type | scope | created | removed |
> +----------------+------+------------+-------+---------------------+---------------------+
> | cs-kvm03-local | NULL | Filesystem | HOST | 2022-02-12 15:01:45 | 2022-04-19 02:11:51 |
> +----------------+------+------------+-------+---------------------+——————————+
>
>
> SELECT volume_id, store_id, snapshot_id, store_role, install_path, size, state FROM cloud.snapshot_store_ref WHERE store_id = '5';
> +-----------+----------+-------------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
> | volume_id | store_id | snapshot_id | store_role | install_path | size | state |
> +-----------+----------+-------------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
> | 155 | 5 | 109 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/66210b31-7445-4586-89ea-52a37f5b1ab5 | 85899345920 | Ready |
> | 155 | 5 | 117 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/a2c4fbbb-12b2-4c43-9647-375f6e334a46 | 85899345920 | Ready |
> | 155 | 5 | 127 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/794d6dff-ba45-4a56-8377-1e48034bf014 | 85899345920 | Ready |
> | 155 | 5 | 138 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/38648d51-bb19-49a2-b01f-12c4e6ed4529 | 85899345920 | Ready |
> +-----------+----------+-------------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
>
>
> All 155 snapshots have the same account_id and domain_id.
>
>
> SELECT id, uuid, name, status, path, created, removed, location_type FROM cloud.snapshots WHERE volume_id = '155';
> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------------------+---------------+
> | id | uuid | name | status | path | created | removed | location_type |
> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------------------+---------------+
> | 101 | 13870bbd-d7d7-4664-a22f-f09c80a56d41 | TS01_ROOT-110_20220414054855 | Destroyed | NULL | 2022-04-14 05:48:55 | NULL | NULL |
> | 109 | 77d84bf6-f601-44a6-a239-821f735a7f48 | TS01_ROOT-110_20220415054855 | BackedUp | NULL | 2022-04-15 05:48:55 | NULL | NULL |
> | 117 | 7db02f43-a9d8-4c73-905a-1524d90f570a | TS01_ROOT-110_20220416054855 | BackedUp | NULL | 2022-04-16 05:48:55 | NULL | NULL |
> | 127 | 47288ab9-2da2-40f2-98e3-9ced3f8e87df | TS01_ROOT-110_20220417054855 | BackedUp | NULL | 2022-04-17 05:48:55 | NULL | NULL |
> | 138 | e9639f4a-fa09-4698-849b-572011045993 | TS01_ROOT-110_20220418054855 | BackedUp | NULL | 2022-04-18 05:48:55 | NULL | NULL |
> | 146 | 875d0298-6ed7-43fc-a468-da6042388ac0 | TS01_ROOT-110_20220419054747 | BackedUp | NULL | 2022-04-19 05:47:47 | NULL | NULL |
> | 154 | 0122c9b3-a9f8-473f-a8d7-6e69ba77265b | TS01_ROOT-110_20220420054748 | Destroyed | NULL | 2022-04-20 05:47:48 | NULL | NULL |
> | 157 | fe2248c6-aa80-4e9f-b539-7f495f1ff39f | Manual - Apr 20 | Destroyed | NULL | 2022-04-21 01:50:16 | NULL | NULL |
> | 163 | 32923ad7-cbe3-4708-95d8-23eca921d711 | TS01_ROOT-110_20220421054817 | Destroyed | NULL | 2022-04-21 05:48:17 | NULL | NULL |
> | 172 | 6e03ea90-e8b0-4405-aed6-7bd17a40de57 | TS01_ROOT-110_20220422054933 | Destroyed | NULL | 2022-04-22 05:49:33 | NULL | NULL |
> | 181 | 9059a26a-0e4e-4d46-95f2-b0a9236ba90a | TS01_ROOT-110_20220423054933 | Destroyed | NULL | 2022-04-23 05:49:33 | NULL | NULL |
> | 193 | c9828cbc-f282-4dfe-b4cf-9df792c522c2 | TS01_ROOT-110_20220424054934 | BackedUp | NULL | 2022-04-24 05:49:34 | NULL | NULL |
> | 207 | 2e5dfd48-c4de-45ff-bbe1-4cf0d90bb698 | TS01_ROOT-110_20220425054934 | Destroyed | NULL | 2022-04-25 05:49:34 | NULL | NULL |
> | 217 | 99d9d9ff-44ae-41a9-9761-f722b370c01c | TS01_ROOT-110_20220426054934 | Destroyed | NULL | 2022-04-26 05:49:34 | NULL | NULL |
> | 227 | 1df14a7b-ba56-4d26-aeb4-c79d4de978fc | TS01_ROOT-110_20220427054934 | BackedUp | NULL | 2022-04-27 05:49:34 | NULL | NULL |
> | 237 | 657479e0-3587-4a72-a8c8-bd7dff54a5e4 | TS01_ROOT-110_20220428054934 | BackedUp | NULL | 2022-04-28 05:49:34 | NULL | NULL |
> | 248 | 2ef0b539-be01-4b0c-9fa5-5fc9c60a6a96 | TS01_ROOT-110_20220429054935 | BackedUp | NULL | 2022-04-29 05:49:35 | NULL | NULL |
> | 257 | 8ab75270-8e3d-471e-8e4a-8d5f3cbc6bc1 | TS01_ROOT-110_20220430054935 | BackedUp | NULL | 2022-04-30 05:49:35 | NULL | NULL |
> | 259 | a1e77bec-8fc7-457f-b0a5-961c339487f7 | TS01_ROOT-110_20220430060935 | Destroyed | NULL | 2022-04-30 06:09:35 | 2022-05-09 20:02:38 | NULL |
> | 313 | 8ad0c04a-1d74-4d51-8223-a6aa3a82d62c | TS01_ROOT-110_20220509015742 | BackedUp | NULL | 2022-05-09 01:57:42 | NULL | NULL |
> | 327 | 0f5e9366-747f-4a34-bec4-1fd31c461ac3 | TS01_ROOT-110_20220509170242 | BackedUp | NULL | 2022-05-09 17:02:42 | NULL | NULL |
> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------------------+---------------+
>
> How would you suggest I resolve the issue moving forward?
>
> I tried the following with error…
>
> (nimbus) 🐱 > delete snapshot id=109
> {
> "accountid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838",
> "cmd": "org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd",
> "completed": "2022-05-11T11:32:08-0400",
> "created": "2022-05-11T11:32:08-0400",
> "jobid": "29661b2b-46c5-42bd-9974-d1a226f3fe68",
> "jobinstanceid": "77d84bf6-f601-44a6-a239-821f735a7f48",
> "jobinstancetype": "Snapshot",
> "jobprocstatus": 0,
> "jobresult": {
> "errorcode": 530,
> "errortext": "Command failed due to Internal Server Error"
> },
> "jobresultcode": 530,
> "jobresulttype": "object",
> "jobstatus": 2,
> "userid": "2cefb708-8bc4-11ec-9c43-001e67fd4838"
> }
> 🙈 Error: async API failed for job 29661b2b-46c5-42bd-9974-d1a226f3fe68
>
> Deleting some table rows in the right sequence?
>
> Same issue with id=117 to id=138, but I was able to delete id=146
>
> Regards,
> Antoine
>
>
>
>> On Apr 27, 2022, at 9:00 AM, Suresh Anaparti <Suresh.Anaparti@shapeblue.com <ma...@shapeblue.com>> wrote:
>>
>> Hi Antoine,
>>
>> The NPE error seems to be due to the primary storage with id '5', when checking for snapshot revertible or not during list snapshots cmd. Is that primary storage removed? Can you check with sql below.
>>
>> SELECT name, uuid, pool_type, scope, created, removed FROM cloud.storage_pool WHERE id = 5
>>
>>
>> I think, the account/user might not have the permissions to list all snapshots. You can check the account / domain details in snapshots table.
>>
>>
>> Regards,
>> Suresh
>>
>> On 21/04/22, 6:50 PM, "Antoine Boucher" <antoineb@haltondc.com <ma...@haltondc.com>> wrote:
>>
>> Thank you Suresh, you will find my answers below,
>>
>>
>>
>>
>>
>>> On Apr 21, 2022, at 1:02 AM, Suresh Anaparti <Suresh.Anaparti@shapeblue.com <ma...@shapeblue.com>> wrote:
>>>
>>> Hi Antoine,
>>>
>>> What is the CloudStack version, where you see this issue.
>>
>> Latest - 4.16.1
>>
>>
>>> When you noticed this issue, is it while listing snapshots for a particular volume, or all volumes? Try list snapshots from API/cmk as well.
>>
>> This is a customer VM, the account may have hit its snapshot limit during a snapshot.
>>
>> cmk list snapshots > only lists 26 snapshots out of 62
>>
>> cmk list snapshots volumeid= > Works for some volume ids, provides null result while the WebUI lists the correct snapshots for others and provides null results while show undefined error for one VM mentioned below
>>
>>
>>>
>>> Also, Check the snapshots details for the volume (with issue) in the db, using the following sql queries.
>>>
>>> 1. SELECT id, uuid, state, pool_id, path, size, removed FROM cloud.volumes WHERE name LIKE '<volume_name>'
>>>
>>
>> mysql> SELECT id, uuid, state, pool_id, path, size, removed FROM cloud.volumes WHERE name LIKE 'ROOT-110';
>> +-----+--------------------------------------+----------+---------+--------------------------------------+-------------+---------------------+
>> | id | uuid | state | pool_id | path | size | removed |
>> +-----+--------------------------------------+----------+---------+--------------------------------------+-------------+---------------------+
>> | 111 | NULL | Expunged | 5 | da585fd4-b10f-480b-9287-45ecb28b6355 | 85899345920 | 2022-04-19 01:29:08 |
>> | 138 | da585fd4-b10f-480b-9287-45ecb28b6355 | Ready | 6 | 04bd5d13-72ca-4240-93ce-ed1b015d3437 | 85899345920 | NULL |
>> +-----+--------------------------------------+----------+---------+--------------------------------------+-------------+---------------------+
>>
>>
>>> 2. SELECT id, uuid, name, status, path, created, removed, location_type FROM cloud.snapshots WHERE volume_id = <volume_id_from_step_1>;
>>
>> mysql> SELECT id, uuid, name, status, path, created, removed, location_type FROM cloud.snapshots WHERE volume_id = '138';
>> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------+---------------+
>> | id | uuid | name | status | path | created | removed | location_type |
>> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------+---------------+
>> | 101 | 13870bbd-d7d7-4664-a22f-f09c80a56d41 | TS01_ROOT-110_20220414054855 | Destroyed | NULL | 2022-04-14 05:48:55 | NULL | NULL |
>> | 109 | 77d84bf6-f601-44a6-a239-821f735a7f48 | TS01_ROOT-110_20220415054855 | BackedUp | NULL | 2022-04-15 05:48:55 | NULL | NULL |
>> | 117 | 7db02f43-a9d8-4c73-905a-1524d90f570a | TS01_ROOT-110_20220416054855 | BackedUp | NULL | 2022-04-16 05:48:55 | NULL | NULL |
>> | 127 | 47288ab9-2da2-40f2-98e3-9ced3f8e87df | TS01_ROOT-110_20220417054855 | BackedUp | NULL | 2022-04-17 05:48:55 | NULL | NULL |
>> | 138 | e9639f4a-fa09-4698-849b-572011045993 | TS01_ROOT-110_20220418054855 | BackedUp | NULL | 2022-04-18 05:48:55 | NULL | NULL |
>> | 146 | 875d0298-6ed7-43fc-a468-da6042388ac0 | TS01_ROOT-110_20220419054747 | BackedUp | NULL | 2022-04-19 05:47:47 | NULL | NULL |
>> | 154 | 0122c9b3-a9f8-473f-a8d7-6e69ba77265b | TS01_ROOT-110_20220420054748 | BackedUp | NULL | 2022-04-20 05:47:48 | NULL | NULL |
>> | 157 | fe2248c6-aa80-4e9f-b539-7f495f1ff39f | Manual - Apr 20 | BackedUp | NULL | 2022-04-21 01:50:16 | NULL | NULL |
>> | 163 | 32923ad7-cbe3-4708-95d8-23eca921d711 | TS01_ROOT-110_20220421054817 | BackedUp | NULL | 2022-04-21 05:48:17 | NULL | NULL |
>> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------+---------------+
>>
>>
>>>
>>> 3. SELECT volume_id, store_id, store_role, install_path, size, state FROM cloud.snapshot_store_ref WHERE snapshot_id = <snapshot_id_from_step_2>
>>>
>>
>> mysql> SELECT volume_id, store_id, store_role, install_path, size, state FROM cloud.snapshot_store_ref WHERE snapshot_id = '101';
>> +-----------+----------+------------+-------------------------------------------------------+-------------+------------+
>> | volume_id | store_id | store_role | install_path | size | state |
>> +-----------+----------+------------+-------------------------------------------------------+-------------+------------+
>> | 138 | 1 | Image | snapshots/16/111/a6b1bead-ca9a-464b-82c1-d631611ad790 | 85899345920 | Destroying |
>> +-----------+----------+------------+-------------------------------------------------------+-------------+------------+
>> 1 row in set (0.00 sec)
>>
>> mysql> SELECT volume_id, store_id, store_role, install_path, size, state FROM cloud.snapshot_store_ref WHERE snapshot_id = '109';
>> +-----------+----------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
>> | volume_id | store_id | store_role | install_path | size | state |
>> +-----------+----------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
>> | 138 | 5 | Primary | /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/66210b31-7445-4586-89ea-52a37f5b1ab5 | 85899345920 | Ready |
>> | 138 | 1 | Image | snapshots/16/111/66210b31-7445-4586-89ea-52a37f5b1ab5 | 85899345920 | Ready |
>> +-----------+----------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
>> 2 rows in set (0.00 sec)
>>
>> Regards,
>> Antoine
>>
>>> Regards,
>>> Suresh
>>>
>>> On 21/04/22, 8:45 AM, "Antoine Boucher" <antoineb@haltondc.com <ma...@haltondc.com>> wrote:
>>>
>>> Unfortunately I spoke too fast. The list is back but listing the snapshots of the volume with issue still show a blank list with error “undefined”.
>>>
>>>
>>>
>>>
>>> On Apr 20, 2022, at 23:11, Antoine Boucher <antoineb@haltondc.com <ma...@haltondc.com>> wrote:
>>>
>>> I was able to recover from the situation by forcing a snapshot on the volume with issue.
>>>
>>> On Apr 20, 2022, at 22:00, Antoine Boucher <antoineb@haltondc.com <ma...@haltondc.com>> wrote:
>>>
>>> I pin-pointed the volume with the issue by going through all volumes and clicking the “view snapshot"
>>>
>>>
>>>> On Apr 20, 2022, at 9:35 PM, Antoine Boucher <antoineb@haltondc.com <ma...@haltondc.com>> wrote:
>>>>
>>>> I’m no longer able to see my list of snapshots (storage > snapshots), instead I see a small popup window with the word “undefined" on an empty list snapshot list.
>>>>
>>>> Has anyone seen this issue? I restart and rebooted the Management server but no resolution.
>>>>
>>>> Here are my logs:
>>>>
>>>> 2022-04-20 21:17:02,428 DEBUG [c.c.a.ApiServlet] (qtp515715487-292:ctx-a220f915) (logid:624389ae) ===START=== 10.101.254.1 -- GET listall=true&page=1&pagesize=20&command=listSnapshots&response=json
>>>> 2022-04-20 21:17:02,439 DEBUG [c.c.a.ApiServer] (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) CIDRs from which account 'Acct[2cee75f9-8bc4-11ec-9c43-001e67fd4838-admin] -- Account {"id": 2, "name": "admin", "uuid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838"}' is allowed to perform API calls: 0.0.0.0/0,::/0
>>>> 2022-04-20 21:17:02,675 ERROR [c.c.a.ApiServer] (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) unhandled exception executing api command: [Ljava.lang.String;@42fa8f7d
>>>> java.lang.NullPointerException
>>>> at org.apache.cloudstack.storage.snapshot.StorageSystemSnapshotStrategy.canHandle(StorageSystemSnapshotStrategy.java:985)
>>>> at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:72)
>>>> at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:69)
>>>> at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.bestMatch(StorageStrategyFactoryImpl.java:95)
>>>> at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.getSnapshotStrategy(StorageStrategyFactoryImpl.java:69)
>>>> at org.apache.cloudstack.storage.snapshot.SnapshotObject.isRevertable(SnapshotObject.java:156)
>>>> at com.cloud.api.ApiResponseHelper.createSnapshotResponse(ApiResponseHelper.java:591)
>>>> at org.apache.cloudstack.api.command.user.snapshot.ListSnapshotsCmd.execute(ListSnapshotsCmd.java:117)
>>>> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
>>>> at com.cloud.api.ApiServer.queueCommand(ApiServer.java:772)
>>>> at com.cloud.api.ApiServer.handleRequest(ApiServer.java:596)
>>>> at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:321)
>>>> at com.cloud.api.ApiServlet$1.run(ApiServlet.java:134)
>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>>> at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:131)
>>>> at com.cloud.api.ApiServlet.doGet(ApiServlet.java:93)
>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
>>>> at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
>>>> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
>>>> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
>>>> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>>>> at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
>>>> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
>>>> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
>>>> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
>>>> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
>>>> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
>>>> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
>>>> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
>>>> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
>>>> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
>>>> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
>>>> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>>>> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
>>>> at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
>>>> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
>>>> at org.eclipse.jetty.server.Server.handle(Server.java:516)
>>>> at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400)
>>>> at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645)
>>>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392)
>>>> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
>>>> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
>>>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
>>>> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
>>>> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
>>>> at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
>>>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
>>>> at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
>>>> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
>>>> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
>>>> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
>>>> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
>>>> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
>>>> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
>>>> at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>> 2022-04-20 21:17:02,677 DEBUG [c.c.a.ApiServlet] (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) ===END=== 10.101.254.1 -- GET listall=true&page=1&pagesize=20&command=listSnapshots&response=json
>>>>
>>>> Antoine Boucher
>>>>
>>>>
>>>> Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential, and may be privileged. If you are not the intended recipient, you are hereby notified that any review, retransmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, and delete this message and any attachments from your system.
>>>>
>>>
>>>
>>
>>
>