You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@storm.apache.org by "Antoine Tran (Jira)" <ji...@apache.org> on 2022/06/03 14:15:00 UTC
[jira] [Created] (STORM-3871) Storm blobstore leak space
Antoine Tran created STORM-3871:
-----------------------------------
Summary: Storm blobstore leak space
Key: STORM-3871
URL: https://issues.apache.org/jira/browse/STORM-3871
Project: Apache Storm
Issue Type: Bug
Components: blobstore
Affects Versions: 2.3.0
Environment: Storm 2.3.0 in Kubernetes 1.21
Storm nimbus have 3 empty dir volumes mapped to /var/lib/kubelet/pods/.../volumes in a 10GB /var limit.
Storm has default blob cleanup strategy.
Reporter: Antoine Tran
We observe after 5 days, with 3 nimbus deployed in HA, that the blostore keeps growing.
Usually we submit topologies each 10min and it ends in around 10 min. The expected result is that the blobstore cleans itself so that topologies older than days are removed. But this is not the case, as seen below (done the 2022-06-03, expected result: only blob from the current day should appear. current behavior: blobs from 2 days ago are still here)
{noformat}
/space/StormApp/current/bin/storm blobstore list 2>&1 | grep o.a.s.c.Blobstore | sort -k7 | perl -anE '$date=localtime($F[6]/1000);shift @F;say join " ", $date, @F'
Wed Jun 1 14:12:43 2022 [main] INFO o.a.s.c.Blobstore - dep-apeframework-3.11.3-8f132aef-b401-4fa6-b07e-3588073da824.jar 1654092763312 [o::r--, o::rwa, u:class org.apache.storm.security.auth.NimbusPrincipal:rwa]
Wed Jun 1 14:13:43 2022 [main] INFO o.a.s.c.Blobstore - dep-apeframework-3.11.3-ae74967a-0def-44a1-8d1b-53a255900239.jar 1654092823330 [o::r--, o::rwa, u:class org.apache.storm.security.auth.NimbusPrincipal:rwa]
Wed Jun 1 14:33:00 2022 [main] INFO o.a.s.c.Blobstore - dep-apeframework-3.11.3-8fa7edcb-5f8a-4a79-8f8d-8c8fd50dbc9e.jar 1654093980305 [o::r--, o::rwa, u:class org.apache.storm.security.auth.NimbusPrincipal:rwa]
Wed Jun 1 15:14:19 2022 [main] INFO o.a.s.c.Blobstore - dep-apeframework-3.11.3-9168e429-2cd6-433c-90d9-6db19d564824.jar 1654096459069 [o::r--, o::rwa, u:class org.apache.storm.security.auth.NimbusPrincipal:rwa]
Wed Jun 1 16:13:49 2022 [main] INFO o.a.s.c.Blobstore - dep-apeframework-3.11.3-9ed7a9fd-2819-4784-a953-c04a65981c59.jar 1654100029294 [o::r--, o::rwa, u:class org.apache.storm.security.auth.NimbusPrincipal:rwa]
Wed Jun 1 22:53:29 2022 [main] INFO o.a.s.c.Blobstore - dep-apeframework-3.11.3-124b606a-0d2b-44a1-9885-4bb1152a98e7.jar 1654124009759 [o::r--, o::rwa, u:class org.apache.storm.security.auth.NimbusPrincipal:rwa]
Wed Jun 1 23:23:59 2022 [main] INFO o.a.s.c.Blobstore - dep-apeframework-3.11.3-be7e85b7-3cf1-495a-83a7-6a28da33af57.jar 1654125839685 [o::r--, o::rwa, u:class org.apache.storm.security.auth.NimbusPrincipal:rwa]
Wed Jun 1 23:33:43 2022 [main] INFO o.a.s.c.Blobstore - dep-apeframework-3.11.3-f799a103-d544-4a31-a680-1d0625fab2b9.jar 1654126423302 [o::r--, o::rwa, u:class org.apache.storm.security.auth.NimbusPrincipal:rwa]
Thu Jun 2 01:13:49 2022 [main] INFO o.a.s.c.Blobstore - dep-apeframework-3.11.3-3edcef03-f7d6-400e-8076-30dc0e5c4bff.jar 1654132429251 [o::r--, o::rwa, u:class org.apache.storm.security.auth.NimbusPrincipal:rwa]
Thu Jun 2 02:03:29 2022 [main] INFO o.a.s.c.Blobstore - dep-apeframework-3.11.3-807000a6-0416-4821-b67d-a1e5e433edce.jar 1654135409776 [o::r--, o::rwa, u:class org.apache.storm.security.auth.NimbusPrincipal:rwa]
Thu Jun 2 02:13:30 2022 [main] INFO o.a.s.c.Blobstore - dep-apeframework-3.11.3-4f7e7439-7788-4b74-81a2-4d6433caed4b.jar 1654136010028 [o::r--, o::rwa, u:class org.apache.storm.security.auth.NimbusPrincipal:rwa]
Thu Jun 2 03:03:59 2022 [main] INFO o.a.s.c.Blobstore - dep-apeframework-3.11.3-038e17f3-7acd-4c19-a212-a095e6c7adba.jar 1654139039870 [o::r--, o::rwa, u:class org.apache.storm.security.auth.NimbusPrincipal:rwa]
Thu Jun 2 07:43:30 2022 [main] INFO o.a.s.c.Blobstore - dep-apeframework-3.11.3-b4fb1761-4f97-4ac7-8785-85e2aefbb66c.jar 1654155810868 [o::r--, o::rwa, u:class org.apache.storm.security.auth.NimbusPrincipal:rwa]
Thu Jun 2 08:33:49 2022 [main] INFO o.a.s.c.Blobstore - dep-apeframework-3.11.3-ac2c4004-d561-44ad-beca-0883ad8980c5.jar 1654158829119 [o::r--, o::rwa, u:class org.apache.storm.security.auth.NimbusPrincipal:rwa]
Thu Jun 2 09:44:00 2022 [main] INFO o.a.s.c.Blobstore - dep-apeframework-3.11.3-31f8b9b4-8f79-47ef-8728-46127baf04a2.jar 1654163040830 [o::r--, o::rwa, u:class org.apache.storm.security.auth.NimbusPrincipal:rwa]
Thu Jun 2 09:53:00 2022 [main] INFO o.a.s.c.Blobstore - dep-apeframework-3.11.3-e373b50f-6282-4eaa-a702-57c8f4d0be55.jar 1654163580973 [o::r--, o::rwa, u:class org.apache.storm.security.auth.NimbusPrincipal:rwa]
Thu Jun 2 10:03:29 2022 [main] INFO o.a.s.c.Blobstore - dep-apeframework-3.11.3-9b0ac8d3-b446-4ce5-89c1-2591cdeb80e6.jar 1654164209828 [o::r--, o::rwa, u:class org.apache.storm.security.auth.NimbusPrincipal:rwa]
Thu Jun 2 11:43:43 2022 [main] INFO o.a.s.c.Blobstore - dep-apeframework-3.11.3-638cbabd-b3ba-46c3-821c-5eb45578c542.jar 1654170223374 [o::r--, o::rwa, u:class org.apache.storm.security.auth.NimbusPrincipal:rwa]
Thu Jun 2 12:13:43 2022 [main] INFO o.a.s.c.Blobstore - dep-apeframework-3.11.3-d8fb8058-d3d6-42eb-9e15-5d3682fcc2c7.jar 1654172023366 [o::r--, o::rwa, u:class org.apache.storm.security.auth.NimbusPrincipal:rwa]
Thu Jun 2 13:02:59 2022 [main] INFO o.a.s.c.Blobstore - dep-apeframework-3.11.3-46e1ff96-da4d-4936-913a-d477d743027a.jar 1654174979704 [o::r--, o::rwa, u:class org.apache.storm.security.auth.NimbusPrincipal:rwa]
Thu Jun 2 16:52:43 2022 [main] INFO o.a.s.c.Blobstore - dep-apeframework-3.11.3-b1b010ee-48d3-4fa5-b6dc-b136a4d54c59.jar 1654188763365 [o::r--, o::rwa, u:class org.apache.storm.security.auth.NimbusPrincipal:rwa]
Thu Jun 2 17:02:59 2022 [main] INFO o.a.s.c.Blobstore - dep-apeframework-3.11.3-d11859ec-7750-434b-a7d1-da5ddd2d68c3.jar 1654189379774 [o::r--, o::rwa, u:class org.apache.storm.security.auth.NimbusPrincipal:rwa]
Thu Jun 2 18:03:00 2022 [main] INFO o.a.s.c.Blobstore - dep-apeframework-3.11.3-22b34179-5dbd-4bf0-aa5d-c21a978fb8c7.jar 1654192980944 [o::r--, o::rwa, u:class org.apache.storm.security.auth.NimbusPrincipal:rwa]
Thu Jun 2 23:43:43 2022 [main] INFO o.a.s.c.Blobstore - dep-apeframework-3.11.3-2b564b9c-1727-404e-9e92-edff6b4799a6.jar 1654213423405 [o::r--, o::rwa, u:class org.apache.storm.security.auth.NimbusPrincipal:rwa]
Fri Jun 3 01:13:43 2022 [main] INFO o.a.s.c.Blobstore - dep-apeframework-3.11.3-c32e9295-abd0-48de-921f-a0d524ca4db2.jar 1654218823466 [o::r--, o::rwa, u:class org.apache.storm.security.auth.NimbusPrincipal:rwa]
Fri Jun 3 05:33:31 2022 [main] INFO o.a.s.c.Blobstore - dep-apeframework-3.11.3-4f88e716-a620-4897-bc80-89e3bfa2282c.jar 1654234411000 [o::r--, o::rwa, u:class org.apache.storm.security.auth.NimbusPrincipal:rwa]
Fri Jun 3 08:43:31 2022 [main] INFO o.a.s.c.Blobstore - dep-apeframework-3.11.3-deb4795a-1983-46ea-a6a3-00457d44d523.jar 1654245811327 [o::r--, o::rwa, u:class org.apache.storm.security.auth.NimbusPrincipal:rwa]
Fri Jun 3 10:23:00 2022 [main] INFO o.a.s.c.Blobstore - dep-apeframework-3.11.3-e887952c-2403-484c-ab53-22ff75a26b9a.jar 1654251780168 [o::r--, o::rwa, u:class org.apache.storm.security.auth.NimbusPrincipal:rwa] {noformat}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)