You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "ivakegg (via GitHub)" <gi...@apache.org> on 2023/05/04 18:05:52 UTC
[GitHub] [accumulo] ivakegg opened a new issue, #3374: Garbage collector page on monitor weirdness
ivakegg opened a new issue, #3374:
URL: https://github.com/apache/accumulo/issues/3374
The garbage collector page dow not seem to be updating despite the logs showing that the collector is obviously running. Also the Duration for the current GC shows as being -54y-135d which I believe is epoch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-ratcliffe commented on issue #3374: Garbage collector page on monitor weirdness
Posted by "keith-ratcliffe (via GitHub)" <gi...@apache.org>.
keith-ratcliffe commented on issue #3374:
URL: https://github.com/apache/accumulo/issues/3374#issuecomment-1551888968
> It'd be helpful to know if the edge case is happening in the REST endpoint or in the web view.
As far as the Duration value for the Current GC, I'm able to confirm that the web view and the rest endpoint agree, i.e., they're reflective of the same negative value, formatted differently of course
> When you say that it doesn't appear to be updating, does it update when you refresh the page? Or not at all?
I am unable to replicate this atm. With auto-refresh enabled, activity updates seem to be appearing with the expected frequency for all the GC types, just that the Duration for Current GC is always the -54y-1XXd value
I'll see if I can get our system patched with your updates from #3403 and then report back
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] DomGarguilo commented on issue #3374: Garbage collector page on monitor weirdness
Posted by "DomGarguilo (via GitHub)" <gi...@apache.org>.
DomGarguilo commented on issue #3374:
URL: https://github.com/apache/accumulo/issues/3374#issuecomment-1538943942
Also was `SimpleGarbageCollector` being run or something else?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on issue #3374: Garbage collector page on monitor weirdness
Posted by "ctubbsii (via GitHub)" <gi...@apache.org>.
ctubbsii commented on issue #3374:
URL: https://github.com/apache/accumulo/issues/3374#issuecomment-1549966823
> ... so it was throwing a java.net.NoRouteToHostException ...
So, you saw this in the GC's logs, then? And the malformed date on the monitor was only occurring when this exception was seen?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] AlbertWhitlock commented on issue #3374: Garbage collector page on monitor weirdness
Posted by "AlbertWhitlock (via GitHub)" <gi...@apache.org>.
AlbertWhitlock commented on issue #3374:
URL: https://github.com/apache/accumulo/issues/3374#issuecomment-1550117327
@ctubbsii So, you saw this in the GC's logs, then? And the malformed date on the monitor was only occurring when this exception was seen?
Correct on both. I saw the exception in the logs and correct, it was only happening when exception was occurring.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on issue #3374: Garbage collector page on monitor weirdness
Posted by "ctubbsii (via GitHub)" <gi...@apache.org>.
ctubbsii commented on issue #3374:
URL: https://github.com/apache/accumulo/issues/3374#issuecomment-1542359125
> > What version? If snapshot, please provide the closest git hash to what you're using.
>
> @ctubbsii It appears that the last external commit we have is [55d972a](https://github.com/apache/accumulo/commit/55d972a4c750c81aba8c59f5841d6369ad40d519)
That is old, but I don't see any updates since then that would have fixed something like this. Is this reproducible?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on issue #3374: Garbage collector page on monitor weirdness
Posted by "ctubbsii (via GitHub)" <gi...@apache.org>.
ctubbsii commented on issue #3374:
URL: https://github.com/apache/accumulo/issues/3374#issuecomment-1550157101
@AlbertWhitlock If you can reliably reproduce the error, can you check to see if it is fixed with #3403 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on issue #3374: Garbage collector page on monitor weirdness
Posted by "ctubbsii (via GitHub)" <gi...@apache.org>.
ctubbsii commented on issue #3374:
URL: https://github.com/apache/accumulo/issues/3374#issuecomment-1548251586
@AlbertWhitlock wrote:
> @ivakegg .... on my instance I also saw the -54y-148d entry. After investigating I found it is because when gc of WAL was kicked off my instance couldn't find all of my volumes. I reconnected to 'new' volume and next gc found the volume and that allowed the value on monitor page to be updated. Perhaps a better 'default' value could be used other than epoch.
What do you mean it couldn't find all your volumes? What did you see that indicated that was related to this?
What do you mean you "reconnected to 'new' volume"?
@ivakegg When you say that it doesn't appear to be updating, does it update when you refresh the page? Or not at all?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on issue #3374: Garbage collector page on monitor weirdness
Posted by "ctubbsii (via GitHub)" <gi...@apache.org>.
ctubbsii commented on issue #3374:
URL: https://github.com/apache/accumulo/issues/3374#issuecomment-1541202556
> Also was `SimpleGarbageCollector` being run or something else?
I don't think our implementation is pluggable. The name is historical, because we changed the implementation internally a long time ago. So, I would assume it's this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on issue #3374: Garbage collector page on monitor weirdness
Posted by "ctubbsii (via GitHub)" <gi...@apache.org>.
ctubbsii commented on issue #3374:
URL: https://github.com/apache/accumulo/issues/3374#issuecomment-1551990097
@keith-ratcliffe I came to the same conclusion. It looks like this was broken by the changes in #2297 . My fix in #3403 only fixes the case where an exception occurred in the last cycle. In those cases, the last cycle could still appear as the current cycle (because it didn't roll over properly) and wouldn't have the finished time set (so it would be 0), resulting in a negative number when computing the duration by subtracting out the start time.
I still need to do more to fix it when it's still currently running.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on issue #3374: Garbage collector page on monitor weirdness
Posted by "ctubbsii (via GitHub)" <gi...@apache.org>.
ctubbsii commented on issue #3374:
URL: https://github.com/apache/accumulo/issues/3374#issuecomment-1535959904
What version? If snapshot, please provide the closest git hash to what you're using.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] AlbertWhitlock commented on issue #3374: Garbage collector page on monitor weirdness
Posted by "AlbertWhitlock (via GitHub)" <gi...@apache.org>.
AlbertWhitlock commented on issue #3374:
URL: https://github.com/apache/accumulo/issues/3374#issuecomment-1549934651
@ctubbsii What do you mean it couldn't find all your volumes? What did you see that indicated that was related to this?
I have 2 instances defined in instance.volumes. When GC was executing it performed cleanup on root table, metadata table, and user table fine. When cleanup of WAL attempted it would try to find the second instance (actual physical machine) and I accidentally had them unconnected so it was throwing a java.net.NoRouteToHostException. Once I reconnected the machines, the GC runs fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ivakegg commented on issue #3374: Garbage collector page on monitor weirdness
Posted by "ivakegg (via GitHub)" <gi...@apache.org>.
ivakegg commented on issue #3374:
URL: https://github.com/apache/accumulo/issues/3374#issuecomment-1542374000
> > > What version? If snapshot, please provide the closest git hash to what you're using.
> >
> >
> > @ctubbsii It appears that the last external commit we have is [55d972a](https://github.com/apache/accumulo/commit/55d972a4c750c81aba8c59f5841d6369ad40d519)
>
> That is old, but I don't see any updates since then that would have fixed something like this. Is this reproducible?
Yes it is
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ivakegg commented on issue #3374: Garbage collector page on monitor weirdness
Posted by "ivakegg (via GitHub)" <gi...@apache.org>.
ivakegg commented on issue #3374:
URL: https://github.com/apache/accumulo/issues/3374#issuecomment-1542268062
@DomGarguilo Yes it was the SimpleGarbageCollector.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ivakegg commented on issue #3374: Garbage collector page on monitor weirdness
Posted by "ivakegg (via GitHub)" <gi...@apache.org>.
ivakegg commented on issue #3374:
URL: https://github.com/apache/accumulo/issues/3374#issuecomment-1542342365
> What version? If snapshot, please provide the closest git hash to what you're using.
@ctubbsii It appears that the last external commit we have is 55d972a4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] AlbertWhitlock commented on issue #3374: Garbage collector page on monitor weirdness
Posted by "AlbertWhitlock (via GitHub)" <gi...@apache.org>.
AlbertWhitlock commented on issue #3374:
URL: https://github.com/apache/accumulo/issues/3374#issuecomment-1547955804
@ivakegg .... on my instance I also saw the -54y-148d entry. After investigating I found it is because when gc of WAL was kicked off my instance couldn't find all of my volumes. I reconnected to 'new' volume and next gc found the volume and that allowed the value on monitor page to be updated. Perhaps a better 'default' value could be used other than epoch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on issue #3374: Garbage collector page on monitor weirdness
Posted by "ctubbsii (via GitHub)" <gi...@apache.org>.
ctubbsii commented on issue #3374:
URL: https://github.com/apache/accumulo/issues/3374#issuecomment-1548257527
If somebody can reproduce this, it would be helpful to know what the contents of `/rest/gc` show when this is happening. It'd be helpful to know if the edge case is happening in the REST endpoint or in the web view.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on issue #3374: Garbage collector page on monitor weirdness
Posted by "ctubbsii (via GitHub)" <gi...@apache.org>.
ctubbsii commented on issue #3374:
URL: https://github.com/apache/accumulo/issues/3374#issuecomment-1552186258
Should be fixed with my latest commit in #3403
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] asfgit closed issue #3374: Garbage collector page on monitor weirdness
Posted by "asfgit (via GitHub)" <gi...@apache.org>.
asfgit closed issue #3374: Garbage collector page on monitor weirdness
URL: https://github.com/apache/accumulo/issues/3374
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org