Expiring the META/repository-* markers used by MarkSweepGarbageCollector ?

[Bertrand Delacretaz <bd...@apache.org>]

Hi,

When a client connects only temporarily to a SharedS3DataStore (for
example) and then goes away, the META/repository-* marker created by
SegmentNodeStoreService is not removed.

This causes MarkSweepGarbageCollector to abort with a "not all
repositories have marked references available" message.

Do people see an issue with adding an expiration time to those
META/repository-* markers?

MarkSweepGarbageCollector can then ignore expired markers, considering
them to belong to long gone clients.

I suppose that the expiration time can be stored as data in the marker
blob, and it would have to be refreshed periodically by then client,
unless configured to never expire.

I can provide a patch for that but wanted to first check for any
issues that I overlooked, as I'm not familiar with that code.

What do people think?

-Bertrand

Re: Expiring the META/repository-* markers used by MarkSweepGarbageCollector ?

[Amit Jain <am...@ieee.org>]

Hi Bertrand,

This approach would work and would be a useful addition if support for
repositories transiently sharing the DataStore is needed.

Though it would be an extremely rare case we need to be mindful about how
to signal a failure to refresh the expiration time and what action is to be
taken as this could lead to data loss for the client (but only for this).

Thanks
Amit

On Fri, May 5, 2017 at 7:26 PM, Bertrand Delacretaz <bd...@apache.org>
wrote:

> Hi,
>
> When a client connects only temporarily to a SharedS3DataStore (for
> example) and then goes away, the META/repository-* marker created by
> SegmentNodeStoreService is not removed.
>
> This causes MarkSweepGarbageCollector to abort with a "not all
> repositories have marked references available" message.
>
> Do people see an issue with adding an expiration time to those
> META/repository-* markers?
>
> MarkSweepGarbageCollector can then ignore expired markers, considering
> them to belong to long gone clients.
>
> I suppose that the expiration time can be stored as data in the marker
> blob, and it would have to be refreshed periodically by then client,
> unless configured to never expire.
>
> I can provide a patch for that but wanted to first check for any
> issues that I overlooked, as I'm not familiar with that code.
>
> What do people think?
>
> -Bertrand
>