You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by Brandon Sara <br...@collectivemedicaltech.com> on 2021/04/09 17:07:19 UTC

Adding auto-deletion to compact task?

I would love it if we could have an automatic deletion option with the compact task, otherwise, a good amount of work can end up being needed just to ensure that a no longer needed `Data-****` folder is removed. Is this something that the maintainers are willing to consider? If so, I’d be willing to help with the implementation of it.

Thanks.
-- 


*NOTICES*:

 

1.  **No PHI in Email**.  Collective Medical policy 
prohibits sending protected health information by email, which may violate 
applicable law. If sending PHI is necessary, please contact me for secure 
delivery instructions.

 

2.  **Confidentiality**.  This message and any 
attachments may be confidential and proprietary. If you received this in 
error, please contact me immediately and delete this message.  

Re: Adding auto-deletion to compact task?

Posted by Andy Seaborne <an...@apache.org>.
Hi Brandon,

There are a number of possible steps after a compaction - delete, 
archive, compress-archive-delete.

This sounds like a natural thing to have but at the same time I'm a bit 
wary of adding features one at a time because a feature is a commitment 
to maintain and some other concern may mean a different approach would 
have been better. Example: maybe different privileges for compact and 
actually deleting data on disk.

With Fuseki/main (it currently does not have the admin functions of 
Fuseki/webapp) we do have the scope to revisit administration.

I've opened a discussion JIRA

   https://issues.apache.org/jira/browse/JENA-2087

for what a new set of administration features for Fuseki/main might 
include. Hope you'll join in.

     Andy

BTW:

Deleting will not work on MS Windows. There is a long standing Java 
issue that memory mapped files do not get released until after the JVM 
exits.

https://bugs.openjdk.java.net/browse/JDK-4715154
https://bugs.openjdk.java.net/browse/JDK-4724038

On 09/04/2021 18:07, Brandon Sara wrote:
> I would love it if we could have an automatic deletion option with the compact task, otherwise, a good amount of work can end up being needed just to ensure that a no longer needed `Data-****` folder is removed. Is this something that the maintainers are willing to consider? If so, I’d be willing to help with the implementation of it.
> 
> Thanks.
>

Re: Adding auto-deletion to compact task?

Posted by Andy Seaborne <an...@apache.org>.

On 12/04/2021 11:07, Rob Vesse wrote:
> Yes such a feature would be a valuable contribution.  It's certainly something that other users have asked for.
> 
> Your main issue is probably going to be verifying that there aren't open transactions against the old versions of the data, Andy might have some ideas/pointers to whether the internals of TDB2 allow for determining that?

There won't be any transactions on the old store - when compaction has 
finished, the system goes into "exclusive" mode to switch the active 
database from old to new.

By the time compaction returns, the old database is no longer used for 
anything (except for the Windows thing).

     Andy

> 
> Rob
> 
> On 09/04/2021, 18:08, "Brandon Sara" <br...@collectivemedicaltech.com> wrote:
> 
>      I would love it if we could have an automatic deletion option with the compact task, otherwise, a good amount of work can end up being needed just to ensure that a no longer needed `Data-****` folder is removed. Is this something that the maintainers are willing to consider? If so, I’d be willing to help with the implementation of it.
> 
>      Thanks.
>      --
> 
> 
>      *NOTICES*:
> 
> 
> 
>      1.  **No PHI in Email**.  Collective Medical policy
>      prohibits sending protected health information by email, which may violate
>      applicable law. If sending PHI is necessary, please contact me for secure
>      delivery instructions.
> 
> 
> 
>      2.  **Confidentiality**.  This message and any
>      attachments may be confidential and proprietary. If you received this in
>      error, please contact me immediately and delete this message.
> 
> 
> 
> 
> 
>

Re: Adding auto-deletion to compact task?

Posted by Rob Vesse <rv...@dotnetrdf.org>.
Yes such a feature would be a valuable contribution.  It's certainly something that other users have asked for.

Your main issue is probably going to be verifying that there aren't open transactions against the old versions of the data, Andy might have some ideas/pointers to whether the internals of TDB2 allow for determining that?

Rob

On 09/04/2021, 18:08, "Brandon Sara" <br...@collectivemedicaltech.com> wrote:

    I would love it if we could have an automatic deletion option with the compact task, otherwise, a good amount of work can end up being needed just to ensure that a no longer needed `Data-****` folder is removed. Is this something that the maintainers are willing to consider? If so, I’d be willing to help with the implementation of it.

    Thanks.
    -- 


    *NOTICES*:



    1.  **No PHI in Email**.  Collective Medical policy 
    prohibits sending protected health information by email, which may violate 
    applicable law. If sending PHI is necessary, please contact me for secure 
    delivery instructions.



    2.  **Confidentiality**.  This message and any 
    attachments may be confidential and proprietary. If you received this in 
    error, please contact me immediately and delete this message.