You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-dev@jackrabbit.apache.org by Chetan Mehrotra <ch...@gmail.com> on 2015/03/10 07:46:02 UTC
Active deletion of 'deleted' Lucene index files from DataStore
without relying on full scale Blob GC
Hi Team,
With storing of Lucene index files within DataStore our usage pattern
of DataStore has changed between JR2 and Oak.
With JR2 the writes were mostly application based i.e. if application
stores a pdf/image file then that would be stored in DataStore. JR2 by
default would not write stuff to DataStore. Further in deployment
where large number of binary content is present then systems tend to
share the DataStore to avoid duplication of storage. In such cases
running Blob GC is a non trivial task as it involves a manual step and
coordination across multiple deployments. Due to this systems tend to
delay frequency of GC
Now with Oak apart from application the Oak system itself *actively*
uses the DataStore to store the index files for Lucene and there the
churn might be much higher i.e. frequency of creation and deletion of
index file is lot higher. This would accelerate the rate of garbage
generation and thus put lot more pressure on the DataStore storage
requirements.
Any thoughts on how to avoid/reduce the requirement to increase the
frequency of Blob GC?
One possible way would be to provide a special cleanup tool which can
look for such old Lucene index files and deletes them directly without
going through the full fledged MarkAndSweep logic
Thoughts?
Chetan Mehrotra
Re: Active deletion of 'deleted' Lucene index files from DataStore
without relying on full scale Blob GC
Posted by Chetan Mehrotra <ch...@gmail.com>.
Thats one approach we can think about. Thinking further with Lucene
design of immutable files things become simpler (ignoring the reindex
case). In normal usage Lucene never reuses the file name and never
modifies any existing file. So we would not have to worry about
reading older revisions. We only need to keep track of deleted file
and blob's referred by them.
So once a file node is marked as deleted we can possibly have a diff
performed (we already do it to detect when index is changed) and
collect blobId from deleted file nodes from previous state. Those can
be safely deleted *after* some time (allowing other cluster nodes to
pickup).
Chetan Mehrotra
On Tue, Mar 10, 2015 at 4:53 PM, Michael Marth <mm...@adobe.com> wrote:
> Could the Lucene indexer explicitly track these files (e.g. as a property in the index definition)? And also take care of removing them? (the latter part is assuming that the same index file is not identical across various definitions)
>
>> On 10 Mar 2015, at 12:18, Chetan Mehrotra <ch...@gmail.com> wrote:
>>
>> On Tue, Mar 10, 2015 at 4:12 PM, Michael Dürig <md...@apache.org> wrote:
>>> The problem is that you don't even have a list of all previous revisions of
>>> the root node state. Revisions are created on the fly and kept as needed.
>>
>> hmm yup. Then we would need to think of some other approach to know
>> all the blobId referred to by the Lucene Index files
>>
>>
>> Chetan Mehrotra
>
Re: Active deletion of 'deleted' Lucene index files from DataStore
without relying on full scale Blob GC
Posted by Michael Marth <mm...@adobe.com>.
Could the Lucene indexer explicitly track these files (e.g. as a property in the index definition)? And also take care of removing them? (the latter part is assuming that the same index file is not identical across various definitions)
> On 10 Mar 2015, at 12:18, Chetan Mehrotra <ch...@gmail.com> wrote:
>
> On Tue, Mar 10, 2015 at 4:12 PM, Michael Dürig <md...@apache.org> wrote:
>> The problem is that you don't even have a list of all previous revisions of
>> the root node state. Revisions are created on the fly and kept as needed.
>
> hmm yup. Then we would need to think of some other approach to know
> all the blobId referred to by the Lucene Index files
>
>
> Chetan Mehrotra
Re: Active deletion of 'deleted' Lucene index files from DataStore
without relying on full scale Blob GC
Posted by Chetan Mehrotra <ch...@gmail.com>.
On Tue, Mar 10, 2015 at 4:12 PM, Michael Dürig <md...@apache.org> wrote:
> The problem is that you don't even have a list of all previous revisions of
> the root node state. Revisions are created on the fly and kept as needed.
hmm yup. Then we would need to think of some other approach to know
all the blobId referred to by the Lucene Index files
Chetan Mehrotra
Re: Active deletion of 'deleted' Lucene index files from DataStore
without relying on full scale Blob GC
Posted by Michael Dürig <md...@apache.org>.
On 10.3.15 11:32 , Chetan Mehrotra wrote:
> On Tue, Mar 10, 2015 at 3:33 PM, Michael Dürig <md...@apache.org> wrote:
>> SegmentMK doesn't even have the concept of a previous revision of a
>> NodeState.
>
> Yes that is to be thought about. I want to read all previous revision
> for path /oak:index/lucene/:data. For segment I believe I would need
> to start at root references for all previous revisions and then read
> along the required path from those root segments to collect previous
> revisions.
The problem is that you don't even have a list of all previous revisions
of the root node state. Revisions are created on the fly and kept as
needed.
Michael
>
> Would that work?
>
> Chetan Mehrotra
>
Re: Active deletion of 'deleted' Lucene index files from DataStore
without relying on full scale Blob GC
Posted by Chetan Mehrotra <ch...@gmail.com>.
On Tue, Mar 10, 2015 at 3:33 PM, Michael Dürig <md...@apache.org> wrote:
> SegmentMK doesn't even have the concept of a previous revision of a
> NodeState.
Yes that is to be thought about. I want to read all previous revision
for path /oak:index/lucene/:data. For segment I believe I would need
to start at root references for all previous revisions and then read
along the required path from those root segments to collect previous
revisions.
Would that work?
Chetan Mehrotra
Re: Active deletion of 'deleted' Lucene index files from DataStore
without relying on full scale Blob GC
Posted by Michael Dürig <md...@apache.org>.
On 10.3.15 10:49 , Chetan Mehrotra wrote:
> For Segment I am
> not sure how to easily read previous revisions of given NodeState
SegmentMK doesn't even have the concept of a previous revision of a
NodeState.
Michael
Re: Active deletion of 'deleted' Lucene index files from DataStore
without relying on full scale Blob GC
Posted by Chetan Mehrotra <ch...@gmail.com>.
On Tue, Mar 10, 2015 at 1:50 PM, Michael Marth <mm...@adobe.com> wrote:
> But I wonder: how do you envision that this new index cleanup would locate indexes in the content-addressed DS
Thats bit tricky. Have rough idea here on how to approach but would
require more thinking here. The approach I am thinking of is
1. Have an index on oak:QueryIndexDefinition
2. Query for all index definition nodes with type=lucene
3. Get the ':data" node and then perform the listing. Each child node
is a Lucene index file representation
For Mongo I can easy read the previous revisions of the jcr:blob
property and then extract the blobId which can be then be deleted via
direct invocation GarbageCollectableBlobStore API. For Segment I am
not sure how to easily read previous revisions of given NodeState
Chetan Mehrotra
Re: Active deletion of 'deleted' Lucene index files from DataStore
without relying on full scale Blob GC
Posted by Michael Marth <mm...@adobe.com>.
Hi Chetan,
I like the idea.
But I wonder: how do you envision that this new index cleanup would locate indexes in the content-addressed DS?
Michael
> On 10 Mar 2015, at 07:46, Chetan Mehrotra <ch...@gmail.com> wrote:
>
> Hi Team,
>
> With storing of Lucene index files within DataStore our usage pattern
> of DataStore has changed between JR2 and Oak.
>
> With JR2 the writes were mostly application based i.e. if application
> stores a pdf/image file then that would be stored in DataStore. JR2 by
> default would not write stuff to DataStore. Further in deployment
> where large number of binary content is present then systems tend to
> share the DataStore to avoid duplication of storage. In such cases
> running Blob GC is a non trivial task as it involves a manual step and
> coordination across multiple deployments. Due to this systems tend to
> delay frequency of GC
>
> Now with Oak apart from application the Oak system itself *actively*
> uses the DataStore to store the index files for Lucene and there the
> churn might be much higher i.e. frequency of creation and deletion of
> index file is lot higher. This would accelerate the rate of garbage
> generation and thus put lot more pressure on the DataStore storage
> requirements.
>
> Any thoughts on how to avoid/reduce the requirement to increase the
> frequency of Blob GC?
>
> One possible way would be to provide a special cleanup tool which can
> look for such old Lucene index files and deletes them directly without
> going through the full fledged MarkAndSweep logic
>
> Thoughts?
>
> Chetan Mehrotra
Re: Active deletion of 'deleted' Lucene index files from DataStore
without relying on full scale Blob GC
Posted by Chetan Mehrotra <ch...@gmail.com>.
To avoid missing this issue opened OAK-2808. Data collected from
recent runs suggest that this aspect would need to be looked into
going forward
Chetan Mehrotra
On Tue, Mar 10, 2015 at 9:49 PM, Thomas Mueller <mu...@adobe.com> wrote:
> Hi,
>
> I think removing binaries directly without going though the GC logic is
> dangerous, because we can't be sure if there are other references. There
> is one exception, it is if each file is guaranteed to be unique. For that,
> we could for example append a unique UUID to each file. The Lucene file
> system implementation would need to be changed for that (write the UUID,
> but ignore it when reading and reading the file size).
>
> Even in that case, there is still a risk, for example if the binary
> _reference_ is copied, or if an old revision is accessed. How do we ensure
> this does not happen?
>
> Regards,
> Thomas
>
>
> On 10/03/15 07:46, "Chetan Mehrotra" <ch...@gmail.com> wrote:
>
>>Hi Team,
>>
>>With storing of Lucene index files within DataStore our usage pattern
>>of DataStore has changed between JR2 and Oak.
>>
>>With JR2 the writes were mostly application based i.e. if application
>>stores a pdf/image file then that would be stored in DataStore. JR2 by
>>default would not write stuff to DataStore. Further in deployment
>>where large number of binary content is present then systems tend to
>>share the DataStore to avoid duplication of storage. In such cases
>>running Blob GC is a non trivial task as it involves a manual step and
>>coordination across multiple deployments. Due to this systems tend to
>>delay frequency of GC
>>
>>Now with Oak apart from application the Oak system itself *actively*
>>uses the DataStore to store the index files for Lucene and there the
>>churn might be much higher i.e. frequency of creation and deletion of
>>index file is lot higher. This would accelerate the rate of garbage
>>generation and thus put lot more pressure on the DataStore storage
>>requirements.
>>
>>Any thoughts on how to avoid/reduce the requirement to increase the
>>frequency of Blob GC?
>>
>>One possible way would be to provide a special cleanup tool which can
>>look for such old Lucene index files and deletes them directly without
>>going through the full fledged MarkAndSweep logic
>>
>>Thoughts?
>>
>>Chetan Mehrotra
>
Re: Active deletion of 'deleted' Lucene index files from DataStore
without relying on full scale Blob GC
Posted by Thomas Mueller <mu...@adobe.com>.
Hi,
I think removing binaries directly without going though the GC logic is
dangerous, because we can't be sure if there are other references. There
is one exception, it is if each file is guaranteed to be unique. For that,
we could for example append a unique UUID to each file. The Lucene file
system implementation would need to be changed for that (write the UUID,
but ignore it when reading and reading the file size).
Even in that case, there is still a risk, for example if the binary
_reference_ is copied, or if an old revision is accessed. How do we ensure
this does not happen?
Regards,
Thomas
On 10/03/15 07:46, "Chetan Mehrotra" <ch...@gmail.com> wrote:
>Hi Team,
>
>With storing of Lucene index files within DataStore our usage pattern
>of DataStore has changed between JR2 and Oak.
>
>With JR2 the writes were mostly application based i.e. if application
>stores a pdf/image file then that would be stored in DataStore. JR2 by
>default would not write stuff to DataStore. Further in deployment
>where large number of binary content is present then systems tend to
>share the DataStore to avoid duplication of storage. In such cases
>running Blob GC is a non trivial task as it involves a manual step and
>coordination across multiple deployments. Due to this systems tend to
>delay frequency of GC
>
>Now with Oak apart from application the Oak system itself *actively*
>uses the DataStore to store the index files for Lucene and there the
>churn might be much higher i.e. frequency of creation and deletion of
>index file is lot higher. This would accelerate the rate of garbage
>generation and thus put lot more pressure on the DataStore storage
>requirements.
>
>Any thoughts on how to avoid/reduce the requirement to increase the
>frequency of Blob GC?
>
>One possible way would be to provide a special cleanup tool which can
>look for such old Lucene index files and deletes them directly without
>going through the full fledged MarkAndSweep logic
>
>Thoughts?
>
>Chetan Mehrotra