TarFile scans whole file in constructor

[Axel Hanikel <ah...@adobe.com>]

Hi,

The org.apache.jackrabbit.oak.plugins.segment.file.TarFile constructor 
reads through the whole tar file, building up an in-memory map from 
segment uuid to the position in the file. This is done each time the 
repository is started, and it takes quite a long time. Shouldn't this 
index be persisted somehow, and be updated when the tar file is being 
written to?

Should I create an issue for this or am I missing anything?

Bye!
     Axel

Re: TarFile scans whole file in constructor

[Axel Hanikel <ah...@adobe.com>]

Hi Jukka

> An sub-task of OAK-593 would be nice.

Ok, done: https://issues.apache.org/jira/browse/OAK-1423

Bye!
     Axel

Re: TarFile scans whole file in constructor

[Jukka Zitting <ju...@gmail.com>]

Hi,

On Fri, Feb 14, 2014 at 8:42 AM, Axel Hanikel <ah...@adobe.com> wrote:
> The org.apache.jackrabbit.oak.plugins.segment.file.TarFile constructor reads
> through the whole tar file, building up an in-memory map from segment uuid
> to the position in the file. This is done each time the repository is
> started, and it takes quite a long time. Shouldn't this index be persisted
> somehow, and be updated when the tar file is being written to?

Yes, an index will most likely be needed. So far one hasn't been added
to keep things simple and to gather more experience about the size
distribution and typical access patterns of the segments.

My idea so far for such an index has been to include a special
directory at the end of each tar file, a bit like the zip format does.
This directory could list the identifier, location and size of all the
segments included in that tar file. The required size would be 24
bytes per segment, so the size overhead would be insignificant
compared to the minimum of 1kB of space taken by each segment in the
tar format.

> Should I create an issue for this or am I missing anything?

An sub-task of OAK-593 would be nice.

BR,

Jukka Zitting

Re: TarFile scans whole file in constructor

[Jukka Zitting <ju...@gmail.com>]

Hi,

On Mon, Feb 17, 2014 at 4:30 AM, Thomas Mueller <mu...@adobe.com> wrote:
> If the tar file only contains a few entries, then I think it's not a
> problem. If the tar file contains many entries, then I think this by
> itself is an issue, as maintaining such an index is problematic.

I'm expecting a normal 256MB tar file to contain 1k - 10k segments, so
an index for such a file would range from a few dozen to a few hundred
kilobytes, i.e. no more than 1% of the required space.

I'm thinking of only adding such an index at the end of a tar file
once it's filled up or explicitly closed, until then the index can be
kept in memory or created by a startup scan.

As the tar files are append-only in nature, index maintenance
shouldn't be too much of a problem. Once the index is written at the
end of a tar file, there'll be no need to update it as new segments
will be written to the next tar file.

> Some statistics about segement sizes based on one example repository:

Thanks for the summary!

As a general rule I've tried to fit as much data in a segment as
possible (including merging multiple small commits in one segment) in
an attempt to keep the median segment size fairly large (> 1kB) as the
tar file format requires at least 1kB of disk space for each non-empty
entry. There's still a few ideas left on how to increase segment
sizes, if it turns out that doing so would be beneficial also for
indexing.

BR,

Jukka Zitting

Re: TarFile scans whole file in constructor

[Thomas Mueller <mu...@adobe.com>]

Hi,

If the tar file only contains a few entries, then I think it's not a
problem. If the tar file contains many entries, then I think this by
itself is an issue, as maintaining such an index is problematic.

I would prefer if the smallest entry is 10 KB, which would result in less
entries, and reading the index should be faster.

Some statistics about segement sizes based on one example repository:

total size: 1'218'330'536 bytes (5 tar files)
average size: 27'630 bytes
total count: 44'093 entries

count, segement size between


17681, 10-99
 6269, 100-999
11397, 1000-9999
 4554, 10000-99999 
 8746, 100000-999999
 4192, 1000000-9999999


Regards,
Thomas



On 14/02/14 14:42, "Axel Hanikel" <ah...@adobe.com> wrote:

>Hi,
>
>The org.apache.jackrabbit.oak.plugins.segment.file.TarFile constructor
>reads through the whole tar file, building up an in-memory map from
>segment uuid to the position in the file. This is done each time the
>repository is started, and it takes quite a long time. Shouldn't this
>index be persisted somehow, and be updated when the tar file is being
>written to?
>
>Should I create an issue for this or am I missing anything?
>
>Bye!
>     Axel
>
>