You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-dev@jackrabbit.apache.org by Jukka Zitting <ju...@gmail.com> on 2013/04/26 14:15:04 UTC
Re: Large flat commit problems
Hi,
On Wed, Feb 27, 2013 at 12:24 PM, Jukka Zitting <ju...@gmail.com> wrote:
> Added 167000 pages in 467 seconds (2.80ms/page)
> Imported 167404 pages in 1799 seconds (10.75ms/page)
Here's an update on the latest status with the Wikipedia import benchmark:
$ java -Xmx1500m -jar oak-run/target/oak-run-0.7-SNAPSHOT.jar \
benchmark --wikipedia=simplewiki-20130414-pages-articles.xml \
--cache=200 WikipediaImport Oak-Segment
[...]
Added 171000 pages in 166 seconds (0.97ms/page)
Imported 171382 pages in 355 seconds (2.07ms/page)
[...]
Traversed 171382 pages in 27 seconds (0.16ms/page)
Pretty good progress here.
> There are still a few problems, most notably the fact the index update
> hook operates directly on the plain MemoryNodeBuilder used by the
> current SegmentMK, so it won't benefit from the automatic purging of
> large change-sets and thus ends up requiring lots of memory during the
> massive final save() call. Something like a SegmentNodeBuilder with
> similar internal purge logic like what we already prototyped in
> KernelNodeState should solve that issue.
This is still an issue, see the -Xmx1500m I used for the import.
> The other big issue is the large amount of time spent processing the
> commit hooks. The one hook approach I outlined earlier should help us
> there.
The work we've done here with the Editor mechanism is clearly paying
off as the commit hooks are now taking some 53% of the import time,
down from 74% two months ago, even when we've been adding more
functionality there.
BR,
Jukka Zitting
Re: Large flat commit problems
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Mon, Apr 29, 2013 at 10:17 AM, Lukas Eder <ma...@adobe.com> wrote:
> Do comparisons with Jackrabbit exist?
Not for this particular benchmark, since Jackrabbit 2.x is unable to
deal with such a large transaction (~400MB of non-binary content) or
the flat hierarchy (170k child nodes).
Some Jackrabbit comparisons are included in the other benchmark thread.
BR,
Jukka Zitting
Re: Large flat commit problems
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Fri, Apr 26, 2013 at 3:15 PM, Jukka Zitting <ju...@gmail.com> wrote:
> Imported 171382 pages in 355 seconds (2.07ms/page)
For comparison, here's the result if I change the benchmark code to
call save() after every 1k pages:
Imported 171382 pages in 1154 seconds (6.74ms/page)
BR,
Jukka Zitting
RE: Large flat commit problems
Posted by Marcel Reutegger <mr...@adobe.com>.
> And with the TarMK:
>
> Added 171000 pages in 36 seconds (0.21ms/page)
> Imported 171382 pages in 54 seconds (0.32ms/page)
> [...]
> Traversed 171382 pages in 2 seconds (0.01ms/page)
>
> I particularly like that last line. :-)
very nice!
regards
marcel
Re: Large flat commit problems
Posted by Thomas Mueller <mu...@adobe.com>.
Hi,
Great! If we continue like this, the TarMK will next month have negative
values :-)
Regards,
Thomas
On 5/30/13 2:45 PM, "Jukka Zitting" <ju...@gmail.com> wrote:
>Hi,
>
>On Fri, Apr 26, 2013 at 3:15 PM, Jukka Zitting <ju...@gmail.com>
>wrote:
>> On Wed, Feb 27, 2013 at 12:24 PM, Jukka Zitting
>><ju...@gmail.com> wrote:
>>> Added 167000 pages in 467 seconds (2.80ms/page)
>>> Imported 167404 pages in 1799 seconds (10.75ms/page)
>>
>> Added 171000 pages in 166 seconds (0.97ms/page)
>> Imported 171382 pages in 355 seconds (2.07ms/page)
>> [...]
>> Traversed 171382 pages in 27 seconds (0.16ms/page)
>>
>> Pretty good progress here.
>
>Getting better still:
>
> Added 171000 pages in 61 seconds (0.36ms/page)
> Imported 171382 pages in 194 seconds (1.14ms/page)
> [...]
> Traversed 171382 pages in 26 seconds (0.16ms/page)
>
>And with the TarMK:
>
> Added 171000 pages in 36 seconds (0.21ms/page)
> Imported 171382 pages in 54 seconds (0.32ms/page)
> [...]
> Traversed 171382 pages in 2 seconds (0.01ms/page)
>
>I particularly like that last line. :-)
>
>BR,
>
>Jukka Zitting
Re: Large flat commit problems
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Fri, Apr 26, 2013 at 3:15 PM, Jukka Zitting <ju...@gmail.com> wrote:
> On Wed, Feb 27, 2013 at 12:24 PM, Jukka Zitting <ju...@gmail.com> wrote:
>> Added 167000 pages in 467 seconds (2.80ms/page)
>> Imported 167404 pages in 1799 seconds (10.75ms/page)
>
> Added 171000 pages in 166 seconds (0.97ms/page)
> Imported 171382 pages in 355 seconds (2.07ms/page)
> [...]
> Traversed 171382 pages in 27 seconds (0.16ms/page)
>
> Pretty good progress here.
Getting better still:
Added 171000 pages in 61 seconds (0.36ms/page)
Imported 171382 pages in 194 seconds (1.14ms/page)
[...]
Traversed 171382 pages in 26 seconds (0.16ms/page)
And with the TarMK:
Added 171000 pages in 36 seconds (0.21ms/page)
Imported 171382 pages in 54 seconds (0.32ms/page)
[...]
Traversed 171382 pages in 2 seconds (0.01ms/page)
I particularly like that last line. :-)
BR,
Jukka Zitting
Re: Large flat commit problems
Posted by Lukas Eder <ma...@adobe.com>.
Hi,
On 4/26/13 2:15 PM, "Jukka Zitting" <ju...@gmail.com> wrote:
>Hi,
>
>On Wed, Feb 27, 2013 at 12:24 PM, Jukka Zitting <ju...@gmail.com>
>wrote:
>> Added 167000 pages in 467 seconds (2.80ms/page)
>> Imported 167404 pages in 1799 seconds (10.75ms/page)
>
>Here's an update on the latest status with the Wikipedia import benchmark:
>
> $ java -Xmx1500m -jar oak-run/target/oak-run-0.7-SNAPSHOT.jar \
> benchmark --wikipedia=simplewiki-20130414-pages-articles.xml \
> --cache=200 WikipediaImport Oak-Segment
> [...]
> Added 171000 pages in 166 seconds (0.97ms/page)
> Imported 171382 pages in 355 seconds (2.07ms/page)
> [...]
> Traversed 171382 pages in 27 seconds (0.16ms/page)
>
>Pretty good progress here.
Those are impressive numbers. Do comparisons with Jackrabbit exist?
Cheers
Lukas
>> There are still a few problems, most notably the fact the index update
>> hook operates directly on the plain MemoryNodeBuilder used by the
>> current SegmentMK, so it won't benefit from the automatic purging of
>> large change-sets and thus ends up requiring lots of memory during the
>> massive final save() call. Something like a SegmentNodeBuilder with
>> similar internal purge logic like what we already prototyped in
>> KernelNodeState should solve that issue.
>
>This is still an issue, see the -Xmx1500m I used for the import.
>
>> The other big issue is the large amount of time spent processing the
>> commit hooks. The one hook approach I outlined earlier should help us
>> there.
>
>The work we've done here with the Editor mechanism is clearly paying
>off as the commit hooks are now taking some 53% of the import time,
>down from 74% two months ago, even when we've been adding more
>functionality there.
>
>BR,
>
>Jukka Zitting