You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jena.apache.org by "Andy Seaborne (JIRA)" <ji...@apache.org> on 2018/06/01 09:53:00 UTC

[jira] [Commented] (JENA-1552) Bulk loader for TDB2 (phased loading)

    [ https://issues.apache.org/jira/browse/JENA-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497799#comment-16497799 ] 

Andy Seaborne commented on JENA-1552:
-------------------------------------

One side effect of the parallel loader is that it takes over the whole machine. The loading process can make the machine unusable for interactive work, for example: typing and mouse clicks get delayed for multiple seconds. 

The phased loading algorithm provides a way to control the amount of resources used at the same time.



> Bulk loader for TDB2 (phased loading)
> -------------------------------------
>
>                 Key: JENA-1552
>                 URL: https://issues.apache.org/jira/browse/JENA-1552
>             Project: Apache Jena
>          Issue Type: Improvement
>          Components: TDB2
>            Reporter: Andy Seaborne
>            Assignee: Andy Seaborne
>            Priority: Major
>
> Following on from JENA-1550, this ticket is for phased loading which combined features of the sequential loader and the parallel loader.
> When building all the persistent datastructures (parallel loader), the work on different indexes at the same time is competing for hardware resources, RAM and I/O bandwidth.  As the size to load grows, this becomes a noticeable slowdown.
> The sequential loader is the other extreme of the design spectrum. It does work on one index at a time so as to maximize caching efficiency.
> Phased loading has parallel operation per phase and splits work into subsets of indexes.
> At 200m and loading to rotational disk, an experimental phased loader working with 2 indexes at a time, starts to become faster than parallel on the same hardware as used for the [figures in JENA-1550|https://issues.apache.org/jira/browse/JENA-1550#comment-16484269] (56K parallel, 76K phased).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)