You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jena.apache.org by Andy Seaborne <an...@apache.org> on 2021/10/23 13:30:38 UTC

JENA-2175 - xloader

xloader (the "x" is from "eXternal") is TDB1 tdbloader2 rebranded, a 
version ported to TDB2, and with some performance and intermediate 
resource consumption improvements.

xloader is not a replacement for regular TDB1 and TDB2 loaders.

The goal of xloader is scale of loading on modest hardware, loading to 
spinning disk (no SSD required), with one 4-core processor.  Scale is 1 
billion+ triples/quads.

It has loaded 1 billion BSBM triples in 7.5 hours into a TDB2 database.

There are some more improvements to that in-progress that not yet in the 
Jena codebase.

It is not as fast as other TDB loaders; it is designed to avoid the slow 
down effects that naively loading indexes on disk causes which can 
reduce loading to a standstill, and even crashing the machine due to 
overload.

     Andy