Re: tdbloader2 performance for 1B+ triples

[Andy Seaborne <an...@apache.org>]

On 10/08/12 06:34, Michael Brunnbauer wrote:
>
> Hello Andy,
>
> [tdbloader2]
>
> On Thu, Aug 09, 2012 at 06:53:59PM +0200, Michael Brunnbauer wrote:
>> INFO  Add: 55,550,000 Data (Batch: 52 / Avg: 8,785)
>> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
>> Any idea what a good value for -Xmx for 1B+ triples would be ?
>> I will try with 16384 now.
>
> -Xmx16384M throws the memory error after 478 mio triples:
>
> INFO  Add: 478,600,000 Data (Batch: 247 / Avg: 13,627)
> Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded

478 million / 16G heap

This is bizarre.

Previously, with 32G heap:
> INFO  Add: 55,500,000 Data (Batch: 98 / Avg: 10,335)
> INFO    Elapsed: 5,369.59 seconds [2012/08/09 17:45:44 CEST]
> INFO  Add: 55,550,000 Data (Batch: 52 / Avg: 8,785)
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

which is 55 million, a lot less than when you decreased the heap size.

This morning, I have loaded (this is the end of the data phase, 
rformatted.):

11:46:38 INFO  loader               ::
    Add: 747,400,000 Data (Batch: 187,969 / Avg: 131,924)
11:46:43 INFO  loader               ::
    Total: 747,436,151 tuples : 5,669.75 seconds :
    131,828.81 tuples/sec [2012/08/10 11:46:43 UTC]

with no change to tdbloader2 other than fix the classpath setting bug so 
it's -Xmx1200M.

The machine is a 34G machine in Amazon - I even forgot to halt the large 
dataset it is hosting but it's not public yet and only the odd developer 
is testing against it.


What is the data like?  The data shape should only affect the building 
of the node table.

Many long literals? (might explain why the default setting was not enough)

but that does not explain why decreasing the heap size means it gets 
further.

Unrelated:

I have noticed the parameters to sort(1) could be a lot better ...

e.g.
--buffer-size=50%  --parallel=3

I'll try that out but you're crashing out in the data phase before index 
creation.

	Andy

>
> Regards,
>
> Michael Brunnbauer
>

Re: tdbloader2 performance for 1B+ triples

[Andy Seaborne <an...@apache.org>]

On 10/08/12 18:40, Michael Brunnbauer wrote:

> It's an nquads file with 4,9 mio named graphs

OK - that's a difference - may be related.  I'll take a look although I 
need to create some test data first.

If it is this, then a smaller heap and more manageable size of data is 
something I can do.


- data we crawled from the web
> for foaf-search.net with the file url as graph. Many foaf profiles with blank
> nodes but also other data that uses foaf:name. There are also other nquad
> files with DBpedia quads but these should be processed much later.
>
>> Many long literals? (might explain why the default setting was not enough)
>
> I don't know. I can check with a SPARQL query on an older TDB version
> of the data if you want.

It would be useful to know because the node table cache growing might 
(guess) be squeezing the rest of the system.  It's count based so large 
literals mean the cache takes more bytes.

	Andy

>
> Regards,
>
> Michael Brunnbauer
>

Re: tdbloader2 performance for 1B+ triples

[Michael Brunnbauer <br...@netestate.de>]

Hello Andy,

On Fri, Aug 10, 2012 at 01:57:39PM +0100, Andy Seaborne wrote:
> Previously, with 32G heap:

No. with 2048M heap.

> What is the data like?  The data shape should only affect the building 
> of the node table.

It's an nquads file with 4,9 mio named graphs - data we crawled from the web
for foaf-search.net with the file url as graph. Many foaf profiles with blank 
nodes but also other data that uses foaf:name. There are also other nquad 
files with DBpedia quads but these should be processed much later.

> Many long literals? (might explain why the default setting was not enough)

I don't know. I can check with a SPARQL query on an older TDB version
of the data if you want.

Regards,

Michael Brunnbauer

-- 
++  Michael Brunnbauer
++  netEstate GmbH
++  Geisenhausener Straße 11a
++  81379 München
++  Tel +49 89 32 19 77 80
++  Fax +49 89 32 19 77 89 
++  E-Mail brunni@netestate.de
++  http://www.netestate.de/
++
++  Sitz: München, HRB Nr.142452 (Handelsregister B München)
++  USt-IdNr. DE221033342
++  Geschäftsführer: Michael Brunnbauer, Franz Brunnbauer
++  Prokurist: Dipl. Kfm. (Univ.) Markus Hendel