You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by Joshua Greben <jg...@stanford.edu> on 2013/02/25 21:07:30 UTC
Loading Large TripleStore using TDB
Hello All,
I am new to this list and to Jena and was wondering if anyone could offer advice for loading a large triplestore.
I am trying to load 670M Ntriples into a store using tdbloader on a single machine with 64-bit hardware and 8GB of memory. However, I am running into a massive slowdown. When the load starts the tdbloader is processing around 30K tps but by the time it has loaded 130M triples it can essentially no longer load any more and slows down to 2300 tps. At that point I have to kill the process because it will basically never finish.
Is 8GB of memory enough or is there a more efficient way to load this data? I am trying to load the data into a single DB location. Should I be splitting up the triples and loading them into different DBs?
Advice from anyone who has experience successfully loading a large triplestore is much appreciated.
Thanks!
- Josh
Joshua Greben
Library Systems Programmer & Analyst
Stanford University Libraries
(650) 714-1937
jgreben@stanford.edu
Re: Loading Large TripleStore using TDB
Posted by Andy Seaborne <an...@apache.org>.
On 26/02/13 19:00, Bill Roberts wrote:
> Since it's N-triples and so one triple per line, why not use unix
> utilities (eg 'split') to divide it into lots of smaller chunks and
> do a series of tdbloader uploads. Should be fairly straightforward
> to script in bash or other scripting language of your choice. That
> should have a lower memory requirement and so avoid the massive
> slowdown. Or am I missing something?
Maybe, but. It can avoid the situation where machine starts swapping
processes if not configured to balance the memory mapped files with
applications running but the effect is moved elsewhere.
But loading that is not into an empty dataset is not optimized.
Both tdbloader andtdbloader2 treat an empty dataset differently and
directly manipulate either indexes or files.
Restarting to load a new batch will
(1) not use optimized loading
(2) not use any part of the datastructures that might have been cached.
The problem at large scale / relatively small RAM is that the effective
cache is small. Lack of caching will be similar to swapping.
Andy
>
> Bill
>
> On 26 Feb 2013, at 18:23, Aaron Coburn <ac...@amherst.edu> wrote:
>
>> I recently had a need to load ~225M triples into a TDB triplestore,
>> and when allocating only ~12G to the triple loader, I experienced
>> the very same slowdowns you described. As an alternative, I just
>> reserved an on-demand, high memory (i.e. ~60GB) instance in the
>> public cloud, and the processing completed in only a few hours. I
>> then just moved the files onto my local server and proceeded from
>> there.
>>
>> Aaron Coburn
>>
>>
>> On Feb 25, 2013, at 1:25 PM, Andy Seaborne <an...@apache.org>
>> wrote:
>>
>>> On 25/02/13 20:07, Joshua Greben wrote:
>>>> Hello All,
>>>>
>>>> I am new to this list and to Jena and was wondering if anyone
>>>> could offer advice for loading a large triplestore.
>>>>
>>>> I am trying to load 670M Ntriples into a store using tdbloader
>>>> on a single machine with 64-bit hardware and 8GB of memory.
>>>> However, I am running into a massive slowdown. When the load
>>>> starts the tdbloader is processing around 30K tps but by the
>>>> time it has loaded 130M triples it can essentially no longer
>>>> load any more and slows down to 2300 tps. At that point I have
>>>> to kill the process because it will basically never finish.
>>>>
>>>> Is 8GB of memory enough or is there a more efficient way to
>>>> load this data? I am trying to load the data into a single DB
>>>> location. Should I be splitting up the triples and loading them
>>>> into different DBs?
>>>>
>>>> Advice from anyone who has experience successfully loading a
>>>> large triplestore is much appreciated.
>>>
>>> Only 8G is pushing it somewhat for 670M triples. It will finish;
>>> it will take a very long time. Faster loads have been reported
>>> by using a larger machine (e.g. Freebase in 8 hours on a IBM
>>> Power7 and 48G RAM).
>>>
>>> tdbloader2 (Linux only) may get you there a bit quicker but
>>> really you need a bigger machine.
>>>
>>> Once built, you can copy the dataset as files to other machines.
>>>
>>> Andy
>>>
>>>>
>>>> Thanks!
>>>>
>>>> - Josh
>>>>
>>>>
>>>>
>>>> Joshua Greben Library Systems Programmer & Analyst Stanford
>>>> University Libraries (650) 714-1937 jgreben@stanford.edu
>>>>
>>>>
>>>>
>>>
>>
>
Re: Loading Large TripleStore using TDB
Posted by Andy Seaborne <an...@apache.org>.
On 29/03/13 19:45, Joshua Greben wrote:
> Yes, there are a lot of large literals.
The general slowness is probably this then - I still don't really see
why it goes as slow later on
If the data is available, I'll take a look. I don't have some hidden
plan about speeding up large literals but if there is something that can
be done, I'd like to do it.
(tempted to run a separate 2 caches: one for large, one for small literals)
>
> I am using tdbloader.
tdbloader2 may do better but the large literals thing has an impact on
it as well (sometimes less so).
>
> The OS is Red Hat 6.3. The hardware is definitely shared, but I am
> not sure what the other applications are. I will have to ask the Sys
> Admin. If memory serves me the hardware has something like 90-ish GB
> or RAM, so I currently have the lion's share. All the RAM was being
> used by the tdbloader at the time. I will also have to inquire about
> a possible ulimit on memory-mapped files.
Good to know - if the java process is huge (it will be larger than the
machine as it includes the file part of memory file areas) then
> I don't think that I mentioned this, but I gave 3200M each as the
> JVM_ARGS (Xms and Xmx). Before that I tried giving it 60G but it
> started swapping on the GC. Maybe there is a happy median?
3G or large enough for your large literals in the node cache (no easy to
estimate I'm afraid).
The memory mapped files aren't in the heap so a very large heap is
actually bad as it reduces the space for file caching.
>
> Thanks again for the pointers and advice.
>
> - Josh
Re: Loading Large TripleStore using TDB
Posted by Joshua Greben <jg...@stanford.edu>.
Yes, there are a lot of large literals.
I am using tdbloader.
The OS is Red Hat 6.3. The hardware is definitely shared, but I am not sure what the other applications are. I will have to ask the Sys Admin. If memory serves me the hardware has something like 90-ish GB or RAM, so I currently have the lion's share. All the RAM was being used by the tdbloader at the time. I will also have to inquire about a possible ulimit on memory-mapped files.
I don't think that I mentioned this, but I gave 3200M each as the JVM_ARGS (Xms and Xmx). Before that I tried giving it 60G but it started swapping on the GC. Maybe there is a happy median?
Thanks again for the pointers and advice.
- Josh
On Mar 29, 2013, at 4:50 AM, Andy Seaborne wrote:
> On 27/03/13 20:46, David Jordan wrote:
>>
>> What indexes exist during the load?
>
> SPO, POS, OSP -- it's a fixed set for TDB (OK, it is changeable but not the facility is not exposed).
>
> More below ...
>
>>
>> On Mar 27, 2013, at 4:40 PM, Joshua Greben wrote:
>>
>>> Hello all,
>>>
>>> I just wanted to give an update on how my loading of 670M triples was going, or in this case not going.
>>>
>>> I ran the riot --time --sink script on my 6.3 GB nt.gz file and this was the result:
>>> 14,128.77 sec 676,740,132 triples 47,898.03 TPS
>
> 47K is slowish - are there lost of large literals?
>
> But it may reflect the fact that I/O on the VM is slow.
>
> (you may have answered this - the gap in thread reflects a gap in my memory :-)
>
>>>
>>> According to this result it should finish in 4 hours, however I am still seeing the same slowdown as before, without any swapping. Here are the first and last few lines of the log file (with only the number of triples processed and the TPS displayed):
>>>
>
> Which loader are you trying?
>
>>> 50,000 15,165
>>> 100,000 21,886
>>> 150,000 25,231
>>> 200,000 27,074
>>> 250,000 28,344
>>> 300,000 29,620
>>> 350,000 30,261
>>> 400,000 31,063
>>> 450,000 31,768
>>> 500,000 31,752
>
> That's all quite slow - for a common distribution of triples, it can be going at 70-80K at this point.
>
>>> ...
>>> 266,350,000 1,564
>>> 266,400,000 1,551
>>> 266,450,000 1,539
>>> 266,500,000 1,522
>>>
>>>
>>> My new machine (still a VM) has 64GB RAM and uses 4 2.3Ghz processors. Any ideas why I can't get a decent processing time? It seems like I would be able to load ~225M triples into three stores over the course of a day or two, but I would rather be able to have one massive store if that is even possible.
>
> That is surprisingly slow. I do loads of 200M-300M size on a smaller box.
>
> What's the OS? Is the hardware shared? (yes - it happens - I've seen setups of 8G VMs ... 4 on an 8G box)? And is VM on hardware with more than 64G?
>
> Is all the RAM getting used? Some Linux setups ulimit the space allowed for memory mapped files so you can give all the RAM you like but it does not get used.
>
> Andy
>
>
>>>
>>> Any ideas appreciated.
>>>
>>> Thanks
>>>
>>> - Josh
>>>
>>> On Feb 26, 2013, at 2:02 PM, Andy Seaborne wrote:
>>>
>>>> Joshua,
>>>>
>>>> If you're in a VM you have another layer trying to help and, in my experience, it does not. Sometimes, maliciously so. [*]
>>>>
>>>> And sharing a real machine can mean you are contending for real resources like disk or memory bandwidth.
>>>>
>>>> In that setup, whatever you do, make sure the VM is not swapping on the real machine. That will make memory mapped files very bad (a sort of double swapping effect).
>>>>
>>>> Reading the N-triples file is unlikely to be the bottle neck. The VM should not make too much difference - only large chunks of stream read I/O is being done (though I have encountered VMs that seem to make that a cost).
>>>>
>>>> You can test this with
>>>>
>>>> riot --time --sink ... files ...
>>>>
>>>> On this machine I'm on currently, a 3y old desktop 8G RAM, no SSD, I get 100K-150K triples/s BSBM from .gz.nt files. BSBM isn't the faster to parse as it has some moderately long literals (or "more bytes to copy" as the parser sees it).
>>>>
>>>> A bit faster from .nt but not enough to make it worth decompressing.
>>>>
>>>> .. just tried ...
>>>>
>>>> ~ >> riot --time --sink ~/Datasets/BSBM/bsbm-25m.nt.gz
>>>> ... 179.50 sec 25,000,250 triples 139,280.26 TPS
>>>>
>>>> ~ >> gzip -d < ~/Datasets/BSBM/bsbm-25m.nt.gz > X.nt
>>>> ~ >> riot --time --sink X.nt
>>>> ... 168.74 sec 25,000,250 triples 148,157.53 TPS
>>>>
>>>> The parser streams ... only a ridiculous proportion of bNodes in large datasets should cause it to slow down.
>>>>
>>>> Please let us know how it goes - it's all useful to build up a pool of experiences.
>>>>
>>>> Andy
>>>>
>>>> [*] as you might guess, I've encountered various "issues" on VMs in the past due to under or mis-provisioning for a database server. AWS is OK, subject to not getting one of the occasional duff machines that people report; I've had one of these once.
>>>>
>>>>
>>>
>>
>
Re: Loading Large TripleStore using TDB
Posted by Andy Seaborne <an...@apache.org>.
On 27/03/13 20:46, David Jordan wrote:
>
> What indexes exist during the load?
SPO, POS, OSP -- it's a fixed set for TDB (OK, it is changeable but not
the facility is not exposed).
More below ...
>
> On Mar 27, 2013, at 4:40 PM, Joshua Greben wrote:
>
>> Hello all,
>>
>> I just wanted to give an update on how my loading of 670M triples was going, or in this case not going.
>>
>> I ran the riot --time --sink script on my 6.3 GB nt.gz file and this was the result:
>> 14,128.77 sec 676,740,132 triples 47,898.03 TPS
47K is slowish - are there lost of large literals?
But it may reflect the fact that I/O on the VM is slow.
(you may have answered this - the gap in thread reflects a gap in my
memory :-)
>>
>> According to this result it should finish in 4 hours, however I am still seeing the same slowdown as before, without any swapping. Here are the first and last few lines of the log file (with only the number of triples processed and the TPS displayed):
>>
Which loader are you trying?
>> 50,000 15,165
>> 100,000 21,886
>> 150,000 25,231
>> 200,000 27,074
>> 250,000 28,344
>> 300,000 29,620
>> 350,000 30,261
>> 400,000 31,063
>> 450,000 31,768
>> 500,000 31,752
That's all quite slow - for a common distribution of triples, it can be
going at 70-80K at this point.
>> ...
>> 266,350,000 1,564
>> 266,400,000 1,551
>> 266,450,000 1,539
>> 266,500,000 1,522
>>
>>
>> My new machine (still a VM) has 64GB RAM and uses 4 2.3Ghz processors. Any ideas why I can't get a decent processing time? It seems like I would be able to load ~225M triples into three stores over the course of a day or two, but I would rather be able to have one massive store if that is even possible.
That is surprisingly slow. I do loads of 200M-300M size on a smaller box.
What's the OS? Is the hardware shared? (yes - it happens - I've seen
setups of 8G VMs ... 4 on an 8G box)? And is VM on hardware with more
than 64G?
Is all the RAM getting used? Some Linux setups ulimit the space allowed
for memory mapped files so you can give all the RAM you like but it does
not get used.
Andy
>>
>> Any ideas appreciated.
>>
>> Thanks
>>
>> - Josh
>>
>> On Feb 26, 2013, at 2:02 PM, Andy Seaborne wrote:
>>
>>> Joshua,
>>>
>>> If you're in a VM you have another layer trying to help and, in my experience, it does not. Sometimes, maliciously so. [*]
>>>
>>> And sharing a real machine can mean you are contending for real resources like disk or memory bandwidth.
>>>
>>> In that setup, whatever you do, make sure the VM is not swapping on the real machine. That will make memory mapped files very bad (a sort of double swapping effect).
>>>
>>> Reading the N-triples file is unlikely to be the bottle neck. The VM should not make too much difference - only large chunks of stream read I/O is being done (though I have encountered VMs that seem to make that a cost).
>>>
>>> You can test this with
>>>
>>> riot --time --sink ... files ...
>>>
>>> On this machine I'm on currently, a 3y old desktop 8G RAM, no SSD, I get 100K-150K triples/s BSBM from .gz.nt files. BSBM isn't the faster to parse as it has some moderately long literals (or "more bytes to copy" as the parser sees it).
>>>
>>> A bit faster from .nt but not enough to make it worth decompressing.
>>>
>>> .. just tried ...
>>>
>>> ~ >> riot --time --sink ~/Datasets/BSBM/bsbm-25m.nt.gz
>>> ... 179.50 sec 25,000,250 triples 139,280.26 TPS
>>>
>>> ~ >> gzip -d < ~/Datasets/BSBM/bsbm-25m.nt.gz > X.nt
>>> ~ >> riot --time --sink X.nt
>>> ... 168.74 sec 25,000,250 triples 148,157.53 TPS
>>>
>>> The parser streams ... only a ridiculous proportion of bNodes in large datasets should cause it to slow down.
>>>
>>> Please let us know how it goes - it's all useful to build up a pool of experiences.
>>>
>>> Andy
>>>
>>> [*] as you might guess, I've encountered various "issues" on VMs in the past due to under or mis-provisioning for a database server. AWS is OK, subject to not getting one of the occasional duff machines that people report; I've had one of these once.
>>>
>>>
>>
>
Re: Loading Large TripleStore using TDB
Posted by David Jordan <da...@bellsouth.net>.
What indexes exist during the load?
On Mar 27, 2013, at 4:40 PM, Joshua Greben wrote:
> Hello all,
>
> I just wanted to give an update on how my loading of 670M triples was going, or in this case not going.
>
> I ran the riot --time --sink script on my 6.3 GB nt.gz file and this was the result:
> 14,128.77 sec 676,740,132 triples 47,898.03 TPS
>
> According to this result it should finish in 4 hours, however I am still seeing the same slowdown as before, without any swapping. Here are the first and last few lines of the log file (with only the number of triples processed and the TPS displayed):
>
> 50,000 15,165
> 100,000 21,886
> 150,000 25,231
> 200,000 27,074
> 250,000 28,344
> 300,000 29,620
> 350,000 30,261
> 400,000 31,063
> 450,000 31,768
> 500,000 31,752
> ...
> 266,350,000 1,564
> 266,400,000 1,551
> 266,450,000 1,539
> 266,500,000 1,522
>
>
> My new machine (still a VM) has 64GB RAM and uses 4 2.3Ghz processors. Any ideas why I can't get a decent processing time? It seems like I would be able to load ~225M triples into three stores over the course of a day or two, but I would rather be able to have one massive store if that is even possible.
>
> Any ideas appreciated.
>
> Thanks
>
> - Josh
>
> On Feb 26, 2013, at 2:02 PM, Andy Seaborne wrote:
>
>> Joshua,
>>
>> If you're in a VM you have another layer trying to help and, in my experience, it does not. Sometimes, maliciously so. [*]
>>
>> And sharing a real machine can mean you are contending for real resources like disk or memory bandwidth.
>>
>> In that setup, whatever you do, make sure the VM is not swapping on the real machine. That will make memory mapped files very bad (a sort of double swapping effect).
>>
>> Reading the N-triples file is unlikely to be the bottle neck. The VM should not make too much difference - only large chunks of stream read I/O is being done (though I have encountered VMs that seem to make that a cost).
>>
>> You can test this with
>>
>> riot --time --sink ... files ...
>>
>> On this machine I'm on currently, a 3y old desktop 8G RAM, no SSD, I get 100K-150K triples/s BSBM from .gz.nt files. BSBM isn't the faster to parse as it has some moderately long literals (or "more bytes to copy" as the parser sees it).
>>
>> A bit faster from .nt but not enough to make it worth decompressing.
>>
>> .. just tried ...
>>
>> ~ >> riot --time --sink ~/Datasets/BSBM/bsbm-25m.nt.gz
>> ... 179.50 sec 25,000,250 triples 139,280.26 TPS
>>
>> ~ >> gzip -d < ~/Datasets/BSBM/bsbm-25m.nt.gz > X.nt
>> ~ >> riot --time --sink X.nt
>> ... 168.74 sec 25,000,250 triples 148,157.53 TPS
>>
>> The parser streams ... only a ridiculous proportion of bNodes in large datasets should cause it to slow down.
>>
>> Please let us know how it goes - it's all useful to build up a pool of experiences.
>>
>> Andy
>>
>> [*] as you might guess, I've encountered various "issues" on VMs in the past due to under or mis-provisioning for a database server. AWS is OK, subject to not getting one of the occasional duff machines that people report; I've had one of these once.
>>
>>
>
Re: Loading Large TripleStore using TDB
Posted by Joshua Greben <jg...@stanford.edu>.
Hello all,
I just wanted to give an update on how my loading of 670M triples was going, or in this case not going.
I ran the riot --time --sink script on my 6.3 GB nt.gz file and this was the result:
14,128.77 sec 676,740,132 triples 47,898.03 TPS
According to this result it should finish in 4 hours, however I am still seeing the same slowdown as before, without any swapping. Here are the first and last few lines of the log file (with only the number of triples processed and the TPS displayed):
50,000 15,165
100,000 21,886
150,000 25,231
200,000 27,074
250,000 28,344
300,000 29,620
350,000 30,261
400,000 31,063
450,000 31,768
500,000 31,752
...
266,350,000 1,564
266,400,000 1,551
266,450,000 1,539
266,500,000 1,522
My new machine (still a VM) has 64GB RAM and uses 4 2.3Ghz processors. Any ideas why I can't get a decent processing time? It seems like I would be able to load ~225M triples into three stores over the course of a day or two, but I would rather be able to have one massive store if that is even possible.
Any ideas appreciated.
Thanks
- Josh
On Feb 26, 2013, at 2:02 PM, Andy Seaborne wrote:
> Joshua,
>
> If you're in a VM you have another layer trying to help and, in my experience, it does not. Sometimes, maliciously so. [*]
>
> And sharing a real machine can mean you are contending for real resources like disk or memory bandwidth.
>
> In that setup, whatever you do, make sure the VM is not swapping on the real machine. That will make memory mapped files very bad (a sort of double swapping effect).
>
> Reading the N-triples file is unlikely to be the bottle neck. The VM should not make too much difference - only large chunks of stream read I/O is being done (though I have encountered VMs that seem to make that a cost).
>
> You can test this with
>
> riot --time --sink ... files ...
>
> On this machine I'm on currently, a 3y old desktop 8G RAM, no SSD, I get 100K-150K triples/s BSBM from .gz.nt files. BSBM isn't the faster to parse as it has some moderately long literals (or "more bytes to copy" as the parser sees it).
>
> A bit faster from .nt but not enough to make it worth decompressing.
>
> .. just tried ...
>
> ~ >> riot --time --sink ~/Datasets/BSBM/bsbm-25m.nt.gz
> ... 179.50 sec 25,000,250 triples 139,280.26 TPS
>
> ~ >> gzip -d < ~/Datasets/BSBM/bsbm-25m.nt.gz > X.nt
> ~ >> riot --time --sink X.nt
> ... 168.74 sec 25,000,250 triples 148,157.53 TPS
>
> The parser streams ... only a ridiculous proportion of bNodes in large datasets should cause it to slow down.
>
> Please let us know how it goes - it's all useful to build up a pool of experiences.
>
> Andy
>
> [*] as you might guess, I've encountered various "issues" on VMs in the past due to under or mis-provisioning for a database server. AWS is OK, subject to not getting one of the occasional duff machines that people report; I've had one of these once.
>
>
Re: Loading Large TripleStore using TDB
Posted by Andy Seaborne <an...@apache.org>.
Joshua,
If you're in a VM you have another layer trying to help and, in my
experience, it does not. Sometimes, maliciously so. [*]
And sharing a real machine can mean you are contending for real
resources like disk or memory bandwidth.
In that setup, whatever you do, make sure the VM is not swapping on the
real machine. That will make memory mapped files very bad (a sort of
double swapping effect).
Reading the N-triples file is unlikely to be the bottle neck. The VM
should not make too much difference - only large chunks of stream read
I/O is being done (though I have encountered VMs that seem to make that
a cost).
You can test this with
riot --time --sink ... files ...
On this machine I'm on currently, a 3y old desktop 8G RAM, no SSD, I get
100K-150K triples/s BSBM from .gz.nt files. BSBM isn't the faster to
parse as it has some moderately long literals (or "more bytes to copy"
as the parser sees it).
A bit faster from .nt but not enough to make it worth decompressing.
.. just tried ...
~ >> riot --time --sink ~/Datasets/BSBM/bsbm-25m.nt.gz
... 179.50 sec 25,000,250 triples 139,280.26 TPS
~ >> gzip -d < ~/Datasets/BSBM/bsbm-25m.nt.gz > X.nt
~ >> riot --time --sink X.nt
... 168.74 sec 25,000,250 triples 148,157.53 TPS
The parser streams ... only a ridiculous proportion of bNodes in large
datasets should cause it to slow down.
Please let us know how it goes - it's all useful to build up a pool of
experiences.
Andy
[*] as you might guess, I've encountered various "issues" on VMs in the
past due to under or mis-provisioning for a database server. AWS is OK,
subject to not getting one of the occasional duff machines that people
report; I've had one of these once.
On 26/02/13 21:31, Joshua Greben wrote:
> Bill,
>
> I tried this approach using a bash script, split, and loop through
> the split files. I tried splitting them into 67 10M line files and 7
> 100M line files respectively, in the hopes that the process would
> recover more memory from the JVM garbage collector with each serial
> load and give a boost to the overall process. In both cases the
> result was pretty much the same as it worked through the files. The
> developers can correct me here but I believe that the slowdown occurs
> when writing to the DB location as opposed to reading from the
> NTriple file.
>
> I am now experimenting with loading the 7 100M line files into 7 DB
> locations and then working with a union query just to get something
> going while I wait on a VM or dedicated server with more RAM. My
> sysadmin wanted to start off by allocating 16G to the VM to see how
> it goes, but now that I read Aaron's comment (thanks Aaron!) I think
> I will forward that info along and try to get 40-60G right away.
>
> - Josh
>
> On Feb 26, 2013, at 11:00 AM, Bill Roberts wrote:
>
>> Since it's N-triples and so one triple per line, why not use unix
>> utilities (eg 'split') to divide it into lots of smaller chunks and
>> do a series of tdbloader uploads. Should be fairly straightforward
>> to script in bash or other scripting language of your choice. That
>> should have a lower memory requirement and so avoid the massive
>> slowdown. Or am I missing something?
>>
>> Bill
>>
>> On 26 Feb 2013, at 18:23, Aaron Coburn <ac...@amherst.edu>
>> wrote:
>>
>>> I recently had a need to load ~225M triples into a TDB
>>> triplestore, and when allocating only ~12G to the triple loader,
>>> I experienced the very same slowdowns you described. As an
>>> alternative, I just reserved an on-demand, high memory (i.e.
>>> ~60GB) instance in the public cloud, and the processing completed
>>> in only a few hours. I then just moved the files onto my local
>>> server and proceeded from there.
>>>
>>> Aaron Coburn
>>>
>>>
>>> On Feb 25, 2013, at 1:25 PM, Andy Seaborne <an...@apache.org>
>>> wrote:
>>>
>>>> On 25/02/13 20:07, Joshua Greben wrote:
>>>>> Hello All,
>>>>>
>>>>> I am new to this list and to Jena and was wondering if anyone
>>>>> could offer advice for loading a large triplestore.
>>>>>
>>>>> I am trying to load 670M Ntriples into a store using
>>>>> tdbloader on a single machine with 64-bit hardware and 8GB of
>>>>> memory. However, I am running into a massive slowdown. When
>>>>> the load starts the tdbloader is processing around 30K tps
>>>>> but by the time it has loaded 130M triples it can essentially
>>>>> no longer load any more and slows down to 2300 tps. At that
>>>>> point I have to kill the process because it will basically
>>>>> never finish.
>>>>>
>>>>> Is 8GB of memory enough or is there a more efficient way to
>>>>> load this data? I am trying to load the data into a single DB
>>>>> location. Should I be splitting up the triples and loading
>>>>> them into different DBs?
>>>>>
>>>>> Advice from anyone who has experience successfully loading a
>>>>> large triplestore is much appreciated.
>>>>
>>>> Only 8G is pushing it somewhat for 670M triples. It will
>>>> finish; it will take a very long time. Faster loads have been
>>>> reported by using a larger machine (e.g. Freebase in 8 hours on
>>>> a IBM Power7 and 48G RAM).
>>>>
>>>> tdbloader2 (Linux only) may get you there a bit quicker but
>>>> really you need a bigger machine.
>>>>
>>>> Once built, you can copy the dataset as files to other
>>>> machines.
>>>>
>>>> Andy
>>>>
>>>>>
>>>>> Thanks!
>>>>>
>>>>> - Josh
>>>>>
>>>>>
>>>>>
>>>>> Joshua Greben Library Systems Programmer & Analyst Stanford
>>>>> University Libraries (650) 714-1937 jgreben@stanford.edu
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Loading Large TripleStore using TDB
Posted by Joshua Greben <jg...@stanford.edu>.
Bill,
I tried this approach using a bash script, split, and loop through the split files. I tried splitting them into 67 10M line files and 7 100M line files respectively, in the hopes that the process would recover more memory from the JVM garbage collector with each serial load and give a boost to the overall process. In both cases the result was pretty much the same as it worked through the files. The developers can correct me here but I believe that the slowdown occurs when writing to the DB location as opposed to reading from the NTriple file.
I am now experimenting with loading the 7 100M line files into 7 DB locations and then working with a union query just to get something going while I wait on a VM or dedicated server with more RAM. My sysadmin wanted to start off by allocating 16G to the VM to see how it goes, but now that I read Aaron's comment (thanks Aaron!) I think I will forward that info along and try to get 40-60G right away.
- Josh
On Feb 26, 2013, at 11:00 AM, Bill Roberts wrote:
> Since it's N-triples and so one triple per line, why not use unix utilities (eg 'split') to divide it into lots of smaller chunks and do a series of tdbloader uploads. Should be fairly straightforward to script in bash or other scripting language of your choice. That should have a lower memory requirement and so avoid the massive slowdown. Or am I missing something?
>
> Bill
>
> On 26 Feb 2013, at 18:23, Aaron Coburn <ac...@amherst.edu> wrote:
>
>> I recently had a need to load ~225M triples into a TDB triplestore, and when allocating only ~12G to the triple loader, I experienced the very same slowdowns you described. As an alternative, I just reserved an on-demand, high memory (i.e. ~60GB) instance in the public cloud, and the processing completed in only a few hours. I then just moved the files onto my local server and proceeded from there.
>>
>> Aaron Coburn
>>
>>
>> On Feb 25, 2013, at 1:25 PM, Andy Seaborne <an...@apache.org> wrote:
>>
>>> On 25/02/13 20:07, Joshua Greben wrote:
>>>> Hello All,
>>>>
>>>> I am new to this list and to Jena and was wondering if anyone could
>>>> offer advice for loading a large triplestore.
>>>>
>>>> I am trying to load 670M Ntriples into a store using tdbloader on a
>>>> single machine with 64-bit hardware and 8GB of memory. However, I am
>>>> running into a massive slowdown. When the load starts the tdbloader
>>>> is processing around 30K tps but by the time it has loaded 130M
>>>> triples it can essentially no longer load any more and slows down to
>>>> 2300 tps. At that point I have to kill the process because it will
>>>> basically never finish.
>>>>
>>>> Is 8GB of memory enough or is there a more efficient way to load this
>>>> data? I am trying to load the data into a single DB location. Should
>>>> I be splitting up the triples and loading them into different DBs?
>>>>
>>>> Advice from anyone who has experience successfully loading a large
>>>> triplestore is much appreciated.
>>>
>>> Only 8G is pushing it somewhat for 670M triples. It will finish; it will take a very long time. Faster loads have been reported by using a larger machine (e.g. Freebase in 8 hours on a IBM Power7 and 48G RAM).
>>>
>>> tdbloader2 (Linux only) may get you there a bit quicker but really you need a bigger machine.
>>>
>>> Once built, you can copy the dataset as files to other machines.
>>>
>>> Andy
>>>
>>>>
>>>> Thanks!
>>>>
>>>> - Josh
>>>>
>>>>
>>>>
>>>> Joshua Greben Library Systems Programmer & Analyst Stanford
>>>> University Libraries (650) 714-1937 jgreben@stanford.edu
>>>>
>>>>
>>>>
>>>
>>
>
Re: Loading Large TripleStore using TDB
Posted by Bill Roberts <bi...@swirrl.com>.
Since it's N-triples and so one triple per line, why not use unix utilities (eg 'split') to divide it into lots of smaller chunks and do a series of tdbloader uploads. Should be fairly straightforward to script in bash or other scripting language of your choice. That should have a lower memory requirement and so avoid the massive slowdown. Or am I missing something?
Bill
On 26 Feb 2013, at 18:23, Aaron Coburn <ac...@amherst.edu> wrote:
> I recently had a need to load ~225M triples into a TDB triplestore, and when allocating only ~12G to the triple loader, I experienced the very same slowdowns you described. As an alternative, I just reserved an on-demand, high memory (i.e. ~60GB) instance in the public cloud, and the processing completed in only a few hours. I then just moved the files onto my local server and proceeded from there.
>
> Aaron Coburn
>
>
> On Feb 25, 2013, at 1:25 PM, Andy Seaborne <an...@apache.org> wrote:
>
>> On 25/02/13 20:07, Joshua Greben wrote:
>>> Hello All,
>>>
>>> I am new to this list and to Jena and was wondering if anyone could
>>> offer advice for loading a large triplestore.
>>>
>>> I am trying to load 670M Ntriples into a store using tdbloader on a
>>> single machine with 64-bit hardware and 8GB of memory. However, I am
>>> running into a massive slowdown. When the load starts the tdbloader
>>> is processing around 30K tps but by the time it has loaded 130M
>>> triples it can essentially no longer load any more and slows down to
>>> 2300 tps. At that point I have to kill the process because it will
>>> basically never finish.
>>>
>>> Is 8GB of memory enough or is there a more efficient way to load this
>>> data? I am trying to load the data into a single DB location. Should
>>> I be splitting up the triples and loading them into different DBs?
>>>
>>> Advice from anyone who has experience successfully loading a large
>>> triplestore is much appreciated.
>>
>> Only 8G is pushing it somewhat for 670M triples. It will finish; it will take a very long time. Faster loads have been reported by using a larger machine (e.g. Freebase in 8 hours on a IBM Power7 and 48G RAM).
>>
>> tdbloader2 (Linux only) may get you there a bit quicker but really you need a bigger machine.
>>
>> Once built, you can copy the dataset as files to other machines.
>>
>> Andy
>>
>>>
>>> Thanks!
>>>
>>> - Josh
>>>
>>>
>>>
>>> Joshua Greben Library Systems Programmer & Analyst Stanford
>>> University Libraries (650) 714-1937 jgreben@stanford.edu
>>>
>>>
>>>
>>
>
Bulk loading [was: Loading Large TripleStore using TDB]
Posted by Andy Seaborne <an...@apache.org>.
On 26/02/13 19:09, Dominique Brezinski wrote:
> This is an interesting subject. What are the steps done and how is the work
> partitioned when loading data? I ask because if everything goes well, I
> will be loading tens of billions of triples a day into a Redshift-backed
> SDB store. Using SQL to load into Redshift is inefficient, so I plan to
> prep the data for insertion into the temp tables into flat files that are
> written into S3, then use the COPY command for distributed load. From there
> the standard INSERT INTO .... SELECT pattern can be used for the Nodes and
> Triples/Quads tables. I want to leverage all the existing code to process
> the triples, then instead of directly inserting, write batches to S3. S3
> really likes 32-64MB chunks.
>
> I will read the code, but if anybody has thoughts on the pattern, or some
> hard-won tips, it would be much appreciated.
>
> Dom
Hi Dom,
There are certainly things you can do to bulk load more efficiently than
naive insertion.
Thoughts, nothing deep:
It might be worth using the naive layout of triple table with strings
for subject/predicate/object.
If you use "layout 2" style, hash is better because it does not require
global coordination to allocate node ids. If you want to use index
numbers, some kind of block allocation scheme will work (allocate blocks
of ids to a processor and it can allocate within that block quickly).
The default hash is only 64 bits, which is good up to aroud a billion
items IIRC before the probability of a clash starts rising. You'll
possibly have to add some bits here.
The prepare-and-load scheme you outline is going to be far better to do
large scale loads if SQL is load - SDB over SQL is going to be worse.
Do let me know how you get on - sounds interesting.
Andy
Re: Loading Large TripleStore using TDB
Posted by Dominique Brezinski <do...@gmail.com>.
This is an interesting subject. What are the steps done and how is the work
partitioned when loading data? I ask because if everything goes well, I
will be loading tens of billions of triples a day into a Redshift-backed
SDB store. Using SQL to load into Redshift is inefficient, so I plan to
prep the data for insertion into the temp tables into flat files that are
written into S3, then use the COPY command for distributed load. From there
the standard INSERT INTO .... SELECT pattern can be used for the Nodes and
Triples/Quads tables. I want to leverage all the existing code to process
the triples, then instead of directly inserting, write batches to S3. S3
really likes 32-64MB chunks.
I will read the code, but if anybody has thoughts on the pattern, or some
hard-won tips, it would be much appreciated.
Dom
On Tue, Feb 26, 2013 at 10:23 AM, Aaron Coburn <ac...@amherst.edu> wrote:
> I recently had a need to load ~225M triples into a TDB triplestore, and
> when allocating only ~12G to the triple loader, I experienced the very same
> slowdowns you described. As an alternative, I just reserved an on-demand,
> high memory (i.e. ~60GB) instance in the public cloud, and the processing
> completed in only a few hours. I then just moved the files onto my local
> server and proceeded from there.
>
> Aaron Coburn
>
>
> On Feb 25, 2013, at 1:25 PM, Andy Seaborne <an...@apache.org> wrote:
>
> > On 25/02/13 20:07, Joshua Greben wrote:
> >> Hello All,
> >>
> >> I am new to this list and to Jena and was wondering if anyone could
> >> offer advice for loading a large triplestore.
> >>
> >> I am trying to load 670M Ntriples into a store using tdbloader on a
> >> single machine with 64-bit hardware and 8GB of memory. However, I am
> >> running into a massive slowdown. When the load starts the tdbloader
> >> is processing around 30K tps but by the time it has loaded 130M
> >> triples it can essentially no longer load any more and slows down to
> >> 2300 tps. At that point I have to kill the process because it will
> >> basically never finish.
> >>
> >> Is 8GB of memory enough or is there a more efficient way to load this
> >> data? I am trying to load the data into a single DB location. Should
> >> I be splitting up the triples and loading them into different DBs?
> >>
> >> Advice from anyone who has experience successfully loading a large
> >> triplestore is much appreciated.
> >
> > Only 8G is pushing it somewhat for 670M triples. It will finish; it
> will take a very long time. Faster loads have been reported by using a
> larger machine (e.g. Freebase in 8 hours on a IBM Power7 and 48G RAM).
> >
> > tdbloader2 (Linux only) may get you there a bit quicker but really you
> need a bigger machine.
> >
> > Once built, you can copy the dataset as files to other machines.
> >
> > Andy
> >
> >>
> >> Thanks!
> >>
> >> - Josh
> >>
> >>
> >>
> >> Joshua Greben Library Systems Programmer & Analyst Stanford
> >> University Libraries (650) 714-1937 jgreben@stanford.edu
> >>
> >>
> >>
> >
>
>
Re: Loading Large TripleStore using TDB
Posted by Aaron Coburn <ac...@amherst.edu>.
I recently had a need to load ~225M triples into a TDB triplestore, and when allocating only ~12G to the triple loader, I experienced the very same slowdowns you described. As an alternative, I just reserved an on-demand, high memory (i.e. ~60GB) instance in the public cloud, and the processing completed in only a few hours. I then just moved the files onto my local server and proceeded from there.
Aaron Coburn
On Feb 25, 2013, at 1:25 PM, Andy Seaborne <an...@apache.org> wrote:
> On 25/02/13 20:07, Joshua Greben wrote:
>> Hello All,
>>
>> I am new to this list and to Jena and was wondering if anyone could
>> offer advice for loading a large triplestore.
>>
>> I am trying to load 670M Ntriples into a store using tdbloader on a
>> single machine with 64-bit hardware and 8GB of memory. However, I am
>> running into a massive slowdown. When the load starts the tdbloader
>> is processing around 30K tps but by the time it has loaded 130M
>> triples it can essentially no longer load any more and slows down to
>> 2300 tps. At that point I have to kill the process because it will
>> basically never finish.
>>
>> Is 8GB of memory enough or is there a more efficient way to load this
>> data? I am trying to load the data into a single DB location. Should
>> I be splitting up the triples and loading them into different DBs?
>>
>> Advice from anyone who has experience successfully loading a large
>> triplestore is much appreciated.
>
> Only 8G is pushing it somewhat for 670M triples. It will finish; it will take a very long time. Faster loads have been reported by using a larger machine (e.g. Freebase in 8 hours on a IBM Power7 and 48G RAM).
>
> tdbloader2 (Linux only) may get you there a bit quicker but really you need a bigger machine.
>
> Once built, you can copy the dataset as files to other machines.
>
> Andy
>
>>
>> Thanks!
>>
>> - Josh
>>
>>
>>
>> Joshua Greben Library Systems Programmer & Analyst Stanford
>> University Libraries (650) 714-1937 jgreben@stanford.edu
>>
>>
>>
>
Re: Loading Large TripleStore using TDB
Posted by Andy Seaborne <an...@apache.org>.
On 25/02/13 20:07, Joshua Greben wrote:
> Hello All,
>
> I am new to this list and to Jena and was wondering if anyone could
> offer advice for loading a large triplestore.
>
> I am trying to load 670M Ntriples into a store using tdbloader on a
> single machine with 64-bit hardware and 8GB of memory. However, I am
> running into a massive slowdown. When the load starts the tdbloader
> is processing around 30K tps but by the time it has loaded 130M
> triples it can essentially no longer load any more and slows down to
> 2300 tps. At that point I have to kill the process because it will
> basically never finish.
>
> Is 8GB of memory enough or is there a more efficient way to load this
> data? I am trying to load the data into a single DB location. Should
> I be splitting up the triples and loading them into different DBs?
>
> Advice from anyone who has experience successfully loading a large
> triplestore is much appreciated.
Only 8G is pushing it somewhat for 670M triples. It will finish; it
will take a very long time. Faster loads have been reported by using a
larger machine (e.g. Freebase in 8 hours on a IBM Power7 and 48G RAM).
tdbloader2 (Linux only) may get you there a bit quicker but really you
need a bigger machine.
Once built, you can copy the dataset as files to other machines.
Andy
>
> Thanks!
>
> - Josh
>
>
>
> Joshua Greben Library Systems Programmer & Analyst Stanford
> University Libraries (650) 714-1937 jgreben@stanford.edu
>
>
>