You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@jena.apache.org by Rose Beck <ro...@gmail.com> on 2013/12/21 01:39:35 UTC

Loading dbpedia.nq: exception in thread main gc overhead limit exceeded

Hi,

I am trying to load dbpedia in n-quads form (from BTC-2012 dataset). I am
using apache-jena-2.10.0. And for loading I used the following command (in
ubuntu 12.04 server, with :
$ cd apache-jena-2.10.0
$

./bin/tdbloader

--loc=/home/
jenaDbpedia -v /home/jyotil/dbpedia.nq

However, after loading a significant number of quads I am getting the
following error:
exception in thread main gc overhead limit exceeded?
        at
com.hp.hpl.jena.tdb.index.bplustree.BPTreeNode.get(BPTreeNode.java:154)
        at
com.hp.hpl.jena.tdb.index.bplustree.BPTreeNode.internalInsert(BPTreeNode.java:447)
        at
com.hp.hpl.jena.tdb.index.bplustree.BPTreeNode.insert(BPTreeNode.java:212)
        at
com.hp.hpl.jena.tdb.index.bplustree.BPlusTree.addAndReturnOld(BPlusTree.java:328)
        at
com.hp.hpl.jena.tdb.index.bplustree.BPlusTree.add(BPlusTree.java:320)
        at
com.hp.hpl.jena.tdb.index.TupleIndexRecord.performAdd(TupleIndexRecord.java:60)
        at
com.hp.hpl.jena.tdb.index.TupleIndexBase.add(TupleIndexBase.java:64)
        at com.hp.hpl.jena.tdb.index.TupleTable.add(TupleTable.java:96)
        at
com.hp.hpl.jena.tdb.nodetable.NodeTupleTableConcrete.addRow(NodeTupleTableConcrete.java:87)
        at
com.hp.hpl.jena.tdb.store.bulkloader.LoaderNodeTupleTable.load(LoaderNodeTupleTable.java:110)
        at
com.hp.hpl.jena.tdb.store.bulkloader.BulkLoader$DestinationDSG.process(BulkLoader.java:265)
        at
com.hp.hpl.jena.tdb.store.bulkloader.BulkLoader$DestinationDSG.quad(BulkLoader.java:257)
        at
org.apache.jena.riot.lang.LangNQuads.runParser(LangNQuads.java:56)
        at org.apache.jena.riot.lang.LangBase.parse(LangBase.java:42)
        at org.apache.jena.riot.RiotReader.parse(RiotReader.java:116)
        at org.apache.jena.riot.RiotReader.parse(RiotReader.java:93)
        at org.apache.jena.riot.RiotReader.parse(RiotReader.java:66)
        at
com.hp.hpl.jena.tdb.store.bulkloader.BulkLoader.loadQuads$(BulkLoader.java:160)
        at
com.hp.hpl.jena.tdb.store.bulkloader.BulkLoader.loadDataset(BulkLoader.java:121)
        at com.hp.hpl.jena.tdb.TDBLoader.loadDataset$(TDBLoader.java:284)
        at com.hp.hpl.jena.tdb.TDBLoader.loadDataset(TDBLoader.java:197)
        at com.hp.hpl.jena.tdb.TDBLoader.load(TDBLoader.java:76)


Can someone please help me as to where am I going wrong?

Re: Loading dbpedia.nq: exception in thread main gc overhead limit exceeded

Posted by Andy Seaborne <an...@apache.org>.

On 21/12/13 03:19, Rose Beck wrote:
> I tried setting using
> export JAVA_OPTS ="-Xms32768m -Xmx32768m"

Try JVM_ARGS

Also, don't set it to 32G - try about 2-4G.

export JVM_ARGS="-Xmx2048m"

A lot of the space TDB loading needs is not in the heap, it's memory 
mapped files which are outside the java heap.  If you set the heap 
large, it will slow down because not much space is left for OS file caching.

	Andy

>
> the maximum heap size to 32 GB. Still I am getting the same error.  I am
> not able to understand as to where am I making the error?
>
>
> On Sat, Dec 21, 2013 at 6:09 AM, Rose Beck <ro...@gmail.com> wrote:
>
>> Hi,
>>
>> I am trying to load dbpedia in n-quads form (from BTC-2012 dataset). I am
>> using apache-jena-2.10.0. And for loading I used the following command (in
>> ubuntu 12.04 server, with :
>> $ cd apache-jena-2.10.0
>> $
>> ./bin/tdbloader
>> --loc=/home/
>> jenaDbpedia -v /home/jyotil/dbpedia.nq
>>
>> However, after loading a significant number of quads I am getting the
>> following error:
>> exception in thread main gc overhead limit exceeded?
>>           at
>> com.hp.hpl.jena.tdb.index.bplustree.BPTreeNode.get(BPTreeNode.java:154)
>>          at
>> com.hp.hpl.jena.tdb.index.bplustree.BPTreeNode.internalInsert(BPTreeNode.java:447)
>>          at
>> com.hp.hpl.jena.tdb.index.bplustree.BPTreeNode.insert(BPTreeNode.java:212)
>>          at
>> com.hp.hpl.jena.tdb.index.bplustree.BPlusTree.addAndReturnOld(BPlusTree.java:328)
>>          at
>> com.hp.hpl.jena.tdb.index.bplustree.BPlusTree.add(BPlusTree.java:320)
>>          at
>> com.hp.hpl.jena.tdb.index.TupleIndexRecord.performAdd(TupleIndexRecord.java:60)
>>          at
>> com.hp.hpl.jena.tdb.index.TupleIndexBase.add(TupleIndexBase.java:64)
>>          at com.hp.hpl.jena.tdb.index.TupleTable.add(TupleTable.java:96)
>>          at
>> com.hp.hpl.jena.tdb.nodetable.NodeTupleTableConcrete.addRow(NodeTupleTableConcrete.java:87)
>>          at
>> com.hp.hpl.jena.tdb.store.bulkloader.LoaderNodeTupleTable.load(LoaderNodeTupleTable.java:110)
>>          at
>> com.hp.hpl.jena.tdb.store.bulkloader.BulkLoader$DestinationDSG.process(BulkLoader.java:265)
>>          at
>> com.hp.hpl.jena.tdb.store.bulkloader.BulkLoader$DestinationDSG.quad(BulkLoader.java:257)
>>          at
>> org.apache.jena.riot.lang.LangNQuads.runParser(LangNQuads.java:56)
>>          at org.apache.jena.riot.lang.LangBase.parse(LangBase.java:42)
>>          at org.apache.jena.riot.RiotReader.parse(RiotReader.java:116)
>>          at org.apache.jena.riot.RiotReader.parse(RiotReader.java:93)
>>          at org.apache.jena.riot.RiotReader.parse(RiotReader.java:66)
>>          at
>> com.hp.hpl.jena.tdb.store.bulkloader.BulkLoader.loadQuads$(BulkLoader.java:160)
>>          at
>> com.hp.hpl.jena.tdb.store.bulkloader.BulkLoader.loadDataset(BulkLoader.java:121)
>>          at com.hp.hpl.jena.tdb.TDBLoader.loadDataset$(TDBLoader.java:284)
>>          at com.hp.hpl.jena.tdb.TDBLoader.loadDataset(TDBLoader.java:197)
>>          at com.hp.hpl.jena.tdb.TDBLoader.load(TDBLoader.java:76)
>>
>>
>> Can someone please help me as to where am I going wrong?
>>
>>
>>
>

Re: Loading dbpedia.nq: exception in thread main gc overhead limit exceeded

Posted by Rose Beck <ro...@gmail.com>.

I tried setting using
export JAVA_OPTS ="-Xms32768m -Xmx32768m"

the maximum heap size to 32 GB. Still I am getting the same error.  I am
not able to understand as to where am I making the error?


On Sat, Dec 21, 2013 at 6:09 AM, Rose Beck <ro...@gmail.com> wrote:

> Hi,
>
> I am trying to load dbpedia in n-quads form (from BTC-2012 dataset). I am
> using apache-jena-2.10.0. And for loading I used the following command (in
> ubuntu 12.04 server, with :
> $ cd apache-jena-2.10.0
> $
> ./bin/tdbloader
> --loc=/home/
> jenaDbpedia -v /home/jyotil/dbpedia.nq
>
> However, after loading a significant number of quads I am getting the
> following error:
> exception in thread main gc overhead limit exceeded?
>          at
> com.hp.hpl.jena.tdb.index.bplustree.BPTreeNode.get(BPTreeNode.java:154)
>         at
> com.hp.hpl.jena.tdb.index.bplustree.BPTreeNode.internalInsert(BPTreeNode.java:447)
>         at
> com.hp.hpl.jena.tdb.index.bplustree.BPTreeNode.insert(BPTreeNode.java:212)
>         at
> com.hp.hpl.jena.tdb.index.bplustree.BPlusTree.addAndReturnOld(BPlusTree.java:328)
>         at
> com.hp.hpl.jena.tdb.index.bplustree.BPlusTree.add(BPlusTree.java:320)
>         at
> com.hp.hpl.jena.tdb.index.TupleIndexRecord.performAdd(TupleIndexRecord.java:60)
>         at
> com.hp.hpl.jena.tdb.index.TupleIndexBase.add(TupleIndexBase.java:64)
>         at com.hp.hpl.jena.tdb.index.TupleTable.add(TupleTable.java:96)
>         at
> com.hp.hpl.jena.tdb.nodetable.NodeTupleTableConcrete.addRow(NodeTupleTableConcrete.java:87)
>         at
> com.hp.hpl.jena.tdb.store.bulkloader.LoaderNodeTupleTable.load(LoaderNodeTupleTable.java:110)
>         at
> com.hp.hpl.jena.tdb.store.bulkloader.BulkLoader$DestinationDSG.process(BulkLoader.java:265)
>         at
> com.hp.hpl.jena.tdb.store.bulkloader.BulkLoader$DestinationDSG.quad(BulkLoader.java:257)
>         at
> org.apache.jena.riot.lang.LangNQuads.runParser(LangNQuads.java:56)
>         at org.apache.jena.riot.lang.LangBase.parse(LangBase.java:42)
>         at org.apache.jena.riot.RiotReader.parse(RiotReader.java:116)
>         at org.apache.jena.riot.RiotReader.parse(RiotReader.java:93)
>         at org.apache.jena.riot.RiotReader.parse(RiotReader.java:66)
>         at
> com.hp.hpl.jena.tdb.store.bulkloader.BulkLoader.loadQuads$(BulkLoader.java:160)
>         at
> com.hp.hpl.jena.tdb.store.bulkloader.BulkLoader.loadDataset(BulkLoader.java:121)
>         at com.hp.hpl.jena.tdb.TDBLoader.loadDataset$(TDBLoader.java:284)
>         at com.hp.hpl.jena.tdb.TDBLoader.loadDataset(TDBLoader.java:197)
>         at com.hp.hpl.jena.tdb.TDBLoader.load(TDBLoader.java:76)
>
>
> Can someone please help me as to where am I going wrong?
>
>
>