You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by lookman sanni <lo...@gmail.com> on 2016/08/10 21:49:59 UTC

Out Of Memory Error at Statement Addition

Hi all,

I am trying to add statements to a default model from a self generated RDF
file (5.3 million of lines) though a
simple: model.read(fileinputstream,base).

However, I am getting a Garbage Collector overhead limit exceeded error
'see below). Is it something anyone has come across before ? I have tried
adding run arguments to increase heap size, but it does not seem to be
effective.

Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit
exceeded
at java.util.Arrays.copyOfRange(Unknown Source)
at java.lang.String.<init>(Unknown Source)
at org.apache.jena.iri.impl.LexerPath.yytext(LexerPath.java:420)
at org.apache.jena.iri.impl.AbsLexer.rule(AbsLexer.java:81)
at org.apache.jena.iri.impl.LexerPath.yylex(LexerPath.java:689)
at org.apache.jena.iri.impl.AbsLexer.analyse(AbsLexer.java:52)
at org.apache.jena.iri.impl.Parser.<init>(Parser.java:108)
at org.apache.jena.iri.impl.IRIImpl.<init>(IRIImpl.java:65)
at
org.apache.jena.iri.impl.AbsIRIFactoryImpl.create(AbsIRIFactoryImpl.java:40)
at org.apache.jena.iri.impl.IRIFactoryImpl.create(IRIFactoryImpl.java:264)
at
org.apache.jena.rdfxml.xmlinput.impl.URIReference.fromQName(URIReference.java:180)
at
org.apache.jena.rdfxml.xmlinput.states.WantPropertyElement.startElement(WantPropertyElement.java:65)
at
org.apache.jena.rdfxml.xmlinput.impl.XMLHandler.startElement(XMLHandler.java:111)
at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown Source)
at org.apache.xerces.impl.XMLNamespaceBinder.handleStartElement(Unknown
Source)
at org.apache.xerces.impl.XMLNamespaceBinder.startElement(Unknown Source)
at
org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown
Source)
at
org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown
Source)
at
org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown
Source)
at org.apache.xerces.parsers.DTDConfiguration.parse(Unknown Source)
at org.apache.xerces.parsers.DTDConfiguration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at
org.apache.jena.rdfxml.xmlinput.impl.RDFXMLParser.parse(RDFXMLParser.java:150)
at org.apache.jena.rdfxml.xmlinput.ARP.load(ARP.java:118)
at org.apache.jena.riot.lang.LangRDFXML.parse(LangRDFXML.java:142)
at
org.apache.jena.riot.RDFParserRegistry$ReaderRIOTLang.read(RDFParserRegistry.java:179)
at org.apache.jena.riot.RDFDataMgr.process(RDFDataMgr.java:861)
at org.apache.jena.riot.RDFDataMgr.read(RDFDataMgr.java:259)
at org.apache.jena.riot.RDFDataMgr.read(RDFDataMgr.java:245)
at org.apache.jena.riot.adapters.RDFReaderRIOT.read(RDFReaderRIOT.java:69)
at org.apache.jena.rdf.model.impl.ModelCom.read(ModelCom.java:251)


Thanks in advance for any help.

-- 
Best Regards

Lookman SANNI

Re: Out Of Memory Error at Statement Addition

Posted by lookman sanni <lo...@gmail.com>.
You got me right Andy, I am trying to read a large RDF/XML into a memory
model. I will experiment TDB as well.

Thank you.

On Thu, Aug 11, 2016 at 2:37 PM, Andy Seaborne <an...@apache.org> wrote:

> Hi - the description doesn't say what your doing. Are you reading a large
> RDF/XML file into a memory model?
>
> If so:
>
> 1/ More RAM. if you are using a memory model, then large data means more
> heap needed.
>
> But parsing RDF/XML is not fast anyway so ...
>
> 2/ Load into a database such as TDB
>
> Parse once. Use many times.
>
>         Andy
>
>
>
> On 11/08/16 08:34, lookman sanni wrote:
>
>> Finally works, with a heap size increased to 3072. Any other alternative ?
>>
>> Thanks.
>>
>>
>> On Wed, Aug 10, 2016 at 11:49 PM, lookman sanni <lo...@gmail.com>
>> wrote:
>>
>> Hi all,
>>>
>>> I am trying to add statements to a default model from a self generated
>>> RDF
>>> file (5.3 million of lines) though a simple: model.read(
>>> fileinputstream,base).
>>>
>>> However, I am getting a Garbage Collector overhead limit exceeded error
>>> 'see below). Is it something anyone has come across before ? I have tried
>>> adding run arguments to increase heap size, but it does not seem to be
>>> effective.
>>>
>>> Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit
>>> exceeded
>>> at java.util.Arrays.copyOfRange(Unknown Source)
>>> at java.lang.String.<init>(Unknown Source)
>>> at org.apache.jena.iri.impl.LexerPath.yytext(LexerPath.java:420)
>>> at org.apache.jena.iri.impl.AbsLexer.rule(AbsLexer.java:81)
>>> at org.apache.jena.iri.impl.LexerPath.yylex(LexerPath.java:689)
>>> at org.apache.jena.iri.impl.AbsLexer.analyse(AbsLexer.java:52)
>>> at org.apache.jena.iri.impl.Parser.<init>(Parser.java:108)
>>> at org.apache.jena.iri.impl.IRIImpl.<init>(IRIImpl.java:65)
>>> at org.apache.jena.iri.impl.AbsIRIFactoryImpl.create(
>>> AbsIRIFactoryImpl.java:40)
>>> at org.apache.jena.iri.impl.IRIFactoryImpl.create(IRIFactoryImp
>>> l.java:264)
>>> at org.apache.jena.rdfxml.xmlinput.impl.URIReference.
>>> fromQName(URIReference.java:180)
>>> at org.apache.jena.rdfxml.xmlinput.states.WantPropertyElement.
>>> startElement(WantPropertyElement.java:65)
>>> at org.apache.jena.rdfxml.xmlinput.impl.XMLHandler.
>>> startElement(XMLHandler.java:111)
>>> at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown
>>> Source)
>>> at org.apache.xerces.impl.XMLNamespaceBinder.handleStartElement(Unknown
>>> Source)
>>> at org.apache.xerces.impl.XMLNamespaceBinder.startElement(Unknown
>>> Source)
>>> at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.
>>> scanStartElement(Unknown
>>> Source)
>>> at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl
>>> $FragmentContentDispatcher.dispatch(Unknown Source)
>>> at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.
>>> scanDocument(Unknown
>>> Source)
>>> at org.apache.xerces.parsers.DTDConfiguration.parse(Unknown Source)
>>> at org.apache.xerces.parsers.DTDConfiguration.parse(Unknown Source)
>>> at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
>>> at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
>>> at org.apache.jena.rdfxml.xmlinput.impl.RDFXMLParser.
>>> parse(RDFXMLParser.java:150)
>>> at org.apache.jena.rdfxml.xmlinput.ARP.load(ARP.java:118)
>>> at org.apache.jena.riot.lang.LangRDFXML.parse(LangRDFXML.java:142)
>>> at org.apache.jena.riot.RDFParserRegistry$ReaderRIOTLang.read(
>>> RDFParserRegistry.java:179)
>>> at org.apache.jena.riot.RDFDataMgr.process(RDFDataMgr.java:861)
>>> at org.apache.jena.riot.RDFDataMgr.read(RDFDataMgr.java:259)
>>> at org.apache.jena.riot.RDFDataMgr.read(RDFDataMgr.java:245)
>>> at org.apache.jena.riot.adapters.RDFReaderRIOT.read(RDFReaderRI
>>> OT.java:69)
>>> at org.apache.jena.rdf.model.impl.ModelCom.read(ModelCom.java:251)
>>>
>>>
>>> Thanks in advance for any help.
>>>
>>> --
>>> Best Regards
>>>
>>> Lookman SANNI
>>>
>>>
>>
>>
>>
>


-- 
Best Regards

Lookman SANNI

Re: Out Of Memory Error at Statement Addition

Posted by Andy Seaborne <an...@apache.org>.
Hi - the description doesn't say what your doing. Are you reading a 
large RDF/XML file into a memory model?

If so:

1/ More RAM. if you are using a memory model, then large data means more 
heap needed.

But parsing RDF/XML is not fast anyway so ...

2/ Load into a database such as TDB

Parse once. Use many times.

	Andy


On 11/08/16 08:34, lookman sanni wrote:
> Finally works, with a heap size increased to 3072. Any other alternative ?
>
> Thanks.
>
>
> On Wed, Aug 10, 2016 at 11:49 PM, lookman sanni <lo...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> I am trying to add statements to a default model from a self generated RDF
>> file (5.3 million of lines) though a simple: model.read(
>> fileinputstream,base).
>>
>> However, I am getting a Garbage Collector overhead limit exceeded error
>> 'see below). Is it something anyone has come across before ? I have tried
>> adding run arguments to increase heap size, but it does not seem to be
>> effective.
>>
>> Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit
>> exceeded
>> at java.util.Arrays.copyOfRange(Unknown Source)
>> at java.lang.String.<init>(Unknown Source)
>> at org.apache.jena.iri.impl.LexerPath.yytext(LexerPath.java:420)
>> at org.apache.jena.iri.impl.AbsLexer.rule(AbsLexer.java:81)
>> at org.apache.jena.iri.impl.LexerPath.yylex(LexerPath.java:689)
>> at org.apache.jena.iri.impl.AbsLexer.analyse(AbsLexer.java:52)
>> at org.apache.jena.iri.impl.Parser.<init>(Parser.java:108)
>> at org.apache.jena.iri.impl.IRIImpl.<init>(IRIImpl.java:65)
>> at org.apache.jena.iri.impl.AbsIRIFactoryImpl.create(
>> AbsIRIFactoryImpl.java:40)
>> at org.apache.jena.iri.impl.IRIFactoryImpl.create(IRIFactoryImpl.java:264)
>> at org.apache.jena.rdfxml.xmlinput.impl.URIReference.
>> fromQName(URIReference.java:180)
>> at org.apache.jena.rdfxml.xmlinput.states.WantPropertyElement.
>> startElement(WantPropertyElement.java:65)
>> at org.apache.jena.rdfxml.xmlinput.impl.XMLHandler.
>> startElement(XMLHandler.java:111)
>> at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown
>> Source)
>> at org.apache.xerces.impl.XMLNamespaceBinder.handleStartElement(Unknown
>> Source)
>> at org.apache.xerces.impl.XMLNamespaceBinder.startElement(Unknown Source)
>> at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown
>> Source)
>> at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl
>> $FragmentContentDispatcher.dispatch(Unknown Source)
>> at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown
>> Source)
>> at org.apache.xerces.parsers.DTDConfiguration.parse(Unknown Source)
>> at org.apache.xerces.parsers.DTDConfiguration.parse(Unknown Source)
>> at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
>> at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
>> at org.apache.jena.rdfxml.xmlinput.impl.RDFXMLParser.
>> parse(RDFXMLParser.java:150)
>> at org.apache.jena.rdfxml.xmlinput.ARP.load(ARP.java:118)
>> at org.apache.jena.riot.lang.LangRDFXML.parse(LangRDFXML.java:142)
>> at org.apache.jena.riot.RDFParserRegistry$ReaderRIOTLang.read(
>> RDFParserRegistry.java:179)
>> at org.apache.jena.riot.RDFDataMgr.process(RDFDataMgr.java:861)
>> at org.apache.jena.riot.RDFDataMgr.read(RDFDataMgr.java:259)
>> at org.apache.jena.riot.RDFDataMgr.read(RDFDataMgr.java:245)
>> at org.apache.jena.riot.adapters.RDFReaderRIOT.read(RDFReaderRIOT.java:69)
>> at org.apache.jena.rdf.model.impl.ModelCom.read(ModelCom.java:251)
>>
>>
>> Thanks in advance for any help.
>>
>> --
>> Best Regards
>>
>> Lookman SANNI
>>
>
>
>

Re: Out Of Memory Error at Statement Addition

Posted by lookman sanni <lo...@gmail.com>.
Finally works, with a heap size increased to 3072. Any other alternative ?

Thanks.


On Wed, Aug 10, 2016 at 11:49 PM, lookman sanni <lo...@gmail.com>
wrote:

> Hi all,
>
> I am trying to add statements to a default model from a self generated RDF
> file (5.3 million of lines) though a simple: model.read(
> fileinputstream,base).
>
> However, I am getting a Garbage Collector overhead limit exceeded error
> 'see below). Is it something anyone has come across before ? I have tried
> adding run arguments to increase heap size, but it does not seem to be
> effective.
>
> Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit
> exceeded
> at java.util.Arrays.copyOfRange(Unknown Source)
> at java.lang.String.<init>(Unknown Source)
> at org.apache.jena.iri.impl.LexerPath.yytext(LexerPath.java:420)
> at org.apache.jena.iri.impl.AbsLexer.rule(AbsLexer.java:81)
> at org.apache.jena.iri.impl.LexerPath.yylex(LexerPath.java:689)
> at org.apache.jena.iri.impl.AbsLexer.analyse(AbsLexer.java:52)
> at org.apache.jena.iri.impl.Parser.<init>(Parser.java:108)
> at org.apache.jena.iri.impl.IRIImpl.<init>(IRIImpl.java:65)
> at org.apache.jena.iri.impl.AbsIRIFactoryImpl.create(
> AbsIRIFactoryImpl.java:40)
> at org.apache.jena.iri.impl.IRIFactoryImpl.create(IRIFactoryImpl.java:264)
> at org.apache.jena.rdfxml.xmlinput.impl.URIReference.
> fromQName(URIReference.java:180)
> at org.apache.jena.rdfxml.xmlinput.states.WantPropertyElement.
> startElement(WantPropertyElement.java:65)
> at org.apache.jena.rdfxml.xmlinput.impl.XMLHandler.
> startElement(XMLHandler.java:111)
> at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown
> Source)
> at org.apache.xerces.impl.XMLNamespaceBinder.handleStartElement(Unknown
> Source)
> at org.apache.xerces.impl.XMLNamespaceBinder.startElement(Unknown Source)
> at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown
> Source)
> at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl
> $FragmentContentDispatcher.dispatch(Unknown Source)
> at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown
> Source)
> at org.apache.xerces.parsers.DTDConfiguration.parse(Unknown Source)
> at org.apache.xerces.parsers.DTDConfiguration.parse(Unknown Source)
> at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
> at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
> at org.apache.jena.rdfxml.xmlinput.impl.RDFXMLParser.
> parse(RDFXMLParser.java:150)
> at org.apache.jena.rdfxml.xmlinput.ARP.load(ARP.java:118)
> at org.apache.jena.riot.lang.LangRDFXML.parse(LangRDFXML.java:142)
> at org.apache.jena.riot.RDFParserRegistry$ReaderRIOTLang.read(
> RDFParserRegistry.java:179)
> at org.apache.jena.riot.RDFDataMgr.process(RDFDataMgr.java:861)
> at org.apache.jena.riot.RDFDataMgr.read(RDFDataMgr.java:259)
> at org.apache.jena.riot.RDFDataMgr.read(RDFDataMgr.java:245)
> at org.apache.jena.riot.adapters.RDFReaderRIOT.read(RDFReaderRIOT.java:69)
> at org.apache.jena.rdf.model.impl.ModelCom.read(ModelCom.java:251)
>
>
> Thanks in advance for any help.
>
> --
> Best Regards
>
> Lookman SANNI
>



-- 
Best Regards

Lookman SANNI