You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by Nikolaos Konstantinou <nk...@live.com> on 2013/10/29 09:02:07 UTC
Writing big models to disk
Dear all,
I am trying to write an RDF file to disk, using the following code:
resultModel.write(new BufferedWriter(new FileWriter("file.rdf")), "N3");
All works ok when the file is less than 100mb.
When the file is larger, this has a strange behaviour: the program does not throw an exception, java runtime does not throw an out-of-memory error, and yet the file does not get written. I have increased java memory to -Xmx3072m
The same model can be successfully stored using SDB (>1.5 million triples), so I guess the problem is not in the model. I guess it is a general Java issue? (http://www.java-forums.org/new-java/42031-writing-huge-sized-file-data-more-than-100mb-output-stream-converting-byte.html)
Did anyone else have this issue? Has anyone succeeded in writing rdf files larger than 100mb to disk (and how)?
Thank you in advance for your support.
Best regards,Nikolaos Konstantinou
RE: Writing big models to disk
Posted by Nikolaos Konstantinou <nk...@live.com>.
Thanks a lot for the prompt answer, it works now.
All I had to do was close the stream afterwards. The following code does it, using jena 2.10.1:
BufferedWriter out = new BufferedWriter(new FileWriter("file.rdf"));resultModel.write(out, "N-TRIPLE");out.close();
Best,Nikos
> Date: Tue, 29 Oct 2013 09:34:16 +0000
> From: andy@apache.org
> To: users@jena.apache.org
> Subject: Re: Writing big models to disk
>
> On 29/10/13 08:02, Nikolaos Konstantinou wrote:
> > Dear all,
> > I am trying to write an RDF file to disk, using the following code:
>
> Which version of Jena is this?
>
> > resultModel.write(new BufferedWriter(new FileWriter("file.rdf")), "N3");
>
> Aside: Better top flush the BufferedWriter explicitly:
>
> BufferedWriter x = new BufferedWriter(....)
> resultModel.write(x, "N3") ;
> x.flush() ;
>
> or close it explicitly.
>
> > All works ok when the file is less than 100mb.
> > When the file is larger, this has a strange behaviour: the program does not throw an exception, java runtime does not throw an out-of-memory error, and yet the file does not get written. I have increased java memory to -Xmx3072m
> > The same model can be successfully stored using SDB (>1.5 million triples), so I guess the problem is not in the model. I guess it is a general Java issue? (http://www.java-forums.org/new-java/42031-writing-huge-sized-file-data-more-than-100mb-output-stream-converting-byte.html)
> > Did anyone else have this issue? Has anyone succeeded in writing rdf files larger than 100mb to disk (and how)?
>
> Are you sure it is no just being very, very slow?
>
> http://jena.apache.org/documentation/io/rdf-output.html
>
> "N3" (better, "TTL") is trying to pretty print the data which requires
> analysing the data first. That can involve of small probes in the data
> which for a database is terrible for performance especially if done
> outside a transaction.
>
> You need to use a format that does less analysis e.g.
> RDFFormat.TURTLE_BLOCKS with RDFDataMgr.
>
>
> > Thank you in advance for your support.
> > Best regards,Nikolaos Konstantinou
> >
> >
>
> Is resultModel stored in SDB?
>
> Try the following:
>
> OutputStream out = new FileOutputStream
> RDFDataMgr.write(out, resultModel, RDFFormat.NTRIPLES) ;
> // Or lang.NTRIPLES
>
> which is the best scaling output. RDFFormat.TURTLE_BLOCKS isn't bad though.
>
> Andy
>
Re: UnSubscription
Posted by Rob Vesse <rv...@dotnetrdf.org>.
Send a blank email to users-unsubscribe@jena.apache.org
For full instructions on mailing lists at Apache see
http://www.apache.org/foundation/mailinglists.html
Rob
On 29/10/2013 10:05, "Samir Hamitouche" <21...@etu.unicaen.fr> wrote:
>
>Hello,
>
>please i want UnSubscription, can you help me to remove my email from
>this list of users of jena please i have any more memory space to receive
>messages
>
>
>thank you very much
>
>Best regards
>
>Samir
>
>
>
>----- Mail original -----
>De: "Andy Seaborne" <an...@apache.org>
>À: users@jena.apache.org
>Envoyé: Mardi 29 Octobre 2013 10:34:16
>Objet: Re: Writing big models to disk
>
>On 29/10/13 08:02, Nikolaos Konstantinou wrote:
>> Dear all,
>> I am trying to write an RDF file to disk, using the following code:
>
>Which version of Jena is this?
>
>> resultModel.write(new BufferedWriter(new FileWriter("file.rdf")), "N3");
>
>Aside: Better top flush the BufferedWriter explicitly:
>
>BufferedWriter x = new BufferedWriter(....)
>resultModel.write(x, "N3") ;
>x.flush() ;
>
>or close it explicitly.
>
>> All works ok when the file is less than 100mb.
>> When the file is larger, this has a strange behaviour: the program does
>>not throw an exception, java runtime does not throw an out-of-memory
>>error, and yet the file does not get written. I have increased java
>>memory to -Xmx3072m
>> The same model can be successfully stored using SDB (>1.5 million
>>triples), so I guess the problem is not in the model. I guess it is a
>>general Java issue?
>>(http://www.java-forums.org/new-java/42031-writing-huge-sized-file-data-m
>>ore-than-100mb-output-stream-converting-byte.html)
>> Did anyone else have this issue? Has anyone succeeded in writing rdf
>>files larger than 100mb to disk (and how)?
>
>Are you sure it is no just being very, very slow?
>
>http://jena.apache.org/documentation/io/rdf-output.html
>
>"N3" (better, "TTL") is trying to pretty print the data which requires
>analysing the data first. That can involve of small probes in the data
>which for a database is terrible for performance especially if done
>outside a transaction.
>
>You need to use a format that does less analysis e.g.
>RDFFormat.TURTLE_BLOCKS with RDFDataMgr.
>
>
>> Thank you in advance for your support.
>> Best regards,Nikolaos Konstantinou
>>
>>
>
>Is resultModel stored in SDB?
>
>Try the following:
>
>OutputStream out = new FileOutputStream
>RDFDataMgr.write(out, resultModel, RDFFormat.NTRIPLES) ;
>// Or lang.NTRIPLES
>
>which is the best scaling output. RDFFormat.TURTLE_BLOCKS isn't bad
>though.
>
> Andy
>
UnSubscription
Posted by Samir Hamitouche <21...@etu.unicaen.fr>.
Hello,
please i want UnSubscription, can you help me to remove my email from this list of users of jena please i have any more memory space to receive messages
thank you very much
Best regards
Samir
----- Mail original -----
De: "Andy Seaborne" <an...@apache.org>
À: users@jena.apache.org
Envoyé: Mardi 29 Octobre 2013 10:34:16
Objet: Re: Writing big models to disk
On 29/10/13 08:02, Nikolaos Konstantinou wrote:
> Dear all,
> I am trying to write an RDF file to disk, using the following code:
Which version of Jena is this?
> resultModel.write(new BufferedWriter(new FileWriter("file.rdf")), "N3");
Aside: Better top flush the BufferedWriter explicitly:
BufferedWriter x = new BufferedWriter(....)
resultModel.write(x, "N3") ;
x.flush() ;
or close it explicitly.
> All works ok when the file is less than 100mb.
> When the file is larger, this has a strange behaviour: the program does not throw an exception, java runtime does not throw an out-of-memory error, and yet the file does not get written. I have increased java memory to -Xmx3072m
> The same model can be successfully stored using SDB (>1.5 million triples), so I guess the problem is not in the model. I guess it is a general Java issue? (http://www.java-forums.org/new-java/42031-writing-huge-sized-file-data-more-than-100mb-output-stream-converting-byte.html)
> Did anyone else have this issue? Has anyone succeeded in writing rdf files larger than 100mb to disk (and how)?
Are you sure it is no just being very, very slow?
http://jena.apache.org/documentation/io/rdf-output.html
"N3" (better, "TTL") is trying to pretty print the data which requires
analysing the data first. That can involve of small probes in the data
which for a database is terrible for performance especially if done
outside a transaction.
You need to use a format that does less analysis e.g.
RDFFormat.TURTLE_BLOCKS with RDFDataMgr.
> Thank you in advance for your support.
> Best regards,Nikolaos Konstantinou
>
>
Is resultModel stored in SDB?
Try the following:
OutputStream out = new FileOutputStream
RDFDataMgr.write(out, resultModel, RDFFormat.NTRIPLES) ;
// Or lang.NTRIPLES
which is the best scaling output. RDFFormat.TURTLE_BLOCKS isn't bad though.
Andy
Re: Writing big models to disk
Posted by Andy Seaborne <an...@apache.org>.
On 29/10/13 08:02, Nikolaos Konstantinou wrote:
> Dear all,
> I am trying to write an RDF file to disk, using the following code:
Which version of Jena is this?
> resultModel.write(new BufferedWriter(new FileWriter("file.rdf")), "N3");
Aside: Better top flush the BufferedWriter explicitly:
BufferedWriter x = new BufferedWriter(....)
resultModel.write(x, "N3") ;
x.flush() ;
or close it explicitly.
> All works ok when the file is less than 100mb.
> When the file is larger, this has a strange behaviour: the program does not throw an exception, java runtime does not throw an out-of-memory error, and yet the file does not get written. I have increased java memory to -Xmx3072m
> The same model can be successfully stored using SDB (>1.5 million triples), so I guess the problem is not in the model. I guess it is a general Java issue? (http://www.java-forums.org/new-java/42031-writing-huge-sized-file-data-more-than-100mb-output-stream-converting-byte.html)
> Did anyone else have this issue? Has anyone succeeded in writing rdf files larger than 100mb to disk (and how)?
Are you sure it is no just being very, very slow?
http://jena.apache.org/documentation/io/rdf-output.html
"N3" (better, "TTL") is trying to pretty print the data which requires
analysing the data first. That can involve of small probes in the data
which for a database is terrible for performance especially if done
outside a transaction.
You need to use a format that does less analysis e.g.
RDFFormat.TURTLE_BLOCKS with RDFDataMgr.
> Thank you in advance for your support.
> Best regards,Nikolaos Konstantinou
>
>
Is resultModel stored in SDB?
Try the following:
OutputStream out = new FileOutputStream
RDFDataMgr.write(out, resultModel, RDFFormat.NTRIPLES) ;
// Or lang.NTRIPLES
which is the best scaling output. RDFFormat.TURTLE_BLOCKS isn't bad though.
Andy