You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by Michael Brunnbauer <br...@netestate.de> on 2013/01/27 12:21:18 UTC
jena-fuseki-0.2.5: java.lang.InternalError during SPARQL update
hi all
the following exception occured during a SPARQL update on jena-fuseki-0.2.5
with jdk-7u11-linux-x64. The system has been doing SPARQL updates without
problem with jena-fuseki-0.2.5 since October and with jdk-7u11-linux-x64 since
Jan 15. I have updated the 32bit glibc yesterday but it should not be involved
as this is a 64bit version of Java. I guess only Oracle can do something here ?
00:43:16 INFO Fuseki :: [10654] POST http://ts.foaf-search.net:3030/crawl/update
00:43:53 WARN SPARQL_Update$HttpActionUpdate :: Transaction still active in endWriter - no commit or abort seen (forced abort)
00:43:53 WARN Fuseki :: [10654] RC = 500 : a fault occurred in a recent unsafe memory access operation in compiled Java code
java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
at com.hp.hpl.jena.tdb.base.file.BlockAccessBase.check(BlockAccessBase.java:118)
at com.hp.hpl.jena.tdb.base.file.BlockAccessMapped.read(BlockAccessMapped.java:93)
at com.hp.hpl.jena.tdb.base.block.BlockMgrFileAccess.getBlock(BlockMgrFileAccess.java:81)
at com.hp.hpl.jena.tdb.base.block.BlockMgrFileAccess.getRead(BlockMgrFileAccess.java:70)
at com.hp.hpl.jena.tdb.base.block.BlockMgrWrapper.getRead(BlockMgrWrapper.java:52)
at com.hp.hpl.jena.tdb.transaction.BlockMgrJournal.getRead(BlockMgrJournal.java:137)
at com.hp.hpl.jena.tdb.base.page.PageBlockMgr.getRead(PageBlockMgr.java:68)
at com.hp.hpl.jena.tdb.index.bplustree.BPTreeNode.getMgrRead(BPTreeNode.java:164)
at com.hp.hpl.jena.tdb.index.bplustree.BPTreeNode.get(BPTreeNode.java:154)
at com.hp.hpl.jena.tdb.index.bplustree.BPTreeNode.internalInsert(BPTreeNode.java:447)
at com.hp.hpl.jena.tdb.index.bplustree.BPTreeNode.internalInsert(BPTreeNode.java:468)
at com.hp.hpl.jena.tdb.index.bplustree.BPTreeNode.internalInsert(BPTreeNode.java:468)
at com.hp.hpl.jena.tdb.index.bplustree.BPTreeNode.internalInsert(BPTreeNode.java:468)
at com.hp.hpl.jena.tdb.index.bplustree.BPTreeNode.insert(BPTreeNode.java:212)
at com.hp.hpl.jena.tdb.index.bplustree.BPlusTree.addAndReturnOld(BPlusTree.java:328)
at com.hp.hpl.jena.tdb.index.bplustree.BPlusTree.add(BPlusTree.java:320)
at com.hp.hpl.jena.tdb.index.TupleIndexRecord.performAdd(TupleIndexRecord.java:60)
at com.hp.hpl.jena.tdb.index.TupleIndexBase.add(TupleIndexBase.java:64)
at com.hp.hpl.jena.tdb.index.TupleTable.add(TupleTable.java:70)
at com.hp.hpl.jena.tdb.nodetable.NodeTupleTableConcrete.addRow(NodeTupleTableConcrete.java:87)
at com.hp.hpl.jena.tdb.store.QuadTable.add(QuadTable.java:63)
at com.hp.hpl.jena.tdb.store.QuadTable.add(QuadTable.java:57)
at com.hp.hpl.jena.tdb.store.GraphNamedTDB._performAdd(GraphNamedTDB.java:83)
at com.hp.hpl.jena.tdb.store.GraphTDBBase.performAdd(GraphTDBBase.java:77)
at com.hp.hpl.jena.graph.impl.SimpleBulkUpdateHandler.add(SimpleBulkUpdateHandler.java:63)
at com.hp.hpl.jena.graph.impl.SimpleBulkUpdateHandler.addIterator(SimpleBulkUpdateHandler.java:75)
at com.hp.hpl.jena.graph.impl.SimpleBulkUpdateHandler.add(SimpleBulkUpdateHandler.java:87)
at com.hp.hpl.jena.graph.impl.SimpleBulkUpdateHandler.add(SimpleBulkUpdateHandler.java:81)
at com.hp.hpl.jena.sparql.modify.UpdateEngineWorker.visit(UpdateEngineWorker.java:161)
at com.hp.hpl.jena.sparql.modify.request.UpdateLoad.visit(UpdateLoad.java:59)
at com.hp.hpl.jena.sparql.modify.UpdateEngineMain.execute(UpdateEngineMain.java:40)
at com.hp.hpl.jena.sparql.modify.UpdateProcessorBase.execute(UpdateProcessorBase.java:56)
at com.hp.hpl.jena.update.UpdateAction.execute$(UpdateAction.java:334)
at com.hp.hpl.jena.update.UpdateAction.execute(UpdateAction.java:327)
at com.hp.hpl.jena.update.UpdateAction.execute(UpdateAction.java:307)
at com.hp.hpl.jena.update.UpdateAction.execute(UpdateAction.java:257)
at org.apache.jena.fuseki.servlets.SPARQL_Update.execute(SPARQL_Update.java:240)
at org.apache.jena.fuseki.servlets.SPARQL_Update.executeForm(SPARQL_Update.java:230)
at org.apache.jena.fuseki.servlets.SPARQL_Update.perform(SPARQL_Update.java:118)
at org.apache.jena.fuseki.servlets.SPARQL_ServletBase.doCommonWorker(SPARQL_ServletBase.java:117)
at org.apache.jena.fuseki.servlets.SPARQL_ServletBase.doCommon(SPARQL_ServletBase.java:67)
at org.apache.jena.fuseki.servlets.SPARQL_Update.doPost(SPARQL_Update.java:84)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:598)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:442)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1033)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:369)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:967)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
at org.eclipse.jetty.server.Server.handle(Server.java:358)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452)
at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:47)
at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:894)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:948)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:851)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:66)
at org.eclipse.jetty.server.nio.BlockingChannelConnector$BlockingChannelEndPoint.run(BlockingChannelConnector.java:293)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538)
at java.lang.Thread.run(Thread.java:722)
00:43:53 INFO Fuseki :: [10654] 500 a fault occurred in a recent unsafe memory access operation in compiled Java code
Regards,
Michael Brunnbauer
--
++ Michael Brunnbauer
++ netEstate GmbH
++ Geisenhausener Straße 11a
++ 81379 München
++ Tel +49 89 32 19 77 80
++ Fax +49 89 32 19 77 89
++ E-Mail brunni@netestate.de
++ http://www.netestate.de/
++
++ Sitz: München, HRB Nr.142452 (Handelsregister B München)
++ USt-IdNr. DE221033342
++ Geschäftsführer: Michael Brunnbauer, Franz Brunnbauer
++ Prokurist: Dipl. Kfm. (Univ.) Markus Hendel
Re: jena-fuseki-0.2.5: java.lang.InternalError during SPARQL update
Posted by Michael Brunnbauer <br...@netestate.de>.
Hello Andy,
On Sun, Jan 27, 2013 at 12:18:09PM +0000, Andy Seaborne wrote:
> >thank you. Can you see from the stack trace if my TDB is still consistent ?
> It should be OK - it's inside a transaction and the update is lost; the
> transaction is aborted.
Good :-)
> How often is this happening? Predictably?
This was the first time and the process doing the updates stopped due to it.
I think I will go back to Java 6 and not do any updates until then. This
server is doing some other stuff and I have not seen signs of buggy hardware.
> BTW - is this machine virtualized or running on real hardware?
real hardware.
Regards,
Michael Brunnbauer
--
++ Michael Brunnbauer
++ netEstate GmbH
++ Geisenhausener Straße 11a
++ 81379 München
++ Tel +49 89 32 19 77 80
++ Fax +49 89 32 19 77 89
++ E-Mail brunni@netestate.de
++ http://www.netestate.de/
++
++ Sitz: München, HRB Nr.142452 (Handelsregister B München)
++ USt-IdNr. DE221033342
++ Geschäftsführer: Michael Brunnbauer, Franz Brunnbauer
++ Prokurist: Dipl. Kfm. (Univ.) Markus Hendel
Re: jena-fuseki-0.2.5: java.lang.InternalError during SPARQL update
Posted by Andy Seaborne <an...@apache.org>.
On 29/01/13 09:37, Michael Brunnbauer wrote:
>
> Hello Andy,
>
> the problem was a bad block on the disk (I should have checked this first).
> It was not automatically corrected as TDB tried to read from it and failed
> (drives only relocate bad blocks on write). I will throw away the TDB on this
> disk as it is most probably inconsistent and use the backup.
Michael - thanks for the update.
The transaction journal entries are CRC'ed but the main index blocks
aren't. Something to consider at a time when the disk format changes (a
major revision change).
Andy
Re: jena-fuseki-0.2.5: java.lang.InternalError during SPARQL update
Posted by Michael Brunnbauer <br...@netestate.de>.
Hello Andy,
the problem was a bad block on the disk (I should have checked this first).
It was not automatically corrected as TDB tried to read from it and failed
(drives only relocate bad blocks on write). I will throw away the TDB on this
disk as it is most probably inconsistent and use the backup.
Regards,
Michael Brunnbauer
On Sun, Jan 27, 2013 at 12:18:09PM +0000, Andy Seaborne wrote:
> On 27/01/13 11:45, Michael Brunnbauer wrote:
> >
> >Hello Andy,
> >
> >thank you. Can you see from the stack trace if my TDB is still consistent ?
>
> It should be OK - it's inside a transaction and the update is lost; the
> transaction is aborted.
>
> The bad memory access is in reading the main database as part of the
> update, not a write to the main DB.
>
> Server restart is a good idea - it may be a heisenbug (pseudo-random
> event - depends on access patterns upto that point).
>
> How often is this happening? Predictably?
>
> If the same error occurred during the commit phase of the transaction,
> it would be a different story. But even then, restart will replay the
> journal and the different pattern of usage may well mean it succeeds.
>
> BTW - is this machine virtualized or running on real hardware?
>
> Andy
>
> >
> >Regards,
> >
> >Michael Brunnbauer
> >
> >On Sun, Jan 27, 2013 at 11:35:28AM +0000, Andy Seaborne wrote:
> >>On 27/01/13 11:21, Michael Brunnbauer wrote:
> >>>
> >>>hi all
> >>>
> >>>the following exception occured during a SPARQL update on
> >>>jena-fuseki-0.2.5
> >>>with jdk-7u11-linux-x64. The system has been doing SPARQL updates without
> >>>problem with jena-fuseki-0.2.5 since October and with jdk-7u11-linux-x64
> >>>since
> >>>Jan 15. I have updated the 32bit glibc yesterday but it should not be
> >>>involved
> >>>as this is a 64bit version of Java. I guess only Oracle can do something
> >>>here ?
> >>
> >>Yes - sorry - nothing Fuseki can do. It is very frustrating when java
> >>goes bad.
> >>
> >>Memory mapped IO is returning errors.
> >>
> >>You could go back an earlier JDK7 or use a JDK6.
> >>
> >>But it might also be that your hardware is decaying or (unlikely but
> >>possible) it's an OS bug that 7u11 happens to provoke.
> >>
> >> Andy
> >>
> >>>
> >>>00:43:16 INFO Fuseki :: [10654] POST
> >>>http://ts.foaf-search.net:3030/crawl/update
> >>>00:43:53 WARN SPARQL_Update$HttpActionUpdate :: Transaction still active
> >>>in endWriter - no commit or abort seen (forced abort)
> >>>00:43:53 WARN Fuseki :: [10654] RC = 500 : a fault
> >>>occurred
> >>>in a recent unsafe memory access operation in compiled Java code
> >>>java.lang.InternalError: a fault occurred in a recent unsafe memory
> >>>access
> >>>operation in compiled Java code
> >>> at
> >>> com.hp.hpl.jena.tdb.base.file.BlockAccessBase.check(BlockAccessBase.java:118)
> >>> at
> >>> com.hp.hpl.jena.tdb.base.file.BlockAccessMapped.read(BlockAccessMapped.java:93)
> >
--
++ Michael Brunnbauer
++ netEstate GmbH
++ Geisenhausener Straße 11a
++ 81379 München
++ Tel +49 89 32 19 77 80
++ Fax +49 89 32 19 77 89
++ E-Mail brunni@netestate.de
++ http://www.netestate.de/
++
++ Sitz: München, HRB Nr.142452 (Handelsregister B München)
++ USt-IdNr. DE221033342
++ Geschäftsführer: Michael Brunnbauer, Franz Brunnbauer
++ Prokurist: Dipl. Kfm. (Univ.) Markus Hendel
Re: jena-fuseki-0.2.5: java.lang.InternalError during SPARQL update
Posted by Andy Seaborne <an...@apache.org>.
On 27/01/13 11:45, Michael Brunnbauer wrote:
>
> Hello Andy,
>
> thank you. Can you see from the stack trace if my TDB is still consistent ?
It should be OK - it's inside a transaction and the update is lost; the
transaction is aborted.
The bad memory access is in reading the main database as part of the
update, not a write to the main DB.
Server restart is a good idea - it may be a heisenbug (pseudo-random
event - depends on access patterns upto that point).
How often is this happening? Predictably?
If the same error occurred during the commit phase of the transaction,
it would be a different story. But even then, restart will replay the
journal and the different pattern of usage may well mean it succeeds.
BTW - is this machine virtualized or running on real hardware?
Andy
>
> Regards,
>
> Michael Brunnbauer
>
> On Sun, Jan 27, 2013 at 11:35:28AM +0000, Andy Seaborne wrote:
>> On 27/01/13 11:21, Michael Brunnbauer wrote:
>>>
>>> hi all
>>>
>>> the following exception occured during a SPARQL update on jena-fuseki-0.2.5
>>> with jdk-7u11-linux-x64. The system has been doing SPARQL updates without
>>> problem with jena-fuseki-0.2.5 since October and with jdk-7u11-linux-x64
>>> since
>>> Jan 15. I have updated the 32bit glibc yesterday but it should not be
>>> involved
>>> as this is a 64bit version of Java. I guess only Oracle can do something
>>> here ?
>>
>> Yes - sorry - nothing Fuseki can do. It is very frustrating when java
>> goes bad.
>>
>> Memory mapped IO is returning errors.
>>
>> You could go back an earlier JDK7 or use a JDK6.
>>
>> But it might also be that your hardware is decaying or (unlikely but
>> possible) it's an OS bug that 7u11 happens to provoke.
>>
>> Andy
>>
>>>
>>> 00:43:16 INFO Fuseki :: [10654] POST
>>> http://ts.foaf-search.net:3030/crawl/update
>>> 00:43:53 WARN SPARQL_Update$HttpActionUpdate :: Transaction still active
>>> in endWriter - no commit or abort seen (forced abort)
>>> 00:43:53 WARN Fuseki :: [10654] RC = 500 : a fault occurred
>>> in a recent unsafe memory access operation in compiled Java code
>>> java.lang.InternalError: a fault occurred in a recent unsafe memory access
>>> operation in compiled Java code
>>> at
>>> com.hp.hpl.jena.tdb.base.file.BlockAccessBase.check(BlockAccessBase.java:118)
>>> at
>>> com.hp.hpl.jena.tdb.base.file.BlockAccessMapped.read(BlockAccessMapped.java:93)
>
Re: jena-fuseki-0.2.5: java.lang.InternalError during SPARQL update
Posted by Michael Brunnbauer <br...@netestate.de>.
Hello Andy,
thank you. Can you see from the stack trace if my TDB is still consistent ?
Regards,
Michael Brunnbauer
On Sun, Jan 27, 2013 at 11:35:28AM +0000, Andy Seaborne wrote:
> On 27/01/13 11:21, Michael Brunnbauer wrote:
> >
> >hi all
> >
> >the following exception occured during a SPARQL update on jena-fuseki-0.2.5
> >with jdk-7u11-linux-x64. The system has been doing SPARQL updates without
> >problem with jena-fuseki-0.2.5 since October and with jdk-7u11-linux-x64
> >since
> >Jan 15. I have updated the 32bit glibc yesterday but it should not be
> >involved
> >as this is a 64bit version of Java. I guess only Oracle can do something
> >here ?
>
> Yes - sorry - nothing Fuseki can do. It is very frustrating when java
> goes bad.
>
> Memory mapped IO is returning errors.
>
> You could go back an earlier JDK7 or use a JDK6.
>
> But it might also be that your hardware is decaying or (unlikely but
> possible) it's an OS bug that 7u11 happens to provoke.
>
> Andy
>
> >
> >00:43:16 INFO Fuseki :: [10654] POST
> >http://ts.foaf-search.net:3030/crawl/update
> >00:43:53 WARN SPARQL_Update$HttpActionUpdate :: Transaction still active
> >in endWriter - no commit or abort seen (forced abort)
> >00:43:53 WARN Fuseki :: [10654] RC = 500 : a fault occurred
> >in a recent unsafe memory access operation in compiled Java code
> >java.lang.InternalError: a fault occurred in a recent unsafe memory access
> >operation in compiled Java code
> > at
> > com.hp.hpl.jena.tdb.base.file.BlockAccessBase.check(BlockAccessBase.java:118)
> > at
> > com.hp.hpl.jena.tdb.base.file.BlockAccessMapped.read(BlockAccessMapped.java:93)
--
++ Michael Brunnbauer
++ netEstate GmbH
++ Geisenhausener Straße 11a
++ 81379 München
++ Tel +49 89 32 19 77 80
++ Fax +49 89 32 19 77 89
++ E-Mail brunni@netestate.de
++ http://www.netestate.de/
++
++ Sitz: München, HRB Nr.142452 (Handelsregister B München)
++ USt-IdNr. DE221033342
++ Geschäftsführer: Michael Brunnbauer, Franz Brunnbauer
++ Prokurist: Dipl. Kfm. (Univ.) Markus Hendel
Re: jena-fuseki-0.2.5: java.lang.InternalError during SPARQL update
Posted by Andy Seaborne <an...@apache.org>.
On 27/01/13 11:21, Michael Brunnbauer wrote:
>
> hi all
>
> the following exception occured during a SPARQL update on jena-fuseki-0.2.5
> with jdk-7u11-linux-x64. The system has been doing SPARQL updates without
> problem with jena-fuseki-0.2.5 since October and with jdk-7u11-linux-x64 since
> Jan 15. I have updated the 32bit glibc yesterday but it should not be involved
> as this is a 64bit version of Java. I guess only Oracle can do something here ?
Yes - sorry - nothing Fuseki can do. It is very frustrating when java
goes bad.
Memory mapped IO is returning errors.
You could go back an earlier JDK7 or use a JDK6.
But it might also be that your hardware is decaying or (unlikely but
possible) it's an OS bug that 7u11 happens to provoke.
Andy
>
> 00:43:16 INFO Fuseki :: [10654] POST http://ts.foaf-search.net:3030/crawl/update
> 00:43:53 WARN SPARQL_Update$HttpActionUpdate :: Transaction still active in endWriter - no commit or abort seen (forced abort)
> 00:43:53 WARN Fuseki :: [10654] RC = 500 : a fault occurred in a recent unsafe memory access operation in compiled Java code
> java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
> at com.hp.hpl.jena.tdb.base.file.BlockAccessBase.check(BlockAccessBase.java:118)
> at com.hp.hpl.jena.tdb.base.file.BlockAccessMapped.read(BlockAccessMapped.java:93)