You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by Bill Roberts <bi...@swirrl.com> on 2012/03/29 10:48:24 UTC

rebuild TDB indexes?

I had a problem while loading a file to Fuseki via the graph protocol, where the load was interrupted part-way through.  It seems to have resulted in the indexes being corrupted.  If I try to access the relevant graph, Fuseki throws an exception when trying a SELECT:

09:30:40 ERROR BindingTDB           :: get1(?o)
java.util.NoSuchElementException
	at org.openjena.riot.tokens.TokenizerText.next(TokenizerText.java:95)
	at com.hp.hpl.jena.tdb.nodetable.NodecSSE.decode(NodecSSE.java:91)
	at com.hp.hpl.jena.tdb.lib.NodeLib.decode(NodeLib.java:89)
	at com.hp.hpl.jena.tdb.lib.NodeLib.fetchDecode(NodeLib.java:71)
	at com.hp.hpl.jena.tdb.nodetable.NodeTableNative.readNodeByNodeId(NodeTableNative.java:158)
	at com.hp.hpl.jena.tdb.nodetable.NodeTableNative._retrieveNodeByNodeId(NodeTableNative.java:85)
	at com.hp.hpl.jena.tdb.nodetable.NodeTableNative.getNodeForNodeId(NodeTableNative.java:58)
	at com.hp.hpl.jena.tdb.nodetable.NodeTableCache._retrieveNodeByNodeId(NodeTableCache.java:87)
	at com.hp.hpl.jena.tdb.nodetable.NodeTableCache.getNodeForNodeId(NodeTableCache.java:58)
	at com.hp.hpl.jena.tdb.nodetable.NodeTableWrapper.getNodeForNodeId(NodeTableWrapper.java:44)
	at com.hp.hpl.jena.tdb.nodetable.NodeTableInline.getNodeForNodeId(NodeTableInline.java:55)
	at com.hp.hpl.jena.tdb.solver.BindingTDB.get1(BindingTDB.java:96)
	at com.hp.hpl.jena.sparql.engine.binding.BindingBase.get(BindingBase.java:126)
	at com.hp.hpl.jena.sparql.engine.binding.BindingProjectBase.get1(BindingProjectBase.java:54)
	at com.hp.hpl.jena.sparql.engine.binding.BindingBase.get(BindingBase.java:126)
	at com.hp.hpl.jena.sparql.core.ResultBinding._get(ResultBinding.java:44)
	at com.hp.hpl.jena.sparql.core.QuerySolutionBase.get(QuerySolutionBase.java:20)
	at com.hp.hpl.jena.sparql.resultset.ResultSetApply.apply(ResultSetApply.java:35)
	at com.hp.hpl.jena.sparql.resultset.JSONOutput.format(JSONOutput.java:23)
	at com.hp.hpl.jena.query.ResultSetFormatter.outputAsJSON(ResultSetFormatter.java:584)
	at org.openjena.fuseki.servlets.ResponseResultSet$2.output(ResponseResultSet.java:121)
	at org.openjena.fuseki.servlets.ResponseResultSet.output(ResponseResultSet.java:215)
	at org.openjena.fuseki.servlets.ResponseResultSet.jsonOutput(ResponseResultSet.java:263)
	at org.openjena.fuseki.servlets.ResponseResultSet.doResponseResultSet(ResponseResultSet.java:116)
	at org.openjena.fuseki.servlets.SPARQL_Query.sendResults(SPARQL_Query.java:276)
	at org.openjena.fuseki.servlets.SPARQL_Query.execute(SPARQL_Query.java:212)
	at org.openjena.fuseki.servlets.SPARQL_Query.executeWithParameter(SPARQL_Query.java:177)
	at org.openjena.fuseki.servlets.SPARQL_Query.perform(SPARQL_Query.java:97)
	at org.openjena.fuseki.servlets.SPARQL_ServletBase.doCommon(SPARQL_ServletBase.java:95)
	at org.openjena.fuseki.servlets.SPARQL_Query.doGet(SPARQL_Query.java:78)

Other parts of the database seem ok.

Is this likely to be an index issue?
Is there a way to rebuild the indexes without reloading all of the data?
Any tips for futher diagnosis?

My fuseki version is a few months old - presumably would be a good idea to update to most recent snapshot?

Thanks

Bill

Re: rebuild TDB indexes?

Posted by Bill Roberts <bi...@swirrl.com>.
Thanks Paolo.  I will certainly upgrade to the release version, in any case.


On 29 Mar 2012, at 11:01, Paolo Castagna wrote:

> Hi Bill,
> I do not have all the answers to your questions...
> 
> Bill Roberts wrote:
> [...]
>> Any tips for futher diagnosis?
> 
> One thing I always wanted when I found myself in your situation is a
> "verifier" tool I can run against an index which tells me if an index
> is effectively corrupted and if so which part is corrupted.
> 
> The idea would be also... can it be repaired?
> 
> My first attempt at this is just very quick & dirty and nothing which
> can be seriously used, you can find it here:
> https://github.com/castagna/tdbloader4/blob/f5363fa49d16a04a362898c1a5084ade620ee81b/src/test/java/dev/TDBVerifier.java
> I guess if there is demand for it, it could be improved.
> 
> However, Andy put a lot of effort to make sure such a tool is not
> necessary! :-)
> 
>> My fuseki version is a few months old - presumably would be a good idea to update to most recent snapshot?
> 
> Fuseki has now been released, here:
> http://www.apache.org/dist/incubator/jena/jena-fuseki-0.2.1-incubating/
> 
> Paolo

Re: rebuild TDB indexes?

Posted by Paolo Castagna <ca...@googlemail.com>.
Hi Bill,
I do not have all the answers to your questions...

Bill Roberts wrote:
[...]
> Any tips for futher diagnosis?

One thing I always wanted when I found myself in your situation is a
"verifier" tool I can run against an index which tells me if an index
is effectively corrupted and if so which part is corrupted.

The idea would be also... can it be repaired?

My first attempt at this is just very quick & dirty and nothing which
can be seriously used, you can find it here:
https://github.com/castagna/tdbloader4/blob/f5363fa49d16a04a362898c1a5084ade620ee81b/src/test/java/dev/TDBVerifier.java
I guess if there is demand for it, it could be improved.

However, Andy put a lot of effort to make sure such a tool is not
necessary! :-)

> My fuseki version is a few months old - presumably would be a good idea to update to most recent snapshot?

Fuseki has now been released, here:
http://www.apache.org/dist/incubator/jena/jena-fuseki-0.2.1-incubating/

Paolo

Re: rebuild TDB indexes?

Posted by Andy Seaborne <an...@apache.org>.
On 29/03/12 15:45, Andy Seaborne wrote:
> Fixed in development 0.9.1-SNAPSHOT.

Confusing :-)

The fix is in TDB with some support code in ARQ.

Fuseki 0.2.2-SNAPSHOT has all the right stuff in it.

	Andy

Re: rebuild TDB indexes?

Posted by Andy Seaborne <an...@apache.org>.
Hi Bill,

On 29/03/12 09:48, Bill Roberts wrote:
> I had a problem while loading a file to Fuseki via the graph protocol, where the load was interrupted part-way through.  It seems to have resulted in the indexes being corrupted.  If I try to access the relevant graph, Fuseki throws an exception when trying a SELECT:
>
> 09:30:40 ERROR BindingTDB           :: get1(?o)
> java.util.NoSuchElementException
> 	at org.openjena.riot.tokens.TokenizerText.next(TokenizerText.java:95)
> 	at com.hp.hpl.jena.tdb.nodetable.NodecSSE.decode(NodecSSE.java:91)
> 	at com.hp.hpl.jena.tdb.lib.NodeLib.decode(NodeLib.java:89)
> 	at com.hp.hpl.jena.tdb.lib.NodeLib.fetchDecode(NodeLib.java:71)
> 	at com.hp.hpl.jena.tdb.nodetable.NodeTableNative.readNodeByNodeId(NodeTableNative.java:158)
> 	at com.hp.hpl.jena.tdb.nodetable.NodeTableNative._retrieveNodeByNodeId(NodeTableNative.java:85)
> 	at com.hp.hpl.jena.tdb.nodetable.NodeTableNative.getNodeForNodeId(NodeTableNative.java:58)
> 	at com.hp.hpl.jena.tdb.nodetable.NodeTableCache._retrieveNodeByNodeId(NodeTableCache.java:87)
> 	at com.hp.hpl.jena.tdb.nodetable.NodeTableCache.getNodeForNodeId(NodeTableCache.java:58)
> 	at com.hp.hpl.jena.tdb.nodetable.NodeTableWrapper.getNodeForNodeId(NodeTableWrapper.java:44)
> 	at com.hp.hpl.jena.tdb.nodetable.NodeTableInline.getNodeForNodeId(NodeTableInline.java:55)
> 	at com.hp.hpl.jena.tdb.solver.BindingTDB.get1(BindingTDB.java:96)
> 	at com.hp.hpl.jena.sparql.engine.binding.BindingBase.get(BindingBase.java:126)
> 	at com.hp.hpl.jena.sparql.engine.binding.BindingProjectBase.get1(BindingProjectBase.java:54)
> 	at com.hp.hpl.jena.sparql.engine.binding.BindingBase.get(BindingBase.java:126)
> 	at com.hp.hpl.jena.sparql.core.ResultBinding._get(ResultBinding.java:44)
> 	at com.hp.hpl.jena.sparql.core.QuerySolutionBase.get(QuerySolutionBase.java:20)
> 	at com.hp.hpl.jena.sparql.resultset.ResultSetApply.apply(ResultSetApply.java:35)
> 	at com.hp.hpl.jena.sparql.resultset.JSONOutput.format(JSONOutput.java:23)
> 	at com.hp.hpl.jena.query.ResultSetFormatter.outputAsJSON(ResultSetFormatter.java:584)
> 	at org.openjena.fuseki.servlets.ResponseResultSet$2.output(ResponseResultSet.java:121)
> 	at org.openjena.fuseki.servlets.ResponseResultSet.output(ResponseResultSet.java:215)
> 	at org.openjena.fuseki.servlets.ResponseResultSet.jsonOutput(ResponseResultSet.java:263)
> 	at org.openjena.fuseki.servlets.ResponseResultSet.doResponseResultSet(ResponseResultSet.java:116)
> 	at org.openjena.fuseki.servlets.SPARQL_Query.sendResults(SPARQL_Query.java:276)
> 	at org.openjena.fuseki.servlets.SPARQL_Query.execute(SPARQL_Query.java:212)
> 	at org.openjena.fuseki.servlets.SPARQL_Query.executeWithParameter(SPARQL_Query.java:177)
> 	at org.openjena.fuseki.servlets.SPARQL_Query.perform(SPARQL_Query.java:97)
> 	at org.openjena.fuseki.servlets.SPARQL_ServletBase.doCommon(SPARQL_ServletBase.java:95)
> 	at org.openjena.fuseki.servlets.SPARQL_Query.doGet(SPARQL_Query.java:78)
>
> Other parts of the database seem ok.
>
> Is this likely to be an index issue?

It's a problem with the node table, specifically the data for the nodes 
themselves.

Two possibilities occur to me:

1/ The original data was loaded awhile ago when there was one of the 
transaction bugs

2/ We discovered and fixed recently a problem that occurs when bad 
Unicode (e.g. when a codepoint is one half of a surrogate pair without 
the other half) made a mess of the node table.  Fixed in development 
0.9.1-SNAPSHOT.

This looks like it could be the same thing.

Could you let me have a copy of the nodes.dat file please?

> Is there a way to rebuild the indexes without reloading all of the data?
> Any tips for futher diagnosis?
>
> My fuseki version is a few months old - presumably would be a good idea to update to most recent snapshot?

"fuseki-server --version" tells you (and me!) which version it is.

A good idea to update but the node table is broken, so just changing 
software will not fix the problem.

	Andy

>
> Thanks
>
> Bill
>
>
>
>