You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by "Dr. André Lanka" <ma...@dr-lanka.de> on 2013/04/10 16:23:13 UTC
How to completely detach a ResultSet from a TDB model
Dear Jena community,
we didn't find a way (in the original Jena API 2.10) to completely
detach a result set from the underlying model. It's possible to detach
it from the QueryExecution by using ResultSetFactory.copyResults but the
bindings still point to the TDB backed model.
By this we encountered some issues as we store results sets for a long
time and the read transaction that created the results stays alive (even
if we calls commit or end). Having this read transaction opened leads to
problems when you have many writers after that (stack usage, huge
journal files, ...).
We finally found a way to convert TDB backed result set to a completely
detached ResultSetRewindable. If you find this useful, it's attached below.
Cheers
André
package com.hojoki.core.jena;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import com.hp.hpl.jena.graph.Graph;
import com.hp.hpl.jena.query.QuerySolution;
import com.hp.hpl.jena.query.ResultSet;
import com.hp.hpl.jena.query.ResultSetRewindable;
import com.hp.hpl.jena.rdf.model.Model;
import com.hp.hpl.jena.sparql.core.ResultBinding;
import com.hp.hpl.jena.sparql.core.Var;
import com.hp.hpl.jena.sparql.engine.binding.Binding;
import com.hp.hpl.jena.sparql.engine.binding.BindingFactory;
import com.hp.hpl.jena.tdb.store.GraphTDB;
public class ResultsetUtil {
public static boolean isTDBBacked(ResultSet resultSet) {
Model model = resultSet.getResourceModel();
if (model != null) {
Graph g = model.getGraph();
return g instanceof GraphTDB;
}
return false;
}
public static ResultSetRewindable
createDetachedRewindableResultSet(ResultSet resultSet) {
if(resultSet instanceof HojokiResultSet)
return (HojokiResultSet) resultSet;
return isTDBBacked(resultSet) ?
detachAndConvertToRewindable(resultSet) : convertToRewindable(resultSet);
}
private static ResultSetRewindable convertToRewindable(ResultSet
resultSet) {
return toHojokiResultSet(resultSet, false);
}
private static ResultSetRewindable
detachAndConvertToRewindable(ResultSet resultSet) {
return toHojokiResultSet(resultSet, true);
}
public static ResultSetRewindable toHojokiResultSet(ResultSet set,
boolean detachBindings) {
List<String> resultVars = set.getResultVars();
final List<Binding> bindings = new ArrayList<Binding>();
while (set.hasNext()) {
Binding binding = set.nextBinding();
bindings.add(detachBindings ? detachedBinding(binding) : binding);
}
return new HojokiResultSet(resultVars, null, bindings);
}
protected static Binding detachedBinding(Binding binding) {
Iterator<Var> varsIt = binding.vars();
Binding initial = BindingFactory.binding();
while (varsIt.hasNext()) {
Var var = varsIt.next();
initial = BindingFactory.binding(initial, var, binding.get(var));
}
return initial;
}
}
class HojokiResultSet implements ResultSetRewindable {
private List<String> resultVars;
private int rowNumber;
private Model model;
private final List<Binding> bindings;
private Iterator<Binding> iter;
public HojokiResultSet(List<String> resultVars, Model m,
List<Binding> bindings) {
this.bindings = bindings;
this.resultVars = resultVars;
model = m;
reset();
}
/**
* @throws UnsupportedOperationException
* Always thrown.
*/
@Override
public void remove() throws java.lang.UnsupportedOperationException {
throw new UnsupportedOperationException(this.getClass().getName() +
".remove");
}
/**
* Is there another possibility?
*/
@Override
public boolean hasNext() {
return iter.hasNext();
}
@Override
public Binding nextBinding() {
Binding binding = iter.next();
if (binding != null)
rowNumber++;
return binding;
}
/**
* Moves onto the next result possibility. The returned object is
actual the
* binding for this result.
*/
@Override
public QuerySolution nextSolution() {
return new ResultBinding(model, nextBinding());
}
/** Moves onto the next result possibility. */
@Override
public QuerySolution next() {
return nextSolution();
}
/**
* Return the "row number" - a count of the number of possibilities
returned
* so far. Remains valid (as the total number of possibilities) after the
* iterator ends.
*/
@Override
public int getRowNumber() {
return rowNumber;
}
/**
* Get the variable names for the projection
*/
@Override
public List<String> getResultVars() {
return resultVars;
}
public Model getModel() {
return model;
}
@Override
public Model getResourceModel() {
return getModel();
}
@Override
public void reset() {
iter = bindings.iterator();
rowNumber = 0;
}
@Override
public int size() {
return bindings.size();
}
}
--
Dr. André Lanka * 0178 / 134 44 47 * http://dr-lanka.de
Re: How to completely detach a ResultSet from a TDB model
Posted by Andy Seaborne <an...@apache.org>.
On 12/04/13 10:19, "Dr. André Lanka" wrote:
> Hello,
>
> On 11.04.2013 14:13, Andy Seaborne wrote:
>> On 10/04/13 15:23, "Dr. André Lanka" wrote:
>> To break the connection,
>>
>> 1/ use ResultSet.nextBinding() which returns the low level, unconnected
>> graph Nodes.
>
> It doesn't help. I tried to "copy" the bindings into a new resultSet
> using ResultSet.nextBinding(). Then I deleted the store (to detect
> really all connections) and still got a NPE (which is logged and
> swallowed in BindingTDB [StackTrace below]).
>
> So I think recreating the bindings seems to be necessary in our case.
Ah - yes, I remember now - it uses a delayed node resolution. When you
touch the node entry, it goes and gets the node string. Internally,
nodes don't always get used (projected away, count(*) aggregation) so
this avoids work that will not be used.
It is possible that the top level binding ought to force materialization
as Bindigns are yielded. Added to QueryEngineTDB.eval.
The case of only using some of the variables in a binding in results
processing is slightly worse although nodes are highly cached so it
shouldn't be too bad. Let us know if it does not work for you.
Andy
Re: How to completely detach a ResultSet from a TDB model
Posted by "Dr. André Lanka" <ma...@dr-lanka.de>.
Hello,
On 11.04.2013 14:13, Andy Seaborne wrote:
> On 10/04/13 15:23, "Dr. André Lanka" wrote:
> To break the connection,
>
> 1/ use ResultSet.nextBinding() which returns the low level, unconnected
> graph Nodes.
It doesn't help. I tried to "copy" the bindings into a new resultSet
using ResultSet.nextBinding(). Then I deleted the store (to detect
really all connections) and still got a NPE (which is logged and
swallowed in BindingTDB [StackTrace below]).
So I think recreating the bindings seems to be necessary in our case.
> 2/ Much like your code - you could use ResultSetStream itself, create a
> QueryIterator of the Bindings passing in a memory model to the
> constructor of ResultSetStream.
Simple passing seems not to be enough (because of BindingTDB).
> I'll change ResultSetStream to take a Iterator<binding> in place of
> QueryIterator (which implements Iterator<binding>) to make it easier.
>
>
> 3/ Processing style - consume the ResultSet inside the transaction and
> pass a different datastructure passed end(). You'll need to use up the
> query results anyway.
We need to cache resultSets for performance reasons without leaving the
read transaction open. However, the code I sent creates another
datastructure, so we did essentially 3/ :-)
Best
André
get1(?s)
java.lang.NullPointerException
at
com.hp.hpl.jena.tdb.nodetable.NodeTableCache.toString(NodeTableCache.java:258)
at java.lang.String.valueOf(String.java:2826)
at java.lang.StringBuilder.append(StringBuilder.java:115)
at
com.hp.hpl.jena.tdb.nodetable.NodeTableCache._retrieveNodeByNodeId(NodeTableCache.java:101)
at
com.hp.hpl.jena.tdb.nodetable.NodeTableCache.getNodeForNodeId(NodeTableCache.java:74)
at
com.hp.hpl.jena.tdb.nodetable.NodeTableWrapper.getNodeForNodeId(NodeTableWrapper.java:55)
at
com.hp.hpl.jena.tdb.nodetable.NodeTableInline.getNodeForNodeId(NodeTableInline.java:67)
at
com.hp.hpl.jena.tdb.transaction.NodeTableTrans.getNodeForNodeId(NodeTableTrans.java:111)
at
com.hp.hpl.jena.tdb.nodetable.NodeTableWrapper.getNodeForNodeId(NodeTableWrapper.java:55)
at
com.hp.hpl.jena.tdb.nodetable.NodeTableInline.getNodeForNodeId(NodeTableInline.java:67)
at com.hp.hpl.jena.tdb.solver.BindingTDB.get1(BindingTDB.java:123)
at
com.hp.hpl.jena.sparql.engine.binding.BindingBase.get(BindingBase.java:123)
at
com.hp.hpl.jena.sparql.engine.binding.BindingProjectBase.get1(BindingProjectBase.java:65)
at
com.hp.hpl.jena.sparql.engine.binding.BindingBase.get(BindingBase.java:123)
at com.hp.hpl.jena.sparql.core.ResultBinding._get(ResultBinding.java:57)
at
com.hp.hpl.jena.sparql.core.QuerySolutionBase.get(QuerySolutionBase.java:33)
--
Dr. André Lanka * 0178 / 134 44 47 * http://dr-lanka.de
Re: How to completely detach a ResultSet from a TDB model
Posted by Andy Seaborne <an...@apache.org>.
On 10/04/13 15:23, "Dr. André Lanka" wrote:
> Dear Jena community,
>
> we didn't find a way (in the original Jena API 2.10) to completely
> detach a result set from the underlying model. It's possible to detach
> it from the QueryExecution by using ResultSetFactory.copyResults but the
> bindings still point to the TDB backed model.
>
> By this we encountered some issues as we store results sets for a long
> time and the read transaction that created the results stays alive (even
> if we calls commit or end). Having this read transaction opened leads to
> problems when you have many writers after that (stack usage, huge
> journal files, ...).
>
> We finally found a way to convert TDB backed result set to a completely
> detached ResultSetRewindable. If you find this useful, it's attached below.
>
> Cheers
> André
Good point - it's a bit of a tradeoff. The result sets API supports
mixing SPARQL with other Jena API calls:
Resource r = row.getResource("var") ;
r.listProperties() ;
to do that "r" needs to know the model.
To break the connection,
1/ use ResultSet.nextBinding() which returns the low level, unconnected
graph Nodes.
2/ Much like your code - you could use ResultSetStream itself, create a
QueryIterator of the Bindings passing in a memory model to the
constructor of ResultSetStream.
I'll change ResultSetStream to take a Iterator<binding> in place of
QueryIterator (which implements Iterator<binding>) to make it easier.
3/ Processing style - consume the ResultSet inside the transaction and
pass a different datastructure passed end(). You'll need to use up the
query results anyway.
Andy