You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jena.apache.org by Claude Warren <cl...@xenei.com> on 2014/05/26 16:48:38 UTC
TDB transaction weirdness.
I have code that uses Jena-Libs 2.11.1 and executes the following block:
(edited for brevity)
dataset.begin( ReadWrite.WRITE);
try {
Model model = dataset.getDefaultModel();
Property[] props = ( create an array of properties in Model)
String dcsId = "someStringValue";
int cnt = 0;
..... (get an iterator on a set of rows from a db table)
while (rIter.hasNext())
{
cnt++;
Row r = rIter.next();
Resource eventR = model.createResource( makeURI( eventUri,
r.getColumn(dcsId).toString()), typeURI);
for (int i=0;i<cols.size();i++)
{
Object o = r.getColumn(i);
if (props[i] != null && o != null)
{
String val = o.toString();
if (val.length()>0)
{
eventR.addProperty( props[i], val); // <-- line 134 as noted
in call stack below
}
}
}
// commit every 500 entries
if (cnt % 500 == 0) {
System.out.println( cnt );
dataset.commit();
dataset.begin( ReadWrite.WRITE );
}
}
}
}
}
finally {
dataset.commit();
}
The table I am reading from contains several 10's of thousands of rows. I
am basically converting from relational table to RDF graph.
However, after the first 500 subjects are committed (5000 +/- triples) I
run into the following problem while adding the 6th property to subject
502.
DEBUG [main] (MySQLTableDef.java:261) - Query String: SELECT ... <--
Initial debugging statement
500 <-- output from commit block in code above
ERROR [main] (Log.java:94) - Not active: 1
ERROR [main] (Log.java:94) - **** Not active: 1
Now, I know this is a transaction error. But I think I have a transaction,
as I restart one right after the commit after the 500th record. Log is
made in the BlockMgrJournal.checkActive() and the call stack appears as
follows:
BlockMgrJournal.checkActive() line: 306
BlockMgrJournal._promote(Block) line: 220
BlockMgrJournal.promote(Block) line: 215
BPTreeRecordsMgr(PageBlockMgr<T>).promote(Page) line: 110
BPTreeRecords.promote() line: 119
BPTreeRecords.internalInsert(Record) line: 131
BPTreeNode.internalInsert(Record) line: 468
BPTreeNode.internalInsert(Record) line: 468
BPTreeNode.insert(BPTreeNode, Record) line: 212
BPlusTree.addAndReturnOld(Record) line: 328
BPlusTree.add(Record) line: 320
TupleIndexRecord.performAdd(Tuple<NodeId>) line: 60
TupleIndexRecord(TupleIndexBase).add(Tuple<NodeId>) line: 64
TupleTable.add(Tuple<NodeId>) line: 96
NodeTupleTableConcrete.addRow(Node...) line: 87
TripleTable.add(Node, Node, Node) line: 58
DatasetGraphTDB.addToDftGraph(Node, Node, Node) line: 100
DatasetGraphTDB(DatasetGraphTriplesQuads).add(Node, Node, Node, Node) line:
47
GraphTDB(GraphView).performAdd(Triple) line: 140
GraphTDB.performAdd(Triple) line: 87
GraphTDB(GraphBase).add(Triple) line: 202
ModelCom.add(Resource, Property, RDFNode) line: 1159
ModelCom.add(Resource, Property, String, String, boolean) line: 161
ModelCom.add(Resource, Property, String) line: 149
ResourceImpl.addProperty(Property, String) line: 241
JenaSink.updateTable(UpdateBlock) line: 134
Any help fixing this problem would be appreciated.
Claude
--
I like: Like Like - The likeliest place on the web<http://like-like.xenei.com>
LinkedIn: http://www.linkedin.com/in/claudewarren
Re: TDB transaction weirdness.
Posted by Andy Seaborne <an...@apache.org>.
Not that I know of - the only mac-oddity I can recall is the fact that
"du" counts file sizes, and not space used, which on sparse files (and
TDB files are sparse files to start with) means the filespace looks huge
when in fact it is not.
But I don't use a mac.
Andy
On 30/05/14 20:41, Claude Warren wrote:
> I tried to get an example but it does not fail on my linux box. Are there
> any known problem with Mac? I'll try it on the Mac next week when I get
> back to work.
>
> Claude
>
>
> On Mon, May 26, 2014 at 7:22 PM, Andy Seaborne <an...@apache.org> wrote:
>
>> On 26/05/14 18:57, Claude Warren wrote:
>>
>>> I was looking at how I could get a minimal set but the data is business
>>> confidential and I am not certian that changing it would produce the same
>>> issue. I will see what I can do about a minimal set -- though I expect
>>> that will still be large.
>>>
>>> Claude
>>>
>>
>> I would be surprised if there were no effects except at scale although
>> where it's going to show first is not clear to me.
>>
>> If you want to try something amusing possibly with side effects ... but
>> SPARQL query on the model would by the general engine in Node space (it
>> can't see that the graoh is TDB-backed).
>>
>> Graph g = GraphView.createDefaultGraph(ds.asDatasetGraph()) ;
>> model = ModelFactory.createModelForGraph(g) ;
>>
>> which inverts the stack to put the view over the top of the switching
>> datasetgraph.
>>
>> Getting the named model for the union may work but uses more RAM to get
>> set semantics.
>>
>> Andy
>>
>>
>>
>>>
>>> On Mon, May 26, 2014 at 5:55 PM, Andy Seaborne <an...@apache.org> wrote:
>>>
>>> On 26/05/14 15:54, Claude Warren wrote:
>>>>
>>>> Look like I needed to add a model=dataset.getDefaultModel() statement
>>>>> to
>>>>> the commit block os that it reads:
>>>>>
>>>>>
>>>>> if (cnt % 500 == 0) {
>>>>>
>>>>> System.out.println( cnt );
>>>>>
>>>>> dataset.commit();
>>>>>
>>>>> dataset.begin( ReadWrite.WRITE );
>>>>>
>>>>> model=dataset.getDefaultModel();
>>>>>
>>>>> }
>>>>>
>>>>> Why? I would expect that the model would track the transaction set on
>>>>> the
>>>>> dataset.
>>>>>
>>>>>
>>>> Should be possible but it's hard to balance compatibility
>>>> (non-transactional use) with being about to mix TDB graphs into general
>>>> purpose dataset and also get efficiency (get SPARQL processing to be
>>>> handled by TDB, code that calls .getDefaultModel and not assigning to a
>>>> local variable).
>>>>
>>>> Come Jena3, this could be improved with changes to the application
>>>> contract for the more corner cases (single thread overlapping
>>>> transactions
>>>> for example). Maybe drop the direct SPARQL execution of TDB graphs in
>>>> general purpose, mixed datasets and resort to general SPARQL execution.
>>>>
>>>> It's not just Models - Resources have an implicit graph.
>>>>
>>>> What would be really helpful is to have an example test case that
>>>> illustrated some aspect of the problem which was compact and standalone.
>>>>
>>>> Andy
>>>>
>>>>
>>>>
>>>>> Claude
>>>>>
>>>>>
>>>>> On Mon, May 26, 2014 at 3:48 PM, Claude Warren <cl...@xenei.com>
>>>>> wrote:
>>>>>
>>>>> I have code that uses Jena-Libs 2.11.1 and executes the following
>>>>> block:
>>>>>
>>>>>> (edited for brevity)
>>>>>>
>>>>>> dataset.begin( ReadWrite.WRITE);
>>>>>>
>>>>>> try {
>>>>>>
>>>>>> Model model = dataset.getDefaultModel();
>>>>>>
>>>>>> Property[] props = ( create an array of properties in Model)
>>>>>>
>>>>>> String dcsId = "someStringValue";
>>>>>>
>>>>>> int cnt = 0;
>>>>>> ..... (get an iterator on a set of rows from a db table)
>>>>>>
>>>>>> while (rIter.hasNext())
>>>>>>
>>>>>> {
>>>>>>
>>>>>> cnt++;
>>>>>>
>>>>>> Row r = rIter.next();
>>>>>>
>>>>>> Resource eventR = model.createResource( makeURI( eventUri,
>>>>>> r.getColumn(dcsId).toString()), typeURI);
>>>>>>
>>>>>> for (int i=0;i<cols.size();i++)
>>>>>>
>>>>>> {
>>>>>>
>>>>>> Object o = r.getColumn(i);
>>>>>>
>>>>>> if (props[i] != null && o != null)
>>>>>>
>>>>>> {
>>>>>>
>>>>>> String val = o.toString();
>>>>>>
>>>>>> if (val.length()>0)
>>>>>>
>>>>>> {
>>>>>>
>>>>>> eventR.addProperty( props[i], val); // <-- line 134 as
>>>>>> noted
>>>>>> in call stack below
>>>>>>
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>> // commit every 500 entries
>>>>>>
>>>>>> if (cnt % 500 == 0) {
>>>>>>
>>>>>> System.out.println( cnt );
>>>>>>
>>>>>> dataset.commit();
>>>>>>
>>>>>> dataset.begin( ReadWrite.WRITE );
>>>>>>
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>> finally {
>>>>>>
>>>>>> dataset.commit();
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> The table I am reading from contains several 10's of thousands of rows.
>>>>>> I
>>>>>> am basically converting from relational table to RDF graph.
>>>>>>
>>>>>> However, after the first 500 subjects are committed (5000 +/- triples)
>>>>>> I
>>>>>> run into the following problem while adding the 6th property to subject
>>>>>> 502.
>>>>>>
>>>>>>
>>>>>> DEBUG [main] (MySQLTableDef.java:261) - Query String: SELECT ... <--
>>>>>> Initial debugging statement
>>>>>>
>>>>>> 500 <-- output from commit block in code above
>>>>>>
>>>>>> ERROR [main] (Log.java:94) - Not active: 1
>>>>>>
>>>>>> ERROR [main] (Log.java:94) - **** Not active: 1
>>>>>>
>>>>>>
>>>>>> Now, I know this is a transaction error. But I think I have a
>>>>>> transaction, as I restart one right after the commit after the 500th
>>>>>> record. Log is made in the BlockMgrJournal.checkActive() and the call
>>>>>> stack appears as follows:
>>>>>> BlockMgrJournal.checkActive() line: 306
>>>>>> BlockMgrJournal._promote(Block) line: 220
>>>>>> BlockMgrJournal.promote(Block) line: 215
>>>>>> BPTreeRecordsMgr(PageBlockMgr<T>).promote(Page) line: 110
>>>>>> BPTreeRecords.promote() line: 119
>>>>>> BPTreeRecords.internalInsert(Record) line: 131
>>>>>> BPTreeNode.internalInsert(Record) line: 468
>>>>>> BPTreeNode.internalInsert(Record) line: 468
>>>>>> BPTreeNode.insert(BPTreeNode, Record) line: 212
>>>>>> BPlusTree.addAndReturnOld(Record) line: 328
>>>>>> BPlusTree.add(Record) line: 320
>>>>>> TupleIndexRecord.performAdd(Tuple<NodeId>) line: 60
>>>>>> TupleIndexRecord(TupleIndexBase).add(Tuple<NodeId>) line: 64
>>>>>> TupleTable.add(Tuple<NodeId>) line: 96
>>>>>> NodeTupleTableConcrete.addRow(Node...) line: 87
>>>>>> TripleTable.add(Node, Node, Node) line: 58
>>>>>> DatasetGraphTDB.addToDftGraph(Node, Node, Node) line: 100
>>>>>> DatasetGraphTDB(DatasetGraphTriplesQuads).add(Node, Node, Node, Node)
>>>>>> line: 47
>>>>>> GraphTDB(GraphView).performAdd(Triple) line: 140
>>>>>> GraphTDB.performAdd(Triple) line: 87
>>>>>> GraphTDB(GraphBase).add(Triple) line: 202
>>>>>> ModelCom.add(Resource, Property, RDFNode) line: 1159
>>>>>> ModelCom.add(Resource, Property, String, String, boolean) line: 161
>>>>>> ModelCom.add(Resource, Property, String) line: 149
>>>>>> ResourceImpl.addProperty(Property, String) line: 241
>>>>>> JenaSink.updateTable(UpdateBlock) line: 134
>>>>>>
>>>>>>
>>>>>> Any help fixing this problem would be appreciated.
>>>>>>
>>>>>> Claude
>>>>>> --
>>>>>> I like: Like Like - The likeliest place on the web<
>>>>>> http://like-like.xenei.com>
>>>>>> LinkedIn: http://www.linkedin.com/in/claudewarren
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
>
Re: TDB transaction weirdness.
Posted by Claude Warren <cl...@xenei.com>.
I tried to get an example but it does not fail on my linux box. Are there
any known problem with Mac? I'll try it on the Mac next week when I get
back to work.
Claude
On Mon, May 26, 2014 at 7:22 PM, Andy Seaborne <an...@apache.org> wrote:
> On 26/05/14 18:57, Claude Warren wrote:
>
>> I was looking at how I could get a minimal set but the data is business
>> confidential and I am not certian that changing it would produce the same
>> issue. I will see what I can do about a minimal set -- though I expect
>> that will still be large.
>>
>> Claude
>>
>
> I would be surprised if there were no effects except at scale although
> where it's going to show first is not clear to me.
>
> If you want to try something amusing possibly with side effects ... but
> SPARQL query on the model would by the general engine in Node space (it
> can't see that the graoh is TDB-backed).
>
> Graph g = GraphView.createDefaultGraph(ds.asDatasetGraph()) ;
> model = ModelFactory.createModelForGraph(g) ;
>
> which inverts the stack to put the view over the top of the switching
> datasetgraph.
>
> Getting the named model for the union may work but uses more RAM to get
> set semantics.
>
> Andy
>
>
>
>>
>> On Mon, May 26, 2014 at 5:55 PM, Andy Seaborne <an...@apache.org> wrote:
>>
>> On 26/05/14 15:54, Claude Warren wrote:
>>>
>>> Look like I needed to add a model=dataset.getDefaultModel() statement
>>>> to
>>>> the commit block os that it reads:
>>>>
>>>>
>>>> if (cnt % 500 == 0) {
>>>>
>>>> System.out.println( cnt );
>>>>
>>>> dataset.commit();
>>>>
>>>> dataset.begin( ReadWrite.WRITE );
>>>>
>>>> model=dataset.getDefaultModel();
>>>>
>>>> }
>>>>
>>>> Why? I would expect that the model would track the transaction set on
>>>> the
>>>> dataset.
>>>>
>>>>
>>> Should be possible but it's hard to balance compatibility
>>> (non-transactional use) with being about to mix TDB graphs into general
>>> purpose dataset and also get efficiency (get SPARQL processing to be
>>> handled by TDB, code that calls .getDefaultModel and not assigning to a
>>> local variable).
>>>
>>> Come Jena3, this could be improved with changes to the application
>>> contract for the more corner cases (single thread overlapping
>>> transactions
>>> for example). Maybe drop the direct SPARQL execution of TDB graphs in
>>> general purpose, mixed datasets and resort to general SPARQL execution.
>>>
>>> It's not just Models - Resources have an implicit graph.
>>>
>>> What would be really helpful is to have an example test case that
>>> illustrated some aspect of the problem which was compact and standalone.
>>>
>>> Andy
>>>
>>>
>>>
>>>> Claude
>>>>
>>>>
>>>> On Mon, May 26, 2014 at 3:48 PM, Claude Warren <cl...@xenei.com>
>>>> wrote:
>>>>
>>>> I have code that uses Jena-Libs 2.11.1 and executes the following
>>>> block:
>>>>
>>>>> (edited for brevity)
>>>>>
>>>>> dataset.begin( ReadWrite.WRITE);
>>>>>
>>>>> try {
>>>>>
>>>>> Model model = dataset.getDefaultModel();
>>>>>
>>>>> Property[] props = ( create an array of properties in Model)
>>>>>
>>>>> String dcsId = "someStringValue";
>>>>>
>>>>> int cnt = 0;
>>>>> ..... (get an iterator on a set of rows from a db table)
>>>>>
>>>>> while (rIter.hasNext())
>>>>>
>>>>> {
>>>>>
>>>>> cnt++;
>>>>>
>>>>> Row r = rIter.next();
>>>>>
>>>>> Resource eventR = model.createResource( makeURI( eventUri,
>>>>> r.getColumn(dcsId).toString()), typeURI);
>>>>>
>>>>> for (int i=0;i<cols.size();i++)
>>>>>
>>>>> {
>>>>>
>>>>> Object o = r.getColumn(i);
>>>>>
>>>>> if (props[i] != null && o != null)
>>>>>
>>>>> {
>>>>>
>>>>> String val = o.toString();
>>>>>
>>>>> if (val.length()>0)
>>>>>
>>>>> {
>>>>>
>>>>> eventR.addProperty( props[i], val); // <-- line 134 as
>>>>> noted
>>>>> in call stack below
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> // commit every 500 entries
>>>>>
>>>>> if (cnt % 500 == 0) {
>>>>>
>>>>> System.out.println( cnt );
>>>>>
>>>>> dataset.commit();
>>>>>
>>>>> dataset.begin( ReadWrite.WRITE );
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> finally {
>>>>>
>>>>> dataset.commit();
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> The table I am reading from contains several 10's of thousands of rows.
>>>>> I
>>>>> am basically converting from relational table to RDF graph.
>>>>>
>>>>> However, after the first 500 subjects are committed (5000 +/- triples)
>>>>> I
>>>>> run into the following problem while adding the 6th property to subject
>>>>> 502.
>>>>>
>>>>>
>>>>> DEBUG [main] (MySQLTableDef.java:261) - Query String: SELECT ... <--
>>>>> Initial debugging statement
>>>>>
>>>>> 500 <-- output from commit block in code above
>>>>>
>>>>> ERROR [main] (Log.java:94) - Not active: 1
>>>>>
>>>>> ERROR [main] (Log.java:94) - **** Not active: 1
>>>>>
>>>>>
>>>>> Now, I know this is a transaction error. But I think I have a
>>>>> transaction, as I restart one right after the commit after the 500th
>>>>> record. Log is made in the BlockMgrJournal.checkActive() and the call
>>>>> stack appears as follows:
>>>>> BlockMgrJournal.checkActive() line: 306
>>>>> BlockMgrJournal._promote(Block) line: 220
>>>>> BlockMgrJournal.promote(Block) line: 215
>>>>> BPTreeRecordsMgr(PageBlockMgr<T>).promote(Page) line: 110
>>>>> BPTreeRecords.promote() line: 119
>>>>> BPTreeRecords.internalInsert(Record) line: 131
>>>>> BPTreeNode.internalInsert(Record) line: 468
>>>>> BPTreeNode.internalInsert(Record) line: 468
>>>>> BPTreeNode.insert(BPTreeNode, Record) line: 212
>>>>> BPlusTree.addAndReturnOld(Record) line: 328
>>>>> BPlusTree.add(Record) line: 320
>>>>> TupleIndexRecord.performAdd(Tuple<NodeId>) line: 60
>>>>> TupleIndexRecord(TupleIndexBase).add(Tuple<NodeId>) line: 64
>>>>> TupleTable.add(Tuple<NodeId>) line: 96
>>>>> NodeTupleTableConcrete.addRow(Node...) line: 87
>>>>> TripleTable.add(Node, Node, Node) line: 58
>>>>> DatasetGraphTDB.addToDftGraph(Node, Node, Node) line: 100
>>>>> DatasetGraphTDB(DatasetGraphTriplesQuads).add(Node, Node, Node, Node)
>>>>> line: 47
>>>>> GraphTDB(GraphView).performAdd(Triple) line: 140
>>>>> GraphTDB.performAdd(Triple) line: 87
>>>>> GraphTDB(GraphBase).add(Triple) line: 202
>>>>> ModelCom.add(Resource, Property, RDFNode) line: 1159
>>>>> ModelCom.add(Resource, Property, String, String, boolean) line: 161
>>>>> ModelCom.add(Resource, Property, String) line: 149
>>>>> ResourceImpl.addProperty(Property, String) line: 241
>>>>> JenaSink.updateTable(UpdateBlock) line: 134
>>>>>
>>>>>
>>>>> Any help fixing this problem would be appreciated.
>>>>>
>>>>> Claude
>>>>> --
>>>>> I like: Like Like - The likeliest place on the web<
>>>>> http://like-like.xenei.com>
>>>>> LinkedIn: http://www.linkedin.com/in/claudewarren
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>
--
I like: Like Like - The likeliest place on the web
<http://like-like.xenei.com>
LinkedIn: http://www.linkedin.com/in/claudewarren
Re: TDB transaction weirdness.
Posted by Andy Seaborne <an...@apache.org>.
On 26/05/14 18:57, Claude Warren wrote:
> I was looking at how I could get a minimal set but the data is business
> confidential and I am not certian that changing it would produce the same
> issue. I will see what I can do about a minimal set -- though I expect
> that will still be large.
>
> Claude
I would be surprised if there were no effects except at scale although
where it's going to show first is not clear to me.
If you want to try something amusing possibly with side effects ... but
SPARQL query on the model would by the general engine in Node space (it
can't see that the graoh is TDB-backed).
Graph g = GraphView.createDefaultGraph(ds.asDatasetGraph()) ;
model = ModelFactory.createModelForGraph(g) ;
which inverts the stack to put the view over the top of the switching
datasetgraph.
Getting the named model for the union may work but uses more RAM to get
set semantics.
Andy
>
>
> On Mon, May 26, 2014 at 5:55 PM, Andy Seaborne <an...@apache.org> wrote:
>
>> On 26/05/14 15:54, Claude Warren wrote:
>>
>>> Look like I needed to add a model=dataset.getDefaultModel() statement to
>>> the commit block os that it reads:
>>>
>>>
>>> if (cnt % 500 == 0) {
>>>
>>> System.out.println( cnt );
>>>
>>> dataset.commit();
>>>
>>> dataset.begin( ReadWrite.WRITE );
>>>
>>> model=dataset.getDefaultModel();
>>>
>>> }
>>>
>>> Why? I would expect that the model would track the transaction set on the
>>> dataset.
>>>
>>
>> Should be possible but it's hard to balance compatibility
>> (non-transactional use) with being about to mix TDB graphs into general
>> purpose dataset and also get efficiency (get SPARQL processing to be
>> handled by TDB, code that calls .getDefaultModel and not assigning to a
>> local variable).
>>
>> Come Jena3, this could be improved with changes to the application
>> contract for the more corner cases (single thread overlapping transactions
>> for example). Maybe drop the direct SPARQL execution of TDB graphs in
>> general purpose, mixed datasets and resort to general SPARQL execution.
>>
>> It's not just Models - Resources have an implicit graph.
>>
>> What would be really helpful is to have an example test case that
>> illustrated some aspect of the problem which was compact and standalone.
>>
>> Andy
>>
>>
>>>
>>> Claude
>>>
>>>
>>> On Mon, May 26, 2014 at 3:48 PM, Claude Warren <cl...@xenei.com> wrote:
>>>
>>> I have code that uses Jena-Libs 2.11.1 and executes the following block:
>>>> (edited for brevity)
>>>>
>>>> dataset.begin( ReadWrite.WRITE);
>>>>
>>>> try {
>>>>
>>>> Model model = dataset.getDefaultModel();
>>>>
>>>> Property[] props = ( create an array of properties in Model)
>>>>
>>>> String dcsId = "someStringValue";
>>>>
>>>> int cnt = 0;
>>>> ..... (get an iterator on a set of rows from a db table)
>>>>
>>>> while (rIter.hasNext())
>>>>
>>>> {
>>>>
>>>> cnt++;
>>>>
>>>> Row r = rIter.next();
>>>>
>>>> Resource eventR = model.createResource( makeURI( eventUri,
>>>> r.getColumn(dcsId).toString()), typeURI);
>>>>
>>>> for (int i=0;i<cols.size();i++)
>>>>
>>>> {
>>>>
>>>> Object o = r.getColumn(i);
>>>>
>>>> if (props[i] != null && o != null)
>>>>
>>>> {
>>>>
>>>> String val = o.toString();
>>>>
>>>> if (val.length()>0)
>>>>
>>>> {
>>>>
>>>> eventR.addProperty( props[i], val); // <-- line 134 as
>>>> noted
>>>> in call stack below
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>> // commit every 500 entries
>>>>
>>>> if (cnt % 500 == 0) {
>>>>
>>>> System.out.println( cnt );
>>>>
>>>> dataset.commit();
>>>>
>>>> dataset.begin( ReadWrite.WRITE );
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>> finally {
>>>>
>>>> dataset.commit();
>>>>
>>>> }
>>>>
>>>>
>>>> The table I am reading from contains several 10's of thousands of rows.
>>>> I
>>>> am basically converting from relational table to RDF graph.
>>>>
>>>> However, after the first 500 subjects are committed (5000 +/- triples) I
>>>> run into the following problem while adding the 6th property to subject
>>>> 502.
>>>>
>>>>
>>>> DEBUG [main] (MySQLTableDef.java:261) - Query String: SELECT ... <--
>>>> Initial debugging statement
>>>>
>>>> 500 <-- output from commit block in code above
>>>>
>>>> ERROR [main] (Log.java:94) - Not active: 1
>>>>
>>>> ERROR [main] (Log.java:94) - **** Not active: 1
>>>>
>>>>
>>>> Now, I know this is a transaction error. But I think I have a
>>>> transaction, as I restart one right after the commit after the 500th
>>>> record. Log is made in the BlockMgrJournal.checkActive() and the call
>>>> stack appears as follows:
>>>> BlockMgrJournal.checkActive() line: 306
>>>> BlockMgrJournal._promote(Block) line: 220
>>>> BlockMgrJournal.promote(Block) line: 215
>>>> BPTreeRecordsMgr(PageBlockMgr<T>).promote(Page) line: 110
>>>> BPTreeRecords.promote() line: 119
>>>> BPTreeRecords.internalInsert(Record) line: 131
>>>> BPTreeNode.internalInsert(Record) line: 468
>>>> BPTreeNode.internalInsert(Record) line: 468
>>>> BPTreeNode.insert(BPTreeNode, Record) line: 212
>>>> BPlusTree.addAndReturnOld(Record) line: 328
>>>> BPlusTree.add(Record) line: 320
>>>> TupleIndexRecord.performAdd(Tuple<NodeId>) line: 60
>>>> TupleIndexRecord(TupleIndexBase).add(Tuple<NodeId>) line: 64
>>>> TupleTable.add(Tuple<NodeId>) line: 96
>>>> NodeTupleTableConcrete.addRow(Node...) line: 87
>>>> TripleTable.add(Node, Node, Node) line: 58
>>>> DatasetGraphTDB.addToDftGraph(Node, Node, Node) line: 100
>>>> DatasetGraphTDB(DatasetGraphTriplesQuads).add(Node, Node, Node, Node)
>>>> line: 47
>>>> GraphTDB(GraphView).performAdd(Triple) line: 140
>>>> GraphTDB.performAdd(Triple) line: 87
>>>> GraphTDB(GraphBase).add(Triple) line: 202
>>>> ModelCom.add(Resource, Property, RDFNode) line: 1159
>>>> ModelCom.add(Resource, Property, String, String, boolean) line: 161
>>>> ModelCom.add(Resource, Property, String) line: 149
>>>> ResourceImpl.addProperty(Property, String) line: 241
>>>> JenaSink.updateTable(UpdateBlock) line: 134
>>>>
>>>>
>>>> Any help fixing this problem would be appreciated.
>>>>
>>>> Claude
>>>> --
>>>> I like: Like Like - The likeliest place on the web<
>>>> http://like-like.xenei.com>
>>>> LinkedIn: http://www.linkedin.com/in/claudewarren
>>>>
>>>>
>>>
>>>
>>>
>>
>
>
Re: TDB transaction weirdness.
Posted by Claude Warren <cl...@xenei.com>.
I was looking at how I could get a minimal set but the data is business
confidential and I am not certian that changing it would produce the same
issue. I will see what I can do about a minimal set -- though I expect
that will still be large.
Claude
On Mon, May 26, 2014 at 5:55 PM, Andy Seaborne <an...@apache.org> wrote:
> On 26/05/14 15:54, Claude Warren wrote:
>
>> Look like I needed to add a model=dataset.getDefaultModel() statement to
>> the commit block os that it reads:
>>
>>
>> if (cnt % 500 == 0) {
>>
>> System.out.println( cnt );
>>
>> dataset.commit();
>>
>> dataset.begin( ReadWrite.WRITE );
>>
>> model=dataset.getDefaultModel();
>>
>> }
>>
>> Why? I would expect that the model would track the transaction set on the
>> dataset.
>>
>
> Should be possible but it's hard to balance compatibility
> (non-transactional use) with being about to mix TDB graphs into general
> purpose dataset and also get efficiency (get SPARQL processing to be
> handled by TDB, code that calls .getDefaultModel and not assigning to a
> local variable).
>
> Come Jena3, this could be improved with changes to the application
> contract for the more corner cases (single thread overlapping transactions
> for example). Maybe drop the direct SPARQL execution of TDB graphs in
> general purpose, mixed datasets and resort to general SPARQL execution.
>
> It's not just Models - Resources have an implicit graph.
>
> What would be really helpful is to have an example test case that
> illustrated some aspect of the problem which was compact and standalone.
>
> Andy
>
>
>>
>> Claude
>>
>>
>> On Mon, May 26, 2014 at 3:48 PM, Claude Warren <cl...@xenei.com> wrote:
>>
>> I have code that uses Jena-Libs 2.11.1 and executes the following block:
>>> (edited for brevity)
>>>
>>> dataset.begin( ReadWrite.WRITE);
>>>
>>> try {
>>>
>>> Model model = dataset.getDefaultModel();
>>>
>>> Property[] props = ( create an array of properties in Model)
>>>
>>> String dcsId = "someStringValue";
>>>
>>> int cnt = 0;
>>> ..... (get an iterator on a set of rows from a db table)
>>>
>>> while (rIter.hasNext())
>>>
>>> {
>>>
>>> cnt++;
>>>
>>> Row r = rIter.next();
>>>
>>> Resource eventR = model.createResource( makeURI( eventUri,
>>> r.getColumn(dcsId).toString()), typeURI);
>>>
>>> for (int i=0;i<cols.size();i++)
>>>
>>> {
>>>
>>> Object o = r.getColumn(i);
>>>
>>> if (props[i] != null && o != null)
>>>
>>> {
>>>
>>> String val = o.toString();
>>>
>>> if (val.length()>0)
>>>
>>> {
>>>
>>> eventR.addProperty( props[i], val); // <-- line 134 as
>>> noted
>>> in call stack below
>>>
>>> }
>>>
>>> }
>>>
>>> }
>>>
>>> // commit every 500 entries
>>>
>>> if (cnt % 500 == 0) {
>>>
>>> System.out.println( cnt );
>>>
>>> dataset.commit();
>>>
>>> dataset.begin( ReadWrite.WRITE );
>>>
>>> }
>>>
>>> }
>>>
>>> }
>>>
>>> }
>>>
>>> }
>>>
>>> finally {
>>>
>>> dataset.commit();
>>>
>>> }
>>>
>>>
>>> The table I am reading from contains several 10's of thousands of rows.
>>> I
>>> am basically converting from relational table to RDF graph.
>>>
>>> However, after the first 500 subjects are committed (5000 +/- triples) I
>>> run into the following problem while adding the 6th property to subject
>>> 502.
>>>
>>>
>>> DEBUG [main] (MySQLTableDef.java:261) - Query String: SELECT ... <--
>>> Initial debugging statement
>>>
>>> 500 <-- output from commit block in code above
>>>
>>> ERROR [main] (Log.java:94) - Not active: 1
>>>
>>> ERROR [main] (Log.java:94) - **** Not active: 1
>>>
>>>
>>> Now, I know this is a transaction error. But I think I have a
>>> transaction, as I restart one right after the commit after the 500th
>>> record. Log is made in the BlockMgrJournal.checkActive() and the call
>>> stack appears as follows:
>>> BlockMgrJournal.checkActive() line: 306
>>> BlockMgrJournal._promote(Block) line: 220
>>> BlockMgrJournal.promote(Block) line: 215
>>> BPTreeRecordsMgr(PageBlockMgr<T>).promote(Page) line: 110
>>> BPTreeRecords.promote() line: 119
>>> BPTreeRecords.internalInsert(Record) line: 131
>>> BPTreeNode.internalInsert(Record) line: 468
>>> BPTreeNode.internalInsert(Record) line: 468
>>> BPTreeNode.insert(BPTreeNode, Record) line: 212
>>> BPlusTree.addAndReturnOld(Record) line: 328
>>> BPlusTree.add(Record) line: 320
>>> TupleIndexRecord.performAdd(Tuple<NodeId>) line: 60
>>> TupleIndexRecord(TupleIndexBase).add(Tuple<NodeId>) line: 64
>>> TupleTable.add(Tuple<NodeId>) line: 96
>>> NodeTupleTableConcrete.addRow(Node...) line: 87
>>> TripleTable.add(Node, Node, Node) line: 58
>>> DatasetGraphTDB.addToDftGraph(Node, Node, Node) line: 100
>>> DatasetGraphTDB(DatasetGraphTriplesQuads).add(Node, Node, Node, Node)
>>> line: 47
>>> GraphTDB(GraphView).performAdd(Triple) line: 140
>>> GraphTDB.performAdd(Triple) line: 87
>>> GraphTDB(GraphBase).add(Triple) line: 202
>>> ModelCom.add(Resource, Property, RDFNode) line: 1159
>>> ModelCom.add(Resource, Property, String, String, boolean) line: 161
>>> ModelCom.add(Resource, Property, String) line: 149
>>> ResourceImpl.addProperty(Property, String) line: 241
>>> JenaSink.updateTable(UpdateBlock) line: 134
>>>
>>>
>>> Any help fixing this problem would be appreciated.
>>>
>>> Claude
>>> --
>>> I like: Like Like - The likeliest place on the web<
>>> http://like-like.xenei.com>
>>> LinkedIn: http://www.linkedin.com/in/claudewarren
>>>
>>>
>>
>>
>>
>
--
I like: Like Like - The likeliest place on the web<http://like-like.xenei.com>
LinkedIn: http://www.linkedin.com/in/claudewarren
Re: TDB transaction weirdness.
Posted by Andy Seaborne <an...@apache.org>.
On 26/05/14 15:54, Claude Warren wrote:
> Look like I needed to add a model=dataset.getDefaultModel() statement to
> the commit block os that it reads:
>
>
> if (cnt % 500 == 0) {
>
> System.out.println( cnt );
>
> dataset.commit();
>
> dataset.begin( ReadWrite.WRITE );
>
> model=dataset.getDefaultModel();
>
> }
>
> Why? I would expect that the model would track the transaction set on the
> dataset.
Should be possible but it's hard to balance compatibility
(non-transactional use) with being about to mix TDB graphs into general
purpose dataset and also get efficiency (get SPARQL processing to be
handled by TDB, code that calls .getDefaultModel and not assigning to a
local variable).
Come Jena3, this could be improved with changes to the application
contract for the more corner cases (single thread overlapping
transactions for example). Maybe drop the direct SPARQL execution of
TDB graphs in general purpose, mixed datasets and resort to general
SPARQL execution.
It's not just Models - Resources have an implicit graph.
What would be really helpful is to have an example test case that
illustrated some aspect of the problem which was compact and standalone.
Andy
>
>
> Claude
>
>
> On Mon, May 26, 2014 at 3:48 PM, Claude Warren <cl...@xenei.com> wrote:
>
>> I have code that uses Jena-Libs 2.11.1 and executes the following block:
>> (edited for brevity)
>>
>> dataset.begin( ReadWrite.WRITE);
>>
>> try {
>>
>> Model model = dataset.getDefaultModel();
>>
>> Property[] props = ( create an array of properties in Model)
>>
>> String dcsId = "someStringValue";
>>
>> int cnt = 0;
>> ..... (get an iterator on a set of rows from a db table)
>>
>> while (rIter.hasNext())
>>
>> {
>>
>> cnt++;
>>
>> Row r = rIter.next();
>>
>> Resource eventR = model.createResource( makeURI( eventUri,
>> r.getColumn(dcsId).toString()), typeURI);
>>
>> for (int i=0;i<cols.size();i++)
>>
>> {
>>
>> Object o = r.getColumn(i);
>>
>> if (props[i] != null && o != null)
>>
>> {
>>
>> String val = o.toString();
>>
>> if (val.length()>0)
>>
>> {
>>
>> eventR.addProperty( props[i], val); // <-- line 134 as noted
>> in call stack below
>>
>> }
>>
>> }
>>
>> }
>>
>> // commit every 500 entries
>>
>> if (cnt % 500 == 0) {
>>
>> System.out.println( cnt );
>>
>> dataset.commit();
>>
>> dataset.begin( ReadWrite.WRITE );
>>
>> }
>>
>> }
>>
>> }
>>
>> }
>>
>> }
>>
>> finally {
>>
>> dataset.commit();
>>
>> }
>>
>>
>> The table I am reading from contains several 10's of thousands of rows. I
>> am basically converting from relational table to RDF graph.
>>
>> However, after the first 500 subjects are committed (5000 +/- triples) I
>> run into the following problem while adding the 6th property to subject
>> 502.
>>
>>
>> DEBUG [main] (MySQLTableDef.java:261) - Query String: SELECT ... <--
>> Initial debugging statement
>>
>> 500 <-- output from commit block in code above
>>
>> ERROR [main] (Log.java:94) - Not active: 1
>>
>> ERROR [main] (Log.java:94) - **** Not active: 1
>>
>>
>> Now, I know this is a transaction error. But I think I have a
>> transaction, as I restart one right after the commit after the 500th
>> record. Log is made in the BlockMgrJournal.checkActive() and the call
>> stack appears as follows:
>> BlockMgrJournal.checkActive() line: 306
>> BlockMgrJournal._promote(Block) line: 220
>> BlockMgrJournal.promote(Block) line: 215
>> BPTreeRecordsMgr(PageBlockMgr<T>).promote(Page) line: 110
>> BPTreeRecords.promote() line: 119
>> BPTreeRecords.internalInsert(Record) line: 131
>> BPTreeNode.internalInsert(Record) line: 468
>> BPTreeNode.internalInsert(Record) line: 468
>> BPTreeNode.insert(BPTreeNode, Record) line: 212
>> BPlusTree.addAndReturnOld(Record) line: 328
>> BPlusTree.add(Record) line: 320
>> TupleIndexRecord.performAdd(Tuple<NodeId>) line: 60
>> TupleIndexRecord(TupleIndexBase).add(Tuple<NodeId>) line: 64
>> TupleTable.add(Tuple<NodeId>) line: 96
>> NodeTupleTableConcrete.addRow(Node...) line: 87
>> TripleTable.add(Node, Node, Node) line: 58
>> DatasetGraphTDB.addToDftGraph(Node, Node, Node) line: 100
>> DatasetGraphTDB(DatasetGraphTriplesQuads).add(Node, Node, Node, Node)
>> line: 47
>> GraphTDB(GraphView).performAdd(Triple) line: 140
>> GraphTDB.performAdd(Triple) line: 87
>> GraphTDB(GraphBase).add(Triple) line: 202
>> ModelCom.add(Resource, Property, RDFNode) line: 1159
>> ModelCom.add(Resource, Property, String, String, boolean) line: 161
>> ModelCom.add(Resource, Property, String) line: 149
>> ResourceImpl.addProperty(Property, String) line: 241
>> JenaSink.updateTable(UpdateBlock) line: 134
>>
>>
>> Any help fixing this problem would be appreciated.
>>
>> Claude
>> --
>> I like: Like Like - The likeliest place on the web<http://like-like.xenei.com>
>> LinkedIn: http://www.linkedin.com/in/claudewarren
>>
>
>
>
Re: TDB transaction weirdness.
Posted by Claude Warren <cl...@xenei.com>.
Look like I needed to add a model=dataset.getDefaultModel() statement to
the commit block os that it reads:
if (cnt % 500 == 0) {
System.out.println( cnt );
dataset.commit();
dataset.begin( ReadWrite.WRITE );
model=dataset.getDefaultModel();
}
Why? I would expect that the model would track the transaction set on the
dataset.
Claude
On Mon, May 26, 2014 at 3:48 PM, Claude Warren <cl...@xenei.com> wrote:
> I have code that uses Jena-Libs 2.11.1 and executes the following block:
> (edited for brevity)
>
> dataset.begin( ReadWrite.WRITE);
>
> try {
>
> Model model = dataset.getDefaultModel();
>
> Property[] props = ( create an array of properties in Model)
>
> String dcsId = "someStringValue";
>
> int cnt = 0;
> ..... (get an iterator on a set of rows from a db table)
>
> while (rIter.hasNext())
>
> {
>
> cnt++;
>
> Row r = rIter.next();
>
> Resource eventR = model.createResource( makeURI( eventUri,
> r.getColumn(dcsId).toString()), typeURI);
>
> for (int i=0;i<cols.size();i++)
>
> {
>
> Object o = r.getColumn(i);
>
> if (props[i] != null && o != null)
>
> {
>
> String val = o.toString();
>
> if (val.length()>0)
>
> {
>
> eventR.addProperty( props[i], val); // <-- line 134 as noted
> in call stack below
>
> }
>
> }
>
> }
>
> // commit every 500 entries
>
> if (cnt % 500 == 0) {
>
> System.out.println( cnt );
>
> dataset.commit();
>
> dataset.begin( ReadWrite.WRITE );
>
> }
>
> }
>
> }
>
> }
>
> }
>
> finally {
>
> dataset.commit();
>
> }
>
>
> The table I am reading from contains several 10's of thousands of rows. I
> am basically converting from relational table to RDF graph.
>
> However, after the first 500 subjects are committed (5000 +/- triples) I
> run into the following problem while adding the 6th property to subject
> 502.
>
>
> DEBUG [main] (MySQLTableDef.java:261) - Query String: SELECT ... <--
> Initial debugging statement
>
> 500 <-- output from commit block in code above
>
> ERROR [main] (Log.java:94) - Not active: 1
>
> ERROR [main] (Log.java:94) - **** Not active: 1
>
>
> Now, I know this is a transaction error. But I think I have a
> transaction, as I restart one right after the commit after the 500th
> record. Log is made in the BlockMgrJournal.checkActive() and the call
> stack appears as follows:
> BlockMgrJournal.checkActive() line: 306
> BlockMgrJournal._promote(Block) line: 220
> BlockMgrJournal.promote(Block) line: 215
> BPTreeRecordsMgr(PageBlockMgr<T>).promote(Page) line: 110
> BPTreeRecords.promote() line: 119
> BPTreeRecords.internalInsert(Record) line: 131
> BPTreeNode.internalInsert(Record) line: 468
> BPTreeNode.internalInsert(Record) line: 468
> BPTreeNode.insert(BPTreeNode, Record) line: 212
> BPlusTree.addAndReturnOld(Record) line: 328
> BPlusTree.add(Record) line: 320
> TupleIndexRecord.performAdd(Tuple<NodeId>) line: 60
> TupleIndexRecord(TupleIndexBase).add(Tuple<NodeId>) line: 64
> TupleTable.add(Tuple<NodeId>) line: 96
> NodeTupleTableConcrete.addRow(Node...) line: 87
> TripleTable.add(Node, Node, Node) line: 58
> DatasetGraphTDB.addToDftGraph(Node, Node, Node) line: 100
> DatasetGraphTDB(DatasetGraphTriplesQuads).add(Node, Node, Node, Node)
> line: 47
> GraphTDB(GraphView).performAdd(Triple) line: 140
> GraphTDB.performAdd(Triple) line: 87
> GraphTDB(GraphBase).add(Triple) line: 202
> ModelCom.add(Resource, Property, RDFNode) line: 1159
> ModelCom.add(Resource, Property, String, String, boolean) line: 161
> ModelCom.add(Resource, Property, String) line: 149
> ResourceImpl.addProperty(Property, String) line: 241
> JenaSink.updateTable(UpdateBlock) line: 134
>
>
> Any help fixing this problem would be appreciated.
>
> Claude
> --
> I like: Like Like - The likeliest place on the web<http://like-like.xenei.com>
> LinkedIn: http://www.linkedin.com/in/claudewarren
>
--
I like: Like Like - The likeliest place on the web<http://like-like.xenei.com>
LinkedIn: http://www.linkedin.com/in/claudewarren