You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by Andy Seaborne <an...@apache.org> on 2014/04/06 18:32:47 UTC
Re: BlockException: No such block
On 27/02/14 17:08, Steve Groeger wrote:> Hi Andy,
>
> What is the best way to get the code to you?
>
>
> Thanks
> Steve Groeger
I'm not getting failures anymore. Ouf course, deep joy, it's a
different JDK build (not that I think this is JDk related).
Can you provide exactly the code you have been using?
Thanks
Andy
On 28/02/14 23:04, Andy Seaborne wrote:
> On 27/02/14 20:11, Steve Groeger wrote:
>> Hi Andy,
>>
>> I ran the test again with the additional code you requested. It took a
>> long time to fail, over 4.3 million reads but it eventually did. The
>> error
>> we saw was:
>>
>> reader3: Models do not compare expected size: 1000 was: 778
>>
>> java.lang.IllegalArgumentException: ObjectFile.read[nodes]: Bad read: -1
>> at com.hp.hpl.jena.tdb.base.objectfile.ObjectFileStorage.read(
>> ObjectFileStorage.java:311)
>> at com.hp.hpl.jena.tdb.lib.NodeLib.fetchDecode(NodeLib.java:79)
> ...
>
>>
>> Hope this allows you to move closer to determining the issue.
>
> The fact it takes such a long time is a bit of a nuisanse. I hoped that
> nulling out would generate an earlier error with "read -1" (it might
> have been reading valid but deleted NodeIds before : -1 is not a valid id).
>
> We need to converge on exactly the same code to be testing - things are
> a bit clearer but it will need some staring at the code.
>
> Andy
>
>>
>> Thanks
>> Steve Groeger
>>
>>
>>
>>
>> From: Andy Seaborne <an...@apache.org>
>> To: users@jena.apache.org,
>> Date: 27/02/2014 15:09
>> Subject: Re: BlockException: No such block
>>
>>
>>
>> On 27/02/14 13:50, Steve Groeger wrote:
>>> Hi Andy,
>>>
>>> All the latest tests I was running were setting SystemTDB.NullOut =
>> true;
>>>
>>> I will add the code you specified to our tests and re-run them and will
>>> send you our results shortly.
>>
>> Thanks - if send the code as well, I'll try to run it at my end. Even
>> though I wasn't managing to catch this nearly as frequently as you, this
>> new situation might be better.
>>
>> Andy
>>
>>>
>>> Thanks
>>> Steve Groeger
>>> IBM WebSphere Service Registry Development
>>> MP 211, Hursley
>>> Tel: (44) 1962 816911 Mobex: 279990 Mobile: 07718 517 129
>>> Fax (44) 1962 816800
>>> Lotus Notes: Steve Groeger/UK/IBM
>>> Internet: groeges@uk.ibm.com
>>>
>>>
>>>
>>> From: Andy Seaborne <an...@apache.org>
>>> To: users@jena.apache.org,
>>> Date: 27/02/2014 13:36
>>> Subject: Re: BlockException: No such block
>>>
>>>
>>>
>>> Hi Steve,
>>>
>>> Thanks for the update.
>>>
>>> On 25/02/14 15:15, Steve Groeger wrote:
>>>> Hi Andy,
>>>>
>>>> I picked up a snapshot of 2.11.2 and tried running our test using these
>>>> libraries.
>>>>
>>>> Adding the changes you suggested seems to have stopped the exception "
>>>> BlockException: No such block", the but we are now seeing other
>>> symptoms,
>>>> which we saw previously but not as frequently as we are now.
>>>> The change in behaviour seems to be result of the SystemTDB.NullOut =
>>> true
>>>> ; as I also tried various version of the listStatements and this gave
>>> the
>>>> same results, What does the SystemTDB.NullOut=true do?
>>>
>>> NullOut makes the B+Tree code overwrite areas of blocks that are now
>>> unused because of a delete with some value (FillByte = 0xFF). In normal
>>> operation, it just leaves it as junk, but of NullOut is set it goes and
>>> sets the bytes to a specific value, and one which does not make sense.
>>>
>>> I would not expect it to change the frequency of errors, but it would
>>> change the symptom.
>>>
>>>> What we are now seeing is data not being returned correctly.
>>>> We initially populate the database with a Resource with 1000
>> Properties,
>>>> then in the ReaderThread we query the database, using the following:
>>>>
>>>> StmtIterator found =
>>>> model.listStatements((Resource)null,
>>>> (Property)null, (RDFNode)null);
>>>>
>>>> we are expecting to get back 1000 statements when we perform the
>>>>
>>>> count = found.toList().size();
>>>>
>>>> however we occasionally get a batch of results that show 0 or non 1000
>>>> statements being returned, then after several more reads this goes back
>>> to
>>>> getting 1000 statements and everything continues as normal.
>>>
>>> I'm not clear whether that is with or without NullOut set true.
>>>
>>>>
>>>> We need to determine why we occasionally dont get the 1000 statements
>>>> returned that we are expecting. Any advice you caould provide would be
>>>> great.
>>>
>>> It looks as if under some unusual conditions, either the B+Tree branch
>>> node or leaf node is not being updated correctly for deletions.
>>> ("unusual" as this code is not new and for anyone doing deletes, heavily
>>> used).
>>>
>>> That gives us something to go on.
>>>
>>> Could you please add some code that not only counts the StmtIterator but
>>> also touches every part of the statement. Execute with
>>>
>>> SystemTDB.NullOut=true
>>>
>>> There is some delayed evaluation under the covers and this forces that
>>> evaluation.
>>>
>>> StmtIterator found = ....
>>> long count = 0 ;
>>> while(found.hasNext()) {
>>> count++ ;
>>> Statement s = found.next();
>>> try {
>>> Node subj = s.getSubject().asNode() ;
>>> Node pred = s.getPredicate().asNode() ;
>>> Node obj = s.getObject().asNode() ;
>>> } catch (Exception ex) {
>>> ex.printStackTrace(System.err) ;
>>> System.exit(99) ;
>>> }
>>> }
>>>
>>> Warning - this may slow the execution down, or not, depending on cache
>>> efficiencies.
>>>
>>> Andy
>>>
>>>>
>>>> Thanks
>>>> Steve Groeger
>>>> IBM WebSphere Service Registry Development
>>>> MP 211, Hursley
>>>> Tel: (44) 1962 816911 Mobex: 279990 Mobile: 07718 517 129
>>>> Fax (44) 1962 816800
>>>> Lotus Notes: Steve Groeger/UK/IBM
>>>> Internet: groeges@uk.ibm.com
>>>
>>>
>>>
>>> Unless stated otherwise above:
>>> IBM United Kingdom Limited - Registered in England and Wales with number
>>> 741598.
>>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
>> 3AU
>>>
>>
>>
>>
>> Unless stated otherwise above:
>> IBM United Kingdom Limited - Registered in England and Wales with number
>> 741598.
>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
>> 3AU
>>
>
Re: BlockException: No such block
Posted by Andy Seaborne <an...@apache.org>.
Ping?
On 06/04/14 17:32, Andy Seaborne wrote:
> On 27/02/14 17:08, Steve Groeger wrote:> Hi Andy,
> >
> > What is the best way to get the code to you?
> >
> >
> > Thanks
> > Steve Groeger
>
> I'm not getting failures anymore. Ouf course, deep joy, it's a
> different JDK build (not that I think this is JDk related).
>
> Can you provide exactly the code you have been using?
>
> Thanks
> Andy
>
> On 28/02/14 23:04, Andy Seaborne wrote:
>> On 27/02/14 20:11, Steve Groeger wrote:
>>> Hi Andy,
>>>
>>> I ran the test again with the additional code you requested. It took a
>>> long time to fail, over 4.3 million reads but it eventually did. The
>>> error
>>> we saw was:
>>>
>>> reader3: Models do not compare expected size: 1000 was: 778
>>>
>>> java.lang.IllegalArgumentException: ObjectFile.read[nodes]: Bad read: -1
>>> at com.hp.hpl.jena.tdb.base.objectfile.ObjectFileStorage.read(
>>> ObjectFileStorage.java:311)
>>> at com.hp.hpl.jena.tdb.lib.NodeLib.fetchDecode(NodeLib.java:79)
>> ...
>>
>>>
>>> Hope this allows you to move closer to determining the issue.
>>
>> The fact it takes such a long time is a bit of a nuisanse. I hoped that
>> nulling out would generate an earlier error with "read -1" (it might
>> have been reading valid but deleted NodeIds before : -1 is not a valid
>> id).
>>
>> We need to converge on exactly the same code to be testing - things are
>> a bit clearer but it will need some staring at the code.
>>
>> Andy
>>
>>>
>>> Thanks
>>> Steve Groeger
>>>
>>>
>>>
>>>
>>> From: Andy Seaborne <an...@apache.org>
>>> To: users@jena.apache.org,
>>> Date: 27/02/2014 15:09
>>> Subject: Re: BlockException: No such block
>>>
>>>
>>>
>>> On 27/02/14 13:50, Steve Groeger wrote:
>>>> Hi Andy,
>>>>
>>>> All the latest tests I was running were setting SystemTDB.NullOut =
>>> true;
>>>>
>>>> I will add the code you specified to our tests and re-run them and will
>>>> send you our results shortly.
>>>
>>> Thanks - if send the code as well, I'll try to run it at my end. Even
>>> though I wasn't managing to catch this nearly as frequently as you, this
>>> new situation might be better.
>>>
>>> Andy
>>>
>>>>
>>>> Thanks
>>>> Steve Groeger
>>>> IBM WebSphere Service Registry Development
>>>> MP 211, Hursley
>>>> Tel: (44) 1962 816911 Mobex: 279990 Mobile: 07718 517 129
>>>> Fax (44) 1962 816800
>>>> Lotus Notes: Steve Groeger/UK/IBM
>>>> Internet: groeges@uk.ibm.com
>>>>
>>>>
>>>>
>>>> From: Andy Seaborne <an...@apache.org>
>>>> To: users@jena.apache.org,
>>>> Date: 27/02/2014 13:36
>>>> Subject: Re: BlockException: No such block
>>>>
>>>>
>>>>
>>>> Hi Steve,
>>>>
>>>> Thanks for the update.
>>>>
>>>> On 25/02/14 15:15, Steve Groeger wrote:
>>>>> Hi Andy,
>>>>>
>>>>> I picked up a snapshot of 2.11.2 and tried running our test using
>>>>> these
>>>>> libraries.
>>>>>
>>>>> Adding the changes you suggested seems to have stopped the exception "
>>>>> BlockException: No such block", the but we are now seeing other
>>>> symptoms,
>>>>> which we saw previously but not as frequently as we are now.
>>>>> The change in behaviour seems to be result of the SystemTDB.NullOut =
>>>> true
>>>>> ; as I also tried various version of the listStatements and this gave
>>>> the
>>>>> same results, What does the SystemTDB.NullOut=true do?
>>>>
>>>> NullOut makes the B+Tree code overwrite areas of blocks that are now
>>>> unused because of a delete with some value (FillByte = 0xFF). In
>>>> normal
>>>> operation, it just leaves it as junk, but of NullOut is set it goes and
>>>> sets the bytes to a specific value, and one which does not make sense.
>>>>
>>>> I would not expect it to change the frequency of errors, but it would
>>>> change the symptom.
>>>>
>>>>> What we are now seeing is data not being returned correctly.
>>>>> We initially populate the database with a Resource with 1000
>>> Properties,
>>>>> then in the ReaderThread we query the database, using the following:
>>>>>
>>>>> StmtIterator found =
>>>>> model.listStatements((Resource)null,
>>>>> (Property)null, (RDFNode)null);
>>>>>
>>>>> we are expecting to get back 1000 statements when we perform the
>>>>>
>>>>> count = found.toList().size();
>>>>>
>>>>> however we occasionally get a batch of results that show 0 or non 1000
>>>>> statements being returned, then after several more reads this goes
>>>>> back
>>>> to
>>>>> getting 1000 statements and everything continues as normal.
>>>>
>>>> I'm not clear whether that is with or without NullOut set true.
>>>>
>>>>>
>>>>> We need to determine why we occasionally dont get the 1000 statements
>>>>> returned that we are expecting. Any advice you caould provide would be
>>>>> great.
>>>>
>>>> It looks as if under some unusual conditions, either the B+Tree branch
>>>> node or leaf node is not being updated correctly for deletions.
>>>> ("unusual" as this code is not new and for anyone doing deletes,
>>>> heavily
>>>> used).
>>>>
>>>> That gives us something to go on.
>>>>
>>>> Could you please add some code that not only counts the StmtIterator
>>>> but
>>>> also touches every part of the statement. Execute with
>>>>
>>>> SystemTDB.NullOut=true
>>>>
>>>> There is some delayed evaluation under the covers and this forces that
>>>> evaluation.
>>>>
>>>> StmtIterator found = ....
>>>> long count = 0 ;
>>>> while(found.hasNext()) {
>>>> count++ ;
>>>> Statement s = found.next();
>>>> try {
>>>> Node subj = s.getSubject().asNode() ;
>>>> Node pred = s.getPredicate().asNode() ;
>>>> Node obj = s.getObject().asNode() ;
>>>> } catch (Exception ex) {
>>>> ex.printStackTrace(System.err) ;
>>>> System.exit(99) ;
>>>> }
>>>> }
>>>>
>>>> Warning - this may slow the execution down, or not, depending on cache
>>>> efficiencies.
>>>>
>>>> Andy
>>>>
>>>>>
>>>>> Thanks
>>>>> Steve Groeger
>>>>> IBM WebSphere Service Registry Development
>>>>> MP 211, Hursley
>>>>> Tel: (44) 1962 816911 Mobex: 279990 Mobile: 07718 517 129
>>>>> Fax (44) 1962 816800
>>>>> Lotus Notes: Steve Groeger/UK/IBM
>>>>> Internet: groeges@uk.ibm.com
>>>>
>>>>
>>>>
>>>> Unless stated otherwise above:
>>>> IBM United Kingdom Limited - Registered in England and Wales with
>>>> number
>>>> 741598.
>>>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
>>> 3AU
>>>>
>>>
>>>
>>>
>>> Unless stated otherwise above:
>>> IBM United Kingdom Limited - Registered in England and Wales with number
>>> 741598.
>>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
>>> 3AU
>>>
>>
>