You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by "Dr. André Lanka" <ma...@dr-lanka.de> on 2013/02/25 10:05:33 UTC
TDB: Large pile of BlockMgrJournals
Hello,
we encounter some problems while writing to TDB stores (0.9.4). We get
some StackOverflowExceptions now and then. Many objects of
BlockMgrJournal seems to overlay each other.
The problem seems to be solved by calling
StoreConnection.getExisting(Location).flush()
periodically.
We wonder, if there is also an internal heuristic that determines when
the journal files get transferred to the base files or if manually
flushing is always necessary?
Thanks
André
--
Dr. André Lanka * 0178 / 134 44 47 * http://dr-lanka.de
Re: TDB: Large pile of BlockMgrJournals
Posted by Andy Seaborne <an...@apache.org>.
"Dr. André Lanka" wrote:
> Now and then we get query timeouts (5 seconds limit). Would this
> "irritate" the transaction management?
No. That's fine and reasonable.
> Hi Andy,
>
> On 01.03.2013 11:03, "Dr. André Lanka" wrote:
>> Perhaps we misuse the API? : For writers, we use two transactions (both
>> read and write) within the same thread,
>
>
> We reduced (but could not completely prevent) the long living reader.
> Just to double-check: Is it appropriate to use two datasets (one read
> and one write transaction) in parallel in the same thread?
That's fine - there are tests in the test suite for multiple
transactions on the same thread in nested and unnested configurations.
Your occasional long-lived read transaction sounds more like a
JVM/scheduling issue under load with java not progressing a thread for
some while.
Andy
>
> Thanks
> André
>
>
>>
>> I'll keep you posted, about the reason of the problem.
>>
>> Best
>> André
>>
>>
>>
>> On 25.02.2013 23:16, Andy Seaborne wrote:
>>> On 25/02/13 09:05, "Dr. André Lanka" wrote:
>>>> Hello,
>>>>
>>>> we encounter some problems while writing to TDB stores (0.9.4). We get
>>>> some StackOverflowExceptions now and then. Many objects of
>>>> BlockMgrJournal seems to overlay each other.
>>>> The problem seems to be solved by calling
>>>>
>>>> StoreConnection.getExisting(Location).flush()
>>>>
>>>> periodically.
>>>>
>>>> We wonder, if there is also an internal heuristic that determines when
>>>> the journal files get transferred to the base files or if manually
>>>> flushing is always necessary?
>>>
>>> Do you have a sample stack trace? I can't tell what going on from the
>>> description.
>>>
>>> You get a stack of BlockMgrJournals under two conditions:
>>>
>>> 1/ Changes from multiple writers are batched together and the
>>> transactions only applied to the permanent database every few writers
>>> commits. This is not official API but you can, currently set this
>>> TransactionManager.QueueBatchSize
>>> the default is 10.
>>>
>>> (Fuseki sets it to zero for increased stability when running many
>>> datasets in one server)
>>>
>>> I would not have thought this on it's own would cause stack overflow
>>>
>>> 2/ But if there is a read transaction about, the writer changes can't be
>>> written back (it's a write-ahead logging system).
>>>
>>> If you have long-lived reader, then they may be blocking the writer
>>> commits finally clearing up (this is after the commit of the
>>> transaction).
>>>
>>> > StoreConnection.getExisting(Location).flush()
>>>
>>> Hmm - looking at that, it needs a "synchronized" (on the Transaction
>>> manager underlying operation)
>>>
>>> Andy
>>>
>>>>
>>>> Thanks
>>>> André
>>>>
>>>>
>>>
>>
>
Re: TDB: Large pile of BlockMgrJournals
Posted by "Dr. André Lanka" <ma...@dr-lanka.de>.
Hi Andy,
On 01.03.2013 11:03, "Dr. André Lanka" wrote:
> Perhaps we misuse the API? : For writers, we use two transactions (both
> read and write) within the same thread,
We reduced (but could not completely prevent) the long living reader.
Just to double-check: Is it appropriate to use two datasets (one read
and one write transaction) in parallel in the same thread?
Thanks
André
>
> I'll keep you posted, about the reason of the problem.
>
> Best
> André
>
>
>
> On 25.02.2013 23:16, Andy Seaborne wrote:
>> On 25/02/13 09:05, "Dr. André Lanka" wrote:
>>> Hello,
>>>
>>> we encounter some problems while writing to TDB stores (0.9.4). We get
>>> some StackOverflowExceptions now and then. Many objects of
>>> BlockMgrJournal seems to overlay each other.
>>> The problem seems to be solved by calling
>>>
>>> StoreConnection.getExisting(Location).flush()
>>>
>>> periodically.
>>>
>>> We wonder, if there is also an internal heuristic that determines when
>>> the journal files get transferred to the base files or if manually
>>> flushing is always necessary?
>>
>> Do you have a sample stack trace? I can't tell what going on from the
>> description.
>>
>> You get a stack of BlockMgrJournals under two conditions:
>>
>> 1/ Changes from multiple writers are batched together and the
>> transactions only applied to the permanent database every few writers
>> commits. This is not official API but you can, currently set this
>> TransactionManager.QueueBatchSize
>> the default is 10.
>>
>> (Fuseki sets it to zero for increased stability when running many
>> datasets in one server)
>>
>> I would not have thought this on it's own would cause stack overflow
>>
>> 2/ But if there is a read transaction about, the writer changes can't be
>> written back (it's a write-ahead logging system).
>>
>> If you have long-lived reader, then they may be blocking the writer
>> commits finally clearing up (this is after the commit of the
>> transaction).
>>
>> > StoreConnection.getExisting(Location).flush()
>>
>> Hmm - looking at that, it needs a "synchronized" (on the Transaction
>> manager underlying operation)
>>
>> Andy
>>
>>>
>>> Thanks
>>> André
>>>
>>>
>>
>
--
Dr. André Lanka * 0178 / 134 44 47 * http://dr-lanka.de
Re: TDB: Large pile of BlockMgrJournals
Posted by "Dr. André Lanka" <ma...@dr-lanka.de>.
Just to clarify this:
> For writers, we use two transactions (both read and write) within the
> same thread,
We create two different datasets:
newEntry.dataset = TDBFactory.createDataset(newEntry.location);
newEntry.dataset.begin(writeAccess ? ReadWrite.WRITE : ReadWrite.READ);
if(writeAccess) {
newEntry.readDataset=TDBFactory.createDataset(newEntry.location);
newEntry.readDataset.begin(ReadWrite.READ);
}
Regarding the long living reader, we have the following SysTxnState,
when the writer starts its work (the three occurrences from today):
[com.hojoki.semantic.component.rdfstore.impl.TDBCacheListener:113]:
There are active transaction (what is NOT supposed to occur). I'll close
each transaction alive. Current state: Active (R=1 W=0) : Finished
(R=48479, WC=7687, WA=0) Queue 4
[com.hojoki.semantic.component.rdfstore.impl.TDBCacheListener:113]:
There are active transaction (what is NOT supposed to occur). I'll close
each transaction alive. Current state: Active (R=1 W=0) : Finished
(R=7964, WC=22, WA=0) Queue 2
[com.hojoki.semantic.component.rdfstore.impl.TDBCacheListener:113]:
There are active transaction (what is NOT supposed to occur). I'll close
each transaction alive. Current state: Active (R=1 W=0) : Finished
(R=1254, WC=200, WA=0) Queue 0
Perhaps this helps
André
On 01.03.2013 11:03, "Dr. André Lanka" wrote:
> Hello,
>
> sorry for my late reply. I did some additional tests and tried to avoid
> the problem in our systems as far as possible. It turned out that you
> are completely right: A long living reader causes the problem. I still
> don't know which circumstances provoke the problem. It looks like a race
> condition since we do millions of transactions each day and get only a
> handful readers that don't finish.
>
> As a work-around I cancel any read transaction before a writer starts
> his work. Next, I'll will investigate which thread doesn't come to an
> end. Now and then we get query timeouts (5 seconds limit). Would this
> "irritate" the transaction management?
>
> Perhaps we misuse the API? : For writers, we use two transactions (both
> read and write) within the same thread, as need to access the "original"
> (i.e. untouched) model at the end of our write operation. Could this
> make trouble?
>
> I'll keep you posted, about the reason of the problem.
>
> Best
> André
>
>
>
> On 25.02.2013 23:16, Andy Seaborne wrote:
>> On 25/02/13 09:05, "Dr. André Lanka" wrote:
>>> Hello,
>>>
>>> we encounter some problems while writing to TDB stores (0.9.4). We get
>>> some StackOverflowExceptions now and then. Many objects of
>>> BlockMgrJournal seems to overlay each other.
>>> The problem seems to be solved by calling
>>>
>>> StoreConnection.getExisting(Location).flush()
>>>
>>> periodically.
>>>
>>> We wonder, if there is also an internal heuristic that determines when
>>> the journal files get transferred to the base files or if manually
>>> flushing is always necessary?
>>
>> Do you have a sample stack trace? I can't tell what going on from the
>> description.
>>
>> You get a stack of BlockMgrJournals under two conditions:
>>
>> 1/ Changes from multiple writers are batched together and the
>> transactions only applied to the permanent database every few writers
>> commits. This is not official API but you can, currently set this
>> TransactionManager.QueueBatchSize
>> the default is 10.
>>
>> (Fuseki sets it to zero for increased stability when running many
>> datasets in one server)
>>
>> I would not have thought this on it's own would cause stack overflow
>>
>> 2/ But if there is a read transaction about, the writer changes can't be
>> written back (it's a write-ahead logging system).
>>
>> If you have long-lived reader, then they may be blocking the writer
>> commits finally clearing up (this is after the commit of the
>> transaction).
>>
>> > StoreConnection.getExisting(Location).flush()
>>
>> Hmm - looking at that, it needs a "synchronized" (on the Transaction
>> manager underlying operation)
>>
>> Andy
>>
>>>
>>> Thanks
>>> André
>>>
>>>
>>
>
--
Dr. André Lanka * 0178 / 134 44 47 * http://dr-lanka.de
Re: TDB: Large pile of BlockMgrJournals
Posted by "Dr. André Lanka" <ma...@dr-lanka.de>.
Hello,
sorry for my late reply. I did some additional tests and tried to avoid
the problem in our systems as far as possible. It turned out that you
are completely right: A long living reader causes the problem. I still
don't know which circumstances provoke the problem. It looks like a race
condition since we do millions of transactions each day and get only a
handful readers that don't finish.
As a work-around I cancel any read transaction before a writer starts
his work. Next, I'll will investigate which thread doesn't come to an
end. Now and then we get query timeouts (5 seconds limit). Would this
"irritate" the transaction management?
Perhaps we misuse the API? : For writers, we use two transactions (both
read and write) within the same thread, as need to access the "original"
(i.e. untouched) model at the end of our write operation. Could this
make trouble?
I'll keep you posted, about the reason of the problem.
Best
André
On 25.02.2013 23:16, Andy Seaborne wrote:
> On 25/02/13 09:05, "Dr. André Lanka" wrote:
>> Hello,
>>
>> we encounter some problems while writing to TDB stores (0.9.4). We get
>> some StackOverflowExceptions now and then. Many objects of
>> BlockMgrJournal seems to overlay each other.
>> The problem seems to be solved by calling
>>
>> StoreConnection.getExisting(Location).flush()
>>
>> periodically.
>>
>> We wonder, if there is also an internal heuristic that determines when
>> the journal files get transferred to the base files or if manually
>> flushing is always necessary?
>
> Do you have a sample stack trace? I can't tell what going on from the
> description.
>
> You get a stack of BlockMgrJournals under two conditions:
>
> 1/ Changes from multiple writers are batched together and the
> transactions only applied to the permanent database every few writers
> commits. This is not official API but you can, currently set this
> TransactionManager.QueueBatchSize
> the default is 10.
>
> (Fuseki sets it to zero for increased stability when running many
> datasets in one server)
>
> I would not have thought this on it's own would cause stack overflow
>
> 2/ But if there is a read transaction about, the writer changes can't be
> written back (it's a write-ahead logging system).
>
> If you have long-lived reader, then they may be blocking the writer
> commits finally clearing up (this is after the commit of the transaction).
>
> > StoreConnection.getExisting(Location).flush()
>
> Hmm - looking at that, it needs a "synchronized" (on the Transaction
> manager underlying operation)
>
> Andy
>
>>
>> Thanks
>> André
>>
>>
>
--
Dr. André Lanka * 0178 / 134 44 47 * http://dr-lanka.de
Re: TDB: Large pile of BlockMgrJournals
Posted by Andy Seaborne <an...@apache.org>.
On 25/02/13 09:05, "Dr. André Lanka" wrote:
> Hello,
>
> we encounter some problems while writing to TDB stores (0.9.4). We get
> some StackOverflowExceptions now and then. Many objects of
> BlockMgrJournal seems to overlay each other.
> The problem seems to be solved by calling
>
> StoreConnection.getExisting(Location).flush()
>
> periodically.
>
> We wonder, if there is also an internal heuristic that determines when
> the journal files get transferred to the base files or if manually
> flushing is always necessary?
Do you have a sample stack trace? I can't tell what going on from the
description.
You get a stack of BlockMgrJournals under two conditions:
1/ Changes from multiple writers are batched together and the
transactions only applied to the permanent database every few writers
commits. This is not official API but you can, currently set this
TransactionManager.QueueBatchSize
the default is 10.
(Fuseki sets it to zero for increased stability when running many
datasets in one server)
I would not have thought this on it's own would cause stack overflow
2/ But if there is a read transaction about, the writer changes can't be
written back (it's a write-ahead logging system).
If you have long-lived reader, then they may be blocking the writer
commits finally clearing up (this is after the commit of the transaction).
> StoreConnection.getExisting(Location).flush()
Hmm - looking at that, it needs a "synchronized" (on the Transaction
manager underlying operation)
Andy
>
> Thanks
> André
>
>