You are viewing a plain text version of this content. The canonical link for it is here.
Posted to slide-dev@jakarta.apache.org by Oliver Zeigermann <oz...@c1-fse.de> on 2003/10/09 13:03:44 UTC
New transactional file store
Dear Slide community,
my employer "C1 Financial Services GmbH" has donated a transactional
file store to the Slide project. It should be in the CVS by now and be
scheduled for the 2.0 release. It is based on the reference
implementation slidestore.file.XMLFileDescriptorsStore and
slidestore.reference.FileContentStore. The replacements for theses
stores are org.apache.slide.store.txfile.TxXMLFileDescriptorsStore for
the descriptors and org.apache.slide.store.txfile.TxFileContentStore for
the content.
Its application includes development and out-of-the-box releases.
To achieve transactional support all changes are done to a working
directory first. Upon rollback, this directory will simply be deleted.
Upon commit it will be relinked or copied to the main store directory
using a mechanism that guarantees atomicy. That's why besides the main
store directory as known from XMLFileDescriptorsStore and
FileContentStore you will have to specify a working directory as well.
Be sure to have it located in the same filesystem as the main directory.
This allows for relinks instead of slower and more error-prone copies.
As an example, the definition part in Domain.xml may look similar to this:
<definition>
<store name="tx">
<nodestore
classname="org.apache.slide.store.txfile.TxXMLFileDescriptorsStore">
<!--nodestore classname="slidestore.file.XMLFileDescriptorsStore"-->
<parameter name="rootpath">E:/tmp/test/store/node</parameter>
<parameter name="workpath">E:/tmp/test/work/node</parameter>
<parameter name="encoding">UTF-8</parameter>
</nodestore>
<securitystore>
<reference store="nodestore"/>
</securitystore>
<lockstore>
<reference store="nodestore"/>
</lockstore>
<revisiondescriptorsstore>
<reference store="nodestore"/>
</revisiondescriptorsstore>
<revisiondescriptorstore>
<reference store="nodestore"/>
</revisiondescriptorstore>
<contentstore
classname="org.apache.slide.store.txfile.TxFileContentStore">
<!--contentstore classname="slidestore.reference.FileContentStore"-->
<parameter name="rootpath">E:/tmp/test/store/content</parameter>
<parameter name="workpath">E:/tmp/test/work/content</parameter>
</contentstore>
</store>
<scope match="/" store="tx"/>
</definition>
There still seem to be some bugs left, presumably mainly taken over from
the reference implementation. We will do our best to further analyze and
fix them.
Regards,
Oliver
---------------------------------------------------------------------
To unsubscribe, e-mail: slide-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: slide-dev-help@jakarta.apache.org
Re: New transactional file store
Posted by Oliver Zeigermann <oz...@c1-fse.de>.
Oliver Zeigermann wrote:
> You are right. This exception should have been displayed in logging, but
> ResourceManagerException can not handle nested exceptions to be
> comaptible with Java 1.3. Sorry :(
> Need to fix this, but do not know how, maybe incorporating the stack
> trace in the message...
Fixed this now. Cause will be added to the message itself.
Peter, could you check the patch in for me?
Thanks in advance,
Oliver
Re: New transactional file store
Posted by Oliver Zeigermann <oz...@c1-fse.de>.
Oliver Zeigermann wrote:
> We know the store is not in stable state, yet. So, please continue to
> report errors.
Next step to a stable store.
FIXES:
AbstractTxFileStoreService:
- Transaction will be marked for rollback upon system error or deadlock
- minor flaws
StoreLogger:
- logging of message and throwable will result in two log methods in
Slide logging
TxFileContentStore:
- adaptions to new system error handling in AbstractTxFileStoreService
- streams get closed in finally blocks to assure all locks in
FileResouceManager get freed
- minor flaws
TxXMLFileDescriptorsStore:
- adaptions to new system error handling in AbstractTxFileStoreService
XMLResourceDescriptor:
- adaptions to new system error handling in AbstractTxFileStoreService
- streams get closed in finally blocks to assure all locks in
FileResouceManager get freed
- made it thread safe
- added a tiny hack (which is to be investigated further)
FileResourceManager:
- freeing locks and resource when tx is marked for rollback to avoid
dangling locks when app misses to actually roll back the tx
Peter, would you...? I hardly dare to ask...
Thanks in advance,
Oliver
Re: New transactional file store
Posted by Oliver Zeigermann <oz...@c1-fse.de>.
Hi Martin,
glad to here it works now :)
We know the store is not in stable state, yet. So, please continue to
report errors.
We have released the store early in development as we depend on people
like you to help us make it stable.
Oliver
Martin Holz wrote:
> Hello Oliver,
>
> sorry about the noise. It was a configuration error. I had two
> stores, which used the same working directory.
> Now everything is fine.
>
> Oliver Zeigermann <oz...@c1-fse.de> writes:
>
>
>>Martin Holz writes:
>>
>>>A minor point, shouldn't you use
>>
>>> File dir = new File(workDir,txId)
>>>instead of
>>> File dir = new File(workDir + "/" + txId)
>>>at line 11397 to make sure the program runs on Windows?.
>>
>>Well, I actually developed the store on Windows. So it does run on
>>Windows :)
>>
>>Did you mean the "/" vs. "\" - this is no problem on Windows any more
>
>
> Yes, but I don't do much development on Windows.
>
>
>>Would you say using ctor File(String, String) generally is better
>>practice than concatenating paths using "/"?
>
>
> That's what the people at Sun thought. In theory it would protect
> you against strange operating systems.
>
> --
> Martin Holz <ho...@fiz-chemie.de>
>
> Softwareentwicklung / Vernetztes Studium - Chemie
> FIZ CHEMIE Berlin
> Franklinstrasse 11
> D-10587 Berlin
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: slide-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: slide-dev-help@jakarta.apache.org
>
>
> .
>
---------------------------------------------------------------------
To unsubscribe, e-mail: slide-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: slide-dev-help@jakarta.apache.org
Re: New transactional file store
Posted by Martin Holz <ho...@fiz-chemie.de>.
Hello Oliver,
sorry about the noise. It was a configuration error. I had two
stores, which used the same working directory.
Now everything is fine.
Oliver Zeigermann <oz...@c1-fse.de> writes:
> Martin Holz writes:
> > A minor point, shouldn't you use
>
> > File dir = new File(workDir,txId)
> > instead of
> > File dir = new File(workDir + "/" + txId)
> > at line 11397 to make sure the program runs on Windows?.
>
> Well, I actually developed the store on Windows. So it does run on
> Windows :)
>
> Did you mean the "/" vs. "\" - this is no problem on Windows any more
Yes, but I don't do much development on Windows.
> Would you say using ctor File(String, String) generally is better
> practice than concatenating paths using "/"?
That's what the people at Sun thought. In theory it would protect
you against strange operating systems.
--
Martin Holz <ho...@fiz-chemie.de>
Softwareentwicklung / Vernetztes Studium - Chemie
FIZ CHEMIE Berlin
Franklinstrasse 11
D-10587 Berlin
---------------------------------------------------------------------
To unsubscribe, e-mail: slide-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: slide-dev-help@jakarta.apache.org
Re: New transactional file store
Posted by Oliver Zeigermann <oz...@c1-fse.de>.
Hallo Martin,
thanks for the quick reply.
Martin Holz wrote:
> Oliver Zeigermann <oz...@c1-fse.de> writes:
>
>
>>>>There still seem to be some bugs left, presumably mainly taken over from
>>>>the reference implementation. We will do our best to further analyze and
>>>>fix them.
>>>
>>>FileResourceManager.saveState() fails for me sometimes,
>>
>>>because it tries to create a file in a nonexisting directory.
>>
>>
>>Could you give a stacktrace and/or any information under which
>>circumstances this happens?
>
>
> I did a simple put.
> Adding a e.printStackTrace at line 1406() gives the following output.
You are right. This exception should have been displayed in logging, but
ResourceManagerException can not handle nested exceptions to be
comaptible with Java 1.3. Sorry :(
Need to fix this, but do not know how, maybe incorporating the stack
trace in the message...
> The work directory exists, but work/Thread-13-1066205894564-11
> does not exist. I checked, that the directory does not exists
> using dir.exists(). Creating this directory will cure the problem
> (or at least the syntoms).
This would only cure the symptom. The code on lines 441 and 442
> context = new TransactionContext(txId);
> context.init();
ensure directories are created. It looks like the transaction has been
cleared before there is a commit. Are you sure there is no other error
before this one?
I could not reproduce the error, but do not think this is due to NFS and
Linux, but my last resort will be to check it under this environment.
As a first approach, you might try to increase the timeout in line 117
as 2 seconds actually are far too little. Try 200 seconds to be on the
save side. Maybe this helps:
> protected static final int DEFAULT_TIMEOUT_MSECS = 200000;
> I am using Java 1.4.1 on a Suse Linux 8.2 / Intel Single CPU.
> Both the work and the data directory live on the same partition,
> which is mounted via NFS.
>
> java.io.FileNotFoundException: /home/holz/slidedata/data/work/Thread-13-1066205894564-11/transaction.log
> (No such file or directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.<init>(FileOutputStream.java:176)
> at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
> at org.apache.slide.store.txfile.rm.impl.FileResourceManager$TransactionContext.saveState(FileRe
> sourceManager.java:1401)
> at org.apache.slide.store.txfile.rm.impl.FileResourceManager.commitTransaction(FileResourceManag
> er.java:561)
> at org.apache.slide.store.txfile.AbstractTxFileStoreService.commit(AbstractTxFileStoreService.ja
> va:180)
> at org.apache.slide.transaction.SlideTransaction.commit(SlideTransaction.java:347)
> at org.apache.slide.transaction.SlideTransactionManager.commit(SlideTransactionManager.java:230)
> at org.apache.slide.common.NamespaceAccessTokenImpl.commit(NamespaceAccessTokenImpl.java:462)
> at org.apache.slide.webdav.method.AbstractWebdavMethod.run(AbstractWebdavMethod.java:356)
> at org.apache.slide.webdav.WebdavServlet.service(WebdavServlet.java:201)
>
>
> Finally a HeuristicMixedException is thrown and no more operations are possible,
> until I restart the webdav servlet.
>
> A minor point, shouldn't you use
> File dir = new File(workDir,txId)
> instead of
> File dir = new File(workDir + "/" + txId)
> at line 11397 to make sure the program runs on Windows?.
Well, I actually developed the store on Windows. So it does run on
Windows :)
Did you mean the "/" vs. "\" - this is no problem on Windows any more -
or something else I was missing?
Would you say using ctor File(String, String) generally is better
practice than concatenating paths using "/"? What if you have more than
two paths to concat like in 1397:
> String statePath = workDir + "/" + txId + "/" + CONTEXT_FILE;
I am just wondering and discussing this issue, I do not try to defend my
stuff...
I will come back to you when I have more information. For nowt hanks for
you cooperation :)
Oliver
---------------------------------------------------------------------
To unsubscribe, e-mail: slide-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: slide-dev-help@jakarta.apache.org
Re: New transactional file store
Posted by Martin Holz <ho...@fiz-chemie.de>.
Oliver Zeigermann <oz...@c1-fse.de> writes:
>
> >>There still seem to be some bugs left, presumably mainly taken over from
> >>the reference implementation. We will do our best to further analyze and
> >>fix them.
> > FileResourceManager.saveState() fails for me sometimes,
>
> > because it tries to create a file in a nonexisting directory.
>
>
> Could you give a stacktrace and/or any information under which
> circumstances this happens?
I did a simple put.
Adding a e.printStackTrace at line 1406() gives the following output.
The work directory exists, but work/Thread-13-1066205894564-11
does not exist. I checked, that the directory does not exists
using dir.exists(). Creating this directory will cure the problem
(or at least the syntoms).
I am using Java 1.4.1 on a Suse Linux 8.2 / Intel Single CPU.
Both the work and the data directory live on the same partition,
which is mounted via NFS.
java.io.FileNotFoundException: /home/holz/slidedata/data/work/Thread-13-1066205894564-11/transaction.log
(No such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:176)
at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
at org.apache.slide.store.txfile.rm.impl.FileResourceManager$TransactionContext.saveState(FileRe
sourceManager.java:1401)
at org.apache.slide.store.txfile.rm.impl.FileResourceManager.commitTransaction(FileResourceManag
er.java:561)
at org.apache.slide.store.txfile.AbstractTxFileStoreService.commit(AbstractTxFileStoreService.ja
va:180)
at org.apache.slide.transaction.SlideTransaction.commit(SlideTransaction.java:347)
at org.apache.slide.transaction.SlideTransactionManager.commit(SlideTransactionManager.java:230)
at org.apache.slide.common.NamespaceAccessTokenImpl.commit(NamespaceAccessTokenImpl.java:462)
at org.apache.slide.webdav.method.AbstractWebdavMethod.run(AbstractWebdavMethod.java:356)
at org.apache.slide.webdav.WebdavServlet.service(WebdavServlet.java:201)
Finally a HeuristicMixedException is thrown and no more operations are possible,
until I restart the webdav servlet.
A minor point, shouldn't you use
File dir = new File(workDir,txId)
instead of
File dir = new File(workDir + "/" + txId)
at line 11397 to make sure the program runs on Windows?.
--
Martin Holz <ho...@fiz-chemie.de>
Softwareentwicklung / Vernetztes Studium - Chemie
FIZ CHEMIE Berlin
Franklinstrasse 11
D-10587 Berlin
---------------------------------------------------------------------
To unsubscribe, e-mail: slide-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: slide-dev-help@jakarta.apache.org
Re: New transactional file store
Posted by Oliver Zeigermann <oz...@c1-fse.de>.
Martin Holz wrote:
>>my employer "C1 Financial Services GmbH" has donated a transactional
>>file store to the Slide project. It should be in the CVS by now and be
>>scheduled for the 2.0 release. It is based on the reference
>>implementation slidestore.file.XMLFileDescriptorsStore and
>>slidestore.reference.FileContentStore. The replacements for theses
>>stores are org.apache.slide.store.txfile.TxXMLFileDescriptorsStore for
>>the descriptors and org.apache.slide.store.txfile.TxFileContentStore for
>>the content.
>
>
> Great work. Thank you very much.
Thanks! You are welcome.
>
>>There still seem to be some bugs left, presumably mainly taken over from
>>the reference implementation. We will do our best to further analyze and
>>fix them.
>
>
> FileResourceManager.saveState() fails for me sometimes,
> because it tries to create a file in a nonexisting
> directory.
Could you give a stacktrace and/or any information under which
circumstances this happens?
Thanks in advance,
Oliver
---------------------------------------------------------------------
To unsubscribe, e-mail: slide-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: slide-dev-help@jakarta.apache.org
Re: New transactional file store
Posted by Martin Holz <ho...@fiz-chemie.de>.
Oliver Zeigermann <oz...@c1-fse.de> writes:
> Dear Slide community,
>
> my employer "C1 Financial Services GmbH" has donated a transactional
> file store to the Slide project. It should be in the CVS by now and be
> scheduled for the 2.0 release. It is based on the reference
> implementation slidestore.file.XMLFileDescriptorsStore and
> slidestore.reference.FileContentStore. The replacements for theses
> stores are org.apache.slide.store.txfile.TxXMLFileDescriptorsStore for
> the descriptors and org.apache.slide.store.txfile.TxFileContentStore for
> the content.
Great work. Thank you very much.
> There still seem to be some bugs left, presumably mainly taken over from
> the reference implementation. We will do our best to further analyze and
> fix them.
FileResourceManager.saveState() fails for me sometimes,
because it tries to create a file in a nonexisting
directory.
--
Martin Holz <ho...@fiz-chemie.de>
Softwareentwicklung / Vernetztes Studium - Chemie
FIZ CHEMIE Berlin
Franklinstrasse 11
D-10587 Berlin
---------------------------------------------------------------------
To unsubscribe, e-mail: slide-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: slide-dev-help@jakarta.apache.org
Re: New transactional file store
Posted by Oliver Zeigermann <oz...@c1-fse.de>.
Stuff I forgot to mention:
IMPORTANT: Every store needs a working directory it is guarantees to
have exclusive access to.
The root (main) directory might be shared with other application.
Although, concurrent writing from other applications is discouraged as
they might interfere with each other.
It might also be worth noting that this new store should be fully
compatible with slidestore.reference.FileContentStore and
slidestore.file.XMLFileDescriptorsStore. Configured to the same
rootpath, data stored by these should work with the new store and
vice-versa.
As for isolation: The underlying FileResourceManager supports
READ_COMMITED and REPEATABLE_READ. Both can be configured by per
transaction. If all transactions use REPEATABLE_READ when writing,
serializability is guaranteed. That is how
org.apache.slide.store.txfile.TxXMLFileDescriptorsStore and
org.apache.slide.store.txfile.TxFileContentStore use it, so all access
should be completely isolated.
Oliver
Oliver Zeigermann wrote:
> Dear Slide community,
>
> my employer "C1 Financial Services GmbH" has donated a transactional
> file store to the Slide project. It should be in the CVS by now and be
> scheduled for the 2.0 release. It is based on the reference
> implementation slidestore.file.XMLFileDescriptorsStore and
> slidestore.reference.FileContentStore. The replacements for theses
> stores are org.apache.slide.store.txfile.TxXMLFileDescriptorsStore for
> the descriptors and org.apache.slide.store.txfile.TxFileContentStore for
> the content.
>
> Its application includes development and out-of-the-box releases.
>
> To achieve transactional support all changes are done to a working
> directory first. Upon rollback, this directory will simply be deleted.
> Upon commit it will be relinked or copied to the main store directory
> using a mechanism that guarantees atomicy. That's why besides the main
> store directory as known from XMLFileDescriptorsStore and
> FileContentStore you will have to specify a working directory as well.
> Be sure to have it located in the same filesystem as the main directory.
> This allows for relinks instead of slower and more error-prone copies.
>
> As an example, the definition part in Domain.xml may look similar to this:
>
> <definition>
> <store name="tx">
> <nodestore
> classname="org.apache.slide.store.txfile.TxXMLFileDescriptorsStore">
> <!--nodestore classname="slidestore.file.XMLFileDescriptorsStore"-->
> <parameter name="rootpath">E:/tmp/test/store/node</parameter>
> <parameter name="workpath">E:/tmp/test/work/node</parameter>
> <parameter name="encoding">UTF-8</parameter>
> </nodestore>
> <securitystore>
> <reference store="nodestore"/>
> </securitystore>
> <lockstore>
> <reference store="nodestore"/>
> </lockstore>
> <revisiondescriptorsstore>
> <reference store="nodestore"/>
> </revisiondescriptorsstore>
> <revisiondescriptorstore>
> <reference store="nodestore"/>
> </revisiondescriptorstore>
> <contentstore
> classname="org.apache.slide.store.txfile.TxFileContentStore">
> <!--contentstore classname="slidestore.reference.FileContentStore"-->
> <parameter name="rootpath">E:/tmp/test/store/content</parameter>
> <parameter name="workpath">E:/tmp/test/work/content</parameter>
> </contentstore>
> </store>
> <scope match="/" store="tx"/>
> </definition>
>
> There still seem to be some bugs left, presumably mainly taken over from
> the reference implementation. We will do our best to further analyze and
> fix them.
>
> Regards,
>
> Oliver
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: slide-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: slide-user-help@jakarta.apache.org
>
>
> .
>
---------------------------------------------------------------------
To unsubscribe, e-mail: slide-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: slide-user-help@jakarta.apache.org