You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by Eric Charles <er...@apache.org> on 2010/11/30 06:41:05 UTC
[Discuss] mailbox-store responsibility
Hi,
The mailbox-store provides base/common logic to all stores.
There seems to be a trend to add more shared functions in it, namely:
1.- LastUidTracker replacing the getLastUid() method that was
implemented in each store.
2.-MailboxPathLocker used when creating/renaming mailbox. The extra
classes really brings bring ease when reading code and allows
implementation for distributed locking.
3.- MailboxIndex for search optimization (ongoing work)
I was also thinking to a 4th area, the UID management (UidProvider) we
talked before on https://issues.apache.org/jira/browse/IMAP-193.
Currently, UID are assigned by the store impl (if you use JPA, it must
be JPA and so on...). This is done in different ways (see code after for
jpa/jcr/maidir).
We could define a UidProvider to get rid of different locking mechanism
and have an efficient and distributed way to manage/generate the UIDs.
JPA/JCR/... and could benefit from this default implementation. (JPA
could probably be ameliorated with a auto_increment rather than a
database lock to manage the UID, but I think there will be a non-optimum
usage of the key space).
We may also discuss a layer implementing distributed MailboxListeners
(to have a base for distributed James, and transfer the needed
LastUidTracker,... events).
Are there any other areas we could define as the mailbox-store
responsibility?
Tks,
Eric
Locking for new UID for JPAMessageMapper
======================================
protected long reserveUid(JPAMailbox mailbox) {
EntityManager manager = getEntityManager();
// we need to set a pessimistic write lock to be sure we don't
get any problems with dirty reads etc.
JPAMailbox m = manager.find(JPAMailbox.class,
mailbox.getMailboxId(), LockModeType.PESSIMISTIC_WRITE);
manager.refresh(m);
m.consumeUid();
manager.persist(m);
return m.getLastUid();
}
}
Locking for new UID for JCRMessageMapper
=======================================
protected long reserveNextUid(Mailbox<String> mailbox) throws
RepositoryException, InterruptedException {
long result = getNodeLocker().execute(new
NodeLocker.NodeLockedExecution<Long>() {
public Long execute(Node node) throws RepositoryException {
Property uidProp =
node.getProperty(JCRMailbox.LASTUID_PROPERTY);
long uid = uidProp.getLong();
uid++;
uidProp.setValue(uid);
uidProp.getSession().save();
return uid;
}
public boolean isDeepLocked() {
return true;
}
}, getSession().getNodeByIdentifier(mailbox.getMailboxId()),
Long.class);
return result;
}
Locking for new UID for MaildirMessageMapper (in MaildirFolder)
=========================================================
/**
* Locks the uidList file and if it has retrieved the lock, returns it.
* Make sure to call unlockUidList() in a finally block afterwards.
* @return The locked uidList
*/
public File lockUidList() {
FileLock.lock(uidFile);
return uidFile;
}
/**
* Unlocks the uidList file if it has been locked before.
*/
public void unlockUidList() {
FileLock.unlock(uidFile);
}
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [Discuss] mailbox-store responsibility
Posted by Eric Charles <er...@apache.org>.
Hi Norman,
Yep, lucene available for all store impl would be great.
I feel that a shared distributed UIDProvider (also available for all
store impl) and the more "generic" distributed MailboxListener have
something in common (the one could be implemented by the second).
Tks,
Eric
On 30/11/2010 07:40, Norman Maurer wrote:
> Hi Eric,
>
> comments inside..
>
> 2010/11/30 Eric Charles<er...@apache.org>:
>> Hi,
>>
>> The mailbox-store provides base/common logic to all stores.
>> There seems to be a trend to add more shared functions in it, namely:
>>
>> 1.- LastUidTracker replacing the getLastUid() method that was implemented in
>> each store.
>>
>> 2.-MailboxPathLocker used when creating/renaming mailbox. The extra classes
>> really brings bring ease when reading code and allows implementation for
>> distributed locking.
>>
>> 3.- MailboxIndex for search optimization (ongoing work)
> I think you talk about the lucene index stuff which I proposed for
> Maildir. I would be in favor to make it generic so that
> implementations can reuse or use their own search index if they like..
> So I think we are "in-line" here..
>
>> I was also thinking to a 4th area, the UID management (UidProvider) we
>> talked before on https://issues.apache.org/jira/browse/IMAP-193.
>> Currently, UID are assigned by the store impl (if you use JPA, it must be
>> JPA and so on...). This is done in different ways (see code after for
>> jpa/jcr/maidir).
>> We could define a UidProvider to get rid of different locking mechanism and
>> have an efficient and distributed way to manage/generate the UIDs.
>> JPA/JCR/... and could benefit from this default implementation. (JPA could
>> probably be ameliorated with a auto_increment rather than a database lock to
>> manage the UID, but I think there will be a non-optimum usage of the key
>> space).
> That makes sense. In most cases we could also just use the
> MailboxPathLocker with an AtomicLong for the generation. Just read the
> last uid for a mailbox on select or startup and then use the
> AtomicLong.
>
>
>> We may also discuss a layer implementing distributed MailboxListeners (to
>> have a base for distributed James, and transfer the needed
>> LastUidTracker,... events).
> I already have something in mind here
>
>
>> Are there any other areas we could define as the mailbox-store
>> responsibility?
>>
>> Tks,
>>
>> Eric
>>
>> Locking for new UID for JPAMessageMapper
>> ======================================
>> protected long reserveUid(JPAMailbox mailbox) {
>> EntityManager manager = getEntityManager();
>>
>> // we need to set a pessimistic write lock to be sure we don't get
>> any problems with dirty reads etc.
>> JPAMailbox m = manager.find(JPAMailbox.class, mailbox.getMailboxId(),
>> LockModeType.PESSIMISTIC_WRITE);
>> manager.refresh(m);
>> m.consumeUid();
>> manager.persist(m);
>> return m.getLastUid();
>> }
>> }
>>
>> Locking for new UID for JCRMessageMapper
>> =======================================
>> protected long reserveNextUid(Mailbox<String> mailbox) throws
>> RepositoryException, InterruptedException {
>> long result = getNodeLocker().execute(new
>> NodeLocker.NodeLockedExecution<Long>() {
>>
>> public Long execute(Node node) throws RepositoryException {
>> Property uidProp =
>> node.getProperty(JCRMailbox.LASTUID_PROPERTY);
>> long uid = uidProp.getLong();
>> uid++;
>> uidProp.setValue(uid);
>> uidProp.getSession().save();
>> return uid;
>> }
>>
>> public boolean isDeepLocked() {
>> return true;
>> }
>>
>> }, getSession().getNodeByIdentifier(mailbox.getMailboxId()),
>> Long.class);
>> return result;
>>
>> }
>>
>>
>> Locking for new UID for MaildirMessageMapper (in MaildirFolder)
>> =========================================================
>>
>> /**
>> * Locks the uidList file and if it has retrieved the lock, returns it.
>> * Make sure to call unlockUidList() in a finally block afterwards.
>> * @return The locked uidList
>> */
>> public File lockUidList() {
>> FileLock.lock(uidFile);
>> return uidFile;
>> }
>>
>> /**
>> * Unlocks the uidList file if it has been locked before.
>> */
>> public void unlockUidList() {
>> FileLock.unlock(uidFile);
>> }
>>
>>
>>
> Thx for your comments!
>
> Bye,
> Norman
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [Discuss] mailbox-store responsibility
Posted by Norman Maurer <no...@apache.org>.
Hi Eric,
comments inside..
2010/11/30 Eric Charles <er...@apache.org>:
> Hi,
>
> The mailbox-store provides base/common logic to all stores.
> There seems to be a trend to add more shared functions in it, namely:
>
> 1.- LastUidTracker replacing the getLastUid() method that was implemented in
> each store.
>
> 2.-MailboxPathLocker used when creating/renaming mailbox. The extra classes
> really brings bring ease when reading code and allows implementation for
> distributed locking.
>
> 3.- MailboxIndex for search optimization (ongoing work)
I think you talk about the lucene index stuff which I proposed for
Maildir. I would be in favor to make it generic so that
implementations can reuse or use their own search index if they like..
So I think we are "in-line" here..
>
> I was also thinking to a 4th area, the UID management (UidProvider) we
> talked before on https://issues.apache.org/jira/browse/IMAP-193.
> Currently, UID are assigned by the store impl (if you use JPA, it must be
> JPA and so on...). This is done in different ways (see code after for
> jpa/jcr/maidir).
> We could define a UidProvider to get rid of different locking mechanism and
> have an efficient and distributed way to manage/generate the UIDs.
> JPA/JCR/... and could benefit from this default implementation. (JPA could
> probably be ameliorated with a auto_increment rather than a database lock to
> manage the UID, but I think there will be a non-optimum usage of the key
> space).
That makes sense. In most cases we could also just use the
MailboxPathLocker with an AtomicLong for the generation. Just read the
last uid for a mailbox on select or startup and then use the
AtomicLong.
>
> We may also discuss a layer implementing distributed MailboxListeners (to
> have a base for distributed James, and transfer the needed
> LastUidTracker,... events).
I already have something in mind here
>
> Are there any other areas we could define as the mailbox-store
> responsibility?
>
> Tks,
>
> Eric
>
> Locking for new UID for JPAMessageMapper
> ======================================
> protected long reserveUid(JPAMailbox mailbox) {
> EntityManager manager = getEntityManager();
>
> // we need to set a pessimistic write lock to be sure we don't get
> any problems with dirty reads etc.
> JPAMailbox m = manager.find(JPAMailbox.class, mailbox.getMailboxId(),
> LockModeType.PESSIMISTIC_WRITE);
> manager.refresh(m);
> m.consumeUid();
> manager.persist(m);
> return m.getLastUid();
> }
> }
>
> Locking for new UID for JCRMessageMapper
> =======================================
> protected long reserveNextUid(Mailbox<String> mailbox) throws
> RepositoryException, InterruptedException {
> long result = getNodeLocker().execute(new
> NodeLocker.NodeLockedExecution<Long>() {
>
> public Long execute(Node node) throws RepositoryException {
> Property uidProp =
> node.getProperty(JCRMailbox.LASTUID_PROPERTY);
> long uid = uidProp.getLong();
> uid++;
> uidProp.setValue(uid);
> uidProp.getSession().save();
> return uid;
> }
>
> public boolean isDeepLocked() {
> return true;
> }
>
> }, getSession().getNodeByIdentifier(mailbox.getMailboxId()),
> Long.class);
> return result;
>
> }
>
>
> Locking for new UID for MaildirMessageMapper (in MaildirFolder)
> =========================================================
>
> /**
> * Locks the uidList file and if it has retrieved the lock, returns it.
> * Make sure to call unlockUidList() in a finally block afterwards.
> * @return The locked uidList
> */
> public File lockUidList() {
> FileLock.lock(uidFile);
> return uidFile;
> }
>
> /**
> * Unlocks the uidList file if it has been locked before.
> */
> public void unlockUidList() {
> FileLock.unlock(uidFile);
> }
>
>
>
Thx for your comments!
Bye,
Norman
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org