You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by "Norman Maurer (JIRA)" <se...@james.apache.org> on 2010/08/13 09:51:17 UTC
[jira] Created: (IMAP-193) UIDNEXT generation is a big performance
killer and should get reworked
UIDNEXT generation is a big performance killer and should get reworked
----------------------------------------------------------------------
Key: IMAP-193
URL: https://issues.apache.org/jira/browse/IMAP-193
Project: JAMES Imap
Issue Type: Improvement
Reporter: Norman Maurer
Assignee: Norman Maurer
In the store api we generate the uid for a new message and guaranteer that the Mailbox.getLastUid() method will return this value after it is stored. For this we save the message and the mailbox and use for example a row lock (in jpa land) for this to get rid of ghost-reads etc.
This is a big performance killer and is not needed at all. In the rfc its clearly stated that the used uid for the next saved message must just be equal or bigger then the UIDNEXT value.
>From rfc:
Note: The next unique identifier value is intended to
provide a means for a client to determine whether any
messages have been delivered to the mailbox since the
previous time it checked this value. It is not intended to
provide any guarantee that any message will have this
unique identifier. A client can only assume, at the time
that it obtains the next unique identifier value, that
messages arriving after that time will have a UID greater
than or equal to that value.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[jira] Updated: (IMAP-193) UIDNEXT generation is a big performance
killer and should get reworked
Posted by "Norman Maurer (JIRA)" <se...@james.apache.org>.
[ https://issues.apache.org/jira/browse/IMAP-193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Norman Maurer updated IMAP-193:
-------------------------------
Fix Version/s: 0.2
Affects Version/s: 0.1
> UIDNEXT generation is a big performance killer and should get reworked
> ----------------------------------------------------------------------
>
> Key: IMAP-193
> URL: https://issues.apache.org/jira/browse/IMAP-193
> Project: JAMES Imap
> Issue Type: Improvement
> Affects Versions: 0.1
> Reporter: Norman Maurer
> Assignee: Norman Maurer
> Fix For: 0.2
>
>
> In the store api we generate the uid for a new message and guaranteer that the Mailbox.getLastUid() method will return this value after it is stored. For this we save the message and the mailbox and use for example a row lock (in jpa land) for this to get rid of ghost-reads etc.
> This is a big performance killer and is not needed at all. In the rfc its clearly stated that the used uid for the next saved message must just be equal or bigger then the UIDNEXT value.
> From rfc:
> Note: The next unique identifier value is intended to
> provide a means for a client to determine whether any
> messages have been delivered to the mailbox since the
> previous time it checked this value. It is not intended to
> provide any guarantee that any message will have this
> unique identifier. A client can only assume, at the time
> that it obtains the next unique identifier value, that
> messages arriving after that time will have a UID greater
> than or equal to that value.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[jira] Updated: (IMAP-193) UIDNEXT generation is a big performance
killer and should get reworked
Posted by "Norman Maurer (JIRA)" <se...@james.apache.org>.
[ https://issues.apache.org/jira/browse/IMAP-193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Norman Maurer updated IMAP-193:
-------------------------------
Fix Version/s: (was: 0.2)
0.2-M1
> UIDNEXT generation is a big performance killer and should get reworked
> ----------------------------------------------------------------------
>
> Key: IMAP-193
> URL: https://issues.apache.org/jira/browse/IMAP-193
> Project: JAMES Imap
> Issue Type: Improvement
> Affects Versions: 0.1
> Reporter: Norman Maurer
> Assignee: Norman Maurer
> Fix For: 0.2-M1
>
>
> In the store api we generate the uid for a new message and guaranteer that the Mailbox.getLastUid() method will return this value after it is stored. For this we save the message and the mailbox and use for example a row lock (in jpa land) for this to get rid of ghost-reads etc.
> This is a big performance killer and is not needed at all. In the rfc its clearly stated that the used uid for the next saved message must just be equal or bigger then the UIDNEXT value.
> From rfc:
> Note: The next unique identifier value is intended to
> provide a means for a client to determine whether any
> messages have been delivered to the mailbox since the
> previous time it checked this value. It is not intended to
> provide any guarantee that any message will have this
> unique identifier. A client can only assume, at the time
> that it obtains the next unique identifier value, that
> messages arriving after that time will have a UID greater
> than or equal to that value.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[jira] Commented: (IMAP-193) UIDNEXT generation is a big
performance killer and should get reworked
Posted by "Norman Maurer (JIRA)" <se...@james.apache.org>.
[ https://issues.apache.org/jira/browse/IMAP-193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913684#action_12913684 ]
Norman Maurer commented on IMAP-193:
------------------------------------
Ok, I think I found a good solution for this after thinking a bit about it today. Its so "simple" that it was really hard to get it ;)
So here what we should do:
* Let us remove the Mailbox.getLastUid() method and not make it persistent.
* Simply return "1" when the Mailbox is selected first. After that on the next append to the mailbox we can set the nextUid value to returned uid + 1.
* Register a MailboxListener to the Mailbox when selecting it to get notified by append operations to the mailbox (in concurrent sessions) and set it when a append happen.
Then the implementations are free to use whatever method/pattern they like to generate the uid on append. For example in JPA we could just use a auto_increment field. This would allow us to get rid of locking the row and so give better performance. For sure we need to update the unit tests to allow this kind of generation of uid when append.
I think this is a way better then what we do now and is fully rfc conform (See RFC snipped in previous comment)
WDYT ?
> UIDNEXT generation is a big performance killer and should get reworked
> ----------------------------------------------------------------------
>
> Key: IMAP-193
> URL: https://issues.apache.org/jira/browse/IMAP-193
> Project: JAMES Imap
> Issue Type: Improvement
> Reporter: Norman Maurer
> Assignee: Norman Maurer
>
> In the store api we generate the uid for a new message and guaranteer that the Mailbox.getLastUid() method will return this value after it is stored. For this we save the message and the mailbox and use for example a row lock (in jpa land) for this to get rid of ghost-reads etc.
> This is a big performance killer and is not needed at all. In the rfc its clearly stated that the used uid for the next saved message must just be equal or bigger then the UIDNEXT value.
> From rfc:
> Note: The next unique identifier value is intended to
> provide a means for a client to determine whether any
> messages have been delivered to the mailbox since the
> previous time it checked this value. It is not intended to
> provide any guarantee that any message will have this
> unique identifier. A client can only assume, at the time
> that it obtains the next unique identifier value, that
> messages arriving after that time will have a UID greater
> than or equal to that value.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[jira] Commented: (IMAP-193) UIDNEXT generation is a big
performance killer and should get reworked
Posted by "Eric Charles (JIRA)" <se...@james.apache.org>.
[ https://issues.apache.org/jira/browse/IMAP-193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913985#action_12913985 ]
Eric Charles commented on IMAP-193:
-----------------------------------
I had to reread a few times to realize the meaning of:
*It is not intended to provide any guarantee that any message will have this unique identifier*
So UIDNEXT is a mechanism to know if new messages are delivered in that mailbox.
The only constraint is that it must be lower or equal than the UID that will be given to a message coming in the future.
So the "simple", and still difficult to find :) way you described maps the requirements.
The JPA example is a simple one because you can use the auto_increment provided by the DB.
For store not having that function such as maildir, jcr, nosql,...we should try to work with injection, meaning that an interface representing the UidNextProvider (whatever its name is).
Some mailbox store such as JPA would be injected with a JPAUidNextProvider, others such as JCR,... could use a FileUidNextProvider for example, or whate.
I suppose the injected component would be a singleton, would work with a map (mailboxId, lastUid) and it will have to be thread-safe.
How do you see the impl ?
> UIDNEXT generation is a big performance killer and should get reworked
> ----------------------------------------------------------------------
>
> Key: IMAP-193
> URL: https://issues.apache.org/jira/browse/IMAP-193
> Project: JAMES Imap
> Issue Type: Improvement
> Affects Versions: 0.1
> Reporter: Norman Maurer
> Assignee: Norman Maurer
> Fix For: 0.2
>
>
> In the store api we generate the uid for a new message and guaranteer that the Mailbox.getLastUid() method will return this value after it is stored. For this we save the message and the mailbox and use for example a row lock (in jpa land) for this to get rid of ghost-reads etc.
> This is a big performance killer and is not needed at all. In the rfc its clearly stated that the used uid for the next saved message must just be equal or bigger then the UIDNEXT value.
> From rfc:
> Note: The next unique identifier value is intended to
> provide a means for a client to determine whether any
> messages have been delivered to the mailbox since the
> previous time it checked this value. It is not intended to
> provide any guarantee that any message will have this
> unique identifier. A client can only assume, at the time
> that it obtains the next unique identifier value, that
> messages arriving after that time will have a UID greater
> than or equal to that value.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[jira] Reopened: (IMAP-193) UIDNEXT generation is a big performance
killer and should get reworked
Posted by "Norman Maurer (JIRA)" <se...@james.apache.org>.
[ https://issues.apache.org/jira/browse/IMAP-193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Norman Maurer reopened IMAP-193:
--------------------------------
This broke Maildir implementation.. I need to find out why :(
> UIDNEXT generation is a big performance killer and should get reworked
> ----------------------------------------------------------------------
>
> Key: IMAP-193
> URL: https://issues.apache.org/jira/browse/IMAP-193
> Project: JAMES Imap
> Issue Type: Improvement
> Affects Versions: 0.1
> Reporter: Norman Maurer
> Assignee: Norman Maurer
> Fix For: 0.2
>
>
> In the store api we generate the uid for a new message and guaranteer that the Mailbox.getLastUid() method will return this value after it is stored. For this we save the message and the mailbox and use for example a row lock (in jpa land) for this to get rid of ghost-reads etc.
> This is a big performance killer and is not needed at all. In the rfc its clearly stated that the used uid for the next saved message must just be equal or bigger then the UIDNEXT value.
> From rfc:
> Note: The next unique identifier value is intended to
> provide a means for a client to determine whether any
> messages have been delivered to the mailbox since the
> previous time it checked this value. It is not intended to
> provide any guarantee that any message will have this
> unique identifier. A client can only assume, at the time
> that it obtains the next unique identifier value, that
> messages arriving after that time will have a UID greater
> than or equal to that value.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[jira] Commented: (IMAP-193) UIDNEXT generation is a big
performance killer and should get reworked
Posted by "Norman Maurer (JIRA)" <se...@james.apache.org>.
[ https://issues.apache.org/jira/browse/IMAP-193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12914033#action_12914033 ]
Norman Maurer commented on IMAP-193:
------------------------------------
Yeah thats right... and we should not forgit this. But with auto_increment in the composite key we will get one "sequence" per mailbox.
> UIDNEXT generation is a big performance killer and should get reworked
> ----------------------------------------------------------------------
>
> Key: IMAP-193
> URL: https://issues.apache.org/jira/browse/IMAP-193
> Project: JAMES Imap
> Issue Type: Improvement
> Affects Versions: 0.1
> Reporter: Norman Maurer
> Assignee: Norman Maurer
> Fix For: 0.2
>
>
> In the store api we generate the uid for a new message and guaranteer that the Mailbox.getLastUid() method will return this value after it is stored. For this we save the message and the mailbox and use for example a row lock (in jpa land) for this to get rid of ghost-reads etc.
> This is a big performance killer and is not needed at all. In the rfc its clearly stated that the used uid for the next saved message must just be equal or bigger then the UIDNEXT value.
> From rfc:
> Note: The next unique identifier value is intended to
> provide a means for a client to determine whether any
> messages have been delivered to the mailbox since the
> previous time it checked this value. It is not intended to
> provide any guarantee that any message will have this
> unique identifier. A client can only assume, at the time
> that it obtains the next unique identifier value, that
> messages arriving after that time will have a UID greater
> than or equal to that value.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[jira] Resolved: (IMAP-193) UIDNEXT generation is a big performance
killer and should get reworked
Posted by "Norman Maurer (JIRA)" <se...@james.apache.org>.
[ https://issues.apache.org/jira/browse/IMAP-193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Norman Maurer resolved IMAP-193.
--------------------------------
Resolution: Fixed
Was fixed..
> UIDNEXT generation is a big performance killer and should get reworked
> ----------------------------------------------------------------------
>
> Key: IMAP-193
> URL: https://issues.apache.org/jira/browse/IMAP-193
> Project: JAMES Imap
> Issue Type: Improvement
> Affects Versions: 0.1
> Reporter: Norman Maurer
> Assignee: Norman Maurer
> Fix For: 0.2
>
>
> In the store api we generate the uid for a new message and guaranteer that the Mailbox.getLastUid() method will return this value after it is stored. For this we save the message and the mailbox and use for example a row lock (in jpa land) for this to get rid of ghost-reads etc.
> This is a big performance killer and is not needed at all. In the rfc its clearly stated that the used uid for the next saved message must just be equal or bigger then the UIDNEXT value.
> From rfc:
> Note: The next unique identifier value is intended to
> provide a means for a client to determine whether any
> messages have been delivered to the mailbox since the
> previous time it checked this value. It is not intended to
> provide any guarantee that any message will have this
> unique identifier. A client can only assume, at the time
> that it obtains the next unique identifier value, that
> messages arriving after that time will have a UID greater
> than or equal to that value.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[jira] Commented: (IMAP-193) UIDNEXT generation is a big
performance killer and should get reworked
Posted by "Eric Charles (JIRA)" <se...@james.apache.org>.
[ https://issues.apache.org/jira/browse/IMAP-193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913998#action_12913998 ]
Eric Charles commented on IMAP-193:
-----------------------------------
back again.
With the auto_increment to generate UID (for JPA), we may fall in the pitfall described on
One drawback of this approach is described in "IMAP4 Implementation Recommendations" http://www.rfc-editor.org/rfc/rfc2683.txt
Some server implementations have attempted to make UIDs unique across
the entire server. This is inadvisable, in that it limits the life
of UIDs unnecessarily. The UID is a 32-bit number and will run out
in reasonably finite time if it's global across the server. If you
assign UIDs sequentially in one mailbox, you will not have to start
re-using them until you have had, at one time or another, 2**32
different messages in that mailbox. In the global case, you will
have to reuse them once you have had, at one time or another, 2**32
different messages in the entire mail store. Suppose your server has
around 8000 users registered (2**13). That gives an average of 2**19
UIDs per user. Suppose each user gets 32 messages (2**5) per day.
That gives you 2**14 days (16000+ days = about 45 years) before you
run out. That may seem like enough, but multiply the usage just a
little (a lot of spam, a lot of mailing list subscriptions, more
users) and you limit yourself too much.
> UIDNEXT generation is a big performance killer and should get reworked
> ----------------------------------------------------------------------
>
> Key: IMAP-193
> URL: https://issues.apache.org/jira/browse/IMAP-193
> Project: JAMES Imap
> Issue Type: Improvement
> Affects Versions: 0.1
> Reporter: Norman Maurer
> Assignee: Norman Maurer
> Fix For: 0.2
>
>
> In the store api we generate the uid for a new message and guaranteer that the Mailbox.getLastUid() method will return this value after it is stored. For this we save the message and the mailbox and use for example a row lock (in jpa land) for this to get rid of ghost-reads etc.
> This is a big performance killer and is not needed at all. In the rfc its clearly stated that the used uid for the next saved message must just be equal or bigger then the UIDNEXT value.
> From rfc:
> Note: The next unique identifier value is intended to
> provide a means for a client to determine whether any
> messages have been delivered to the mailbox since the
> previous time it checked this value. It is not intended to
> provide any guarantee that any message will have this
> unique identifier. A client can only assume, at the time
> that it obtains the next unique identifier value, that
> messages arriving after that time will have a UID greater
> than or equal to that value.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[jira] Resolved: (IMAP-193) UIDNEXT generation is a big performance
killer and should get reworked
Posted by "Norman Maurer (JIRA)" <se...@james.apache.org>.
[ https://issues.apache.org/jira/browse/IMAP-193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Norman Maurer resolved IMAP-193.
--------------------------------
Resolution: Fixed
committed
> UIDNEXT generation is a big performance killer and should get reworked
> ----------------------------------------------------------------------
>
> Key: IMAP-193
> URL: https://issues.apache.org/jira/browse/IMAP-193
> Project: JAMES Imap
> Issue Type: Improvement
> Affects Versions: 0.1
> Reporter: Norman Maurer
> Assignee: Norman Maurer
> Fix For: 0.2
>
>
> In the store api we generate the uid for a new message and guaranteer that the Mailbox.getLastUid() method will return this value after it is stored. For this we save the message and the mailbox and use for example a row lock (in jpa land) for this to get rid of ghost-reads etc.
> This is a big performance killer and is not needed at all. In the rfc its clearly stated that the used uid for the next saved message must just be equal or bigger then the UIDNEXT value.
> From rfc:
> Note: The next unique identifier value is intended to
> provide a means for a client to determine whether any
> messages have been delivered to the mailbox since the
> previous time it checked this value. It is not intended to
> provide any guarantee that any message will have this
> unique identifier. A client can only assume, at the time
> that it obtains the next unique identifier value, that
> messages arriving after that time will have a UID greater
> than or equal to that value.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[jira] Commented: (IMAP-193) UIDNEXT generation is a big
performance killer and should get reworked
Posted by "Eric Charles (JIRA)" <se...@james.apache.org>.
[ https://issues.apache.org/jira/browse/IMAP-193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913993#action_12913993 ]
Eric Charles commented on IMAP-193:
-----------------------------------
oops, I was a bit confused.
The *UidNextProvider* would be by default a MemoryUidNextProvider.
For stores without auto_increment, I was also thinking to a *UidProvider* that would be injected.
This would be also injected with the choice impl (could be FileUidProvider, JPAUidProvider,... or whatever an imaginative developer could think to :).
> UIDNEXT generation is a big performance killer and should get reworked
> ----------------------------------------------------------------------
>
> Key: IMAP-193
> URL: https://issues.apache.org/jira/browse/IMAP-193
> Project: JAMES Imap
> Issue Type: Improvement
> Affects Versions: 0.1
> Reporter: Norman Maurer
> Assignee: Norman Maurer
> Fix For: 0.2
>
>
> In the store api we generate the uid for a new message and guaranteer that the Mailbox.getLastUid() method will return this value after it is stored. For this we save the message and the mailbox and use for example a row lock (in jpa land) for this to get rid of ghost-reads etc.
> This is a big performance killer and is not needed at all. In the rfc its clearly stated that the used uid for the next saved message must just be equal or bigger then the UIDNEXT value.
> From rfc:
> Note: The next unique identifier value is intended to
> provide a means for a client to determine whether any
> messages have been delivered to the mailbox since the
> previous time it checked this value. It is not intended to
> provide any guarantee that any message will have this
> unique identifier. A client can only assume, at the time
> that it obtains the next unique identifier value, that
> messages arriving after that time will have a UID greater
> than or equal to that value.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org