You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by Soren Hilmer <so...@tietoenator.com> on 2005/09/06 14:40:12 UTC
Spoolmanager blues
Hi,
I am having trouble with the JamesSpoolManager in the trunk.
I experience mails hanging in the spool, it looks like the offending piece of
code is the return statement in line 418. The reason I suspect that line is
that I cannot reproduce the effect if I remove the line.
Have anyone seen something similar, or can someone (Stefano) provide the
rationale behind this particular return statement.
I will do some more debugging to better understand what happens.
regards
Søren
--
Søren Hilmer, M.Sc.
R&D manager Phone: +45 72 30 64 00
TietoEnator IT+ A/S Fax: +45 72 30 64 02
Ved Lunden 12 Direct: +45 72 30 64 57
DK-8230 Åbyhøj Email: soren.hilmer <at> tietoenator.com
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Spoolmanager blues
Posted by Soren Hilmer <so...@tietoenator.com>.
On Tuesday 06 September 2005 15:12, Stefano Bagnara wrote:
> > I am having trouble with the JamesSpoolManager in the trunk.
> > I experience mails hanging in the spool, it looks like the
> > offending piece of code is the return statement in line 418.
> > The reason I suspect that line is that I cannot reproduce the
> > effect if I remove the line.
>
> If you remove the return you change the current behaviour.
Yes, I realize that. I just remembered (like Noel) that I had seen a patch a
while back which included this exact return, commented out.
>
> Currently each spool thread get a message from the spool, run it in the
> processor associated with the current spool and returns.
>
> If you remove the return then a single thread will bring the mail to GHOST
> in a single processing. At the end of each processor the LinearProcessor
> will store each mail in the spool (currently there are no drowbacks in
> storing the same message multiple time without accepting them again, but I
> did not changed it because I'm not sure it is a good thing)
>
> IMHO the problem is in the spool.unlock implementation for the file
> repository.
Yes, you are probably right.
>
> Stefano
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
--
Søren Hilmer, M.Sc.
R&D manager Phone: +45 72 30 64 00
TietoEnator IT+ A/S Fax: +45 72 30 64 02
Ved Lunden 12 Direct: +45 72 30 64 57
DK-8230 Åbyhøj Email: soren.hilmer <at> tietoenator.com
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Spoolmanager blues
Posted by Stefano Bagnara <ap...@bago.org>.
> I am having trouble with the JamesSpoolManager in the trunk.
> I experience mails hanging in the spool, it looks like the
> offending piece of code is the return statement in line 418.
> The reason I suspect that line is that I cannot reproduce the
> effect if I remove the line.
If you remove the return you change the current behaviour.
Currently each spool thread get a message from the spool, run it in the
processor associated with the current spool and returns.
If you remove the return then a single thread will bring the mail to GHOST
in a single processing. At the end of each processor the LinearProcessor
will store each mail in the spool (currently there are no drowbacks in
storing the same message multiple time without accepting them again, but I
did not changed it because I'm not sure it is a good thing)
IMHO the problem is in the spool.unlock implementation for the file
repository.
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Spoolmanager blues
Posted by Soren Hilmer <so...@tietoenator.com>.
I have now tried to downgrade to 2.2.0, and I cannot reproduce the effect
there (only tried filerepositories on this version).
I also tried using Derby/db on trunk, and saw "just" the delay.
I will try out your patch.
--Søren
On Wednesday 07 September 2005 11:30, Stefano Bagnara wrote:
> > > Can you try downgrading to 2.2.0 and verify wether the
> >
> > issue is there
> >
> > > or not?
> >
> > Will do that today.
> >
> > > Can you try using db/derby to check wether the issue is
> >
> > there or not?
> >
> >
> > Sure.
>
> I'm testing a patch to both JDBC and Avalon repositories.
> It seems working better than before. I'll run my stress-tests and
> eventually commit the code so you can test it, too!
>
> Stefano
>
> ------
>
>
> Index:
> james/src/java/org/apache/james/mailrepository/AvalonMailRepository.java
> ===================================================================
> ---
> james/src/java/org/apache/james/mailrepository/AvalonMailRepository.java
> (revision 233041)
> +++
> james/src/java/org/apache/james/mailrepository/AvalonMailRepository.java
> (working copy)
> @@ -205,6 +205,7 @@
> // synchronized (this) {
> // notifyAll();
> // }
> + notify();
> return true;
> } else {
> return false;
> Index:
> james/src/java/org/apache/james/mailrepository/JDBCMailRepository.java
> ===================================================================
> --- james/src/java/org/apache/james/mailrepository/JDBCMailRepository.java
> (revision 233041)
> +++ james/src/java/org/apache/james/mailrepository/JDBCMailRepository.java
> (working copy)
> @@ -494,7 +494,7 @@
> *
> * @return true if successfully released the lock, false otherwise
> */
> - public synchronized boolean unlock(String key) {
> + public boolean unlock(String key) {
> if (lock.unlock(key)) {
> if ((DEEP_DEBUG) && (getLogger().isDebugEnabled())) {
> StringBuffer debugBuffer =
> @@ -508,6 +508,7 @@
> getLogger().debug(debugBuffer.toString());
> }
> // notifyAll();
> + notify();
> return true;
> } else {
> return false;
> @@ -521,7 +522,7 @@
> *
> * @return true if successfully obtained the lock, false otherwise
> */
> - public synchronized boolean lock(String key) {
> + public boolean lock(String key) {
> if (lock.lock(key)) {
> if ((DEEP_DEBUG) && (getLogger().isDebugEnabled())) {
> StringBuffer debugBuffer =
> @@ -546,7 +547,17 @@
> */
> public void store(MailImpl mc) throws MessagingException {
> Connection conn = null;
> + boolean wasLocked = true;
> + String key = mc.getName();
> try {
> + synchronized(this) {
> + wasLocked = lock.isLocked(key);
> +
> + if (!wasLocked) {
> + //If it wasn't locked, we want a lock during the
> store
> + lock.lock(key);
> + }
> + }
> conn = datasource.getConnection();
>
> //Need to determine whether need to insert this record, or
> update it.
> @@ -775,6 +786,10 @@
> throw new MessagingException("Exception caught while storing
> mail Container: " + e);
> } finally {
> theJDBCUtil.closeJDBCConnection(conn);
> + if (!wasLocked) {
> + //If it wasn't locked, we need to now unlock
> + lock.unlock(key);
> + }
> }
> }
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
--
Søren Hilmer, M.Sc.
R&D manager Phone: +45 72 30 64 00
TietoEnator IT+ A/S Fax: +45 72 30 64 02
Ved Lunden 12 Direct: +45 72 30 64 57
DK-8230 Åbyhøj Email: soren.hilmer <at> tietoenator.com
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Spoolmanager blues
Posted by Stefano Bagnara <ap...@bago.org>.
> > Can you try downgrading to 2.2.0 and verify wether the
> issue is there
> > or not?
>
> Will do that today.
>
> > Can you try using db/derby to check wether the issue is
> there or not?
> >
>
> Sure.
I'm testing a patch to both JDBC and Avalon repositories.
It seems working better than before. I'll run my stress-tests and eventually
commit the code so you can test it, too!
Stefano
------
Index:
james/src/java/org/apache/james/mailrepository/AvalonMailRepository.java
===================================================================
--- james/src/java/org/apache/james/mailrepository/AvalonMailRepository.java
(revision 233041)
+++ james/src/java/org/apache/james/mailrepository/AvalonMailRepository.java
(working copy)
@@ -205,6 +205,7 @@
// synchronized (this) {
// notifyAll();
// }
+ notify();
return true;
} else {
return false;
Index:
james/src/java/org/apache/james/mailrepository/JDBCMailRepository.java
===================================================================
--- james/src/java/org/apache/james/mailrepository/JDBCMailRepository.java
(revision 233041)
+++ james/src/java/org/apache/james/mailrepository/JDBCMailRepository.java
(working copy)
@@ -494,7 +494,7 @@
*
* @return true if successfully released the lock, false otherwise
*/
- public synchronized boolean unlock(String key) {
+ public boolean unlock(String key) {
if (lock.unlock(key)) {
if ((DEEP_DEBUG) && (getLogger().isDebugEnabled())) {
StringBuffer debugBuffer =
@@ -508,6 +508,7 @@
getLogger().debug(debugBuffer.toString());
}
// notifyAll();
+ notify();
return true;
} else {
return false;
@@ -521,7 +522,7 @@
*
* @return true if successfully obtained the lock, false otherwise
*/
- public synchronized boolean lock(String key) {
+ public boolean lock(String key) {
if (lock.lock(key)) {
if ((DEEP_DEBUG) && (getLogger().isDebugEnabled())) {
StringBuffer debugBuffer =
@@ -546,7 +547,17 @@
*/
public void store(MailImpl mc) throws MessagingException {
Connection conn = null;
+ boolean wasLocked = true;
+ String key = mc.getName();
try {
+ synchronized(this) {
+ wasLocked = lock.isLocked(key);
+
+ if (!wasLocked) {
+ //If it wasn't locked, we want a lock during the
store
+ lock.lock(key);
+ }
+ }
conn = datasource.getConnection();
//Need to determine whether need to insert this record, or
update it.
@@ -775,6 +786,10 @@
throw new MessagingException("Exception caught while storing
mail Container: " + e);
} finally {
theJDBCUtil.closeJDBCConnection(conn);
+ if (!wasLocked) {
+ //If it wasn't locked, we need to now unlock
+ lock.unlock(key);
+ }
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Spoolmanager blues
Posted by Soren Hilmer <so...@tietoenator.com>.
On Tuesday 06 September 2005 15:25, Stefano Bagnara wrote:
> > > I'm experiencing spooling issues with file repositories too.
> >
> > Well, I haven't until now :-(
>
> Can you try downgrading to 2.2.0 and verify wether the issue is there or
> not?
Will do that today.
> Can you try using db/derby to check wether the issue is there or not?
>
Sure.
> My tests results are that filerepositories have this problem in 2.2.0 and
> in trunk. Db is working in both.
>
> > > Can you try using the db/derby repositories and see wether
> >
> > you see the
> >
> > > same problem?
> >
> > I could, but I need file repositories for my production
> > deployments, so I will try to figure this one out instead.
>
> I would start looking for differences between JDBCMailRepository and
> AvalonMailRepository (e.g: JDBCMailRepository.unlock is synchronized)
> More: AvalonMailRepository.store does lock/unlock a message not already
> locked while JDBCMailRepository does not lock/unlock such messages.
>
> I've experienced issues with DB repositories also, but they just delayed
> the spooling by 60 seconds (probably, for db, the notify does not work
> fine).
>
> I don't use FileRepositories in production because my stresstests provided
> exceptions in locking/unlocking EVERY time:
> http://issues.apache.org/jira/browse/JAMES-397
>
> Stefano
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
--
Søren Hilmer, M.Sc.
R&D manager Phone: +45 72 30 64 00
TietoEnator IT+ A/S Fax: +45 72 30 64 02
Ved Lunden 12 Direct: +45 72 30 64 57
DK-8230 Åbyhøj Email: soren.hilmer <at> tietoenator.com
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Spoolmanager blues
Posted by Stefano Bagnara <ap...@bago.org>.
> > I'm experiencing spooling issues with file repositories too.
>
> Well, I haven't until now :-(
Can you try downgrading to 2.2.0 and verify wether the issue is there or
not?
Can you try using db/derby to check wether the issue is there or not?
My tests results are that filerepositories have this problem in 2.2.0 and in
trunk. Db is working in both.
> > Can you try using the db/derby repositories and see wether
> you see the
> > same problem?
>
> I could, but I need file repositories for my production
> deployments, so I will try to figure this one out instead.
I would start looking for differences between JDBCMailRepository and
AvalonMailRepository (e.g: JDBCMailRepository.unlock is synchronized)
More: AvalonMailRepository.store does lock/unlock a message not already
locked while JDBCMailRepository does not lock/unlock such messages.
I've experienced issues with DB repositories also, but they just delayed the
spooling by 60 seconds (probably, for db, the notify does not work fine).
I don't use FileRepositories in production because my stresstests provided
exceptions in locking/unlocking EVERY time:
http://issues.apache.org/jira/browse/JAMES-397
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Spoolmanager blues
Posted by Soren Hilmer <so...@tietoenator.com>.
On Tuesday 06 September 2005 14:59, Stefano Bagnara wrote:
> > Hi,
> >
> > I am having trouble with the JamesSpoolManager in the trunk.
> > I experience mails hanging in the spool, it looks like the
> > offending piece of code is the return statement in line 418.
> > The reason I suspect that line is that I cannot reproduce the
> > effect if I remove the line.
> >
> > Have anyone seen something similar, or can someone (Stefano)
> > provide the rationale behind this particular return statement.
> > I will do some more debugging to better understand what happens.
>
> I'm experiencing spooling issues with file repositories too.
Well, I haven't until now :-(
>
> The return statement is there since long before I committed my patches ;-)
> (the annotation show that the return line has been written on may 2001). In
> fact I only done minor changes to the spoolmanager (it now just gets
> mailetloader/matcher loader as avalon services instead of running its own).
>
> I've also tested 2.2.0 and I've experienced the same spool problems using
> file repositories.
>
> IMHO file repositories lock/unlock is not (has never been?) working fine.
Probably true.
>
> If you send more mails you will see that the one in the spool will start
> being elaborated.
Yes, just tried that, maybe as you say it has allways been there, only I have
just noticed it now.
>
> Can you try using the db/derby repositories and see wether you see the same
> problem?
I could, but I need file repositories for my production deployments, so I will
try to figure this one out instead.
--Søren
>
> Stefano
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
--
Søren Hilmer, M.Sc.
R&D manager Phone: +45 72 30 64 00
TietoEnator IT+ A/S Fax: +45 72 30 64 02
Ved Lunden 12 Direct: +45 72 30 64 57
DK-8230 Åbyhøj Email: soren.hilmer <at> tietoenator.com
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
RE: Spoolmanager blues
Posted by "Noel J. Bergman" <no...@devtech.com>.
> > > I'm experiencing spooling issues with file repositories too.
> > Actually, I have occassionally seen the same symptom with JDBC, too.
> Simple delay of messages or full hangs of single messages?
Just the simple delay.
As an aside, we need to make sure that dbfile is working with Derby. It is
a common production configuration to use a database with a single file
location for all message content.
> > The problem *appears* to be with notify(). This is why I
> > have wanted us to simplify the logic by having one spool
> > thread and plain worker threads that just process what
> > they are given.
> it's not an easy change mainly because the contract between
> mail/spool repositories and james is not clear enough.
We should define it. We own those contracts.
> What is threadsafe in a repository?
The exposed put/get type operations should be thread safe.
> What is the correct use of locking/unlocking?
Ideally, we should be able to eliminate much of it, and isolate any
remaining notion of it. A single spool thread / multiple worker pool change
should be helpful in this regard.
> Can I store the same message in 2 different repositories?
I'd like to be able to say no, but I suspect that we need for some version
of the answer to be yes.
> Is it safe to store the same message in the same repository
> multiple times?
We need to distinguish between calling store() and actually storing copies.
> Currently we run the notify when the message is stored but
> when the message is stored we still got the lock on the Mail
I'll try to find time to look at the spooler code again.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Spoolmanager blues
Posted by Stefano Bagnara <ap...@bago.org>.
> > Soren wrote:
> > > I am having trouble with the JamesSpoolManager in the trunk.
> > > I experience mails hanging in the spool, it looks like
> the offending
> > > piece of code is the return statement in line 418.
>
> To be clear, that is the return statement in
> process(MailImpl) that does a return after a processor has
> successfully handled the message?
Yes.
> > > The reason I suspect that line is that I cannot reproduce
> the effect
> > > if I remove the line.
>
> For some reason, I had thought that Stefano was changing the
> spooler to process the entire pipeline for an e-mail in one
> shot, rather than queuing for each processor, but since he
> just said that he hadn't made the change, and obviously the
> code doesn't have it, perhaps it was just something he had
> talked about. I seem to recall being concerned about that change.
I just talked about it.
I also changed it locally and attached a patch to a JIRA issue but it was
not working so I commented this on JIRA and never applied changes that way.
> > I'm experiencing spooling issues with file repositories too.
>
> Actually, I have occassionally seen the same symptom with JDBC, too.
Simple delay of messages or full hangs of single messages?
> > If you send more mails you will see that the one in the spool will
> > start being elaborated.
>
> The problem *appears* to be with notify(). This is why I
> have wanted us to simplify the logic by having one spool
> thread and plain worker threads that just process what they are given.
I now understand better the spooling mechanism and it's not an easy change
mainly because the contract between mail/spool repositories and james is not
clear enough. What is threadsafe in a repository? What is the correct use of
locking/unlocking? Can I store the same message in 2 different repositories?
Is it safe to store the same message in the same repository multiple times?
Currently we run the notify when the message is stored but when the message
is stored we still got the lock on the Mail (unless it was generated by a
matcher split/duplicate or by a new sendmail).
When we run the unlock there is no notify so the thread will wait for
another mail and no other spoolthread will get the message we just stored.
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
RE: Spoolmanager blues
Posted by "Noel J. Bergman" <no...@devtech.com>.
Stefano Bagnara wrote:
> Soren wrote:
> > I am having trouble with the JamesSpoolManager in the trunk.
> > I experience mails hanging in the spool, it looks like the
> > offending piece of code is the return statement in line 418.
To be clear, that is the return statement in process(MailImpl) that does a
return after a processor has successfully handled the message?
> > The reason I suspect that line is that I cannot reproduce
> > the effect if I remove the line.
For some reason, I had thought that Stefano was changing the spooler to
process the entire pipeline for an e-mail in one shot, rather than queuing
for each processor, but since he just said that he hadn't made the change,
and obviously the code doesn't have it, perhaps it was just something he had
talked about. I seem to recall being concerned about that change.
> I'm experiencing spooling issues with file repositories too.
Actually, I have occassionally seen the same symptom with JDBC, too.
> If you send more mails you will see that the one in the spool
> will start being elaborated.
The problem *appears* to be with notify(). This is why I have wanted us to
simplify the logic by having one spool thread and plain worker threads that
just process what they are given.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Spoolmanager blues
Posted by Stefano Bagnara <ap...@bago.org>.
> Hi,
>
> I am having trouble with the JamesSpoolManager in the trunk.
> I experience mails hanging in the spool, it looks like the
> offending piece of code is the return statement in line 418.
> The reason I suspect that line is that I cannot reproduce the
> effect if I remove the line.
>
> Have anyone seen something similar, or can someone (Stefano)
> provide the rationale behind this particular return statement.
> I will do some more debugging to better understand what happens.
I'm experiencing spooling issues with file repositories too.
The return statement is there since long before I committed my patches ;-)
(the annotation show that the return line has been written on may 2001). In
fact I only done minor changes to the spoolmanager (it now just gets
mailetloader/matcher loader as avalon services instead of running its own).
I've also tested 2.2.0 and I've experienced the same spool problems using
file repositories.
IMHO file repositories lock/unlock is not (has never been?) working fine.
If you send more mails you will see that the one in the spool will start
being elaborated.
Can you try using the db/derby repositories and see wether you see the same
problem?
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org