You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-user@james.apache.org by "rafael.munoz" <ra...@gmail.com> on 2008/08/21 15:08:54 UTC
RemoteDelivery mailet questions
Hi
I'm trying to refactor all the retry code from the RemoteDelivery mailtet
(an idea I got from a recent mail from Stefano Bagnara) to a
GenericRetryMailet easy to extend and use.
To do that I 'm reading and trying to understand the RemoteDelivery code but
I have some doubts and I hope someone can answer them:
- There is a comment that says that the store of the mail on the spool
triggers a inmediately delivery. How is this done? I can't find anything on
the AvalonMailRepository/AvalonSpoolRepository that indicates how this is
done ... It seems that the mail is only store in the spool and it will be
deliver in the same way that the failed mails, when a spool thread read it
and deliver it.
CODE:
//Set it to try to deliver (in a separate thread) immediately
(triggered by storage)
outgoing.store(mail);
- Someone can explain to me the ContainterUtil.dispose(mail) method? The
dispose method sets the MimeMessage internal field of mail to null and call
ContainerUtil.dispose(MimeMessage), but this second call has no sense,
doesn't it? The dispose method only has sense with objects that implements
Disposable and MimeMessage doesn't implements this interface.
regards and thanks in advance,
Rafael Munoz
--
View this message in context: http://www.nabble.com/RemoteDelivery-mailet-questions-tp19088399p19088399.html
Sent from the James - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
RE: James uses up almost all the memory any suggestions
Posted by Hillel Bilman <hb...@ecommunicate.co.za>.
Thanks, we will try with JDK 6 and see what happens.
-----Original Message-----
From: Stefano Bagnara [mailto:apache@bago.org]
Sent: 21 August 2008 12:43 PM
To: Hillel Bilman
Subject: Re: James uses up almost all the memory any suggestions
Hillel Bilman ha scritto:
> Hi Stefano,
>
> Thanks for replying.
>
> 1. I agree, the goal for using free was to show that after stopping James
> about 7GB was now available to the OS.
> The PS command showed James as using about 95% of the available memory.
>
> 2. I agree
>
> 3 and 4.:
> Based on your remark, I don't understand how Java gets access to so much
> memory.
> java version "1.5.0_15"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_15-b04)
> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_15-b04, mixed mode)
>
> Would you recommend we move to JDK 6?
>
> We are using james-server.sh to keep James running and we are using the
> default -Xmx,
> how does one find out the default Xmx the JVM is using?
If the JVM uses 7GB and you didn't set Xmx then this is not related to
JAMES but to the JVM. So please make sure you are using the latest
version of the JVM for your platform.
64bit platforms are supported since less time so they are more likely
hit by issues.
Try with different jvms and I bet this problem will disappear.
> 5. From the URL
> http://java.sun.com/developer/technicalArticles/Programming/HPROF.html
>
> It appears we should use javac -J-agentlib:hprof=heap=dump jamesClassName
or
Try first with jstat command:
http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jstat.html
This will give you informations about the java memory pools and their usage.
> 6. If one of our mailets throws exceptions that were not caught, could
that
> cause the problem?
Should not.
But your mailets could allocate objects they never free.
I think they are not your mailets because a JVM cannot use 8GB of memory
because of bad classes if you don't add an Xmx to allow it to use so
much memory.
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: James uses up almost all the memory any suggestions
Posted by Hillel Bilman <hb...@ecommunicate.co.za>.
Hi Stefano,
Thanks for replying.
1. I agree, the goal for using free was to show that after stopping James
about 7GB was now available to the OS.
The PS command showed James as using about 95% of the available memory.
2. I agree
3 and 4.:
Based on your remark, I don't understand how Java gets access to so much
memory.
java version "1.5.0_15"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_15-b04)
Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_15-b04, mixed mode)
Would you recommend we move to JDK 6?
We are using james-server.sh to keep James running and we are using the
default -Xmx,
how does one find out the default Xmx the JVM is using?
5. From the URL
http://java.sun.com/developer/technicalArticles/Programming/HPROF.html
It appears we should use javac -J-agentlib:hprof=heap=dump jamesClassName or
javac -J-agentlib:hprof=heap=sites jamesClassName
What james Class Name should we use in this command to do the dump or
continual profiling?
OR Do you recommend a better command to find the Heap details?
6. If one of our mailets throws exceptions that were not caught, could that
cause the problem?
Rgds
>>Re: James uses up almost all the memory any suggestions
Stefano Bagnara
Thu, 21 Aug 2008 00:21:33 -0700
Hillel Bilman ha scritto:
Hi,
We are using james-2.3.1 on Linux 64 bit Redhat with 8GB of RAM.
Using free:
total used free shared
buffers cached
Mem: 8174276 8129120 45156 0 2696 42560
-/+ buffers/cache: 8083864 90412
Swap: 16386248 8255480 8130768
We stopped James and now there is 7856744 memory available.
This means the server starts to go slowly.
Using free:
total used free shared
buffers cached
Mem: 8174276 374404 7799872 0 3500 53372
-/+ buffers/cache: 317532 7856744
Swap: 16386248 915912 15470336
If one of our mailets throws an exception could that cause it?
Why does this happen? How can we prevent it?
> 1st: memory consumption for a linux tool is not measured with "free". Free
tells you the memory used by the operating system as a whole.
Please check the memory with "ps", first and tell us what you see.
2nd: it is the normal behaviour for linux to use all of the memory you have
because it uses the free memory as a cache when it understand it is not used
by programs.
3rd: a java program will never use 8GB unless you configured java to use so
much memory. The default Xmx (maximum heap size) for a java program depends
on the JVM and the platform but I guess it never is more than 256MB, so if
you didn't alter the Xmx manually it is impossible for james to use 8gb.
4th: whenever you set a larger -Xmx then java will be allowed to use more
memory and in case it will need once the memory will be allocated by java
and never returned to the OS.
5th: you can get an heap dump from the james jvm just before stopping it to
understand what there is in the JVM memory ("jheap" if you are on a recent
jvm).
I guess there's a lot for you to think and reply before I go on with more
points.
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: RemoteDelivery mailet questions
Posted by Stefano Bagnara <ap...@bago.org>.
rafael.munoz ha scritto:
> Hi
>
> Ok, thanks a lot. I'll look into it.
>
> The reason I care about it it's because I was thinking of using a
> ExecutorService (and the JDK 5.0 concurrent classes) instead of instancing
> and managing by hand all the waiting threads.
I'd say it won't work: The spool locking is based on the thread
locking/synchronization.
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: RemoteDelivery mailet questions
Posted by "rafael.munoz" <ra...@gmail.com>.
Hi
Ok, thanks a lot. I'll look into it.
The reason I care about it it's because I was thinking of using a
ExecutorService (and the JDK 5.0 concurrent classes) instead of instancing
and managing by hand all the waiting threads.
regards,
Rafa
Stefano Bagnara-2 wrote:
>
> rafael.munoz ha scritto:
>> Hi
>>
>> Sorry to bother you again (I guess you are now thinking to answer me with
>> a
>> 'oh, please, shut up' ;) ) but I just can't find it on the
>> AvalonMailRepository (the one I'm using) or in any of the other
>> MailRepository or SpoolRepository implementations. Can you point me out
>> to
>> the exact code that implement the feature?
>>
>> In fact, I don't exactly understand how the spool is going to know about
>> the
>> waiting threads that are own and manage by the RemoteDelivery mailet.
>
> AbstractMailRepository.store():165
> ---
> if (!wasLocked) {
> // If it wasn't locked, we need to unlock
> unlock(key);
> synchronized (this) {
> notify();
> }
> }
> -----
>
> If you want to understand why it happens on the first attempt you'll
> have to track all of the "lock()" and "wasLocked" use, but I don't
> understand why you should care of all this low-level stuff for your
> refactoring.
>
> Stefano
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
>
>
--
View this message in context: http://www.nabble.com/RemoteDelivery-mailet-questions-tp19088399p19092001.html
Sent from the James - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: RemoteDelivery mailet questions
Posted by Stefano Bagnara <ap...@bago.org>.
rafael.munoz ha scritto:
> Hi
>
> Sorry to bother you again (I guess you are now thinking to answer me with a
> 'oh, please, shut up' ;) ) but I just can't find it on the
> AvalonMailRepository (the one I'm using) or in any of the other
> MailRepository or SpoolRepository implementations. Can you point me out to
> the exact code that implement the feature?
>
> In fact, I don't exactly understand how the spool is going to know about the
> waiting threads that are own and manage by the RemoteDelivery mailet.
AbstractMailRepository.store():165
---
if (!wasLocked) {
// If it wasn't locked, we need to unlock
unlock(key);
synchronized (this) {
notify();
}
}
-----
If you want to understand why it happens on the first attempt you'll
have to track all of the "lock()" and "wasLocked" use, but I don't
understand why you should care of all this low-level stuff for your
refactoring.
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: RemoteDelivery mailet questions
Posted by "rafael.munoz" <ra...@gmail.com>.
Hi
Sorry to bother you again (I guess you are now thinking to answer me with a
'oh, please, shut up' ;) ) but I just can't find it on the
AvalonMailRepository (the one I'm using) or in any of the other
MailRepository or SpoolRepository implementations. Can you point me out to
the exact code that implement the feature?
In fact, I don't exactly understand how the spool is going to know about the
waiting threads that are own and manage by the RemoteDelivery mailet.
regards and again thanks for you patience,
Rafa
Stefano Bagnara-2 wrote:
>
> rafael.munoz ha scritto:
>> Hi
>>
>> Thanks for the answer. Comments inline.
>>
>>> Yes. The first delivery happens like the retries.
>>> The only difference is that the first store "notify" a waiting thread
>>> (if available) and this will start the delivery ASAP. Otherwise the
>>> standard behaviour is polling.
>>
>> Yeah, this is what I understand by the comment in the code but I can't
>> find
>> how the store method notify the waiting threads to start the delivery.
>> It's
>> inside the spool objects or something?
>
> Yes. It is managed by the spoolrepository implementation.
>
> Stefano
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
>
>
--
View this message in context: http://www.nabble.com/RemoteDelivery-mailet-questions-tp19088399p19090753.html
Sent from the James - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: RemoteDelivery mailet questions
Posted by Stefano Bagnara <ap...@bago.org>.
rafael.munoz ha scritto:
> Hi
>
> Thanks for the answer. Comments inline.
>
>> Yes. The first delivery happens like the retries.
>> The only difference is that the first store "notify" a waiting thread
>> (if available) and this will start the delivery ASAP. Otherwise the
>> standard behaviour is polling.
>
> Yeah, this is what I understand by the comment in the code but I can't find
> how the store method notify the waiting threads to start the delivery. It's
> inside the spool objects or something?
Yes. It is managed by the spoolrepository implementation.
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: RemoteDelivery mailet questions
Posted by "rafael.munoz" <ra...@gmail.com>.
Hi
Thanks for the answer. Comments inline.
>Yes. The first delivery happens like the retries.
>The only difference is that the first store "notify" a waiting thread
>(if available) and this will start the delivery ASAP. Otherwise the
>standard behaviour is polling.
Yeah, this is what I understand by the comment in the code but I can't find
how the store method notify the waiting threads to start the delivery. It's
inside the spool objects or something?
>In specific a MailImpl contains a MimeMessage, but again, you don't know
>what kind of MimeMessage and in this case is probably a
>MimeMessageCopyOnWriteProxy that wraps a MimeMessageWrapper. All of them
>allocates resources.
>Failing to call dispose will result in file descriptors leaks and/or
>memory leaks.
Understood.
regards,
Rafael Munoz
Stefano Bagnara-2 wrote:
>
> rafael.munoz ha scritto:
>> Hi
>>
>> I'm trying to refactor all the retry code from the RemoteDelivery mailtet
>> (an idea I got from a recent mail from Stefano Bagnara) to a
>> GenericRetryMailet easy to extend and use.
>>
>> To do that I 'm reading and trying to understand the RemoteDelivery code
>> but
>> I have some doubts and I hope someone can answer them:
>>
>> - There is a comment that says that the store of the mail on the spool
>> triggers a inmediately delivery. How is this done? I can't find anything
>> on
>> the AvalonMailRepository/AvalonSpoolRepository that indicates how this is
>> done ... It seems that the mail is only store in the spool and it will be
>> deliver in the same way that the failed mails, when a spool thread read
>> it
>> and deliver it.
>> CODE:
>> //Set it to try to deliver (in a separate thread)
>> immediately
>> (triggered by storage)
>> outgoing.store(mail);
>
> Yes. The first delivery happens like the retries.
> The only difference is that the first store "notify" a waiting thread
> (if available) and this will start the delivery ASAP. Otherwise the
> standard behaviour is polling.
>
>
>
>
>> - Someone can explain to me the ContainterUtil.dispose(mail) method? The
>> dispose method sets the MimeMessage internal field of mail to null and
>> call
>> ContainerUtil.dispose(MimeMessage), but this second call has no sense,
>> doesn't it? The dispose method only has sense with objects that
>> implements
>> Disposable and MimeMessage doesn't implements this interface.
>
> you don't know what "mail" is.
> In our case mail is a MailImpl message. It requires to be disposed to
> disallocate resources.
> In specific a MailImpl contains a MimeMessage, but again, you don't know
> what kind of MimeMessage and in this case is probably a
> MimeMessageCopyOnWriteProxy that wraps a MimeMessageWrapper. All of them
> allocates resources.
> Failing to call dispose will result in file descriptors leaks and/or
> memory leaks.
>
> Stefano
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
>
>
--
View this message in context: http://www.nabble.com/RemoteDelivery-mailet-questions-tp19088399p19089767.html
Sent from the James - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: RemoteDelivery mailet questions
Posted by Stefano Bagnara <ap...@bago.org>.
rafael.munoz ha scritto:
> Hi
>
> I'm trying to refactor all the retry code from the RemoteDelivery mailtet
> (an idea I got from a recent mail from Stefano Bagnara) to a
> GenericRetryMailet easy to extend and use.
>
> To do that I 'm reading and trying to understand the RemoteDelivery code but
> I have some doubts and I hope someone can answer them:
>
> - There is a comment that says that the store of the mail on the spool
> triggers a inmediately delivery. How is this done? I can't find anything on
> the AvalonMailRepository/AvalonSpoolRepository that indicates how this is
> done ... It seems that the mail is only store in the spool and it will be
> deliver in the same way that the failed mails, when a spool thread read it
> and deliver it.
> CODE:
> //Set it to try to deliver (in a separate thread) immediately
> (triggered by storage)
> outgoing.store(mail);
Yes. The first delivery happens like the retries.
The only difference is that the first store "notify" a waiting thread
(if available) and this will start the delivery ASAP. Otherwise the
standard behaviour is polling.
> - Someone can explain to me the ContainterUtil.dispose(mail) method? The
> dispose method sets the MimeMessage internal field of mail to null and call
> ContainerUtil.dispose(MimeMessage), but this second call has no sense,
> doesn't it? The dispose method only has sense with objects that implements
> Disposable and MimeMessage doesn't implements this interface.
you don't know what "mail" is.
In our case mail is a MailImpl message. It requires to be disposed to
disallocate resources.
In specific a MailImpl contains a MimeMessage, but again, you don't know
what kind of MimeMessage and in this case is probably a
MimeMessageCopyOnWriteProxy that wraps a MimeMessageWrapper. All of them
allocates resources.
Failing to call dispose will result in file descriptors leaks and/or
memory leaks.
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org