You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-user@james.apache.org by Vincenzo Gianferrari Pini <vi...@praxis.it> on 2006/08/01 09:44:57 UTC

Re: FileStreamStore file is left in the spool

Very strange, this problem *never* occurred to me.

The only possible difference with your test is that I use file based
repositories for everything (spool, outgoing, inbox), but db for user
accounts.
I will try with the default for 2.3 changing everything to file.

Vincenzo

Stefano Bagnara wrote:

>FWIW I can reproduce the issue with current 2.3rc1 and current trunk.
>
>Config file as default for 2.3, repositories changed to file in trunk
>(we still have derby as default in trunk).
>
>I connect to remote manager, create an user. Connect to smtp server and
>send a message to that user.
>
>The XXX.Repository.FileStreamStore file is stuck there, in the spool.
>If I restart James it got cleaned by the cleaning procedure in
>AvalonMailRepository.
>
>This seems to be a bug. Every file that is spooled remain there until a
>restart. This is diskspace leak.
>
>I tried manually calling from all the places the
>ContainerUtil.dispose(mail) and set mail to null to be sure that the
>remove were called and I can see that remove is called twice on that
>file but it is not removed. It seems that a stream is still opened when
>we try to remove it and it's not removed.
>
>I've not investigated more on this. I only use filerepository in a small
>production server that I rarely check. Noel, Vincenzo, can you check
>what does it happen in the long term? Maybe this is a test for
>Postage... spool folder should increase it size in a linear way if this
>bug is there.
>
>Stefano
>
>Noel J. Bergman wrote:
>  
>
>>Vincenzo Gianferrari Pini wrote:
>>
>>    
>>
>>>My file based repositories are always clean, and the
>>>FileStreamStore files are automatically deleted as
>>>soon as sent.
>>>      
>>>
>>>It has always been so for me in the last 3 years,
>>>in 2.1, 2.2 and now in 2.3.
>>>      
>>>
>>I have on very rare occassions seen a stray file.  The more common one would
>>be a stray .m64 file in temp/.
>>
>>	--- Noel
>>    
>>
>
>  
>


---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org


Re: FileStreamStore file is left in the spool

Posted by Stefano Bagnara <ap...@bago.org>.
Even Norman is not able to reproduce it.
This could be platform specific: I'm able to reproduce it under windows.

This would also make sense because in unix os you can remove opened
files and they are really removed when the file is closed while under
windows an opened file cannot be deleted.

What OS are you testing?
Stefano

Vincenzo Gianferrari Pini wrote:
> Very strange, this problem *never* occurred to me.
> 
> The only possible difference with your test is that I use file based
> repositories for everything (spool, outgoing, inbox), but db for user
> accounts.
> I will try with the default for 2.3 changing everything to file.
> 
> Vincenzo
> 
> Stefano Bagnara wrote:
> 
>> FWIW I can reproduce the issue with current 2.3rc1 and current trunk.
>>
>> Config file as default for 2.3, repositories changed to file in trunk
>> (we still have derby as default in trunk).
>>
>> I connect to remote manager, create an user. Connect to smtp server and
>> send a message to that user.
>>
>> The XXX.Repository.FileStreamStore file is stuck there, in the spool.
>> If I restart James it got cleaned by the cleaning procedure in
>> AvalonMailRepository.
>>
>> This seems to be a bug. Every file that is spooled remain there until a
>> restart. This is diskspace leak.
>>
>> I tried manually calling from all the places the
>> ContainerUtil.dispose(mail) and set mail to null to be sure that the
>> remove were called and I can see that remove is called twice on that
>> file but it is not removed. It seems that a stream is still opened when
>> we try to remove it and it's not removed.
>>
>> I've not investigated more on this. I only use filerepository in a small
>> production server that I rarely check. Noel, Vincenzo, can you check
>> what does it happen in the long term? Maybe this is a test for
>> Postage... spool folder should increase it size in a linear way if this
>> bug is there.
>>
>> Stefano



---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org


Re: FileStreamStore file is left in the spool

Posted by Stefano Bagnara <ap...@bago.org>.
Hi Iwasa,

I've been able to reproduce the issue you described and I think I fixed 
it. Let's hope this was the last place locking that file!

Let me know if you are able to build the current branch from here:
http://svn.apache.org/repos/asf/james/server/branches/v2.3/
or if you prefer me to make a build for you to test.

Stefano

Iwasa Kazmi wrote:
> Thank you Noel,
> I've tried 2.3.0-dev on Windows XP.
> 
> In my environment, the problem appears if the recipients have
> both of local account and non-local account.
> 
> I sent a message to a local user, and no files were left
> in the spool.
> Next I sent a message to an account of the external server,
> and the spool was still clean.
> Next I sent a message to both of above accounts.
> FileStreamStore file was left in the spool.
> 
> I'm using Redirect mailet to deliver copy of the outgoing/incoming
> messages to local users.
> In such case too, FileStreamStore file seems not to be deleted.
> 
> Regards,
> Iwasa Kazmi
> 
> Noel J. Bergman:
>>> I would strongly suspect that this is a Windows issue.
>>
>> Yes, the file was still open, and *nix allows deleting the open file 
>> whereas
>> MS-Windows does not.
>>
>> We believe that this issue is now fixed.
>>
>> I've uploaded a build to http://people.apache.org/~noel/james for 
>> people to
>> test in their environment.  The build includes all cumulative fixes to 
>> date.
>>
>>     --- Noel



---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org


Re: FileStreamStore file is left in the spool

Posted by Stefano Bagnara <ap...@bago.org>.
Thank you for providing details to reproduce the issue.
Now I setup a test using your config and let's try to fix this.

I'll post here again asap.

Stefano

Iwasa Kazmi wrote:
> Thank you,
> 
> I confirmed that the issue was solved in normal use.
> But...
> I inserted Redirect mailet in the "transport" processor.
> (see config.xml.diff)
> It duplicates a message for local-user before the RemoteDelivery runs.
> Then I sent a mail to the external server via james.
> FileStreamStore file in the spool was not deleted.
> 
> I think there are much people who use more complex configuration.
> The file-base repository would need more tests.
> 
> Regards,
> Iwasa Kazmi
> 
> Noel J. Bergman:
>>> I've tried 2.3.0-dev on Windows XP.
>>
>>> In my environment, the problem [still] appears
>>
>> We made another change this morning based upon your report.  I'm in the
>> process of uploading now.  Look for a new build within the next 30 
>> minutes
>> or so, depending upon how long the upload takes.
>>
>> Same place: http://people.apache.org/~noel/james
>>
>>     --- Noel
>>
> 
> 
> ------------------------------------------------------------------------
> 
> --- config.xml.orig	2006-08-05 15:44:44.578125000 +0900
> +++ config.xml	2006-08-05 15:59:14.031250000 +0900
> @@ -51,7 +51,7 @@
>        <!-- -->
>        <!-- Warning: If you are using fetchmail it is important to include the -->
>        <!-- fetched domains in the server name list to prevent looping.       -->
> -      <servernames autodetect="true" autodetectIP="true">
> +      <servernames autodetect="false" autodetectIP="true">
>  <!-- CONFIRM? -->
>           <servername>localhost</servername>
>        </servernames>
> @@ -540,6 +540,14 @@
>              <notice>550 - Requested action not taken: relaying denied</notice>
>           </mailet>
>  
> +         <!-- Redirect outgoing message -->
> +         <mailet match="All" class="Redirect">
> +            <recipients> user1@localhost </recipients>
> +            <inline> unaltered </inline>
> +            <to> unaltered </to>
> +            <passThrough> true </passThrough>
> +         </mailet>
> +
>           <!-- Attempt remote delivery using the specified repository for the spool, -->
>           <!-- using delay time to retry delivery and the maximum number of retries -->
>           <mailet match="All" class="RemoteDelivery">
> 
> 
> ------------------------------------------------------------------------
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org


Re: FileStreamStore file is left in the spool

Posted by Stefano Bagnara <ap...@bago.org>.
Hi Iwasa,

Thank you again for your tests!
As you can read here I think I fixed it:
http://issues.apache.org/jira/browse/JAMES-584?page=all

You can find the code in branches/v2.3, or wait until Norman or Noel 
prepare a test build for this.

I really hope this is the last time we have to talk about this bug :-)

Let us know if this finally works.
Stefano

PS: the problem is not the file-based repository but the remaining core 
code. The file based repository under windows make it only much more 
visible. Under linux or using other repositories this only creates a 
minor leak and it is difficult to be noticed.

Iwasa Kazmi wrote:
> Thank you,
> 
> I confirmed that the issue was solved in normal use.
> But...
> I inserted Redirect mailet in the "transport" processor.
> (see config.xml.diff)
> It duplicates a message for local-user before the RemoteDelivery runs.
> Then I sent a mail to the external server via james.
> FileStreamStore file in the spool was not deleted.
> 
> I think there are much people who use more complex configuration.
> The file-base repository would need more tests.
> 
> Regards,
> Iwasa Kazmi



---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org


Re: FileStreamStore file is left in the spool

Posted by Norman Maurer <nm...@byteaction.de>.
Hi,

i just prepare a test build. I upload it at the moment. So you can grab
it from :
http://people.apache.org/~norman/james/

The upload should be done in about 20 minutes..

Thx for testing 

bye
Norman

Am Samstag, den 05.08.2006, 14:48 +0200 schrieb Stefano Bagnara:
> Hi Iwasa,
> 
> Thank you again for your tests!
> As you can read here I think I fixed it:
> http://issues.apache.org/jira/browse/JAMES-584?page=all
> 
> You can find the code in branches/v2.3, or wait until Norman or Noel 
> prepare a test build for this.
> 
> I really hope this is the last time we have to talk about this bug :-)
> 
> Let us know if this finally works.
> Stefano
> 
> PS: the problem is not the file-based repository but the remaining core 
> code. The file based repository under windows make it only much more 
> visible. Under linux or using other repositories this only creates a 
> minor leak and it is difficult to be noticed.
> 
> Iwasa Kazmi wrote:
> > Thank you,
> > 
> > I confirmed that the issue was solved in normal use.
> > But...
> > I inserted Redirect mailet in the "transport" processor.
> > (see config.xml.diff)
> > It duplicates a message for local-user before the RemoteDelivery runs.
> > Then I sent a mail to the external server via james.
> > FileStreamStore file in the spool was not deleted.
> > 
> > I think there are much people who use more complex configuration.
> > The file-base repository would need more tests.
> > 
> > Regards,
> > Iwasa Kazmi
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
> 
> !EXCUBATOR:1,44d493c643388955741453!

Re: FileStreamStore file is left in the spool

Posted by Iwasa Kazmi <iw...@cosmo-system.jp>.
Thank you,

I confirmed that the issue was solved in normal use.
But...
I inserted Redirect mailet in the "transport" processor.
(see config.xml.diff)
It duplicates a message for local-user before the RemoteDelivery runs.
Then I sent a mail to the external server via james.
FileStreamStore file in the spool was not deleted.

I think there are much people who use more complex configuration.
The file-base repository would need more tests.

Regards,
Iwasa Kazmi

Noel J. Bergman:
>> I've tried 2.3.0-dev on Windows XP.
> 
>> In my environment, the problem [still] appears
> 
> We made another change this morning based upon your report.  I'm in the
> process of uploading now.  Look for a new build within the next 30 minutes
> or so, depending upon how long the upload takes.
> 
> Same place: http://people.apache.org/~noel/james
> 
> 	--- Noel
> 


RE: FileStreamStore file is left in the spool

Posted by "Noel J. Bergman" <no...@devtech.com>.
> I've tried 2.3.0-dev on Windows XP.

> In my environment, the problem [still] appears

We made another change this morning based upon your report.  I'm in the
process of uploading now.  Look for a new build within the next 30 minutes
or so, depending upon how long the upload takes.

Same place: http://people.apache.org/~noel/james

	--- Noel



---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org


Re: FileStreamStore file is left in the spool

Posted by Iwasa Kazmi <iw...@cosmo-system.jp>.
Thank you Noel,
I've tried 2.3.0-dev on Windows XP.

In my environment, the problem appears if the recipients have
both of local account and non-local account.

I sent a message to a local user, and no files were left
in the spool.
Next I sent a message to an account of the external server,
and the spool was still clean.
Next I sent a message to both of above accounts.
FileStreamStore file was left in the spool.

I'm using Redirect mailet to deliver copy of the outgoing/incoming
messages to local users.
In such case too, FileStreamStore file seems not to be deleted.

Regards,
Iwasa Kazmi

Noel J. Bergman:
>> I would strongly suspect that this is a Windows issue.
> 
> Yes, the file was still open, and *nix allows deleting the open file whereas
> MS-Windows does not.
> 
> We believe that this issue is now fixed.
> 
> I've uploaded a build to http://people.apache.org/~noel/james for people to
> test in their environment.  The build includes all cumulative fixes to date.
> 
> 	--- Noel
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org


RE: FileStreamStore file is left in the spool

Posted by "Noel J. Bergman" <no...@devtech.com>.
Iwasa Kazmi wrote:

> I did a simple test of rc2 with my usual settings,
> and no problem was found.
> Thanks a lot.

No, thank YOU.  :-)  We very much appreciate your efforts to help make JAMES
better for everyone.  :-)

	--- Noel



---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org


Re: FileStreamStore file is left in the spool

Posted by Iwasa Kazmi <iw...@cosmo-system.jp>.
I'm sorry to late.
I did a simple test of rc2 with my usual settings,
and no problem was found.
Thanks a lot.

Iwasa Kazmi

Norman Maurer:
> Hi Iwasa,
> Can you confirm that the Bug is now resolved ?
> 
> You can grab "fixed" jars from:
> http://people.apache.org/~norman/james/
> 
> bye
> Norman

---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org


Re: FileStreamStore file is left in the spool

Posted by Norman Maurer <nm...@byteaction.de>.
Hi Iwasa,
Can you confirm that the Bug is now resolved ?

You can grab "fixed" jars from:
http://people.apache.org/~norman/james/

bye
Norman


RE: FileStreamStore file is left in the spool

Posted by "Noel J. Bergman" <no...@devtech.com>.
> I would strongly suspect that this is a Windows issue.

Yes, the file was still open, and *nix allows deleting the open file whereas
MS-Windows does not.

We believe that this issue is now fixed.

I've uploaded a build to http://people.apache.org/~noel/james for people to
test in their environment.  The build includes all cumulative fixes to date.

	--- Noel



---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org


Re: FileStreamStore file is left in the spool

Posted by Serge Knystautas <sk...@gmail.com>.
On 8/1/06, Stefano Bagnara <ap...@bago.org> wrote:
> Even Norman is not able to reproduce it.
> This could be platform specific: I'm able to reproduce it under windows.
>
> This would also make sense because in unix os you can remove opened
> files and they are really removed when the file is closed while under
> windows an opened file cannot be deleted.
>
> What OS are you testing?

I would strongly suspect that this is a Windows issue.  I've always
seen Java be slow to close stream and delete files on Windows.  Maybe
the latest release is doing things fast enough that linux can delete
the file, but not for windows.

-- 
Serge Knystautas
Lokitech >> software . strategy . design >> http://www.lokitech.com
p. 301.656.5501
e. sergek@lokitech.com

---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org