You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@commons.apache.org by Mario Ivankovits <ma...@ops.co.at> on 2006/06/01 11:11:31 UTC
Re: [commons vfs]: Handling large packages, memory error
Hi!
> When extracting large zip packages (~1.6Gb), Java heap space error is
> thrown. Currently used attribute -Xmx500M isn't enough. Memory usage rises
> gradually during extraction (slowly, but surely). We have a need to handle
> packages up to 2Gb size.
>
Do this zip file contains many entries?
If this is the case then we are out of luck currently. VFS reads the
whole directory structure in memory.
I dont see what we can do to avoid this without introducing a HUGE
performance impact.
If this is not the case it would be nice if you could start up a
profiler to see what uses so much memory, maybe its again a
java.util.zip limitation? Did you try some simple code using
java.util.zip directly?
Ciao,
Mario
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: [commons vfs]: Handling large packages, memory error
Posted by hrjk <ha...@zeniitti.net>.
Hi.
We've found the problem. The error message was somewhat misleading. It
seems that the extracted zip package contained too long directory paths
for even WinZip to handle. So basically extracting this package to too
deep directory structure would cause the heap size exception.
Thanks for your help.
BR,
-hrjk
> Hi!
>> Also, we've tried to clear cache with
>> getFilesCache().clear(usedFileObject.getFileSystem()). Even this doesn't
>> seem to help.
>>
> Are you sure you get rid of EVERY fileObject you had processed during
> extraction of the zip file?
> In case of zip files there is a global list (on filesystem level) which
> holds all entries.
> It should be garbage collected as soon as you dont hold any reference.
>
> Clear the cache isnt required as it holds only SoftReferences and the vm
> should discard them too.
>
>> Do you see this as a potential VFS problem?
>>
> I wont rule out that you catched a bug here. Its complicated stuff, and
> hard to debug :-(
>
>> Could you point me to a working profiler?
> I use the yourkit profiler: http://www.yourkit.com/overview/index.jsp
>
>
> Ciao,
> Mario
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: commons-user-help@jakarta.apache.org
>
--
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: [commons vfs]: Handling large packages, memory error
Posted by hrjk <ha...@zeniitti.net>.
Hi.
Things are looking even more interesting. The error (Java Heap Size)
occurs while reading a zip package with a LOT of files. Zip package size
is ~150Mb.
However, this error does not occur, when we're developing through Eclipse
3.1.2. But as soon as we shift to running client on JRE 1.5.0_03, we get
the error.
Any ideas?
Thanks.
BR,
-hrjk
> Hi again.
>
> We haven't got rid of this issue. It seems like applying
> Java.util.zip-library will handle this zip package correctly.
>
> The zip package at hand contains up to 200 characters long paths. Can this
> cause VFS problems? I know Windows has a restriction of path length to 260
> characters.
>
> We're still running into Java heap size errors. The actual zip contains a
> lot of files, but is relatively small (~150Mb).
>
> The cache should be cleared properly. We're even calling clear() in
> several occasions to prevent memory errors. I can't see a way how we could
> be referencing to any FileObjects longer than absolutely needed.
>
> Any ideas?
>
> BR,
> -hrjk
>
>> Hi!
>>> Also, we've tried to clear cache with
>>> getFilesCache().clear(usedFileObject.getFileSystem()). Even this
>>> doesn't
>>> seem to help.
>>>
>> Are you sure you get rid of EVERY fileObject you had processed during
>> extraction of the zip file?
>> In case of zip files there is a global list (on filesystem level) which
>> holds all entries.
>> It should be garbage collected as soon as you dont hold any reference.
>>
>> Clear the cache isnt required as it holds only SoftReferences and the vm
>> should discard them too.
>>
>>> Do you see this as a potential VFS problem?
>>>
>> I wont rule out that you catched a bug here. Its complicated stuff, and
>> hard to debug :-(
>>
>>> Could you point me to a working profiler?
>> I use the yourkit profiler: http://www.yourkit.com/overview/index.jsp
>>
>>
>> Ciao,
>> Mario
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
>> For additional commands, e-mail: commons-user-help@jakarta.apache.org
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: commons-user-help@jakarta.apache.org
>
--
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: [commons vfs]: Handling large packages, memory error
Posted by hrjk <ha...@zeniitti.net>.
Hi again.
We haven't got rid of this issue. It seems like applying
Java.util.zip-library will handle this zip package correctly.
The zip package at hand contains up to 200 characters long paths. Can this
cause VFS problems? I know Windows has a restriction of path length to 260
characters.
We're still running into Java heap size errors. The actual zip contains a
lot of files, but is relatively small (~150Mb).
The cache should be cleared properly. We're even calling clear() in
several occasions to prevent memory errors. I can't see a way how we could
be referencing to any FileObjects longer than absolutely needed.
Any ideas?
BR,
-hrjk
> Hi!
>> Also, we've tried to clear cache with
>> getFilesCache().clear(usedFileObject.getFileSystem()). Even this doesn't
>> seem to help.
>>
> Are you sure you get rid of EVERY fileObject you had processed during
> extraction of the zip file?
> In case of zip files there is a global list (on filesystem level) which
> holds all entries.
> It should be garbage collected as soon as you dont hold any reference.
>
> Clear the cache isnt required as it holds only SoftReferences and the vm
> should discard them too.
>
>> Do you see this as a potential VFS problem?
>>
> I wont rule out that you catched a bug here. Its complicated stuff, and
> hard to debug :-(
>
>> Could you point me to a working profiler?
> I use the yourkit profiler: http://www.yourkit.com/overview/index.jsp
>
>
> Ciao,
> Mario
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: commons-user-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: [commons vfs]: Handling large packages, memory error
Posted by Mario Ivankovits <ma...@ops.co.at>.
Hi!
> Also, we've tried to clear cache with
> getFilesCache().clear(usedFileObject.getFileSystem()). Even this doesn't
> seem to help.
>
Are you sure you get rid of EVERY fileObject you had processed during
extraction of the zip file?
In case of zip files there is a global list (on filesystem level) which
holds all entries.
It should be garbage collected as soon as you dont hold any reference.
Clear the cache isnt required as it holds only SoftReferences and the vm
should discard them too.
> Do you see this as a potential VFS problem?
>
I wont rule out that you catched a bug here. Its complicated stuff, and
hard to debug :-(
> Could you point me to a working profiler?
I use the yourkit profiler: http://www.yourkit.com/overview/index.jsp
Ciao,
Mario
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: [commons vfs]: Handling large packages, memory error
Posted by hrjk <ha...@zeniitti.net>.
Hi.
> Hi!
>> When extracting large zip packages (~1.6Gb), Java heap space error is
>> thrown. Currently used attribute -Xmx500M isn't enough. Memory usage
>> rises
>> gradually during extraction (slowly, but surely). We have a need to
>> handle
>> packages up to 2Gb size.
>>
> Do this zip file contains many entries?
> If this is the case then we are out of luck currently. VFS reads the
> whole directory structure in memory.
> I dont see what we can do to avoid this without introducing a HUGE
> performance impact.
Yes, the zip files contain a lot of entries. However, we're not
experiencing problems when we extract say 1 big zip. The problems occur
later, when we have extracted like 5 of them. It seems like the VFS
doesn't free the used memory, even when the object used is out of scope.
Also, we've tried to clear cache with
getFilesCache().clear(usedFileObject.getFileSystem()). Even this doesn't
seem to help.
Do you see this as a potential VFS problem?
> If this is not the case it would be nice if you could start up a
> profiler to see what uses so much memory, maybe its again a
> java.util.zip limitation? Did you try some simple code using
> java.util.zip directly?
Could you point me to a working profiler? I have tried to found some for
Eclipse 3.1.2, but I'm out of luck.
Thanks a lot for your help.
BR,
-hrjk
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org