You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@commons.apache.org by JW...@FAMILYDOLLAR.com on 2005/09/30 23:05:43 UTC
[vfs] Out of memory when doing getChildren() or findFiles(FileSel
ector)
This is a large directory and doing getChildren() or findFiles() even if the
includeFile method of the selector returns false it will take 50MB of the
heap.
I assume its parsing the output of the ls -l command into a StringBuffer or
something.
And after it finishes, it doesnt clear the memory from the heap, even if a
call System.gc() and schedule a garbage collection run.
Any help is appreciated.
class IndexThread extends Thread {
private ArrayList<String> fileNames;
public IndexThread(){fileNames = new ArrayList<String>();}
public void run(){
while(go){
Calendar c = Calendar.getInstance();
System.out.println("Indexing Service
Started: " + c.get(Calendar.DAY_OF_MONTH)+ "/"+ c.get(Calendar.MONTH) + "/"
+ c.get(Calendar.YEAR) +" "+c.get(Calendar.HOUR_OF_DAY)+
":"+c.get(Calendar.MINUTE)+":"+c.get(Calendar.MILLISECOND));
FileSystemManager fsm = null;
try {
//get a FileSystemManager from the
static class VFS (part of API)
fsm = VFS.getManager();
Iterator it =
fileMountPoints.iterator();
while(it.hasNext()){
//get the next file mount
point
// ex data is
"ftp://jwilliam@password/server/path/to/files"
String loc =
(String)it.next();
//resolve the URL into a
FileObject...
remoteFO =
fsm.resolveFile(loc);
//process the files and
folders in this directory
processChildren(remoteFO,
fsm);
}
//close the File Object ( attempt to
clear the cache and free memory )
remoteFO.close();
} catch (FileSystemException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}finally{
if(fsm!= null)
//an attempt to clear the
cache..
fsm.getFilesCache().close();
}
c = Calendar.getInstance();
System.out.println("Indexing Service Ended:
" + c.get(Calendar.DAY_OF_MONTH)+ "/"+ c.get(Calendar.MONTH) + "/" +
c.get(Calendar.YEAR) +" "+c.get(Calendar.HOUR_OF_DAY)+
":"+c.get(Calendar.MINUTE)+":"+c.get(Calendar.MILLISECOND));
try {
//sleeping thread...
Thread.sleep(Long.parseLong(props.getProperty("index.sleep")));
} catch (NumberFormatException e) {
System.out.println("index.sleep is
missing in index.config");
} catch (InterruptedException e) {
//stop the thread upon interruption
go = false;
}
}
}
private void processChildren(FileObject remoteFO,
FileSystemManager fsm) {
try {
remoteFO.findFiles(new FileSelector(){
public boolean
includeFile(FileSelectInfo arg0) throws Exception {
System.out.println(arg0.getFile().getName().toString());
if(!vde.fileInContents(arg0.getFile().getName().toString())){
if(arg0.getFile().getType() == FileType.FILE){
System.out.println("NEW FILE!" + arg0.getFile().getName().getBaseName());
//fileNames.add(arg0.getFile().getName().toString());
arg0.getFile().getFileSystem().getFileSystemManager().getFilesCache().close(
);
}
}
return false;
}
public boolean
traverseDescendents(FileSelectInfo arg0) throws Exception {
return true;
}
});
for(String name : fileNames){
System.out.println("begining process of children");
FileObject theFile =
fsm.resolveFile(name);
indexFile(name,
theFile.getContent().getInputStream());
theFile.close();
}
fsm.getFilesCache().close();
fsm = null;
System.gc();
} catch (FileSystemException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
Joe Williams
E-Commerce Analyst
Family Dollar
o:(704)849-7512
-----------------------------------------
************************************************************
NOTE:
This e-mail message contains PRIVILEGED and CONFIDENTIAL information
and is intended only for the use of the specific individual or
individuals to which it is addressed. If you are not an intended
recipient of this e-mail, you are hereby notified that any unauthorized
use, dissemination or copying of this e-mail or the information
contained herein or attached hereto is strictly prohibited. If you
receive this e-mail in error, notify the person named above by reply
e-mail and please delete it. Thank you.
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [vfs] Out of memory when doing getChildren() or findFiles(FileSel
ector)
Posted by Mario Ivankovits <ma...@ops.co.at>.
Mario Ivankovits wrote:
> Do you have any problems other than the amout of heap used?
Ooops - sorry, your subject says you get an "out of memory".
I'll have a look at it.
---
Mario
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [vfs] Out of memory when doing getChildren() or findFiles(FileSel
ector)
Posted by Mario Ivankovits <ma...@ops.co.at>.
JWilliams@FAMILYDOLLAR.com wrote:
>This is a large directory and doing getChildren() or findFiles() even if the
>includeFile method of the selector returns false it will take 50MB of the
>heap.
>
VFS utilize a cache for faster lookup on subsequent resolves.
This cache uses SoftReferences and they are freed as soon as the JVM
gets low of memory and no-one holds a strong reference to the
corresponding file objects.
Do you have any problems other than the amout of heap used?
---
Mario
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [vfs] Out of memory when doing getChildren() or findFiles(FileSel
ector)
Posted by Mario Ivankovits <ma...@ops.co.at>.
JWilliams@FAMILYDOLLAR.com wrote:
> This is a large directory and doing getChildren() or findFiles() even if the
> includeFile method of the selector returns false it will take 50MB of the
> heap.
>
I tried some things, but didnt found any fault.
Which filesystem do you use? (local file, ftp, ...)
If this is a REALLY HUGE directory, then the problem can be that almost
all used filesystem library returns the list of files in an array.
There isnt much VFS can do here then.
If this array gets too large (in bytes used) then you get the out of
memory. The only thing you can do then is to add -Xmx256M (or whatever
needed) to your jvm startup script.
---
Mario
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org