You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by luocanrao <lu...@sohu.com> on 2010/01/30 08:31:03 UTC
MMapDirectory need twice more virtual memory than actually need?
Environment: 64 bit linux,memory 8G
When I used pmap instruction to see virtual memory, I found two big anon
memory which grows with the index file size.
I had the two following pictures to show the problem, can you explain?
This is why I got out of memory exception in 32 bit machine.
Because my index file size is only 800m.
Anon memory
picture
Indexfile
memory picture
答复: MMapDirectory need twice more virtual memory than actually need?
Posted by luocanrao <lu...@sohu.com>.
Ps
I use Sun JVMs, I will try setUseUnmap(true) to see whether anon memory disappear.
-----邮件原件-----
发件人: Uwe Schindler [mailto:uwe@thetaphi.de]
发送时间: 2010年1月30日 16:47
收件人: java-user@lucene.apache.org
主题: RE: MMapDirectory need twice more virtual memory than actually need?
MMapDirectory should only be used on 64bit machines with 64bit JVMs. This is described in the Javadocs.
The additional sizes are caused by reopens and have their reason in specialties with the Java VM, described in 2.9/3.0 Javadocs of the class, please take a look. You can also enable a workaround for the missing “unmap” call.
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
<http://www.thetaphi.de/> http://www.thetaphi.de
eMail: uwe@thetaphi.de
From: luocanrao [mailto:luocan19826164@sohu.com]
Sent: Saturday, January 30, 2010 8:31 AM
To: java-user@lucene.apache.org
Subject: MMapDirectory need twice more virtual memory than actually need?
Environment: 64 bit linux,memory 8G
When I used pmap instruction to see virtual memory, I found two big anon memory which grows with the index file size.
I had the two following pictures to show the problem, can you explain?
This is why I got out of memory exception in 32 bit machine. Because my index file size is only 800m.
Anon memory picture
Indexfile memory picture
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: MMapDirectory need twice more virtual memory than actually need?
Posted by Uwe Schindler <uw...@thetaphi.de>.
I am sorry, we know nothing about how the JVM internally handles this. But we do not force it to map into memory! MMapDirectory just maps but does not call MappedByteBuffer.load(). So it is completely handled by the kernel when something is swapped in or not.
I have a server with 32 GB RAM that mapps about 100 GB files using JAVA's MappedByteBuffer into its address space and it still runs :-) (Solaris 64 bit, Java 5, no unmap hack).
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: luocanrao [mailto:luocan19826164@sohu.com]
> Sent: Saturday, January 30, 2010 12:58 PM
> To: java-user@lucene.apache.org
> Subject: 答复: MMapDirectory need twice more virtual memory than
> actually need?
>
> I found this in http://osdir.com/ml/linux-kernel/2009-10/msg03503.html
>
> If application does mlockall(MCL_FUTURE) it is no longer possible to
> mmap
> file bigger than main memory or allocate big area of anonymous memory
> in a thread safe manner. Sometimes it is desirable to lock everything
> related to program execution into memory, but still be able to mmap
> big file or allocate huge amount of memory and allow OS to swap them on
> demand. MAP_UNLOCKED allows to do that
>
> if mmap method use MAP_UNLOCKED flag, this can expain everything.
> Is that right?
>
> -----邮件原件-----
> 发件人: Uwe Schindler [mailto:uwe@thetaphi.de]
> 发送时间: 2010年1月30日 16:47
> 收件人: java-user@lucene.apache.org
> 主题: RE: MMapDirectory need twice more virtual memory than actually
> need?
>
> MMapDirectory should only be used on 64bit machines with 64bit JVMs.
> This is described in the Javadocs.
>
> The additional sizes are caused by reopens and have their reason in
> specialties with the Java VM, described in 2.9/3.0 Javadocs of the
> class, please take a look. You can also enable a workaround for the
> missing “unmap” call.
>
>
>
> -----
>
> Uwe Schindler
>
> H.-H.-Meier-Allee 63, D-28213 Bremen
>
> <http://www.thetaphi.de/> http://www.thetaphi.de
>
> eMail: uwe@thetaphi.de
>
>
>
> From: luocanrao [mailto:luocan19826164@sohu.com]
> Sent: Saturday, January 30, 2010 8:31 AM
> To: java-user@lucene.apache.org
> Subject: MMapDirectory need twice more virtual memory than actually
> need?
>
>
>
> Environment: 64 bit linux,memory 8G
>
> When I used pmap instruction to see virtual memory, I found two big
> anon memory which grows with the index file size.
>
> I had the two following pictures to show the problem, can you explain?
>
> This is why I got out of memory exception in 32 bit machine.
> Because my index file size is only 800m.
>
> Anon
> memory picture
>
>
>
>
> Indexfile memory picture
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
答复: MMapDirectory need twice more virtual memory than actually need?
Posted by luocanrao <lu...@sohu.com>.
I found this in http://osdir.com/ml/linux-kernel/2009-10/msg03503.html
If application does mlockall(MCL_FUTURE) it is no longer possible to mmap
file bigger than main memory or allocate big area of anonymous memory
in a thread safe manner. Sometimes it is desirable to lock everything
related to program execution into memory, but still be able to mmap
big file or allocate huge amount of memory and allow OS to swap them on
demand. MAP_UNLOCKED allows to do that
if mmap method use MAP_UNLOCKED flag, this can expain everything.
Is that right?
-----邮件原件-----
发件人: Uwe Schindler [mailto:uwe@thetaphi.de]
发送时间: 2010年1月30日 16:47
收件人: java-user@lucene.apache.org
主题: RE: MMapDirectory need twice more virtual memory than actually need?
MMapDirectory should only be used on 64bit machines with 64bit JVMs. This is described in the Javadocs.
The additional sizes are caused by reopens and have their reason in specialties with the Java VM, described in 2.9/3.0 Javadocs of the class, please take a look. You can also enable a workaround for the missing “unmap” call.
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
<http://www.thetaphi.de/> http://www.thetaphi.de
eMail: uwe@thetaphi.de
From: luocanrao [mailto:luocan19826164@sohu.com]
Sent: Saturday, January 30, 2010 8:31 AM
To: java-user@lucene.apache.org
Subject: MMapDirectory need twice more virtual memory than actually need?
Environment: 64 bit linux,memory 8G
When I used pmap instruction to see virtual memory, I found two big anon memory which grows with the index file size.
I had the two following pictures to show the problem, can you explain?
This is why I got out of memory exception in 32 bit machine. Because my index file size is only 800m.
Anon memory picture
Indexfile memory picture
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: MMapDirectory need twice more virtual memory than actually need?
Posted by Uwe Schindler <uw...@thetaphi.de>.
I am sorry, I cannot see your attached pictures, can you put them on a web server and link here? In general with 64 bit operating systems the usage of address space is no problem at all, because no real memory is wasted it is just allocated address space and there is enough available :-)
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: luocanrao [mailto:luocan19826164@sohu.com]
> Sent: Saturday, January 30, 2010 10:37 AM
> To: java-user@lucene.apache.org
> Subject: 答复: MMapDirectory need twice more virtual memory than
> actually need?
>
> Uwe Schindler ,thanks for your reply.
> First,I did not use reopens method, even when I start the my program, I
> can see this two big anon memory at once.
> Second,if I miss unmap,I will see some other index file map to virtual
> memory other than anon memory.
>
> -----邮件原件-----
> 发件人: Uwe Schindler [mailto:uwe@thetaphi.de]
> 发送时间: 2010年1月30日 16:47
> 收件人: java-user@lucene.apache.org
> 主题: RE: MMapDirectory need twice more virtual memory than actually
> need?
>
> MMapDirectory should only be used on 64bit machines with 64bit JVMs.
> This is described in the Javadocs.
>
> The additional sizes are caused by reopens and have their reason in
> specialties with the Java VM, described in 2.9/3.0 Javadocs of the
> class, please take a look. You can also enable a workaround for the
> missing “unmap” call.
>
>
>
> -----
>
> Uwe Schindler
>
> H.-H.-Meier-Allee 63, D-28213 Bremen
>
> <http://www.thetaphi.de/> http://www.thetaphi.de
>
> eMail: uwe@thetaphi.de
>
>
>
> From: luocanrao [mailto:luocan19826164@sohu.com]
> Sent: Saturday, January 30, 2010 8:31 AM
> To: java-user@lucene.apache.org
> Subject: MMapDirectory need twice more virtual memory than actually
> need?
>
>
>
> Environment: 64 bit linux,memory 8G
>
> When I used pmap instruction to see virtual memory, I found two big
> anon memory which grows with the index file size.
>
> I had the two following pictures to show the problem, can you explain?
>
> This is why I got out of memory exception in 32 bit machine.
> Because my index file size is only 800m.
>
> Anon
> memory picture
>
>
>
>
> Indexfile memory picture
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
答复: MMapDirectory need twice more virtual memory than actually need?
Posted by luocanrao <lu...@sohu.com>.
Uwe Schindler ,thanks for your reply.
First,I did not use reopens method, even when I start the my program, I can see this two big anon memory at once.
Second,if I miss unmap,I will see some other index file map to virtual memory other than anon memory.
-----邮件原件-----
发件人: Uwe Schindler [mailto:uwe@thetaphi.de]
发送时间: 2010年1月30日 16:47
收件人: java-user@lucene.apache.org
主题: RE: MMapDirectory need twice more virtual memory than actually need?
MMapDirectory should only be used on 64bit machines with 64bit JVMs. This is described in the Javadocs.
The additional sizes are caused by reopens and have their reason in specialties with the Java VM, described in 2.9/3.0 Javadocs of the class, please take a look. You can also enable a workaround for the missing “unmap” call.
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
<http://www.thetaphi.de/> http://www.thetaphi.de
eMail: uwe@thetaphi.de
From: luocanrao [mailto:luocan19826164@sohu.com]
Sent: Saturday, January 30, 2010 8:31 AM
To: java-user@lucene.apache.org
Subject: MMapDirectory need twice more virtual memory than actually need?
Environment: 64 bit linux,memory 8G
When I used pmap instruction to see virtual memory, I found two big anon memory which grows with the index file size.
I had the two following pictures to show the problem, can you explain?
This is why I got out of memory exception in 32 bit machine. Because my index file size is only 800m.
Anon memory picture
Indexfile memory picture
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: MMapDirectory need twice more virtual memory than actually need?
Posted by Uwe Schindler <uw...@thetaphi.de>.
MMapDirectory should only be used on 64bit machines with 64bit JVMs. This is described in the Javadocs.
The additional sizes are caused by reopens and have their reason in specialties with the Java VM, described in 2.9/3.0 Javadocs of the class, please take a look. You can also enable a workaround for the missing “unmap” call.
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
<http://www.thetaphi.de/> http://www.thetaphi.de
eMail: uwe@thetaphi.de
From: luocanrao [mailto:luocan19826164@sohu.com]
Sent: Saturday, January 30, 2010 8:31 AM
To: java-user@lucene.apache.org
Subject: MMapDirectory need twice more virtual memory than actually need?
Environment: 64 bit linux,memory 8G
When I used pmap instruction to see virtual memory, I found two big anon memory which grows with the index file size.
I had the two following pictures to show the problem, can you explain?
This is why I got out of memory exception in 32 bit machine. Because my index file size is only 800m.
Anon memory picture
Indexfile memory picture