TestHugeRamFile

[TJ Kolev <tj...@gmail.com>]

Greetings,

I thought I'd run the NUnit tests, and TestHugeFile() (in
/Test/Store/TestHugeRamFile.cs) fails:

System.OutOfMemoryException: Exception of type
'System.OutOfMemoryException' was thrown.
at Lucene.Net.Store.RAMFile.AddBuffer(Int32 size) in
D:\svn_lucene\src\Lucene.Net\Store\RAMFile.cs: line 85
at Lucene.Net.Store.RAMOutputStream.SwitchCurrentBuffer() in
D:\svn_lucene\src\Lucene.Net\Store\RAMOutputStream.cs: line 152
at Lucene.Net.Store.RAMOutputStream.WriteBytes(Byte[] b, Int32 offset,
Int32 len) in D:\svn_lucene\src\Lucene.Net\Store\RAMOutputStream.cs:
line 136
at Lucene.Net.Store.TestHugeRamFile.TestHugeFile() in
TestHugeRamFile.cs: line 83

The test tries to write out 2 * int.MaxValue bytes to RAMOutputStream.
I can't make it work for int.MaxValue bytes. It does not even work for
0.9. It works for 0.8 of int.MaxValue:

   private static readonly long MAX_VALUE = (long) (0.8 *
System.Int32.MaxValue);

I am running on .Net 3.5, Windows Server 2003, 4 GB RAM.

So, what are the expectations for this test? In my case, I was
expecting it to complete, since I had more memory than the needed 2
GB. And yet it failed. Then I found this article:
http://technet.microsoft.com/en-us/library/bb124810.aspx and I set the
/3GB switch. Same results. I don't know what is limiting the CLR from
allocating more memory.

Also, the test harness has its own class DenseRAMFile : RAMFile, with
the only purpose of doing its own NewBuffer(int size). However, the
base method virtual byte[] NewBuffer(int size) is not used, so
DenseRAMFile doesn't get its shot either. I am suspecting that the
intention was to have line 85 in /Lucene.Net/Store/RAMFile.cs in
AddBuffer(int size) look like this:

    byte[] buffer = NewBuffer(size); // instead of = new byte[size];

If this change is made, then TestHugeFile() passes, but as the
comments say, DenseRAMFile reuses a byte[] buffer, and does not
allocate huge amounts of memory, so it is not a huge ram test at
all...

tjk :)

Re: TestHugeRamFile

[TJ Kolev <tj...@gmail.com>]

I have come to the conclusion that the test as it is is not realistic
on a 32 bit system (at least on my Windows server 2003). 2 *
int.MaxValue is 4 GB, which is the maximum addressable space on a 32
bit system. No way the test can ever be able to have that much on its
own. Also, 2 GB is what user processes are allowed (the /3GB switch
looks hacky and I still could not make the test work), and that amount
is also in the ball park of impossible to be claimed by the test. On
my machine 0.8 * int.MaxValue sometimes succeeds, sometimes not - I
guess depending on what else I am running, and possible memory
fragmentation (although the test tries to get 1K chunks at a time). So
I figured 0.7 * int.MaxValue is a reasonable expectations for this
test on a 32 bit machine.

To that effect, how about this change to the test:

		[TestFixtureSetUp]
		public void FixtureSetup()
		{
			if (Is64bitPlatform())
			{
				MAX_VALUE =  2L * Int32.MaxValue;
			}
			else
			{
				MAX_VALUE = (long)(0.7 * Int32.MaxValue);
			}
		}

		private bool Is64bitPlatform()
		{
			return Marshal.SizeOf(typeof(IntPtr)) == 8;
		}

I would have preferred to have an #ifdef and check for the target
platform at compile time, but it seems such symbol is not provided by
the compiler.

tjk :)

On Mon, Jan 12, 2009 at 2:56 PM, TJ Kolev <tj...@gmail.com> wrote:
> Greetings,
>
> I thought I'd run the NUnit tests, and TestHugeFile() (in
> /Test/Store/TestHugeRamFile.cs) fails:
>
> System.OutOfMemoryException: Exception of type
> 'System.OutOfMemoryException' was thrown.
> at Lucene.Net.Store.RAMFile.AddBuffer(Int32 size) in
> D:\svn_lucene\src\Lucene.Net\Store\RAMFile.cs: line 85
> at Lucene.Net.Store.RAMOutputStream.SwitchCurrentBuffer() in
> D:\svn_lucene\src\Lucene.Net\Store\RAMOutputStream.cs: line 152
> at Lucene.Net.Store.RAMOutputStream.WriteBytes(Byte[] b, Int32 offset,
> Int32 len) in D:\svn_lucene\src\Lucene.Net\Store\RAMOutputStream.cs:
> line 136
> at Lucene.Net.Store.TestHugeRamFile.TestHugeFile() in
> TestHugeRamFile.cs: line 83
>
> The test tries to write out 2 * int.MaxValue bytes to RAMOutputStream.
> I can't make it work for int.MaxValue bytes. It does not even work for
> 0.9. It works for 0.8 of int.MaxValue:
>
>   private static readonly long MAX_VALUE = (long) (0.8 *
> System.Int32.MaxValue);
>
> I am running on .Net 3.5, Windows Server 2003, 4 GB RAM.
>
> So, what are the expectations for this test? In my case, I was
> expecting it to complete, since I had more memory than the needed 2
> GB. And yet it failed. Then I found this article:
> http://technet.microsoft.com/en-us/library/bb124810.aspx and I set the
> /3GB switch. Same results. I don't know what is limiting the CLR from
> allocating more memory.
>
> Also, the test harness has its own class DenseRAMFile : RAMFile, with
> the only purpose of doing its own NewBuffer(int size). However, the
> base method virtual byte[] NewBuffer(int size) is not used, so
> DenseRAMFile doesn't get its shot either. I am suspecting that the
> intention was to have line 85 in /Lucene.Net/Store/RAMFile.cs in
> AddBuffer(int size) look like this:
>
>    byte[] buffer = NewBuffer(size); // instead of = new byte[size];
>
> If this change is made, then TestHugeFile() passes, but as the
> comments say, DenseRAMFile reuses a byte[] buffer, and does not
> allocate huge amounts of memory, so it is not a huge ram test at
> all...
>
> tjk :)
>