[lucy-user] Not enough storage is available to process this command in Win 2012

[Thomas den Braber <th...@mediafiler.com>]

Hallo,

I have done some installations on Windows 2012 with a 32bit Strawberry Perl 
(64 bit is not yet ready) and I get frequent error like:

Not enough storage is available to process this command. 
(D:\sites\index\lcyindx10007\seg_3/sort-45.ord).
It looks like this error happens if there are many results returned from a 
query that needs to be sorted in a particular order.

There is enough memory available in the server and I have seen it on 
different installations (servers).
I use Strawberry Perl  5.16 (32bit) with modperl 2.08 and Apache 2.2.19 (32 
bit) from apachelounche (compiled with vc2010)

Do you have any idea how to fix that ? Maybe upgrading to 64bit will help, I 
will try that when the 64 bit is ready but that can take some time. It would 
be nice if there was an easy fix.

-
Thomas

Re: [lucy-user] Not enough storage is available to process this command in Win 2012

[Marvin Humphrey <ma...@rectangular.com>]

On Mon, Mar 23, 2015 at 9:15 AM, Thomas den Braber <th...@delos.nl> wrote:
>> Gory details: To support large files under 32-bit operating systems, we
>> avoid mapping the entire file into memory and instead use a "sliding
>> window" technique where only a portion of the file is mapped.
>
> And under 64 bit, is the "sliding window" option also used ?

It is not.  We simply map the whole file, because there's no possibility of
running out of address space.

Marvin Humphrey

Re: [lucy-user] Not enough storage is available to process this command in Win 2012

[Thomas den Braber <th...@delos.nl>]

> I don't recall having seen this particular error message before, but I 
believe
> that it occurs when `MapViewOfFile` (the Windows analogue to `mmap`) 
fails.

That is correct, I also see these errors in the logs.

> Gory details: To support large files under 32-bit operating systems, we 
avoid
> mapping the entire file into memory and instead use a "sliding window"
> technique where only a portion of the file is mapped.

And under 64 bit, is the "sliding window" option also used ?

> I speculate that
> if you have a lot of sortable fields, a lot more of the index is getting
> mapped into memory than we would like.

I have about 15 sortable fields at the moment, I will test with reducing the 
number to a minimum of 5 fields and check if it makes a difference.


> I don't recall having tested Lucy under 64-bit Windows and I'm not sure 
what
> it would take to make it work.  (Moving to 64-bit Unix would probably 
solve
> the problem, for whatever that's worth.)

Yes I prefer Linux but some companies have no knowledge of Linux, that makes 
do the installations on Windows.
> In the meantime, another possibility might be to cut down on the number of
> segments in the index.  Try making a copy of a problematic index and 
calling
> Indexer#optimize before Indexer#commit, which will cause the consolidation 
of
> all segments into one.  (This doesn't actually "optimize" much but it 
might
> help in this case.)

Yes I already done that but it still goes wrong.

I will do some more tests. If I have more results, I will post them.

Thanks for your support,

Thomas den Braber

Re: [lucy-user] Not enough storage is available to process this command in Win 2012

[Marvin Humphrey <ma...@rectangular.com>]

On Tue, Mar 17, 2015 at 9:46 AM, Thomas den Braber
<th...@mediafiler.com> wrote:
> Hallo,
>
> I have done some installations on Windows 2012 with a 32bit Strawberry Perl
> (64 bit is not yet ready) and I get frequent error like:
>
> Not enough storage is available to process this command.
> (D:\sites\index\lcyindx10007\seg_3/sort-45.ord).

I don't recall having seen this particular error message before, but I believe
that it occurs when `MapViewOfFile` (the Windows analogue to `mmap`) fails.

Gory details: To support large files under 32-bit operating systems, we avoid
mapping the entire file into memory and instead use a "sliding window"
technique where only a portion of the file is mapped.  But large indexes with
sort caches put pressure on the design because the minimum amount of address
space you can map is fairly large (I think 64k as opposed to typically 4k on
Unixen), and portions of each sort cache have to be mapped.  I speculate that
if you have a lot of sortable fields, a lot more of the index is getting
mapped into memory than we would like.  I also wonder whether the way
`MapViewOfFile` handles overlapping regions is optimal.

> It looks like this error happens if there are many results returned from a
> query that needs to be sorted in a particular order.
>
> There is enough memory available in the server and I have seen it on
> different installations (servers).
> I use Strawberry Perl  5.16 (32bit) with modperl 2.08 and Apache 2.2.19 (32
> bit) from apachelounche (compiled with vc2010)
>
> Do you have any idea how to fix that ? Maybe upgrading to 64bit will help, I
> will try that when the 64 bit is ready but that can take some time. It would
> be nice if there was an easy fix.

I don't recall having tested Lucy under 64-bit Windows and I'm not sure what
it would take to make it work.  (Moving to 64-bit Unix would probably solve
the problem, for whatever that's worth.)

In the meantime, another possibility might be to cut down on the number of
segments in the index.  Try making a copy of a problematic index and calling
Indexer#optimize before Indexer#commit, which will cause the consolidation of
all segments into one.  (This doesn't actually "optimize" much but it might
help in this case.)

Marvin Humphrey