Windows Server 2003 4GB file issue not solved

[Rob Pettefar <rp...@gpslsolutions.com>]

Hi guys
I am using Windows 7 Ultimate and I managed to get round the 4GB 
database limit using this installer:
https://github.com/dch/couchdb/downloads/setup-couchdb-1.0.2_otp_R14B01_spidermonkey_1.8.5.exe
With this I have a 5.8GB database that is happily running, even if I 
restart the service.

However when this was applied to a Windows Server 2003 Standard Edition 
machine the 4GB limit still seems to exist.
Is there something funky with Server 2003 that would cause this?
Is there a binary or installer available that would work instead?

Also do database files that grow to the magical 4GB limit become corrupt 
or are they usable on other setups without this issue?
(I have to ask as the 2003 Server machine is over in china and will take 
an age to transfer it for examination)

Any help on this matter would be greatly appreciated.
Rob

Re: Windows Server 2003 4GB file issue not solved

[Rob Pettefar <rp...@gpslsolutions.com>]

  Hi there guys
I did a full clean install of everything and things are working nicely 
again.
I think we were getting false negatives when testing with the 4GB 
database file. I didn't know that it would overwrite, I just assumed it 
was append only. This would probably explain why it can't be read properly.

Thanks for your help in resolving this
Rob

On 16/02/2011 09:00, Dave Cottlehuber wrote:
> On 16 February 2011 01:12, Rob Pettefar<rp...@gpslsolutions.com>  wrote:
>> Hi guys
>> I am using Windows 7 Ultimate and I managed to get round the 4GB database
>> limit using this installer:
>> https://github.com/dch/couchdb/downloads/setup-couchdb-1.0.2_otp_R14B01_spidermonkey_1.8.5.exe
>> With this I have a 5.8GB database that is happily running, even if I restart
>> the service.
>>
>> However when this was applied to a Windows Server 2003 Standard Edition
>> machine the 4GB limit still seems to exist.
> Hi Rob
>
> If this is still causing errors then it's pretty serious -
> over-written (lost) data at the head of the couch db file is the worst
> case scenario. The 1.0.2 binary was checked for this issue on both 32
> and 64bit platforms&  came up clean.
>
> Firstly more detail please -  William of Ockham[3] is betting on the first two
> - was this a clean install of CouchDB into a new folder, or
> overwriting an existing install (not recommended)?
> - are you running on an NTFS volume for /var/lib/couchdb/*?
> - what caused you to see a 4GiB limit? errors, logs, unexpected behaviour?
>
> History -
> The 4GiB error is caused by inability in erlang 13B04 and earlier
> versions to append data to a file that is already>  4 GiB on windows
> only. This means that:
>
> 1) A running CouchDB instance can't grow a db file from<  4 GiB to
> over 4 GiB without crashing.
>
> 2) On restart, the first time that>  4GiB db file has a reason to
> write - e.g. you add a doc - it starts over-writing not from the end
> (append) but from the beginning of the DB = lost header = possible
> lost data (I think depending if compaction is needed or not).
>
> This was first resolved in 14B01 only with Juhani's patch.
>
> 3 checks you can do on a *clean* folder install of CouchDB
>
> 1) confirm what erl/werl version you are using
> 2) confirm there are no issues at erlang level using werl.exe shell
> - create a large file (I use one around 128MiB)&  use Juhani's quick
> checker [1] below from the erlang shell to drive it past 4GiB. This
> should be successful. Note the size&  the content of the header of the
> file.
> - re-run the quick checker again - each time a segment is written you
> would normally (e.g. in 1 above) see the file size grow. It does not,
> and examining the header shows you have overwritten it.
>
> 3) less direct but more couchy:
> - make a large DB just under 4GiB = 4096 * 1024 * 1024 -1 max size. I
> upload many docs with large attachments for this.
> - stop couchdb&  record header and filesize of db.
> - upload more docs&  increase the db size past 4GiB -- if you can....
> - watch couch/erlang die if error is present
> http://friendpaste.com/6moY1vyUyIsIX4t5N4hZ1B?rev=626638353230
> - stop couchdb&  record header and filesize of db.
> - start couchdb&  relaunch the script again.
> - confirm header changed, and filesize has not, even though docs were written.
>
>> Is there something funky with Server 2003 that would cause this?
> Not to my knowledge. It would be an OS bug in this case if the API is
> inconsistent between XP/2003/2008 32 and 64 bit.
>
>> Is there a binary or installer available that would work instead?
> At the moment I suspect
>
>> Also do database files that grow to the magical 4GB limit become corrupt or
>> are they usable on other setups without this issue?
> All fine so long as you don't restart. Best is to stop couch, move the
> db files safely, and upgrade.
>
>> (I have to ask as the 2003 Server machine is over in china and will take an
>> age to transfer it for examination)
>>
>> Any help on this matter would be greatly appreciated.
>> Rob
>>
> Look for me on irc d_ch&  we can go through the gory bits. I'm in UTC
> + 12 but very keen to help here.
> A+
> Dave
>
> [3] http://www.phys.ncku.edu.tw/mirrors/physicsfaq/General/occam.html
> [2] http://friendpaste.com/6moY1vyUyIsIX4t5N4hZ1B?rev=626638353230
> [1] filetest.erl
>
> -module(filetest).
> -export([main/0]).
> main() ->
> {ok, Binary}=file:read_file("128MiB"),
> {ok, WriteDescr} = file:open("grow_past_4_GiB.file", [raw, append]),
>
> loop(1000, WriteDescr,Binary),
> file:close(WriteDescr).
>
> loop(0,_NotNeeded,_NotNeeded) ->  ok;
>
> loop(N,WriteDescr,Binary) ->
> file:write(WriteDescr,Binary),
> io:format("wrote ~w\n", [N]),
> loop(N-1,WriteDescr,Binary).

Re: Windows Server 2003 4GB file issue not solved

[Dave Cottlehuber <da...@muse.net.nz>]

On 16 February 2011 01:12, Rob Pettefar <rp...@gpslsolutions.com> wrote:
> Hi guys
> I am using Windows 7 Ultimate and I managed to get round the 4GB database
> limit using this installer:
> https://github.com/dch/couchdb/downloads/setup-couchdb-1.0.2_otp_R14B01_spidermonkey_1.8.5.exe
> With this I have a 5.8GB database that is happily running, even if I restart
> the service.
>
> However when this was applied to a Windows Server 2003 Standard Edition
> machine the 4GB limit still seems to exist.

Hi Rob

If this is still causing errors then it's pretty serious -
over-written (lost) data at the head of the couch db file is the worst
case scenario. The 1.0.2 binary was checked for this issue on both 32
and 64bit platforms & came up clean.

Firstly more detail please -  William of Ockham[3] is betting on the first two
- was this a clean install of CouchDB into a new folder, or
overwriting an existing install (not recommended)?
- are you running on an NTFS volume for /var/lib/couchdb/*?
- what caused you to see a 4GiB limit? errors, logs, unexpected behaviour?

History -
The 4GiB error is caused by inability in erlang 13B04 and earlier
versions to append data to a file that is already > 4 GiB on windows
only. This means that:

1) A running CouchDB instance can't grow a db file from < 4 GiB to
over 4 GiB without crashing.

2) On restart, the first time that  > 4GiB db file has a reason to
write - e.g. you add a doc - it starts over-writing not from the end
(append) but from the beginning of the DB = lost header = possible
lost data (I think depending if compaction is needed or not).

This was first resolved in 14B01 only with Juhani's patch.

3 checks you can do on a *clean* folder install of CouchDB

1) confirm what erl/werl version you are using
2) confirm there are no issues at erlang level using werl.exe shell
- create a large file (I use one around 128MiB) & use Juhani's quick
checker [1] below from the erlang shell to drive it past 4GiB. This
should be successful. Note the size & the content of the header of the
file.
- re-run the quick checker again - each time a segment is written you
would normally (e.g. in 1 above) see the file size grow. It does not,
and examining the header shows you have overwritten it.

3) less direct but more couchy:
- make a large DB just under 4GiB = 4096 * 1024 * 1024 -1 max size. I
upload many docs with large attachments for this.
- stop couchdb & record header and filesize of db.
- upload more docs & increase the db size past 4GiB -- if you can....
- watch couch/erlang die if error is present
http://friendpaste.com/6moY1vyUyIsIX4t5N4hZ1B?rev=626638353230
- stop couchdb & record header and filesize of db.
- start couchdb & relaunch the script again.
- confirm header changed, and filesize has not, even though docs were written.

> Is there something funky with Server 2003 that would cause this?

Not to my knowledge. It would be an OS bug in this case if the API is
inconsistent between XP/2003/2008 32 and 64 bit.

> Is there a binary or installer available that would work instead?
At the moment I suspect

> Also do database files that grow to the magical 4GB limit become corrupt or
> are they usable on other setups without this issue?

All fine so long as you don't restart. Best is to stop couch, move the
db files safely, and upgrade.

> (I have to ask as the 2003 Server machine is over in china and will take an
> age to transfer it for examination)
>
> Any help on this matter would be greatly appreciated.
> Rob
>

Look for me on irc d_ch & we can go through the gory bits. I'm in UTC
+ 12 but very keen to help here.
A+
Dave

[3] http://www.phys.ncku.edu.tw/mirrors/physicsfaq/General/occam.html
[2] http://friendpaste.com/6moY1vyUyIsIX4t5N4hZ1B?rev=626638353230
[1] filetest.erl

-module(filetest).
-export([main/0]).
main() ->
{ok, Binary}=file:read_file("128MiB"),
{ok, WriteDescr} = file:open("grow_past_4_GiB.file", [raw, append]),

loop(1000, WriteDescr,Binary),
file:close(WriteDescr).

loop(0,_NotNeeded,_NotNeeded) -> ok;

loop(N,WriteDescr,Binary) ->
file:write(WriteDescr,Binary),
io:format("wrote ~w\n", [N]),
loop(N-1,WriteDescr,Binary).

Re: Windows Server 2003 4GB file issue not solved

[Nikolai Teofilov <n....@gmail.com>]

Rob,

Are you sure you have removed the old version? Sometimes the erlang is still has running processes and it is just suggestion that you are running still the old version in the background.
Reinstall and reboot could help ... It is "Windows" 

Another suggestion run the couchdb not as windows service. 

http://support.microsoft.com/kb/283037
Is this somehow related to your problem?

  
Regards
Nikolai


On 15.02.2011, at 13:12, Rob Pettefar wrote:

> Hi guys
> I am using Windows 7 Ultimate and I managed to get round the 4GB database limit using this installer:
> https://github.com/dch/couchdb/downloads/setup-couchdb-1.0.2_otp_R14B01_spidermonkey_1.8.5.exe
> With this I have a 5.8GB database that is happily running, even if I restart the service.
> 
> However when this was applied to a Windows Server 2003 Standard Edition machine the 4GB limit still seems to exist.
> Is there something funky with Server 2003 that would cause this?
> Is there a binary or installer available that would work instead?
> 
> Also do database files that grow to the magical 4GB limit become corrupt or are they usable on other setups without this issue?
> (I have to ask as the 2003 Server machine is over in china and will take an age to transfer it for examination)
> 
> Any help on this matter would be greatly appreciated.
> Rob