You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Nick North <no...@gmail.com> on 2012/05/15 14:55:09 UTC
Replicated database size
I'm curious about the size of replicated CouchDb databases in comparison to
each other. I have four databases, each with pull replications from the
other three, but they report quite different data sizes. Two of them say:
{"db_name":"hydra","doc_count":1489060,"doc_del_count":2754893,"update_seq":6998882,"purge_seq":0,"compact_running":false,"disk_size":3213656193,"data_size":1395943755,"instance_start_time":"1337067567481841","disk_format_version":6,"committed_update_seq":6998882}
While the other two say this - note the difference in data_size:
{"db_name":"hydra","doc_count":1489441,"doc_del_count":2755302,"update_seq":4375865,"purge_seq":0,"compact_running":false,"disk_size":7599413027,"data_size":7265993199,"instance_start_time":"1337014746154865","disk_format_version":6,"committed_update_seq":4375865}
(There is some discrepancy in the doc_count because new documents are
being posted continuously, and some went in in between fetching stats
for the various instances.) Other possibly relevant information:
- All the replications appear to be in working order so I don't
believe there is a backlog of documents waiting to be replicated.
- The database has just one design view and whether or not it has
been queried does not seem to make any difference to whether the
database is "large" or "small".
- Compaction makes little difference, in that the "large" instances
always remain much larger than the "small" ones.
- Everything is running CouchDb 1.2 on Windows: the "small"
instances on Windows 7 and Windows Vista, and the "large" ones on
Windows Server 2008.
- File_compression is set to "snappy" in all cases and there are no
attachments anywhere.
Can anyone suggest what might be going on here? My best guess is that
it's to do with file compression on Windows Server but that is a
guess, so I'm intending to do some experimentation with the other file
compression options. I'd be grateful for any thoughts, as I'm planning
out disk requirements for a system with ten times the capacity of the
current one, and would very much like to be do that with some
certainty about file sizes. Thanks in advance for any help,
Nick North
Re: Replicated database size
Posted by Nick North <no...@gmail.com>.
Good luck, and many thanks for looking into this. For your sake, I wish it
had been a unique quirk on my setup.
Nick
On 22 May 2012 11:26, Dave Cottlehuber <da...@muse.net.nz> wrote:
> On 17 May 2012 12:07, Nick North <no...@gmail.com> wrote:
>
> > I'm getting this behaviour on 2008R2 and plain old 2008, and all my
> > instances are running the R15B build as a service.
> >
> > Nick
> >
> > On 17 May 2012 11:00, Dave Cottlehuber <da...@muse.net.nz> wrote:
> >
> > > Weird. What release of w2008 server is it? i'll spin one up & see if I
> > can
> > > get the same result. Is this the R15B or the R14B04 build? Presumably
> > > running as a service, right?
> > >
> > > On 17 May 2012 10:08, Nick North <no...@gmail.com> wrote:
> > >
> > > > Looking at erl.exe with procexp shows that both Win7 and Server 2008
> > are
> > > > using 10.0.40219.325. Installing your vcredist version asked if I
> > wanted
> > > to
> > > > repair the existing installation, so I let it go ahead. Unfortunately
> > it
> > > > made no difference: after installation the version in use was still
> > > > 10.0.40219.325 and the tests failed in the same way.
> > > >
> > > > Nick
> > > >
> > > > On 16 May 2012 22:58, Dave Cottlehuber <da...@muse.net.nz> wrote:
> > > >
> > > > > On 16 May 2012 20:37, Nick North <no...@gmail.com> wrote:
> > > > >
> > > > > > Thanks for replying - that test does show up a problem. It works
> > fine
> > > > on
> > > > > my
> > > > > > Win7 machine but, on Windows Server, says:
> > > > > >
> > > > > > Erlang R15B (erts-5.9) [source] [smp:2:2] [async-threads:0]
> > > > > >
> > > > > > Eshell V5.9 (abort with ^G)
> > > > > > 1> application:load(snappy).
> > > > > > ok
> > > > > > 2> application:start(snappy).
> > > > > > ok
> > > > > > 3> c(snappy_tests).
> > > > > > {ok,snappy_tests}
> > > > > > 4> snappy_tests:test().
> > > > > > snappy_tests: compression_test_...*failed*
> > > > > > ::snappy_nif_not_loaded
> > > > > >
> > > > > >
> > > > > > snappy_tests: decompression_test_...*failed*
> > > > > > ::snappy_nif_not_loaded
> > > > > >
> > > > > >
> > > > > > =======================================================
> > > > > > Failed: 2. Skipped: 0. Passed: 0.
> > > > > > error
> > > > > >
> > > > > > Both systems have snappy_nif.dll, in the
> > > > %COUCHDB%\lib\snappy-1.0.3\priv
> > > > > > directory and they look to be the same file, at least as far as
> can
> > > be
> > > > > > determined from Windows Explorer.
> > > > > >
> > > > > > C++ runtime versions are fairly different. The server has 2008
> x64
> > > and
> > > > > x86,
> > > > > > and 2010 x86. The Win7 machine has 2005 x64, 2008 x64 and x86,
> 2010
> > > x64
> > > > > and
> > > > > > x86 and the C++ 11 Beta x64. There are various sub-versions of
> > each,
> > > > and
> > > > > I
> > > > > > can give more details if that is useful.
> > > > > >
> > > > > >
> > > > > I'd like to know *exactly* what the server is using for
> > > vcredist_x86.exe
> > > > > (VC10 version
> > > > > only). You can see this with sigcheck[1] if you have the original
> > > > vcredist
> > > > > to hand.
> > > > >
> > > > > When erlang/couch is running you can use listdlls[2] or procexp[3]
> to
> > > > peek
> > > > > in and
> > > > > see what runtime it has decided to use.
> > > > >
> > > > > After all that, try installing this vcredist version[4] and seeing
> if
> > > > that
> > > > > resolves things.
> > > > >
> > > > > There are numerous bugs in the dll load path stuff on windows, and
> > IIRC
> > > > > different
> > > > > manifestations of that across OS versions. Let's hope this is not
> > the
> > > > > issue here!
> > > > >
> > > > > Finally, screw MS for releasing the same filename with 3
> consecutive
> > > VC++
> > > > > releases..
> > > > > that's plain dumb. I'll reserve my vituperation for twitter.
> > > > >
> > > > > A+
> > > > > Dave
> > > > >
> > > > > [1]: http://technet.microsoft.com/en-us/sysinternals/bb897441<
> > > > > http://technet.microsoft.com/en-us/sysinternals/bb897441.aspx>
> > > > > [2]: http://technet.microsoft.com/en-us/sysinternals/bb896656
> > > > > [3]: http://technet.microsoft.com/en-us/sysinternals/bb896653
> > > > > [4]:
> > > > >
> > > > >
> > > >
> > >
> >
> http://download.microsoft.com/download/1/6/5/165255E7-1014-4D0A-B094-B6A430A6BFFC/vcredist_x86.exe
> > > > > [5]: msdn.microsoft.com/en-us/library/windows/desktop/ms682586
> > > > >
> > > >
> > >
> >
>
> I can confirm that, snappy definitely fails on Windows 2008 R2 server. I'm
> still downloading SDKs and dependencies to see why.
>
> A+
> Dave
>
Re: Replicated database size
Posted by Dave Cottlehuber <da...@muse.net.nz>.
On 23 May 2012 09:55, Nick North <no...@gmail.com> wrote:
> Success! I put it my 2008 R2 machine and both the snappy tests and the
> entire test suite passed, so it looks good. As recommended, I won't use
> this in production until it's officially blessed - deflate seems to be
> doing a reasonable job in the meantime. Many thanks for finding and fixing
> the problem.
>
> Nick
>
>
I still don't know *why* it fails only on server, but its definitely a bug
needs squishing.
A+
Dave
Re: Replicated database size
Posted by Nick North <no...@gmail.com>.
Success! I put it my 2008 R2 machine and both the snappy tests and the
entire test suite passed, so it looks good. As recommended, I won't use
this in production until it's officially blessed - deflate seems to be
doing a reasonable job in the meantime. Many thanks for finding and fixing
the problem.
Nick
On 23 May 2012 00:01, Dave Cottlehuber <da...@muse.net.nz> wrote:
> On 22 May 2012 12:26, Dave Cottlehuber <da...@muse.net.nz> wrote:
>
> > On 17 May 2012 12:07, Nick North <no...@gmail.com> wrote:
> >
> >> I'm getting this behaviour on 2008R2 and plain old 2008, and all my
> >> instances are running the R15B build as a service.
> >>
> >> Nick
> >>
> >> On 17 May 2012 11:00, Dave Cottlehuber <da...@muse.net.nz> wrote:
> >>
> >> > Weird. What release of w2008 server is it? i'll spin one up & see if I
> >> can
> >> > get the same result. Is this the R15B or the R14B04 build? Presumably
> >> > running as a service, right?
> >> >
> >> > On 17 May 2012 10:08, Nick North <no...@gmail.com> wrote:
> >> >
> >> > > Looking at erl.exe with procexp shows that both Win7 and Server 2008
> >> are
> >> > > using 10.0.40219.325. Installing your vcredist version asked if I
> >> wanted
> >> > to
> >> > > repair the existing installation, so I let it go ahead.
> Unfortunately
> >> it
> >> > > made no difference: after installation the version in use was still
> >> > > 10.0.40219.325 and the tests failed in the same way.
> >> > >
> >> > > Nick
> >> > >
> >> > > On 16 May 2012 22:58, Dave Cottlehuber <da...@muse.net.nz> wrote:
> >> > >
> >> > > > On 16 May 2012 20:37, Nick North <no...@gmail.com> wrote:
> >> > > >
> >> > > > > Thanks for replying - that test does show up a problem. It works
> >> fine
> >> > > on
> >> > > > my
> >> > > > > Win7 machine but, on Windows Server, says:
> >> > > > >
> >> > > > > Erlang R15B (erts-5.9) [source] [smp:2:2] [async-threads:0]
> >> > > > >
> >> > > > > Eshell V5.9 (abort with ^G)
> >> > > > > 1> application:load(snappy).
> >> > > > > ok
> >> > > > > 2> application:start(snappy).
> >> > > > > ok
> >> > > > > 3> c(snappy_tests).
> >> > > > > {ok,snappy_tests}
> >> > > > > 4> snappy_tests:test().
> >> > > > > snappy_tests: compression_test_...*failed*
> >> > > > > ::snappy_nif_not_loaded
> >> > > > >
> >> > > > >
> >> > > > > snappy_tests: decompression_test_...*failed*
> >> > > > > ::snappy_nif_not_loaded
> >> > > > >
> >> > > > >
> >> > > > > =======================================================
> >> > > > > Failed: 2. Skipped: 0. Passed: 0.
> >> > > > > error
> >> > > > >
> >> > > > > Both systems have snappy_nif.dll, in the
> >> > > %COUCHDB%\lib\snappy-1.0.3\priv
> >> > > > > directory and they look to be the same file, at least as far as
> >> can
> >> > be
> >> > > > > determined from Windows Explorer.
> >> > > > >
> >> > > > > C++ runtime versions are fairly different. The server has 2008
> x64
> >> > and
> >> > > > x86,
> >> > > > > and 2010 x86. The Win7 machine has 2005 x64, 2008 x64 and x86,
> >> 2010
> >> > x64
> >> > > > and
> >> > > > > x86 and the C++ 11 Beta x64. There are various sub-versions of
> >> each,
> >> > > and
> >> > > > I
> >> > > > > can give more details if that is useful.
> >> > > > >
> >> > > > >
> >> > > > I'd like to know *exactly* what the server is using for
> >> > vcredist_x86.exe
> >> > > > (VC10 version
> >> > > > only). You can see this with sigcheck[1] if you have the original
> >> > > vcredist
> >> > > > to hand.
> >> > > >
> >> > > > When erlang/couch is running you can use listdlls[2] or procexp[3]
> >> to
> >> > > peek
> >> > > > in and
> >> > > > see what runtime it has decided to use.
> >> > > >
> >> > > > After all that, try installing this vcredist version[4] and seeing
> >> if
> >> > > that
> >> > > > resolves things.
> >> > > >
> >> > > > There are numerous bugs in the dll load path stuff on windows, and
> >> IIRC
> >> > > > different
> >> > > > manifestations of that across OS versions. Let's hope this is not
> >> the
> >> > > > issue here!
> >> > > >
> >> > > > Finally, screw MS for releasing the same filename with 3
> consecutive
> >> > VC++
> >> > > > releases..
> >> > > > that's plain dumb. I'll reserve my vituperation for twitter.
> >> > > >
> >> > > > A+
> >> > > > Dave
> >> > > >
> >> > > > [1]: http://technet.microsoft.com/en-us/sysinternals/bb897441<
> >> > > > http://technet.microsoft.com/en-us/sysinternals/bb897441.aspx>
> >> > > > [2]: http://technet.microsoft.com/en-us/sysinternals/bb896656
> >> > > > [3]: http://technet.microsoft.com/en-us/sysinternals/bb896653
> >> > > > [4]:
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> http://download.microsoft.com/download/1/6/5/165255E7-1014-4D0A-B094-B6A430A6BFFC/vcredist_x86.exe
> >> > > > [5]: msdn.microsoft.com/en-us/library/windows/desktop/ms682586
> >> > > >
> >> > >
> >> >
> >>
> >
> > I can confirm that, snappy definitely fails on Windows 2008 R2 server.
> I'm
> > still downloading SDKs and dependencies to see why.
> >
> > A+
> > Dave
> >
> >
> Nick, here's a (very) fresh build [1], not for prod use but it
> should allow you to confirm if the original issue is fixed.
> It's R15B01 with static OpenSSL included, + a workaround
> for the issue [2]. The check is to run depends.exe against
> the snappy_nif.dll and confirm the *100d.DLL runtimes are gone.
>
> Now I am not yet sure if the fix *cough* hack I applied
> is a minimal fix or not so please just use this for testing.
>
> You can install it alongside e.g. c:\tmp\1.2.0, tweak local.ini
>
> [httpd]
> port = 5985
>
> and run the .bat file (no service reqd) without screwing your
> production stuff. The same test as before is sufficient.
>
> A+
> Dave
>
> [1]:
> https://www.dropbox.com/sh/jeifcxpbtpo78ak/Yri8o1U6ig/snapshots/20120523
> [2]: https://issues.apache.org/jira/browse/COUCHDB-1482
> [3]: http://www.cmake.org/
>
Re: Replicated database size
Posted by Dave Cottlehuber <da...@muse.net.nz>.
On 22 May 2012 12:26, Dave Cottlehuber <da...@muse.net.nz> wrote:
> On 17 May 2012 12:07, Nick North <no...@gmail.com> wrote:
>
>> I'm getting this behaviour on 2008R2 and plain old 2008, and all my
>> instances are running the R15B build as a service.
>>
>> Nick
>>
>> On 17 May 2012 11:00, Dave Cottlehuber <da...@muse.net.nz> wrote:
>>
>> > Weird. What release of w2008 server is it? i'll spin one up & see if I
>> can
>> > get the same result. Is this the R15B or the R14B04 build? Presumably
>> > running as a service, right?
>> >
>> > On 17 May 2012 10:08, Nick North <no...@gmail.com> wrote:
>> >
>> > > Looking at erl.exe with procexp shows that both Win7 and Server 2008
>> are
>> > > using 10.0.40219.325. Installing your vcredist version asked if I
>> wanted
>> > to
>> > > repair the existing installation, so I let it go ahead. Unfortunately
>> it
>> > > made no difference: after installation the version in use was still
>> > > 10.0.40219.325 and the tests failed in the same way.
>> > >
>> > > Nick
>> > >
>> > > On 16 May 2012 22:58, Dave Cottlehuber <da...@muse.net.nz> wrote:
>> > >
>> > > > On 16 May 2012 20:37, Nick North <no...@gmail.com> wrote:
>> > > >
>> > > > > Thanks for replying - that test does show up a problem. It works
>> fine
>> > > on
>> > > > my
>> > > > > Win7 machine but, on Windows Server, says:
>> > > > >
>> > > > > Erlang R15B (erts-5.9) [source] [smp:2:2] [async-threads:0]
>> > > > >
>> > > > > Eshell V5.9 (abort with ^G)
>> > > > > 1> application:load(snappy).
>> > > > > ok
>> > > > > 2> application:start(snappy).
>> > > > > ok
>> > > > > 3> c(snappy_tests).
>> > > > > {ok,snappy_tests}
>> > > > > 4> snappy_tests:test().
>> > > > > snappy_tests: compression_test_...*failed*
>> > > > > ::snappy_nif_not_loaded
>> > > > >
>> > > > >
>> > > > > snappy_tests: decompression_test_...*failed*
>> > > > > ::snappy_nif_not_loaded
>> > > > >
>> > > > >
>> > > > > =======================================================
>> > > > > Failed: 2. Skipped: 0. Passed: 0.
>> > > > > error
>> > > > >
>> > > > > Both systems have snappy_nif.dll, in the
>> > > %COUCHDB%\lib\snappy-1.0.3\priv
>> > > > > directory and they look to be the same file, at least as far as
>> can
>> > be
>> > > > > determined from Windows Explorer.
>> > > > >
>> > > > > C++ runtime versions are fairly different. The server has 2008 x64
>> > and
>> > > > x86,
>> > > > > and 2010 x86. The Win7 machine has 2005 x64, 2008 x64 and x86,
>> 2010
>> > x64
>> > > > and
>> > > > > x86 and the C++ 11 Beta x64. There are various sub-versions of
>> each,
>> > > and
>> > > > I
>> > > > > can give more details if that is useful.
>> > > > >
>> > > > >
>> > > > I'd like to know *exactly* what the server is using for
>> > vcredist_x86.exe
>> > > > (VC10 version
>> > > > only). You can see this with sigcheck[1] if you have the original
>> > > vcredist
>> > > > to hand.
>> > > >
>> > > > When erlang/couch is running you can use listdlls[2] or procexp[3]
>> to
>> > > peek
>> > > > in and
>> > > > see what runtime it has decided to use.
>> > > >
>> > > > After all that, try installing this vcredist version[4] and seeing
>> if
>> > > that
>> > > > resolves things.
>> > > >
>> > > > There are numerous bugs in the dll load path stuff on windows, and
>> IIRC
>> > > > different
>> > > > manifestations of that across OS versions. Let's hope this is not
>> the
>> > > > issue here!
>> > > >
>> > > > Finally, screw MS for releasing the same filename with 3 consecutive
>> > VC++
>> > > > releases..
>> > > > that's plain dumb. I'll reserve my vituperation for twitter.
>> > > >
>> > > > A+
>> > > > Dave
>> > > >
>> > > > [1]: http://technet.microsoft.com/en-us/sysinternals/bb897441<
>> > > > http://technet.microsoft.com/en-us/sysinternals/bb897441.aspx>
>> > > > [2]: http://technet.microsoft.com/en-us/sysinternals/bb896656
>> > > > [3]: http://technet.microsoft.com/en-us/sysinternals/bb896653
>> > > > [4]:
>> > > >
>> > > >
>> > >
>> >
>> http://download.microsoft.com/download/1/6/5/165255E7-1014-4D0A-B094-B6A430A6BFFC/vcredist_x86.exe
>> > > > [5]: msdn.microsoft.com/en-us/library/windows/desktop/ms682586
>> > > >
>> > >
>> >
>>
>
> I can confirm that, snappy definitely fails on Windows 2008 R2 server. I'm
> still downloading SDKs and dependencies to see why.
>
> A+
> Dave
>
>
Nick, here's a (very) fresh build [1], not for prod use but it
should allow you to confirm if the original issue is fixed.
It's R15B01 with static OpenSSL included, + a workaround
for the issue [2]. The check is to run depends.exe against
the snappy_nif.dll and confirm the *100d.DLL runtimes are gone.
Now I am not yet sure if the fix *cough* hack I applied
is a minimal fix or not so please just use this for testing.
You can install it alongside e.g. c:\tmp\1.2.0, tweak local.ini
[httpd]
port = 5985
and run the .bat file (no service reqd) without screwing your
production stuff. The same test as before is sufficient.
A+
Dave
[1]:
https://www.dropbox.com/sh/jeifcxpbtpo78ak/Yri8o1U6ig/snapshots/20120523
[2]: https://issues.apache.org/jira/browse/COUCHDB-1482
[3]: http://www.cmake.org/
Re: Replicated database size
Posted by Dave Cottlehuber <da...@muse.net.nz>.
On 17 May 2012 12:07, Nick North <no...@gmail.com> wrote:
> I'm getting this behaviour on 2008R2 and plain old 2008, and all my
> instances are running the R15B build as a service.
>
> Nick
>
> On 17 May 2012 11:00, Dave Cottlehuber <da...@muse.net.nz> wrote:
>
> > Weird. What release of w2008 server is it? i'll spin one up & see if I
> can
> > get the same result. Is this the R15B or the R14B04 build? Presumably
> > running as a service, right?
> >
> > On 17 May 2012 10:08, Nick North <no...@gmail.com> wrote:
> >
> > > Looking at erl.exe with procexp shows that both Win7 and Server 2008
> are
> > > using 10.0.40219.325. Installing your vcredist version asked if I
> wanted
> > to
> > > repair the existing installation, so I let it go ahead. Unfortunately
> it
> > > made no difference: after installation the version in use was still
> > > 10.0.40219.325 and the tests failed in the same way.
> > >
> > > Nick
> > >
> > > On 16 May 2012 22:58, Dave Cottlehuber <da...@muse.net.nz> wrote:
> > >
> > > > On 16 May 2012 20:37, Nick North <no...@gmail.com> wrote:
> > > >
> > > > > Thanks for replying - that test does show up a problem. It works
> fine
> > > on
> > > > my
> > > > > Win7 machine but, on Windows Server, says:
> > > > >
> > > > > Erlang R15B (erts-5.9) [source] [smp:2:2] [async-threads:0]
> > > > >
> > > > > Eshell V5.9 (abort with ^G)
> > > > > 1> application:load(snappy).
> > > > > ok
> > > > > 2> application:start(snappy).
> > > > > ok
> > > > > 3> c(snappy_tests).
> > > > > {ok,snappy_tests}
> > > > > 4> snappy_tests:test().
> > > > > snappy_tests: compression_test_...*failed*
> > > > > ::snappy_nif_not_loaded
> > > > >
> > > > >
> > > > > snappy_tests: decompression_test_...*failed*
> > > > > ::snappy_nif_not_loaded
> > > > >
> > > > >
> > > > > =======================================================
> > > > > Failed: 2. Skipped: 0. Passed: 0.
> > > > > error
> > > > >
> > > > > Both systems have snappy_nif.dll, in the
> > > %COUCHDB%\lib\snappy-1.0.3\priv
> > > > > directory and they look to be the same file, at least as far as can
> > be
> > > > > determined from Windows Explorer.
> > > > >
> > > > > C++ runtime versions are fairly different. The server has 2008 x64
> > and
> > > > x86,
> > > > > and 2010 x86. The Win7 machine has 2005 x64, 2008 x64 and x86, 2010
> > x64
> > > > and
> > > > > x86 and the C++ 11 Beta x64. There are various sub-versions of
> each,
> > > and
> > > > I
> > > > > can give more details if that is useful.
> > > > >
> > > > >
> > > > I'd like to know *exactly* what the server is using for
> > vcredist_x86.exe
> > > > (VC10 version
> > > > only). You can see this with sigcheck[1] if you have the original
> > > vcredist
> > > > to hand.
> > > >
> > > > When erlang/couch is running you can use listdlls[2] or procexp[3] to
> > > peek
> > > > in and
> > > > see what runtime it has decided to use.
> > > >
> > > > After all that, try installing this vcredist version[4] and seeing if
> > > that
> > > > resolves things.
> > > >
> > > > There are numerous bugs in the dll load path stuff on windows, and
> IIRC
> > > > different
> > > > manifestations of that across OS versions. Let's hope this is not
> the
> > > > issue here!
> > > >
> > > > Finally, screw MS for releasing the same filename with 3 consecutive
> > VC++
> > > > releases..
> > > > that's plain dumb. I'll reserve my vituperation for twitter.
> > > >
> > > > A+
> > > > Dave
> > > >
> > > > [1]: http://technet.microsoft.com/en-us/sysinternals/bb897441<
> > > > http://technet.microsoft.com/en-us/sysinternals/bb897441.aspx>
> > > > [2]: http://technet.microsoft.com/en-us/sysinternals/bb896656
> > > > [3]: http://technet.microsoft.com/en-us/sysinternals/bb896653
> > > > [4]:
> > > >
> > > >
> > >
> >
> http://download.microsoft.com/download/1/6/5/165255E7-1014-4D0A-B094-B6A430A6BFFC/vcredist_x86.exe
> > > > [5]: msdn.microsoft.com/en-us/library/windows/desktop/ms682586
> > > >
> > >
> >
>
I can confirm that, snappy definitely fails on Windows 2008 R2 server. I'm
still downloading SDKs and dependencies to see why.
A+
Dave
Re: Replicated database size
Posted by Nick North <no...@gmail.com>.
I'm getting this behaviour on 2008R2 and plain old 2008, and all my
instances are running the R15B build as a service.
Nick
On 17 May 2012 11:00, Dave Cottlehuber <da...@muse.net.nz> wrote:
> Weird. What release of w2008 server is it? i'll spin one up & see if I can
> get the same result. Is this the R15B or the R14B04 build? Presumably
> running as a service, right?
>
> On 17 May 2012 10:08, Nick North <no...@gmail.com> wrote:
>
> > Looking at erl.exe with procexp shows that both Win7 and Server 2008 are
> > using 10.0.40219.325. Installing your vcredist version asked if I wanted
> to
> > repair the existing installation, so I let it go ahead. Unfortunately it
> > made no difference: after installation the version in use was still
> > 10.0.40219.325 and the tests failed in the same way.
> >
> > Nick
> >
> > On 16 May 2012 22:58, Dave Cottlehuber <da...@muse.net.nz> wrote:
> >
> > > On 16 May 2012 20:37, Nick North <no...@gmail.com> wrote:
> > >
> > > > Thanks for replying - that test does show up a problem. It works fine
> > on
> > > my
> > > > Win7 machine but, on Windows Server, says:
> > > >
> > > > Erlang R15B (erts-5.9) [source] [smp:2:2] [async-threads:0]
> > > >
> > > > Eshell V5.9 (abort with ^G)
> > > > 1> application:load(snappy).
> > > > ok
> > > > 2> application:start(snappy).
> > > > ok
> > > > 3> c(snappy_tests).
> > > > {ok,snappy_tests}
> > > > 4> snappy_tests:test().
> > > > snappy_tests: compression_test_...*failed*
> > > > ::snappy_nif_not_loaded
> > > >
> > > >
> > > > snappy_tests: decompression_test_...*failed*
> > > > ::snappy_nif_not_loaded
> > > >
> > > >
> > > > =======================================================
> > > > Failed: 2. Skipped: 0. Passed: 0.
> > > > error
> > > >
> > > > Both systems have snappy_nif.dll, in the
> > %COUCHDB%\lib\snappy-1.0.3\priv
> > > > directory and they look to be the same file, at least as far as can
> be
> > > > determined from Windows Explorer.
> > > >
> > > > C++ runtime versions are fairly different. The server has 2008 x64
> and
> > > x86,
> > > > and 2010 x86. The Win7 machine has 2005 x64, 2008 x64 and x86, 2010
> x64
> > > and
> > > > x86 and the C++ 11 Beta x64. There are various sub-versions of each,
> > and
> > > I
> > > > can give more details if that is useful.
> > > >
> > > >
> > > I'd like to know *exactly* what the server is using for
> vcredist_x86.exe
> > > (VC10 version
> > > only). You can see this with sigcheck[1] if you have the original
> > vcredist
> > > to hand.
> > >
> > > When erlang/couch is running you can use listdlls[2] or procexp[3] to
> > peek
> > > in and
> > > see what runtime it has decided to use.
> > >
> > > After all that, try installing this vcredist version[4] and seeing if
> > that
> > > resolves things.
> > >
> > > There are numerous bugs in the dll load path stuff on windows, and IIRC
> > > different
> > > manifestations of that across OS versions. Let's hope this is not the
> > > issue here!
> > >
> > > Finally, screw MS for releasing the same filename with 3 consecutive
> VC++
> > > releases..
> > > that's plain dumb. I'll reserve my vituperation for twitter.
> > >
> > > A+
> > > Dave
> > >
> > > [1]: http://technet.microsoft.com/en-us/sysinternals/bb897441<
> > > http://technet.microsoft.com/en-us/sysinternals/bb897441.aspx>
> > > [2]: http://technet.microsoft.com/en-us/sysinternals/bb896656
> > > [3]: http://technet.microsoft.com/en-us/sysinternals/bb896653
> > > [4]:
> > >
> > >
> >
> http://download.microsoft.com/download/1/6/5/165255E7-1014-4D0A-B094-B6A430A6BFFC/vcredist_x86.exe
> > > [5]: msdn.microsoft.com/en-us/library/windows/desktop/ms682586
> > >
> >
>
Re: Replicated database size
Posted by Dave Cottlehuber <da...@muse.net.nz>.
Weird. What release of w2008 server is it? i'll spin one up & see if I can
get the same result. Is this the R15B or the R14B04 build? Presumably
running as a service, right?
On 17 May 2012 10:08, Nick North <no...@gmail.com> wrote:
> Looking at erl.exe with procexp shows that both Win7 and Server 2008 are
> using 10.0.40219.325. Installing your vcredist version asked if I wanted to
> repair the existing installation, so I let it go ahead. Unfortunately it
> made no difference: after installation the version in use was still
> 10.0.40219.325 and the tests failed in the same way.
>
> Nick
>
> On 16 May 2012 22:58, Dave Cottlehuber <da...@muse.net.nz> wrote:
>
> > On 16 May 2012 20:37, Nick North <no...@gmail.com> wrote:
> >
> > > Thanks for replying - that test does show up a problem. It works fine
> on
> > my
> > > Win7 machine but, on Windows Server, says:
> > >
> > > Erlang R15B (erts-5.9) [source] [smp:2:2] [async-threads:0]
> > >
> > > Eshell V5.9 (abort with ^G)
> > > 1> application:load(snappy).
> > > ok
> > > 2> application:start(snappy).
> > > ok
> > > 3> c(snappy_tests).
> > > {ok,snappy_tests}
> > > 4> snappy_tests:test().
> > > snappy_tests: compression_test_...*failed*
> > > ::snappy_nif_not_loaded
> > >
> > >
> > > snappy_tests: decompression_test_...*failed*
> > > ::snappy_nif_not_loaded
> > >
> > >
> > > =======================================================
> > > Failed: 2. Skipped: 0. Passed: 0.
> > > error
> > >
> > > Both systems have snappy_nif.dll, in the
> %COUCHDB%\lib\snappy-1.0.3\priv
> > > directory and they look to be the same file, at least as far as can be
> > > determined from Windows Explorer.
> > >
> > > C++ runtime versions are fairly different. The server has 2008 x64 and
> > x86,
> > > and 2010 x86. The Win7 machine has 2005 x64, 2008 x64 and x86, 2010 x64
> > and
> > > x86 and the C++ 11 Beta x64. There are various sub-versions of each,
> and
> > I
> > > can give more details if that is useful.
> > >
> > >
> > I'd like to know *exactly* what the server is using for vcredist_x86.exe
> > (VC10 version
> > only). You can see this with sigcheck[1] if you have the original
> vcredist
> > to hand.
> >
> > When erlang/couch is running you can use listdlls[2] or procexp[3] to
> peek
> > in and
> > see what runtime it has decided to use.
> >
> > After all that, try installing this vcredist version[4] and seeing if
> that
> > resolves things.
> >
> > There are numerous bugs in the dll load path stuff on windows, and IIRC
> > different
> > manifestations of that across OS versions. Let's hope this is not the
> > issue here!
> >
> > Finally, screw MS for releasing the same filename with 3 consecutive VC++
> > releases..
> > that's plain dumb. I'll reserve my vituperation for twitter.
> >
> > A+
> > Dave
> >
> > [1]: http://technet.microsoft.com/en-us/sysinternals/bb897441<
> > http://technet.microsoft.com/en-us/sysinternals/bb897441.aspx>
> > [2]: http://technet.microsoft.com/en-us/sysinternals/bb896656
> > [3]: http://technet.microsoft.com/en-us/sysinternals/bb896653
> > [4]:
> >
> >
> http://download.microsoft.com/download/1/6/5/165255E7-1014-4D0A-B094-B6A430A6BFFC/vcredist_x86.exe
> > [5]: msdn.microsoft.com/en-us/library/windows/desktop/ms682586
> >
>
Re: Replicated database size
Posted by Nick North <no...@gmail.com>.
Looking at erl.exe with procexp shows that both Win7 and Server 2008 are
using 10.0.40219.325. Installing your vcredist version asked if I wanted to
repair the existing installation, so I let it go ahead. Unfortunately it
made no difference: after installation the version in use was still
10.0.40219.325 and the tests failed in the same way.
Nick
On 16 May 2012 22:58, Dave Cottlehuber <da...@muse.net.nz> wrote:
> On 16 May 2012 20:37, Nick North <no...@gmail.com> wrote:
>
> > Thanks for replying - that test does show up a problem. It works fine on
> my
> > Win7 machine but, on Windows Server, says:
> >
> > Erlang R15B (erts-5.9) [source] [smp:2:2] [async-threads:0]
> >
> > Eshell V5.9 (abort with ^G)
> > 1> application:load(snappy).
> > ok
> > 2> application:start(snappy).
> > ok
> > 3> c(snappy_tests).
> > {ok,snappy_tests}
> > 4> snappy_tests:test().
> > snappy_tests: compression_test_...*failed*
> > ::snappy_nif_not_loaded
> >
> >
> > snappy_tests: decompression_test_...*failed*
> > ::snappy_nif_not_loaded
> >
> >
> > =======================================================
> > Failed: 2. Skipped: 0. Passed: 0.
> > error
> >
> > Both systems have snappy_nif.dll, in the %COUCHDB%\lib\snappy-1.0.3\priv
> > directory and they look to be the same file, at least as far as can be
> > determined from Windows Explorer.
> >
> > C++ runtime versions are fairly different. The server has 2008 x64 and
> x86,
> > and 2010 x86. The Win7 machine has 2005 x64, 2008 x64 and x86, 2010 x64
> and
> > x86 and the C++ 11 Beta x64. There are various sub-versions of each, and
> I
> > can give more details if that is useful.
> >
> >
> I'd like to know *exactly* what the server is using for vcredist_x86.exe
> (VC10 version
> only). You can see this with sigcheck[1] if you have the original vcredist
> to hand.
>
> When erlang/couch is running you can use listdlls[2] or procexp[3] to peek
> in and
> see what runtime it has decided to use.
>
> After all that, try installing this vcredist version[4] and seeing if that
> resolves things.
>
> There are numerous bugs in the dll load path stuff on windows, and IIRC
> different
> manifestations of that across OS versions. Let's hope this is not the
> issue here!
>
> Finally, screw MS for releasing the same filename with 3 consecutive VC++
> releases..
> that's plain dumb. I'll reserve my vituperation for twitter.
>
> A+
> Dave
>
> [1]: http://technet.microsoft.com/en-us/sysinternals/bb897441<
> http://technet.microsoft.com/en-us/sysinternals/bb897441.aspx>
> [2]: http://technet.microsoft.com/en-us/sysinternals/bb896656
> [3]: http://technet.microsoft.com/en-us/sysinternals/bb896653
> [4]:
>
> http://download.microsoft.com/download/1/6/5/165255E7-1014-4D0A-B094-B6A430A6BFFC/vcredist_x86.exe
> [5]: msdn.microsoft.com/en-us/library/windows/desktop/ms682586
>
Re: Replicated database size
Posted by Dave Cottlehuber <da...@muse.net.nz>.
On 16 May 2012 20:37, Nick North <no...@gmail.com> wrote:
> Thanks for replying - that test does show up a problem. It works fine on my
> Win7 machine but, on Windows Server, says:
>
> Erlang R15B (erts-5.9) [source] [smp:2:2] [async-threads:0]
>
> Eshell V5.9 (abort with ^G)
> 1> application:load(snappy).
> ok
> 2> application:start(snappy).
> ok
> 3> c(snappy_tests).
> {ok,snappy_tests}
> 4> snappy_tests:test().
> snappy_tests: compression_test_...*failed*
> ::snappy_nif_not_loaded
>
>
> snappy_tests: decompression_test_...*failed*
> ::snappy_nif_not_loaded
>
>
> =======================================================
> Failed: 2. Skipped: 0. Passed: 0.
> error
>
> Both systems have snappy_nif.dll, in the %COUCHDB%\lib\snappy-1.0.3\priv
> directory and they look to be the same file, at least as far as can be
> determined from Windows Explorer.
>
> C++ runtime versions are fairly different. The server has 2008 x64 and x86,
> and 2010 x86. The Win7 machine has 2005 x64, 2008 x64 and x86, 2010 x64 and
> x86 and the C++ 11 Beta x64. There are various sub-versions of each, and I
> can give more details if that is useful.
>
>
I'd like to know *exactly* what the server is using for vcredist_x86.exe
(VC10 version
only). You can see this with sigcheck[1] if you have the original vcredist
to hand.
When erlang/couch is running you can use listdlls[2] or procexp[3] to peek
in and
see what runtime it has decided to use.
After all that, try installing this vcredist version[4] and seeing if that
resolves things.
There are numerous bugs in the dll load path stuff on windows, and IIRC
different
manifestations of that across OS versions. Let's hope this is not the
issue here!
Finally, screw MS for releasing the same filename with 3 consecutive VC++
releases..
that's plain dumb. I'll reserve my vituperation for twitter.
A+
Dave
[1]: http://technet.microsoft.com/en-us/sysinternals/bb897441<http://technet.microsoft.com/en-us/sysinternals/bb897441.aspx>
[2]: http://technet.microsoft.com/en-us/sysinternals/bb896656
[3]: http://technet.microsoft.com/en-us/sysinternals/bb896653
[4]:
http://download.microsoft.com/download/1/6/5/165255E7-1014-4D0A-B094-B6A430A6BFFC/vcredist_x86.exe
[5]: msdn.microsoft.com/en-us/library/windows/desktop/ms682586
Re: Replicated database size
Posted by Nick North <no...@gmail.com>.
Thanks for replying - that test does show up a problem. It works fine on my
Win7 machine but, on Windows Server, says:
Erlang R15B (erts-5.9) [source] [smp:2:2] [async-threads:0]
Eshell V5.9 (abort with ^G)
1> application:load(snappy).
ok
2> application:start(snappy).
ok
3> c(snappy_tests).
{ok,snappy_tests}
4> snappy_tests:test().
snappy_tests: compression_test_...*failed*
::snappy_nif_not_loaded
snappy_tests: decompression_test_...*failed*
::snappy_nif_not_loaded
=======================================================
Failed: 2. Skipped: 0. Passed: 0.
error
Both systems have snappy_nif.dll, in the %COUCHDB%\lib\snappy-1.0.3\priv
directory and they look to be the same file, at least as far as can be
determined from Windows Explorer.
C++ runtime versions are fairly different. The server has 2008 x64 and x86,
and 2010 x86. The Win7 machine has 2005 x64, 2008 x64 and x86, 2010 x64 and
x86 and the C++ 11 Beta x64. There are various sub-versions of each, and I
can give more details if that is useful.
Thanks for your help with this, and do let me know what else I can do to
track down the problem.
Nick
On 16 May 2012 18:36, Dave Cottlehuber <da...@muse.net.nz> wrote:
> On 16 May 2012 16:49, Nick North <no...@gmail.com> wrote:
>
> > Following up on my own email, this seems to be an issue with snappy on
> > Windows Server 2008. When I changed the file_compression setting to
> > deflate_6, the "large" databases went down from 7GB to 1GB after
> > compaction. I'm not entirely sure if this counts as a bug so I won't
> raise
> > an issue on it.
> >
>
> Something's not right here. Are you able to run a couple tests on both
> instances?
>
>
> https://www.dropbox.com/sh/jeifcxpbtpo78ak/wvnm1MHmP9/testing/snappy_tests.erl
>
> Copy this file to your %COUCHDB%/bin/ folder. Then run werl.exe shell (i.e.
> not couch).
>
> Erlang R15B (erts-5.9) [source] [smp:2:2] [async-threads:0]
>
> Eshell V5.9 (abort with ^G)
> 1> application:load(snappy).
> ok
> 2> application:start(snappy).
> ok
> 3> c(snappy_tests).
> {ok,snappy_tests}
> 4> snappy_tests:test().
> All 2 tests passed.
> ok
> 5>
>
> Hopefully somewhere in there it spits out a sensible error message & we can
> work from there.
>
> Also if there's a way to find what version(s) of vcredist_x86 are installed
> on the
> server versions vs the other ones, that might help.
>
> By the way: kudos to whoever wrote the code to deal with file_compression.
> > When I changed file_compression to deflate_6, the system happily worked
> > with the existing, supposedly snappy-compressed databases, and converted
> > format on the next compaction. That could have gone wrong in several
> ways,
> > but didn't, so thank you.
> >
> > That is Filipe's work.
>
> A+
> Dave
>
Re: Replicated database size
Posted by Dave Cottlehuber <da...@muse.net.nz>.
On 16 May 2012 16:49, Nick North <no...@gmail.com> wrote:
> Following up on my own email, this seems to be an issue with snappy on
> Windows Server 2008. When I changed the file_compression setting to
> deflate_6, the "large" databases went down from 7GB to 1GB after
> compaction. I'm not entirely sure if this counts as a bug so I won't raise
> an issue on it.
>
Something's not right here. Are you able to run a couple tests on both
instances?
https://www.dropbox.com/sh/jeifcxpbtpo78ak/wvnm1MHmP9/testing/snappy_tests.erl
Copy this file to your %COUCHDB%/bin/ folder. Then run werl.exe shell (i.e.
not couch).
Erlang R15B (erts-5.9) [source] [smp:2:2] [async-threads:0]
Eshell V5.9 (abort with ^G)
1> application:load(snappy).
ok
2> application:start(snappy).
ok
3> c(snappy_tests).
{ok,snappy_tests}
4> snappy_tests:test().
All 2 tests passed.
ok
5>
Hopefully somewhere in there it spits out a sensible error message & we can
work from there.
Also if there's a way to find what version(s) of vcredist_x86 are installed
on the
server versions vs the other ones, that might help.
By the way: kudos to whoever wrote the code to deal with file_compression.
> When I changed file_compression to deflate_6, the system happily worked
> with the existing, supposedly snappy-compressed databases, and converted
> format on the next compaction. That could have gone wrong in several ways,
> but didn't, so thank you.
>
> That is Filipe's work.
A+
Dave
Re: Replicated database size
Posted by Nick North <no...@gmail.com>.
Following up on my own email, this seems to be an issue with snappy on
Windows Server 2008. When I changed the file_compression setting to
deflate_6, the "large" databases went down from 7GB to 1GB after
compaction. I'm not entirely sure if this counts as a bug so I won't raise
an issue on it.
By the way: kudos to whoever wrote the code to deal with file_compression.
When I changed file_compression to deflate_6, the system happily worked
with the existing, supposedly snappy-compressed databases, and converted
format on the next compaction. That could have gone wrong in several ways,
but didn't, so thank you.
Nick
On 15 May 2012 13:55, Nick North <no...@gmail.com> wrote:
> I'm curious about the size of replicated CouchDb databases in comparison
> to each other. I have four databases, each with pull replications from the
> other three, but they report quite different data sizes. Two of them say:
>
> {"db_name":"hydra","doc_count":1489060,"doc_del_count":2754893,"update_seq":6998882,"purge_seq":0,"compact_running":false,"disk_size":3213656193,"data_size":1395943755,"instance_start_time":"1337067567481841","disk_format_version":6,"committed_update_seq":6998882}
>
> While the other two say this - note the difference in data_size:
>
> {"db_name":"hydra","doc_count":1489441,"doc_del_count":2755302,"update_seq":4375865,"purge_seq":0,"compact_running":false,"disk_size":7599413027,"data_size":7265993199,"instance_start_time":"1337014746154865","disk_format_version":6,"committed_update_seq":4375865}
>
> (There is some discrepancy in the doc_count because new documents are being posted continuously, and some went in in between fetching stats for the various instances.) Other possibly relevant information:
>
>
> - All the replications appear to be in working order so I don't believe there is a backlog of documents waiting to be replicated.
> - The database has just one design view and whether or not it has been queried does not seem to make any difference to whether the database is "large" or "small".
> - Compaction makes little difference, in that the "large" instances always remain much larger than the "small" ones.
> - Everything is running CouchDb 1.2 on Windows: the "small" instances on Windows 7 and Windows Vista, and the "large" ones on Windows Server 2008.
> - File_compression is set to "snappy" in all cases and there are no attachments anywhere.
>
> Can anyone suggest what might be going on here? My best guess is that it's to do with file compression on Windows Server but that is a guess, so I'm intending to do some experimentation with the other file compression options. I'd be grateful for any thoughts, as I'm planning out disk requirements for a system with ten times the capacity of the current one, and would very much like to be do that with some certainty about file sizes. Thanks in advance for any help,
>
> Nick North
>
>
>