You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Paul Davis <pa...@gmail.com> on 2016/07/12 20:37:15 UTC
Re: 2.0 & Windows: status update
The logs posted at [1] show that we're seeing OS processes die with an
exit code of 4. The most likely place I can find that that comes from
is couchspawnkillable_win.c [2] which is nicely Windows specific so
would do a lot to explain why we don't see it on *nix systems.
Unfortunately other than pointing out that the subprocess creation
seems to fail I don't have any idea or suggestion on how to debug
further.
[1] https://gist.github.com/anonymous/f2a94234195f007c3049e27d942482c1
[2] https://github.com/apache/couchdb-couch/blob/master/priv/spawnkillable/couchspawnkillable_win.c#L106
On Wed, Jun 29, 2016 at 4:21 PM, Sebastian Rothbucher
<se...@googlemail.com> wrote:
> Hi all,
>
> just to follow up on that: there is another PR coming up (
> https://github.com/apache/couchdb/pull/427) that tests for some more fixes
> and brings even more stability. In the meantime, deleting dev/lib is indeed
> the best way to produce reliable results. So is switching between
> auth-tests-wip and master. But there's progress => it might all end up
> nicely on master.
>
> Good luck, thanks & best
> Sebastian
>
>
> On Mon, Jun 20, 2016 at 5:26 PM, Nick North <no...@gmail.com> wrote:
>
>> Thanks Sebastian. I'm looking at eunit at the moment, but hope to come back
>> to these.
>>
>> Nick
>>
>> On Sun, 19 Jun 2016 at 23:01 Sebastian Rothbucher <
>> sebastianrothbucher@googlemail.com> wrote:
>>
>> > Hi Joan, Nick,
>> >
>> > the following gist provides a current run of the test against the latest
>> > master of CouchDB - and the latest tests (from the auth-tests-wip
>> branch):
>> >
>> >
>> https://gist.github.com/sebastianrothbucher/efa3a992bd4de9996b4125da82a7e0de
>> > Maybe you can use them
>> >
>> > Here's what I did to get both latest tests and latest code:
>> > git checkout master
>> > ./configure -c --disable-docs --disable-fauxton
>> > make clean
>> > make
>> > git checkout auth-tests-wip
>> >
>> > Currently, make javascript seems not optimal as one tests (needs
>> > investigation) seems to mess up the setup for all that's following.
>> Hence,
>> > I took this drastic measure to produce the logs:
>> >
>> > for t in test/javascript/tests/*.js; do rm -rf dev/lib; dev/run -n 1 -q
>> > --with-admin-party-please test/javascript/run $t 2>&1 | tee -a
>> jstest2.log;
>> > done
>> >
>> > Maybe it makes sense for you to start w/ something similar to produce
>> some
>> > meaningful results.
>> >
>> > Best
>> > Sebastian
>> >
>> > On Sun, Jun 19, 2016 at 6:04 PM, Nick North <no...@gmail.com> wrote:
>> >
>> > > Sorry - I meant a single node cluster in that last message. And I meant
>> > to
>> > > sign my name correctly.
>> > >
>> > > Nick
>> > >
>> > > On Sun, 19 Jun 2016 at 16:56 Nick North <no...@gmail.com> wrote:
>> > >
>> > > > I'm trying these tests now, and find that there are still a lot of JS
>> > > > failures with a single cluster. Many of them look suspiciously
>> similar
>> > at
>> > > > an initial glance, but I hope to look in more detail tomorrow.
>> > > >
>> > > > Nicj
>> > > >
>> > > > On Sun, 19 Jun 2016 at 15:20 Jan Lehnardt <ja...@apache.org> wrote:
>> > > >
>> > > >>
>> > > >> > On 17 Jun 2016, at 22:48, Joan Touzet <wo...@apache.org> wrote:
>> > > >> >
>> > > >> > Hello everyone,
>> > > >> >
>> > > >> > I'd like to update the community on the status of the 2.0 port to
>> > > >> Microsoft Windows. There are three parts to this email: the build
>> > > >> tools/chain themselves, support in CouchDB for the Windows build
>> > > process,
>> > > >> and testing results. I'll cover them in that order.
>> > > >> >
>> > > >> > -Joan
>> > > >> >
>> > > >> > Build Tools/Chain
>> > > >> > =================
>> > > >> > ** TL;DR: New glazier repo to join couchdb, contains scripts and
>> > > README
>> > > >> to build CouchDB 2.0 on Windows.
>> > > >> >
>> > > >> > Our work to date has been going on in Dave Cottlehuber's glazier
>> > > >> repository at
>> > > >> >
>> > > >> > https://github.com/dch/glazier/tree/release/couchdb_2.0
>> > > >> >
>> > > >> > The reason for the extra repository is that the Windows build
>> > process
>> > > >> is *very* ugly, involving 3 distinct build chains (Visual Studio,
>> > Cygwin
>> > > >> and the Mozilla Build system) to build all of the necessary
>> > > prerequisites.
>> > > >> The repository includes a number of support scripts to set up that
>> > > >> environment, a README with a detailed walkthrough, and some patches
>> > > >> necessary to the prerequisites to get them to build under the modern
>> > > >> Windows b uild system.
>> > > >> >
>> > > >> > Parenthetically, it _is_ possible to use binary installs for the
>> > > >> prerequisites (OpenSSL, libcurl, Erlang, SM 1.8.5), but Dave, Nick
>> > North
>> > > >> and I have evolved the glazier system over a number of years and
>> it's
>> > > >> proven quite effective. Plus, we don't have to worry about the
>> > > provenance
>> > > >> of any of the binaries since we build everything from source
>> directly,
>> > > and
>> > > >> that's important when we put up an unofficial Windows build for
>> > > download at
>> > > >> https://couchdb.apache.org/ .
>> > > >> >
>> > > >> > Good news: as of today I've requested and Infra has created a new
>> > > >> apache couchdb-glazier repo, and it's my intent to mirror
>> dch/glazier
>> > > over
>> > > >> into the ASF's repo once things have stabilized a bit more (PR and
>> > > merge of
>> > > >> the release/couchdb_2.0 branch, and pending progress on steps 2 and
>> 3
>> > > >> below). Dave and I did an audit of the repository as it stands, and
>> > > since
>> > > >> all checkins come from CouchDB contributors already, we are good to
>> go
>> > > from
>> > > >> a licensing perspective.
>> > > >> >
>> > > >> >
>> > > >> > Overall CouchDB Windows support
>> > > >> > ===============================
>> > > >> > ** TL;DR: Windows support in 2.0 a priority, conversion of
>> top-level
>> > > >> Makefile in progress.
>> > > >> >
>> > > >> > There are two aspects to native CouchDB Windows support. The first
>> > is
>> > > >> anything within the CouchDB code itself that assumes a Unix-like
>> > > >> environment. Fortunately, most of these problems have been worked
>> out
>> > in
>> > > >> prior releases. I'm not aware of any outstanding issues here (except
>> > one
>> > > >> point below under test results).
>> > > >> >
>> > > >> > The other aspect is the build setup within the couchdb repo
>> itself.
>> > > >> I've already converted the bash configure script into a PowerShell
>> > > >> configure script that works fine. However, the Makefile has bashisms
>> > in
>> > > it
>> > > >> and assumes GNU Make. I've started a conversion of this into Windows
>> > > NMake
>> > > >> format and will submit a PR for a Makefile.win in due course.
>> > > >> >
>> > > >> > I want to answer two frequent questions we get here before they
>> get
>> > > >> re-asked:
>> > > >> >
>> > > >> > 1) Why not use a cygwin environment to retain compatibility with
>> > the
>> > > >> Unix build process? The answer is that performance suffers, the
>> build
>> > > chain
>> > > >> is onerous, there are link-time problems when trying to link against
>> > > things
>> > > >> built using Visual Studio, and there are still assumptions on paths
>> > that
>> > > >> don't work out. We can't get away from making Windows-specific
>> > > >> customizations to the build process anyway, so we might as well take
>> > the
>> > > >> extra step and support the build process properly. It's not THAT
>> much
>> > > work
>> > > >> to convert the Makefile and configure script, and our top-level
>> > Makefile
>> > > >> really isn't much more than a shell script anyway (every target is a
>> > > .PHONY
>> > > >> target!). In fact, a TODO for an enterprising developer might be to
>> > > rewrite
>> > > >> our top-level Makefile/Makefile.win as a Python script that "does
>> the
>> > > right
>> > > >> thing" on both platforms, the same way our dev/run script works
>> today.
>> > > >> >
>> > > >> > 2) Why not use the new "Bash and Ubuntu on Windows" functionality
>> > > >> Microsoft has announced for Windows 10? There are two distinct
>> > problems
>> > > >> here. The first is that there is a very large install base still of
>> > > Windows
>> > > >> 7 and 8 (and Windows Server) machines that cannot run this
>> subsystem.
>> > > The
>> > > >> second is that Microsoft themselves say this about the
>> functionality:
>> > > >> >
>> > > >> > "Second, while you’ll be able to run native Bash and many
>> Linux
>> > > >> command-line tools on Windows, it’s important to note that this is a
>> > > >> developer toolset to help you write and build all your code for all
>> > your
>> > > >> scenarios and platforms. This is not a server platform upon which
>> you
>> > > will
>> > > >> host websites, run server infrastructure, etc."
>> > > >> >
>> > > >> > Given this strong warning from Microsoft themselves (which hints
>> at
>> > > >> performance consideratings), and the fact that download statistics
>> > show
>> > > an
>> > > >> equal number of downloads of the CouchDB .tar source and the Windows
>> > > .zip
>> > > >> installer from our couchdb.apache.org website, we need to consider
>> > that
>> > > >> people are running CouchDB on Windows not just as a developer tool
>> but
>> > > as a
>> > > >> fully-fledged server. As such it behooves us to build it "properly"
>> > as a
>> > > >> normal Windows binary/service.
>> > > >> >
>> > > >>
>> > > >> Great progress Joan! Thank you! :)
>> > > >>
>> > > >> > Test Results
>> > > >> > ============
>> > > >> > ** TL;DR: Lots of things are failing. Joan needs help interpreting
>> > the
>> > > >> results or she will go around the bend.
>> > > >> >
>> > > >> > Here are the current test results in gist form.
>> > > >> >
>> > > >> > EUnit:
>> > > >> https://gist.github.com/anonymous/3203ed27c60cf3da4f0f0d5bff731722
>> > > >> >
>> > > >> > JS tests:
>> > > >> https://gist.github.com/anonymous/93b0b70ed445ca4043a63140f8d219bf
>> > > >> >
>> > > >> > For the EUnit tests, everything other than os_process stuff seems
>> to
>> > > be
>> > > >> working. Honestly, I think we can release without os_process support
>> > on
>> > > >> Windows, though I should file a bug to track this. I am actually
>> > > inclined
>> > > >> to disable os_process support on Windows and the related eunit tests
>> > > rather
>> > > >> than fix this rarely-needed functionality, unless someone on this
>> list
>> > > >> objects.
>> > > >>
>> > > >> You are probably thinking about CouchDB Externals, which definitely
>> > use
>> > > >> os_process functionality and which I’d also be fine with dropping
>> > > support
>> > > >> for Windows for now, but os_process is also used by the view server,
>> > so
>> > > we
>> > > >> should definitely get them passing.
>> > > >>
>> > > >>
>> > > >> > For the JS tests, a *lot* is failing. I need to know how much this
>> > > >> differs from a Linux/OSX baseline today, can anyone help me follow
>> up
>> > > here?
>> > > >>
>> > > >> Can you try running these against a -n 1 cluster? We are not set up
>> to
>> > > >> run JS tests against more nodes at this point.
>> > > >>
>> > > >> On master/unix most if not all JS tests should either pass or
>> skipped
>> > > >> with a TODO message.
>> > > >>
>> > > >>
>> > > >> Best
>> > > >> Jan
>> > > >> --
>> > > >>
>> > > >>
>> > > >>
>> > >
>> >
>>
Re: 2.0 & Windows: status update (all tests passing)
Posted by Garren Smith <ga...@apache.org>.
Awesome. Thanks Joan this is great. Another step closer to 2.0
On Wednesday, 03 August 2016, Joan Touzet <wo...@apache.org> wrote:
> One last update. All of the work on COUCHDB-3080 is now complete.
> We have a full Windows installer that logs to a logfile and runs
> CouchDB as a Windows service. Finally, the dch/glazier repo has
> been migrated to the ASF (couchdb-glazier).
>
> I will release a new installer when RC3 hits the streets. Until then,
> if you want the latest installer, you can build it yourself :)
>
> All the best,
> Joan
>
> ----- Original Message -----
> From: "Joan Touzet" <wohali@apache.org <javascript:;>>
> To: dev@couchdb.apache.org <javascript:;>
> Sent: Friday, July 15, 2016 6:58:39 PM
> Subject: Re: 2.0 & Windows: status update (all tests passing)
>
> Trimming the long thread for readability...
>
> Good news, everyone! As of the latest checkin to master, CouchDB 2.0
> builds correctly on Windows 64-bit! All eunit and javascript tests
> pass (or are skipped intentionally).
>
> This is a huge milestone! I especially want to thank Nick North and
> Dave Cottlehuber for helping me get this far.
>
> What's left is that rebar generate fails completely. Until this is
> working I can't generate a full release build, I can't finish the
> installer work, and I don't want to push dch/glazier into
> asf/couchdb-glazier until all of that is complete.
>
> Still, we're in the home stretch.
>
> -Joan
>
Re: 2.0 & Windows: status update (all tests passing)
Posted by Joan Touzet <wo...@apache.org>.
One last update. All of the work on COUCHDB-3080 is now complete.
We have a full Windows installer that logs to a logfile and runs
CouchDB as a Windows service. Finally, the dch/glazier repo has
been migrated to the ASF (couchdb-glazier).
I will release a new installer when RC3 hits the streets. Until then,
if you want the latest installer, you can build it yourself :)
All the best,
Joan
----- Original Message -----
From: "Joan Touzet" <wo...@apache.org>
To: dev@couchdb.apache.org
Sent: Friday, July 15, 2016 6:58:39 PM
Subject: Re: 2.0 & Windows: status update (all tests passing)
Trimming the long thread for readability...
Good news, everyone! As of the latest checkin to master, CouchDB 2.0
builds correctly on Windows 64-bit! All eunit and javascript tests
pass (or are skipped intentionally).
This is a huge milestone! I especially want to thank Nick North and
Dave Cottlehuber for helping me get this far.
What's left is that rebar generate fails completely. Until this is
working I can't generate a full release build, I can't finish the
installer work, and I don't want to push dch/glazier into
asf/couchdb-glazier until all of that is complete.
Still, we're in the home stretch.
-Joan
Re: 2.0 & Windows: status update (all tests passing)
Posted by Joan Touzet <wo...@apache.org>.
Trimming the long thread for readability...
Good news, everyone! As of the latest checkin to master, CouchDB 2.0
builds correctly on Windows 64-bit! All eunit and javascript tests
pass (or are skipped intentionally).
This is a huge milestone! I especially want to thank Nick North and
Dave Cottlehuber for helping me get this far.
What's left is that rebar generate fails completely. Until this is
working I can't generate a full release build, I can't finish the
installer work, and I don't want to push dch/glazier into
asf/couchdb-glazier until all of that is complete.
Still, we're in the home stretch.
-Joan
Re: 2.0 & Windows: status update
Posted by Nick North <no...@gmail.com>.
Ah - my mistake, sorry: I didn't merge it after all. The PR is outstanding
at https://github.com/apache/couchdb-jiffy/pull/3, with some discussion
between @kxepal and myself. There's a note to Paul there asking wiser heads
than mine to make a decision on whether to add it upstream.
Nick
On Wed, 13 Jul 2016 at 18:00 Nick North <no...@gmail.com> wrote:
> Maybe I did something stupid with the PR. If I can get to a PC this
> evening I'll take a look.
>
> Nick
> On Wed, 13 Jul 2016 at 17:41, Paul Davis <pa...@gmail.com>
> wrote:
>
>> That Jiffy error does seem like a conversion issue either from integer
>> widths or endianess. I don't remember seeing a patch for that on
>> Windows and looking back I'm not seeing anything from after 2013 that
>> looks related (in davisp/jiffy). Looking at the apache/jiffy repo we
>> use merges so I'd think it'd be unlikely to have been overwritten but
>> I guess its always possible.
>>
>> Paul
>>
>> On Wed, Jul 13, 2016 at 3:31 AM, Nick North <no...@gmail.com> wrote:
>> > I think I submitted a PR for that Jiffy problem a while back: it's some
>> > 32/64 bit issue. But I thought I'd also merged it. Unfortunately I'm
>> away
>> > from any useful tools for the next few days so can't check myself, but
>> take
>> > a look at the activity on the couchdb-jiffy repository. It's possible
>> that
>> > there has been a pull from upstream that overwrote my change.
>> >
>> > Nick
>> >
>> > On Wed, 13 Jul 2016 at 09:24 Joan Touzet <wo...@apache.org> wrote:
>> >
>> >> I took a look at the eunit failures and found that the entire
>> >> couchdb_os_daemon_test module fails due to issues with how it expects
>> >> to launch daemons.
>> >>
>> >> The main issue is that the primary test harness is a .escript file,
>> which
>> >> on *nix is magically parsed via the #!/usr/bin/escript header. On
>> Windows
>> >> we are just trying to directly launch the .escript file which fails
>> (since
>> >> Windows has no idea how to execute *.escript files). There are also .sh
>> >> scripts that are part of the test harness that will not run correctly.
>> >>
>> >> I've submitted a PR to simply bypass this entire test module for now.
>> It
>> >> sure would be swell to make it work but it'll be a fair bit of
>> fiddling,
>> >> especially in a way that makes eunit happy, to get it to work.
>> >>
>> >> Besides, there are other tests that drive the mrview os daemon; we will
>> >> see massive failures in the JS test suite if the entire os daemon
>> launching
>> >> process fails.
>> >>
>> >> https://github.com/apache/couchdb-couch/pull/184 hopefully will land
>> soon
>> >> allowing the main couch eunit tests to pass.
>> >>
>> >> More worryingly we have a failure in jiffy on Windows:
>> >>
>> >>
>> >>
>> https://gist.github.com/anonymous/c796d4b048efa90b17b1f43008c59783#file-gistfile1-txt-L381-L389
>> >>
>> >> Anyone who can help look at this one? (Paul?)
>> >>
>> >> -Joan
>> >>
>> >> ----- Original Message -----
>> >> From: "Joan Touzet" <wo...@apache.org>
>> >> To: dev@couchdb.apache.org
>> >> Sent: Tuesday, July 12, 2016 6:02:12 PM
>> >> Subject: Re: 2.0 & Windows: status update
>> >>
>> >> Thanks, Paul. I'm starting to look at this today.
>> >>
>> >> In better news, current Windows JS tests now match the *nix JS test
>> >> results.
>> >> Only one test, replication.js, is failing (ignoring the ignored tests).
>> >>
>> >> Results:
>> >> https://gist.github.com/anonymous/8e236848a89af440d3c56569e81f4829
>> >>
>> >> Mr. Newson is looking at this failure right now and says we may be able
>> >> to improve upon the testing methodology.
>> >>
>> >> -Joan
>> >>
>> >> ----- Original Message -----
>> >> From: "Paul Davis" <pa...@gmail.com>
>> >> To: dev@couchdb.apache.org
>> >> Sent: Tuesday, July 12, 2016 4:37:15 PM
>> >> Subject: Re: 2.0 & Windows: status update
>> >>
>> >> The logs posted at [1] show that we're seeing OS processes die with an
>> >> exit code of 4. The most likely place I can find that that comes from
>> >> is couchspawnkillable_win.c [2] which is nicely Windows specific so
>> >> would do a lot to explain why we don't see it on *nix systems.
>> >> Unfortunately other than pointing out that the subprocess creation
>> >> seems to fail I don't have any idea or suggestion on how to debug
>> >> further.
>> >>
>> >> [1] https://gist.github.com/anonymous/f2a94234195f007c3049e27d942482c1
>> >> [2]
>> >>
>> https://github.com/apache/couchdb-couch/blob/master/priv/spawnkillable/couchspawnkillable_win.c#L106
>> >>
>> >> On Wed, Jun 29, 2016 at 4:21 PM, Sebastian Rothbucher
>> >> <se...@googlemail.com> wrote:
>> >> > Hi all,
>> >> >
>> >> > just to follow up on that: there is another PR coming up (
>> >> > https://github.com/apache/couchdb/pull/427) that tests for some more
>> >> fixes
>> >> > and brings even more stability. In the meantime, deleting dev/lib is
>> >> indeed
>> >> > the best way to produce reliable results. So is switching between
>> >> > auth-tests-wip and master. But there's progress => it might all end
>> up
>> >> > nicely on master.
>> >> >
>> >> > Good luck, thanks & best
>> >> > Sebastian
>> >> >
>> >> >
>> >> > On Mon, Jun 20, 2016 at 5:26 PM, Nick North <no...@gmail.com>
>> wrote:
>> >> >
>> >> >> Thanks Sebastian. I'm looking at eunit at the moment, but hope to
>> come
>> >> back
>> >> >> to these.
>> >> >>
>> >> >> Nick
>> >> >>
>> >> >> On Sun, 19 Jun 2016 at 23:01 Sebastian Rothbucher <
>> >> >> sebastianrothbucher@googlemail.com> wrote:
>> >> >>
>> >> >> > Hi Joan, Nick,
>> >> >> >
>> >> >> > the following gist provides a current run of the test against the
>> >> latest
>> >> >> > master of CouchDB - and the latest tests (from the auth-tests-wip
>> >> >> branch):
>> >> >> >
>> >> >> >
>> >> >>
>> >>
>> https://gist.github.com/sebastianrothbucher/efa3a992bd4de9996b4125da82a7e0de
>> >> >> > Maybe you can use them
>> >> >> >
>> >> >> > Here's what I did to get both latest tests and latest code:
>> >> >> > git checkout master
>> >> >> > ./configure -c --disable-docs --disable-fauxton
>> >> >> > make clean
>> >> >> > make
>> >> >> > git checkout auth-tests-wip
>> >> >> >
>> >> >> > Currently, make javascript seems not optimal as one tests (needs
>> >> >> > investigation) seems to mess up the setup for all that's
>> following.
>> >> >> Hence,
>> >> >> > I took this drastic measure to produce the logs:
>> >> >> >
>> >> >> > for t in test/javascript/tests/*.js; do rm -rf dev/lib; dev/run
>> -n 1
>> >> -q
>> >> >> > --with-admin-party-please test/javascript/run $t 2>&1 | tee -a
>> >> >> jstest2.log;
>> >> >> > done
>> >> >> >
>> >> >> > Maybe it makes sense for you to start w/ something similar to
>> produce
>> >> >> some
>> >> >> > meaningful results.
>> >> >> >
>> >> >> > Best
>> >> >> > Sebastian
>> >> >> >
>> >> >> > On Sun, Jun 19, 2016 at 6:04 PM, Nick North <no...@gmail.com>
>> >> wrote:
>> >> >> >
>> >> >> > > Sorry - I meant a single node cluster in that last message. And
>> I
>> >> meant
>> >> >> > to
>> >> >> > > sign my name correctly.
>> >> >> > >
>> >> >> > > Nick
>> >> >> > >
>> >> >> > > On Sun, 19 Jun 2016 at 16:56 Nick North <no...@gmail.com>
>> wrote:
>> >> >> > >
>> >> >> > > > I'm trying these tests now, and find that there are still a
>> lot
>> >> of JS
>> >> >> > > > failures with a single cluster. Many of them look suspiciously
>> >> >> similar
>> >> >> > at
>> >> >> > > > an initial glance, but I hope to look in more detail tomorrow.
>> >> >> > > >
>> >> >> > > > Nicj
>> >> >> > > >
>> >> >> > > > On Sun, 19 Jun 2016 at 15:20 Jan Lehnardt <ja...@apache.org>
>> wrote:
>> >> >> > > >
>> >> >> > > >>
>> >> >> > > >> > On 17 Jun 2016, at 22:48, Joan Touzet <wo...@apache.org>
>> >> wrote:
>> >> >> > > >> >
>> >> >> > > >> > Hello everyone,
>> >> >> > > >> >
>> >> >> > > >> > I'd like to update the community on the status of the 2.0
>> port
>> >> to
>> >> >> > > >> Microsoft Windows. There are three parts to this email: the
>> build
>> >> >> > > >> tools/chain themselves, support in CouchDB for the Windows
>> build
>> >> >> > > process,
>> >> >> > > >> and testing results. I'll cover them in that order.
>> >> >> > > >> >
>> >> >> > > >> > -Joan
>> >> >> > > >> >
>> >> >> > > >> > Build Tools/Chain
>> >> >> > > >> > =================
>> >> >> > > >> > ** TL;DR: New glazier repo to join couchdb, contains
>> scripts
>> >> and
>> >> >> > > README
>> >> >> > > >> to build CouchDB 2.0 on Windows.
>> >> >> > > >> >
>> >> >> > > >> > Our work to date has been going on in Dave Cottlehuber's
>> >> glazier
>> >> >> > > >> repository at
>> >> >> > > >> >
>> >> >> > > >> > https://github.com/dch/glazier/tree/release/couchdb_2.0
>> >> >> > > >> >
>> >> >> > > >> > The reason for the extra repository is that the Windows
>> build
>> >> >> > process
>> >> >> > > >> is *very* ugly, involving 3 distinct build chains (Visual
>> Studio,
>> >> >> > Cygwin
>> >> >> > > >> and the Mozilla Build system) to build all of the necessary
>> >> >> > > prerequisites.
>> >> >> > > >> The repository includes a number of support scripts to set up
>> >> that
>> >> >> > > >> environment, a README with a detailed walkthrough, and some
>> >> patches
>> >> >> > > >> necessary to the prerequisites to get them to build under the
>> >> modern
>> >> >> > > >> Windows b uild system.
>> >> >> > > >> >
>> >> >> > > >> > Parenthetically, it _is_ possible to use binary installs
>> for
>> >> the
>> >> >> > > >> prerequisites (OpenSSL, libcurl, Erlang, SM 1.8.5), but Dave,
>> >> Nick
>> >> >> > North
>> >> >> > > >> and I have evolved the glazier system over a number of years
>> and
>> >> >> it's
>> >> >> > > >> proven quite effective. Plus, we don't have to worry about
>> the
>> >> >> > > provenance
>> >> >> > > >> of any of the binaries since we build everything from source
>> >> >> directly,
>> >> >> > > and
>> >> >> > > >> that's important when we put up an unofficial Windows build
>> for
>> >> >> > > download at
>> >> >> > > >> https://couchdb.apache.org/ .
>> >> >> > > >> >
>> >> >> > > >> > Good news: as of today I've requested and Infra has
>> created a
>> >> new
>> >> >> > > >> apache couchdb-glazier repo, and it's my intent to mirror
>> >> >> dch/glazier
>> >> >> > > over
>> >> >> > > >> into the ASF's repo once things have stabilized a bit more
>> (PR
>> >> and
>> >> >> > > merge of
>> >> >> > > >> the release/couchdb_2.0 branch, and pending progress on
>> steps 2
>> >> and
>> >> >> 3
>> >> >> > > >> below). Dave and I did an audit of the repository as it
>> stands,
>> >> and
>> >> >> > > since
>> >> >> > > >> all checkins come from CouchDB contributors already, we are
>> good
>> >> to
>> >> >> go
>> >> >> > > from
>> >> >> > > >> a licensing perspective.
>> >> >> > > >> >
>> >> >> > > >> >
>> >> >> > > >> > Overall CouchDB Windows support
>> >> >> > > >> > ===============================
>> >> >> > > >> > ** TL;DR: Windows support in 2.0 a priority, conversion of
>> >> >> top-level
>> >> >> > > >> Makefile in progress.
>> >> >> > > >> >
>> >> >> > > >> > There are two aspects to native CouchDB Windows support.
>> The
>> >> first
>> >> >> > is
>> >> >> > > >> anything within the CouchDB code itself that assumes a
>> Unix-like
>> >> >> > > >> environment. Fortunately, most of these problems have been
>> worked
>> >> >> out
>> >> >> > in
>> >> >> > > >> prior releases. I'm not aware of any outstanding issues here
>> >> (except
>> >> >> > one
>> >> >> > > >> point below under test results).
>> >> >> > > >> >
>> >> >> > > >> > The other aspect is the build setup within the couchdb repo
>> >> >> itself.
>> >> >> > > >> I've already converted the bash configure script into a
>> >> PowerShell
>> >> >> > > >> configure script that works fine. However, the Makefile has
>> >> bashisms
>> >> >> > in
>> >> >> > > it
>> >> >> > > >> and assumes GNU Make. I've started a conversion of this into
>> >> Windows
>> >> >> > > NMake
>> >> >> > > >> format and will submit a PR for a Makefile.win in due course.
>> >> >> > > >> >
>> >> >> > > >> > I want to answer two frequent questions we get here before
>> they
>> >> >> get
>> >> >> > > >> re-asked:
>> >> >> > > >> >
>> >> >> > > >> > 1) Why not use a cygwin environment to retain
>> compatibility
>> >> with
>> >> >> > the
>> >> >> > > >> Unix build process? The answer is that performance suffers,
>> the
>> >> >> build
>> >> >> > > chain
>> >> >> > > >> is onerous, there are link-time problems when trying to link
>> >> against
>> >> >> > > things
>> >> >> > > >> built using Visual Studio, and there are still assumptions on
>> >> paths
>> >> >> > that
>> >> >> > > >> don't work out. We can't get away from making
>> Windows-specific
>> >> >> > > >> customizations to the build process anyway, so we might as
>> well
>> >> take
>> >> >> > the
>> >> >> > > >> extra step and support the build process properly. It's not
>> THAT
>> >> >> much
>> >> >> > > work
>> >> >> > > >> to convert the Makefile and configure script, and our
>> top-level
>> >> >> > Makefile
>> >> >> > > >> really isn't much more than a shell script anyway (every
>> target
>> >> is a
>> >> >> > > .PHONY
>> >> >> > > >> target!). In fact, a TODO for an enterprising developer
>> might be
>> >> to
>> >> >> > > rewrite
>> >> >> > > >> our top-level Makefile/Makefile.win as a Python script that
>> "does
>> >> >> the
>> >> >> > > right
>> >> >> > > >> thing" on both platforms, the same way our dev/run script
>> works
>> >> >> today.
>> >> >> > > >> >
>> >> >> > > >> > 2) Why not use the new "Bash and Ubuntu on Windows"
>> >> functionality
>> >> >> > > >> Microsoft has announced for Windows 10? There are two
>> distinct
>> >> >> > problems
>> >> >> > > >> here. The first is that there is a very large install base
>> still
>> >> of
>> >> >> > > Windows
>> >> >> > > >> 7 and 8 (and Windows Server) machines that cannot run this
>> >> >> subsystem.
>> >> >> > > The
>> >> >> > > >> second is that Microsoft themselves say this about the
>> >> >> functionality:
>> >> >> > > >> >
>> >> >> > > >> > "Second, while you’ll be able to run native Bash and
>> many
>> >> >> Linux
>> >> >> > > >> command-line tools on Windows, it’s important to note that
>> this
>> >> is a
>> >> >> > > >> developer toolset to help you write and build all your code
>> for
>> >> all
>> >> >> > your
>> >> >> > > >> scenarios and platforms. This is not a server platform upon
>> which
>> >> >> you
>> >> >> > > will
>> >> >> > > >> host websites, run server infrastructure, etc."
>> >> >> > > >> >
>> >> >> > > >> > Given this strong warning from Microsoft themselves (which
>> >> hints
>> >> >> at
>> >> >> > > >> performance consideratings), and the fact that download
>> >> statistics
>> >> >> > show
>> >> >> > > an
>> >> >> > > >> equal number of downloads of the CouchDB .tar source and the
>> >> Windows
>> >> >> > > .zip
>> >> >> > > >> installer from our couchdb.apache.org website, we need to
>> >> consider
>> >> >> > that
>> >> >> > > >> people are running CouchDB on Windows not just as a developer
>> >> tool
>> >> >> but
>> >> >> > > as a
>> >> >> > > >> fully-fledged server. As such it behooves us to build it
>> >> "properly"
>> >> >> > as a
>> >> >> > > >> normal Windows binary/service.
>> >> >> > > >> >
>> >> >> > > >>
>> >> >> > > >> Great progress Joan! Thank you! :)
>> >> >> > > >>
>> >> >> > > >> > Test Results
>> >> >> > > >> > ============
>> >> >> > > >> > ** TL;DR: Lots of things are failing. Joan needs help
>> >> interpreting
>> >> >> > the
>> >> >> > > >> results or she will go around the bend.
>> >> >> > > >> >
>> >> >> > > >> > Here are the current test results in gist form.
>> >> >> > > >> >
>> >> >> > > >> > EUnit:
>> >> >> > > >>
>> >> https://gist.github.com/anonymous/3203ed27c60cf3da4f0f0d5bff731722
>> >> >> > > >> >
>> >> >> > > >> > JS tests:
>> >> >> > > >>
>> >> https://gist.github.com/anonymous/93b0b70ed445ca4043a63140f8d219bf
>> >> >> > > >> >
>> >> >> > > >> > For the EUnit tests, everything other than os_process stuff
>> >> seems
>> >> >> to
>> >> >> > > be
>> >> >> > > >> working. Honestly, I think we can release without os_process
>> >> support
>> >> >> > on
>> >> >> > > >> Windows, though I should file a bug to track this. I am
>> actually
>> >> >> > > inclined
>> >> >> > > >> to disable os_process support on Windows and the related
>> eunit
>> >> tests
>> >> >> > > rather
>> >> >> > > >> than fix this rarely-needed functionality, unless someone on
>> this
>> >> >> list
>> >> >> > > >> objects.
>> >> >> > > >>
>> >> >> > > >> You are probably thinking about CouchDB Externals, which
>> >> definitely
>> >> >> > use
>> >> >> > > >> os_process functionality and which I’d also be fine with
>> dropping
>> >> >> > > support
>> >> >> > > >> for Windows for now, but os_process is also used by the view
>> >> server,
>> >> >> > so
>> >> >> > > we
>> >> >> > > >> should definitely get them passing.
>> >> >> > > >>
>> >> >> > > >>
>> >> >> > > >> > For the JS tests, a *lot* is failing. I need to know how
>> much
>> >> this
>> >> >> > > >> differs from a Linux/OSX baseline today, can anyone help me
>> >> follow
>> >> >> up
>> >> >> > > here?
>> >> >> > > >>
>> >> >> > > >> Can you try running these against a -n 1 cluster? We are not
>> set
>> >> up
>> >> >> to
>> >> >> > > >> run JS tests against more nodes at this point.
>> >> >> > > >>
>> >> >> > > >> On master/unix most if not all JS tests should either pass or
>> >> >> skipped
>> >> >> > > >> with a TODO message.
>> >> >> > > >>
>> >> >> > > >>
>> >> >> > > >> Best
>> >> >> > > >> Jan
>> >> >> > > >> --
>> >> >> > > >>
>> >> >> > > >>
>> >> >> > > >>
>> >> >> > >
>> >> >> >
>> >> >>
>> >>
>>
>
Re: 2.0 & Windows: status update
Posted by Nick North <no...@gmail.com>.
Maybe I did something stupid with the PR. If I can get to a PC this evening
I'll take a look.
Nick
On Wed, 13 Jul 2016 at 17:41, Paul Davis <pa...@gmail.com>
wrote:
> That Jiffy error does seem like a conversion issue either from integer
> widths or endianess. I don't remember seeing a patch for that on
> Windows and looking back I'm not seeing anything from after 2013 that
> looks related (in davisp/jiffy). Looking at the apache/jiffy repo we
> use merges so I'd think it'd be unlikely to have been overwritten but
> I guess its always possible.
>
> Paul
>
> On Wed, Jul 13, 2016 at 3:31 AM, Nick North <no...@gmail.com> wrote:
> > I think I submitted a PR for that Jiffy problem a while back: it's some
> > 32/64 bit issue. But I thought I'd also merged it. Unfortunately I'm away
> > from any useful tools for the next few days so can't check myself, but
> take
> > a look at the activity on the couchdb-jiffy repository. It's possible
> that
> > there has been a pull from upstream that overwrote my change.
> >
> > Nick
> >
> > On Wed, 13 Jul 2016 at 09:24 Joan Touzet <wo...@apache.org> wrote:
> >
> >> I took a look at the eunit failures and found that the entire
> >> couchdb_os_daemon_test module fails due to issues with how it expects
> >> to launch daemons.
> >>
> >> The main issue is that the primary test harness is a .escript file,
> which
> >> on *nix is magically parsed via the #!/usr/bin/escript header. On
> Windows
> >> we are just trying to directly launch the .escript file which fails
> (since
> >> Windows has no idea how to execute *.escript files). There are also .sh
> >> scripts that are part of the test harness that will not run correctly.
> >>
> >> I've submitted a PR to simply bypass this entire test module for now. It
> >> sure would be swell to make it work but it'll be a fair bit of fiddling,
> >> especially in a way that makes eunit happy, to get it to work.
> >>
> >> Besides, there are other tests that drive the mrview os daemon; we will
> >> see massive failures in the JS test suite if the entire os daemon
> launching
> >> process fails.
> >>
> >> https://github.com/apache/couchdb-couch/pull/184 hopefully will land
> soon
> >> allowing the main couch eunit tests to pass.
> >>
> >> More worryingly we have a failure in jiffy on Windows:
> >>
> >>
> >>
> https://gist.github.com/anonymous/c796d4b048efa90b17b1f43008c59783#file-gistfile1-txt-L381-L389
> >>
> >> Anyone who can help look at this one? (Paul?)
> >>
> >> -Joan
> >>
> >> ----- Original Message -----
> >> From: "Joan Touzet" <wo...@apache.org>
> >> To: dev@couchdb.apache.org
> >> Sent: Tuesday, July 12, 2016 6:02:12 PM
> >> Subject: Re: 2.0 & Windows: status update
> >>
> >> Thanks, Paul. I'm starting to look at this today.
> >>
> >> In better news, current Windows JS tests now match the *nix JS test
> >> results.
> >> Only one test, replication.js, is failing (ignoring the ignored tests).
> >>
> >> Results:
> >> https://gist.github.com/anonymous/8e236848a89af440d3c56569e81f4829
> >>
> >> Mr. Newson is looking at this failure right now and says we may be able
> >> to improve upon the testing methodology.
> >>
> >> -Joan
> >>
> >> ----- Original Message -----
> >> From: "Paul Davis" <pa...@gmail.com>
> >> To: dev@couchdb.apache.org
> >> Sent: Tuesday, July 12, 2016 4:37:15 PM
> >> Subject: Re: 2.0 & Windows: status update
> >>
> >> The logs posted at [1] show that we're seeing OS processes die with an
> >> exit code of 4. The most likely place I can find that that comes from
> >> is couchspawnkillable_win.c [2] which is nicely Windows specific so
> >> would do a lot to explain why we don't see it on *nix systems.
> >> Unfortunately other than pointing out that the subprocess creation
> >> seems to fail I don't have any idea or suggestion on how to debug
> >> further.
> >>
> >> [1] https://gist.github.com/anonymous/f2a94234195f007c3049e27d942482c1
> >> [2]
> >>
> https://github.com/apache/couchdb-couch/blob/master/priv/spawnkillable/couchspawnkillable_win.c#L106
> >>
> >> On Wed, Jun 29, 2016 at 4:21 PM, Sebastian Rothbucher
> >> <se...@googlemail.com> wrote:
> >> > Hi all,
> >> >
> >> > just to follow up on that: there is another PR coming up (
> >> > https://github.com/apache/couchdb/pull/427) that tests for some more
> >> fixes
> >> > and brings even more stability. In the meantime, deleting dev/lib is
> >> indeed
> >> > the best way to produce reliable results. So is switching between
> >> > auth-tests-wip and master. But there's progress => it might all end up
> >> > nicely on master.
> >> >
> >> > Good luck, thanks & best
> >> > Sebastian
> >> >
> >> >
> >> > On Mon, Jun 20, 2016 at 5:26 PM, Nick North <no...@gmail.com>
> wrote:
> >> >
> >> >> Thanks Sebastian. I'm looking at eunit at the moment, but hope to
> come
> >> back
> >> >> to these.
> >> >>
> >> >> Nick
> >> >>
> >> >> On Sun, 19 Jun 2016 at 23:01 Sebastian Rothbucher <
> >> >> sebastianrothbucher@googlemail.com> wrote:
> >> >>
> >> >> > Hi Joan, Nick,
> >> >> >
> >> >> > the following gist provides a current run of the test against the
> >> latest
> >> >> > master of CouchDB - and the latest tests (from the auth-tests-wip
> >> >> branch):
> >> >> >
> >> >> >
> >> >>
> >>
> https://gist.github.com/sebastianrothbucher/efa3a992bd4de9996b4125da82a7e0de
> >> >> > Maybe you can use them
> >> >> >
> >> >> > Here's what I did to get both latest tests and latest code:
> >> >> > git checkout master
> >> >> > ./configure -c --disable-docs --disable-fauxton
> >> >> > make clean
> >> >> > make
> >> >> > git checkout auth-tests-wip
> >> >> >
> >> >> > Currently, make javascript seems not optimal as one tests (needs
> >> >> > investigation) seems to mess up the setup for all that's following.
> >> >> Hence,
> >> >> > I took this drastic measure to produce the logs:
> >> >> >
> >> >> > for t in test/javascript/tests/*.js; do rm -rf dev/lib; dev/run -n
> 1
> >> -q
> >> >> > --with-admin-party-please test/javascript/run $t 2>&1 | tee -a
> >> >> jstest2.log;
> >> >> > done
> >> >> >
> >> >> > Maybe it makes sense for you to start w/ something similar to
> produce
> >> >> some
> >> >> > meaningful results.
> >> >> >
> >> >> > Best
> >> >> > Sebastian
> >> >> >
> >> >> > On Sun, Jun 19, 2016 at 6:04 PM, Nick North <no...@gmail.com>
> >> wrote:
> >> >> >
> >> >> > > Sorry - I meant a single node cluster in that last message. And I
> >> meant
> >> >> > to
> >> >> > > sign my name correctly.
> >> >> > >
> >> >> > > Nick
> >> >> > >
> >> >> > > On Sun, 19 Jun 2016 at 16:56 Nick North <no...@gmail.com>
> wrote:
> >> >> > >
> >> >> > > > I'm trying these tests now, and find that there are still a lot
> >> of JS
> >> >> > > > failures with a single cluster. Many of them look suspiciously
> >> >> similar
> >> >> > at
> >> >> > > > an initial glance, but I hope to look in more detail tomorrow.
> >> >> > > >
> >> >> > > > Nicj
> >> >> > > >
> >> >> > > > On Sun, 19 Jun 2016 at 15:20 Jan Lehnardt <ja...@apache.org>
> wrote:
> >> >> > > >
> >> >> > > >>
> >> >> > > >> > On 17 Jun 2016, at 22:48, Joan Touzet <wo...@apache.org>
> >> wrote:
> >> >> > > >> >
> >> >> > > >> > Hello everyone,
> >> >> > > >> >
> >> >> > > >> > I'd like to update the community on the status of the 2.0
> port
> >> to
> >> >> > > >> Microsoft Windows. There are three parts to this email: the
> build
> >> >> > > >> tools/chain themselves, support in CouchDB for the Windows
> build
> >> >> > > process,
> >> >> > > >> and testing results. I'll cover them in that order.
> >> >> > > >> >
> >> >> > > >> > -Joan
> >> >> > > >> >
> >> >> > > >> > Build Tools/Chain
> >> >> > > >> > =================
> >> >> > > >> > ** TL;DR: New glazier repo to join couchdb, contains scripts
> >> and
> >> >> > > README
> >> >> > > >> to build CouchDB 2.0 on Windows.
> >> >> > > >> >
> >> >> > > >> > Our work to date has been going on in Dave Cottlehuber's
> >> glazier
> >> >> > > >> repository at
> >> >> > > >> >
> >> >> > > >> > https://github.com/dch/glazier/tree/release/couchdb_2.0
> >> >> > > >> >
> >> >> > > >> > The reason for the extra repository is that the Windows
> build
> >> >> > process
> >> >> > > >> is *very* ugly, involving 3 distinct build chains (Visual
> Studio,
> >> >> > Cygwin
> >> >> > > >> and the Mozilla Build system) to build all of the necessary
> >> >> > > prerequisites.
> >> >> > > >> The repository includes a number of support scripts to set up
> >> that
> >> >> > > >> environment, a README with a detailed walkthrough, and some
> >> patches
> >> >> > > >> necessary to the prerequisites to get them to build under the
> >> modern
> >> >> > > >> Windows b uild system.
> >> >> > > >> >
> >> >> > > >> > Parenthetically, it _is_ possible to use binary installs for
> >> the
> >> >> > > >> prerequisites (OpenSSL, libcurl, Erlang, SM 1.8.5), but Dave,
> >> Nick
> >> >> > North
> >> >> > > >> and I have evolved the glazier system over a number of years
> and
> >> >> it's
> >> >> > > >> proven quite effective. Plus, we don't have to worry about the
> >> >> > > provenance
> >> >> > > >> of any of the binaries since we build everything from source
> >> >> directly,
> >> >> > > and
> >> >> > > >> that's important when we put up an unofficial Windows build
> for
> >> >> > > download at
> >> >> > > >> https://couchdb.apache.org/ .
> >> >> > > >> >
> >> >> > > >> > Good news: as of today I've requested and Infra has created
> a
> >> new
> >> >> > > >> apache couchdb-glazier repo, and it's my intent to mirror
> >> >> dch/glazier
> >> >> > > over
> >> >> > > >> into the ASF's repo once things have stabilized a bit more (PR
> >> and
> >> >> > > merge of
> >> >> > > >> the release/couchdb_2.0 branch, and pending progress on steps
> 2
> >> and
> >> >> 3
> >> >> > > >> below). Dave and I did an audit of the repository as it
> stands,
> >> and
> >> >> > > since
> >> >> > > >> all checkins come from CouchDB contributors already, we are
> good
> >> to
> >> >> go
> >> >> > > from
> >> >> > > >> a licensing perspective.
> >> >> > > >> >
> >> >> > > >> >
> >> >> > > >> > Overall CouchDB Windows support
> >> >> > > >> > ===============================
> >> >> > > >> > ** TL;DR: Windows support in 2.0 a priority, conversion of
> >> >> top-level
> >> >> > > >> Makefile in progress.
> >> >> > > >> >
> >> >> > > >> > There are two aspects to native CouchDB Windows support. The
> >> first
> >> >> > is
> >> >> > > >> anything within the CouchDB code itself that assumes a
> Unix-like
> >> >> > > >> environment. Fortunately, most of these problems have been
> worked
> >> >> out
> >> >> > in
> >> >> > > >> prior releases. I'm not aware of any outstanding issues here
> >> (except
> >> >> > one
> >> >> > > >> point below under test results).
> >> >> > > >> >
> >> >> > > >> > The other aspect is the build setup within the couchdb repo
> >> >> itself.
> >> >> > > >> I've already converted the bash configure script into a
> >> PowerShell
> >> >> > > >> configure script that works fine. However, the Makefile has
> >> bashisms
> >> >> > in
> >> >> > > it
> >> >> > > >> and assumes GNU Make. I've started a conversion of this into
> >> Windows
> >> >> > > NMake
> >> >> > > >> format and will submit a PR for a Makefile.win in due course.
> >> >> > > >> >
> >> >> > > >> > I want to answer two frequent questions we get here before
> they
> >> >> get
> >> >> > > >> re-asked:
> >> >> > > >> >
> >> >> > > >> > 1) Why not use a cygwin environment to retain compatibility
> >> with
> >> >> > the
> >> >> > > >> Unix build process? The answer is that performance suffers,
> the
> >> >> build
> >> >> > > chain
> >> >> > > >> is onerous, there are link-time problems when trying to link
> >> against
> >> >> > > things
> >> >> > > >> built using Visual Studio, and there are still assumptions on
> >> paths
> >> >> > that
> >> >> > > >> don't work out. We can't get away from making Windows-specific
> >> >> > > >> customizations to the build process anyway, so we might as
> well
> >> take
> >> >> > the
> >> >> > > >> extra step and support the build process properly. It's not
> THAT
> >> >> much
> >> >> > > work
> >> >> > > >> to convert the Makefile and configure script, and our
> top-level
> >> >> > Makefile
> >> >> > > >> really isn't much more than a shell script anyway (every
> target
> >> is a
> >> >> > > .PHONY
> >> >> > > >> target!). In fact, a TODO for an enterprising developer might
> be
> >> to
> >> >> > > rewrite
> >> >> > > >> our top-level Makefile/Makefile.win as a Python script that
> "does
> >> >> the
> >> >> > > right
> >> >> > > >> thing" on both platforms, the same way our dev/run script
> works
> >> >> today.
> >> >> > > >> >
> >> >> > > >> > 2) Why not use the new "Bash and Ubuntu on Windows"
> >> functionality
> >> >> > > >> Microsoft has announced for Windows 10? There are two distinct
> >> >> > problems
> >> >> > > >> here. The first is that there is a very large install base
> still
> >> of
> >> >> > > Windows
> >> >> > > >> 7 and 8 (and Windows Server) machines that cannot run this
> >> >> subsystem.
> >> >> > > The
> >> >> > > >> second is that Microsoft themselves say this about the
> >> >> functionality:
> >> >> > > >> >
> >> >> > > >> > "Second, while you’ll be able to run native Bash and
> many
> >> >> Linux
> >> >> > > >> command-line tools on Windows, it’s important to note that
> this
> >> is a
> >> >> > > >> developer toolset to help you write and build all your code
> for
> >> all
> >> >> > your
> >> >> > > >> scenarios and platforms. This is not a server platform upon
> which
> >> >> you
> >> >> > > will
> >> >> > > >> host websites, run server infrastructure, etc."
> >> >> > > >> >
> >> >> > > >> > Given this strong warning from Microsoft themselves (which
> >> hints
> >> >> at
> >> >> > > >> performance consideratings), and the fact that download
> >> statistics
> >> >> > show
> >> >> > > an
> >> >> > > >> equal number of downloads of the CouchDB .tar source and the
> >> Windows
> >> >> > > .zip
> >> >> > > >> installer from our couchdb.apache.org website, we need to
> >> consider
> >> >> > that
> >> >> > > >> people are running CouchDB on Windows not just as a developer
> >> tool
> >> >> but
> >> >> > > as a
> >> >> > > >> fully-fledged server. As such it behooves us to build it
> >> "properly"
> >> >> > as a
> >> >> > > >> normal Windows binary/service.
> >> >> > > >> >
> >> >> > > >>
> >> >> > > >> Great progress Joan! Thank you! :)
> >> >> > > >>
> >> >> > > >> > Test Results
> >> >> > > >> > ============
> >> >> > > >> > ** TL;DR: Lots of things are failing. Joan needs help
> >> interpreting
> >> >> > the
> >> >> > > >> results or she will go around the bend.
> >> >> > > >> >
> >> >> > > >> > Here are the current test results in gist form.
> >> >> > > >> >
> >> >> > > >> > EUnit:
> >> >> > > >>
> >> https://gist.github.com/anonymous/3203ed27c60cf3da4f0f0d5bff731722
> >> >> > > >> >
> >> >> > > >> > JS tests:
> >> >> > > >>
> >> https://gist.github.com/anonymous/93b0b70ed445ca4043a63140f8d219bf
> >> >> > > >> >
> >> >> > > >> > For the EUnit tests, everything other than os_process stuff
> >> seems
> >> >> to
> >> >> > > be
> >> >> > > >> working. Honestly, I think we can release without os_process
> >> support
> >> >> > on
> >> >> > > >> Windows, though I should file a bug to track this. I am
> actually
> >> >> > > inclined
> >> >> > > >> to disable os_process support on Windows and the related eunit
> >> tests
> >> >> > > rather
> >> >> > > >> than fix this rarely-needed functionality, unless someone on
> this
> >> >> list
> >> >> > > >> objects.
> >> >> > > >>
> >> >> > > >> You are probably thinking about CouchDB Externals, which
> >> definitely
> >> >> > use
> >> >> > > >> os_process functionality and which I’d also be fine with
> dropping
> >> >> > > support
> >> >> > > >> for Windows for now, but os_process is also used by the view
> >> server,
> >> >> > so
> >> >> > > we
> >> >> > > >> should definitely get them passing.
> >> >> > > >>
> >> >> > > >>
> >> >> > > >> > For the JS tests, a *lot* is failing. I need to know how
> much
> >> this
> >> >> > > >> differs from a Linux/OSX baseline today, can anyone help me
> >> follow
> >> >> up
> >> >> > > here?
> >> >> > > >>
> >> >> > > >> Can you try running these against a -n 1 cluster? We are not
> set
> >> up
> >> >> to
> >> >> > > >> run JS tests against more nodes at this point.
> >> >> > > >>
> >> >> > > >> On master/unix most if not all JS tests should either pass or
> >> >> skipped
> >> >> > > >> with a TODO message.
> >> >> > > >>
> >> >> > > >>
> >> >> > > >> Best
> >> >> > > >> Jan
> >> >> > > >> --
> >> >> > > >>
> >> >> > > >>
> >> >> > > >>
> >> >> > >
> >> >> >
> >> >>
> >>
>
Re: 2.0 & Windows: status update
Posted by Paul Davis <pa...@gmail.com>.
That Jiffy error does seem like a conversion issue either from integer
widths or endianess. I don't remember seeing a patch for that on
Windows and looking back I'm not seeing anything from after 2013 that
looks related (in davisp/jiffy). Looking at the apache/jiffy repo we
use merges so I'd think it'd be unlikely to have been overwritten but
I guess its always possible.
Paul
On Wed, Jul 13, 2016 at 3:31 AM, Nick North <no...@gmail.com> wrote:
> I think I submitted a PR for that Jiffy problem a while back: it's some
> 32/64 bit issue. But I thought I'd also merged it. Unfortunately I'm away
> from any useful tools for the next few days so can't check myself, but take
> a look at the activity on the couchdb-jiffy repository. It's possible that
> there has been a pull from upstream that overwrote my change.
>
> Nick
>
> On Wed, 13 Jul 2016 at 09:24 Joan Touzet <wo...@apache.org> wrote:
>
>> I took a look at the eunit failures and found that the entire
>> couchdb_os_daemon_test module fails due to issues with how it expects
>> to launch daemons.
>>
>> The main issue is that the primary test harness is a .escript file, which
>> on *nix is magically parsed via the #!/usr/bin/escript header. On Windows
>> we are just trying to directly launch the .escript file which fails (since
>> Windows has no idea how to execute *.escript files). There are also .sh
>> scripts that are part of the test harness that will not run correctly.
>>
>> I've submitted a PR to simply bypass this entire test module for now. It
>> sure would be swell to make it work but it'll be a fair bit of fiddling,
>> especially in a way that makes eunit happy, to get it to work.
>>
>> Besides, there are other tests that drive the mrview os daemon; we will
>> see massive failures in the JS test suite if the entire os daemon launching
>> process fails.
>>
>> https://github.com/apache/couchdb-couch/pull/184 hopefully will land soon
>> allowing the main couch eunit tests to pass.
>>
>> More worryingly we have a failure in jiffy on Windows:
>>
>>
>> https://gist.github.com/anonymous/c796d4b048efa90b17b1f43008c59783#file-gistfile1-txt-L381-L389
>>
>> Anyone who can help look at this one? (Paul?)
>>
>> -Joan
>>
>> ----- Original Message -----
>> From: "Joan Touzet" <wo...@apache.org>
>> To: dev@couchdb.apache.org
>> Sent: Tuesday, July 12, 2016 6:02:12 PM
>> Subject: Re: 2.0 & Windows: status update
>>
>> Thanks, Paul. I'm starting to look at this today.
>>
>> In better news, current Windows JS tests now match the *nix JS test
>> results.
>> Only one test, replication.js, is failing (ignoring the ignored tests).
>>
>> Results:
>> https://gist.github.com/anonymous/8e236848a89af440d3c56569e81f4829
>>
>> Mr. Newson is looking at this failure right now and says we may be able
>> to improve upon the testing methodology.
>>
>> -Joan
>>
>> ----- Original Message -----
>> From: "Paul Davis" <pa...@gmail.com>
>> To: dev@couchdb.apache.org
>> Sent: Tuesday, July 12, 2016 4:37:15 PM
>> Subject: Re: 2.0 & Windows: status update
>>
>> The logs posted at [1] show that we're seeing OS processes die with an
>> exit code of 4. The most likely place I can find that that comes from
>> is couchspawnkillable_win.c [2] which is nicely Windows specific so
>> would do a lot to explain why we don't see it on *nix systems.
>> Unfortunately other than pointing out that the subprocess creation
>> seems to fail I don't have any idea or suggestion on how to debug
>> further.
>>
>> [1] https://gist.github.com/anonymous/f2a94234195f007c3049e27d942482c1
>> [2]
>> https://github.com/apache/couchdb-couch/blob/master/priv/spawnkillable/couchspawnkillable_win.c#L106
>>
>> On Wed, Jun 29, 2016 at 4:21 PM, Sebastian Rothbucher
>> <se...@googlemail.com> wrote:
>> > Hi all,
>> >
>> > just to follow up on that: there is another PR coming up (
>> > https://github.com/apache/couchdb/pull/427) that tests for some more
>> fixes
>> > and brings even more stability. In the meantime, deleting dev/lib is
>> indeed
>> > the best way to produce reliable results. So is switching between
>> > auth-tests-wip and master. But there's progress => it might all end up
>> > nicely on master.
>> >
>> > Good luck, thanks & best
>> > Sebastian
>> >
>> >
>> > On Mon, Jun 20, 2016 at 5:26 PM, Nick North <no...@gmail.com> wrote:
>> >
>> >> Thanks Sebastian. I'm looking at eunit at the moment, but hope to come
>> back
>> >> to these.
>> >>
>> >> Nick
>> >>
>> >> On Sun, 19 Jun 2016 at 23:01 Sebastian Rothbucher <
>> >> sebastianrothbucher@googlemail.com> wrote:
>> >>
>> >> > Hi Joan, Nick,
>> >> >
>> >> > the following gist provides a current run of the test against the
>> latest
>> >> > master of CouchDB - and the latest tests (from the auth-tests-wip
>> >> branch):
>> >> >
>> >> >
>> >>
>> https://gist.github.com/sebastianrothbucher/efa3a992bd4de9996b4125da82a7e0de
>> >> > Maybe you can use them
>> >> >
>> >> > Here's what I did to get both latest tests and latest code:
>> >> > git checkout master
>> >> > ./configure -c --disable-docs --disable-fauxton
>> >> > make clean
>> >> > make
>> >> > git checkout auth-tests-wip
>> >> >
>> >> > Currently, make javascript seems not optimal as one tests (needs
>> >> > investigation) seems to mess up the setup for all that's following.
>> >> Hence,
>> >> > I took this drastic measure to produce the logs:
>> >> >
>> >> > for t in test/javascript/tests/*.js; do rm -rf dev/lib; dev/run -n 1
>> -q
>> >> > --with-admin-party-please test/javascript/run $t 2>&1 | tee -a
>> >> jstest2.log;
>> >> > done
>> >> >
>> >> > Maybe it makes sense for you to start w/ something similar to produce
>> >> some
>> >> > meaningful results.
>> >> >
>> >> > Best
>> >> > Sebastian
>> >> >
>> >> > On Sun, Jun 19, 2016 at 6:04 PM, Nick North <no...@gmail.com>
>> wrote:
>> >> >
>> >> > > Sorry - I meant a single node cluster in that last message. And I
>> meant
>> >> > to
>> >> > > sign my name correctly.
>> >> > >
>> >> > > Nick
>> >> > >
>> >> > > On Sun, 19 Jun 2016 at 16:56 Nick North <no...@gmail.com> wrote:
>> >> > >
>> >> > > > I'm trying these tests now, and find that there are still a lot
>> of JS
>> >> > > > failures with a single cluster. Many of them look suspiciously
>> >> similar
>> >> > at
>> >> > > > an initial glance, but I hope to look in more detail tomorrow.
>> >> > > >
>> >> > > > Nicj
>> >> > > >
>> >> > > > On Sun, 19 Jun 2016 at 15:20 Jan Lehnardt <ja...@apache.org> wrote:
>> >> > > >
>> >> > > >>
>> >> > > >> > On 17 Jun 2016, at 22:48, Joan Touzet <wo...@apache.org>
>> wrote:
>> >> > > >> >
>> >> > > >> > Hello everyone,
>> >> > > >> >
>> >> > > >> > I'd like to update the community on the status of the 2.0 port
>> to
>> >> > > >> Microsoft Windows. There are three parts to this email: the build
>> >> > > >> tools/chain themselves, support in CouchDB for the Windows build
>> >> > > process,
>> >> > > >> and testing results. I'll cover them in that order.
>> >> > > >> >
>> >> > > >> > -Joan
>> >> > > >> >
>> >> > > >> > Build Tools/Chain
>> >> > > >> > =================
>> >> > > >> > ** TL;DR: New glazier repo to join couchdb, contains scripts
>> and
>> >> > > README
>> >> > > >> to build CouchDB 2.0 on Windows.
>> >> > > >> >
>> >> > > >> > Our work to date has been going on in Dave Cottlehuber's
>> glazier
>> >> > > >> repository at
>> >> > > >> >
>> >> > > >> > https://github.com/dch/glazier/tree/release/couchdb_2.0
>> >> > > >> >
>> >> > > >> > The reason for the extra repository is that the Windows build
>> >> > process
>> >> > > >> is *very* ugly, involving 3 distinct build chains (Visual Studio,
>> >> > Cygwin
>> >> > > >> and the Mozilla Build system) to build all of the necessary
>> >> > > prerequisites.
>> >> > > >> The repository includes a number of support scripts to set up
>> that
>> >> > > >> environment, a README with a detailed walkthrough, and some
>> patches
>> >> > > >> necessary to the prerequisites to get them to build under the
>> modern
>> >> > > >> Windows b uild system.
>> >> > > >> >
>> >> > > >> > Parenthetically, it _is_ possible to use binary installs for
>> the
>> >> > > >> prerequisites (OpenSSL, libcurl, Erlang, SM 1.8.5), but Dave,
>> Nick
>> >> > North
>> >> > > >> and I have evolved the glazier system over a number of years and
>> >> it's
>> >> > > >> proven quite effective. Plus, we don't have to worry about the
>> >> > > provenance
>> >> > > >> of any of the binaries since we build everything from source
>> >> directly,
>> >> > > and
>> >> > > >> that's important when we put up an unofficial Windows build for
>> >> > > download at
>> >> > > >> https://couchdb.apache.org/ .
>> >> > > >> >
>> >> > > >> > Good news: as of today I've requested and Infra has created a
>> new
>> >> > > >> apache couchdb-glazier repo, and it's my intent to mirror
>> >> dch/glazier
>> >> > > over
>> >> > > >> into the ASF's repo once things have stabilized a bit more (PR
>> and
>> >> > > merge of
>> >> > > >> the release/couchdb_2.0 branch, and pending progress on steps 2
>> and
>> >> 3
>> >> > > >> below). Dave and I did an audit of the repository as it stands,
>> and
>> >> > > since
>> >> > > >> all checkins come from CouchDB contributors already, we are good
>> to
>> >> go
>> >> > > from
>> >> > > >> a licensing perspective.
>> >> > > >> >
>> >> > > >> >
>> >> > > >> > Overall CouchDB Windows support
>> >> > > >> > ===============================
>> >> > > >> > ** TL;DR: Windows support in 2.0 a priority, conversion of
>> >> top-level
>> >> > > >> Makefile in progress.
>> >> > > >> >
>> >> > > >> > There are two aspects to native CouchDB Windows support. The
>> first
>> >> > is
>> >> > > >> anything within the CouchDB code itself that assumes a Unix-like
>> >> > > >> environment. Fortunately, most of these problems have been worked
>> >> out
>> >> > in
>> >> > > >> prior releases. I'm not aware of any outstanding issues here
>> (except
>> >> > one
>> >> > > >> point below under test results).
>> >> > > >> >
>> >> > > >> > The other aspect is the build setup within the couchdb repo
>> >> itself.
>> >> > > >> I've already converted the bash configure script into a
>> PowerShell
>> >> > > >> configure script that works fine. However, the Makefile has
>> bashisms
>> >> > in
>> >> > > it
>> >> > > >> and assumes GNU Make. I've started a conversion of this into
>> Windows
>> >> > > NMake
>> >> > > >> format and will submit a PR for a Makefile.win in due course.
>> >> > > >> >
>> >> > > >> > I want to answer two frequent questions we get here before they
>> >> get
>> >> > > >> re-asked:
>> >> > > >> >
>> >> > > >> > 1) Why not use a cygwin environment to retain compatibility
>> with
>> >> > the
>> >> > > >> Unix build process? The answer is that performance suffers, the
>> >> build
>> >> > > chain
>> >> > > >> is onerous, there are link-time problems when trying to link
>> against
>> >> > > things
>> >> > > >> built using Visual Studio, and there are still assumptions on
>> paths
>> >> > that
>> >> > > >> don't work out. We can't get away from making Windows-specific
>> >> > > >> customizations to the build process anyway, so we might as well
>> take
>> >> > the
>> >> > > >> extra step and support the build process properly. It's not THAT
>> >> much
>> >> > > work
>> >> > > >> to convert the Makefile and configure script, and our top-level
>> >> > Makefile
>> >> > > >> really isn't much more than a shell script anyway (every target
>> is a
>> >> > > .PHONY
>> >> > > >> target!). In fact, a TODO for an enterprising developer might be
>> to
>> >> > > rewrite
>> >> > > >> our top-level Makefile/Makefile.win as a Python script that "does
>> >> the
>> >> > > right
>> >> > > >> thing" on both platforms, the same way our dev/run script works
>> >> today.
>> >> > > >> >
>> >> > > >> > 2) Why not use the new "Bash and Ubuntu on Windows"
>> functionality
>> >> > > >> Microsoft has announced for Windows 10? There are two distinct
>> >> > problems
>> >> > > >> here. The first is that there is a very large install base still
>> of
>> >> > > Windows
>> >> > > >> 7 and 8 (and Windows Server) machines that cannot run this
>> >> subsystem.
>> >> > > The
>> >> > > >> second is that Microsoft themselves say this about the
>> >> functionality:
>> >> > > >> >
>> >> > > >> > "Second, while you’ll be able to run native Bash and many
>> >> Linux
>> >> > > >> command-line tools on Windows, it’s important to note that this
>> is a
>> >> > > >> developer toolset to help you write and build all your code for
>> all
>> >> > your
>> >> > > >> scenarios and platforms. This is not a server platform upon which
>> >> you
>> >> > > will
>> >> > > >> host websites, run server infrastructure, etc."
>> >> > > >> >
>> >> > > >> > Given this strong warning from Microsoft themselves (which
>> hints
>> >> at
>> >> > > >> performance consideratings), and the fact that download
>> statistics
>> >> > show
>> >> > > an
>> >> > > >> equal number of downloads of the CouchDB .tar source and the
>> Windows
>> >> > > .zip
>> >> > > >> installer from our couchdb.apache.org website, we need to
>> consider
>> >> > that
>> >> > > >> people are running CouchDB on Windows not just as a developer
>> tool
>> >> but
>> >> > > as a
>> >> > > >> fully-fledged server. As such it behooves us to build it
>> "properly"
>> >> > as a
>> >> > > >> normal Windows binary/service.
>> >> > > >> >
>> >> > > >>
>> >> > > >> Great progress Joan! Thank you! :)
>> >> > > >>
>> >> > > >> > Test Results
>> >> > > >> > ============
>> >> > > >> > ** TL;DR: Lots of things are failing. Joan needs help
>> interpreting
>> >> > the
>> >> > > >> results or she will go around the bend.
>> >> > > >> >
>> >> > > >> > Here are the current test results in gist form.
>> >> > > >> >
>> >> > > >> > EUnit:
>> >> > > >>
>> https://gist.github.com/anonymous/3203ed27c60cf3da4f0f0d5bff731722
>> >> > > >> >
>> >> > > >> > JS tests:
>> >> > > >>
>> https://gist.github.com/anonymous/93b0b70ed445ca4043a63140f8d219bf
>> >> > > >> >
>> >> > > >> > For the EUnit tests, everything other than os_process stuff
>> seems
>> >> to
>> >> > > be
>> >> > > >> working. Honestly, I think we can release without os_process
>> support
>> >> > on
>> >> > > >> Windows, though I should file a bug to track this. I am actually
>> >> > > inclined
>> >> > > >> to disable os_process support on Windows and the related eunit
>> tests
>> >> > > rather
>> >> > > >> than fix this rarely-needed functionality, unless someone on this
>> >> list
>> >> > > >> objects.
>> >> > > >>
>> >> > > >> You are probably thinking about CouchDB Externals, which
>> definitely
>> >> > use
>> >> > > >> os_process functionality and which I’d also be fine with dropping
>> >> > > support
>> >> > > >> for Windows for now, but os_process is also used by the view
>> server,
>> >> > so
>> >> > > we
>> >> > > >> should definitely get them passing.
>> >> > > >>
>> >> > > >>
>> >> > > >> > For the JS tests, a *lot* is failing. I need to know how much
>> this
>> >> > > >> differs from a Linux/OSX baseline today, can anyone help me
>> follow
>> >> up
>> >> > > here?
>> >> > > >>
>> >> > > >> Can you try running these against a -n 1 cluster? We are not set
>> up
>> >> to
>> >> > > >> run JS tests against more nodes at this point.
>> >> > > >>
>> >> > > >> On master/unix most if not all JS tests should either pass or
>> >> skipped
>> >> > > >> with a TODO message.
>> >> > > >>
>> >> > > >>
>> >> > > >> Best
>> >> > > >> Jan
>> >> > > >> --
>> >> > > >>
>> >> > > >>
>> >> > > >>
>> >> > >
>> >> >
>> >>
>>
Re: 2.0 & Windows: status update
Posted by Nick North <no...@gmail.com>.
I think I submitted a PR for that Jiffy problem a while back: it's some
32/64 bit issue. But I thought I'd also merged it. Unfortunately I'm away
from any useful tools for the next few days so can't check myself, but take
a look at the activity on the couchdb-jiffy repository. It's possible that
there has been a pull from upstream that overwrote my change.
Nick
On Wed, 13 Jul 2016 at 09:24 Joan Touzet <wo...@apache.org> wrote:
> I took a look at the eunit failures and found that the entire
> couchdb_os_daemon_test module fails due to issues with how it expects
> to launch daemons.
>
> The main issue is that the primary test harness is a .escript file, which
> on *nix is magically parsed via the #!/usr/bin/escript header. On Windows
> we are just trying to directly launch the .escript file which fails (since
> Windows has no idea how to execute *.escript files). There are also .sh
> scripts that are part of the test harness that will not run correctly.
>
> I've submitted a PR to simply bypass this entire test module for now. It
> sure would be swell to make it work but it'll be a fair bit of fiddling,
> especially in a way that makes eunit happy, to get it to work.
>
> Besides, there are other tests that drive the mrview os daemon; we will
> see massive failures in the JS test suite if the entire os daemon launching
> process fails.
>
> https://github.com/apache/couchdb-couch/pull/184 hopefully will land soon
> allowing the main couch eunit tests to pass.
>
> More worryingly we have a failure in jiffy on Windows:
>
>
> https://gist.github.com/anonymous/c796d4b048efa90b17b1f43008c59783#file-gistfile1-txt-L381-L389
>
> Anyone who can help look at this one? (Paul?)
>
> -Joan
>
> ----- Original Message -----
> From: "Joan Touzet" <wo...@apache.org>
> To: dev@couchdb.apache.org
> Sent: Tuesday, July 12, 2016 6:02:12 PM
> Subject: Re: 2.0 & Windows: status update
>
> Thanks, Paul. I'm starting to look at this today.
>
> In better news, current Windows JS tests now match the *nix JS test
> results.
> Only one test, replication.js, is failing (ignoring the ignored tests).
>
> Results:
> https://gist.github.com/anonymous/8e236848a89af440d3c56569e81f4829
>
> Mr. Newson is looking at this failure right now and says we may be able
> to improve upon the testing methodology.
>
> -Joan
>
> ----- Original Message -----
> From: "Paul Davis" <pa...@gmail.com>
> To: dev@couchdb.apache.org
> Sent: Tuesday, July 12, 2016 4:37:15 PM
> Subject: Re: 2.0 & Windows: status update
>
> The logs posted at [1] show that we're seeing OS processes die with an
> exit code of 4. The most likely place I can find that that comes from
> is couchspawnkillable_win.c [2] which is nicely Windows specific so
> would do a lot to explain why we don't see it on *nix systems.
> Unfortunately other than pointing out that the subprocess creation
> seems to fail I don't have any idea or suggestion on how to debug
> further.
>
> [1] https://gist.github.com/anonymous/f2a94234195f007c3049e27d942482c1
> [2]
> https://github.com/apache/couchdb-couch/blob/master/priv/spawnkillable/couchspawnkillable_win.c#L106
>
> On Wed, Jun 29, 2016 at 4:21 PM, Sebastian Rothbucher
> <se...@googlemail.com> wrote:
> > Hi all,
> >
> > just to follow up on that: there is another PR coming up (
> > https://github.com/apache/couchdb/pull/427) that tests for some more
> fixes
> > and brings even more stability. In the meantime, deleting dev/lib is
> indeed
> > the best way to produce reliable results. So is switching between
> > auth-tests-wip and master. But there's progress => it might all end up
> > nicely on master.
> >
> > Good luck, thanks & best
> > Sebastian
> >
> >
> > On Mon, Jun 20, 2016 at 5:26 PM, Nick North <no...@gmail.com> wrote:
> >
> >> Thanks Sebastian. I'm looking at eunit at the moment, but hope to come
> back
> >> to these.
> >>
> >> Nick
> >>
> >> On Sun, 19 Jun 2016 at 23:01 Sebastian Rothbucher <
> >> sebastianrothbucher@googlemail.com> wrote:
> >>
> >> > Hi Joan, Nick,
> >> >
> >> > the following gist provides a current run of the test against the
> latest
> >> > master of CouchDB - and the latest tests (from the auth-tests-wip
> >> branch):
> >> >
> >> >
> >>
> https://gist.github.com/sebastianrothbucher/efa3a992bd4de9996b4125da82a7e0de
> >> > Maybe you can use them
> >> >
> >> > Here's what I did to get both latest tests and latest code:
> >> > git checkout master
> >> > ./configure -c --disable-docs --disable-fauxton
> >> > make clean
> >> > make
> >> > git checkout auth-tests-wip
> >> >
> >> > Currently, make javascript seems not optimal as one tests (needs
> >> > investigation) seems to mess up the setup for all that's following.
> >> Hence,
> >> > I took this drastic measure to produce the logs:
> >> >
> >> > for t in test/javascript/tests/*.js; do rm -rf dev/lib; dev/run -n 1
> -q
> >> > --with-admin-party-please test/javascript/run $t 2>&1 | tee -a
> >> jstest2.log;
> >> > done
> >> >
> >> > Maybe it makes sense for you to start w/ something similar to produce
> >> some
> >> > meaningful results.
> >> >
> >> > Best
> >> > Sebastian
> >> >
> >> > On Sun, Jun 19, 2016 at 6:04 PM, Nick North <no...@gmail.com>
> wrote:
> >> >
> >> > > Sorry - I meant a single node cluster in that last message. And I
> meant
> >> > to
> >> > > sign my name correctly.
> >> > >
> >> > > Nick
> >> > >
> >> > > On Sun, 19 Jun 2016 at 16:56 Nick North <no...@gmail.com> wrote:
> >> > >
> >> > > > I'm trying these tests now, and find that there are still a lot
> of JS
> >> > > > failures with a single cluster. Many of them look suspiciously
> >> similar
> >> > at
> >> > > > an initial glance, but I hope to look in more detail tomorrow.
> >> > > >
> >> > > > Nicj
> >> > > >
> >> > > > On Sun, 19 Jun 2016 at 15:20 Jan Lehnardt <ja...@apache.org> wrote:
> >> > > >
> >> > > >>
> >> > > >> > On 17 Jun 2016, at 22:48, Joan Touzet <wo...@apache.org>
> wrote:
> >> > > >> >
> >> > > >> > Hello everyone,
> >> > > >> >
> >> > > >> > I'd like to update the community on the status of the 2.0 port
> to
> >> > > >> Microsoft Windows. There are three parts to this email: the build
> >> > > >> tools/chain themselves, support in CouchDB for the Windows build
> >> > > process,
> >> > > >> and testing results. I'll cover them in that order.
> >> > > >> >
> >> > > >> > -Joan
> >> > > >> >
> >> > > >> > Build Tools/Chain
> >> > > >> > =================
> >> > > >> > ** TL;DR: New glazier repo to join couchdb, contains scripts
> and
> >> > > README
> >> > > >> to build CouchDB 2.0 on Windows.
> >> > > >> >
> >> > > >> > Our work to date has been going on in Dave Cottlehuber's
> glazier
> >> > > >> repository at
> >> > > >> >
> >> > > >> > https://github.com/dch/glazier/tree/release/couchdb_2.0
> >> > > >> >
> >> > > >> > The reason for the extra repository is that the Windows build
> >> > process
> >> > > >> is *very* ugly, involving 3 distinct build chains (Visual Studio,
> >> > Cygwin
> >> > > >> and the Mozilla Build system) to build all of the necessary
> >> > > prerequisites.
> >> > > >> The repository includes a number of support scripts to set up
> that
> >> > > >> environment, a README with a detailed walkthrough, and some
> patches
> >> > > >> necessary to the prerequisites to get them to build under the
> modern
> >> > > >> Windows b uild system.
> >> > > >> >
> >> > > >> > Parenthetically, it _is_ possible to use binary installs for
> the
> >> > > >> prerequisites (OpenSSL, libcurl, Erlang, SM 1.8.5), but Dave,
> Nick
> >> > North
> >> > > >> and I have evolved the glazier system over a number of years and
> >> it's
> >> > > >> proven quite effective. Plus, we don't have to worry about the
> >> > > provenance
> >> > > >> of any of the binaries since we build everything from source
> >> directly,
> >> > > and
> >> > > >> that's important when we put up an unofficial Windows build for
> >> > > download at
> >> > > >> https://couchdb.apache.org/ .
> >> > > >> >
> >> > > >> > Good news: as of today I've requested and Infra has created a
> new
> >> > > >> apache couchdb-glazier repo, and it's my intent to mirror
> >> dch/glazier
> >> > > over
> >> > > >> into the ASF's repo once things have stabilized a bit more (PR
> and
> >> > > merge of
> >> > > >> the release/couchdb_2.0 branch, and pending progress on steps 2
> and
> >> 3
> >> > > >> below). Dave and I did an audit of the repository as it stands,
> and
> >> > > since
> >> > > >> all checkins come from CouchDB contributors already, we are good
> to
> >> go
> >> > > from
> >> > > >> a licensing perspective.
> >> > > >> >
> >> > > >> >
> >> > > >> > Overall CouchDB Windows support
> >> > > >> > ===============================
> >> > > >> > ** TL;DR: Windows support in 2.0 a priority, conversion of
> >> top-level
> >> > > >> Makefile in progress.
> >> > > >> >
> >> > > >> > There are two aspects to native CouchDB Windows support. The
> first
> >> > is
> >> > > >> anything within the CouchDB code itself that assumes a Unix-like
> >> > > >> environment. Fortunately, most of these problems have been worked
> >> out
> >> > in
> >> > > >> prior releases. I'm not aware of any outstanding issues here
> (except
> >> > one
> >> > > >> point below under test results).
> >> > > >> >
> >> > > >> > The other aspect is the build setup within the couchdb repo
> >> itself.
> >> > > >> I've already converted the bash configure script into a
> PowerShell
> >> > > >> configure script that works fine. However, the Makefile has
> bashisms
> >> > in
> >> > > it
> >> > > >> and assumes GNU Make. I've started a conversion of this into
> Windows
> >> > > NMake
> >> > > >> format and will submit a PR for a Makefile.win in due course.
> >> > > >> >
> >> > > >> > I want to answer two frequent questions we get here before they
> >> get
> >> > > >> re-asked:
> >> > > >> >
> >> > > >> > 1) Why not use a cygwin environment to retain compatibility
> with
> >> > the
> >> > > >> Unix build process? The answer is that performance suffers, the
> >> build
> >> > > chain
> >> > > >> is onerous, there are link-time problems when trying to link
> against
> >> > > things
> >> > > >> built using Visual Studio, and there are still assumptions on
> paths
> >> > that
> >> > > >> don't work out. We can't get away from making Windows-specific
> >> > > >> customizations to the build process anyway, so we might as well
> take
> >> > the
> >> > > >> extra step and support the build process properly. It's not THAT
> >> much
> >> > > work
> >> > > >> to convert the Makefile and configure script, and our top-level
> >> > Makefile
> >> > > >> really isn't much more than a shell script anyway (every target
> is a
> >> > > .PHONY
> >> > > >> target!). In fact, a TODO for an enterprising developer might be
> to
> >> > > rewrite
> >> > > >> our top-level Makefile/Makefile.win as a Python script that "does
> >> the
> >> > > right
> >> > > >> thing" on both platforms, the same way our dev/run script works
> >> today.
> >> > > >> >
> >> > > >> > 2) Why not use the new "Bash and Ubuntu on Windows"
> functionality
> >> > > >> Microsoft has announced for Windows 10? There are two distinct
> >> > problems
> >> > > >> here. The first is that there is a very large install base still
> of
> >> > > Windows
> >> > > >> 7 and 8 (and Windows Server) machines that cannot run this
> >> subsystem.
> >> > > The
> >> > > >> second is that Microsoft themselves say this about the
> >> functionality:
> >> > > >> >
> >> > > >> > "Second, while you’ll be able to run native Bash and many
> >> Linux
> >> > > >> command-line tools on Windows, it’s important to note that this
> is a
> >> > > >> developer toolset to help you write and build all your code for
> all
> >> > your
> >> > > >> scenarios and platforms. This is not a server platform upon which
> >> you
> >> > > will
> >> > > >> host websites, run server infrastructure, etc."
> >> > > >> >
> >> > > >> > Given this strong warning from Microsoft themselves (which
> hints
> >> at
> >> > > >> performance consideratings), and the fact that download
> statistics
> >> > show
> >> > > an
> >> > > >> equal number of downloads of the CouchDB .tar source and the
> Windows
> >> > > .zip
> >> > > >> installer from our couchdb.apache.org website, we need to
> consider
> >> > that
> >> > > >> people are running CouchDB on Windows not just as a developer
> tool
> >> but
> >> > > as a
> >> > > >> fully-fledged server. As such it behooves us to build it
> "properly"
> >> > as a
> >> > > >> normal Windows binary/service.
> >> > > >> >
> >> > > >>
> >> > > >> Great progress Joan! Thank you! :)
> >> > > >>
> >> > > >> > Test Results
> >> > > >> > ============
> >> > > >> > ** TL;DR: Lots of things are failing. Joan needs help
> interpreting
> >> > the
> >> > > >> results or she will go around the bend.
> >> > > >> >
> >> > > >> > Here are the current test results in gist form.
> >> > > >> >
> >> > > >> > EUnit:
> >> > > >>
> https://gist.github.com/anonymous/3203ed27c60cf3da4f0f0d5bff731722
> >> > > >> >
> >> > > >> > JS tests:
> >> > > >>
> https://gist.github.com/anonymous/93b0b70ed445ca4043a63140f8d219bf
> >> > > >> >
> >> > > >> > For the EUnit tests, everything other than os_process stuff
> seems
> >> to
> >> > > be
> >> > > >> working. Honestly, I think we can release without os_process
> support
> >> > on
> >> > > >> Windows, though I should file a bug to track this. I am actually
> >> > > inclined
> >> > > >> to disable os_process support on Windows and the related eunit
> tests
> >> > > rather
> >> > > >> than fix this rarely-needed functionality, unless someone on this
> >> list
> >> > > >> objects.
> >> > > >>
> >> > > >> You are probably thinking about CouchDB Externals, which
> definitely
> >> > use
> >> > > >> os_process functionality and which I’d also be fine with dropping
> >> > > support
> >> > > >> for Windows for now, but os_process is also used by the view
> server,
> >> > so
> >> > > we
> >> > > >> should definitely get them passing.
> >> > > >>
> >> > > >>
> >> > > >> > For the JS tests, a *lot* is failing. I need to know how much
> this
> >> > > >> differs from a Linux/OSX baseline today, can anyone help me
> follow
> >> up
> >> > > here?
> >> > > >>
> >> > > >> Can you try running these against a -n 1 cluster? We are not set
> up
> >> to
> >> > > >> run JS tests against more nodes at this point.
> >> > > >>
> >> > > >> On master/unix most if not all JS tests should either pass or
> >> skipped
> >> > > >> with a TODO message.
> >> > > >>
> >> > > >>
> >> > > >> Best
> >> > > >> Jan
> >> > > >> --
> >> > > >>
> >> > > >>
> >> > > >>
> >> > >
> >> >
> >>
>
Re: 2.0 & Windows: status update
Posted by Joan Touzet <wo...@apache.org>.
I took a look at the eunit failures and found that the entire
couchdb_os_daemon_test module fails due to issues with how it expects
to launch daemons.
The main issue is that the primary test harness is a .escript file, which
on *nix is magically parsed via the #!/usr/bin/escript header. On Windows
we are just trying to directly launch the .escript file which fails (since
Windows has no idea how to execute *.escript files). There are also .sh
scripts that are part of the test harness that will not run correctly.
I've submitted a PR to simply bypass this entire test module for now. It
sure would be swell to make it work but it'll be a fair bit of fiddling,
especially in a way that makes eunit happy, to get it to work.
Besides, there are other tests that drive the mrview os daemon; we will
see massive failures in the JS test suite if the entire os daemon launching
process fails.
https://github.com/apache/couchdb-couch/pull/184 hopefully will land soon
allowing the main couch eunit tests to pass.
More worryingly we have a failure in jiffy on Windows:
https://gist.github.com/anonymous/c796d4b048efa90b17b1f43008c59783#file-gistfile1-txt-L381-L389
Anyone who can help look at this one? (Paul?)
-Joan
----- Original Message -----
From: "Joan Touzet" <wo...@apache.org>
To: dev@couchdb.apache.org
Sent: Tuesday, July 12, 2016 6:02:12 PM
Subject: Re: 2.0 & Windows: status update
Thanks, Paul. I'm starting to look at this today.
In better news, current Windows JS tests now match the *nix JS test results.
Only one test, replication.js, is failing (ignoring the ignored tests).
Results: https://gist.github.com/anonymous/8e236848a89af440d3c56569e81f4829
Mr. Newson is looking at this failure right now and says we may be able
to improve upon the testing methodology.
-Joan
----- Original Message -----
From: "Paul Davis" <pa...@gmail.com>
To: dev@couchdb.apache.org
Sent: Tuesday, July 12, 2016 4:37:15 PM
Subject: Re: 2.0 & Windows: status update
The logs posted at [1] show that we're seeing OS processes die with an
exit code of 4. The most likely place I can find that that comes from
is couchspawnkillable_win.c [2] which is nicely Windows specific so
would do a lot to explain why we don't see it on *nix systems.
Unfortunately other than pointing out that the subprocess creation
seems to fail I don't have any idea or suggestion on how to debug
further.
[1] https://gist.github.com/anonymous/f2a94234195f007c3049e27d942482c1
[2] https://github.com/apache/couchdb-couch/blob/master/priv/spawnkillable/couchspawnkillable_win.c#L106
On Wed, Jun 29, 2016 at 4:21 PM, Sebastian Rothbucher
<se...@googlemail.com> wrote:
> Hi all,
>
> just to follow up on that: there is another PR coming up (
> https://github.com/apache/couchdb/pull/427) that tests for some more fixes
> and brings even more stability. In the meantime, deleting dev/lib is indeed
> the best way to produce reliable results. So is switching between
> auth-tests-wip and master. But there's progress => it might all end up
> nicely on master.
>
> Good luck, thanks & best
> Sebastian
>
>
> On Mon, Jun 20, 2016 at 5:26 PM, Nick North <no...@gmail.com> wrote:
>
>> Thanks Sebastian. I'm looking at eunit at the moment, but hope to come back
>> to these.
>>
>> Nick
>>
>> On Sun, 19 Jun 2016 at 23:01 Sebastian Rothbucher <
>> sebastianrothbucher@googlemail.com> wrote:
>>
>> > Hi Joan, Nick,
>> >
>> > the following gist provides a current run of the test against the latest
>> > master of CouchDB - and the latest tests (from the auth-tests-wip
>> branch):
>> >
>> >
>> https://gist.github.com/sebastianrothbucher/efa3a992bd4de9996b4125da82a7e0de
>> > Maybe you can use them
>> >
>> > Here's what I did to get both latest tests and latest code:
>> > git checkout master
>> > ./configure -c --disable-docs --disable-fauxton
>> > make clean
>> > make
>> > git checkout auth-tests-wip
>> >
>> > Currently, make javascript seems not optimal as one tests (needs
>> > investigation) seems to mess up the setup for all that's following.
>> Hence,
>> > I took this drastic measure to produce the logs:
>> >
>> > for t in test/javascript/tests/*.js; do rm -rf dev/lib; dev/run -n 1 -q
>> > --with-admin-party-please test/javascript/run $t 2>&1 | tee -a
>> jstest2.log;
>> > done
>> >
>> > Maybe it makes sense for you to start w/ something similar to produce
>> some
>> > meaningful results.
>> >
>> > Best
>> > Sebastian
>> >
>> > On Sun, Jun 19, 2016 at 6:04 PM, Nick North <no...@gmail.com> wrote:
>> >
>> > > Sorry - I meant a single node cluster in that last message. And I meant
>> > to
>> > > sign my name correctly.
>> > >
>> > > Nick
>> > >
>> > > On Sun, 19 Jun 2016 at 16:56 Nick North <no...@gmail.com> wrote:
>> > >
>> > > > I'm trying these tests now, and find that there are still a lot of JS
>> > > > failures with a single cluster. Many of them look suspiciously
>> similar
>> > at
>> > > > an initial glance, but I hope to look in more detail tomorrow.
>> > > >
>> > > > Nicj
>> > > >
>> > > > On Sun, 19 Jun 2016 at 15:20 Jan Lehnardt <ja...@apache.org> wrote:
>> > > >
>> > > >>
>> > > >> > On 17 Jun 2016, at 22:48, Joan Touzet <wo...@apache.org> wrote:
>> > > >> >
>> > > >> > Hello everyone,
>> > > >> >
>> > > >> > I'd like to update the community on the status of the 2.0 port to
>> > > >> Microsoft Windows. There are three parts to this email: the build
>> > > >> tools/chain themselves, support in CouchDB for the Windows build
>> > > process,
>> > > >> and testing results. I'll cover them in that order.
>> > > >> >
>> > > >> > -Joan
>> > > >> >
>> > > >> > Build Tools/Chain
>> > > >> > =================
>> > > >> > ** TL;DR: New glazier repo to join couchdb, contains scripts and
>> > > README
>> > > >> to build CouchDB 2.0 on Windows.
>> > > >> >
>> > > >> > Our work to date has been going on in Dave Cottlehuber's glazier
>> > > >> repository at
>> > > >> >
>> > > >> > https://github.com/dch/glazier/tree/release/couchdb_2.0
>> > > >> >
>> > > >> > The reason for the extra repository is that the Windows build
>> > process
>> > > >> is *very* ugly, involving 3 distinct build chains (Visual Studio,
>> > Cygwin
>> > > >> and the Mozilla Build system) to build all of the necessary
>> > > prerequisites.
>> > > >> The repository includes a number of support scripts to set up that
>> > > >> environment, a README with a detailed walkthrough, and some patches
>> > > >> necessary to the prerequisites to get them to build under the modern
>> > > >> Windows b uild system.
>> > > >> >
>> > > >> > Parenthetically, it _is_ possible to use binary installs for the
>> > > >> prerequisites (OpenSSL, libcurl, Erlang, SM 1.8.5), but Dave, Nick
>> > North
>> > > >> and I have evolved the glazier system over a number of years and
>> it's
>> > > >> proven quite effective. Plus, we don't have to worry about the
>> > > provenance
>> > > >> of any of the binaries since we build everything from source
>> directly,
>> > > and
>> > > >> that's important when we put up an unofficial Windows build for
>> > > download at
>> > > >> https://couchdb.apache.org/ .
>> > > >> >
>> > > >> > Good news: as of today I've requested and Infra has created a new
>> > > >> apache couchdb-glazier repo, and it's my intent to mirror
>> dch/glazier
>> > > over
>> > > >> into the ASF's repo once things have stabilized a bit more (PR and
>> > > merge of
>> > > >> the release/couchdb_2.0 branch, and pending progress on steps 2 and
>> 3
>> > > >> below). Dave and I did an audit of the repository as it stands, and
>> > > since
>> > > >> all checkins come from CouchDB contributors already, we are good to
>> go
>> > > from
>> > > >> a licensing perspective.
>> > > >> >
>> > > >> >
>> > > >> > Overall CouchDB Windows support
>> > > >> > ===============================
>> > > >> > ** TL;DR: Windows support in 2.0 a priority, conversion of
>> top-level
>> > > >> Makefile in progress.
>> > > >> >
>> > > >> > There are two aspects to native CouchDB Windows support. The first
>> > is
>> > > >> anything within the CouchDB code itself that assumes a Unix-like
>> > > >> environment. Fortunately, most of these problems have been worked
>> out
>> > in
>> > > >> prior releases. I'm not aware of any outstanding issues here (except
>> > one
>> > > >> point below under test results).
>> > > >> >
>> > > >> > The other aspect is the build setup within the couchdb repo
>> itself.
>> > > >> I've already converted the bash configure script into a PowerShell
>> > > >> configure script that works fine. However, the Makefile has bashisms
>> > in
>> > > it
>> > > >> and assumes GNU Make. I've started a conversion of this into Windows
>> > > NMake
>> > > >> format and will submit a PR for a Makefile.win in due course.
>> > > >> >
>> > > >> > I want to answer two frequent questions we get here before they
>> get
>> > > >> re-asked:
>> > > >> >
>> > > >> > 1) Why not use a cygwin environment to retain compatibility with
>> > the
>> > > >> Unix build process? The answer is that performance suffers, the
>> build
>> > > chain
>> > > >> is onerous, there are link-time problems when trying to link against
>> > > things
>> > > >> built using Visual Studio, and there are still assumptions on paths
>> > that
>> > > >> don't work out. We can't get away from making Windows-specific
>> > > >> customizations to the build process anyway, so we might as well take
>> > the
>> > > >> extra step and support the build process properly. It's not THAT
>> much
>> > > work
>> > > >> to convert the Makefile and configure script, and our top-level
>> > Makefile
>> > > >> really isn't much more than a shell script anyway (every target is a
>> > > .PHONY
>> > > >> target!). In fact, a TODO for an enterprising developer might be to
>> > > rewrite
>> > > >> our top-level Makefile/Makefile.win as a Python script that "does
>> the
>> > > right
>> > > >> thing" on both platforms, the same way our dev/run script works
>> today.
>> > > >> >
>> > > >> > 2) Why not use the new "Bash and Ubuntu on Windows" functionality
>> > > >> Microsoft has announced for Windows 10? There are two distinct
>> > problems
>> > > >> here. The first is that there is a very large install base still of
>> > > Windows
>> > > >> 7 and 8 (and Windows Server) machines that cannot run this
>> subsystem.
>> > > The
>> > > >> second is that Microsoft themselves say this about the
>> functionality:
>> > > >> >
>> > > >> > "Second, while you’ll be able to run native Bash and many
>> Linux
>> > > >> command-line tools on Windows, it’s important to note that this is a
>> > > >> developer toolset to help you write and build all your code for all
>> > your
>> > > >> scenarios and platforms. This is not a server platform upon which
>> you
>> > > will
>> > > >> host websites, run server infrastructure, etc."
>> > > >> >
>> > > >> > Given this strong warning from Microsoft themselves (which hints
>> at
>> > > >> performance consideratings), and the fact that download statistics
>> > show
>> > > an
>> > > >> equal number of downloads of the CouchDB .tar source and the Windows
>> > > .zip
>> > > >> installer from our couchdb.apache.org website, we need to consider
>> > that
>> > > >> people are running CouchDB on Windows not just as a developer tool
>> but
>> > > as a
>> > > >> fully-fledged server. As such it behooves us to build it "properly"
>> > as a
>> > > >> normal Windows binary/service.
>> > > >> >
>> > > >>
>> > > >> Great progress Joan! Thank you! :)
>> > > >>
>> > > >> > Test Results
>> > > >> > ============
>> > > >> > ** TL;DR: Lots of things are failing. Joan needs help interpreting
>> > the
>> > > >> results or she will go around the bend.
>> > > >> >
>> > > >> > Here are the current test results in gist form.
>> > > >> >
>> > > >> > EUnit:
>> > > >> https://gist.github.com/anonymous/3203ed27c60cf3da4f0f0d5bff731722
>> > > >> >
>> > > >> > JS tests:
>> > > >> https://gist.github.com/anonymous/93b0b70ed445ca4043a63140f8d219bf
>> > > >> >
>> > > >> > For the EUnit tests, everything other than os_process stuff seems
>> to
>> > > be
>> > > >> working. Honestly, I think we can release without os_process support
>> > on
>> > > >> Windows, though I should file a bug to track this. I am actually
>> > > inclined
>> > > >> to disable os_process support on Windows and the related eunit tests
>> > > rather
>> > > >> than fix this rarely-needed functionality, unless someone on this
>> list
>> > > >> objects.
>> > > >>
>> > > >> You are probably thinking about CouchDB Externals, which definitely
>> > use
>> > > >> os_process functionality and which I’d also be fine with dropping
>> > > support
>> > > >> for Windows for now, but os_process is also used by the view server,
>> > so
>> > > we
>> > > >> should definitely get them passing.
>> > > >>
>> > > >>
>> > > >> > For the JS tests, a *lot* is failing. I need to know how much this
>> > > >> differs from a Linux/OSX baseline today, can anyone help me follow
>> up
>> > > here?
>> > > >>
>> > > >> Can you try running these against a -n 1 cluster? We are not set up
>> to
>> > > >> run JS tests against more nodes at this point.
>> > > >>
>> > > >> On master/unix most if not all JS tests should either pass or
>> skipped
>> > > >> with a TODO message.
>> > > >>
>> > > >>
>> > > >> Best
>> > > >> Jan
>> > > >> --
>> > > >>
>> > > >>
>> > > >>
>> > >
>> >
>>
Re: 2.0 & Windows: status update
Posted by Joan Touzet <wo...@apache.org>.
Thanks, Paul. I'm starting to look at this today.
In better news, current Windows JS tests now match the *nix JS test results.
Only one test, replication.js, is failing (ignoring the ignored tests).
Results: https://gist.github.com/anonymous/8e236848a89af440d3c56569e81f4829
Mr. Newson is looking at this failure right now and says we may be able
to improve upon the testing methodology.
-Joan
----- Original Message -----
From: "Paul Davis" <pa...@gmail.com>
To: dev@couchdb.apache.org
Sent: Tuesday, July 12, 2016 4:37:15 PM
Subject: Re: 2.0 & Windows: status update
The logs posted at [1] show that we're seeing OS processes die with an
exit code of 4. The most likely place I can find that that comes from
is couchspawnkillable_win.c [2] which is nicely Windows specific so
would do a lot to explain why we don't see it on *nix systems.
Unfortunately other than pointing out that the subprocess creation
seems to fail I don't have any idea or suggestion on how to debug
further.
[1] https://gist.github.com/anonymous/f2a94234195f007c3049e27d942482c1
[2] https://github.com/apache/couchdb-couch/blob/master/priv/spawnkillable/couchspawnkillable_win.c#L106
On Wed, Jun 29, 2016 at 4:21 PM, Sebastian Rothbucher
<se...@googlemail.com> wrote:
> Hi all,
>
> just to follow up on that: there is another PR coming up (
> https://github.com/apache/couchdb/pull/427) that tests for some more fixes
> and brings even more stability. In the meantime, deleting dev/lib is indeed
> the best way to produce reliable results. So is switching between
> auth-tests-wip and master. But there's progress => it might all end up
> nicely on master.
>
> Good luck, thanks & best
> Sebastian
>
>
> On Mon, Jun 20, 2016 at 5:26 PM, Nick North <no...@gmail.com> wrote:
>
>> Thanks Sebastian. I'm looking at eunit at the moment, but hope to come back
>> to these.
>>
>> Nick
>>
>> On Sun, 19 Jun 2016 at 23:01 Sebastian Rothbucher <
>> sebastianrothbucher@googlemail.com> wrote:
>>
>> > Hi Joan, Nick,
>> >
>> > the following gist provides a current run of the test against the latest
>> > master of CouchDB - and the latest tests (from the auth-tests-wip
>> branch):
>> >
>> >
>> https://gist.github.com/sebastianrothbucher/efa3a992bd4de9996b4125da82a7e0de
>> > Maybe you can use them
>> >
>> > Here's what I did to get both latest tests and latest code:
>> > git checkout master
>> > ./configure -c --disable-docs --disable-fauxton
>> > make clean
>> > make
>> > git checkout auth-tests-wip
>> >
>> > Currently, make javascript seems not optimal as one tests (needs
>> > investigation) seems to mess up the setup for all that's following.
>> Hence,
>> > I took this drastic measure to produce the logs:
>> >
>> > for t in test/javascript/tests/*.js; do rm -rf dev/lib; dev/run -n 1 -q
>> > --with-admin-party-please test/javascript/run $t 2>&1 | tee -a
>> jstest2.log;
>> > done
>> >
>> > Maybe it makes sense for you to start w/ something similar to produce
>> some
>> > meaningful results.
>> >
>> > Best
>> > Sebastian
>> >
>> > On Sun, Jun 19, 2016 at 6:04 PM, Nick North <no...@gmail.com> wrote:
>> >
>> > > Sorry - I meant a single node cluster in that last message. And I meant
>> > to
>> > > sign my name correctly.
>> > >
>> > > Nick
>> > >
>> > > On Sun, 19 Jun 2016 at 16:56 Nick North <no...@gmail.com> wrote:
>> > >
>> > > > I'm trying these tests now, and find that there are still a lot of JS
>> > > > failures with a single cluster. Many of them look suspiciously
>> similar
>> > at
>> > > > an initial glance, but I hope to look in more detail tomorrow.
>> > > >
>> > > > Nicj
>> > > >
>> > > > On Sun, 19 Jun 2016 at 15:20 Jan Lehnardt <ja...@apache.org> wrote:
>> > > >
>> > > >>
>> > > >> > On 17 Jun 2016, at 22:48, Joan Touzet <wo...@apache.org> wrote:
>> > > >> >
>> > > >> > Hello everyone,
>> > > >> >
>> > > >> > I'd like to update the community on the status of the 2.0 port to
>> > > >> Microsoft Windows. There are three parts to this email: the build
>> > > >> tools/chain themselves, support in CouchDB for the Windows build
>> > > process,
>> > > >> and testing results. I'll cover them in that order.
>> > > >> >
>> > > >> > -Joan
>> > > >> >
>> > > >> > Build Tools/Chain
>> > > >> > =================
>> > > >> > ** TL;DR: New glazier repo to join couchdb, contains scripts and
>> > > README
>> > > >> to build CouchDB 2.0 on Windows.
>> > > >> >
>> > > >> > Our work to date has been going on in Dave Cottlehuber's glazier
>> > > >> repository at
>> > > >> >
>> > > >> > https://github.com/dch/glazier/tree/release/couchdb_2.0
>> > > >> >
>> > > >> > The reason for the extra repository is that the Windows build
>> > process
>> > > >> is *very* ugly, involving 3 distinct build chains (Visual Studio,
>> > Cygwin
>> > > >> and the Mozilla Build system) to build all of the necessary
>> > > prerequisites.
>> > > >> The repository includes a number of support scripts to set up that
>> > > >> environment, a README with a detailed walkthrough, and some patches
>> > > >> necessary to the prerequisites to get them to build under the modern
>> > > >> Windows b uild system.
>> > > >> >
>> > > >> > Parenthetically, it _is_ possible to use binary installs for the
>> > > >> prerequisites (OpenSSL, libcurl, Erlang, SM 1.8.5), but Dave, Nick
>> > North
>> > > >> and I have evolved the glazier system over a number of years and
>> it's
>> > > >> proven quite effective. Plus, we don't have to worry about the
>> > > provenance
>> > > >> of any of the binaries since we build everything from source
>> directly,
>> > > and
>> > > >> that's important when we put up an unofficial Windows build for
>> > > download at
>> > > >> https://couchdb.apache.org/ .
>> > > >> >
>> > > >> > Good news: as of today I've requested and Infra has created a new
>> > > >> apache couchdb-glazier repo, and it's my intent to mirror
>> dch/glazier
>> > > over
>> > > >> into the ASF's repo once things have stabilized a bit more (PR and
>> > > merge of
>> > > >> the release/couchdb_2.0 branch, and pending progress on steps 2 and
>> 3
>> > > >> below). Dave and I did an audit of the repository as it stands, and
>> > > since
>> > > >> all checkins come from CouchDB contributors already, we are good to
>> go
>> > > from
>> > > >> a licensing perspective.
>> > > >> >
>> > > >> >
>> > > >> > Overall CouchDB Windows support
>> > > >> > ===============================
>> > > >> > ** TL;DR: Windows support in 2.0 a priority, conversion of
>> top-level
>> > > >> Makefile in progress.
>> > > >> >
>> > > >> > There are two aspects to native CouchDB Windows support. The first
>> > is
>> > > >> anything within the CouchDB code itself that assumes a Unix-like
>> > > >> environment. Fortunately, most of these problems have been worked
>> out
>> > in
>> > > >> prior releases. I'm not aware of any outstanding issues here (except
>> > one
>> > > >> point below under test results).
>> > > >> >
>> > > >> > The other aspect is the build setup within the couchdb repo
>> itself.
>> > > >> I've already converted the bash configure script into a PowerShell
>> > > >> configure script that works fine. However, the Makefile has bashisms
>> > in
>> > > it
>> > > >> and assumes GNU Make. I've started a conversion of this into Windows
>> > > NMake
>> > > >> format and will submit a PR for a Makefile.win in due course.
>> > > >> >
>> > > >> > I want to answer two frequent questions we get here before they
>> get
>> > > >> re-asked:
>> > > >> >
>> > > >> > 1) Why not use a cygwin environment to retain compatibility with
>> > the
>> > > >> Unix build process? The answer is that performance suffers, the
>> build
>> > > chain
>> > > >> is onerous, there are link-time problems when trying to link against
>> > > things
>> > > >> built using Visual Studio, and there are still assumptions on paths
>> > that
>> > > >> don't work out. We can't get away from making Windows-specific
>> > > >> customizations to the build process anyway, so we might as well take
>> > the
>> > > >> extra step and support the build process properly. It's not THAT
>> much
>> > > work
>> > > >> to convert the Makefile and configure script, and our top-level
>> > Makefile
>> > > >> really isn't much more than a shell script anyway (every target is a
>> > > .PHONY
>> > > >> target!). In fact, a TODO for an enterprising developer might be to
>> > > rewrite
>> > > >> our top-level Makefile/Makefile.win as a Python script that "does
>> the
>> > > right
>> > > >> thing" on both platforms, the same way our dev/run script works
>> today.
>> > > >> >
>> > > >> > 2) Why not use the new "Bash and Ubuntu on Windows" functionality
>> > > >> Microsoft has announced for Windows 10? There are two distinct
>> > problems
>> > > >> here. The first is that there is a very large install base still of
>> > > Windows
>> > > >> 7 and 8 (and Windows Server) machines that cannot run this
>> subsystem.
>> > > The
>> > > >> second is that Microsoft themselves say this about the
>> functionality:
>> > > >> >
>> > > >> > "Second, while you’ll be able to run native Bash and many
>> Linux
>> > > >> command-line tools on Windows, it’s important to note that this is a
>> > > >> developer toolset to help you write and build all your code for all
>> > your
>> > > >> scenarios and platforms. This is not a server platform upon which
>> you
>> > > will
>> > > >> host websites, run server infrastructure, etc."
>> > > >> >
>> > > >> > Given this strong warning from Microsoft themselves (which hints
>> at
>> > > >> performance consideratings), and the fact that download statistics
>> > show
>> > > an
>> > > >> equal number of downloads of the CouchDB .tar source and the Windows
>> > > .zip
>> > > >> installer from our couchdb.apache.org website, we need to consider
>> > that
>> > > >> people are running CouchDB on Windows not just as a developer tool
>> but
>> > > as a
>> > > >> fully-fledged server. As such it behooves us to build it "properly"
>> > as a
>> > > >> normal Windows binary/service.
>> > > >> >
>> > > >>
>> > > >> Great progress Joan! Thank you! :)
>> > > >>
>> > > >> > Test Results
>> > > >> > ============
>> > > >> > ** TL;DR: Lots of things are failing. Joan needs help interpreting
>> > the
>> > > >> results or she will go around the bend.
>> > > >> >
>> > > >> > Here are the current test results in gist form.
>> > > >> >
>> > > >> > EUnit:
>> > > >> https://gist.github.com/anonymous/3203ed27c60cf3da4f0f0d5bff731722
>> > > >> >
>> > > >> > JS tests:
>> > > >> https://gist.github.com/anonymous/93b0b70ed445ca4043a63140f8d219bf
>> > > >> >
>> > > >> > For the EUnit tests, everything other than os_process stuff seems
>> to
>> > > be
>> > > >> working. Honestly, I think we can release without os_process support
>> > on
>> > > >> Windows, though I should file a bug to track this. I am actually
>> > > inclined
>> > > >> to disable os_process support on Windows and the related eunit tests
>> > > rather
>> > > >> than fix this rarely-needed functionality, unless someone on this
>> list
>> > > >> objects.
>> > > >>
>> > > >> You are probably thinking about CouchDB Externals, which definitely
>> > use
>> > > >> os_process functionality and which I’d also be fine with dropping
>> > > support
>> > > >> for Windows for now, but os_process is also used by the view server,
>> > so
>> > > we
>> > > >> should definitely get them passing.
>> > > >>
>> > > >>
>> > > >> > For the JS tests, a *lot* is failing. I need to know how much this
>> > > >> differs from a Linux/OSX baseline today, can anyone help me follow
>> up
>> > > here?
>> > > >>
>> > > >> Can you try running these against a -n 1 cluster? We are not set up
>> to
>> > > >> run JS tests against more nodes at this point.
>> > > >>
>> > > >> On master/unix most if not all JS tests should either pass or
>> skipped
>> > > >> with a TODO message.
>> > > >>
>> > > >>
>> > > >> Best
>> > > >> Jan
>> > > >> --
>> > > >>
>> > > >>
>> > > >>
>> > >
>> >
>>