2.0 & Windows: status update

[Joan Touzet <wo...@apache.org>]

Hello everyone,

I'd like to update the community on the status of the 2.0 port to Microsoft Windows. There are three parts to this email: the build tools/chain themselves, support in CouchDB for the Windows build process, and testing results. I'll cover them in that order.

-Joan

Build Tools/Chain
=================
** TL;DR: New glazier repo to join couchdb, contains scripts and README to build CouchDB 2.0 on Windows.

Our work to date has been going on in Dave Cottlehuber's glazier repository at 

  https://github.com/dch/glazier/tree/release/couchdb_2.0

The reason for the extra repository is that the Windows build process is *very* ugly, involving 3 distinct build chains (Visual Studio, Cygwin and the Mozilla Build system) to build all of the necessary prerequisites. The repository includes a number of support scripts to set up that environment, a README with a detailed walkthrough, and some patches necessary to the prerequisites to get them to build under the modern Windows b uild system.

Parenthetically, it _is_ possible to use binary installs for the prerequisites (OpenSSL, libcurl, Erlang, SM 1.8.5), but Dave, Nick North and I have evolved the glazier system over a number of years and it's proven quite effective. Plus, we don't have to worry about the provenance of any of the binaries since we build everything from source directly, and that's important when we put up an unofficial Windows build for download at https://couchdb.apache.org/ .

Good news: as of today I've requested and Infra has created a new apache couchdb-glazier repo, and it's my intent to mirror dch/glazier over into the ASF's repo once things have stabilized a bit more (PR and merge of the release/couchdb_2.0 branch, and pending progress on steps 2 and 3 below). Dave and I did an audit of the repository as it stands, and since all checkins come from CouchDB contributors already, we are good to go from a licensing perspective.


Overall CouchDB Windows support
===============================
** TL;DR: Windows support in 2.0 a priority, conversion of top-level Makefile in progress.

There are two aspects to native CouchDB Windows support. The first is anything within the CouchDB code itself that assumes a Unix-like environment. Fortunately, most of these problems have been worked out in prior releases. I'm not aware of any outstanding issues here (except one point below under test results).

The other aspect is the build setup within the couchdb repo itself. I've already converted the bash configure script into a PowerShell configure script that works fine. However, the Makefile has bashisms in it and assumes GNU Make. I've started a conversion of this into Windows NMake format and will submit a PR for a Makefile.win in due course.

I want to answer two frequent questions we get here before they get re-asked:

  1) Why not use a cygwin environment to retain compatibility with the Unix build process? The answer is that performance suffers, the build chain is onerous, there are link-time problems when trying to link against things built using Visual Studio, and there are still assumptions on paths that don't work out. We can't get away from making Windows-specific customizations to the build process anyway, so we might as well take the extra step and support the build process properly. It's not THAT much work to convert the Makefile and configure script, and our top-level Makefile really isn't much more than a shell script anyway (every target is a .PHONY target!). In fact, a TODO for an enterprising developer might be to rewrite our top-level Makefile/Makefile.win as a Python script that "does the right thing" on both platforms, the same way our dev/run script works today.

  2) Why not use the new "Bash and Ubuntu on Windows" functionality Microsoft has announced for Windows 10? There are two distinct problems here. The first is that there is a very large install base still of Windows 7 and 8 (and Windows Server) machines that cannot run this subsystem. The second is that Microsoft themselves say this about the functionality:

     "Second, while you’ll be able to run native Bash and many Linux command-line tools on Windows, it’s important to note that this is a developer toolset to help you write and build all your code for all your scenarios and platforms. This is not a server platform upon which you will host websites, run server infrastructure, etc."

Given this strong warning from Microsoft themselves (which hints at performance consideratings), and the fact that download statistics show an equal number of downloads of the CouchDB .tar source and the Windows .zip installer from our couchdb.apache.org website, we need to consider that people are running CouchDB on Windows not just as a developer tool but as a fully-fledged server. As such it behooves us to build it "properly" as a normal Windows binary/service.


Test Results
============
** TL;DR: Lots of things are failing. Joan needs help interpreting the results or she will go around the bend.

Here are the current test results in gist form.

EUnit: https://gist.github.com/anonymous/3203ed27c60cf3da4f0f0d5bff731722

JS tests: https://gist.github.com/anonymous/93b0b70ed445ca4043a63140f8d219bf

For the EUnit tests, everything other than os_process stuff seems to be working. Honestly, I think we can release without os_process support on Windows, though I should file a bug to track this. I am actually inclined to disable os_process support on Windows and the related eunit tests rather than fix this rarely-needed functionality, unless someone on this list objects.

For the JS tests, a *lot* is failing. I need to know how much this differs from a Linux/OSX baseline today, can anyone help me follow up here?

Re: 2.0 & Windows: status update

["Reddy B." <re...@live.fr>]

Regarding the quote below, I just wanted to mention that this is indeed 
how we use CouchDb here : as a production database on Windows Server.  
We both develop on Windows and host on Windows. The first-class support 
and stability of CouchDb on the platform largely contributed to our 
interest in the solution.


On 17/06/2016 22:48, Joan Touzet wrote:
> we need to consider that people are running CouchDB on Windows not just as a developer tool but as a fully-fledged server

Re: 2.0 & Windows: status update (all tests passing)

[Garren Smith <ga...@apache.org>]

Awesome. Thanks Joan this is great. Another step closer to 2.0

On Wednesday, 03 August 2016, Joan Touzet <wo...@apache.org> wrote:

> One last update. All of the work on COUCHDB-3080 is now complete.
> We have a full Windows installer that logs to a logfile and runs
> CouchDB as a Windows service. Finally, the dch/glazier repo has
> been migrated to the ASF (couchdb-glazier).
>
> I will release a new installer when RC3 hits the streets. Until then,
> if you want the latest installer, you can build it yourself :)
>
> All the best,
> Joan
>
> ----- Original Message -----
> From: "Joan Touzet" <wohali@apache.org <javascript:;>>
> To: dev@couchdb.apache.org <javascript:;>
> Sent: Friday, July 15, 2016 6:58:39 PM
> Subject: Re: 2.0 & Windows: status update (all tests passing)
>
> Trimming the long thread for readability...
>
> Good news, everyone! As of the latest checkin to master, CouchDB 2.0
> builds correctly on Windows 64-bit! All eunit and javascript tests
> pass (or are skipped intentionally).
>
> This is a huge milestone! I especially want to thank Nick North and
> Dave Cottlehuber for helping me get this far.
>
> What's left is that rebar generate fails completely. Until this is
> working I can't generate a full release build, I can't finish the
> installer work, and I don't want to push dch/glazier into
> asf/couchdb-glazier until all of that is complete.
>
> Still, we're in the home stretch.
>
> -Joan
>

Re: 2.0 & Windows: status update (all tests passing)

[Joan Touzet <wo...@apache.org>]

One last update. All of the work on COUCHDB-3080 is now complete.
We have a full Windows installer that logs to a logfile and runs
CouchDB as a Windows service. Finally, the dch/glazier repo has
been migrated to the ASF (couchdb-glazier).

I will release a new installer when RC3 hits the streets. Until then,
if you want the latest installer, you can build it yourself :)

All the best,
Joan

----- Original Message -----
From: "Joan Touzet" <wo...@apache.org>
To: dev@couchdb.apache.org
Sent: Friday, July 15, 2016 6:58:39 PM
Subject: Re: 2.0 & Windows: status update (all tests passing)

Trimming the long thread for readability...

Good news, everyone! As of the latest checkin to master, CouchDB 2.0
builds correctly on Windows 64-bit! All eunit and javascript tests
pass (or are skipped intentionally). 

This is a huge milestone! I especially want to thank Nick North and
Dave Cottlehuber for helping me get this far.

What's left is that rebar generate fails completely. Until this is
working I can't generate a full release build, I can't finish the
installer work, and I don't want to push dch/glazier into
asf/couchdb-glazier until all of that is complete.

Still, we're in the home stretch.

-Joan

Re: 2.0 & Windows: status update (all tests passing)

[Joan Touzet <wo...@apache.org>]

Trimming the long thread for readability...

Good news, everyone! As of the latest checkin to master, CouchDB 2.0
builds correctly on Windows 64-bit! All eunit and javascript tests
pass (or are skipped intentionally). 

This is a huge milestone! I especially want to thank Nick North and
Dave Cottlehuber for helping me get this far.

What's left is that rebar generate fails completely. Until this is
working I can't generate a full release build, I can't finish the
installer work, and I don't want to push dch/glazier into
asf/couchdb-glazier until all of that is complete.

Still, we're in the home stretch.

-Joan

Re: 2.0 & Windows: status update

[Nick North <no...@gmail.com>]

Ah - my mistake, sorry: I didn't merge it after all. The PR is outstanding
at https://github.com/apache/couchdb-jiffy/pull/3, with some discussion
between @kxepal and myself. There's a note to Paul there asking wiser heads
than mine to make a decision on whether to add it upstream.

Nick

On Wed, 13 Jul 2016 at 18:00 Nick North <no...@gmail.com> wrote:

> Maybe I did something stupid with the PR. If I can get to a PC this
> evening I'll take a look.
>
> Nick
> On Wed, 13 Jul 2016 at 17:41, Paul Davis <pa...@gmail.com>
> wrote:
>
>> That Jiffy error does seem like a conversion issue either from integer
>> widths or endianess. I don't remember seeing a patch for that on
>> Windows and looking back I'm not seeing anything from after 2013 that
>> looks related (in davisp/jiffy). Looking at the apache/jiffy repo we
>> use merges so I'd think it'd be unlikely to have been overwritten but
>> I guess its always possible.
>>
>> Paul
>>
>> On Wed, Jul 13, 2016 at 3:31 AM, Nick North <no...@gmail.com> wrote:
>> > I think I submitted a PR for that Jiffy problem a while back: it's some
>> > 32/64 bit issue. But I thought I'd also merged it. Unfortunately I'm
>> away
>> > from any useful tools for the next few days so can't check myself, but
>> take
>> > a look at the activity on the couchdb-jiffy repository. It's possible
>> that
>> > there has been a pull from upstream that overwrote my change.
>> >
>> > Nick
>> >
>> > On Wed, 13 Jul 2016 at 09:24 Joan Touzet <wo...@apache.org> wrote:
>> >
>> >> I took a look at the eunit failures and found that the entire
>> >> couchdb_os_daemon_test module fails due to issues with how it expects
>> >> to launch daemons.
>> >>
>> >> The main issue is that the primary test harness is a .escript file,
>> which
>> >> on *nix is magically parsed via the #!/usr/bin/escript header. On
>> Windows
>> >> we are just trying to directly launch the .escript file which fails
>> (since
>> >> Windows has no idea how to execute *.escript files). There are also .sh
>> >> scripts that are part of the test harness that will not run correctly.
>> >>
>> >> I've submitted a PR to simply bypass this entire test module for now.
>> It
>> >> sure would be swell to make it work but it'll be a fair bit of
>> fiddling,
>> >> especially in a way that makes eunit happy, to get it to work.
>> >>
>> >> Besides, there are other tests that drive the mrview os daemon; we will
>> >> see massive failures in the JS test suite if the entire os daemon
>> launching
>> >> process fails.
>> >>
>> >> https://github.com/apache/couchdb-couch/pull/184 hopefully will land
>> soon
>> >> allowing the main couch eunit tests to pass.
>> >>
>> >> More worryingly we have a failure in jiffy on Windows:
>> >>
>> >>
>> >>
>> https://gist.github.com/anonymous/c796d4b048efa90b17b1f43008c59783#file-gistfile1-txt-L381-L389
>> >>
>> >> Anyone who can help look at this one? (Paul?)
>> >>
>> >> -Joan
>> >>
>> >> ----- Original Message -----
>> >> From: "Joan Touzet" <wo...@apache.org>
>> >> To: dev@couchdb.apache.org
>> >> Sent: Tuesday, July 12, 2016 6:02:12 PM
>> >> Subject: Re: 2.0 & Windows: status update
>> >>
>> >> Thanks, Paul. I'm starting to look at this today.
>> >>
>> >> In better news, current Windows JS tests now match the *nix JS test
>> >> results.
>> >> Only one test, replication.js, is failing (ignoring the ignored tests).
>> >>
>> >> Results:
>> >> https://gist.github.com/anonymous/8e236848a89af440d3c56569e81f4829
>> >>
>> >> Mr. Newson is looking at this failure right now and says we may be able
>> >> to improve upon the testing methodology.
>> >>
>> >> -Joan
>> >>
>> >> ----- Original Message -----
>> >> From: "Paul Davis" <pa...@gmail.com>
>> >> To: dev@couchdb.apache.org
>> >> Sent: Tuesday, July 12, 2016 4:37:15 PM
>> >> Subject: Re: 2.0 & Windows: status update
>> >>
>> >> The logs posted at [1] show that we're seeing OS processes die with an
>> >> exit code of 4. The most likely place I can find that that comes from
>> >> is couchspawnkillable_win.c [2] which is nicely Windows specific so
>> >> would do a lot to explain why we don't see it on *nix systems.
>> >> Unfortunately other than pointing out that the subprocess creation
>> >> seems to fail I don't have any idea or suggestion on how to debug
>> >> further.
>> >>
>> >> [1] https://gist.github.com/anonymous/f2a94234195f007c3049e27d942482c1
>> >> [2]
>> >>
>> https://github.com/apache/couchdb-couch/blob/master/priv/spawnkillable/couchspawnkillable_win.c#L106
>> >>
>> >> On Wed, Jun 29, 2016 at 4:21 PM, Sebastian Rothbucher
>> >> <se...@googlemail.com> wrote:
>> >> > Hi all,
>> >> >
>> >> > just to follow up on that: there is another PR coming up (
>> >> > https://github.com/apache/couchdb/pull/427) that tests for some more
>> >> fixes
>> >> > and brings even more stability. In the meantime, deleting dev/lib is
>> >> indeed
>> >> > the best way to produce reliable results. So is switching between
>> >> > auth-tests-wip and master. But there's progress => it might all end
>> up
>> >> > nicely on master.
>> >> >
>> >> > Good luck, thanks & best
>> >> >     Sebastian
>> >> >
>> >> >
>> >> > On Mon, Jun 20, 2016 at 5:26 PM, Nick North <no...@gmail.com>
>> wrote:
>> >> >
>> >> >> Thanks Sebastian. I'm looking at eunit at the moment, but hope to
>> come
>> >> back
>> >> >> to these.
>> >> >>
>> >> >> Nick
>> >> >>
>> >> >> On Sun, 19 Jun 2016 at 23:01 Sebastian Rothbucher <
>> >> >> sebastianrothbucher@googlemail.com> wrote:
>> >> >>
>> >> >> > Hi Joan, Nick,
>> >> >> >
>> >> >> > the following gist provides a current run of the test against the
>> >> latest
>> >> >> > master of CouchDB - and the latest tests (from the auth-tests-wip
>> >> >> branch):
>> >> >> >
>> >> >> >
>> >> >>
>> >>
>> https://gist.github.com/sebastianrothbucher/efa3a992bd4de9996b4125da82a7e0de
>> >> >> > Maybe you can use them
>> >> >> >
>> >> >> > Here's what I did to get both latest tests and latest code:
>> >> >> > git checkout master
>> >> >> > ./configure -c --disable-docs --disable-fauxton
>> >> >> > make clean
>> >> >> > make
>> >> >> > git checkout auth-tests-wip
>> >> >> >
>> >> >> > Currently, make javascript seems not optimal as one tests (needs
>> >> >> > investigation) seems to mess up the setup for all that's
>> following.
>> >> >> Hence,
>> >> >> > I took this drastic measure to produce the logs:
>> >> >> >
>> >> >> > for t in test/javascript/tests/*.js; do rm -rf dev/lib; dev/run
>> -n 1
>> >> -q
>> >> >> > --with-admin-party-please test/javascript/run $t 2>&1 | tee -a
>> >> >> jstest2.log;
>> >> >> > done
>> >> >> >
>> >> >> > Maybe it makes sense for you to start w/ something similar to
>> produce
>> >> >> some
>> >> >> > meaningful results.
>> >> >> >
>> >> >> > Best
>> >> >> >    Sebastian
>> >> >> >
>> >> >> > On Sun, Jun 19, 2016 at 6:04 PM, Nick North <no...@gmail.com>
>> >> wrote:
>> >> >> >
>> >> >> > > Sorry - I meant a single node cluster in that last message. And
>> I
>> >> meant
>> >> >> > to
>> >> >> > > sign my name correctly.
>> >> >> > >
>> >> >> > > Nick
>> >> >> > >
>> >> >> > > On Sun, 19 Jun 2016 at 16:56 Nick North <no...@gmail.com>
>> wrote:
>> >> >> > >
>> >> >> > > > I'm trying these tests now, and find that there are still a
>> lot
>> >> of JS
>> >> >> > > > failures with a single cluster. Many of them look suspiciously
>> >> >> similar
>> >> >> > at
>> >> >> > > > an initial glance, but I hope to look in more detail tomorrow.
>> >> >> > > >
>> >> >> > > > Nicj
>> >> >> > > >
>> >> >> > > > On Sun, 19 Jun 2016 at 15:20 Jan Lehnardt <ja...@apache.org>
>> wrote:
>> >> >> > > >
>> >> >> > > >>
>> >> >> > > >> > On 17 Jun 2016, at 22:48, Joan Touzet <wo...@apache.org>
>> >> wrote:
>> >> >> > > >> >
>> >> >> > > >> > Hello everyone,
>> >> >> > > >> >
>> >> >> > > >> > I'd like to update the community on the status of the 2.0
>> port
>> >> to
>> >> >> > > >> Microsoft Windows. There are three parts to this email: the
>> build
>> >> >> > > >> tools/chain themselves, support in CouchDB for the Windows
>> build
>> >> >> > > process,
>> >> >> > > >> and testing results. I'll cover them in that order.
>> >> >> > > >> >
>> >> >> > > >> > -Joan
>> >> >> > > >> >
>> >> >> > > >> > Build Tools/Chain
>> >> >> > > >> > =================
>> >> >> > > >> > ** TL;DR: New glazier repo to join couchdb, contains
>> scripts
>> >> and
>> >> >> > > README
>> >> >> > > >> to build CouchDB 2.0 on Windows.
>> >> >> > > >> >
>> >> >> > > >> > Our work to date has been going on in Dave Cottlehuber's
>> >> glazier
>> >> >> > > >> repository at
>> >> >> > > >> >
>> >> >> > > >> >  https://github.com/dch/glazier/tree/release/couchdb_2.0
>> >> >> > > >> >
>> >> >> > > >> > The reason for the extra repository is that the Windows
>> build
>> >> >> > process
>> >> >> > > >> is *very* ugly, involving 3 distinct build chains (Visual
>> Studio,
>> >> >> > Cygwin
>> >> >> > > >> and the Mozilla Build system) to build all of the necessary
>> >> >> > > prerequisites.
>> >> >> > > >> The repository includes a number of support scripts to set up
>> >> that
>> >> >> > > >> environment, a README with a detailed walkthrough, and some
>> >> patches
>> >> >> > > >> necessary to the prerequisites to get them to build under the
>> >> modern
>> >> >> > > >> Windows b uild system.
>> >> >> > > >> >
>> >> >> > > >> > Parenthetically, it _is_ possible to use binary installs
>> for
>> >> the
>> >> >> > > >> prerequisites (OpenSSL, libcurl, Erlang, SM 1.8.5), but Dave,
>> >> Nick
>> >> >> > North
>> >> >> > > >> and I have evolved the glazier system over a number of years
>> and
>> >> >> it's
>> >> >> > > >> proven quite effective. Plus, we don't have to worry about
>> the
>> >> >> > > provenance
>> >> >> > > >> of any of the binaries since we build everything from source
>> >> >> directly,
>> >> >> > > and
>> >> >> > > >> that's important when we put up an unofficial Windows build
>> for
>> >> >> > > download at
>> >> >> > > >> https://couchdb.apache.org/ .
>> >> >> > > >> >
>> >> >> > > >> > Good news: as of today I've requested and Infra has
>> created a
>> >> new
>> >> >> > > >> apache couchdb-glazier repo, and it's my intent to mirror
>> >> >> dch/glazier
>> >> >> > > over
>> >> >> > > >> into the ASF's repo once things have stabilized a bit more
>> (PR
>> >> and
>> >> >> > > merge of
>> >> >> > > >> the release/couchdb_2.0 branch, and pending progress on
>> steps 2
>> >> and
>> >> >> 3
>> >> >> > > >> below). Dave and I did an audit of the repository as it
>> stands,
>> >> and
>> >> >> > > since
>> >> >> > > >> all checkins come from CouchDB contributors already, we are
>> good
>> >> to
>> >> >> go
>> >> >> > > from
>> >> >> > > >> a licensing perspective.
>> >> >> > > >> >
>> >> >> > > >> >
>> >> >> > > >> > Overall CouchDB Windows support
>> >> >> > > >> > ===============================
>> >> >> > > >> > ** TL;DR: Windows support in 2.0 a priority, conversion of
>> >> >> top-level
>> >> >> > > >> Makefile in progress.
>> >> >> > > >> >
>> >> >> > > >> > There are two aspects to native CouchDB Windows support.
>> The
>> >> first
>> >> >> > is
>> >> >> > > >> anything within the CouchDB code itself that assumes a
>> Unix-like
>> >> >> > > >> environment. Fortunately, most of these problems have been
>> worked
>> >> >> out
>> >> >> > in
>> >> >> > > >> prior releases. I'm not aware of any outstanding issues here
>> >> (except
>> >> >> > one
>> >> >> > > >> point below under test results).
>> >> >> > > >> >
>> >> >> > > >> > The other aspect is the build setup within the couchdb repo
>> >> >> itself.
>> >> >> > > >> I've already converted the bash configure script into a
>> >> PowerShell
>> >> >> > > >> configure script that works fine. However, the Makefile has
>> >> bashisms
>> >> >> > in
>> >> >> > > it
>> >> >> > > >> and assumes GNU Make. I've started a conversion of this into
>> >> Windows
>> >> >> > > NMake
>> >> >> > > >> format and will submit a PR for a Makefile.win in due course.
>> >> >> > > >> >
>> >> >> > > >> > I want to answer two frequent questions we get here before
>> they
>> >> >> get
>> >> >> > > >> re-asked:
>> >> >> > > >> >
>> >> >> > > >> >  1) Why not use a cygwin environment to retain
>> compatibility
>> >> with
>> >> >> > the
>> >> >> > > >> Unix build process? The answer is that performance suffers,
>> the
>> >> >> build
>> >> >> > > chain
>> >> >> > > >> is onerous, there are link-time problems when trying to link
>> >> against
>> >> >> > > things
>> >> >> > > >> built using Visual Studio, and there are still assumptions on
>> >> paths
>> >> >> > that
>> >> >> > > >> don't work out. We can't get away from making
>> Windows-specific
>> >> >> > > >> customizations to the build process anyway, so we might as
>> well
>> >> take
>> >> >> > the
>> >> >> > > >> extra step and support the build process properly. It's not
>> THAT
>> >> >> much
>> >> >> > > work
>> >> >> > > >> to convert the Makefile and configure script, and our
>> top-level
>> >> >> > Makefile
>> >> >> > > >> really isn't much more than a shell script anyway (every
>> target
>> >> is a
>> >> >> > > .PHONY
>> >> >> > > >> target!). In fact, a TODO for an enterprising developer
>> might be
>> >> to
>> >> >> > > rewrite
>> >> >> > > >> our top-level Makefile/Makefile.win as a Python script that
>> "does
>> >> >> the
>> >> >> > > right
>> >> >> > > >> thing" on both platforms, the same way our dev/run script
>> works
>> >> >> today.
>> >> >> > > >> >
>> >> >> > > >> >  2) Why not use the new "Bash and Ubuntu on Windows"
>> >> functionality
>> >> >> > > >> Microsoft has announced for Windows 10? There are two
>> distinct
>> >> >> > problems
>> >> >> > > >> here. The first is that there is a very large install base
>> still
>> >> of
>> >> >> > > Windows
>> >> >> > > >> 7 and 8 (and Windows Server) machines that cannot run this
>> >> >> subsystem.
>> >> >> > > The
>> >> >> > > >> second is that Microsoft themselves say this about the
>> >> >> functionality:
>> >> >> > > >> >
>> >> >> > > >> >     "Second, while you’ll be able to run native Bash and
>> many
>> >> >> Linux
>> >> >> > > >> command-line tools on Windows, it’s important to note that
>> this
>> >> is a
>> >> >> > > >> developer toolset to help you write and build all your code
>> for
>> >> all
>> >> >> > your
>> >> >> > > >> scenarios and platforms. This is not a server platform upon
>> which
>> >> >> you
>> >> >> > > will
>> >> >> > > >> host websites, run server infrastructure, etc."
>> >> >> > > >> >
>> >> >> > > >> > Given this strong warning from Microsoft themselves (which
>> >> hints
>> >> >> at
>> >> >> > > >> performance consideratings), and the fact that download
>> >> statistics
>> >> >> > show
>> >> >> > > an
>> >> >> > > >> equal number of downloads of the CouchDB .tar source and the
>> >> Windows
>> >> >> > > .zip
>> >> >> > > >> installer from our couchdb.apache.org website, we need to
>> >> consider
>> >> >> > that
>> >> >> > > >> people are running CouchDB on Windows not just as a developer
>> >> tool
>> >> >> but
>> >> >> > > as a
>> >> >> > > >> fully-fledged server. As such it behooves us to build it
>> >> "properly"
>> >> >> > as a
>> >> >> > > >> normal Windows binary/service.
>> >> >> > > >> >
>> >> >> > > >>
>> >> >> > > >> Great progress Joan! Thank you! :)
>> >> >> > > >>
>> >> >> > > >> > Test Results
>> >> >> > > >> > ============
>> >> >> > > >> > ** TL;DR: Lots of things are failing. Joan needs help
>> >> interpreting
>> >> >> > the
>> >> >> > > >> results or she will go around the bend.
>> >> >> > > >> >
>> >> >> > > >> > Here are the current test results in gist form.
>> >> >> > > >> >
>> >> >> > > >> > EUnit:
>> >> >> > > >>
>> >> https://gist.github.com/anonymous/3203ed27c60cf3da4f0f0d5bff731722
>> >> >> > > >> >
>> >> >> > > >> > JS tests:
>> >> >> > > >>
>> >> https://gist.github.com/anonymous/93b0b70ed445ca4043a63140f8d219bf
>> >> >> > > >> >
>> >> >> > > >> > For the EUnit tests, everything other than os_process stuff
>> >> seems
>> >> >> to
>> >> >> > > be
>> >> >> > > >> working. Honestly, I think we can release without os_process
>> >> support
>> >> >> > on
>> >> >> > > >> Windows, though I should file a bug to track this. I am
>> actually
>> >> >> > > inclined
>> >> >> > > >> to disable os_process support on Windows and the related
>> eunit
>> >> tests
>> >> >> > > rather
>> >> >> > > >> than fix this rarely-needed functionality, unless someone on
>> this
>> >> >> list
>> >> >> > > >> objects.
>> >> >> > > >>
>> >> >> > > >> You are probably thinking about CouchDB Externals, which
>> >> definitely
>> >> >> > use
>> >> >> > > >> os_process functionality and which I’d also be fine with
>> dropping
>> >> >> > > support
>> >> >> > > >> for Windows for now, but os_process is also used by the view
>> >> server,
>> >> >> > so
>> >> >> > > we
>> >> >> > > >> should definitely get them passing.
>> >> >> > > >>
>> >> >> > > >>
>> >> >> > > >> > For the JS tests, a *lot* is failing. I need to know how
>> much
>> >> this
>> >> >> > > >> differs from a Linux/OSX baseline today, can anyone help me
>> >> follow
>> >> >> up
>> >> >> > > here?
>> >> >> > > >>
>> >> >> > > >> Can you try running these against a -n 1 cluster? We are not
>> set
>> >> up
>> >> >> to
>> >> >> > > >> run JS tests against more nodes at this point.
>> >> >> > > >>
>> >> >> > > >> On master/unix most if not all JS tests should either pass or
>> >> >> skipped
>> >> >> > > >> with a TODO message.
>> >> >> > > >>
>> >> >> > > >>
>> >> >> > > >> Best
>> >> >> > > >> Jan
>> >> >> > > >> --
>> >> >> > > >>
>> >> >> > > >>
>> >> >> > > >>
>> >> >> > >
>> >> >> >
>> >> >>
>> >>
>>
>

Re: 2.0 & Windows: status update

[Nick North <no...@gmail.com>]

Maybe I did something stupid with the PR. If I can get to a PC this evening
I'll take a look.

Nick
On Wed, 13 Jul 2016 at 17:41, Paul Davis <pa...@gmail.com>
wrote:

> That Jiffy error does seem like a conversion issue either from integer
> widths or endianess. I don't remember seeing a patch for that on
> Windows and looking back I'm not seeing anything from after 2013 that
> looks related (in davisp/jiffy). Looking at the apache/jiffy repo we
> use merges so I'd think it'd be unlikely to have been overwritten but
> I guess its always possible.
>
> Paul
>
> On Wed, Jul 13, 2016 at 3:31 AM, Nick North <no...@gmail.com> wrote:
> > I think I submitted a PR for that Jiffy problem a while back: it's some
> > 32/64 bit issue. But I thought I'd also merged it. Unfortunately I'm away
> > from any useful tools for the next few days so can't check myself, but
> take
> > a look at the activity on the couchdb-jiffy repository. It's possible
> that
> > there has been a pull from upstream that overwrote my change.
> >
> > Nick
> >
> > On Wed, 13 Jul 2016 at 09:24 Joan Touzet <wo...@apache.org> wrote:
> >
> >> I took a look at the eunit failures and found that the entire
> >> couchdb_os_daemon_test module fails due to issues with how it expects
> >> to launch daemons.
> >>
> >> The main issue is that the primary test harness is a .escript file,
> which
> >> on *nix is magically parsed via the #!/usr/bin/escript header. On
> Windows
> >> we are just trying to directly launch the .escript file which fails
> (since
> >> Windows has no idea how to execute *.escript files). There are also .sh
> >> scripts that are part of the test harness that will not run correctly.
> >>
> >> I've submitted a PR to simply bypass this entire test module for now. It
> >> sure would be swell to make it work but it'll be a fair bit of fiddling,
> >> especially in a way that makes eunit happy, to get it to work.
> >>
> >> Besides, there are other tests that drive the mrview os daemon; we will
> >> see massive failures in the JS test suite if the entire os daemon
> launching
> >> process fails.
> >>
> >> https://github.com/apache/couchdb-couch/pull/184 hopefully will land
> soon
> >> allowing the main couch eunit tests to pass.
> >>
> >> More worryingly we have a failure in jiffy on Windows:
> >>
> >>
> >>
> https://gist.github.com/anonymous/c796d4b048efa90b17b1f43008c59783#file-gistfile1-txt-L381-L389
> >>
> >> Anyone who can help look at this one? (Paul?)
> >>
> >> -Joan
> >>
> >> ----- Original Message -----
> >> From: "Joan Touzet" <wo...@apache.org>
> >> To: dev@couchdb.apache.org
> >> Sent: Tuesday, July 12, 2016 6:02:12 PM
> >> Subject: Re: 2.0 & Windows: status update
> >>
> >> Thanks, Paul. I'm starting to look at this today.
> >>
> >> In better news, current Windows JS tests now match the *nix JS test
> >> results.
> >> Only one test, replication.js, is failing (ignoring the ignored tests).
> >>
> >> Results:
> >> https://gist.github.com/anonymous/8e236848a89af440d3c56569e81f4829
> >>
> >> Mr. Newson is looking at this failure right now and says we may be able
> >> to improve upon the testing methodology.
> >>
> >> -Joan
> >>
> >> ----- Original Message -----
> >> From: "Paul Davis" <pa...@gmail.com>
> >> To: dev@couchdb.apache.org
> >> Sent: Tuesday, July 12, 2016 4:37:15 PM
> >> Subject: Re: 2.0 & Windows: status update
> >>
> >> The logs posted at [1] show that we're seeing OS processes die with an
> >> exit code of 4. The most likely place I can find that that comes from
> >> is couchspawnkillable_win.c [2] which is nicely Windows specific so
> >> would do a lot to explain why we don't see it on *nix systems.
> >> Unfortunately other than pointing out that the subprocess creation
> >> seems to fail I don't have any idea or suggestion on how to debug
> >> further.
> >>
> >> [1] https://gist.github.com/anonymous/f2a94234195f007c3049e27d942482c1
> >> [2]
> >>
> https://github.com/apache/couchdb-couch/blob/master/priv/spawnkillable/couchspawnkillable_win.c#L106
> >>
> >> On Wed, Jun 29, 2016 at 4:21 PM, Sebastian Rothbucher
> >> <se...@googlemail.com> wrote:
> >> > Hi all,
> >> >
> >> > just to follow up on that: there is another PR coming up (
> >> > https://github.com/apache/couchdb/pull/427) that tests for some more
> >> fixes
> >> > and brings even more stability. In the meantime, deleting dev/lib is
> >> indeed
> >> > the best way to produce reliable results. So is switching between
> >> > auth-tests-wip and master. But there's progress => it might all end up
> >> > nicely on master.
> >> >
> >> > Good luck, thanks & best
> >> >     Sebastian
> >> >
> >> >
> >> > On Mon, Jun 20, 2016 at 5:26 PM, Nick North <no...@gmail.com>
> wrote:
> >> >
> >> >> Thanks Sebastian. I'm looking at eunit at the moment, but hope to
> come
> >> back
> >> >> to these.
> >> >>
> >> >> Nick
> >> >>
> >> >> On Sun, 19 Jun 2016 at 23:01 Sebastian Rothbucher <
> >> >> sebastianrothbucher@googlemail.com> wrote:
> >> >>
> >> >> > Hi Joan, Nick,
> >> >> >
> >> >> > the following gist provides a current run of the test against the
> >> latest
> >> >> > master of CouchDB - and the latest tests (from the auth-tests-wip
> >> >> branch):
> >> >> >
> >> >> >
> >> >>
> >>
> https://gist.github.com/sebastianrothbucher/efa3a992bd4de9996b4125da82a7e0de
> >> >> > Maybe you can use them
> >> >> >
> >> >> > Here's what I did to get both latest tests and latest code:
> >> >> > git checkout master
> >> >> > ./configure -c --disable-docs --disable-fauxton
> >> >> > make clean
> >> >> > make
> >> >> > git checkout auth-tests-wip
> >> >> >
> >> >> > Currently, make javascript seems not optimal as one tests (needs
> >> >> > investigation) seems to mess up the setup for all that's following.
> >> >> Hence,
> >> >> > I took this drastic measure to produce the logs:
> >> >> >
> >> >> > for t in test/javascript/tests/*.js; do rm -rf dev/lib; dev/run -n
> 1
> >> -q
> >> >> > --with-admin-party-please test/javascript/run $t 2>&1 | tee -a
> >> >> jstest2.log;
> >> >> > done
> >> >> >
> >> >> > Maybe it makes sense for you to start w/ something similar to
> produce
> >> >> some
> >> >> > meaningful results.
> >> >> >
> >> >> > Best
> >> >> >    Sebastian
> >> >> >
> >> >> > On Sun, Jun 19, 2016 at 6:04 PM, Nick North <no...@gmail.com>
> >> wrote:
> >> >> >
> >> >> > > Sorry - I meant a single node cluster in that last message. And I
> >> meant
> >> >> > to
> >> >> > > sign my name correctly.
> >> >> > >
> >> >> > > Nick
> >> >> > >
> >> >> > > On Sun, 19 Jun 2016 at 16:56 Nick North <no...@gmail.com>
> wrote:
> >> >> > >
> >> >> > > > I'm trying these tests now, and find that there are still a lot
> >> of JS
> >> >> > > > failures with a single cluster. Many of them look suspiciously
> >> >> similar
> >> >> > at
> >> >> > > > an initial glance, but I hope to look in more detail tomorrow.
> >> >> > > >
> >> >> > > > Nicj
> >> >> > > >
> >> >> > > > On Sun, 19 Jun 2016 at 15:20 Jan Lehnardt <ja...@apache.org>
> wrote:
> >> >> > > >
> >> >> > > >>
> >> >> > > >> > On 17 Jun 2016, at 22:48, Joan Touzet <wo...@apache.org>
> >> wrote:
> >> >> > > >> >
> >> >> > > >> > Hello everyone,
> >> >> > > >> >
> >> >> > > >> > I'd like to update the community on the status of the 2.0
> port
> >> to
> >> >> > > >> Microsoft Windows. There are three parts to this email: the
> build
> >> >> > > >> tools/chain themselves, support in CouchDB for the Windows
> build
> >> >> > > process,
> >> >> > > >> and testing results. I'll cover them in that order.
> >> >> > > >> >
> >> >> > > >> > -Joan
> >> >> > > >> >
> >> >> > > >> > Build Tools/Chain
> >> >> > > >> > =================
> >> >> > > >> > ** TL;DR: New glazier repo to join couchdb, contains scripts
> >> and
> >> >> > > README
> >> >> > > >> to build CouchDB 2.0 on Windows.
> >> >> > > >> >
> >> >> > > >> > Our work to date has been going on in Dave Cottlehuber's
> >> glazier
> >> >> > > >> repository at
> >> >> > > >> >
> >> >> > > >> >  https://github.com/dch/glazier/tree/release/couchdb_2.0
> >> >> > > >> >
> >> >> > > >> > The reason for the extra repository is that the Windows
> build
> >> >> > process
> >> >> > > >> is *very* ugly, involving 3 distinct build chains (Visual
> Studio,
> >> >> > Cygwin
> >> >> > > >> and the Mozilla Build system) to build all of the necessary
> >> >> > > prerequisites.
> >> >> > > >> The repository includes a number of support scripts to set up
> >> that
> >> >> > > >> environment, a README with a detailed walkthrough, and some
> >> patches
> >> >> > > >> necessary to the prerequisites to get them to build under the
> >> modern
> >> >> > > >> Windows b uild system.
> >> >> > > >> >
> >> >> > > >> > Parenthetically, it _is_ possible to use binary installs for
> >> the
> >> >> > > >> prerequisites (OpenSSL, libcurl, Erlang, SM 1.8.5), but Dave,
> >> Nick
> >> >> > North
> >> >> > > >> and I have evolved the glazier system over a number of years
> and
> >> >> it's
> >> >> > > >> proven quite effective. Plus, we don't have to worry about the
> >> >> > > provenance
> >> >> > > >> of any of the binaries since we build everything from source
> >> >> directly,
> >> >> > > and
> >> >> > > >> that's important when we put up an unofficial Windows build
> for
> >> >> > > download at
> >> >> > > >> https://couchdb.apache.org/ .
> >> >> > > >> >
> >> >> > > >> > Good news: as of today I've requested and Infra has created
> a
> >> new
> >> >> > > >> apache couchdb-glazier repo, and it's my intent to mirror
> >> >> dch/glazier
> >> >> > > over
> >> >> > > >> into the ASF's repo once things have stabilized a bit more (PR
> >> and
> >> >> > > merge of
> >> >> > > >> the release/couchdb_2.0 branch, and pending progress on steps
> 2
> >> and
> >> >> 3
> >> >> > > >> below). Dave and I did an audit of the repository as it
> stands,
> >> and
> >> >> > > since
> >> >> > > >> all checkins come from CouchDB contributors already, we are
> good
> >> to
> >> >> go
> >> >> > > from
> >> >> > > >> a licensing perspective.
> >> >> > > >> >
> >> >> > > >> >
> >> >> > > >> > Overall CouchDB Windows support
> >> >> > > >> > ===============================
> >> >> > > >> > ** TL;DR: Windows support in 2.0 a priority, conversion of
> >> >> top-level
> >> >> > > >> Makefile in progress.
> >> >> > > >> >
> >> >> > > >> > There are two aspects to native CouchDB Windows support. The
> >> first
> >> >> > is
> >> >> > > >> anything within the CouchDB code itself that assumes a
> Unix-like
> >> >> > > >> environment. Fortunately, most of these problems have been
> worked
> >> >> out
> >> >> > in
> >> >> > > >> prior releases. I'm not aware of any outstanding issues here
> >> (except
> >> >> > one
> >> >> > > >> point below under test results).
> >> >> > > >> >
> >> >> > > >> > The other aspect is the build setup within the couchdb repo
> >> >> itself.
> >> >> > > >> I've already converted the bash configure script into a
> >> PowerShell
> >> >> > > >> configure script that works fine. However, the Makefile has
> >> bashisms
> >> >> > in
> >> >> > > it
> >> >> > > >> and assumes GNU Make. I've started a conversion of this into
> >> Windows
> >> >> > > NMake
> >> >> > > >> format and will submit a PR for a Makefile.win in due course.
> >> >> > > >> >
> >> >> > > >> > I want to answer two frequent questions we get here before
> they
> >> >> get
> >> >> > > >> re-asked:
> >> >> > > >> >
> >> >> > > >> >  1) Why not use a cygwin environment to retain compatibility
> >> with
> >> >> > the
> >> >> > > >> Unix build process? The answer is that performance suffers,
> the
> >> >> build
> >> >> > > chain
> >> >> > > >> is onerous, there are link-time problems when trying to link
> >> against
> >> >> > > things
> >> >> > > >> built using Visual Studio, and there are still assumptions on
> >> paths
> >> >> > that
> >> >> > > >> don't work out. We can't get away from making Windows-specific
> >> >> > > >> customizations to the build process anyway, so we might as
> well
> >> take
> >> >> > the
> >> >> > > >> extra step and support the build process properly. It's not
> THAT
> >> >> much
> >> >> > > work
> >> >> > > >> to convert the Makefile and configure script, and our
> top-level
> >> >> > Makefile
> >> >> > > >> really isn't much more than a shell script anyway (every
> target
> >> is a
> >> >> > > .PHONY
> >> >> > > >> target!). In fact, a TODO for an enterprising developer might
> be
> >> to
> >> >> > > rewrite
> >> >> > > >> our top-level Makefile/Makefile.win as a Python script that
> "does
> >> >> the
> >> >> > > right
> >> >> > > >> thing" on both platforms, the same way our dev/run script
> works
> >> >> today.
> >> >> > > >> >
> >> >> > > >> >  2) Why not use the new "Bash and Ubuntu on Windows"
> >> functionality
> >> >> > > >> Microsoft has announced for Windows 10? There are two distinct
> >> >> > problems
> >> >> > > >> here. The first is that there is a very large install base
> still
> >> of
> >> >> > > Windows
> >> >> > > >> 7 and 8 (and Windows Server) machines that cannot run this
> >> >> subsystem.
> >> >> > > The
> >> >> > > >> second is that Microsoft themselves say this about the
> >> >> functionality:
> >> >> > > >> >
> >> >> > > >> >     "Second, while you’ll be able to run native Bash and
> many
> >> >> Linux
> >> >> > > >> command-line tools on Windows, it’s important to note that
> this
> >> is a
> >> >> > > >> developer toolset to help you write and build all your code
> for
> >> all
> >> >> > your
> >> >> > > >> scenarios and platforms. This is not a server platform upon
> which
> >> >> you
> >> >> > > will
> >> >> > > >> host websites, run server infrastructure, etc."
> >> >> > > >> >
> >> >> > > >> > Given this strong warning from Microsoft themselves (which
> >> hints
> >> >> at
> >> >> > > >> performance consideratings), and the fact that download
> >> statistics
> >> >> > show
> >> >> > > an
> >> >> > > >> equal number of downloads of the CouchDB .tar source and the
> >> Windows
> >> >> > > .zip
> >> >> > > >> installer from our couchdb.apache.org website, we need to
> >> consider
> >> >> > that
> >> >> > > >> people are running CouchDB on Windows not just as a developer
> >> tool
> >> >> but
> >> >> > > as a
> >> >> > > >> fully-fledged server. As such it behooves us to build it
> >> "properly"
> >> >> > as a
> >> >> > > >> normal Windows binary/service.
> >> >> > > >> >
> >> >> > > >>
> >> >> > > >> Great progress Joan! Thank you! :)
> >> >> > > >>
> >> >> > > >> > Test Results
> >> >> > > >> > ============
> >> >> > > >> > ** TL;DR: Lots of things are failing. Joan needs help
> >> interpreting
> >> >> > the
> >> >> > > >> results or she will go around the bend.
> >> >> > > >> >
> >> >> > > >> > Here are the current test results in gist form.
> >> >> > > >> >
> >> >> > > >> > EUnit:
> >> >> > > >>
> >> https://gist.github.com/anonymous/3203ed27c60cf3da4f0f0d5bff731722
> >> >> > > >> >
> >> >> > > >> > JS tests:
> >> >> > > >>
> >> https://gist.github.com/anonymous/93b0b70ed445ca4043a63140f8d219bf
> >> >> > > >> >
> >> >> > > >> > For the EUnit tests, everything other than os_process stuff
> >> seems
> >> >> to
> >> >> > > be
> >> >> > > >> working. Honestly, I think we can release without os_process
> >> support
> >> >> > on
> >> >> > > >> Windows, though I should file a bug to track this. I am
> actually
> >> >> > > inclined
> >> >> > > >> to disable os_process support on Windows and the related eunit
> >> tests
> >> >> > > rather
> >> >> > > >> than fix this rarely-needed functionality, unless someone on
> this
> >> >> list
> >> >> > > >> objects.
> >> >> > > >>
> >> >> > > >> You are probably thinking about CouchDB Externals, which
> >> definitely
> >> >> > use
> >> >> > > >> os_process functionality and which I’d also be fine with
> dropping
> >> >> > > support
> >> >> > > >> for Windows for now, but os_process is also used by the view
> >> server,
> >> >> > so
> >> >> > > we
> >> >> > > >> should definitely get them passing.
> >> >> > > >>
> >> >> > > >>
> >> >> > > >> > For the JS tests, a *lot* is failing. I need to know how
> much
> >> this
> >> >> > > >> differs from a Linux/OSX baseline today, can anyone help me
> >> follow
> >> >> up
> >> >> > > here?
> >> >> > > >>
> >> >> > > >> Can you try running these against a -n 1 cluster? We are not
> set
> >> up
> >> >> to
> >> >> > > >> run JS tests against more nodes at this point.
> >> >> > > >>
> >> >> > > >> On master/unix most if not all JS tests should either pass or
> >> >> skipped
> >> >> > > >> with a TODO message.
> >> >> > > >>
> >> >> > > >>
> >> >> > > >> Best
> >> >> > > >> Jan
> >> >> > > >> --
> >> >> > > >>
> >> >> > > >>
> >> >> > > >>
> >> >> > >
> >> >> >
> >> >>
> >>
>

Re: 2.0 & Windows: status update

[Paul Davis <pa...@gmail.com>]

That Jiffy error does seem like a conversion issue either from integer
widths or endianess. I don't remember seeing a patch for that on
Windows and looking back I'm not seeing anything from after 2013 that
looks related (in davisp/jiffy). Looking at the apache/jiffy repo we
use merges so I'd think it'd be unlikely to have been overwritten but
I guess its always possible.

Paul

On Wed, Jul 13, 2016 at 3:31 AM, Nick North <no...@gmail.com> wrote:
> I think I submitted a PR for that Jiffy problem a while back: it's some
> 32/64 bit issue. But I thought I'd also merged it. Unfortunately I'm away
> from any useful tools for the next few days so can't check myself, but take
> a look at the activity on the couchdb-jiffy repository. It's possible that
> there has been a pull from upstream that overwrote my change.
>
> Nick
>
> On Wed, 13 Jul 2016 at 09:24 Joan Touzet <wo...@apache.org> wrote:
>
>> I took a look at the eunit failures and found that the entire
>> couchdb_os_daemon_test module fails due to issues with how it expects
>> to launch daemons.
>>
>> The main issue is that the primary test harness is a .escript file, which
>> on *nix is magically parsed via the #!/usr/bin/escript header. On Windows
>> we are just trying to directly launch the .escript file which fails (since
>> Windows has no idea how to execute *.escript files). There are also .sh
>> scripts that are part of the test harness that will not run correctly.
>>
>> I've submitted a PR to simply bypass this entire test module for now. It
>> sure would be swell to make it work but it'll be a fair bit of fiddling,
>> especially in a way that makes eunit happy, to get it to work.
>>
>> Besides, there are other tests that drive the mrview os daemon; we will
>> see massive failures in the JS test suite if the entire os daemon launching
>> process fails.
>>
>> https://github.com/apache/couchdb-couch/pull/184 hopefully will land soon
>> allowing the main couch eunit tests to pass.
>>
>> More worryingly we have a failure in jiffy on Windows:
>>
>>
>> https://gist.github.com/anonymous/c796d4b048efa90b17b1f43008c59783#file-gistfile1-txt-L381-L389
>>
>> Anyone who can help look at this one? (Paul?)
>>
>> -Joan
>>
>> ----- Original Message -----
>> From: "Joan Touzet" <wo...@apache.org>
>> To: dev@couchdb.apache.org
>> Sent: Tuesday, July 12, 2016 6:02:12 PM
>> Subject: Re: 2.0 & Windows: status update
>>
>> Thanks, Paul. I'm starting to look at this today.
>>
>> In better news, current Windows JS tests now match the *nix JS test
>> results.
>> Only one test, replication.js, is failing (ignoring the ignored tests).
>>
>> Results:
>> https://gist.github.com/anonymous/8e236848a89af440d3c56569e81f4829
>>
>> Mr. Newson is looking at this failure right now and says we may be able
>> to improve upon the testing methodology.
>>
>> -Joan
>>
>> ----- Original Message -----
>> From: "Paul Davis" <pa...@gmail.com>
>> To: dev@couchdb.apache.org
>> Sent: Tuesday, July 12, 2016 4:37:15 PM
>> Subject: Re: 2.0 & Windows: status update
>>
>> The logs posted at [1] show that we're seeing OS processes die with an
>> exit code of 4. The most likely place I can find that that comes from
>> is couchspawnkillable_win.c [2] which is nicely Windows specific so
>> would do a lot to explain why we don't see it on *nix systems.
>> Unfortunately other than pointing out that the subprocess creation
>> seems to fail I don't have any idea or suggestion on how to debug
>> further.
>>
>> [1] https://gist.github.com/anonymous/f2a94234195f007c3049e27d942482c1
>> [2]
>> https://github.com/apache/couchdb-couch/blob/master/priv/spawnkillable/couchspawnkillable_win.c#L106
>>
>> On Wed, Jun 29, 2016 at 4:21 PM, Sebastian Rothbucher
>> <se...@googlemail.com> wrote:
>> > Hi all,
>> >
>> > just to follow up on that: there is another PR coming up (
>> > https://github.com/apache/couchdb/pull/427) that tests for some more
>> fixes
>> > and brings even more stability. In the meantime, deleting dev/lib is
>> indeed
>> > the best way to produce reliable results. So is switching between
>> > auth-tests-wip and master. But there's progress => it might all end up
>> > nicely on master.
>> >
>> > Good luck, thanks & best
>> >     Sebastian
>> >
>> >
>> > On Mon, Jun 20, 2016 at 5:26 PM, Nick North <no...@gmail.com> wrote:
>> >
>> >> Thanks Sebastian. I'm looking at eunit at the moment, but hope to come
>> back
>> >> to these.
>> >>
>> >> Nick
>> >>
>> >> On Sun, 19 Jun 2016 at 23:01 Sebastian Rothbucher <
>> >> sebastianrothbucher@googlemail.com> wrote:
>> >>
>> >> > Hi Joan, Nick,
>> >> >
>> >> > the following gist provides a current run of the test against the
>> latest
>> >> > master of CouchDB - and the latest tests (from the auth-tests-wip
>> >> branch):
>> >> >
>> >> >
>> >>
>> https://gist.github.com/sebastianrothbucher/efa3a992bd4de9996b4125da82a7e0de
>> >> > Maybe you can use them
>> >> >
>> >> > Here's what I did to get both latest tests and latest code:
>> >> > git checkout master
>> >> > ./configure -c --disable-docs --disable-fauxton
>> >> > make clean
>> >> > make
>> >> > git checkout auth-tests-wip
>> >> >
>> >> > Currently, make javascript seems not optimal as one tests (needs
>> >> > investigation) seems to mess up the setup for all that's following.
>> >> Hence,
>> >> > I took this drastic measure to produce the logs:
>> >> >
>> >> > for t in test/javascript/tests/*.js; do rm -rf dev/lib; dev/run -n 1
>> -q
>> >> > --with-admin-party-please test/javascript/run $t 2>&1 | tee -a
>> >> jstest2.log;
>> >> > done
>> >> >
>> >> > Maybe it makes sense for you to start w/ something similar to produce
>> >> some
>> >> > meaningful results.
>> >> >
>> >> > Best
>> >> >    Sebastian
>> >> >
>> >> > On Sun, Jun 19, 2016 at 6:04 PM, Nick North <no...@gmail.com>
>> wrote:
>> >> >
>> >> > > Sorry - I meant a single node cluster in that last message. And I
>> meant
>> >> > to
>> >> > > sign my name correctly.
>> >> > >
>> >> > > Nick
>> >> > >
>> >> > > On Sun, 19 Jun 2016 at 16:56 Nick North <no...@gmail.com> wrote:
>> >> > >
>> >> > > > I'm trying these tests now, and find that there are still a lot
>> of JS
>> >> > > > failures with a single cluster. Many of them look suspiciously
>> >> similar
>> >> > at
>> >> > > > an initial glance, but I hope to look in more detail tomorrow.
>> >> > > >
>> >> > > > Nicj
>> >> > > >
>> >> > > > On Sun, 19 Jun 2016 at 15:20 Jan Lehnardt <ja...@apache.org> wrote:
>> >> > > >
>> >> > > >>
>> >> > > >> > On 17 Jun 2016, at 22:48, Joan Touzet <wo...@apache.org>
>> wrote:
>> >> > > >> >
>> >> > > >> > Hello everyone,
>> >> > > >> >
>> >> > > >> > I'd like to update the community on the status of the 2.0 port
>> to
>> >> > > >> Microsoft Windows. There are three parts to this email: the build
>> >> > > >> tools/chain themselves, support in CouchDB for the Windows build
>> >> > > process,
>> >> > > >> and testing results. I'll cover them in that order.
>> >> > > >> >
>> >> > > >> > -Joan
>> >> > > >> >
>> >> > > >> > Build Tools/Chain
>> >> > > >> > =================
>> >> > > >> > ** TL;DR: New glazier repo to join couchdb, contains scripts
>> and
>> >> > > README
>> >> > > >> to build CouchDB 2.0 on Windows.
>> >> > > >> >
>> >> > > >> > Our work to date has been going on in Dave Cottlehuber's
>> glazier
>> >> > > >> repository at
>> >> > > >> >
>> >> > > >> >  https://github.com/dch/glazier/tree/release/couchdb_2.0
>> >> > > >> >
>> >> > > >> > The reason for the extra repository is that the Windows build
>> >> > process
>> >> > > >> is *very* ugly, involving 3 distinct build chains (Visual Studio,
>> >> > Cygwin
>> >> > > >> and the Mozilla Build system) to build all of the necessary
>> >> > > prerequisites.
>> >> > > >> The repository includes a number of support scripts to set up
>> that
>> >> > > >> environment, a README with a detailed walkthrough, and some
>> patches
>> >> > > >> necessary to the prerequisites to get them to build under the
>> modern
>> >> > > >> Windows b uild system.
>> >> > > >> >
>> >> > > >> > Parenthetically, it _is_ possible to use binary installs for
>> the
>> >> > > >> prerequisites (OpenSSL, libcurl, Erlang, SM 1.8.5), but Dave,
>> Nick
>> >> > North
>> >> > > >> and I have evolved the glazier system over a number of years and
>> >> it's
>> >> > > >> proven quite effective. Plus, we don't have to worry about the
>> >> > > provenance
>> >> > > >> of any of the binaries since we build everything from source
>> >> directly,
>> >> > > and
>> >> > > >> that's important when we put up an unofficial Windows build for
>> >> > > download at
>> >> > > >> https://couchdb.apache.org/ .
>> >> > > >> >
>> >> > > >> > Good news: as of today I've requested and Infra has created a
>> new
>> >> > > >> apache couchdb-glazier repo, and it's my intent to mirror
>> >> dch/glazier
>> >> > > over
>> >> > > >> into the ASF's repo once things have stabilized a bit more (PR
>> and
>> >> > > merge of
>> >> > > >> the release/couchdb_2.0 branch, and pending progress on steps 2
>> and
>> >> 3
>> >> > > >> below). Dave and I did an audit of the repository as it stands,
>> and
>> >> > > since
>> >> > > >> all checkins come from CouchDB contributors already, we are good
>> to
>> >> go
>> >> > > from
>> >> > > >> a licensing perspective.
>> >> > > >> >
>> >> > > >> >
>> >> > > >> > Overall CouchDB Windows support
>> >> > > >> > ===============================
>> >> > > >> > ** TL;DR: Windows support in 2.0 a priority, conversion of
>> >> top-level
>> >> > > >> Makefile in progress.
>> >> > > >> >
>> >> > > >> > There are two aspects to native CouchDB Windows support. The
>> first
>> >> > is
>> >> > > >> anything within the CouchDB code itself that assumes a Unix-like
>> >> > > >> environment. Fortunately, most of these problems have been worked
>> >> out
>> >> > in
>> >> > > >> prior releases. I'm not aware of any outstanding issues here
>> (except
>> >> > one
>> >> > > >> point below under test results).
>> >> > > >> >
>> >> > > >> > The other aspect is the build setup within the couchdb repo
>> >> itself.
>> >> > > >> I've already converted the bash configure script into a
>> PowerShell
>> >> > > >> configure script that works fine. However, the Makefile has
>> bashisms
>> >> > in
>> >> > > it
>> >> > > >> and assumes GNU Make. I've started a conversion of this into
>> Windows
>> >> > > NMake
>> >> > > >> format and will submit a PR for a Makefile.win in due course.
>> >> > > >> >
>> >> > > >> > I want to answer two frequent questions we get here before they
>> >> get
>> >> > > >> re-asked:
>> >> > > >> >
>> >> > > >> >  1) Why not use a cygwin environment to retain compatibility
>> with
>> >> > the
>> >> > > >> Unix build process? The answer is that performance suffers, the
>> >> build
>> >> > > chain
>> >> > > >> is onerous, there are link-time problems when trying to link
>> against
>> >> > > things
>> >> > > >> built using Visual Studio, and there are still assumptions on
>> paths
>> >> > that
>> >> > > >> don't work out. We can't get away from making Windows-specific
>> >> > > >> customizations to the build process anyway, so we might as well
>> take
>> >> > the
>> >> > > >> extra step and support the build process properly. It's not THAT
>> >> much
>> >> > > work
>> >> > > >> to convert the Makefile and configure script, and our top-level
>> >> > Makefile
>> >> > > >> really isn't much more than a shell script anyway (every target
>> is a
>> >> > > .PHONY
>> >> > > >> target!). In fact, a TODO for an enterprising developer might be
>> to
>> >> > > rewrite
>> >> > > >> our top-level Makefile/Makefile.win as a Python script that "does
>> >> the
>> >> > > right
>> >> > > >> thing" on both platforms, the same way our dev/run script works
>> >> today.
>> >> > > >> >
>> >> > > >> >  2) Why not use the new "Bash and Ubuntu on Windows"
>> functionality
>> >> > > >> Microsoft has announced for Windows 10? There are two distinct
>> >> > problems
>> >> > > >> here. The first is that there is a very large install base still
>> of
>> >> > > Windows
>> >> > > >> 7 and 8 (and Windows Server) machines that cannot run this
>> >> subsystem.
>> >> > > The
>> >> > > >> second is that Microsoft themselves say this about the
>> >> functionality:
>> >> > > >> >
>> >> > > >> >     "Second, while you’ll be able to run native Bash and many
>> >> Linux
>> >> > > >> command-line tools on Windows, it’s important to note that this
>> is a
>> >> > > >> developer toolset to help you write and build all your code for
>> all
>> >> > your
>> >> > > >> scenarios and platforms. This is not a server platform upon which
>> >> you
>> >> > > will
>> >> > > >> host websites, run server infrastructure, etc."
>> >> > > >> >
>> >> > > >> > Given this strong warning from Microsoft themselves (which
>> hints
>> >> at
>> >> > > >> performance consideratings), and the fact that download
>> statistics
>> >> > show
>> >> > > an
>> >> > > >> equal number of downloads of the CouchDB .tar source and the
>> Windows
>> >> > > .zip
>> >> > > >> installer from our couchdb.apache.org website, we need to
>> consider
>> >> > that
>> >> > > >> people are running CouchDB on Windows not just as a developer
>> tool
>> >> but
>> >> > > as a
>> >> > > >> fully-fledged server. As such it behooves us to build it
>> "properly"
>> >> > as a
>> >> > > >> normal Windows binary/service.
>> >> > > >> >
>> >> > > >>
>> >> > > >> Great progress Joan! Thank you! :)
>> >> > > >>
>> >> > > >> > Test Results
>> >> > > >> > ============
>> >> > > >> > ** TL;DR: Lots of things are failing. Joan needs help
>> interpreting
>> >> > the
>> >> > > >> results or she will go around the bend.
>> >> > > >> >
>> >> > > >> > Here are the current test results in gist form.
>> >> > > >> >
>> >> > > >> > EUnit:
>> >> > > >>
>> https://gist.github.com/anonymous/3203ed27c60cf3da4f0f0d5bff731722
>> >> > > >> >
>> >> > > >> > JS tests:
>> >> > > >>
>> https://gist.github.com/anonymous/93b0b70ed445ca4043a63140f8d219bf
>> >> > > >> >
>> >> > > >> > For the EUnit tests, everything other than os_process stuff
>> seems
>> >> to
>> >> > > be
>> >> > > >> working. Honestly, I think we can release without os_process
>> support
>> >> > on
>> >> > > >> Windows, though I should file a bug to track this. I am actually
>> >> > > inclined
>> >> > > >> to disable os_process support on Windows and the related eunit
>> tests
>> >> > > rather
>> >> > > >> than fix this rarely-needed functionality, unless someone on this
>> >> list
>> >> > > >> objects.
>> >> > > >>
>> >> > > >> You are probably thinking about CouchDB Externals, which
>> definitely
>> >> > use
>> >> > > >> os_process functionality and which I’d also be fine with dropping
>> >> > > support
>> >> > > >> for Windows for now, but os_process is also used by the view
>> server,
>> >> > so
>> >> > > we
>> >> > > >> should definitely get them passing.
>> >> > > >>
>> >> > > >>
>> >> > > >> > For the JS tests, a *lot* is failing. I need to know how much
>> this
>> >> > > >> differs from a Linux/OSX baseline today, can anyone help me
>> follow
>> >> up
>> >> > > here?
>> >> > > >>
>> >> > > >> Can you try running these against a -n 1 cluster? We are not set
>> up
>> >> to
>> >> > > >> run JS tests against more nodes at this point.
>> >> > > >>
>> >> > > >> On master/unix most if not all JS tests should either pass or
>> >> skipped
>> >> > > >> with a TODO message.
>> >> > > >>
>> >> > > >>
>> >> > > >> Best
>> >> > > >> Jan
>> >> > > >> --
>> >> > > >>
>> >> > > >>
>> >> > > >>
>> >> > >
>> >> >
>> >>
>>

Re: 2.0 & Windows: status update

[Nick North <no...@gmail.com>]

I think I submitted a PR for that Jiffy problem a while back: it's some
32/64 bit issue. But I thought I'd also merged it. Unfortunately I'm away
from any useful tools for the next few days so can't check myself, but take
a look at the activity on the couchdb-jiffy repository. It's possible that
there has been a pull from upstream that overwrote my change.

Nick

On Wed, 13 Jul 2016 at 09:24 Joan Touzet <wo...@apache.org> wrote:

> I took a look at the eunit failures and found that the entire
> couchdb_os_daemon_test module fails due to issues with how it expects
> to launch daemons.
>
> The main issue is that the primary test harness is a .escript file, which
> on *nix is magically parsed via the #!/usr/bin/escript header. On Windows
> we are just trying to directly launch the .escript file which fails (since
> Windows has no idea how to execute *.escript files). There are also .sh
> scripts that are part of the test harness that will not run correctly.
>
> I've submitted a PR to simply bypass this entire test module for now. It
> sure would be swell to make it work but it'll be a fair bit of fiddling,
> especially in a way that makes eunit happy, to get it to work.
>
> Besides, there are other tests that drive the mrview os daemon; we will
> see massive failures in the JS test suite if the entire os daemon launching
> process fails.
>
> https://github.com/apache/couchdb-couch/pull/184 hopefully will land soon
> allowing the main couch eunit tests to pass.
>
> More worryingly we have a failure in jiffy on Windows:
>
>
> https://gist.github.com/anonymous/c796d4b048efa90b17b1f43008c59783#file-gistfile1-txt-L381-L389
>
> Anyone who can help look at this one? (Paul?)
>
> -Joan
>
> ----- Original Message -----
> From: "Joan Touzet" <wo...@apache.org>
> To: dev@couchdb.apache.org
> Sent: Tuesday, July 12, 2016 6:02:12 PM
> Subject: Re: 2.0 & Windows: status update
>
> Thanks, Paul. I'm starting to look at this today.
>
> In better news, current Windows JS tests now match the *nix JS test
> results.
> Only one test, replication.js, is failing (ignoring the ignored tests).
>
> Results:
> https://gist.github.com/anonymous/8e236848a89af440d3c56569e81f4829
>
> Mr. Newson is looking at this failure right now and says we may be able
> to improve upon the testing methodology.
>
> -Joan
>
> ----- Original Message -----
> From: "Paul Davis" <pa...@gmail.com>
> To: dev@couchdb.apache.org
> Sent: Tuesday, July 12, 2016 4:37:15 PM
> Subject: Re: 2.0 & Windows: status update
>
> The logs posted at [1] show that we're seeing OS processes die with an
> exit code of 4. The most likely place I can find that that comes from
> is couchspawnkillable_win.c [2] which is nicely Windows specific so
> would do a lot to explain why we don't see it on *nix systems.
> Unfortunately other than pointing out that the subprocess creation
> seems to fail I don't have any idea or suggestion on how to debug
> further.
>
> [1] https://gist.github.com/anonymous/f2a94234195f007c3049e27d942482c1
> [2]
> https://github.com/apache/couchdb-couch/blob/master/priv/spawnkillable/couchspawnkillable_win.c#L106
>
> On Wed, Jun 29, 2016 at 4:21 PM, Sebastian Rothbucher
> <se...@googlemail.com> wrote:
> > Hi all,
> >
> > just to follow up on that: there is another PR coming up (
> > https://github.com/apache/couchdb/pull/427) that tests for some more
> fixes
> > and brings even more stability. In the meantime, deleting dev/lib is
> indeed
> > the best way to produce reliable results. So is switching between
> > auth-tests-wip and master. But there's progress => it might all end up
> > nicely on master.
> >
> > Good luck, thanks & best
> >     Sebastian
> >
> >
> > On Mon, Jun 20, 2016 at 5:26 PM, Nick North <no...@gmail.com> wrote:
> >
> >> Thanks Sebastian. I'm looking at eunit at the moment, but hope to come
> back
> >> to these.
> >>
> >> Nick
> >>
> >> On Sun, 19 Jun 2016 at 23:01 Sebastian Rothbucher <
> >> sebastianrothbucher@googlemail.com> wrote:
> >>
> >> > Hi Joan, Nick,
> >> >
> >> > the following gist provides a current run of the test against the
> latest
> >> > master of CouchDB - and the latest tests (from the auth-tests-wip
> >> branch):
> >> >
> >> >
> >>
> https://gist.github.com/sebastianrothbucher/efa3a992bd4de9996b4125da82a7e0de
> >> > Maybe you can use them
> >> >
> >> > Here's what I did to get both latest tests and latest code:
> >> > git checkout master
> >> > ./configure -c --disable-docs --disable-fauxton
> >> > make clean
> >> > make
> >> > git checkout auth-tests-wip
> >> >
> >> > Currently, make javascript seems not optimal as one tests (needs
> >> > investigation) seems to mess up the setup for all that's following.
> >> Hence,
> >> > I took this drastic measure to produce the logs:
> >> >
> >> > for t in test/javascript/tests/*.js; do rm -rf dev/lib; dev/run -n 1
> -q
> >> > --with-admin-party-please test/javascript/run $t 2>&1 | tee -a
> >> jstest2.log;
> >> > done
> >> >
> >> > Maybe it makes sense for you to start w/ something similar to produce
> >> some
> >> > meaningful results.
> >> >
> >> > Best
> >> >    Sebastian
> >> >
> >> > On Sun, Jun 19, 2016 at 6:04 PM, Nick North <no...@gmail.com>
> wrote:
> >> >
> >> > > Sorry - I meant a single node cluster in that last message. And I
> meant
> >> > to
> >> > > sign my name correctly.
> >> > >
> >> > > Nick
> >> > >
> >> > > On Sun, 19 Jun 2016 at 16:56 Nick North <no...@gmail.com> wrote:
> >> > >
> >> > > > I'm trying these tests now, and find that there are still a lot
> of JS
> >> > > > failures with a single cluster. Many of them look suspiciously
> >> similar
> >> > at
> >> > > > an initial glance, but I hope to look in more detail tomorrow.
> >> > > >
> >> > > > Nicj
> >> > > >
> >> > > > On Sun, 19 Jun 2016 at 15:20 Jan Lehnardt <ja...@apache.org> wrote:
> >> > > >
> >> > > >>
> >> > > >> > On 17 Jun 2016, at 22:48, Joan Touzet <wo...@apache.org>
> wrote:
> >> > > >> >
> >> > > >> > Hello everyone,
> >> > > >> >
> >> > > >> > I'd like to update the community on the status of the 2.0 port
> to
> >> > > >> Microsoft Windows. There are three parts to this email: the build
> >> > > >> tools/chain themselves, support in CouchDB for the Windows build
> >> > > process,
> >> > > >> and testing results. I'll cover them in that order.
> >> > > >> >
> >> > > >> > -Joan
> >> > > >> >
> >> > > >> > Build Tools/Chain
> >> > > >> > =================
> >> > > >> > ** TL;DR: New glazier repo to join couchdb, contains scripts
> and
> >> > > README
> >> > > >> to build CouchDB 2.0 on Windows.
> >> > > >> >
> >> > > >> > Our work to date has been going on in Dave Cottlehuber's
> glazier
> >> > > >> repository at
> >> > > >> >
> >> > > >> >  https://github.com/dch/glazier/tree/release/couchdb_2.0
> >> > > >> >
> >> > > >> > The reason for the extra repository is that the Windows build
> >> > process
> >> > > >> is *very* ugly, involving 3 distinct build chains (Visual Studio,
> >> > Cygwin
> >> > > >> and the Mozilla Build system) to build all of the necessary
> >> > > prerequisites.
> >> > > >> The repository includes a number of support scripts to set up
> that
> >> > > >> environment, a README with a detailed walkthrough, and some
> patches
> >> > > >> necessary to the prerequisites to get them to build under the
> modern
> >> > > >> Windows b uild system.
> >> > > >> >
> >> > > >> > Parenthetically, it _is_ possible to use binary installs for
> the
> >> > > >> prerequisites (OpenSSL, libcurl, Erlang, SM 1.8.5), but Dave,
> Nick
> >> > North
> >> > > >> and I have evolved the glazier system over a number of years and
> >> it's
> >> > > >> proven quite effective. Plus, we don't have to worry about the
> >> > > provenance
> >> > > >> of any of the binaries since we build everything from source
> >> directly,
> >> > > and
> >> > > >> that's important when we put up an unofficial Windows build for
> >> > > download at
> >> > > >> https://couchdb.apache.org/ .
> >> > > >> >
> >> > > >> > Good news: as of today I've requested and Infra has created a
> new
> >> > > >> apache couchdb-glazier repo, and it's my intent to mirror
> >> dch/glazier
> >> > > over
> >> > > >> into the ASF's repo once things have stabilized a bit more (PR
> and
> >> > > merge of
> >> > > >> the release/couchdb_2.0 branch, and pending progress on steps 2
> and
> >> 3
> >> > > >> below). Dave and I did an audit of the repository as it stands,
> and
> >> > > since
> >> > > >> all checkins come from CouchDB contributors already, we are good
> to
> >> go
> >> > > from
> >> > > >> a licensing perspective.
> >> > > >> >
> >> > > >> >
> >> > > >> > Overall CouchDB Windows support
> >> > > >> > ===============================
> >> > > >> > ** TL;DR: Windows support in 2.0 a priority, conversion of
> >> top-level
> >> > > >> Makefile in progress.
> >> > > >> >
> >> > > >> > There are two aspects to native CouchDB Windows support. The
> first
> >> > is
> >> > > >> anything within the CouchDB code itself that assumes a Unix-like
> >> > > >> environment. Fortunately, most of these problems have been worked
> >> out
> >> > in
> >> > > >> prior releases. I'm not aware of any outstanding issues here
> (except
> >> > one
> >> > > >> point below under test results).
> >> > > >> >
> >> > > >> > The other aspect is the build setup within the couchdb repo
> >> itself.
> >> > > >> I've already converted the bash configure script into a
> PowerShell
> >> > > >> configure script that works fine. However, the Makefile has
> bashisms
> >> > in
> >> > > it
> >> > > >> and assumes GNU Make. I've started a conversion of this into
> Windows
> >> > > NMake
> >> > > >> format and will submit a PR for a Makefile.win in due course.
> >> > > >> >
> >> > > >> > I want to answer two frequent questions we get here before they
> >> get
> >> > > >> re-asked:
> >> > > >> >
> >> > > >> >  1) Why not use a cygwin environment to retain compatibility
> with
> >> > the
> >> > > >> Unix build process? The answer is that performance suffers, the
> >> build
> >> > > chain
> >> > > >> is onerous, there are link-time problems when trying to link
> against
> >> > > things
> >> > > >> built using Visual Studio, and there are still assumptions on
> paths
> >> > that
> >> > > >> don't work out. We can't get away from making Windows-specific
> >> > > >> customizations to the build process anyway, so we might as well
> take
> >> > the
> >> > > >> extra step and support the build process properly. It's not THAT
> >> much
> >> > > work
> >> > > >> to convert the Makefile and configure script, and our top-level
> >> > Makefile
> >> > > >> really isn't much more than a shell script anyway (every target
> is a
> >> > > .PHONY
> >> > > >> target!). In fact, a TODO for an enterprising developer might be
> to
> >> > > rewrite
> >> > > >> our top-level Makefile/Makefile.win as a Python script that "does
> >> the
> >> > > right
> >> > > >> thing" on both platforms, the same way our dev/run script works
> >> today.
> >> > > >> >
> >> > > >> >  2) Why not use the new "Bash and Ubuntu on Windows"
> functionality
> >> > > >> Microsoft has announced for Windows 10? There are two distinct
> >> > problems
> >> > > >> here. The first is that there is a very large install base still
> of
> >> > > Windows
> >> > > >> 7 and 8 (and Windows Server) machines that cannot run this
> >> subsystem.
> >> > > The
> >> > > >> second is that Microsoft themselves say this about the
> >> functionality:
> >> > > >> >
> >> > > >> >     "Second, while you’ll be able to run native Bash and many
> >> Linux
> >> > > >> command-line tools on Windows, it’s important to note that this
> is a
> >> > > >> developer toolset to help you write and build all your code for
> all
> >> > your
> >> > > >> scenarios and platforms. This is not a server platform upon which
> >> you
> >> > > will
> >> > > >> host websites, run server infrastructure, etc."
> >> > > >> >
> >> > > >> > Given this strong warning from Microsoft themselves (which
> hints
> >> at
> >> > > >> performance consideratings), and the fact that download
> statistics
> >> > show
> >> > > an
> >> > > >> equal number of downloads of the CouchDB .tar source and the
> Windows
> >> > > .zip
> >> > > >> installer from our couchdb.apache.org website, we need to
> consider
> >> > that
> >> > > >> people are running CouchDB on Windows not just as a developer
> tool
> >> but
> >> > > as a
> >> > > >> fully-fledged server. As such it behooves us to build it
> "properly"
> >> > as a
> >> > > >> normal Windows binary/service.
> >> > > >> >
> >> > > >>
> >> > > >> Great progress Joan! Thank you! :)
> >> > > >>
> >> > > >> > Test Results
> >> > > >> > ============
> >> > > >> > ** TL;DR: Lots of things are failing. Joan needs help
> interpreting
> >> > the
> >> > > >> results or she will go around the bend.
> >> > > >> >
> >> > > >> > Here are the current test results in gist form.
> >> > > >> >
> >> > > >> > EUnit:
> >> > > >>
> https://gist.github.com/anonymous/3203ed27c60cf3da4f0f0d5bff731722
> >> > > >> >
> >> > > >> > JS tests:
> >> > > >>
> https://gist.github.com/anonymous/93b0b70ed445ca4043a63140f8d219bf
> >> > > >> >
> >> > > >> > For the EUnit tests, everything other than os_process stuff
> seems
> >> to
> >> > > be
> >> > > >> working. Honestly, I think we can release without os_process
> support
> >> > on
> >> > > >> Windows, though I should file a bug to track this. I am actually
> >> > > inclined
> >> > > >> to disable os_process support on Windows and the related eunit
> tests
> >> > > rather
> >> > > >> than fix this rarely-needed functionality, unless someone on this
> >> list
> >> > > >> objects.
> >> > > >>
> >> > > >> You are probably thinking about CouchDB Externals, which
> definitely
> >> > use
> >> > > >> os_process functionality and which I’d also be fine with dropping
> >> > > support
> >> > > >> for Windows for now, but os_process is also used by the view
> server,
> >> > so
> >> > > we
> >> > > >> should definitely get them passing.
> >> > > >>
> >> > > >>
> >> > > >> > For the JS tests, a *lot* is failing. I need to know how much
> this
> >> > > >> differs from a Linux/OSX baseline today, can anyone help me
> follow
> >> up
> >> > > here?
> >> > > >>
> >> > > >> Can you try running these against a -n 1 cluster? We are not set
> up
> >> to
> >> > > >> run JS tests against more nodes at this point.
> >> > > >>
> >> > > >> On master/unix most if not all JS tests should either pass or
> >> skipped
> >> > > >> with a TODO message.
> >> > > >>
> >> > > >>
> >> > > >> Best
> >> > > >> Jan
> >> > > >> --
> >> > > >>
> >> > > >>
> >> > > >>
> >> > >
> >> >
> >>
>

Re: 2.0 & Windows: status update

[Joan Touzet <wo...@apache.org>]

I took a look at the eunit failures and found that the entire
couchdb_os_daemon_test module fails due to issues with how it expects
to launch daemons.

The main issue is that the primary test harness is a .escript file, which
on *nix is magically parsed via the #!/usr/bin/escript header. On Windows
we are just trying to directly launch the .escript file which fails (since
Windows has no idea how to execute *.escript files). There are also .sh
scripts that are part of the test harness that will not run correctly.

I've submitted a PR to simply bypass this entire test module for now. It
sure would be swell to make it work but it'll be a fair bit of fiddling,
especially in a way that makes eunit happy, to get it to work.

Besides, there are other tests that drive the mrview os daemon; we will
see massive failures in the JS test suite if the entire os daemon launching
process fails.

https://github.com/apache/couchdb-couch/pull/184 hopefully will land soon
allowing the main couch eunit tests to pass.

More worryingly we have a failure in jiffy on Windows:

https://gist.github.com/anonymous/c796d4b048efa90b17b1f43008c59783#file-gistfile1-txt-L381-L389

Anyone who can help look at this one? (Paul?)

-Joan

----- Original Message -----
From: "Joan Touzet" <wo...@apache.org>
To: dev@couchdb.apache.org
Sent: Tuesday, July 12, 2016 6:02:12 PM
Subject: Re: 2.0 & Windows: status update

Thanks, Paul. I'm starting to look at this today.

In better news, current Windows JS tests now match the *nix JS test results.
Only one test, replication.js, is failing (ignoring the ignored tests).

Results: https://gist.github.com/anonymous/8e236848a89af440d3c56569e81f4829

Mr. Newson is looking at this failure right now and says we may be able
to improve upon the testing methodology.

-Joan

----- Original Message -----
From: "Paul Davis" <pa...@gmail.com>
To: dev@couchdb.apache.org
Sent: Tuesday, July 12, 2016 4:37:15 PM
Subject: Re: 2.0 & Windows: status update

The logs posted at [1] show that we're seeing OS processes die with an
exit code of 4. The most likely place I can find that that comes from
is couchspawnkillable_win.c [2] which is nicely Windows specific so
would do a lot to explain why we don't see it on *nix systems.
Unfortunately other than pointing out that the subprocess creation
seems to fail I don't have any idea or suggestion on how to debug
further.

[1] https://gist.github.com/anonymous/f2a94234195f007c3049e27d942482c1
[2] https://github.com/apache/couchdb-couch/blob/master/priv/spawnkillable/couchspawnkillable_win.c#L106

On Wed, Jun 29, 2016 at 4:21 PM, Sebastian Rothbucher
<se...@googlemail.com> wrote:
> Hi all,
>
> just to follow up on that: there is another PR coming up (
> https://github.com/apache/couchdb/pull/427) that tests for some more fixes
> and brings even more stability. In the meantime, deleting dev/lib is indeed
> the best way to produce reliable results. So is switching between
> auth-tests-wip and master. But there's progress => it might all end up
> nicely on master.
>
> Good luck, thanks & best
>     Sebastian
>
>
> On Mon, Jun 20, 2016 at 5:26 PM, Nick North <no...@gmail.com> wrote:
>
>> Thanks Sebastian. I'm looking at eunit at the moment, but hope to come back
>> to these.
>>
>> Nick
>>
>> On Sun, 19 Jun 2016 at 23:01 Sebastian Rothbucher <
>> sebastianrothbucher@googlemail.com> wrote:
>>
>> > Hi Joan, Nick,
>> >
>> > the following gist provides a current run of the test against the latest
>> > master of CouchDB - and the latest tests (from the auth-tests-wip
>> branch):
>> >
>> >
>> https://gist.github.com/sebastianrothbucher/efa3a992bd4de9996b4125da82a7e0de
>> > Maybe you can use them
>> >
>> > Here's what I did to get both latest tests and latest code:
>> > git checkout master
>> > ./configure -c --disable-docs --disable-fauxton
>> > make clean
>> > make
>> > git checkout auth-tests-wip
>> >
>> > Currently, make javascript seems not optimal as one tests (needs
>> > investigation) seems to mess up the setup for all that's following.
>> Hence,
>> > I took this drastic measure to produce the logs:
>> >
>> > for t in test/javascript/tests/*.js; do rm -rf dev/lib; dev/run -n 1 -q
>> > --with-admin-party-please test/javascript/run $t 2>&1 | tee -a
>> jstest2.log;
>> > done
>> >
>> > Maybe it makes sense for you to start w/ something similar to produce
>> some
>> > meaningful results.
>> >
>> > Best
>> >    Sebastian
>> >
>> > On Sun, Jun 19, 2016 at 6:04 PM, Nick North <no...@gmail.com> wrote:
>> >
>> > > Sorry - I meant a single node cluster in that last message. And I meant
>> > to
>> > > sign my name correctly.
>> > >
>> > > Nick
>> > >
>> > > On Sun, 19 Jun 2016 at 16:56 Nick North <no...@gmail.com> wrote:
>> > >
>> > > > I'm trying these tests now, and find that there are still a lot of JS
>> > > > failures with a single cluster. Many of them look suspiciously
>> similar
>> > at
>> > > > an initial glance, but I hope to look in more detail tomorrow.
>> > > >
>> > > > Nicj
>> > > >
>> > > > On Sun, 19 Jun 2016 at 15:20 Jan Lehnardt <ja...@apache.org> wrote:
>> > > >
>> > > >>
>> > > >> > On 17 Jun 2016, at 22:48, Joan Touzet <wo...@apache.org> wrote:
>> > > >> >
>> > > >> > Hello everyone,
>> > > >> >
>> > > >> > I'd like to update the community on the status of the 2.0 port to
>> > > >> Microsoft Windows. There are three parts to this email: the build
>> > > >> tools/chain themselves, support in CouchDB for the Windows build
>> > > process,
>> > > >> and testing results. I'll cover them in that order.
>> > > >> >
>> > > >> > -Joan
>> > > >> >
>> > > >> > Build Tools/Chain
>> > > >> > =================
>> > > >> > ** TL;DR: New glazier repo to join couchdb, contains scripts and
>> > > README
>> > > >> to build CouchDB 2.0 on Windows.
>> > > >> >
>> > > >> > Our work to date has been going on in Dave Cottlehuber's glazier
>> > > >> repository at
>> > > >> >
>> > > >> >  https://github.com/dch/glazier/tree/release/couchdb_2.0
>> > > >> >
>> > > >> > The reason for the extra repository is that the Windows build
>> > process
>> > > >> is *very* ugly, involving 3 distinct build chains (Visual Studio,
>> > Cygwin
>> > > >> and the Mozilla Build system) to build all of the necessary
>> > > prerequisites.
>> > > >> The repository includes a number of support scripts to set up that
>> > > >> environment, a README with a detailed walkthrough, and some patches
>> > > >> necessary to the prerequisites to get them to build under the modern
>> > > >> Windows b uild system.
>> > > >> >
>> > > >> > Parenthetically, it _is_ possible to use binary installs for the
>> > > >> prerequisites (OpenSSL, libcurl, Erlang, SM 1.8.5), but Dave, Nick
>> > North
>> > > >> and I have evolved the glazier system over a number of years and
>> it's
>> > > >> proven quite effective. Plus, we don't have to worry about the
>> > > provenance
>> > > >> of any of the binaries since we build everything from source
>> directly,
>> > > and
>> > > >> that's important when we put up an unofficial Windows build for
>> > > download at
>> > > >> https://couchdb.apache.org/ .
>> > > >> >
>> > > >> > Good news: as of today I've requested and Infra has created a new
>> > > >> apache couchdb-glazier repo, and it's my intent to mirror
>> dch/glazier
>> > > over
>> > > >> into the ASF's repo once things have stabilized a bit more (PR and
>> > > merge of
>> > > >> the release/couchdb_2.0 branch, and pending progress on steps 2 and
>> 3
>> > > >> below). Dave and I did an audit of the repository as it stands, and
>> > > since
>> > > >> all checkins come from CouchDB contributors already, we are good to
>> go
>> > > from
>> > > >> a licensing perspective.
>> > > >> >
>> > > >> >
>> > > >> > Overall CouchDB Windows support
>> > > >> > ===============================
>> > > >> > ** TL;DR: Windows support in 2.0 a priority, conversion of
>> top-level
>> > > >> Makefile in progress.
>> > > >> >
>> > > >> > There are two aspects to native CouchDB Windows support. The first
>> > is
>> > > >> anything within the CouchDB code itself that assumes a Unix-like
>> > > >> environment. Fortunately, most of these problems have been worked
>> out
>> > in
>> > > >> prior releases. I'm not aware of any outstanding issues here (except
>> > one
>> > > >> point below under test results).
>> > > >> >
>> > > >> > The other aspect is the build setup within the couchdb repo
>> itself.
>> > > >> I've already converted the bash configure script into a PowerShell
>> > > >> configure script that works fine. However, the Makefile has bashisms
>> > in
>> > > it
>> > > >> and assumes GNU Make. I've started a conversion of this into Windows
>> > > NMake
>> > > >> format and will submit a PR for a Makefile.win in due course.
>> > > >> >
>> > > >> > I want to answer two frequent questions we get here before they
>> get
>> > > >> re-asked:
>> > > >> >
>> > > >> >  1) Why not use a cygwin environment to retain compatibility with
>> > the
>> > > >> Unix build process? The answer is that performance suffers, the
>> build
>> > > chain
>> > > >> is onerous, there are link-time problems when trying to link against
>> > > things
>> > > >> built using Visual Studio, and there are still assumptions on paths
>> > that
>> > > >> don't work out. We can't get away from making Windows-specific
>> > > >> customizations to the build process anyway, so we might as well take
>> > the
>> > > >> extra step and support the build process properly. It's not THAT
>> much
>> > > work
>> > > >> to convert the Makefile and configure script, and our top-level
>> > Makefile
>> > > >> really isn't much more than a shell script anyway (every target is a
>> > > .PHONY
>> > > >> target!). In fact, a TODO for an enterprising developer might be to
>> > > rewrite
>> > > >> our top-level Makefile/Makefile.win as a Python script that "does
>> the
>> > > right
>> > > >> thing" on both platforms, the same way our dev/run script works
>> today.
>> > > >> >
>> > > >> >  2) Why not use the new "Bash and Ubuntu on Windows" functionality
>> > > >> Microsoft has announced for Windows 10? There are two distinct
>> > problems
>> > > >> here. The first is that there is a very large install base still of
>> > > Windows
>> > > >> 7 and 8 (and Windows Server) machines that cannot run this
>> subsystem.
>> > > The
>> > > >> second is that Microsoft themselves say this about the
>> functionality:
>> > > >> >
>> > > >> >     "Second, while you’ll be able to run native Bash and many
>> Linux
>> > > >> command-line tools on Windows, it’s important to note that this is a
>> > > >> developer toolset to help you write and build all your code for all
>> > your
>> > > >> scenarios and platforms. This is not a server platform upon which
>> you
>> > > will
>> > > >> host websites, run server infrastructure, etc."
>> > > >> >
>> > > >> > Given this strong warning from Microsoft themselves (which hints
>> at
>> > > >> performance consideratings), and the fact that download statistics
>> > show
>> > > an
>> > > >> equal number of downloads of the CouchDB .tar source and the Windows
>> > > .zip
>> > > >> installer from our couchdb.apache.org website, we need to consider
>> > that
>> > > >> people are running CouchDB on Windows not just as a developer tool
>> but
>> > > as a
>> > > >> fully-fledged server. As such it behooves us to build it "properly"
>> > as a
>> > > >> normal Windows binary/service.
>> > > >> >
>> > > >>
>> > > >> Great progress Joan! Thank you! :)
>> > > >>
>> > > >> > Test Results
>> > > >> > ============
>> > > >> > ** TL;DR: Lots of things are failing. Joan needs help interpreting
>> > the
>> > > >> results or she will go around the bend.
>> > > >> >
>> > > >> > Here are the current test results in gist form.
>> > > >> >
>> > > >> > EUnit:
>> > > >> https://gist.github.com/anonymous/3203ed27c60cf3da4f0f0d5bff731722
>> > > >> >
>> > > >> > JS tests:
>> > > >> https://gist.github.com/anonymous/93b0b70ed445ca4043a63140f8d219bf
>> > > >> >
>> > > >> > For the EUnit tests, everything other than os_process stuff seems
>> to
>> > > be
>> > > >> working. Honestly, I think we can release without os_process support
>> > on
>> > > >> Windows, though I should file a bug to track this. I am actually
>> > > inclined
>> > > >> to disable os_process support on Windows and the related eunit tests
>> > > rather
>> > > >> than fix this rarely-needed functionality, unless someone on this
>> list
>> > > >> objects.
>> > > >>
>> > > >> You are probably thinking about CouchDB Externals, which definitely
>> > use
>> > > >> os_process functionality and which I’d also be fine with dropping
>> > > support
>> > > >> for Windows for now, but os_process is also used by the view server,
>> > so
>> > > we
>> > > >> should definitely get them passing.
>> > > >>
>> > > >>
>> > > >> > For the JS tests, a *lot* is failing. I need to know how much this
>> > > >> differs from a Linux/OSX baseline today, can anyone help me follow
>> up
>> > > here?
>> > > >>
>> > > >> Can you try running these against a -n 1 cluster? We are not set up
>> to
>> > > >> run JS tests against more nodes at this point.
>> > > >>
>> > > >> On master/unix most if not all JS tests should either pass or
>> skipped
>> > > >> with a TODO message.
>> > > >>
>> > > >>
>> > > >> Best
>> > > >> Jan
>> > > >> --
>> > > >>
>> > > >>
>> > > >>
>> > >
>> >
>>

Re: 2.0 & Windows: status update

[Joan Touzet <wo...@apache.org>]

Thanks, Paul. I'm starting to look at this today.

In better news, current Windows JS tests now match the *nix JS test results.
Only one test, replication.js, is failing (ignoring the ignored tests).

Results: https://gist.github.com/anonymous/8e236848a89af440d3c56569e81f4829

Mr. Newson is looking at this failure right now and says we may be able
to improve upon the testing methodology.

-Joan

----- Original Message -----
From: "Paul Davis" <pa...@gmail.com>
To: dev@couchdb.apache.org
Sent: Tuesday, July 12, 2016 4:37:15 PM
Subject: Re: 2.0 & Windows: status update

The logs posted at [1] show that we're seeing OS processes die with an
exit code of 4. The most likely place I can find that that comes from
is couchspawnkillable_win.c [2] which is nicely Windows specific so
would do a lot to explain why we don't see it on *nix systems.
Unfortunately other than pointing out that the subprocess creation
seems to fail I don't have any idea or suggestion on how to debug
further.

[1] https://gist.github.com/anonymous/f2a94234195f007c3049e27d942482c1
[2] https://github.com/apache/couchdb-couch/blob/master/priv/spawnkillable/couchspawnkillable_win.c#L106

On Wed, Jun 29, 2016 at 4:21 PM, Sebastian Rothbucher
<se...@googlemail.com> wrote:
> Hi all,
>
> just to follow up on that: there is another PR coming up (
> https://github.com/apache/couchdb/pull/427) that tests for some more fixes
> and brings even more stability. In the meantime, deleting dev/lib is indeed
> the best way to produce reliable results. So is switching between
> auth-tests-wip and master. But there's progress => it might all end up
> nicely on master.
>
> Good luck, thanks & best
>     Sebastian
>
>
> On Mon, Jun 20, 2016 at 5:26 PM, Nick North <no...@gmail.com> wrote:
>
>> Thanks Sebastian. I'm looking at eunit at the moment, but hope to come back
>> to these.
>>
>> Nick
>>
>> On Sun, 19 Jun 2016 at 23:01 Sebastian Rothbucher <
>> sebastianrothbucher@googlemail.com> wrote:
>>
>> > Hi Joan, Nick,
>> >
>> > the following gist provides a current run of the test against the latest
>> > master of CouchDB - and the latest tests (from the auth-tests-wip
>> branch):
>> >
>> >
>> https://gist.github.com/sebastianrothbucher/efa3a992bd4de9996b4125da82a7e0de
>> > Maybe you can use them
>> >
>> > Here's what I did to get both latest tests and latest code:
>> > git checkout master
>> > ./configure -c --disable-docs --disable-fauxton
>> > make clean
>> > make
>> > git checkout auth-tests-wip
>> >
>> > Currently, make javascript seems not optimal as one tests (needs
>> > investigation) seems to mess up the setup for all that's following.
>> Hence,
>> > I took this drastic measure to produce the logs:
>> >
>> > for t in test/javascript/tests/*.js; do rm -rf dev/lib; dev/run -n 1 -q
>> > --with-admin-party-please test/javascript/run $t 2>&1 | tee -a
>> jstest2.log;
>> > done
>> >
>> > Maybe it makes sense for you to start w/ something similar to produce
>> some
>> > meaningful results.
>> >
>> > Best
>> >    Sebastian
>> >
>> > On Sun, Jun 19, 2016 at 6:04 PM, Nick North <no...@gmail.com> wrote:
>> >
>> > > Sorry - I meant a single node cluster in that last message. And I meant
>> > to
>> > > sign my name correctly.
>> > >
>> > > Nick
>> > >
>> > > On Sun, 19 Jun 2016 at 16:56 Nick North <no...@gmail.com> wrote:
>> > >
>> > > > I'm trying these tests now, and find that there are still a lot of JS
>> > > > failures with a single cluster. Many of them look suspiciously
>> similar
>> > at
>> > > > an initial glance, but I hope to look in more detail tomorrow.
>> > > >
>> > > > Nicj
>> > > >
>> > > > On Sun, 19 Jun 2016 at 15:20 Jan Lehnardt <ja...@apache.org> wrote:
>> > > >
>> > > >>
>> > > >> > On 17 Jun 2016, at 22:48, Joan Touzet <wo...@apache.org> wrote:
>> > > >> >
>> > > >> > Hello everyone,
>> > > >> >
>> > > >> > I'd like to update the community on the status of the 2.0 port to
>> > > >> Microsoft Windows. There are three parts to this email: the build
>> > > >> tools/chain themselves, support in CouchDB for the Windows build
>> > > process,
>> > > >> and testing results. I'll cover them in that order.
>> > > >> >
>> > > >> > -Joan
>> > > >> >
>> > > >> > Build Tools/Chain
>> > > >> > =================
>> > > >> > ** TL;DR: New glazier repo to join couchdb, contains scripts and
>> > > README
>> > > >> to build CouchDB 2.0 on Windows.
>> > > >> >
>> > > >> > Our work to date has been going on in Dave Cottlehuber's glazier
>> > > >> repository at
>> > > >> >
>> > > >> >  https://github.com/dch/glazier/tree/release/couchdb_2.0
>> > > >> >
>> > > >> > The reason for the extra repository is that the Windows build
>> > process
>> > > >> is *very* ugly, involving 3 distinct build chains (Visual Studio,
>> > Cygwin
>> > > >> and the Mozilla Build system) to build all of the necessary
>> > > prerequisites.
>> > > >> The repository includes a number of support scripts to set up that
>> > > >> environment, a README with a detailed walkthrough, and some patches
>> > > >> necessary to the prerequisites to get them to build under the modern
>> > > >> Windows b uild system.
>> > > >> >
>> > > >> > Parenthetically, it _is_ possible to use binary installs for the
>> > > >> prerequisites (OpenSSL, libcurl, Erlang, SM 1.8.5), but Dave, Nick
>> > North
>> > > >> and I have evolved the glazier system over a number of years and
>> it's
>> > > >> proven quite effective. Plus, we don't have to worry about the
>> > > provenance
>> > > >> of any of the binaries since we build everything from source
>> directly,
>> > > and
>> > > >> that's important when we put up an unofficial Windows build for
>> > > download at
>> > > >> https://couchdb.apache.org/ .
>> > > >> >
>> > > >> > Good news: as of today I've requested and Infra has created a new
>> > > >> apache couchdb-glazier repo, and it's my intent to mirror
>> dch/glazier
>> > > over
>> > > >> into the ASF's repo once things have stabilized a bit more (PR and
>> > > merge of
>> > > >> the release/couchdb_2.0 branch, and pending progress on steps 2 and
>> 3
>> > > >> below). Dave and I did an audit of the repository as it stands, and
>> > > since
>> > > >> all checkins come from CouchDB contributors already, we are good to
>> go
>> > > from
>> > > >> a licensing perspective.
>> > > >> >
>> > > >> >
>> > > >> > Overall CouchDB Windows support
>> > > >> > ===============================
>> > > >> > ** TL;DR: Windows support in 2.0 a priority, conversion of
>> top-level
>> > > >> Makefile in progress.
>> > > >> >
>> > > >> > There are two aspects to native CouchDB Windows support. The first
>> > is
>> > > >> anything within the CouchDB code itself that assumes a Unix-like
>> > > >> environment. Fortunately, most of these problems have been worked
>> out
>> > in
>> > > >> prior releases. I'm not aware of any outstanding issues here (except
>> > one
>> > > >> point below under test results).
>> > > >> >
>> > > >> > The other aspect is the build setup within the couchdb repo
>> itself.
>> > > >> I've already converted the bash configure script into a PowerShell
>> > > >> configure script that works fine. However, the Makefile has bashisms
>> > in
>> > > it
>> > > >> and assumes GNU Make. I've started a conversion of this into Windows
>> > > NMake
>> > > >> format and will submit a PR for a Makefile.win in due course.
>> > > >> >
>> > > >> > I want to answer two frequent questions we get here before they
>> get
>> > > >> re-asked:
>> > > >> >
>> > > >> >  1) Why not use a cygwin environment to retain compatibility with
>> > the
>> > > >> Unix build process? The answer is that performance suffers, the
>> build
>> > > chain
>> > > >> is onerous, there are link-time problems when trying to link against
>> > > things
>> > > >> built using Visual Studio, and there are still assumptions on paths
>> > that
>> > > >> don't work out. We can't get away from making Windows-specific
>> > > >> customizations to the build process anyway, so we might as well take
>> > the
>> > > >> extra step and support the build process properly. It's not THAT
>> much
>> > > work
>> > > >> to convert the Makefile and configure script, and our top-level
>> > Makefile
>> > > >> really isn't much more than a shell script anyway (every target is a
>> > > .PHONY
>> > > >> target!). In fact, a TODO for an enterprising developer might be to
>> > > rewrite
>> > > >> our top-level Makefile/Makefile.win as a Python script that "does
>> the
>> > > right
>> > > >> thing" on both platforms, the same way our dev/run script works
>> today.
>> > > >> >
>> > > >> >  2) Why not use the new "Bash and Ubuntu on Windows" functionality
>> > > >> Microsoft has announced for Windows 10? There are two distinct
>> > problems
>> > > >> here. The first is that there is a very large install base still of
>> > > Windows
>> > > >> 7 and 8 (and Windows Server) machines that cannot run this
>> subsystem.
>> > > The
>> > > >> second is that Microsoft themselves say this about the
>> functionality:
>> > > >> >
>> > > >> >     "Second, while you’ll be able to run native Bash and many
>> Linux
>> > > >> command-line tools on Windows, it’s important to note that this is a
>> > > >> developer toolset to help you write and build all your code for all
>> > your
>> > > >> scenarios and platforms. This is not a server platform upon which
>> you
>> > > will
>> > > >> host websites, run server infrastructure, etc."
>> > > >> >
>> > > >> > Given this strong warning from Microsoft themselves (which hints
>> at
>> > > >> performance consideratings), and the fact that download statistics
>> > show
>> > > an
>> > > >> equal number of downloads of the CouchDB .tar source and the Windows
>> > > .zip
>> > > >> installer from our couchdb.apache.org website, we need to consider
>> > that
>> > > >> people are running CouchDB on Windows not just as a developer tool
>> but
>> > > as a
>> > > >> fully-fledged server. As such it behooves us to build it "properly"
>> > as a
>> > > >> normal Windows binary/service.
>> > > >> >
>> > > >>
>> > > >> Great progress Joan! Thank you! :)
>> > > >>
>> > > >> > Test Results
>> > > >> > ============
>> > > >> > ** TL;DR: Lots of things are failing. Joan needs help interpreting
>> > the
>> > > >> results or she will go around the bend.
>> > > >> >
>> > > >> > Here are the current test results in gist form.
>> > > >> >
>> > > >> > EUnit:
>> > > >> https://gist.github.com/anonymous/3203ed27c60cf3da4f0f0d5bff731722
>> > > >> >
>> > > >> > JS tests:
>> > > >> https://gist.github.com/anonymous/93b0b70ed445ca4043a63140f8d219bf
>> > > >> >
>> > > >> > For the EUnit tests, everything other than os_process stuff seems
>> to
>> > > be
>> > > >> working. Honestly, I think we can release without os_process support
>> > on
>> > > >> Windows, though I should file a bug to track this. I am actually
>> > > inclined
>> > > >> to disable os_process support on Windows and the related eunit tests
>> > > rather
>> > > >> than fix this rarely-needed functionality, unless someone on this
>> list
>> > > >> objects.
>> > > >>
>> > > >> You are probably thinking about CouchDB Externals, which definitely
>> > use
>> > > >> os_process functionality and which I’d also be fine with dropping
>> > > support
>> > > >> for Windows for now, but os_process is also used by the view server,
>> > so
>> > > we
>> > > >> should definitely get them passing.
>> > > >>
>> > > >>
>> > > >> > For the JS tests, a *lot* is failing. I need to know how much this
>> > > >> differs from a Linux/OSX baseline today, can anyone help me follow
>> up
>> > > here?
>> > > >>
>> > > >> Can you try running these against a -n 1 cluster? We are not set up
>> to
>> > > >> run JS tests against more nodes at this point.
>> > > >>
>> > > >> On master/unix most if not all JS tests should either pass or
>> skipped
>> > > >> with a TODO message.
>> > > >>
>> > > >>
>> > > >> Best
>> > > >> Jan
>> > > >> --
>> > > >>
>> > > >>
>> > > >>
>> > >
>> >
>>

Re: 2.0 & Windows: status update

[Paul Davis <pa...@gmail.com>]

The logs posted at [1] show that we're seeing OS processes die with an
exit code of 4. The most likely place I can find that that comes from
is couchspawnkillable_win.c [2] which is nicely Windows specific so
would do a lot to explain why we don't see it on *nix systems.
Unfortunately other than pointing out that the subprocess creation
seems to fail I don't have any idea or suggestion on how to debug
further.

[1] https://gist.github.com/anonymous/f2a94234195f007c3049e27d942482c1
[2] https://github.com/apache/couchdb-couch/blob/master/priv/spawnkillable/couchspawnkillable_win.c#L106

On Wed, Jun 29, 2016 at 4:21 PM, Sebastian Rothbucher
<se...@googlemail.com> wrote:
> Hi all,
>
> just to follow up on that: there is another PR coming up (
> https://github.com/apache/couchdb/pull/427) that tests for some more fixes
> and brings even more stability. In the meantime, deleting dev/lib is indeed
> the best way to produce reliable results. So is switching between
> auth-tests-wip and master. But there's progress => it might all end up
> nicely on master.
>
> Good luck, thanks & best
>     Sebastian
>
>
> On Mon, Jun 20, 2016 at 5:26 PM, Nick North <no...@gmail.com> wrote:
>
>> Thanks Sebastian. I'm looking at eunit at the moment, but hope to come back
>> to these.
>>
>> Nick
>>
>> On Sun, 19 Jun 2016 at 23:01 Sebastian Rothbucher <
>> sebastianrothbucher@googlemail.com> wrote:
>>
>> > Hi Joan, Nick,
>> >
>> > the following gist provides a current run of the test against the latest
>> > master of CouchDB - and the latest tests (from the auth-tests-wip
>> branch):
>> >
>> >
>> https://gist.github.com/sebastianrothbucher/efa3a992bd4de9996b4125da82a7e0de
>> > Maybe you can use them
>> >
>> > Here's what I did to get both latest tests and latest code:
>> > git checkout master
>> > ./configure -c --disable-docs --disable-fauxton
>> > make clean
>> > make
>> > git checkout auth-tests-wip
>> >
>> > Currently, make javascript seems not optimal as one tests (needs
>> > investigation) seems to mess up the setup for all that's following.
>> Hence,
>> > I took this drastic measure to produce the logs:
>> >
>> > for t in test/javascript/tests/*.js; do rm -rf dev/lib; dev/run -n 1 -q
>> > --with-admin-party-please test/javascript/run $t 2>&1 | tee -a
>> jstest2.log;
>> > done
>> >
>> > Maybe it makes sense for you to start w/ something similar to produce
>> some
>> > meaningful results.
>> >
>> > Best
>> >    Sebastian
>> >
>> > On Sun, Jun 19, 2016 at 6:04 PM, Nick North <no...@gmail.com> wrote:
>> >
>> > > Sorry - I meant a single node cluster in that last message. And I meant
>> > to
>> > > sign my name correctly.
>> > >
>> > > Nick
>> > >
>> > > On Sun, 19 Jun 2016 at 16:56 Nick North <no...@gmail.com> wrote:
>> > >
>> > > > I'm trying these tests now, and find that there are still a lot of JS
>> > > > failures with a single cluster. Many of them look suspiciously
>> similar
>> > at
>> > > > an initial glance, but I hope to look in more detail tomorrow.
>> > > >
>> > > > Nicj
>> > > >
>> > > > On Sun, 19 Jun 2016 at 15:20 Jan Lehnardt <ja...@apache.org> wrote:
>> > > >
>> > > >>
>> > > >> > On 17 Jun 2016, at 22:48, Joan Touzet <wo...@apache.org> wrote:
>> > > >> >
>> > > >> > Hello everyone,
>> > > >> >
>> > > >> > I'd like to update the community on the status of the 2.0 port to
>> > > >> Microsoft Windows. There are three parts to this email: the build
>> > > >> tools/chain themselves, support in CouchDB for the Windows build
>> > > process,
>> > > >> and testing results. I'll cover them in that order.
>> > > >> >
>> > > >> > -Joan
>> > > >> >
>> > > >> > Build Tools/Chain
>> > > >> > =================
>> > > >> > ** TL;DR: New glazier repo to join couchdb, contains scripts and
>> > > README
>> > > >> to build CouchDB 2.0 on Windows.
>> > > >> >
>> > > >> > Our work to date has been going on in Dave Cottlehuber's glazier
>> > > >> repository at
>> > > >> >
>> > > >> >  https://github.com/dch/glazier/tree/release/couchdb_2.0
>> > > >> >
>> > > >> > The reason for the extra repository is that the Windows build
>> > process
>> > > >> is *very* ugly, involving 3 distinct build chains (Visual Studio,
>> > Cygwin
>> > > >> and the Mozilla Build system) to build all of the necessary
>> > > prerequisites.
>> > > >> The repository includes a number of support scripts to set up that
>> > > >> environment, a README with a detailed walkthrough, and some patches
>> > > >> necessary to the prerequisites to get them to build under the modern
>> > > >> Windows b uild system.
>> > > >> >
>> > > >> > Parenthetically, it _is_ possible to use binary installs for the
>> > > >> prerequisites (OpenSSL, libcurl, Erlang, SM 1.8.5), but Dave, Nick
>> > North
>> > > >> and I have evolved the glazier system over a number of years and
>> it's
>> > > >> proven quite effective. Plus, we don't have to worry about the
>> > > provenance
>> > > >> of any of the binaries since we build everything from source
>> directly,
>> > > and
>> > > >> that's important when we put up an unofficial Windows build for
>> > > download at
>> > > >> https://couchdb.apache.org/ .
>> > > >> >
>> > > >> > Good news: as of today I've requested and Infra has created a new
>> > > >> apache couchdb-glazier repo, and it's my intent to mirror
>> dch/glazier
>> > > over
>> > > >> into the ASF's repo once things have stabilized a bit more (PR and
>> > > merge of
>> > > >> the release/couchdb_2.0 branch, and pending progress on steps 2 and
>> 3
>> > > >> below). Dave and I did an audit of the repository as it stands, and
>> > > since
>> > > >> all checkins come from CouchDB contributors already, we are good to
>> go
>> > > from
>> > > >> a licensing perspective.
>> > > >> >
>> > > >> >
>> > > >> > Overall CouchDB Windows support
>> > > >> > ===============================
>> > > >> > ** TL;DR: Windows support in 2.0 a priority, conversion of
>> top-level
>> > > >> Makefile in progress.
>> > > >> >
>> > > >> > There are two aspects to native CouchDB Windows support. The first
>> > is
>> > > >> anything within the CouchDB code itself that assumes a Unix-like
>> > > >> environment. Fortunately, most of these problems have been worked
>> out
>> > in
>> > > >> prior releases. I'm not aware of any outstanding issues here (except
>> > one
>> > > >> point below under test results).
>> > > >> >
>> > > >> > The other aspect is the build setup within the couchdb repo
>> itself.
>> > > >> I've already converted the bash configure script into a PowerShell
>> > > >> configure script that works fine. However, the Makefile has bashisms
>> > in
>> > > it
>> > > >> and assumes GNU Make. I've started a conversion of this into Windows
>> > > NMake
>> > > >> format and will submit a PR for a Makefile.win in due course.
>> > > >> >
>> > > >> > I want to answer two frequent questions we get here before they
>> get
>> > > >> re-asked:
>> > > >> >
>> > > >> >  1) Why not use a cygwin environment to retain compatibility with
>> > the
>> > > >> Unix build process? The answer is that performance suffers, the
>> build
>> > > chain
>> > > >> is onerous, there are link-time problems when trying to link against
>> > > things
>> > > >> built using Visual Studio, and there are still assumptions on paths
>> > that
>> > > >> don't work out. We can't get away from making Windows-specific
>> > > >> customizations to the build process anyway, so we might as well take
>> > the
>> > > >> extra step and support the build process properly. It's not THAT
>> much
>> > > work
>> > > >> to convert the Makefile and configure script, and our top-level
>> > Makefile
>> > > >> really isn't much more than a shell script anyway (every target is a
>> > > .PHONY
>> > > >> target!). In fact, a TODO for an enterprising developer might be to
>> > > rewrite
>> > > >> our top-level Makefile/Makefile.win as a Python script that "does
>> the
>> > > right
>> > > >> thing" on both platforms, the same way our dev/run script works
>> today.
>> > > >> >
>> > > >> >  2) Why not use the new "Bash and Ubuntu on Windows" functionality
>> > > >> Microsoft has announced for Windows 10? There are two distinct
>> > problems
>> > > >> here. The first is that there is a very large install base still of
>> > > Windows
>> > > >> 7 and 8 (and Windows Server) machines that cannot run this
>> subsystem.
>> > > The
>> > > >> second is that Microsoft themselves say this about the
>> functionality:
>> > > >> >
>> > > >> >     "Second, while you’ll be able to run native Bash and many
>> Linux
>> > > >> command-line tools on Windows, it’s important to note that this is a
>> > > >> developer toolset to help you write and build all your code for all
>> > your
>> > > >> scenarios and platforms. This is not a server platform upon which
>> you
>> > > will
>> > > >> host websites, run server infrastructure, etc."
>> > > >> >
>> > > >> > Given this strong warning from Microsoft themselves (which hints
>> at
>> > > >> performance consideratings), and the fact that download statistics
>> > show
>> > > an
>> > > >> equal number of downloads of the CouchDB .tar source and the Windows
>> > > .zip
>> > > >> installer from our couchdb.apache.org website, we need to consider
>> > that
>> > > >> people are running CouchDB on Windows not just as a developer tool
>> but
>> > > as a
>> > > >> fully-fledged server. As such it behooves us to build it "properly"
>> > as a
>> > > >> normal Windows binary/service.
>> > > >> >
>> > > >>
>> > > >> Great progress Joan! Thank you! :)
>> > > >>
>> > > >> > Test Results
>> > > >> > ============
>> > > >> > ** TL;DR: Lots of things are failing. Joan needs help interpreting
>> > the
>> > > >> results or she will go around the bend.
>> > > >> >
>> > > >> > Here are the current test results in gist form.
>> > > >> >
>> > > >> > EUnit:
>> > > >> https://gist.github.com/anonymous/3203ed27c60cf3da4f0f0d5bff731722
>> > > >> >
>> > > >> > JS tests:
>> > > >> https://gist.github.com/anonymous/93b0b70ed445ca4043a63140f8d219bf
>> > > >> >
>> > > >> > For the EUnit tests, everything other than os_process stuff seems
>> to
>> > > be
>> > > >> working. Honestly, I think we can release without os_process support
>> > on
>> > > >> Windows, though I should file a bug to track this. I am actually
>> > > inclined
>> > > >> to disable os_process support on Windows and the related eunit tests
>> > > rather
>> > > >> than fix this rarely-needed functionality, unless someone on this
>> list
>> > > >> objects.
>> > > >>
>> > > >> You are probably thinking about CouchDB Externals, which definitely
>> > use
>> > > >> os_process functionality and which I’d also be fine with dropping
>> > > support
>> > > >> for Windows for now, but os_process is also used by the view server,
>> > so
>> > > we
>> > > >> should definitely get them passing.
>> > > >>
>> > > >>
>> > > >> > For the JS tests, a *lot* is failing. I need to know how much this
>> > > >> differs from a Linux/OSX baseline today, can anyone help me follow
>> up
>> > > here?
>> > > >>
>> > > >> Can you try running these against a -n 1 cluster? We are not set up
>> to
>> > > >> run JS tests against more nodes at this point.
>> > > >>
>> > > >> On master/unix most if not all JS tests should either pass or
>> skipped
>> > > >> with a TODO message.
>> > > >>
>> > > >>
>> > > >> Best
>> > > >> Jan
>> > > >> --
>> > > >>
>> > > >>
>> > > >>
>> > >
>> >
>>

Re: 2.0 & Windows: status update

[Sebastian Rothbucher <se...@googlemail.com>]

Hi all,

just to follow up on that: there is another PR coming up (
https://github.com/apache/couchdb/pull/427) that tests for some more fixes
and brings even more stability. In the meantime, deleting dev/lib is indeed
the best way to produce reliable results. So is switching between
auth-tests-wip and master. But there's progress => it might all end up
nicely on master.

Good luck, thanks & best
    Sebastian


On Mon, Jun 20, 2016 at 5:26 PM, Nick North <no...@gmail.com> wrote:

> Thanks Sebastian. I'm looking at eunit at the moment, but hope to come back
> to these.
>
> Nick
>
> On Sun, 19 Jun 2016 at 23:01 Sebastian Rothbucher <
> sebastianrothbucher@googlemail.com> wrote:
>
> > Hi Joan, Nick,
> >
> > the following gist provides a current run of the test against the latest
> > master of CouchDB - and the latest tests (from the auth-tests-wip
> branch):
> >
> >
> https://gist.github.com/sebastianrothbucher/efa3a992bd4de9996b4125da82a7e0de
> > Maybe you can use them
> >
> > Here's what I did to get both latest tests and latest code:
> > git checkout master
> > ./configure -c --disable-docs --disable-fauxton
> > make clean
> > make
> > git checkout auth-tests-wip
> >
> > Currently, make javascript seems not optimal as one tests (needs
> > investigation) seems to mess up the setup for all that's following.
> Hence,
> > I took this drastic measure to produce the logs:
> >
> > for t in test/javascript/tests/*.js; do rm -rf dev/lib; dev/run -n 1 -q
> > --with-admin-party-please test/javascript/run $t 2>&1 | tee -a
> jstest2.log;
> > done
> >
> > Maybe it makes sense for you to start w/ something similar to produce
> some
> > meaningful results.
> >
> > Best
> >    Sebastian
> >
> > On Sun, Jun 19, 2016 at 6:04 PM, Nick North <no...@gmail.com> wrote:
> >
> > > Sorry - I meant a single node cluster in that last message. And I meant
> > to
> > > sign my name correctly.
> > >
> > > Nick
> > >
> > > On Sun, 19 Jun 2016 at 16:56 Nick North <no...@gmail.com> wrote:
> > >
> > > > I'm trying these tests now, and find that there are still a lot of JS
> > > > failures with a single cluster. Many of them look suspiciously
> similar
> > at
> > > > an initial glance, but I hope to look in more detail tomorrow.
> > > >
> > > > Nicj
> > > >
> > > > On Sun, 19 Jun 2016 at 15:20 Jan Lehnardt <ja...@apache.org> wrote:
> > > >
> > > >>
> > > >> > On 17 Jun 2016, at 22:48, Joan Touzet <wo...@apache.org> wrote:
> > > >> >
> > > >> > Hello everyone,
> > > >> >
> > > >> > I'd like to update the community on the status of the 2.0 port to
> > > >> Microsoft Windows. There are three parts to this email: the build
> > > >> tools/chain themselves, support in CouchDB for the Windows build
> > > process,
> > > >> and testing results. I'll cover them in that order.
> > > >> >
> > > >> > -Joan
> > > >> >
> > > >> > Build Tools/Chain
> > > >> > =================
> > > >> > ** TL;DR: New glazier repo to join couchdb, contains scripts and
> > > README
> > > >> to build CouchDB 2.0 on Windows.
> > > >> >
> > > >> > Our work to date has been going on in Dave Cottlehuber's glazier
> > > >> repository at
> > > >> >
> > > >> >  https://github.com/dch/glazier/tree/release/couchdb_2.0
> > > >> >
> > > >> > The reason for the extra repository is that the Windows build
> > process
> > > >> is *very* ugly, involving 3 distinct build chains (Visual Studio,
> > Cygwin
> > > >> and the Mozilla Build system) to build all of the necessary
> > > prerequisites.
> > > >> The repository includes a number of support scripts to set up that
> > > >> environment, a README with a detailed walkthrough, and some patches
> > > >> necessary to the prerequisites to get them to build under the modern
> > > >> Windows b uild system.
> > > >> >
> > > >> > Parenthetically, it _is_ possible to use binary installs for the
> > > >> prerequisites (OpenSSL, libcurl, Erlang, SM 1.8.5), but Dave, Nick
> > North
> > > >> and I have evolved the glazier system over a number of years and
> it's
> > > >> proven quite effective. Plus, we don't have to worry about the
> > > provenance
> > > >> of any of the binaries since we build everything from source
> directly,
> > > and
> > > >> that's important when we put up an unofficial Windows build for
> > > download at
> > > >> https://couchdb.apache.org/ .
> > > >> >
> > > >> > Good news: as of today I've requested and Infra has created a new
> > > >> apache couchdb-glazier repo, and it's my intent to mirror
> dch/glazier
> > > over
> > > >> into the ASF's repo once things have stabilized a bit more (PR and
> > > merge of
> > > >> the release/couchdb_2.0 branch, and pending progress on steps 2 and
> 3
> > > >> below). Dave and I did an audit of the repository as it stands, and
> > > since
> > > >> all checkins come from CouchDB contributors already, we are good to
> go
> > > from
> > > >> a licensing perspective.
> > > >> >
> > > >> >
> > > >> > Overall CouchDB Windows support
> > > >> > ===============================
> > > >> > ** TL;DR: Windows support in 2.0 a priority, conversion of
> top-level
> > > >> Makefile in progress.
> > > >> >
> > > >> > There are two aspects to native CouchDB Windows support. The first
> > is
> > > >> anything within the CouchDB code itself that assumes a Unix-like
> > > >> environment. Fortunately, most of these problems have been worked
> out
> > in
> > > >> prior releases. I'm not aware of any outstanding issues here (except
> > one
> > > >> point below under test results).
> > > >> >
> > > >> > The other aspect is the build setup within the couchdb repo
> itself.
> > > >> I've already converted the bash configure script into a PowerShell
> > > >> configure script that works fine. However, the Makefile has bashisms
> > in
> > > it
> > > >> and assumes GNU Make. I've started a conversion of this into Windows
> > > NMake
> > > >> format and will submit a PR for a Makefile.win in due course.
> > > >> >
> > > >> > I want to answer two frequent questions we get here before they
> get
> > > >> re-asked:
> > > >> >
> > > >> >  1) Why not use a cygwin environment to retain compatibility with
> > the
> > > >> Unix build process? The answer is that performance suffers, the
> build
> > > chain
> > > >> is onerous, there are link-time problems when trying to link against
> > > things
> > > >> built using Visual Studio, and there are still assumptions on paths
> > that
> > > >> don't work out. We can't get away from making Windows-specific
> > > >> customizations to the build process anyway, so we might as well take
> > the
> > > >> extra step and support the build process properly. It's not THAT
> much
> > > work
> > > >> to convert the Makefile and configure script, and our top-level
> > Makefile
> > > >> really isn't much more than a shell script anyway (every target is a
> > > .PHONY
> > > >> target!). In fact, a TODO for an enterprising developer might be to
> > > rewrite
> > > >> our top-level Makefile/Makefile.win as a Python script that "does
> the
> > > right
> > > >> thing" on both platforms, the same way our dev/run script works
> today.
> > > >> >
> > > >> >  2) Why not use the new "Bash and Ubuntu on Windows" functionality
> > > >> Microsoft has announced for Windows 10? There are two distinct
> > problems
> > > >> here. The first is that there is a very large install base still of
> > > Windows
> > > >> 7 and 8 (and Windows Server) machines that cannot run this
> subsystem.
> > > The
> > > >> second is that Microsoft themselves say this about the
> functionality:
> > > >> >
> > > >> >     "Second, while you’ll be able to run native Bash and many
> Linux
> > > >> command-line tools on Windows, it’s important to note that this is a
> > > >> developer toolset to help you write and build all your code for all
> > your
> > > >> scenarios and platforms. This is not a server platform upon which
> you
> > > will
> > > >> host websites, run server infrastructure, etc."
> > > >> >
> > > >> > Given this strong warning from Microsoft themselves (which hints
> at
> > > >> performance consideratings), and the fact that download statistics
> > show
> > > an
> > > >> equal number of downloads of the CouchDB .tar source and the Windows
> > > .zip
> > > >> installer from our couchdb.apache.org website, we need to consider
> > that
> > > >> people are running CouchDB on Windows not just as a developer tool
> but
> > > as a
> > > >> fully-fledged server. As such it behooves us to build it "properly"
> > as a
> > > >> normal Windows binary/service.
> > > >> >
> > > >>
> > > >> Great progress Joan! Thank you! :)
> > > >>
> > > >> > Test Results
> > > >> > ============
> > > >> > ** TL;DR: Lots of things are failing. Joan needs help interpreting
> > the
> > > >> results or she will go around the bend.
> > > >> >
> > > >> > Here are the current test results in gist form.
> > > >> >
> > > >> > EUnit:
> > > >> https://gist.github.com/anonymous/3203ed27c60cf3da4f0f0d5bff731722
> > > >> >
> > > >> > JS tests:
> > > >> https://gist.github.com/anonymous/93b0b70ed445ca4043a63140f8d219bf
> > > >> >
> > > >> > For the EUnit tests, everything other than os_process stuff seems
> to
> > > be
> > > >> working. Honestly, I think we can release without os_process support
> > on
> > > >> Windows, though I should file a bug to track this. I am actually
> > > inclined
> > > >> to disable os_process support on Windows and the related eunit tests
> > > rather
> > > >> than fix this rarely-needed functionality, unless someone on this
> list
> > > >> objects.
> > > >>
> > > >> You are probably thinking about CouchDB Externals, which definitely
> > use
> > > >> os_process functionality and which I’d also be fine with dropping
> > > support
> > > >> for Windows for now, but os_process is also used by the view server,
> > so
> > > we
> > > >> should definitely get them passing.
> > > >>
> > > >>
> > > >> > For the JS tests, a *lot* is failing. I need to know how much this
> > > >> differs from a Linux/OSX baseline today, can anyone help me follow
> up
> > > here?
> > > >>
> > > >> Can you try running these against a -n 1 cluster? We are not set up
> to
> > > >> run JS tests against more nodes at this point.
> > > >>
> > > >> On master/unix most if not all JS tests should either pass or
> skipped
> > > >> with a TODO message.
> > > >>
> > > >>
> > > >> Best
> > > >> Jan
> > > >> --
> > > >>
> > > >>
> > > >>
> > >
> >
>

Re: 2.0 & Windows: status update

[Nick North <no...@gmail.com>]

Thanks Sebastian. I'm looking at eunit at the moment, but hope to come back
to these.

Nick

On Sun, 19 Jun 2016 at 23:01 Sebastian Rothbucher <
sebastianrothbucher@googlemail.com> wrote:

> Hi Joan, Nick,
>
> the following gist provides a current run of the test against the latest
> master of CouchDB - and the latest tests (from the auth-tests-wip branch):
>
> https://gist.github.com/sebastianrothbucher/efa3a992bd4de9996b4125da82a7e0de
> Maybe you can use them
>
> Here's what I did to get both latest tests and latest code:
> git checkout master
> ./configure -c --disable-docs --disable-fauxton
> make clean
> make
> git checkout auth-tests-wip
>
> Currently, make javascript seems not optimal as one tests (needs
> investigation) seems to mess up the setup for all that's following. Hence,
> I took this drastic measure to produce the logs:
>
> for t in test/javascript/tests/*.js; do rm -rf dev/lib; dev/run -n 1 -q
> --with-admin-party-please test/javascript/run $t 2>&1 | tee -a jstest2.log;
> done
>
> Maybe it makes sense for you to start w/ something similar to produce some
> meaningful results.
>
> Best
>    Sebastian
>
> On Sun, Jun 19, 2016 at 6:04 PM, Nick North <no...@gmail.com> wrote:
>
> > Sorry - I meant a single node cluster in that last message. And I meant
> to
> > sign my name correctly.
> >
> > Nick
> >
> > On Sun, 19 Jun 2016 at 16:56 Nick North <no...@gmail.com> wrote:
> >
> > > I'm trying these tests now, and find that there are still a lot of JS
> > > failures with a single cluster. Many of them look suspiciously similar
> at
> > > an initial glance, but I hope to look in more detail tomorrow.
> > >
> > > Nicj
> > >
> > > On Sun, 19 Jun 2016 at 15:20 Jan Lehnardt <ja...@apache.org> wrote:
> > >
> > >>
> > >> > On 17 Jun 2016, at 22:48, Joan Touzet <wo...@apache.org> wrote:
> > >> >
> > >> > Hello everyone,
> > >> >
> > >> > I'd like to update the community on the status of the 2.0 port to
> > >> Microsoft Windows. There are three parts to this email: the build
> > >> tools/chain themselves, support in CouchDB for the Windows build
> > process,
> > >> and testing results. I'll cover them in that order.
> > >> >
> > >> > -Joan
> > >> >
> > >> > Build Tools/Chain
> > >> > =================
> > >> > ** TL;DR: New glazier repo to join couchdb, contains scripts and
> > README
> > >> to build CouchDB 2.0 on Windows.
> > >> >
> > >> > Our work to date has been going on in Dave Cottlehuber's glazier
> > >> repository at
> > >> >
> > >> >  https://github.com/dch/glazier/tree/release/couchdb_2.0
> > >> >
> > >> > The reason for the extra repository is that the Windows build
> process
> > >> is *very* ugly, involving 3 distinct build chains (Visual Studio,
> Cygwin
> > >> and the Mozilla Build system) to build all of the necessary
> > prerequisites.
> > >> The repository includes a number of support scripts to set up that
> > >> environment, a README with a detailed walkthrough, and some patches
> > >> necessary to the prerequisites to get them to build under the modern
> > >> Windows b uild system.
> > >> >
> > >> > Parenthetically, it _is_ possible to use binary installs for the
> > >> prerequisites (OpenSSL, libcurl, Erlang, SM 1.8.5), but Dave, Nick
> North
> > >> and I have evolved the glazier system over a number of years and it's
> > >> proven quite effective. Plus, we don't have to worry about the
> > provenance
> > >> of any of the binaries since we build everything from source directly,
> > and
> > >> that's important when we put up an unofficial Windows build for
> > download at
> > >> https://couchdb.apache.org/ .
> > >> >
> > >> > Good news: as of today I've requested and Infra has created a new
> > >> apache couchdb-glazier repo, and it's my intent to mirror dch/glazier
> > over
> > >> into the ASF's repo once things have stabilized a bit more (PR and
> > merge of
> > >> the release/couchdb_2.0 branch, and pending progress on steps 2 and 3
> > >> below). Dave and I did an audit of the repository as it stands, and
> > since
> > >> all checkins come from CouchDB contributors already, we are good to go
> > from
> > >> a licensing perspective.
> > >> >
> > >> >
> > >> > Overall CouchDB Windows support
> > >> > ===============================
> > >> > ** TL;DR: Windows support in 2.0 a priority, conversion of top-level
> > >> Makefile in progress.
> > >> >
> > >> > There are two aspects to native CouchDB Windows support. The first
> is
> > >> anything within the CouchDB code itself that assumes a Unix-like
> > >> environment. Fortunately, most of these problems have been worked out
> in
> > >> prior releases. I'm not aware of any outstanding issues here (except
> one
> > >> point below under test results).
> > >> >
> > >> > The other aspect is the build setup within the couchdb repo itself.
> > >> I've already converted the bash configure script into a PowerShell
> > >> configure script that works fine. However, the Makefile has bashisms
> in
> > it
> > >> and assumes GNU Make. I've started a conversion of this into Windows
> > NMake
> > >> format and will submit a PR for a Makefile.win in due course.
> > >> >
> > >> > I want to answer two frequent questions we get here before they get
> > >> re-asked:
> > >> >
> > >> >  1) Why not use a cygwin environment to retain compatibility with
> the
> > >> Unix build process? The answer is that performance suffers, the build
> > chain
> > >> is onerous, there are link-time problems when trying to link against
> > things
> > >> built using Visual Studio, and there are still assumptions on paths
> that
> > >> don't work out. We can't get away from making Windows-specific
> > >> customizations to the build process anyway, so we might as well take
> the
> > >> extra step and support the build process properly. It's not THAT much
> > work
> > >> to convert the Makefile and configure script, and our top-level
> Makefile
> > >> really isn't much more than a shell script anyway (every target is a
> > .PHONY
> > >> target!). In fact, a TODO for an enterprising developer might be to
> > rewrite
> > >> our top-level Makefile/Makefile.win as a Python script that "does the
> > right
> > >> thing" on both platforms, the same way our dev/run script works today.
> > >> >
> > >> >  2) Why not use the new "Bash and Ubuntu on Windows" functionality
> > >> Microsoft has announced for Windows 10? There are two distinct
> problems
> > >> here. The first is that there is a very large install base still of
> > Windows
> > >> 7 and 8 (and Windows Server) machines that cannot run this subsystem.
> > The
> > >> second is that Microsoft themselves say this about the functionality:
> > >> >
> > >> >     "Second, while you’ll be able to run native Bash and many Linux
> > >> command-line tools on Windows, it’s important to note that this is a
> > >> developer toolset to help you write and build all your code for all
> your
> > >> scenarios and platforms. This is not a server platform upon which you
> > will
> > >> host websites, run server infrastructure, etc."
> > >> >
> > >> > Given this strong warning from Microsoft themselves (which hints at
> > >> performance consideratings), and the fact that download statistics
> show
> > an
> > >> equal number of downloads of the CouchDB .tar source and the Windows
> > .zip
> > >> installer from our couchdb.apache.org website, we need to consider
> that
> > >> people are running CouchDB on Windows not just as a developer tool but
> > as a
> > >> fully-fledged server. As such it behooves us to build it "properly"
> as a
> > >> normal Windows binary/service.
> > >> >
> > >>
> > >> Great progress Joan! Thank you! :)
> > >>
> > >> > Test Results
> > >> > ============
> > >> > ** TL;DR: Lots of things are failing. Joan needs help interpreting
> the
> > >> results or she will go around the bend.
> > >> >
> > >> > Here are the current test results in gist form.
> > >> >
> > >> > EUnit:
> > >> https://gist.github.com/anonymous/3203ed27c60cf3da4f0f0d5bff731722
> > >> >
> > >> > JS tests:
> > >> https://gist.github.com/anonymous/93b0b70ed445ca4043a63140f8d219bf
> > >> >
> > >> > For the EUnit tests, everything other than os_process stuff seems to
> > be
> > >> working. Honestly, I think we can release without os_process support
> on
> > >> Windows, though I should file a bug to track this. I am actually
> > inclined
> > >> to disable os_process support on Windows and the related eunit tests
> > rather
> > >> than fix this rarely-needed functionality, unless someone on this list
> > >> objects.
> > >>
> > >> You are probably thinking about CouchDB Externals, which definitely
> use
> > >> os_process functionality and which I’d also be fine with dropping
> > support
> > >> for Windows for now, but os_process is also used by the view server,
> so
> > we
> > >> should definitely get them passing.
> > >>
> > >>
> > >> > For the JS tests, a *lot* is failing. I need to know how much this
> > >> differs from a Linux/OSX baseline today, can anyone help me follow up
> > here?
> > >>
> > >> Can you try running these against a -n 1 cluster? We are not set up to
> > >> run JS tests against more nodes at this point.
> > >>
> > >> On master/unix most if not all JS tests should either pass or skipped
> > >> with a TODO message.
> > >>
> > >>
> > >> Best
> > >> Jan
> > >> --
> > >>
> > >>
> > >>
> >
>

Re: 2.0 & Windows: status update

[Sebastian Rothbucher <se...@googlemail.com>]

Hi Joan, Nick,

the following gist provides a current run of the test against the latest
master of CouchDB - and the latest tests (from the auth-tests-wip branch):
https://gist.github.com/sebastianrothbucher/efa3a992bd4de9996b4125da82a7e0de
Maybe you can use them

Here's what I did to get both latest tests and latest code:
git checkout master
./configure -c --disable-docs --disable-fauxton
make clean
make
git checkout auth-tests-wip

Currently, make javascript seems not optimal as one tests (needs
investigation) seems to mess up the setup for all that's following. Hence,
I took this drastic measure to produce the logs:

for t in test/javascript/tests/*.js; do rm -rf dev/lib; dev/run -n 1 -q
--with-admin-party-please test/javascript/run $t 2>&1 | tee -a jstest2.log;
done

Maybe it makes sense for you to start w/ something similar to produce some
meaningful results.

Best
   Sebastian

On Sun, Jun 19, 2016 at 6:04 PM, Nick North <no...@gmail.com> wrote:

> Sorry - I meant a single node cluster in that last message. And I meant to
> sign my name correctly.
>
> Nick
>
> On Sun, 19 Jun 2016 at 16:56 Nick North <no...@gmail.com> wrote:
>
> > I'm trying these tests now, and find that there are still a lot of JS
> > failures with a single cluster. Many of them look suspiciously similar at
> > an initial glance, but I hope to look in more detail tomorrow.
> >
> > Nicj
> >
> > On Sun, 19 Jun 2016 at 15:20 Jan Lehnardt <ja...@apache.org> wrote:
> >
> >>
> >> > On 17 Jun 2016, at 22:48, Joan Touzet <wo...@apache.org> wrote:
> >> >
> >> > Hello everyone,
> >> >
> >> > I'd like to update the community on the status of the 2.0 port to
> >> Microsoft Windows. There are three parts to this email: the build
> >> tools/chain themselves, support in CouchDB for the Windows build
> process,
> >> and testing results. I'll cover them in that order.
> >> >
> >> > -Joan
> >> >
> >> > Build Tools/Chain
> >> > =================
> >> > ** TL;DR: New glazier repo to join couchdb, contains scripts and
> README
> >> to build CouchDB 2.0 on Windows.
> >> >
> >> > Our work to date has been going on in Dave Cottlehuber's glazier
> >> repository at
> >> >
> >> >  https://github.com/dch/glazier/tree/release/couchdb_2.0
> >> >
> >> > The reason for the extra repository is that the Windows build process
> >> is *very* ugly, involving 3 distinct build chains (Visual Studio, Cygwin
> >> and the Mozilla Build system) to build all of the necessary
> prerequisites.
> >> The repository includes a number of support scripts to set up that
> >> environment, a README with a detailed walkthrough, and some patches
> >> necessary to the prerequisites to get them to build under the modern
> >> Windows b uild system.
> >> >
> >> > Parenthetically, it _is_ possible to use binary installs for the
> >> prerequisites (OpenSSL, libcurl, Erlang, SM 1.8.5), but Dave, Nick North
> >> and I have evolved the glazier system over a number of years and it's
> >> proven quite effective. Plus, we don't have to worry about the
> provenance
> >> of any of the binaries since we build everything from source directly,
> and
> >> that's important when we put up an unofficial Windows build for
> download at
> >> https://couchdb.apache.org/ .
> >> >
> >> > Good news: as of today I've requested and Infra has created a new
> >> apache couchdb-glazier repo, and it's my intent to mirror dch/glazier
> over
> >> into the ASF's repo once things have stabilized a bit more (PR and
> merge of
> >> the release/couchdb_2.0 branch, and pending progress on steps 2 and 3
> >> below). Dave and I did an audit of the repository as it stands, and
> since
> >> all checkins come from CouchDB contributors already, we are good to go
> from
> >> a licensing perspective.
> >> >
> >> >
> >> > Overall CouchDB Windows support
> >> > ===============================
> >> > ** TL;DR: Windows support in 2.0 a priority, conversion of top-level
> >> Makefile in progress.
> >> >
> >> > There are two aspects to native CouchDB Windows support. The first is
> >> anything within the CouchDB code itself that assumes a Unix-like
> >> environment. Fortunately, most of these problems have been worked out in
> >> prior releases. I'm not aware of any outstanding issues here (except one
> >> point below under test results).
> >> >
> >> > The other aspect is the build setup within the couchdb repo itself.
> >> I've already converted the bash configure script into a PowerShell
> >> configure script that works fine. However, the Makefile has bashisms in
> it
> >> and assumes GNU Make. I've started a conversion of this into Windows
> NMake
> >> format and will submit a PR for a Makefile.win in due course.
> >> >
> >> > I want to answer two frequent questions we get here before they get
> >> re-asked:
> >> >
> >> >  1) Why not use a cygwin environment to retain compatibility with the
> >> Unix build process? The answer is that performance suffers, the build
> chain
> >> is onerous, there are link-time problems when trying to link against
> things
> >> built using Visual Studio, and there are still assumptions on paths that
> >> don't work out. We can't get away from making Windows-specific
> >> customizations to the build process anyway, so we might as well take the
> >> extra step and support the build process properly. It's not THAT much
> work
> >> to convert the Makefile and configure script, and our top-level Makefile
> >> really isn't much more than a shell script anyway (every target is a
> .PHONY
> >> target!). In fact, a TODO for an enterprising developer might be to
> rewrite
> >> our top-level Makefile/Makefile.win as a Python script that "does the
> right
> >> thing" on both platforms, the same way our dev/run script works today.
> >> >
> >> >  2) Why not use the new "Bash and Ubuntu on Windows" functionality
> >> Microsoft has announced for Windows 10? There are two distinct problems
> >> here. The first is that there is a very large install base still of
> Windows
> >> 7 and 8 (and Windows Server) machines that cannot run this subsystem.
> The
> >> second is that Microsoft themselves say this about the functionality:
> >> >
> >> >     "Second, while you’ll be able to run native Bash and many Linux
> >> command-line tools on Windows, it’s important to note that this is a
> >> developer toolset to help you write and build all your code for all your
> >> scenarios and platforms. This is not a server platform upon which you
> will
> >> host websites, run server infrastructure, etc."
> >> >
> >> > Given this strong warning from Microsoft themselves (which hints at
> >> performance consideratings), and the fact that download statistics show
> an
> >> equal number of downloads of the CouchDB .tar source and the Windows
> .zip
> >> installer from our couchdb.apache.org website, we need to consider that
> >> people are running CouchDB on Windows not just as a developer tool but
> as a
> >> fully-fledged server. As such it behooves us to build it "properly" as a
> >> normal Windows binary/service.
> >> >
> >>
> >> Great progress Joan! Thank you! :)
> >>
> >> > Test Results
> >> > ============
> >> > ** TL;DR: Lots of things are failing. Joan needs help interpreting the
> >> results or she will go around the bend.
> >> >
> >> > Here are the current test results in gist form.
> >> >
> >> > EUnit:
> >> https://gist.github.com/anonymous/3203ed27c60cf3da4f0f0d5bff731722
> >> >
> >> > JS tests:
> >> https://gist.github.com/anonymous/93b0b70ed445ca4043a63140f8d219bf
> >> >
> >> > For the EUnit tests, everything other than os_process stuff seems to
> be
> >> working. Honestly, I think we can release without os_process support on
> >> Windows, though I should file a bug to track this. I am actually
> inclined
> >> to disable os_process support on Windows and the related eunit tests
> rather
> >> than fix this rarely-needed functionality, unless someone on this list
> >> objects.
> >>
> >> You are probably thinking about CouchDB Externals, which definitely use
> >> os_process functionality and which I’d also be fine with dropping
> support
> >> for Windows for now, but os_process is also used by the view server, so
> we
> >> should definitely get them passing.
> >>
> >>
> >> > For the JS tests, a *lot* is failing. I need to know how much this
> >> differs from a Linux/OSX baseline today, can anyone help me follow up
> here?
> >>
> >> Can you try running these against a -n 1 cluster? We are not set up to
> >> run JS tests against more nodes at this point.
> >>
> >> On master/unix most if not all JS tests should either pass or skipped
> >> with a TODO message.
> >>
> >>
> >> Best
> >> Jan
> >> --
> >>
> >>
> >>
>

Re: 2.0 & Windows: status update

[Nick North <no...@gmail.com>]

Sorry - I meant a single node cluster in that last message. And I meant to
sign my name correctly.

Nick

On Sun, 19 Jun 2016 at 16:56 Nick North <no...@gmail.com> wrote:

> I'm trying these tests now, and find that there are still a lot of JS
> failures with a single cluster. Many of them look suspiciously similar at
> an initial glance, but I hope to look in more detail tomorrow.
>
> Nicj
>
> On Sun, 19 Jun 2016 at 15:20 Jan Lehnardt <ja...@apache.org> wrote:
>
>>
>> > On 17 Jun 2016, at 22:48, Joan Touzet <wo...@apache.org> wrote:
>> >
>> > Hello everyone,
>> >
>> > I'd like to update the community on the status of the 2.0 port to
>> Microsoft Windows. There are three parts to this email: the build
>> tools/chain themselves, support in CouchDB for the Windows build process,
>> and testing results. I'll cover them in that order.
>> >
>> > -Joan
>> >
>> > Build Tools/Chain
>> > =================
>> > ** TL;DR: New glazier repo to join couchdb, contains scripts and README
>> to build CouchDB 2.0 on Windows.
>> >
>> > Our work to date has been going on in Dave Cottlehuber's glazier
>> repository at
>> >
>> >  https://github.com/dch/glazier/tree/release/couchdb_2.0
>> >
>> > The reason for the extra repository is that the Windows build process
>> is *very* ugly, involving 3 distinct build chains (Visual Studio, Cygwin
>> and the Mozilla Build system) to build all of the necessary prerequisites.
>> The repository includes a number of support scripts to set up that
>> environment, a README with a detailed walkthrough, and some patches
>> necessary to the prerequisites to get them to build under the modern
>> Windows b uild system.
>> >
>> > Parenthetically, it _is_ possible to use binary installs for the
>> prerequisites (OpenSSL, libcurl, Erlang, SM 1.8.5), but Dave, Nick North
>> and I have evolved the glazier system over a number of years and it's
>> proven quite effective. Plus, we don't have to worry about the provenance
>> of any of the binaries since we build everything from source directly, and
>> that's important when we put up an unofficial Windows build for download at
>> https://couchdb.apache.org/ .
>> >
>> > Good news: as of today I've requested and Infra has created a new
>> apache couchdb-glazier repo, and it's my intent to mirror dch/glazier over
>> into the ASF's repo once things have stabilized a bit more (PR and merge of
>> the release/couchdb_2.0 branch, and pending progress on steps 2 and 3
>> below). Dave and I did an audit of the repository as it stands, and since
>> all checkins come from CouchDB contributors already, we are good to go from
>> a licensing perspective.
>> >
>> >
>> > Overall CouchDB Windows support
>> > ===============================
>> > ** TL;DR: Windows support in 2.0 a priority, conversion of top-level
>> Makefile in progress.
>> >
>> > There are two aspects to native CouchDB Windows support. The first is
>> anything within the CouchDB code itself that assumes a Unix-like
>> environment. Fortunately, most of these problems have been worked out in
>> prior releases. I'm not aware of any outstanding issues here (except one
>> point below under test results).
>> >
>> > The other aspect is the build setup within the couchdb repo itself.
>> I've already converted the bash configure script into a PowerShell
>> configure script that works fine. However, the Makefile has bashisms in it
>> and assumes GNU Make. I've started a conversion of this into Windows NMake
>> format and will submit a PR for a Makefile.win in due course.
>> >
>> > I want to answer two frequent questions we get here before they get
>> re-asked:
>> >
>> >  1) Why not use a cygwin environment to retain compatibility with the
>> Unix build process? The answer is that performance suffers, the build chain
>> is onerous, there are link-time problems when trying to link against things
>> built using Visual Studio, and there are still assumptions on paths that
>> don't work out. We can't get away from making Windows-specific
>> customizations to the build process anyway, so we might as well take the
>> extra step and support the build process properly. It's not THAT much work
>> to convert the Makefile and configure script, and our top-level Makefile
>> really isn't much more than a shell script anyway (every target is a .PHONY
>> target!). In fact, a TODO for an enterprising developer might be to rewrite
>> our top-level Makefile/Makefile.win as a Python script that "does the right
>> thing" on both platforms, the same way our dev/run script works today.
>> >
>> >  2) Why not use the new "Bash and Ubuntu on Windows" functionality
>> Microsoft has announced for Windows 10? There are two distinct problems
>> here. The first is that there is a very large install base still of Windows
>> 7 and 8 (and Windows Server) machines that cannot run this subsystem. The
>> second is that Microsoft themselves say this about the functionality:
>> >
>> >     "Second, while you’ll be able to run native Bash and many Linux
>> command-line tools on Windows, it’s important to note that this is a
>> developer toolset to help you write and build all your code for all your
>> scenarios and platforms. This is not a server platform upon which you will
>> host websites, run server infrastructure, etc."
>> >
>> > Given this strong warning from Microsoft themselves (which hints at
>> performance consideratings), and the fact that download statistics show an
>> equal number of downloads of the CouchDB .tar source and the Windows .zip
>> installer from our couchdb.apache.org website, we need to consider that
>> people are running CouchDB on Windows not just as a developer tool but as a
>> fully-fledged server. As such it behooves us to build it "properly" as a
>> normal Windows binary/service.
>> >
>>
>> Great progress Joan! Thank you! :)
>>
>> > Test Results
>> > ============
>> > ** TL;DR: Lots of things are failing. Joan needs help interpreting the
>> results or she will go around the bend.
>> >
>> > Here are the current test results in gist form.
>> >
>> > EUnit:
>> https://gist.github.com/anonymous/3203ed27c60cf3da4f0f0d5bff731722
>> >
>> > JS tests:
>> https://gist.github.com/anonymous/93b0b70ed445ca4043a63140f8d219bf
>> >
>> > For the EUnit tests, everything other than os_process stuff seems to be
>> working. Honestly, I think we can release without os_process support on
>> Windows, though I should file a bug to track this. I am actually inclined
>> to disable os_process support on Windows and the related eunit tests rather
>> than fix this rarely-needed functionality, unless someone on this list
>> objects.
>>
>> You are probably thinking about CouchDB Externals, which definitely use
>> os_process functionality and which I’d also be fine with dropping support
>> for Windows for now, but os_process is also used by the view server, so we
>> should definitely get them passing.
>>
>>
>> > For the JS tests, a *lot* is failing. I need to know how much this
>> differs from a Linux/OSX baseline today, can anyone help me follow up here?
>>
>> Can you try running these against a -n 1 cluster? We are not set up to
>> run JS tests against more nodes at this point.
>>
>> On master/unix most if not all JS tests should either pass or skipped
>> with a TODO message.
>>
>>
>> Best
>> Jan
>> --
>>
>>
>>

Re: 2.0 & Windows: status update

[Nick North <no...@gmail.com>]

I'm trying these tests now, and find that there are still a lot of JS
failures with a single cluster. Many of them look suspiciously similar at
an initial glance, but I hope to look in more detail tomorrow.

Nicj

On Sun, 19 Jun 2016 at 15:20 Jan Lehnardt <ja...@apache.org> wrote:

>
> > On 17 Jun 2016, at 22:48, Joan Touzet <wo...@apache.org> wrote:
> >
> > Hello everyone,
> >
> > I'd like to update the community on the status of the 2.0 port to
> Microsoft Windows. There are three parts to this email: the build
> tools/chain themselves, support in CouchDB for the Windows build process,
> and testing results. I'll cover them in that order.
> >
> > -Joan
> >
> > Build Tools/Chain
> > =================
> > ** TL;DR: New glazier repo to join couchdb, contains scripts and README
> to build CouchDB 2.0 on Windows.
> >
> > Our work to date has been going on in Dave Cottlehuber's glazier
> repository at
> >
> >  https://github.com/dch/glazier/tree/release/couchdb_2.0
> >
> > The reason for the extra repository is that the Windows build process is
> *very* ugly, involving 3 distinct build chains (Visual Studio, Cygwin and
> the Mozilla Build system) to build all of the necessary prerequisites. The
> repository includes a number of support scripts to set up that environment,
> a README with a detailed walkthrough, and some patches necessary to the
> prerequisites to get them to build under the modern Windows b uild system.
> >
> > Parenthetically, it _is_ possible to use binary installs for the
> prerequisites (OpenSSL, libcurl, Erlang, SM 1.8.5), but Dave, Nick North
> and I have evolved the glazier system over a number of years and it's
> proven quite effective. Plus, we don't have to worry about the provenance
> of any of the binaries since we build everything from source directly, and
> that's important when we put up an unofficial Windows build for download at
> https://couchdb.apache.org/ .
> >
> > Good news: as of today I've requested and Infra has created a new apache
> couchdb-glazier repo, and it's my intent to mirror dch/glazier over into
> the ASF's repo once things have stabilized a bit more (PR and merge of the
> release/couchdb_2.0 branch, and pending progress on steps 2 and 3 below).
> Dave and I did an audit of the repository as it stands, and since all
> checkins come from CouchDB contributors already, we are good to go from a
> licensing perspective.
> >
> >
> > Overall CouchDB Windows support
> > ===============================
> > ** TL;DR: Windows support in 2.0 a priority, conversion of top-level
> Makefile in progress.
> >
> > There are two aspects to native CouchDB Windows support. The first is
> anything within the CouchDB code itself that assumes a Unix-like
> environment. Fortunately, most of these problems have been worked out in
> prior releases. I'm not aware of any outstanding issues here (except one
> point below under test results).
> >
> > The other aspect is the build setup within the couchdb repo itself. I've
> already converted the bash configure script into a PowerShell configure
> script that works fine. However, the Makefile has bashisms in it and
> assumes GNU Make. I've started a conversion of this into Windows NMake
> format and will submit a PR for a Makefile.win in due course.
> >
> > I want to answer two frequent questions we get here before they get
> re-asked:
> >
> >  1) Why not use a cygwin environment to retain compatibility with the
> Unix build process? The answer is that performance suffers, the build chain
> is onerous, there are link-time problems when trying to link against things
> built using Visual Studio, and there are still assumptions on paths that
> don't work out. We can't get away from making Windows-specific
> customizations to the build process anyway, so we might as well take the
> extra step and support the build process properly. It's not THAT much work
> to convert the Makefile and configure script, and our top-level Makefile
> really isn't much more than a shell script anyway (every target is a .PHONY
> target!). In fact, a TODO for an enterprising developer might be to rewrite
> our top-level Makefile/Makefile.win as a Python script that "does the right
> thing" on both platforms, the same way our dev/run script works today.
> >
> >  2) Why not use the new "Bash and Ubuntu on Windows" functionality
> Microsoft has announced for Windows 10? There are two distinct problems
> here. The first is that there is a very large install base still of Windows
> 7 and 8 (and Windows Server) machines that cannot run this subsystem. The
> second is that Microsoft themselves say this about the functionality:
> >
> >     "Second, while you’ll be able to run native Bash and many Linux
> command-line tools on Windows, it’s important to note that this is a
> developer toolset to help you write and build all your code for all your
> scenarios and platforms. This is not a server platform upon which you will
> host websites, run server infrastructure, etc."
> >
> > Given this strong warning from Microsoft themselves (which hints at
> performance consideratings), and the fact that download statistics show an
> equal number of downloads of the CouchDB .tar source and the Windows .zip
> installer from our couchdb.apache.org website, we need to consider that
> people are running CouchDB on Windows not just as a developer tool but as a
> fully-fledged server. As such it behooves us to build it "properly" as a
> normal Windows binary/service.
> >
>
> Great progress Joan! Thank you! :)
>
> > Test Results
> > ============
> > ** TL;DR: Lots of things are failing. Joan needs help interpreting the
> results or she will go around the bend.
> >
> > Here are the current test results in gist form.
> >
> > EUnit:
> https://gist.github.com/anonymous/3203ed27c60cf3da4f0f0d5bff731722
> >
> > JS tests:
> https://gist.github.com/anonymous/93b0b70ed445ca4043a63140f8d219bf
> >
> > For the EUnit tests, everything other than os_process stuff seems to be
> working. Honestly, I think we can release without os_process support on
> Windows, though I should file a bug to track this. I am actually inclined
> to disable os_process support on Windows and the related eunit tests rather
> than fix this rarely-needed functionality, unless someone on this list
> objects.
>
> You are probably thinking about CouchDB Externals, which definitely use
> os_process functionality and which I’d also be fine with dropping support
> for Windows for now, but os_process is also used by the view server, so we
> should definitely get them passing.
>
>
> > For the JS tests, a *lot* is failing. I need to know how much this
> differs from a Linux/OSX baseline today, can anyone help me follow up here?
>
> Can you try running these against a -n 1 cluster? We are not set up to run
> JS tests against more nodes at this point.
>
> On master/unix most if not all JS tests should either pass or skipped with
> a TODO message.
>
>
> Best
> Jan
> --
>
>
>

Re: 2.0 & Windows: status update

[Jan Lehnardt <ja...@apache.org>]

> On 17 Jun 2016, at 22:48, Joan Touzet <wo...@apache.org> wrote:
> 
> Hello everyone,
> 
> I'd like to update the community on the status of the 2.0 port to Microsoft Windows. There are three parts to this email: the build tools/chain themselves, support in CouchDB for the Windows build process, and testing results. I'll cover them in that order.
> 
> -Joan
> 
> Build Tools/Chain
> =================
> ** TL;DR: New glazier repo to join couchdb, contains scripts and README to build CouchDB 2.0 on Windows.
> 
> Our work to date has been going on in Dave Cottlehuber's glazier repository at 
> 
>  https://github.com/dch/glazier/tree/release/couchdb_2.0
> 
> The reason for the extra repository is that the Windows build process is *very* ugly, involving 3 distinct build chains (Visual Studio, Cygwin and the Mozilla Build system) to build all of the necessary prerequisites. The repository includes a number of support scripts to set up that environment, a README with a detailed walkthrough, and some patches necessary to the prerequisites to get them to build under the modern Windows b uild system.
> 
> Parenthetically, it _is_ possible to use binary installs for the prerequisites (OpenSSL, libcurl, Erlang, SM 1.8.5), but Dave, Nick North and I have evolved the glazier system over a number of years and it's proven quite effective. Plus, we don't have to worry about the provenance of any of the binaries since we build everything from source directly, and that's important when we put up an unofficial Windows build for download at https://couchdb.apache.org/ .
> 
> Good news: as of today I've requested and Infra has created a new apache couchdb-glazier repo, and it's my intent to mirror dch/glazier over into the ASF's repo once things have stabilized a bit more (PR and merge of the release/couchdb_2.0 branch, and pending progress on steps 2 and 3 below). Dave and I did an audit of the repository as it stands, and since all checkins come from CouchDB contributors already, we are good to go from a licensing perspective.
> 
> 
> Overall CouchDB Windows support
> ===============================
> ** TL;DR: Windows support in 2.0 a priority, conversion of top-level Makefile in progress.
> 
> There are two aspects to native CouchDB Windows support. The first is anything within the CouchDB code itself that assumes a Unix-like environment. Fortunately, most of these problems have been worked out in prior releases. I'm not aware of any outstanding issues here (except one point below under test results).
> 
> The other aspect is the build setup within the couchdb repo itself. I've already converted the bash configure script into a PowerShell configure script that works fine. However, the Makefile has bashisms in it and assumes GNU Make. I've started a conversion of this into Windows NMake format and will submit a PR for a Makefile.win in due course.
> 
> I want to answer two frequent questions we get here before they get re-asked:
> 
>  1) Why not use a cygwin environment to retain compatibility with the Unix build process? The answer is that performance suffers, the build chain is onerous, there are link-time problems when trying to link against things built using Visual Studio, and there are still assumptions on paths that don't work out. We can't get away from making Windows-specific customizations to the build process anyway, so we might as well take the extra step and support the build process properly. It's not THAT much work to convert the Makefile and configure script, and our top-level Makefile really isn't much more than a shell script anyway (every target is a .PHONY target!). In fact, a TODO for an enterprising developer might be to rewrite our top-level Makefile/Makefile.win as a Python script that "does the right thing" on both platforms, the same way our dev/run script works today.
> 
>  2) Why not use the new "Bash and Ubuntu on Windows" functionality Microsoft has announced for Windows 10? There are two distinct problems here. The first is that there is a very large install base still of Windows 7 and 8 (and Windows Server) machines that cannot run this subsystem. The second is that Microsoft themselves say this about the functionality:
> 
>     "Second, while you’ll be able to run native Bash and many Linux command-line tools on Windows, it’s important to note that this is a developer toolset to help you write and build all your code for all your scenarios and platforms. This is not a server platform upon which you will host websites, run server infrastructure, etc."
> 
> Given this strong warning from Microsoft themselves (which hints at performance consideratings), and the fact that download statistics show an equal number of downloads of the CouchDB .tar source and the Windows .zip installer from our couchdb.apache.org website, we need to consider that people are running CouchDB on Windows not just as a developer tool but as a fully-fledged server. As such it behooves us to build it "properly" as a normal Windows binary/service.
> 

Great progress Joan! Thank you! :)

> Test Results
> ============
> ** TL;DR: Lots of things are failing. Joan needs help interpreting the results or she will go around the bend.
> 
> Here are the current test results in gist form.
> 
> EUnit: https://gist.github.com/anonymous/3203ed27c60cf3da4f0f0d5bff731722
> 
> JS tests: https://gist.github.com/anonymous/93b0b70ed445ca4043a63140f8d219bf
> 
> For the EUnit tests, everything other than os_process stuff seems to be working. Honestly, I think we can release without os_process support on Windows, though I should file a bug to track this. I am actually inclined to disable os_process support on Windows and the related eunit tests rather than fix this rarely-needed functionality, unless someone on this list objects.

You are probably thinking about CouchDB Externals, which definitely use os_process functionality and which I’d also be fine with dropping support for Windows for now, but os_process is also used by the view server, so we should definitely get them passing.


> For the JS tests, a *lot* is failing. I need to know how much this differs from a Linux/OSX baseline today, can anyone help me follow up here?

Can you try running these against a -n 1 cluster? We are not set up to run JS tests against more nodes at this point.

On master/unix most if not all JS tests should either pass or skipped with a TODO message.


Best
Jan
--

Re: 2.0 & Windows: status update

[Joan Touzet <wo...@apache.org>]

*facepalm* :) Nothing private, just...didn't want to clutter the list.

Sorry!

----- Original Message -----
> From: "Joan Touzet" <wo...@apache.org>
> To: dev@couchdb.apache.org
> Sent: Saturday, June 18, 2016 3:56:50 AM
> Subject: Re: 2.0 & Windows: status update
> 
> Replying off list.
[SNIP]

Re: 2.0 & Windows: status update

[Joan Touzet <wo...@apache.org>]

Replying off list.

Nick be sure you get my patch to the couchdb repo itself before you test:

  https://github.com/apache/couchdb/pull/423/files

I'll be reworking this slightly to make Ilya happy but the PR works 
in its current state.

-Joan

----- Original Message -----
> From: "Nick North" <no...@gmail.com>
> To: dev@couchdb.apache.org, "Joan Touzet" <wo...@apache.org>
> Sent: Saturday, June 18, 2016 3:37:50 AM
> Subject: Re: 2.0 & Windows: status update
> 
> Hi Joan. This sounds like an improvement on my last Windows build
> attempt,
> which still had a couple of eunit failures outside the os_process
> area.
> I'll attempt to reproduce your results today, and report back.
> 
> Nick
> 
> On Sat, 18 Jun 2016 at 00:34 Joan Touzet <wo...@apache.org> wrote:
> 
> > Replying to my own email: dev/run is using the wrong path slashes
> > for
> > the path to the javascript/coffeescript view servers. I've
> > submitted a PR
> > and will merge if no one complains.
> >
> > Here is the latest set of test results if anyone still wants to
> > volunteer
> > to help me review them (both Eunit and JS, scroll down for the JS
> > results):
> >
> >   https://gist.github.com/wohali/256d134dea8e79c2050636c7ff044c23
> >
> > I'm especially worried about the few JS tests we're failing, those
> > seem
> > subtle. I need a Linux baseline to compare against. If no one has
> > one, I
> > can see I'll be setting up my own Linux build environment on
> > Monday...
> >
> > -Joan
> >
> 

Re: 2.0 & Windows: status update

[Nick North <no...@gmail.com>]

Hi Joan. This sounds like an improvement on my last Windows build attempt,
which still had a couple of eunit failures outside the os_process area.
I'll attempt to reproduce your results today, and report back.

Nick

On Sat, 18 Jun 2016 at 00:34 Joan Touzet <wo...@apache.org> wrote:

> Replying to my own email: dev/run is using the wrong path slashes for
> the path to the javascript/coffeescript view servers. I've submitted a PR
> and will merge if no one complains.
>
> Here is the latest set of test results if anyone still wants to volunteer
> to help me review them (both Eunit and JS, scroll down for the JS results):
>
>   https://gist.github.com/wohali/256d134dea8e79c2050636c7ff044c23
>
> I'm especially worried about the few JS tests we're failing, those seem
> subtle. I need a Linux baseline to compare against. If no one has one, I
> can see I'll be setting up my own Linux build environment on Monday...
>
> -Joan
>

Re: 2.0 & Windows: status update

[Joan Touzet <wo...@apache.org>]

Replying to my own email: dev/run is using the wrong path slashes for
the path to the javascript/coffeescript view servers. I've submitted a PR
and will merge if no one complains.

Here is the latest set of test results if anyone still wants to volunteer
to help me review them (both Eunit and JS, scroll down for the JS results):

  https://gist.github.com/wohali/256d134dea8e79c2050636c7ff044c23

I'm especially worried about the few JS tests we're failing, those seem
subtle. I need a Linux baseline to compare against. If no one has one, I 
can see I'll be setting up my own Linux build environment on Monday...

-Joan

Re: 2.0 & Windows: status update

[Joan Touzet <wo...@apache.org>]

A small update on the testing front. I've tried replicating the animaldb from
Cloudant to my local server, and things looked very ugly:

https://gist.github.com/anonymous/f2a94234195f007c3049e27d942482c1

I've yet to dig into this failure; I'm posting it here in case anyone has the
time to analyze it on my behalf. (I have about 2 more hours in my work day
today and I have other things to attend to...)

-Joan