Hooks That Use Perl Test::Builder Having Problems with STDERR

[er...@barclayscapital.com]

I'm working on porting a fairly extensive set of CVS hooks to SVN.  The issue that I'm having now is that my pre-commit hook, which runs a Perl script that performs tests based on Test::Builder and Test::More, never show their stderr on the console.  I see it in the svn web server logs, but not on the console.

Has anyone had any experience with a problem like this?  Essentially, the Test::* modules remap stderr and stdout for the purpose of trapping that output to evaluate it for the tests.  Seems like I have dueling file handles.

Still looking into this; I know I don't have a complete understanding of this issue.

Any thoughts on what I might be able to do here?

Eric

_______________________________________________

This e-mail may contain information that is confidential, privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. Unless specifically indicated, this e-mail is not an offer to buy or sell or a solicitation to buy or sell any securities, investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Barclays. Any views or opinions presented are solely those of the author and do not necessarily represent those of Barclays. This e-mail is subject to terms available at the following link: www.barcap.com/emaildisclaimer. By messaging with Barclays you consent to the foregoing.  Barclays Capital is the investment banking division of Barclays Bank PLC, a company registered in England (number 1026167) with its registered office at 1 Churchill Place, London, E14 5HP.  This email may relate to or be sent from other members of the Barclays Group.
_______________________________________________

RE: Hooks That Use Perl Test::Builder Having Problems with STDERR

[er...@barclayscapital.com]

________________________________
From: David Weintraub [mailto:qazwart@gmail.com]
Sent: Wednesday, January 05, 2011 6:04 PM
To: Berg, Eric: IT (NYK)
Cc: users@subversion.apache.org
Subject: Re: Hooks That Use Perl Test::Builder Having Problems with STDERR
On Wed, Jan 5, 2011 at 5:03 PM, <er...@barclayscapital.com>> wrote:
Dave, if you look into how the hooks work, basically, they are passed a repo path and a transaction id that, using svnlook, gives you access to copies of the working files, so it doesn't matter where the hooks run, nor is there any requirement for server/client communication.

I've written quite a few hooks. I have a hook script that implements watches.  8<8<

However, I take it that in order to run the tests, you need these files written to a directory, and you may need dependent files there too. That starts getting a bit more complex than what "svnlook" was built for. In theory, you could checkout a working directory on a hook script, then use "svnlook cat" to update the files that are being committed, and run your tests. It's complex, and can take a long time.

Nonissue.  We've been doing this with CVS hooks for almost 10 years.  It's "just a port" ;(....  and I'm not even going to bring up the issues that I will undoubtedly run into when I get to work on the DEPLOYMENT piece.

Even after all of that, Subversion captures STDOUT and STDERR and they don't get printed out to a console. If you want to see them, you'll have to capture them and then either write them to a logfile, or email them.

This is a larger issue for us, specifically WRT the Test::Builder tests that we've implemented.  The way this was done was to make any Perl class which required tests to be a subclass of our UnitTest class, which runs any subroutines with names that begin with "test_".  Each of these has some number of Test::* tests.  With CVS, we see the progress of the tests as they complete -- whether they succeed or fail.  With SVN, we'd only see failures.

This is where I'm having some issues with the STDERR output.  I haven't dug down deeply enough understand the impace of the "stacked redirection" of SVN's doing a STDERR/OUT capture, then the Test::* mods doing their own redirect.  What i have seen is that the only place that the stderr appears once these tests are run is the web server logs.  A simple test which simply dies in my precommit hook shows up fine to the client's stderr.

Though I do love immediate checkins, I'm not sure where you're going when you suggest that our validations might be better handled some way other than by hooks.  That appears to be the whole reason to have such hooks:  to validate files before allowing a checkin.

How long does it take your pre-commit hook to run? Even a few seconds can seem like an eternity to a developer who is making a few minor changes. If every time a developer does a commit, they have to wait, they simply will stop making commits when they should. And, they'll learn to hate Subversion because it is slow and buggy.

Don't forget that this has been the standard for our checkins here since time immemorial.   Developers are pretty much used to it, if not simply resigned.  I appreciate the stringent code validation regime that enforces a number of checks in addition to the unit tests.

It can take from 30 seconds up to several minutes.  For database code, often, actual temporary deploys are followed by tests to validate what's been deployed are  valid before allowing the checkins to continue.  Again, this seems like a good place NOT to mention our CVS tag-based deploys, which can take a VERY long time to deploy to our ~40 db's, etc.
Remember that Subversion is a version control system which means you can undo stuff that should never have been committed in the first place. Doing tests during the build cycle has lots of advantages:

I'm in the middle here, being the developer responsible for this porting project.  I mentioned the deployment stuff too, which I fear, and which I fear is far less appropriate to be implemented via SVN hooks.  Your input is appreciated.  We could go back-and-forth endlessly, but ultimately, it's not my decision.  There's a lot of inertia influencing this as well, not to mention that this is just one of my projects, and that our organization has discontinued support for CVS, so we're kind of working without a net at the moment.

Not sure how this is going to go, but the input is, again, appreciated.

* You have room to checkout your entire project and have access to all the files. That can make running your tests much simpler to do.
* You can use Hudson as your framework (or another build server). That means the reporting, running, and checking the results are all done for you. There's no reinventing the wheel.
 * Your commits are now much faster.
* Your tests have their own environment and won't interfere with Subversion
* You have a complete log of your tests, and you can review it and see if things are improving.
* You have time and resources for more thorough testing
* You can notify the entire team when there's an issue. There's always a good chance that the problem isn't in the file being committed, but is located elsewhere.
* Most importantly, you'll make your life a lot easier. All you have to do is write the tests and not worry about how you'll run the tests, or how you'll get the output from the tests.

It is definitely possible to do what you want, but it will take a lot of work and will probably cause more problems than it is worth..

--
David Weintraub
qazwart@gmail.com<ma...@gmail.com>

_______________________________________________

This e-mail may contain information that is confidential, privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. Unless specifically indicated, this e-mail is not an offer to buy or sell or a solicitation to buy or sell any securities, investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Barclays. Any views or opinions presented are solely those of the author and do not necessarily represent those of Barclays. This e-mail is subject to terms available at the following link: www.barcap.com/emaildisclaimer. By messaging with Barclays you consent to the foregoing.  Barclays Capital is the investment banking division of Barclays Bank PLC, a company registered in England (number 1026167) with its registered office at 1 Churchill Place, London, E14 5HP.  This email may relate to or be sent from other members of the Barclays Group.
_______________________________________________

Re: Hooks That Use Perl Test::Builder Having Problems with STDERR

[David Weintraub <qa...@gmail.com>]

On Wed, Jan 5, 2011 at 5:03 PM, <er...@barclayscapital.com> wrote:

> Dave, if you look into how the hooks work, basically, they are passed a
> repo path and a transaction id that, using svnlook, gives you access to
> copies of the working files, so it doesn't matter where the hooks run, nor
> is there any requirement for server/client communication.
>

I've written quite a few hooks. I have a hook script that implements
watches. There are many post-commit hooks that set watches, but mine allows
the users to set their own by storing the watchlist for each user in the
Subversion repository. I use "svnlook cat" to parse out the user's watch
list to see if the file being committed is part of the watch. I can do this
without having to write the watchlist file into a directory.

However, I take it that in order to run the tests, you need these files
written to a directory, and you may need dependent files there too. That
starts getting a bit more complex than what "svnlook" was built for. In
theory, you could checkout a working directory on a hook script, then use
"svnlook cat" to update the files that are being committed, and run your
tests. It's complex, and can take a long time.

Even after all of that, Subversion captures STDOUT and STDERR and they don't
get printed out to a console. If you want to see them, you'll have to
capture them and then either write them to a logfile, or email them.

Though I do love immediate checkins, I'm not sure where you're going when
> you suggest that our validations might be better handled some way other than
> by hooks.  That appears to be the whole reason to have such hooks:  to
> validate files before allowing a checkin.


How long does it take your pre-commit hook to run? Even a few seconds can
seem like an eternity to a developer who is making a few minor changes. If
every time a developer does a commit, they have to wait, they simply will
stop making commits when they should. And, they'll learn to hate Subversion
because it is slow and buggy.

Remember that Subversion is a version control system which means you can
undo stuff that should never have been committed in the first place. Doing
tests during the build cycle has lots of advantages:

* You have room to checkout your entire project and have access to all the
files. That can make running your tests much simpler to do.
* You can use Hudson as your framework (or another build server). That means
the reporting, running, and checking the results are all done for you.
There's no reinventing the wheel.
* Your commits are now much faster.
* Your tests have their own environment and won't interfere with Subversion
* You have a complete log of your tests, and you can review it and see if
things are improving.
* You have time and resources for more thorough testing
* You can notify the entire team when there's an issue. There's always a
good chance that the problem isn't in the file being committed, but is
located elsewhere.
* Most importantly, you'll make your life a lot easier. All you have to do
is write the tests and not worry about how you'll run the tests, or how
you'll get the output from the tests.

It is definitely possible to do what you want, but it will take a lot of
work and will probably cause more problems than it is worth..

-- 
David Weintraub
qazwart@gmail.com

Re: Hooks That Use Perl Test::Builder Having Problems with STDERR

[Daniel Shahaf <d....@daniel.shahaf.name>]

eric.berg@barclayscapital.com wrote on Wed, Jan 05, 2011 at 17:03:11 -0500:
> Dave, if you look into how the hooks work, basically, they are passed a repo path and a transaction id that, using svnlook, gives you access to copies of the working files, so it doesn't matter where the hooks run, nor is there any requirement for server/client communication.
> 
> Though I do love immediate checkins, I'm not sure where you're going when you suggest that our validations might be better handled some way other than by hooks.  That appears to be the whole reason to have such hooks:  to validate files before allowing a checkin.
> 
> How would hudson help to validate files in the context of a checkin transaction?
> 

Hudson validates after the commit.  But you could commit to one branch,
have hudson validate, and if validation passes merge to trunk.

(David, if you already said that in your response, sorry for the
duplication)

> Eric
> 
> 
> ________________________________
> From: David Weintraub [mailto:qazwart@gmail.com]
> Sent: Wednesday, January 05, 2011 4:37 PM
> To: Berg, Eric: IT (NYK)
> Cc: users@subversion.apache.org
> Subject: Re: Hooks That Use Perl Test::Builder Having Problems with STDERR
> 
> On Wed, Jan 5, 2011 at 11:30 AM, <er...@barclayscapital.com>> wrote:
> I'm working on porting a fairly extensive set of CVS hooks to SVN.
> 
> Okay. Stop right there. When ever someone mentions "a fairly extensive set" of hooks, I start to think maybe most of what they want shouldn't necessarily be hooks. When hooks are running, the client is sitting there waiting for the results. I've been at one place where it wasn't unusual for a commit to take almost a minute for reformatting, testing, etc. Those were a group of very frustrated developers.
> 
> Besides, how can you even run your tests? The hooks execute on the Subversion server, yet the working directory is on the client There's no communication between the server and client until the hooks are complete.
> 
> > The issue that I'm having now is that my pre-commit hook, which runs a Perl script
> > that performs tests based on Test::Builder and Test::More, never show their stderr
> > on the console.  I see it in the svn web server logs, but not on the console.
> 
> The problem is that Subversion swallow STDOUT and STDERR from the hooks, and Subversion won't display STDERR unless the hook script returns a non-zero exit value, and only then, it displays it on the developer's console. That may be your problem.
> 
> Do you have  continuous build server like Hudson (http://hudson-ci.org)? Even if you don't need to compile code, you can use something like Hudson for running your tests. Hudson can checkout the project from the Subversion repository, run tests, and then report back the results.
> 
> Moving your testing to Hudson will simplify your life and help keep a running record of your test results. Hudson won't interfere with Test::Builder and Test::More, and you'll be able to keep the test results of all of your check ins in one easy to find place.
> 
> --
> David Weintraub
> qazwart@gmail.com<ma...@gmail.com>
> 
> _______________________________________________
> 
> This e-mail may contain information that is confidential, privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. Unless specifically indicated, this e-mail is not an offer to buy or sell or a solicitation to buy or sell any securities, investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Barclays. Any views or opinions presented are solely those of the author and do not necessarily represent those of Barclays. This e-mail is subject to terms available at the following link: www.barcap.com/emaildisclaimer. By messaging with Barclays you consent to the foregoing.  Barclays Capital is the investment banking division of Barclays Bank PLC, a company registered in England (number 1026167) with its registered office at 1 Churchill Place, London, E14 5HP.  This email may relate to or be sent from other members of the Barclays Group.
> _______________________________________________

Re: Hooks That Use Perl Test::Builder Having Problems with STDERR

[Andy Levy <an...@gmail.com>]

On Wed, Jan 5, 2011 at 17:03,  <er...@barclayscapital.com> wrote:
> Dave, if you look into how the hooks work, basically, they are passed a repo
> path and a transaction id that, using svnlook, gives you access to copies of
> the working files, so it doesn't matter where the hooks run, nor is there
> any requirement for server/client communication.

Depending upon what your hooks do, they may require a full working
copy (and thus client executable) to operate.

> Though I do love immediate checkins, I'm not sure where you're going when
> you suggest that our validations might be better handled some way other than
> by hooks.  That appears to be the whole reason to have such hooks:  to
> validate files before allowing a checkin.

One breaking point in my mind is "how long will this take?". If it's a
very fast script to execute then a hook script is probably fine. If
it's a compilation & test suite execution, that could take minutes or
longer. One of my projects takes 15 minutes to do a full compile &
deploy; I can't hold every other commit to the repository for that.

> How would hudson help to validate files in the context of a checkin
> transaction?

A CI server would do the validation after checkin (allowing the
checkin to be very fast), and send notifications in the case of errors
(or success, for that matter).

> ________________________________
> From: David Weintraub [mailto:qazwart@gmail.com]
> Sent: Wednesday, January 05, 2011 4:37 PM
> To: Berg, Eric: IT (NYK)
> Cc: users@subversion.apache.org
> Subject: Re: Hooks That Use Perl Test::Builder Having Problems with STDERR
>
> On Wed, Jan 5, 2011 at 11:30 AM, <er...@barclayscapital.com> wrote:
>>
>> I'm working on porting a fairly extensive set of CVS hooks to SVN.
>
> Okay. Stop right there. When ever someone mentions "a fairly extensive set"
> of hooks, I start to think maybe most of what they want shouldn't
> necessarily be hooks. When hooks are running, the client is sitting there
> waiting for the results. I've been at one place where it wasn't unusual for
> a commit to take almost a minute for reformatting, testing, etc. Those were
> a group of very frustrated developers.
> Besides, how can you even run your tests? The hooks execute on the
> Subversion server, yet the working directory is on the client There's no
> communication between the server and client until the hooks are complete.
>> The issue that I'm having now is that my pre-commit hook, which runs a
>> Perl script
>> that performs tests based on Test::Builder and Test::More, never show
>> their stderr
>> on the console.  I see it in the svn web server logs, but not on the
>> console.
> The problem is that Subversion swallow STDOUT and STDERR from the hooks, and
> Subversion won't display STDERR unless the hook script returns a non-zero
> exit value, and only then, it displays it on the developer's console. That
> may be your problem.
> Do you have  continuous build server like Hudson (http://hudson-ci.org)?
> Even if you don't need to compile code, you can use something like Hudson
> for running your tests. Hudson can checkout the project from the Subversion
> repository, run tests, and then report back the results.
> Moving your testing to Hudson will simplify your life and help keep a
> running record of your test results. Hudson won't interfere with
> Test::Builder and Test::More, and you'll be able to keep the test results of
> all of your check ins in one easy to find place.
> --
> David Weintraub
> qazwart@gmail.com
>
> _______________________________________________
>
>
>
> This e-mail may contain information that is confidential, privileged or
> otherwise protected from disclosure. If you are not an intended recipient of
> this e-mail, do not duplicate or redistribute it by any means. Please delete
> it and any attachments and notify the sender that you have received it in
> error. Unless specifically indicated, this e-mail is not an offer to buy or
> sell or a solicitation to buy or sell any securities, investment products or
> other financial product or service, an official confirmation of any
> transaction, or an official statement of Barclays. Any views or opinions
> presented are solely those of the author and do not necessarily represent
> those of Barclays. This e-mail is subject to terms available at the
> following link: www.barcap.com/emaildisclaimer. By messaging with Barclays
> you consent to the foregoing.  Barclays Capital is the investment banking
> division of Barclays Bank PLC, a company registered in England (number
> 1026167) with its registered office at 1 Churchill Place, London, E14 5HP.
> This email may relate to or be sent from other members of the Barclays
> Group.
>
> _______________________________________________

RE: Hooks That Use Perl Test::Builder Having Problems with STDERR

[er...@barclayscapital.com]

Dave, if you look into how the hooks work, basically, they are passed a repo path and a transaction id that, using svnlook, gives you access to copies of the working files, so it doesn't matter where the hooks run, nor is there any requirement for server/client communication.

Though I do love immediate checkins, I'm not sure where you're going when you suggest that our validations might be better handled some way other than by hooks.  That appears to be the whole reason to have such hooks:  to validate files before allowing a checkin.

How would hudson help to validate files in the context of a checkin transaction?

Eric


________________________________
From: David Weintraub [mailto:qazwart@gmail.com]
Sent: Wednesday, January 05, 2011 4:37 PM
To: Berg, Eric: IT (NYK)
Cc: users@subversion.apache.org
Subject: Re: Hooks That Use Perl Test::Builder Having Problems with STDERR

On Wed, Jan 5, 2011 at 11:30 AM, <er...@barclayscapital.com>> wrote:
I'm working on porting a fairly extensive set of CVS hooks to SVN.

Okay. Stop right there. When ever someone mentions "a fairly extensive set" of hooks, I start to think maybe most of what they want shouldn't necessarily be hooks. When hooks are running, the client is sitting there waiting for the results. I've been at one place where it wasn't unusual for a commit to take almost a minute for reformatting, testing, etc. Those were a group of very frustrated developers.

Besides, how can you even run your tests? The hooks execute on the Subversion server, yet the working directory is on the client There's no communication between the server and client until the hooks are complete.

> The issue that I'm having now is that my pre-commit hook, which runs a Perl script
> that performs tests based on Test::Builder and Test::More, never show their stderr
> on the console.  I see it in the svn web server logs, but not on the console.

The problem is that Subversion swallow STDOUT and STDERR from the hooks, and Subversion won't display STDERR unless the hook script returns a non-zero exit value, and only then, it displays it on the developer's console. That may be your problem.

Do you have  continuous build server like Hudson (http://hudson-ci.org)? Even if you don't need to compile code, you can use something like Hudson for running your tests. Hudson can checkout the project from the Subversion repository, run tests, and then report back the results.

Moving your testing to Hudson will simplify your life and help keep a running record of your test results. Hudson won't interfere with Test::Builder and Test::More, and you'll be able to keep the test results of all of your check ins in one easy to find place.

--
David Weintraub
qazwart@gmail.com<ma...@gmail.com>

_______________________________________________

This e-mail may contain information that is confidential, privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. Unless specifically indicated, this e-mail is not an offer to buy or sell or a solicitation to buy or sell any securities, investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Barclays. Any views or opinions presented are solely those of the author and do not necessarily represent those of Barclays. This e-mail is subject to terms available at the following link: www.barcap.com/emaildisclaimer. By messaging with Barclays you consent to the foregoing.  Barclays Capital is the investment banking division of Barclays Bank PLC, a company registered in England (number 1026167) with its registered office at 1 Churchill Place, London, E14 5HP.  This email may relate to or be sent from other members of the Barclays Group.
_______________________________________________

Re: Hooks That Use Perl Test::Builder Having Problems with STDERR

[David Weintraub <qa...@gmail.com>]

On Wed, Jan 5, 2011 at 11:30 AM, <er...@barclayscapital.com> wrote:

> I'm working on porting a fairly extensive set of CVS hooks to SVN.


Okay. Stop right there. When ever someone mentions "a fairly extensive set"
of hooks, I start to think maybe most of what they want shouldn't
necessarily be hooks. When hooks are running, the client is sitting there
waiting for the results. I've been at one place where it wasn't unusual for
a commit to take almost a minute for reformatting, testing, etc. Those were
a group of very frustrated developers.

Besides, how can you even run your tests? The hooks execute on the
Subversion server, yet the working directory is on the client There's no
communication between the server and client until the hooks are complete.

> The issue that I'm having now is that my pre-commit hook, which runs a
Perl script
> that performs tests based on Test::Builder and Test::More, never show
their stderr
> on the console.  I see it in the svn web server logs, but not on the
console.

The problem is that Subversion swallow STDOUT and STDERR from the hooks, and
Subversion won't display STDERR unless the hook script returns a non-zero
exit value, and only then, it displays it on the developer's console. That
may be your problem.

Do you have  continuous build server like Hudson (http://hudson-ci.org)?
Even if you don't need to compile code, you can use something like Hudson
for running your tests. Hudson can checkout the project from the Subversion
repository, run tests, and then report back the results.

Moving your testing to Hudson will simplify your life and help keep a
running record of your test results. Hudson won't interfere with
Test::Builder and Test::More, and you'll be able to keep the test results of
all of your check ins in one easy to find place.

-- 
David Weintraub
qazwart@gmail.com