Re: svn commit: r1215525 - in /httpd/httpd/trunk: ./ docs/manual/mod/ docs/manual/programs/ modules/debugging/ support/

["William A. Rowe Jr." <wr...@rowe-clan.net>]

On 12/17/2011 11:04 AM, minfrin@apache.org wrote:
> Author: minfrin
> Date: Sat Dec 17 17:03:59 2011
> New Revision: 1215525
> 
> URL: http://svn.apache.org/viewvc?rev=1215525&view=rev
> Log:
> mod_firehose: Add a new debugging module able to record traffic
> passing through the server in such a way that connections and/or
> requests be reconstructed and replayed.

Hmmm?  Slow down cowboy.

You called for a vote on a *subproject*.  A subproject has a very
specific definition.  apreq, mod_ftp, mod_fcgid, mod_arm4 are all
various subprojects of httpd.

None live in httpd/httpd/trunk/

I proposed instead that you directly propose mod_policy, one of the
three modules, as a core httpd module, because it already has a clear
fit and really needs little further review (improvements, sure).

I didn't see any such commentary about mod_firehose.

Issac, Jim and Sander all voted to adopt this subproject.  None of them
proposed that it be introduced into trunk.

NOBODY suggested that this proposed subproject go into trunk.

Re: svn commit: r1215525 - in /httpd/httpd/trunk: ./ docs/manual/mod/ docs/manual/programs/ modules/debugging/ support/

["William A. Rowe Jr." <wr...@rowe-clan.net>]

On 2/29/2012 9:41 AM, Jim Jagielski wrote:
> 
> On Feb 28, 2012, at 10:05 PM, William A. Rowe Jr. wrote:
>>
>> You have only three votes for mod_policy... please fix this today, before I
>> withdraw what starts to look like an ill-cast vote for introducing code with
>> dubious support of the community.
>>
> 
> "only three votes"???

Correction, now two votes for, with one vote against.

My vote was ill-cast, I no longer believe minfrin's behavior is correctable.

Re: svn commit: r1215525 - in /httpd/httpd/trunk: ./ docs/manual/mod/ docs/manual/programs/ modules/debugging/ support/

[Jim Jagielski <ji...@jaguNET.com>]

On Feb 28, 2012, at 10:05 PM, William A. Rowe Jr. wrote:
> 
> You have only three votes for mod_policy... please fix this today, before I
> withdraw what starts to look like an ill-cast vote for introducing code with
> dubious support of the community.
> 

"only three votes"???
 ----

How many does he need??

Re: svn commit: r1215525 - in /httpd/httpd/trunk: ./ docs/manual/mod/ docs/manual/programs/ modules/debugging/ support/

["William A. Rowe Jr." <wr...@rowe-clan.net>]

On 12/18/2011 10:45 AM, Graham Leggett wrote:
> On 17 Dec 2011, at 10:51 PM, William A. Rowe Jr. wrote:
> 
>> I proposed instead that you directly propose mod_policy, one of the
>> three modules, as a core httpd module, because it already has a clear
>> fit and really needs little further review (improvements, sure).
> 
> What you really said was:
> 
> "To clarify, I'm +1 for directly incorporating this into httpd/trunk/modules/test/
> 
> I don't think we need an independent subproject for this particular module, it
> seems rather straightforward.  Default to build-but-don't-load.  Provide an example
> config under docs/conf/extra/httpd-policy.conf"

The subject of what I really said was mod_policy.  Only mod_policy.  Nothing
other than mod_policy.  Do not twist my post.

mod_policy is an expression of HTTP correctness which I suggested for test/,
which httpd aught to have a vested interest in pursuing, as protocol correctness
is one aspect that interests us technically.  modules\test\ is a historic place
for this.

> Given that mod_firehose is significantly simpler than mod_policy, I am struggling to understand given both your stated preference above, and the preferences of others already stated, why suddenly this is a big issue.

That's your evaluation, not mine.  My evaluation is that you invented yet another
wonky data representation and the whole idea of firehose merits further thought
at its new home if httpd is to accept it.

We don't even need mod_firehose.  Maybe it will be nice to have.  If we do want
to adopt it, this implies killing mod_dump_io or refactoring one into the other
since they are largely aspects of the same thing.

Even back in the day I would have insisted this go to modules/experimental.
As it stands, I insist it not go in until it has a community, something you
have yet to demonstrate.

You have only three votes for mod_policy... please fix this today, before I
withdraw what starts to look like an ill-cast vote for introducing code with
dubious support of the community.

Re: svn commit: r1215525 - in /httpd/httpd/trunk: ./ docs/manual/mod/ docs/manual/programs/ modules/debugging/ support/

["William A. Rowe Jr." <wr...@rowe-clan.net>]

On 2/28/2012 5:47 PM, Graham Leggett wrote:
> On 29 Feb 2012, at 12:21 AM, William A. Rowe Jr. wrote:
> 
>> After two months, firehose still didn't obtain another +1, so the vote to
>> incorporate firehose into trunk stands at 3 +1's, 1 -1, and therefore
>> failed the vote for inclusion in trunk.
> 
> I count 4 +1s on the dev@httpd list:
> 
> minfrin, issac, sctemme, jim

For firehose you called a vote for a *** subproject ***

Subject: Proposal: adoption of mod_firehose subproject
Date: Tue, 13 Dec 2011 17:19:16 +0200
Message-Id: <C9...@sharp.fm>

These four people voted to adopt a subproject.  Nobody within that
thread suggested it become a module.  I emailed some supportive questions
seeking more detail, as I have been supportive of all subprojects to gain
attention to code.  Sometimes successful, sometimes not.

You then ninja'ed the thread on the 17th after the voting that this had been
a vote to add a module.  Such behavior isn't becoming of a collaborator.
I then voted -1 due to your bait and switch and asked for who voted
to adopt firehose as a module? ... as I did not trust that the committers
had really consented to maintaining this code in core, especially since
it was only accepted on the slimmest of margin as a standalone subproject.

Jim responded that his was a vote for a module or a subproject, no difference.

Sander did not clarify after 12/13, and did not consent to a module.

Issac did not clarify after 12/13, and did not consent to a module.

"Based on the support expressed, and the expressed preference that the module
be part of httpd, I have updated the documentation for both mod_firehose and
firehose and will commit them to trunk shortly."

There was no said support in that thread for mod_firehose in trunk. Don't piss
on my head and tell me it's raining.

Veto stands, calling your vote fundamentally invalid, and since you are failing
to read my posts, is now unconditional.  Revert this yourself to a subproject
before I simply revert this into void.

Re: svn commit: r1215525 - in /httpd/httpd/trunk: ./ docs/manual/mod/ docs/manual/programs/ modules/debugging/ support/

[Graham Leggett <mi...@sharp.fm>]

On 29 Feb 2012, at 12:21 AM, William A. Rowe Jr. wrote:

> After two months, firehose still didn't obtain another +1, so the vote to
> incorporate firehose into trunk stands at 3 +1's, 1 -1, and therefore
> failed the vote for inclusion in trunk.

I count 4 +1s on the dev@httpd list:

minfrin, issac, sctemme, jim

and igalic on the pmc list. poirier and niq expressed support, but didn't vote. By my reckoning, it passes.

> There are 4 +1's for a firehose subproject at httpd.  If you wish to continue
> this effort you can keep this simple by svn mv'ing those respective files across
> from httpd/httpd/trunk to httpd/mod_firehose/trunk, where it can build a larger
> community within httpd project, and things like the data representation format
> can be evaluated and enhanced by the community.

So far, you are the only one who has questioned the data representation format, claiming that wireshark/pcap could be used instead.

When I invented the firehose I wanted an ASCII based protocol, so that I could look at a firehose being recorded and by inspection see the boundaries between the buckets. When you're debugging httpd or an application server behind httpd the bucket boundaries are important. The protocol used is simply chunked encoding, with additional fields behind the chunked value allowing you to tie the buckets back together, and it is very clear which bits are firehose and which part is the bucket by inspection, on purpose.

In addition, there was a clear need to be able to capture both individual requests, and whole connection streams, and have the ability to choose which. With the individual requests we were able to isolate some tricky protocol edge cases in mod_cache, and with the connections we were able to identify issues with HTTP/1.0 clients and keepalive in a highly loaded service oriented environment.

Further crucial information is dropped buckets, which is either a sign of congestion over the pipe to which these buckets are written, or a sign that the child process crashed unexpectedly.

When you brought up pcap as a possibility, I went and did a whole bunch of research into it. What I discovered is that pcap itself gives you a binary encapsulation format, meaning that you lose the ability to see bucket boundaries, and the ability to debug by inspection. Then I looked for existing client support that might help, and discovered there is none - clients expect the pcap packets to contain clearly identified packets you would expect to find at layer 2, not free form binary that you can expect to find in an HTTP body. I considered hacking up fake TCP packets, but then ran into the problem that I now needed to care whether they were IPV4 or IPV6 packets, and trying to hack up fake packets when you captured individual requests on a long lived connection is a non trivial exercise. At the end of the day you want confidence in your debugging data, and a hacked together reconstruction of TCP is the exact opposite of that.

The HTTP protocol dump is called a "firehose" for a reason - at one point we were recording and analysing hundreds of gigabytes of request data, and firehose had to decode and process that in situ on a production box that had been taken out of the pool without the installation of any scripting language stack (which would have been too slow anyway), or the moving around of the data. Both mod_firehose and firehose are battle tested in that real world scenario.

I believe the pcap based solution you proposed is inferior to what we have now, and I believe there is no compelling reason at this point to attempt any alternative solution.

Regards,
Graham
--

Re: svn commit: r1215525 - in /httpd/httpd/trunk: ./ docs/manual/mod/ docs/manual/programs/ modules/debugging/ support/

["William A. Rowe Jr." <wr...@rowe-clan.net>]

Graham,

After two months, firehose still didn't obtain another +1, so the vote to
incorporate firehose into trunk stands at 3 +1's, 1 -1, and therefore
failed the vote for inclusion in trunk.

There are 4 +1's for a firehose subproject at httpd.  If you wish to continue
this effort you can keep this simple by svn mv'ing those respective files across
from httpd/httpd/trunk to httpd/mod_firehose/trunk, where it can build a larger
community within httpd project, and things like the data representation format
can be evaluated and enhanced by the community.

I would be happy to entertain merging this to trunk once we have contribution
from multiple individuals to the subproject and the consensus is that it will
be supported by the entire community in 2.6.  As an example, only Jeff and I
have recently made any significant commits to the mod_fcgid, so by my own
standards it is still missing at least one more active committer before
mod_fcgid could be considered for inclusion trunk.

FWIW my support to explore a mod_combine subproject is attached, and I'll
clarify that I meant to +1 the subproject, -1 for commit to trunk as a
module (Jim voted +1 to either case) so there are now three votes for a
mod_combine subproject, if you want to proceed with that effort as well.

Yours,

Bill


On 12/20/2011 12:53 PM, William A. Rowe Jr. wrote:
>> On Dec 18, 2011, at 11:45 AM, Graham Leggett wrote:
>>>
>>> Given that mod_firehose is significantly simpler than mod_policy, I am struggling to understand given both your stated preference above, and the preferences of others already stated, why suddenly this is a big issue.
> 
> Simpler?  Far more files are touched.  10 commits to inject it.  A new
> support utility must be maintained by the project.  It introduces a new
> nonstandard data format.  And the module name is not descriptive, which
> is a no-no for httpd core, as far as I'm concerned.
> 
> mod_policy was a single .c file, right?  It doesn't require a support
> binary to function, right?  (Naming it mod_http_policy might be clearer).
> Single commit, plus regen of the docs.  I'm struggling to understand what
> you mean by less simple.
> 
> On 12/18/2011 11:31 AM, Jim Jagielski wrote:
>> FWIW, my vote also applies to incorping direct into httpd repo...
>> It was not "conditional" on it being a "subproject"
> 
> Thanks Jim!
> 
> mod_firehose had still needed one more vote if it was to become part
> of trunk.  It now needs two...
> 
> -1 to this commit.  I'm casting this as a vote, and expressly not vetoing
> the change; *provided it obtains a majority +2 committers who will truly
> review the design and the code*.  (3 +1's are always needed; 4 +1/my -1
> passes for "accepted" as far as I'm concerned).
> 
> My preference for it remains to be a subproject, and the name rethought
> (we have all functional module names in httpd), so that people can start
> adopting it and proving it *before* we inject it into 2.4 branch.
> It would be a shame for it to grow stale for 2-5 years, but the 2.3-dev
> process reflects that not enough committers are active right now to
> ensure newly introduced code is sufficiently thought through.
> 
> Also wondering about the choice of compiling the firehose support tool
> instead of authoring something trivially maintainable in perl or python.
> That's what we did for mod_log_forensic.  Better yet, if it were using
> a standard representation, there isn't much need for the 'firehose'
> support utility to be maintained here.
> 
> I won't trust your code to receive adequate review sitting out on trunk
> for backport... we don't need to dwell on your previous poor designs
> such as r59099... still not corrected by you after 7 years... and still
> blocked by your veto for adequate resolution in httpd 2.4.  Anything
> you design that implies introduction of an API or a data representation
> format is suspect until proven otherwise; particularly in light of your
> failure to reasonably collaborate on implementation details.
> 
> A subproject would give this new code the attention it deserves while
> enabling users, and I'll even go so far as to +1 it's introduction as
> a subproject, apart from httpd trunk, where it can float or sink on
> its own merits.
> 
> Igor and Jim both agreed with you to adopt mod_combine as a subproject,
> so all three modules have green lights as such; only one was green lit
> for trunk.  Please don't repeat this mistake of jamming mod_combine
> into trunk, just yet?  Put it to a vote (or let voters refine their +1
> to indicate that it belongs in trunk).

Re: svn commit: r1215525 - in /httpd/httpd/trunk: ./ docs/manual/mod/ docs/manual/programs/ modules/debugging/ support/

["William A. Rowe Jr." <wr...@rowe-clan.net>]

> On Dec 18, 2011, at 11:45 AM, Graham Leggett wrote:
>>
>> Given that mod_firehose is significantly simpler than mod_policy, I am struggling to understand given both your stated preference above, and the preferences of others already stated, why suddenly this is a big issue.

Simpler?  Far more files are touched.  10 commits to inject it.  A new
support utility must be maintained by the project.  It introduces a new
nonstandard data format.  And the module name is not descriptive, which
is a no-no for httpd core, as far as I'm concerned.

mod_policy was a single .c file, right?  It doesn't require a support
binary to function, right?  (Naming it mod_http_policy might be clearer).
Single commit, plus regen of the docs.  I'm struggling to understand what
you mean by less simple.

On 12/18/2011 11:31 AM, Jim Jagielski wrote:
> FWIW, my vote also applies to incorping direct into httpd repo...
> It was not "conditional" on it being a "subproject"

Thanks Jim!

mod_firehose had still needed one more vote if it was to become part
of trunk.  It now needs two...

-1 to this commit.  I'm casting this as a vote, and expressly not vetoing
the change; *provided it obtains a majority +2 committers who will truly
review the design and the code*.  (3 +1's are always needed; 4 +1/my -1
passes for "accepted" as far as I'm concerned).

My preference for it remains to be a subproject, and the name rethought
(we have all functional module names in httpd), so that people can start
adopting it and proving it *before* we inject it into 2.4 branch.
It would be a shame for it to grow stale for 2-5 years, but the 2.3-dev
process reflects that not enough committers are active right now to
ensure newly introduced code is sufficiently thought through.

Also wondering about the choice of compiling the firehose support tool
instead of authoring something trivially maintainable in perl or python.
That's what we did for mod_log_forensic.  Better yet, if it were using
a standard representation, there isn't much need for the 'firehose'
support utility to be maintained here.

I won't trust your code to receive adequate review sitting out on trunk
for backport... we don't need to dwell on your previous poor designs
such as r59099... still not corrected by you after 7 years... and still
blocked by your veto for adequate resolution in httpd 2.4.  Anything
you design that implies introduction of an API or a data representation
format is suspect until proven otherwise; particularly in light of your
failure to reasonably collaborate on implementation details.

A subproject would give this new code the attention it deserves while
enabling users, and I'll even go so far as to +1 it's introduction as
a subproject, apart from httpd trunk, where it can float or sink on
its own merits.

Igor and Jim both agreed with you to adopt mod_combine as a subproject,
so all three modules have green lights as such; only one was green lit
for trunk.  Please don't repeat this mistake of jamming mod_combine
into trunk, just yet?  Put it to a vote (or let voters refine their +1
to indicate that it belongs in trunk).

Re: svn commit: r1215525 - in /httpd/httpd/trunk: ./ docs/manual/mod/ docs/manual/programs/ modules/debugging/ support/

[Jim Jagielski <ji...@jaguNET.com>]

On Dec 18, 2011, at 11:45 AM, Graham Leggett wrote:

> On 17 Dec 2011, at 10:51 PM, William A. Rowe Jr. wrote:
> 
>> I proposed instead that you directly propose mod_policy, one of the
>> three modules, as a core httpd module, because it already has a clear
>> fit and really needs little further review (improvements, sure).
> 
> What you really said was:
> 
> "To clarify, I'm +1 for directly incorporating this into httpd/trunk/modules/test/
> 
> I don't think we need an independent subproject for this particular module, it
> seems rather straightforward.  Default to build-but-don't-load.  Provide an example
> config under docs/conf/extra/httpd-policy.conf"
> 
> Given that mod_firehose is significantly simpler than mod_policy, I am struggling to understand given both your stated preference above, and the preferences of others already stated, why suddenly this is a big issue.
> 

FWIW, my vote also applies to incorping direct into httpd repo...
It was not "conditional" on it being a "subproject"

Re: svn commit: r1215525 - in /httpd/httpd/trunk: ./ docs/manual/mod/ docs/manual/programs/ modules/debugging/ support/

[Graham Leggett <mi...@sharp.fm>]

On 17 Dec 2011, at 10:51 PM, William A. Rowe Jr. wrote:

> I proposed instead that you directly propose mod_policy, one of the
> three modules, as a core httpd module, because it already has a clear
> fit and really needs little further review (improvements, sure).

What you really said was:

"To clarify, I'm +1 for directly incorporating this into httpd/trunk/modules/test/

I don't think we need an independent subproject for this particular module, it
seems rather straightforward.  Default to build-but-don't-load.  Provide an example
config under docs/conf/extra/httpd-policy.conf"

Given that mod_firehose is significantly simpler than mod_policy, I am struggling to understand given both your stated preference above, and the preferences of others already stated, why suddenly this is a big issue.

Regards,
Graham
--

Re: svn commit: r1215525 - in /httpd/httpd/trunk: ./ docs/manual/mod/ docs/manual/programs/ modules/debugging/ support/

["William A. Rowe Jr." <wr...@rowe-clan.net>]

On 12/18/2011 7:32 AM, Jim Jagielski wrote:
> 
> On Dec 17, 2011, at 3:51 PM, William A. Rowe Jr. wrote:
> 
>> NOBODY suggested that this proposed subproject go into trunk.
> 
> I must have been reading a different thread that you when
> the issue of having these as "subprojects" or "normal modules"
> was discussed and Igor and Dan (at least) indicated the desire
> for them being normal modules (since sub-modules have a tendency
> to be ignored)... No wait, you had to have been reading that because
> you rightly reminded people that large code donations normally
> go thru the Incubator.

I'm not seeing this within the thread "Proposal: adoption of mod_firehose
subproject" or the two other similar posts.  Pointer please?

Thank you for already indicating you reviewed and accept this into
trunk!  If two other committers review these, plus Graham, and agree
w.r.t. to firehose or combine in trunk, that will be plenty for me to
drop my objections.

Perhaps Issac or Sander, who already voted +1, also please share if you
want this in httpd trunk or as a new subproject as already approved, so
we can close this discussion?

If not approved sometime over the next week; Graham, would you please
revert, and introduce firehose as a subproject, instead?

Re: svn commit: r1215525 - in /httpd/httpd/trunk: ./ docs/manual/mod/ docs/manual/programs/ modules/debugging/ support/

["William A. Rowe Jr." <wr...@rowe-clan.net>]

On 12/18/2011 7:32 AM, Jim Jagielski wrote:
> 
> On Dec 17, 2011, at 3:51 PM, William A. Rowe Jr. wrote:
> 
>> NOBODY suggested that this proposed subproject go into trunk.
> 
> I must have been reading a different thread that you when
> the issue of having these as "subprojects" or "normal modules"
> was discussed and Igor and Dan (at least) indicated the desire
> for them being normal modules (since sub-modules have a tendency
> to be ignored)... No wait, you had to have been reading that because
> you rightly reminded people that large code donations normally
> go thru the Incubator.

Ahhh, no but wait, that chatter on a private list (your fault for
disclosure, not mine) did not happen.  Technical discussions that
occur on the private list are void.  Now I see why you and perhaps
even Graham are confused, and to an outside observer, off-base.

I see where there were private concerns expressed about acceptance
of sub-project code.  The fact that I disagreed with those comments,
but published my concerns on the public list, mean that effectively
this is a one sided discussion so far.  Those concerns could be valid,
but don't represent consensus, and until discussion on dev@ happens,
they aren't even concerns.

IMHO it is up to any subproject champion to promote their code for
consideration in trunk if that's where they believe it should end up,
once they demonstrate a community around the code.  That means they
have multiple committers.  Do we really believe this code is more
deserving of a fast track to trunk than mod_fcgid?

At this time these are simply nothing more than a code dump, and
that's not acceptable for mainline trunk.  With five or fewer pmc
supporters of this code out of some 70 PMC members, success doesn't
seem to be written in stone just yet.

Re: svn commit: r1215525 - in /httpd/httpd/trunk: ./ docs/manual/mod/ docs/manual/programs/ modules/debugging/ support/

[Jim Jagielski <ji...@jaguNET.com>]

On Dec 17, 2011, at 3:51 PM, William A. Rowe Jr. wrote:

> 
> NOBODY suggested that this proposed subproject go into trunk.
> 

I must have been reading a different thread that you when
the issue of having these as "subprojects" or "normal modules"
was discussed and Igor and Dan (at least) indicated the desire
for them being normal modules (since sub-modules have a tendency
to be ignored)... No wait, you had to have been reading that because
you rightly reminded people that large code donations normally
go thru the Incubator.