another response to Roy :-)

[Greg Stein <gs...@lyra.org>]

On Thu, Jun 29, 2000 at 10:16:31PM -0700, Roy T. Fielding wrote:
> >Your veto is real bullshit. It is supported only by manufactured excuses,
> >illogical assumptions, and impossible preconditions.
> 
> I don't think this is fair.  We don't need an interface that has all of
> the problems of buff and is just as fragile as existing module writes
> (more if you consider that the layers introduce configurable interaction
> errors that simply didn't exist before). If there is a design on the table
> that can improve the situation, then not desiring an inferior design
> is sufficient justification for a veto.  More to the point, if your
> design doesn't significantly improve the state of our implementation,
> then it isn't worth adding to the cruft to the server.

I don't understand this. What are the "problems of buff" that my patch
introduces? And I really don't understand the fragility point. It started
with a simple, easy-to-use char* callback handler. People wanted more, so I
filled in the slots I had (but didn't code first-time around) for the bucket
stuff. Ryan asked for an example of something else, and I demonstrated that
it was possible (but it isn't part of the patch). I believe the ability to
quickly add each feature demonstrates a sound framework rather than any
fragility.

Presuming the design you're referring to is the bucket-brigades... yes,
there is a design, but no code that implements it. I believe the only
missing, basic requirements is adding a "next" pointer to the buckets in my
patch. There may be some other details, but I think my patch can implement
what you are looking for. I'd love to learn where it specifically falls down.

To your last point: I would not have placed my patch into the STATUS file
for consideration unless I truly believed that it improved the situation.
I'm not about to suggest adding "cruft" to the server.

> The char * interface cannot support sendfile or caching or writing a proxy
> in filters.  It is useless to me, at the same cost as a design that would
> actually work.  I'm not saying it isn't useful for someone -- I am saying
> the design won't accomplish my needs, whereas the bucket brigades does.
> I would veto it myself if I had time to back it up with a fully implemented
> alternative.  Even so, there is no doubt in my mind that adding a dumb
> filtering design to Apache at this point is far far worse than doing
> nothing about filtering in 2.0.

Roy. Please hold for a moment here.

I *do* have buckets. The char* interface is simply an *option* for that
"someone" you refer to. If you want buckets, then apply my patch. If you
want a list-of-buckets, then I'll add a next pointer.

Specifically, take a look at ap_bucket_t in the ap_filter.h file that is
part of my patch. A filter then implements a function that accepts one of
those buckets.

This isn't a dumb filtering interface. It is a bucket-based system with an
option for a simplified interface.

>...
> If you guys spent half as much time commenting your code as you did
> arguing about the edge cases, maybe we wouldn't have to spend so much
> time trying to communicate?  Maybe it would help clarify the intention
> behind some of the interface issues.

This is a good suggestion, but we only have so much time. Both of us hoped
that the code was enough, and the comments and doc would come in a future
pass.

> Like, for example, Greg's apparent
> desire to stick to only what is needed by the ap_rwrite interface, which
> I didn't see from reading the code.  I still don't care about that as
> a design requirement, but at least now I understand a little more about
> why the patch does what it does.

Yes. I want to work within the existing framework, and all existing modules
to be able to send their content (via ap_rwrite and friends) into the filter
chain. I fully expect to later provide an ap_rbucket() interface so that
content generators can optimize their output.

> Here is my problem.  The early patches that were proposed were capable
> of implementing 25% of the stream filter goals, with the ability to
> be extended to 75%.  The latest ones implement maybe 70% with an
> extensibility to 80%.  You seem to be suggesting that it is the first
> number that matters in order to justify applying the patch.  My problem
> is that I don't care about the first number -- I don't want a filter
> interface in Apache that isn't capable of eventually handling 100%
> of the problem space.  That's because I don't want to have to redesign
> the interface after 2.0.0 goes out and everyone discovers that what
> we have sucks.

All right. This is a very reasonable point of view.

But, please... detail the features/qualities of that missing 20% or 30%. If
you want to be completely happy with my patch, then I'll gladly code those
bits for you. But I just don't know what your "100% goals" are... It seems
somewhat unfair to state that one of the posted patches does not meet the
100% goals, yet not explain what those are. How could we ever meet that?

>...
> I can understand why we might want to start with a simple implementation
> under the covers of an ADT.  What I don't understand is the theory that
> we should start with an interface that we know isn't sufficient for the
> task at hand and replace it later on.  We already know the right interface.
> If you want a simple layer on top of that, no problem, but the most
> efficient native interface must be the baseline.

Please educate me, then. Suggest a way to improve on the ap_bucket_t in my
patch. I simply do not understand why it doesn't match the "right interface."

> Greg, I know this is more vague rambling on my part.  Let me make a
> specific example.  Somewhere in your patch (as noted while catching
> up on my e-mail today) is a filter function that is making a test
> on filter->r->connection->aborted.  That is horribly wrong.

Are you referring to the ap_l* functions? The filter callbacks never do.

(or are you thinking of Ryan's patches? in one of his macros, which is used
in the filter callbacks, they refer to the aborted flag)

I'm going to assume you *are* talking about the ap_l* functions... below...

> Consider the case where you have an active proxy implemented in the form
> of a filter where you have one input filter chain reading the inbound
> response and passing it downstream toward the client.  What r?  Which
> connection?  Why the hell does the filter need to know anything about
> the endpoints, let alone assume they are sockets?

The filter doesn't need to know (and doesn't today). The framework doesn't
need to know (but does today). I can remove those tests quite easily. But in
the *current* Apache, it makes a lot of sense. It allows the filter chain to
short-circuit the processing when it finds out that it is no longer needed.

Is that too much knowledge of the endpoint? Sure. But it isn't permanent,
and it certainly isn't a requirement of the design. If the whole chain just
kept going and shoved everything down to BUFF where it finally got
discarded, it would still work fine.

> The patch may work fine, but the design doesn't.  Data flow networks are
> a very well-defined and understood software architecture.  They have a
> single, very important constraint: no filter is allowed to know anything
> about the nature of its upstream or downstream neighbors beyond what
> is defined by the filter's own interface.  That constraint is what makes
> data flow networks highly configurable and reusable.  Those are properties
> that we want from our filters.

Sigh. Roy: none of my example filters have used any knowledge of the end
points. Never have the callbacks ever checked ->aborted. I believe my filter
design matches your requirements.

> The second problem here is that this discussion contains too many
> messages.  Both of you seem to have a need to respond to every message
> on a point-by-point basis,

I know there are too many messages. But look at it from my point of view.
You come along and post a relatively singular message. People that have
tuned out of the discussion stop to read it ("ooh. roy posted. maybe it is
time to see what is happening."). Then they read about how you don't like my
patch because it only provides the char* interface and does not implement
the bucket brigades.

What am I to do? Just let that impression go and sink in? Or do I try to
educate the readers of these emails that, *yes* it *does* have buckets?

When Ryan says the recursive-call design will suck performance wise, yet
that whole "register spill" thing truly doesn't exist in these scenarios?
Let that pass? Give people the wrong impression?

I feel it is important that people understand that I've posted a patch that
meets (IMO) our needs. If they know that, then they may stop and actually
look at the thing and provide one of the needed +1 votes. But if every
message has some comment about how it is broken here or there, or doesn't
meet this need or that, then they will keep skipping over it. We'll just
remain in limbo...

>...
> What you don't seem
> to be realizing is that you have generated so much text that the only people
> who have enough time to both read the discussion and propose alternatives
> are the two of you.  The rest of us just can't keep up.

Oh, I totally understand. That is why I've been posted things with titles
such as "what are the issues?" or "recap" or "summary". To boil all this
stuff down so that (hopefully) people can jump in at that point.

> I'd love to spend
> some time flushing out the bucket brigades design, but instead I am spending
> twelve hours a day reading the e-mail that gets stacked-up behind days
> when I'm travelling, teaching classes, or otherwise doing what my real
> job requires.  I don't want you to slow down, but for crissakes please
> boil down the discussion to the meat and ignore the side-comments if
> they are already covered by a prior message.

I get the feeling that you didn't see the ap_bucket_t stuff in my patch.
Maybe you looked at the very first one? Please look at the latest, available
in the archive with MsgID <20...@lyra.org>. There are some
examples of the filtering scheme in <20...@lyra.org>.

If you could spend just 30 minutes looking over that. Again: it is only 850
lines, most of it quite straight-forward. But sorry: no comments yet.

If that patch does not meet your bucket-brigades thoughts, or you can see
where it can't *ever* meet them, then please explain to me. I will
definitely listen and fix the code.

> And I don't see any reason why we should hurry to get this in 2.0.
> Not even my own personal favorite design is worth rushing into the
> code base -- I'd rather make sure that the existing server works.

And we all do things that are interesting and important to each one of us.
Both Ryan and myself think this would be a really cool, kickass feature for
2.0, and we are both willing to spend a lot of time to see it happen.

I'm guessing that *something* is going to go into 2.0. I hope that you can
at least spend a small amount of time to review the posted patch, and
provide your thoughts. If this is dear to you, and something is going in,
then you'll want to have a look :-)

thx,
-g

-- 
Greg Stein, http://www.lyra.org/

Re: another response to Roy :-)

[Greg Stein <gs...@lyra.org>]

On Sun, Jul 02, 2000 at 10:55:42PM -0700, Dirk-Willem van Gulik wrote:
> On Fri, 30 Jun 2000, Greg Stein wrote:
> 
> >> ..Roy wrote..
> 
> > I don't understand this. What are the "problems of buff" that my patch
> > introduces? And I really don't understand the fragility point. It started
> 
> Interesting reply; as _You_ did not introduce them; we've been struggling
> with them from the days of shambala. 

Gotcha. Well, the BUFF is effectively a layer/filter in my updated patch. I
didn't register a filter for it, nor add it to the response (to minimize the
number of files affected, and to avoid thinking about the proper point in
the control flow to do so). However, inside of http_protocol.c, it
dispatches to BUFF_filter_callback() exactly as if it were a true filter.

Point being, the filter chain doesn't know about BUFF and just tosses the
data at it via another filter. This independence means that we can alter and
tweak BUFF at will (an BUFF_filter_callback's interaction with it) without
affecting the rest of the filter chain.

I did not want to tackle fixing/changing BUFF at *this* point in time.
Creating a clean separation is the first step. There are going to be other
tweakies such as the various points in Apache where we modify the
translations in BUFF, modify the chunking flags, alter timeouts, etc.
Gathering those up is another hunk of work.

> Perhaps this is the core of the issue at hand; some of us want to solve
> some long standing problems whilst adding a layer whereas others just want
> a layer which does what it should do.

I couldn't parse this sentence :-)

Yes, I think cleaning up BUFF would be a Great Thing(tm), too. But it isn't
easy, and a good framework is going to be the first step. I'm quite wary of
it, as the filtering patches' first foray into this area introduced BUFF
bugs. I'd like to see the filters added, then we can start to take pieces
out of BUFF and up into the filter chain; but carefully and individually
reviewable. I've started a mod_xlate to take over the xlate features of
BUFF. I'll post it as an example once it is done.

Cheers,
-g

-- 
Greg Stein, http://www.lyra.org/

Re: another response to Roy :-)

[Dirk-Willem van Gulik <di...@covalent.net>]

On Fri, 30 Jun 2000, Greg Stein wrote:

>> ..Roy wrote..

> I don't understand this. What are the "problems of buff" that my patch
> introduces? And I really don't understand the fragility point. It started

Interesting reply; as _You_ did not introduce them; we've been struggling
with them from the days of shambala. 

Perhaps this is the core of the issue at hand; some of us want to solve
some long standing problems whilst adding a layer whereas others just want
a layer which does what it should do.

Dw