Re: Tomcat 4.0 FAB

[Roy Wilson <de...@bellatlantic.net>]

Jason,

I am a geek aspiring to be a Catalina-performance-geek, but I know I 
appreciated your marketing-speak more than I'm supposed to. :-) 

You mention that the built-in webserver performs well under heavy load. I 
don't doubt it, but would like to know the data source (ApacheCon 
Europe?) and see numbers. In particular, how was the load generated? 
ApacheBench?

Roy Wilson
E-mail: designrw@bellatlantic.net

Re: Can market-first/measure-later sustain TC growth?

[Jason Brittain <ja...@olliance.com>]

Hi again Roy.

Roy Wilson wrote:
> 
> Jason,
> 
> A matter of small importance, and of philosophy, but ...
> 
> <snip>
> 
> > The great thing about this is that
> > it gets some people jazzed about the project without making any
> > real, solid claims.
> 
> I think I'm ok on this.
> 
> > And, about the sentence that says "performs well under heavy
> > load, even compared to the native Apache web server"..  This
> > works the same way, since we're not saying which web server,
> > nor running on which operating system, nor under what conditions.
> > We're not even saying a major version number.  So why is it in
> > here?  Because, I heard through the grapevine that someone ran
> > some meaningful (but not necessarily bullet-proof) performance
> > tests and had this finding: that the version of Catalina that
> > was tested at the time performed about as well on most tests
> > as the version of their native Apache web server did on the same
> > tests.
> 
> But what if there happens to also be in the gv a story about someone who
> has run some meaningful performance tests and found the contrary? To mix
> metaphors, the can-o-worms that you don't want to open gets opened
> anyway, does it not?

So if someone else wants to dispute this claim, they'll first run
into the fact that we're not making any exact claims in the FAB,
only relative statements.  Then, if they want to try to make some
kind of exact claim that Catalina's web server doesn't perform well
in comparison to some version (that they choose) of the Apache web
server, then they'll first need to run some meaningful benchmarks
and publicly post the results.  Most people know that benchmarks
are often misleading, and are often swayed in favor of whoever runs
the benchmark.  So, that would only cause more talk about how the
benchmark should have been tuned, or which tests really should have
been run, etc..  And, I think that's all good because the result
is that more people are thinking and talking about Catalina.  :)
We learn more about how to tune it, how to test it, etc, all in
comparison to the native Apache web server.

The end result of it all is that it is our word against theirs,
and people will believe what they want to believe and what they
can prove on their own machine.  That's about it.
 
> > I personally don't want to spend time validating that,
> > nor would I want to be the one to try to defend the results
> > (it would likely turn into another "Java's not as fast as C"
> > arguement).
> 
> I agree with this, sort of.
> 
> > But, I believe it's possible, and I also believe
> > that Catalina's web server performs quite nicely.
> 
> I am too am inclined to think that Catalina's webserver will perform well
> enough to satisfy someone somewhere, but aren't you claiming more?

This is one thing I find awkward about marketing-speak: it sure
seems like it makes wild, solid claims, but in reality it doesn't.

> > Everyone should keep in mind that the purpose of our FAB is not
> > to prove any performance numbers or to help engineers to
> > understand Catalina's deep implementation details.  It's to make
> > some not-so-technically-saavy readers want to help the project,
> > promote the project, hire people to get it running and use it,
> > etc.  We were primarily thinking of people like:
> 
> >  - CEOs, CTOs, and other management personnel
> >  - Venture Capitalists
> >  - Marketing staff
> >  - [insert random Joe "Non-Technical" Businessman here]
> 
> I agree with the language used for the target audience, but see below.
> 
> > Reallly, people who don't know much about why their company
> > should be interested in this thing called Catalina who aren't
> > technically saavy enough to understand other docs that they
> > can find about it (like the User docs, or other technical docs
> > like the Classloader explanation doc or even the status doc..)
> > How else can we each get permission to work full-time on cool
> > projects like Catalina?  The better Catalina looks to both the
> > technical and non-technical population at large, the more support
> > the project will get overall, I think.
> 
> I think that the marketing-speak can't be broadcast until there is
> technical data to support it in some fashion. Otherwise, perhaps, some
> CEO/CTO/etc., asks a question that ultimately gets answered by a
> technically savvy person such as yourself who (in the absence of
> technical performance docs) might then be forced to reply "there's no
> !#%?! evidence!"

So then the Boss's next reply would be "Well then benchmark it and
have the results ready by friday.", and that means they'll download
it, install it, figure out some way of benchmarking it, and compare
with Apache.  If they get bad numbers, then they'll probably think
that we know something they don't.  And, we just might.

Also, having no "solid" numbers to back up a claim that's actually
a relative statement won't necessarily mean that we're just plain
wrong.

Besides, I think those who benchmark will find that it performs
quite comparably to the native Apache web server of their choice.
Notice, though, that the FAB makes no mention that currently the
Catalina web server doesn't have a plethora of configuration
features like the native Apache web server has.  Of course the
Catalina web server should have less features at this point, it's
a much younger project.  So, even benchmarking doesn't give you
the full picture.  You have to decide what kind of a web server
you need for any given project.  Performance isn't always everything
(but it sure is nice! :).

> I don't see how a market-first/measure-later approach can sustain growth
> in Tomcat support. But, then, maybe you guys are simply getting the
> market-speak in place :-).

Yes, for us it was mostly a matter of having our ducks in a row.
We wanted to be prepared for non-techies asking us the question
"So what's this Catalina thing you mentioned, and why should I
care?".  That was our real motive for writing this kind of text.

--
Jason Brittain
Software Engineer, Olliance Inc.        http://www.Olliance.com
Current Maintainer, Locomotive Project  http://www.Locomotive.org

Can market-first/measure-later sustain TC growth?

[Roy Wilson <de...@bellatlantic.net>]

Jason,

A matter of small importance, and of philosophy, but ...

<snip>


> The great thing about this is that
> it gets some people jazzed about the project without making any
> real, solid claims.

I think I'm ok on this.

> And, about the sentence that says "performs well under heavy
> load, even compared to the native Apache web server"..  This
> works the same way, since we're not saying which web server,
> nor running on which operating system, nor under what conditions.
> We're not even saying a major version number.  So why is it in
> here?  Because, I heard through the grapevine that someone ran
> some meaningful (but not necessarily bullet-proof) performance
> tests and had this finding: that the version of Catalina that
> was tested at the time performed about as well on most tests
> as the version of their native Apache web server did on the same
> tests.

But what if there happens to also be in the gv a story about someone who 
has run some meaningful performance tests and found the contrary? To mix 
metaphors, the can-o-worms that you don't want to open gets opened 
anyway, does it not?  

> I personally don't want to spend time validating that,
> nor would I want to be the one to try to defend the results
> (it would likely turn into another "Java's not as fast as C"
> arguement).  

I agree with this, sort of.

> But, I believe it's possible, and I also believe
> that Catalina's web server performs quite nicely.

I am too am inclined to think that Catalina's webserver will perform well 
enough to satisfy someone somewhere, but aren't you claiming more?  

> Everyone should keep in mind that the purpose of our FAB is not
> to prove any performance numbers or to help engineers to
> understand Catalina's deep implementation details.  It's to make
> some not-so-technically-saavy readers want to help the project,
> promote the project, hire people to get it running and use it,
> etc.  We were primarily thinking of people like:

>  - CEOs, CTOs, and other management personnel
>  - Venture Capitalists
>  - Marketing staff
>  - [insert random Joe "Non-Technical" Businessman here]

I agree with the language used for the target audience, but see below.

> Reallly, people who don't know much about why their company
> should be interested in this thing called Catalina who aren't
> technically saavy enough to understand other docs that they
> can find about it (like the User docs, or other technical docs
> like the Classloader explanation doc or even the status doc..)
> How else can we each get permission to work full-time on cool
> projects like Catalina?  The better Catalina looks to both the
> technical and non-technical population at large, the more support
> the project will get overall, I think.

I think that the marketing-speak can't be broadcast until there is 
technical data to support it in some fashion. Otherwise, perhaps, some 
CEO/CTO/etc., asks a question that ultimately gets answered by a 
technically savvy person such as yourself who (in the absence of 
technical performance docs) might then be forced to reply "there's no 
!#%?! evidence!" 

I don't see how a market-first/measure-later approach can sustain growth 
in Tomcat support. But, then, maybe you guys are simply getting the 
market-speak in place :-).

Roy

-- 
Roy Wilson
E-mail: designrw@bellatlantic.net

Re: Tomcat 4.0 FAB

[Jason Brittain <ja...@olliance.com>]

Hi Roy.

Roy Wilson wrote:
> 
> Jason,
> 
> I am a geek aspiring to be a Catalina-performance-geek, but I know I
> appreciated your marketing-speak more than I'm supposed to. :-)

:)  We're glad you liked it.

> You mention that the built-in webserver performs well under heavy load. I
> don't doubt it, but would like to know the data source (ApacheCon
> Europe?) and see numbers. In particular, how was the load generated?
> ApacheBench?
> 
> Roy Wilson
> E-mail: designrw@bellatlantic.net

Here's the paragraph from our Features and Benefits doc that you're
referring to:

----snip------snip------

Built-in Multiplatform High Performance Web Server

Along with great servlet container features, Catalina comes with its
own high performance HTTP 1.1 web server built in. This new web server,
written entirely in Java, makes it easy to try Catalina without
having to configure a third-party web server. Users can thus deploy
Catalina as a functionally complete application server solution
(excluding database server) without a serious performance
penalty. The built-in web server performs well under heavy load,
even compared to the native Apache web server. And because Catalina
is written completely in Java, it's fully multiplatform, running
on Linux, Windows, Solaris or any other operating system that supports
the Java 2 Platform.

----snip------snip------

Now, keep in mind that one property of marketing-speak is that
everything is relative, and that phrases that sound like exact
claims are nothing of the kind.  :)  "high performance" is a
relative term, just as "performs well under heavy load" is,
since the text makes no claims about what "high" or "well" or
"heavy" really means.  The reason why I think this kind of text
works so well on some folks is that they make the assumption
that "high" means what they (the reader) thinks it means, and
that the writer is making a claim about that kind of performance.
In reality, this text doesn't mean much, except that it means
to boast about Catalina.  :)  The great thing about this is that
it gets some people jazzed about the project without making any
real, solid claims.

And, about the sentence that says "performs well under heavy
load, even compared to the native Apache web server"..  This
works the same way, since we're not saying which web server,
nor running on which operating system, nor under what conditions.
We're not even saying a major version number.  So why is it in
here?  Because, I heard through the grapevine that someone ran
some meaningful (but not necessarily bullet-proof) performance
tests and had this finding: that the version of Catalina that
was tested at the time performed about as well on most tests
as the version of their native Apache web server did on the same
tests.  I personally don't want to spend time validating that,
nor would I want to be the one to try to defend the results
(it would likely turn into another "Java's not as fast as C"
arguement).  But, I believe it's possible, and I also believe
that Catalina's web server performs quite nicely.

Everyone should keep in mind that the purpose of our FAB is not
to prove any performance numbers or to help engineers to
understand Catalina's deep implementation details.  It's to make
some not-so-technically-saavy readers want to help the project,
promote the project, hire people to get it running and use it,
etc.  We were primarily thinking of people like:

 - CEOs, CTOs, and other management personnel
 - Venture Capitalists
 - Marketing staff
 - [insert random Joe "Non-Technical" Businessman here]

Reallly, people who don't know much about why their company
should be interested in this thing called Catalina who aren't
technically saavy enough to understand other docs that they
can find about it (like the User docs, or other technical docs
like the Classloader explanation doc or even the status doc..)
How else can we each get permission to work full-time on cool
projects like Catalina?  The better Catalina looks to both the
technical and non-technical population at large, the more support
the project will get overall, I think.

--
Jason Brittain                          (415)354-6645
Software Engineer, Olliance Inc.        http://www.Olliance.com
Current Maintainer, Locomotive Project  http://www.Locomotive.org