Cocoon 2 assumptions

[Mark Washeim <es...@canuck.com>]

I have one comment regarding some underlying assumptions about 'big sites'.
Explicitly, in the quote, below:
-- Quote --
Overlapping contexts and Chain Mapping 

In Cocoon2, the reactor pattern will be abandoned in favor of a chain
mapping technique. This is based on the fact that the number of different
contracts is limited even for big sites (for example, even if the pages are
millions, they probably all share no more than a few different DTDs and each
DTD has no more than a couple of stylesheets).
--

I am responsible for several such big sites (for instance saab.com and all
the market variants, accounting for 18 markets/nations/languages and, sadly,
marketing departments ). In almost all of the cases I've encountered, the
modeling of business information (even in such traditionally disciplined
areas as IT departments) is deficient.

That's an understatement. The very process of trying to create rationale,
organizational document types (or schemas in a table context) is practically
impossible in all but the most disciplined organizations (which are few and
far between). In addition the demand to add arbitrary (usually marketing
dept. driven) data, with little mind paid to the context and consequences
(these people are driven by 'appearances', not rational thinking) often
causes additional proliferation of document types.

While my companies have succeeded (from a financial vantage point, sadly,
not often from an engineering vantage point) in being able to respond very
quickly to these demands, we are, really, failing our clients. Failing in
that, as the data becomes less structured, it promises to be of less and
less use over time. Of course, they should listen to their own IT
departments and impose some discipline.

In any case, I'm not sure what the answer is. I'm attempting to impose some
discipline on clients by extending one independently (Localization Industry
Standards Association, tmx.dtd) maintained dtd. We are attempting to extend
this by use of a 'catalog' of entities referred to as an addendum (to
tmx.dtd), hence minimizing the deterioration of the base dtd but permitting
ease of extending. That, however, also alows us (or pushes us) to use these
'document fragments' in other contextes (ie, to create a dom from the
catalog for an editing application constrained by a limited context, eg. the
HR departments subset of documents). We're also looking at incorporating
many of the other, standards bodies maintained dtds (by reference). Namely,
such banalities as the purchase order (OMG & OASIS are starting to bulk up,
as are the industry associations).

In any case, I'm a little concerned about the assumptions stated above.
While the assumption is true of very big sites like yahoo.com, it is not
true of many others. Even in the case of Mercedes Benz (in Germany, not the
US) with whom I've worked, where the IT department has been very careful
about this (ie, they used internal business process docs for all business
units to attempt to create, at the time, table schemas and UI guidelines for
common deptartmental tasks) they have failed, often, to maintain the
integrity of data coming into and out of the enterprise.

Sigh, this is a verbose letter to post. I hope it isn't a waste of all of
your energies.

Thanks for all your extraordinary efforts and regards
-- 
Mark (Poetaster) Washeim
Technical Director,
Large Medium, Critical Mass
'In Xanadu did Kublai Khan
 A stately pleasure dome decree . . .'

 

Re: Cocoon 2 assumptions

[Stefano Mazzocchi <st...@apache.org>]

Mark Washeim wrote:
> 
> I have one comment regarding some underlying assumptions about 'big sites'.

Cool.

> Explicitly, in the quote, below:
> -- Quote --
> Overlapping contexts and Chain Mapping
> 
> In Cocoon2, the reactor pattern will be abandoned in favor of a chain
> mapping technique. This is based on the fact that the number of different
> contracts is limited even for big sites (for example, even if the pages are
> millions, they probably all share no more than a few different DTDs and each
> DTD has no more than a couple of stylesheets).
> --
> 
> I am responsible for several such big sites (for instance saab.com and all
> the market variants, accounting for 18 markets/nations/languages and, sadly,
> marketing departments ). In almost all of the cases I've encountered, the
> modeling of business information (even in such traditionally disciplined
> areas as IT departments) is deficient.

I have, by far, less responsibility than that, but I think I imagine
your situation... it's no fun at all  :(
 
> That's an understatement. The very process of trying to create rationale,
> organizational document types (or schemas in a table context) is practically
> impossible in all but the most disciplined organizations (which are few and
> far between). 

I'm doing consulting for a company involved in military research and
web-based information systems... luckily, they have better disciplines.
:) In fact, they come from the SGML world...

> In addition the demand to add arbitrary (usually marketing
> dept. driven) data, with little mind paid to the context and consequences
> (these people are driven by 'appearances', not rational thinking) often
> causes additional proliferation of document types.

DTD proliferation is a plague. _many_ people in the SGML world know
this. I'm far too young to be a "guy from the SGML world", but it's not
difficult to see why this should be avoided like a disease.
 
> While my companies have succeeded (from a financial vantage point, sadly,
> not often from an engineering vantage point) in being able to respond very
> quickly to these demands, we are, really, failing our clients. Failing in
> that, as the data becomes less structured, it promises to be of less and
> less use over time. Of course, they should listen to their own IT
> departments and impose some discipline.

Look: DTDs are contracts. The most economically-successful company in
the history of mankind based his entire business model on breaking
contracts and forcing the market to adopt each new one. Microsoft Office
would not be continously upgraded by companies if Work 2.0 and Work 2000
formats were the same, resulting in billions of dollars of missed gain.

Of course, if the technological contracts fail, the economical contracts
back them up. In some senses, since economical contracts are much more
manageable and understandable by the management-type people, these are
the most solid, even if they are _by_far_ more expensive.

This is, unfortunately, something that cannot be fixed technologically,
unless, people like us get to the point where they have to plan the
creation and maintenance of working contracts: some believe in
commercial software, some don't and believe in open standards, even if
created by a mass of geeks like us.

Who wins? There is no global equation. Unlike some radical people, I
don't believe open source is the solution to everything... but it helps
when it imposes good standards and keeps control on the technological
contracts, reducing your costs of ownership...
 
> In any case, I'm not sure what the answer is. I'm attempting to impose some
> discipline on clients by extending one independently (Localization Industry
> Standards Association, tmx.dtd) maintained dtd. 

DTDs need to be created and maintained by independent organizations, or,
if the company is big enough (for example, NATO), create their own.

> We are attempting to extend
> this by use of a 'catalog' of entities referred to as an addendum (to
> tmx.dtd), hence minimizing the deterioration of the base dtd but permitting
> ease of extending. That, however, also alows us (or pushes us) to use these
> 'document fragments' in other contextes (ie, to create a dom from the
> catalog for an editing application constrained by a limited context, eg. the
> HR departments subset of documents). We're also looking at incorporating
> many of the other, standards bodies maintained dtds (by reference). Namely,
> such banalities as the purchase order (OMG & OASIS are starting to bulk up,
> as are the industry associations).

Mark,

unfortunately, this is new technology. And creating good contracts that
last takes time... lots of time.
 
> In any case, I'm a little concerned about the assumptions stated above.
> While the assumption is true of very big sites like yahoo.com, it is not
> true of many others. Even in the case of Mercedes Benz (in Germany, not the
> US) with whom I've worked, where the IT department has been very careful
> about this (ie, they used internal business process docs for all business
> units to attempt to create, at the time, table schemas and UI guidelines for
> common deptartmental tasks) they have failed, often, to maintain the
> integrity of data coming into and out of the enterprise.

This is another problem. In another consulancy I've made in Germany, we
developed what we called "Adaplets" which are responsible to provide
"whatever -> XML" translation. These adaplets are developed to "adapt"
existing formats or data into the "wisely designed" xml-based system
which, in turn, is designed on very few DTDs (something around 5 for a
_huge_ system).
 
> Sigh, this is a verbose letter to post. I hope it isn't a waste of all of
> your energies.

Not at all, but I don't change my mind: I believe that a good
information system requires DTDs that are a function of the logarithm of
the amount of documents hosted plus a site-dependent DTD starting
number.

This means that a 100 pages site might well have 20 DTDs, but if this
grows to 1.000.000 pages the DTDs grow to something around 30 DTDs, not
more.

Of course, I'm talking for "well behaving" systems. If this is not the
case, other possible scenarios are

1) DTDs grow as SQRT(pages) 
2) DTDs grow as pages

As you see, logarithm, sqrt and linear are the three possible flavors,
here is a simple analisys based on my experience:

1) dtd = log(pages) -> the best and most scalable solution. Requires
_very_ strict discipline on information control. XML validation helps a
lot, but this purity control has a cost and normally not all companies
are willing to pay it, even if turns out to be a lot less expensive in
the long run. In real life, I assume very few companies are able to do
this... but in the future, more and more will adopt strict policies and
abandon soft-contract-based information systems (MS Office, HTML)

2) dtd = (pages)^-n -> something in between. Not optimum, but still
manageable. The °n° coefficient is a measure of the ability to adopt
strict politicies on information validation and adaptation from external
sources. The higher the coefficient, the better the internal politices
and system design, the fewer the number of DTDs, the fewer the
information management costs.

3) dtd = pages -> a total nightmare: I believe no company will survive
on having such a small information efficiency coefficient.

So, in conclusion, you're right: the quoted statement is very optimistic
and reflects an optimal solution. Still, I do believe the Cocoon sitemap
model will work well even in cases where the dtd function is not
logarithmical but has a polynomial coefficient high enough to make the
information management costs "reasonable" for the company involved.

Of course, the bigger the company, the lower the coefficient. Or, in
other terms: your mileage may vary :)

> Thanks for all your extraordinary efforts and regards

You're welcome :)

-- 
Stefano Mazzocchi      One must still have chaos in oneself to be
                          able to give birth to a dancing star.
<st...@apache.org>                             Friedrich Nietzsche
--------------------------------------------------------------------
 Come to the first official Apache Software Foundation Conference!  
------------------------- http://ApacheCon.Com ---------------------