You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@xml.apache.org by Andy Clark <an...@apache.org> on 2002/07/17 17:58:40 UTC
[Announce] The CyberNeko Tools for XNI 2002.07.17 Available
Despite recent mail server problems that kept me off the
list for a few days (and probably returned a bunch of mail
to their senders), I've been hard at work on NekoXNI. Now
I am pleased to announce the release of the CyberNeko
Tools for XNI version 2002.07.17.
This release fixes a variety of small problems and adds
new features and new tools to allow users to use Xerces2
to validate documents using Relax NG.
NekoDTD was updated to include stylesheets to convert DTD
grammars to Relax NG. Also, this release adds a new tool,
called ManekiNeko, that includes a parser configuration
built with a Relax NG validator instead of the standard
DTD/XML Schema validators.
<note>
ManekiNeko does not implement a Relax NG validator. Instead,
it wraps James Clark's Jing validator by converting between
XNI and SAX events. (Even if you don't use RelaxNG, this code
can be used by any application that wants to embed a SAX2
filter in an XNI pipeline.)
</note>
It is my hope that the new tools will enable existing Relax
NG fans to more easily use Relax NG in their applications
as well as introduce this wonderful XML schema language to
a wider audience.
* * *
The following list details all of the changes made to the
NekoXNI tools in this release:
NekoHTML 0.6.5
* Fixed bug in changing character encoding when
"charset=..." is not written in lowercase; and
* mark attributes as "specified".
NekoDTD 0.1.1
* Fixed bugs in DTDx to DTD stylesheet for comments,
processing instructions, and ignore conditionals;
* fixed minor bug in DTDx to Flat stylesheet;
* fixed minor bug in Flat to XML Schema stylesheet; and
* added Flat to RelaxNG stylesheets and documentation.
NekoStyle 0.2
* Modified processor API so to allow relative system
identifiers to be used within documents; and
* added ability to set properties within pipeline scripts.
ManekiNeko 0.1
* Initial writing.
You can download and use each project independently or
download the entire package from the following URL:
http://www.apache.org/~andyc/neko/
Enjoy!
* * *
P.S. In the pipeline for NekoXNI is a pull-parser API and
implementation based on Xerces2 and XNI. I have already
written a buffering parser configuration that guarantees
that one and only one event is returned for each call to
XMLPullParserConfiguration#parse(boolean):boolean. (This
is very important in order to make a true pull-parsing
implementation built from Xerces2 and XNI work correctly.)
Next, I plan on designing a pull-parser API that builds
on the Xerces Native Interface. This API is not meant to
be a competitor to existing pull-parsing projects such
as XPP. Rather, it is meant to show the flexibility and
power of the XNI framework to address other parsing needs
while leveraging the existing library of XNI components.
--
Andy Clark * andyc@apache.org
---------------------------------------------------------------------
In case of troubles, e-mail: webmaster@xml.apache.org
To unsubscribe, e-mail: general-unsubscribe@xml.apache.org
For additional commands, e-mail: general-help@xml.apache.org
allowing client to control buffer use policy [Re: feature to reuse
buffer]
Posted by Aleksander Slominski <as...@cs.indiana.edu>.
Andy Clark wrote:
> Aleksander Slominski wrote:
> > i have implemented something like this already by wrapping
> > real reader with special reader that is cumulating input when
> > asked for. that allows me to keep buffer offsets from
> > XMLString and uses content in cumulative reader buffer
> > even though Xerces may reuse its internal buffer
> > (see below how i implemented CumulativeReader - it
> > works but is inefficient as it adds extra layer of indirection
> > and another buffer potentially for all IO operations).
>
> The inefficiency is precisely the reason why I wouldn't
> want to use this type of approach. Besides, this approach
> has a hard limit of Integer.MAX_VALUE.
hi,
typical usage pattern is to lock buffer when interesting
event happens (for example first call to characters())
and then unlock to retrieve content (for example in endElement())
- that means that client does not need to copy
buffer content (no need to accumulate characters()
content in StringBuffer etc.).
when unlocked the content of the buffer has exactly
the same semantics as currently (when unlocked buffer can be shrank)
however when locked it is then guaranteed that offsets
into buffer content are valid until buffer is unlocked.
therefore only if buffer is locked permanently it will grow indefinitely.
anyway i think it is very useful feature.
> The approach I'm thinking of simply manages the creation
> of buffers, not what is done with them. So the various
> parts of the code would still be in control of their own
> buffers but would simply request new buffers from the
> buffer manager/factory/whatever.
however that means that making sure that
buffer can be safely reused will be very difficult - right?
i think that if buffers are not reused then xerces may
be facing memory problems when parsing large XML input
(for example if client keeps reference to buffers
passed in characters())?
thanks,
alek
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
Re: feature to reuse buffer [Re: [Announce] The CyberNeko Tools forXNI
2002.07.17 Available]
Posted by Andy Clark <an...@apache.org>.
Aleksander Slominski wrote:
> i have implemented something like this already by wrapping
> real reader with special reader that is cumulating input when
> asked for. that allows me to keep buffer offsets from
> XMLString and uses content in cumulative reader buffer
> even though Xerces may reuse its internal buffer
> (see below how i implemented CumulativeReader - it
> works but is inefficient as it adds extra layer of indirection
> and another buffer potentially for all IO operations).
The inefficiency is precisely the reason why I wouldn't
want to use this type of approach. Besides, this approach
has a hard limit of Integer.MAX_VALUE.
The approach I'm thinking of simply manages the creation
of buffers, not what is done with them. So the various
parts of the code would still be in control of their own
buffers but would simply request new buffers from the
buffer manager/factory/whatever.
--
Andy Clark * andyc@apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
Re: feature to reuse buffer [Re: [Announce] The CyberNeko Tools forXNI
2002.07.17 Available]
Posted by Aleksander Slominski <as...@cs.indiana.edu>.
Andy Clark wrote:
> Aleksander Slominski wrote:
> > another great feature would be ability to "lock" parser
> > buffer content so when i receive multiple character() call-backs
> > i know that XMLString offset and length will be valid
> > as parser will just create bigger buffer to keep larger content.
>
> Sounds like you're asking for a more advanced buffer
> management system built into the parser. I was looking
> at implementing the "don't re-use character buffers"
> feature today and it's not as easy as I had hoped. So...
i have implemented something like this already by wrapping
real reader with special reader that is cumulating input when
asked for. that allows me to keep buffer offsets from
XMLString and uses content in cumulative reader buffer
even though Xerces may reuse its internal buffer
(see below how i implemented CumulativeReader - it
works but is inefficient as it adds extra layer of indirection
and another buffer potentially for all IO operations).
> I may end up implementing a system where the application
> can register a buffer manager and all of the components
> would ask the manager for new buffers, etc. Then, using
> this method, the Xerces2 parser would by default re-use
> buffers but a pull parser could install a new buffer
> manager that always created new buffers.
>
> But I need to do some more thinking on the topic...
that would be great!
i think that this is important feature to tune for perfromance.
thanks,
alek
ps. here is how i implemented cumulative reader:
protected class CumulativeReader extends Reader
{
private Reader source;
private boolean cumulative;
private int bufAbsoluteStart;
private int bufAbsoluteEnd;
private char[] buf = new char[10 * 1024];
private int bufStart;
private int bufEnd;
/** Constructs this reader from another reader. */
public CumulativeReader(Reader reader) {
source = reader;
}
public void setCumulative(boolean value) {
cumulative = value;
if(cumulative) {
char[] newBuf = new char[buf.length + 10 * 1024];
if(bufEnd > bufStart) {
System.arraycopy(buf, bufStart, newBuf, 0, bufEnd - bufStart);
}
bufEnd = bufEnd - bufStart;
bufStart = 0;
}
}
public boolean getCumulative() { return cumulative; }
public char[] getCumulativeBuffer() {
return buf;
}
public int getCumulativeBufferAbsoluteStart() {
return bufAbsoluteStart;
}
public int getCumulativeBufferAbsoluteEnd() {
return bufAbsoluteEnd;
}
public int getCumulativeBufferStart() {
return bufStart;
}
public int getCumulativeBufferEnd() {
return bufEnd;
}
//
// Reader methods
//
// ignore closing
public void close() { }
public int read(char[] ch, int offset, int length)
throws IOException
{
// read form original
int ret = source.read(ch, offset, length);
if(ret > 0) {
if(!cumulative) {
buf = ch;
bufStart = offset;
bufEnd = offset + ret;
bufAbsoluteStart = bufAbsoluteEnd;
} else {
// append ch to buf at bufAbsoluteEnd
int newLen = bufEnd + length;
if(buf.length < newLen) {
char[] newBuf = new char[newLen + 10 * 1024];
System.arraycopy(buf, bufStart, newBuf, 0, bufEnd - bufStart);
bufEnd = bufEnd - bufStart;
bufStart = 0;
}
System.arraycopy(ch, offset, buf, bufEnd, ret);
bufEnd += ret;
}
bufAbsoluteEnd += ret;
}
return ret;
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
Re: feature to reuse buffer [Re: [Announce] The CyberNeko Tools for
XNI 2002.07.17 Available]
Posted by Andy Clark <an...@apache.org>.
Aleksander Slominski wrote:
> another great feature would be ability to "lock" parser
> buffer content so when i receive multiple character() call-backs
> i know that XMLString offset and length will be valid
> as parser will just create bigger buffer to keep larger content.
Sounds like you're asking for a more advanced buffer
management system built into the parser. I was looking
at implementing the "don't re-use character buffers"
feature today and it's not as easy as I had hoped. So...
I may end up implementing a system where the application
can register a buffer manager and all of the components
would ask the manager for new buffers, etc. Then, using
this method, the Xerces2 parser would by default re-use
buffers but a pull parser could install a new buffer
manager that always created new buffers.
But I need to do some more thinking on the topic...
--
Andy Clark * andyc@apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
feature to reuse buffer [Re: [Announce] The CyberNeko Tools for XNI
2002.07.17 Available]
Posted by Aleksander Slominski <as...@cs.indiana.edu>.
Andy Clark wrote:
> So I'm thinking of adding a feature to Xerces2 that
> allows applications to set whether the scanner re-
> uses its character buffers. If the scanner doesn't
> reuse buffers, then my converter doesn't have to
> copy anything. Therefore, the performance of the
> pull-parser driven by Xerces2 should be better.
hi,
that would be great- i have hit this problem when
implementing XMLPULL API on top of Xerces 2 XNI.
another great feature would be ability to "lock" parser
buffer content so when i receive multiple character() call-backs
i know that XMLString offset and length will be valid
as parser will just create bigger buffer to keep larger content.
then later i unlock allowing parser to reuse current buffer
(so parser can start writing to buffer from beginning).
it would be interesting to compare if such reuse of growable
buffer is more efficient then fixed buffer (AFAIK it is current
implementation in Xerces) and the option you describe when
scanner never reuses the buffer - how does it sound?
> Of course, you could always write an implementation
> only for the pull-parsing API that wouldn't suffer
> this problem at all. You could even drive the API
> from other pull-parsing APIs like XPP, etc. :)
that is where good layering becomes important for
good performance (lower level must be efficient :-))
and i think about XML pull parsing as being
quite low level and very efficient (similarly to SAX),
natural layer just above XML tokenizer that
can be used to build higher levels efficiently
(like increment xml node trees).
thanks,
alek
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
Re: [Announce] The CyberNeko Tools for XNI 2002.07.17 Available
Posted by Andy Clark <an...@apache.org>.
Elliotte Rusty Harold wrote:
> You are working under the misapprehension that this is your choice to
> make. It isn't. No matter how valid your criticisms of XML are, you do
> not get to decide what is and isn't in the language. That was decided
> years ago by the XML working group and the W3C. Sometimes they decided
> right. Sometimes they decided wrong. But they have decided, and it is
> now up to implementers to follow the spec as it's written, not to write
> their own half-XML/half meat-by-product parser.
Elliotte, you make some good points but I think you're
overreacting. I don't think it's as bad as you say. But
if a user goes to the XPP page and expects that a pull-
parser implementing this interface is fully compliant
with XML 1.0 out of the box, then that's clearly a
problem. But the problem is in the documentation (and
perhaps marketing).
There are many situations where a fully compliant XML
parser is not needed. For example, take any closed
application system that is generating well-formed,
valid XML documents. In this situation, they do not
need fully compliant parsers. And if they chose to use
a fully compliant parser, they would very likely not
hit their transactions/second requirements. However,
I do agree that the programmers that choose this path
must be fully aware of what they are doing.
On a related topic, I've been working on my own pull-
parsing API based on XNI. I have a preliminary design
implemented using the Xerces2 standard parser config
as the driver. It's working quite well and allows all
of the functionality of Xerces2 natively, including
augmentations such as PSVI.
The only downside I'm seeing is performance. But
this isn't due to the API but rather because of my
implementation. The convertor from document handler
callbacks to pull-parser events has to buffer the
data coming from the parser configuration. If you're
processing small documents you probably wouldn't
notice it but as the document size increases, the
constant character array copying kills you.
So I'm thinking of adding a feature to Xerces2 that
allows applications to set whether the scanner re-
uses its character buffers. If the scanner doesn't
reuse buffers, then my converter doesn't have to
copy anything. Therefore, the performance of the
pull-parser driven by Xerces2 should be better.
Of course, you could always write an implementation
only for the pull-parsing API that wouldn't suffer
this problem at all. You could even drive the API
from other pull-parsing APIs like XPP, etc. :)
--
Andy Clark * andyc@apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
Re: [Announce] The CyberNeko Tools for XNI 2002.07.17 Available
Posted by Aleksander Slominski <as...@cs.indiana.edu>.
Elliotte Rusty Harold wrote:
> At 1:52 PM -0400 7/22/02, Aleksander Slominski wrote:
>
> >actually that is part of parser documentation, and for example
> >implementation XMLPULL API based on Xerces 2 XNI
> >will by default be XML 1.0 non validating so user will not need
> >to do anything (and trying to use it in "limited" mode and
> >incompatible with full XML 1.0 mode will be simply not supported)
>
> Perhaps you are confusing validation with the reading of the internal
> DTD subset. They are related but are not the same thing. XML 1.0
> specifically allows parsers not to validate. However, all parsers,
> validating or not, *must* read the internal DTD subset, use it to
> define internal entities and supply default values to attributes, and
> report any well-formedness errors detected therein. This is required,
> even of a non-validating parser.
that is well defined in XML 1.0 and that is why i have said that XMLPULL API
has mode in which it incompatible with full XML 1.0 (validating or non validating)
but allows small and efficient implementations that for example works well in
J2ME environments (so in this context limited subset XML and not full XML
is desirable and even required so API makes it possible). this is choice we made in API
to support full spectrum of implementations from J2ME up to J2EE.
> >> Noted. However, it seems to me that the flaws in API design and the
> >> flaws in implementation both stem from the same false preconceptions.
> >> In particular:
> >>
> >> 1. XML 1.0 correctness is negotiable
> >> 2. Size and speed are more important than a clean design
> >>
> >> Thus the flaws in the API and the implementations are very closely related.
> >
> >IMHO design of XMLPULL API is very clean and choices
> >were made to balance between easy to use API and
> >good performance _and_ small memory footprint and still
> >allow full XML 1.0 support - so API tries to do quite a lot!
>
> If that were accurate, we would not being having this conversation.
> XMLPULL as currently implemented does not provide full XML 1.0
> support. The design choices you've made in the name of good
> performance _and_ small memory footprint have compromised full XML
> 1.0 support.
i do not agree - current XMLPULL API _has_ XML 1.0 compliant mode
(as it exposes XML infoset) just that there are different modes in API
and implementations may be only supporting non full XML 1.0 compliant mode.
therefore i would not say that specialized implementation compromise anything
- we just made informed choices when creating implementations ...
> >ultimately it is up to XMLPULL API users to decide if they
> >like API.
>
> I think that it's more than that. I do not expect API users to be
> experts in XML. I do not expect them to know every last nook and
> cranny of the XML 1.0 spec, just as I do not expect that users of
> java.net.Socket know every detail of the RFCs for TCP/IP. However, I
> do expect that the designers of the java.net.Socket class knew every
> detail of the RFCs for TCP/IP and designed the class appropriately so
> it can be used by non-experts. Similarly, I expect designers of XML
> APIs to know every last nook and cranny of the XML specs and to
> design their APIs so that they can be used by non-experts. I am
> distinctly concerned about bait-and-switch tactics of some APIs
> (XMLPULL is hardly unique here) which present XML as somehow easier
> or different than it really is. Users very well may choose an API
> based on which seems simplest for them, without considering or even
> knowing how to evaluate which API is most correct. Users will assume
> the API designer has done their homework and is offering them a
> correct model of XML.
to make things absolutely clear: XMLPULL API can expose to the user
full XML 1.0 infoset - there is no limitation in API that i know about ...
> It is the responsibility of the API designers
> to see that users trust is not violated by correctly implementing the
> standards they claim to implement.
users does not have to assume anything as everything is documented
(in future we will try to make better job though of explaining
difference between API and what exactly implementations are providing).
thanks,
alek
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
Re: [Announce] The CyberNeko Tools for XNI 2002.07.17 Available
Posted by Elliotte Rusty Harold <el...@metalab.unc.edu>.
At 1:52 PM -0400 7/22/02, Aleksander Slominski wrote:
>actually that is part of parser documentation, and for example
>implementation XMLPULL API based on Xerces 2 XNI
>will by default be XML 1.0 non validating so user will not need
>to do anything (and trying to use it in "limited" mode and
>incompatible with full XML 1.0 mode will be simply not supported)
Perhaps you are confusing validation with the reading of the internal
DTD subset. They are related but are not the same thing. XML 1.0
specifically allows parsers not to validate. However, all parsers,
validating or not, *must* read the internal DTD subset, use it to
define internal entities and supply default values to attributes, and
report any well-formedness errors detected therein. This is required,
even of a non-validating parser.
>> You have convinced me that pull
>> parsing is a useful model in general, which I was extremely skeptical
>> of before. However, I remain convinced that the existing parsers in
>> Java and the XMLPULL API in particular are deeply flawed. Unless you
>> are willing to reconsider the principles that underlie the design of
>> XMLPULL, it will not become a suitable API for XML processing.
>
>u would say that current implementations are not meeting
>your requirements ...
No, I would say they are not meeting the requirements of XML 1.0.
>> >i would like to ask that when making comments about XMLPULL API
>> >to be precise if it is API that has flaws (this is serious problem!)
>> >or particular implementation that needs fixing (that i think is
>> >minor problem).
>> >
>>
>> Noted. However, it seems to me that the flaws in API design and the
>> flaws in implementation both stem from the same false preconceptions.
>> In particular:
>>
>> 1. XML 1.0 correctness is negotiable
>> 2. Size and speed are more important than a clean design
>>
>> Thus the flaws in the API and the implementations are very closely related.
>
>IMHO design of XMLPULL API is very clean and choices
>were made to balance between easy to use API and
>good performance _and_ small memory footprint and still
>allow full XML 1.0 support - so API tries to do quite a lot!
If that were accurate, we would not being having this conversation.
XMLPULL as currently implemented does not provide full XML 1.0
support. The design choices you've made in the name of good
performance _and_ small memory footprint have compromised full XML
1.0 support.
>ultimately it is up to XMLPULL API users to decide if they
>like API.
I think that it's more than that. I do not expect API users to be
experts in XML. I do not expect them to know every last nook and
cranny of the XML 1.0 spec, just as I do not expect that users of
java.net.Socket know every detail of the RFCs for TCP/IP. However, I
do expect that the designers of the java.net.Socket class knew every
detail of the RFCs for TCP/IP and designed the class appropriately so
it can be used by non-experts. Similarly, I expect designers of XML
APIs to know every last nook and cranny of the XML specs and to
design their APIs so that they can be used by non-experts. I am
distinctly concerned about bait-and-switch tactics of some APIs
(XMLPULL is hardly unique here) which present XML as somehow easier
or different than it really is. Users very well may choose an API
based on which seems simplest for them, without considering or even
knowing how to evaluate which API is most correct. Users will assume
the API designer has done their homework and is offering them a
correct model of XML. It is the responsibility of the API designers
to see that users trust is not violated by correctly implementing the
standards they claim to implement.
--
+-----------------------+------------------------+-------------------+
| Elliotte Rusty Harold | elharo@metalab.unc.edu | Writer/Programmer |
+-----------------------+------------------------+-------------------+
| XML in a Nutshell, 2nd Edition (O'Reilly, 2002) |
| http://www.cafeconleche.org/books/xian2/ |
| http://www.amazon.com/exec/obidos/ISBN%3D0596002920/cafeaulaitA/ |
+----------------------------------+---------------------------------+
| Read Cafe au Lait for Java News: http://www.cafeaulait.org/ |
| Read Cafe con Leche for XML News: http://www.cafeconleche.org/ |
+----------------------------------+---------------------------------+
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
Re: [Announce] The CyberNeko Tools for XNI 2002.07.17 Available
Posted by Elliotte Rusty Harold <el...@metalab.unc.edu>.
At 10:26 PM +0200 7/22/02, Stefan Haustein wrote:
>as said earlier, I had started in kXML 1 with event objects
>and polymorphism. It just turned out to be not to very useful
>when actually using the API, since the different event types
>do not have very much in common. Instead, unneccessary
>overhead and code bloat was caused in applications
>that are using the parser.
You're bringing up a different issue, which is whether the design of
XMLPULL is clean and object oriented. As you know, I think it's not.
However, this is completely orthogonal to the question of whether
XMLPULL and its implementations correctly support XML 1.0. You could
fix all the OO problems and still not correctly implement XML 1.0;
and you could fix all the errors in XML parsing and still not have
clean, OO interfaces to the parser. Both are problems with the
existing API. Both would be helped by competition from independent
efforts to develop a pull parser.
>You can easily verify this by converting the simple
>XML-RPC parser example available at http://kxml.org in the
>samples section to an imaginatory event object based
>pull parser of your choice.
One example would prove little. You need a broad range of problems to
see how well the API fits the uses to which XML is put. Right now I'm
working on what I think will be the first correct tree-based API for
processing XML. (More about that on September 17 in New York.) As
part of the testing I'm rewriting every single example in Processing
XML with Java to see how well the API works. That's shown me a number
of ugly areas in my API that I've had to redesign. When I eventually
throw this out to the public, I'll then have to consider all the use
cases everyone else comes up with and see if any of those indicate a
need for redesign.
One thing I have not done yet, and will not do until the API is
stable, is any optimization. To quote Donald Knuth, "Premature
optimization is the root of all evil." Once I am convinced the API is
correct and complete, then and only then will I do profiling to
discover where the speed and space problems are. At that point, I'll
fix them. Before I can do this, I have to make the code right. Making
the code fast before you make it right rarely makes it any faster and
almost never makes it right.
> If the XMLPULL design is flawed
>as you claim, it should be easy to sketch a better
>API that does not just sound more OO but also justifies
>the extra object by making application code using the
>parser better readable.
>
I'm certain it is; and I may yet do this one day, though right now
I'm busy with tree-based APIs. The most important issue is to stop
having so much awareness of state in the parser. XmlPullParser has
way too many methods that only work some of the time, and it's up to
the client programmer to figure out which methods work when and
where. Very ugly. A clean API would both make the application code
using the parser more readable and make it easier to write that code
in the first place.
--
+-----------------------+------------------------+-------------------+
| Elliotte Rusty Harold | elharo@metalab.unc.edu | Writer/Programmer |
+-----------------------+------------------------+-------------------+
| XML in a Nutshell, 2nd Edition (O'Reilly, 2002) |
| http://www.cafeconleche.org/books/xian2/ |
| http://www.amazon.com/exec/obidos/ISBN%3D0596002920/cafeaulaitA/ |
+----------------------------------+---------------------------------+
| Read Cafe au Lait for Java News: http://www.cafeaulait.org/ |
| Read Cafe con Leche for XML News: http://www.cafeconleche.org/ |
+----------------------------------+---------------------------------+
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
Re: [Announce] The CyberNeko Tools for XNI 2002.07.17 Available
Posted by Stefan Haustein <ha...@kimo.cs.uni-dortmund.de>.
>>Elliotte Rusty Harold wrote:
>>
>>You have convinced me that pull
>>parsing is a useful model in general, which I was extremely skeptical
>>of before. However, I remain convinced that the existing parsers in
>>Java and the XMLPULL API in particular are deeply flawed. Unless you
>>are willing to reconsider the principles that underlie the design of
>>XMLPULL, it will not become a suitable API for XML processing.
Elliote,
as said earlier, I had started in kXML 1 with event objects
and polymorphism. It just turned out to be not to very useful
when actually using the API, since the different event types
do not have very much in common. Instead, unneccessary
overhead and code bloat was caused in applications
that are using the parser.
You can easily verify this by converting the simple
XML-RPC parser example available at http://kxml.org in the
samples section to an imaginatory event object based
pull parser of your choice. If the XMLPULL design is flawed
as you claim, it should be easy to sketch a better
API that does not just sound more OO but also justifies
the extra object by making application code using the
parser better readable.
Best,
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
Re: [Announce] The CyberNeko Tools for XNI 2002.07.17 Available
Posted by Aleksander Slominski <as...@cs.indiana.edu>.
Elliotte Rusty Harold wrote:
> The user should not have to make special requests to get a fully
> conformant parser. This should be the default, and it should not be
> possible to produce a non-conformant parser. Anything less is a
> violation of XML 1.0.
actually that is part of parser documentation, and for example
implementation XMLPULL API based on Xerces 2 XNI
will by default be XML 1.0 non validating so user will not need
to do anything (and trying to use it in "limited" mode and
incompatible with full XML 1.0 mode will be simply not supported)
> You have convinced me that pull
> parsing is a useful model in general, which I was extremely skeptical
> of before. However, I remain convinced that the existing parsers in
> Java and the XMLPULL API in particular are deeply flawed. Unless you
> are willing to reconsider the principles that underlie the design of
> XMLPULL, it will not become a suitable API for XML processing.
u would say that current implementations are not meeting
your requirements ...
> >i would like to ask that when making comments about XMLPULL API
> >to be precise if it is API that has flaws (this is serious problem!)
> >or particular implementation that needs fixing (that i think is
> >minor problem).
> >
>
> Noted. However, it seems to me that the flaws in API design and the
> flaws in implementation both stem from the same false preconceptions.
> In particular:
>
> 1. XML 1.0 correctness is negotiable
> 2. Size and speed are more important than a clean design
>
> Thus the flaws in the API and the implementations are very closely related.
IMHO design of XMLPULL API is very clean and choices
were made to balance between easy to use API and
good performance _and_ small memory footprint and still
allow full XML 1.0 support - so API tries to do quite a lot!
ultimately it is up to XMLPULL API users to decide if they
like API. however we think about XMLPULL API
as a building block on which higher level layers can be built
(like supporting push and pull simultaneously - implemented currently
with SAX driver or more OO XML pull event oriented API, or lightweight
incremental DOM APIs or very efficient data binding impl. or
SOAP processors etc.).
if we enable all this to happen then i think that we succeeded ...
thanks,
alek
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
Re: [Announce] The CyberNeko Tools for XNI 2002.07.17 Available
Posted by Elliotte Rusty Harold <el...@metalab.unc.edu>.
At 11:36 PM -0400 7/21/02, Aleksander Slominski wrote:
>i must disagree ...
>
>XML is used now in all possible places and that what was perfect for
>hand-writing XML documents does not necessarily is useful when
>XML is machine processed - in such situation DTD is not only unnecessary
>but adds extra weight to XML parser implementations
>(and size of implementation is very important factor for J2ME ...)
>
You are working under the misapprehension that this is your choice to
make. It isn't. No matter how valid your criticisms of XML are, you
do not get to decide what is and isn't in the language. That was
decided years ago by the XML working group and the W3C. Sometimes
they decided right. Sometimes they decided wrong. But they have
decided, and it is now up to implementers to follow the spec as it's
written, not to write their own half-XML/half meat-by-product parser.
If you are truly convinced that the XML spec is fundamentally broken,
and you are willing to throw out the advantage of working with a
standard language, flawed but a standard nonetheless; then you should
feel free to invent your own markup language and parsers for it. This
is what the MinML and YAML folks have done, for example. However,
please do not tell people that your tools and parsers are XML.
They're not. Misleading people about what they get when they use
XMLPULL does a disservice to your users and the broader XML community.
>moreover XML schemas are replacing DTD so in many situation
>support for DTD is no longer needed at all ...
So what? Schemas do not change any aspect of XML 1.0, or relax any
restrictions on what a conforming parser must provide.
>i think that XML 1.0 conformance is crucial for XML parser
>- with XMLPULL API the user code can request XML 1.0
>validating or non validating parser and if parser is created it
>is guaranteed to be XML 1.0 compliant - short example:
The user should not have to make special requests to get a fully
conformant parser. This should be the default, and it should not be
possible to produce a non-conformant parser. Anything less is a
violation of XML 1.0.
>unfortunately it seems that we may failed in trying to get
>this fundamental API design choice explained ...
I've had this conversation with you before. I understand you quite
well. I understand your choice, and why you made the choice you did.
I *disagree* with your choice, and find that choice to be
fundamentally incompatible with XML. You have convinced me that pull
parsing is a useful model in general, which I was extremely skeptical
of before. However, I remain convinced that the existing parsers in
Java and the XMLPULL API in particular are deeply flawed. Unless you
are willing to reconsider the principles that underlie the design of
XMLPULL, it will not become a suitable API for XML processing.
>in MPX1 i have currently support for proposed XML 1.1 character
>checking as it seems better reflecting current UNICODE usage patterns
>(http://www.w3.org/TR/xml11/#sec2.3)
Thanks for confirming that. This is another area in which the parser
is deliberately not following the spec, and is likely to confuse
users and cause problems for interoperability. If you wish to write
an experimental XML 1.1 parser, feel free. However, please label it
as such so users are not confused. Furthermore, you should realize
that it is likely that even XML 1.1 parsers will still be required to
make full name checks according to XML 1.0 rules for documents that
are not explicitly labelled as version="1.1".
>i would like to ask that when making comments about XMLPULL API
>to be precise if it is API that has flaws (this is serious problem!)
>or particular implementation that needs fixing (that i think is
>minor problem).
>
Noted. However, it seems to me that the flaws in API design and the
flaws in implementation both stem from the same false preconceptions.
In particular:
1. XML 1.0 correctness is negotiable
2. Size and speed are more important than a clean design
Thus the flaws in the API and the implementations are very closely related.
--
+-----------------------+------------------------+-------------------+
| Elliotte Rusty Harold | elharo@metalab.unc.edu | Writer/Programmer |
+-----------------------+------------------------+-------------------+
| XML in a Nutshell, 2nd Edition (O'Reilly, 2002) |
| http://www.cafeconleche.org/books/xian2/ |
| http://www.amazon.com/exec/obidos/ISBN%3D0596002920/cafeaulaitA/ |
+----------------------------------+---------------------------------+
| Read Cafe au Lait for Java News: http://www.cafeaulait.org/ |
| Read Cafe con Leche for XML News: http://www.cafeconleche.org/ |
+----------------------------------+---------------------------------+
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
Re: [Announce] The CyberNeko Tools for XNI 2002.07.17 Available
Posted by Aleksander Slominski <as...@cs.indiana.edu>.
Elliotte Rusty Harold wrote:
> At 4:52 PM -0400 7/19/02, Aleksander Slominski wrote:
>
> >hmm, citing: 'outright violations with XML 1.0, none of which the maintainers
> >seem willing to fix.' - this is pretty strong statement!
> >
> >i am not aware of any XML 1.0 violations in XMLPULL API.
> >the only thing that could possibly make XMLPULL API not
> >100% compatible with XML 1.0 is when PROCESS DOCDECL feature
> >(http://xmlpull.org/v1/doc/features.html#process-docdecl)
> >is set to false and we did only to allow creation of limited XML
> >parsers (mainly for J2ME).
> >
>
> That's a very big one. A parser should not be allowed to turn off
> processing of the internal DTD subset at all. And to make not
> processing it the default?! That's just wrong.
i must disagree ...
XML is used now in all possible places and that what was perfect for
hand-writing XML documents does not necessarily is useful when
XML is machine processed - in such situation DTD is not only unnecessary
but adds extra weight to XML parser implementations
(and size of implementation is very important factor for J2ME ...)
moreover XML schemas are replacing DTD so in many situation
support for DTD is no longer needed at all ...
i think that XML 1.0 conformance is crucial for XML parser
- with XMLPULL API the user code can request XML 1.0
validating or non validating parser and if parser is created it
is guaranteed to be XML 1.0 compliant - short example:
XmlPullParserFactory factory = XmlPullParserFactory.newInstance();
factory.setFeature(XmlPullParser.FEATURE_VALIDATION, true); // request validating parser
//factory.setFeature(XmlPullParser.FEATURE_PROCESS_DOCDECL, true); //to request non-validating parser
factory.setNamespaceAware(true); //request that created parser must support namespaces
XmlPullParser xpp = factory.newPullParser();
if parser implementation supporting features required by user is not available
it will not be created (and newPullParser() will throw exception to indicate it).
we did not make PROCESS_DOCDECL to be true by default
as we wanted to create minimal XML parsing API that can be
gracefully scaled up to full validating XML parsing.
unfortunately it seems that we may failed in trying to get
this fundamental API design choice explained ...
> Worse yet, according to http://www.xmlpull.org/impls.shtml neither of
> the existing implementations even allows you to set that feature to
> true.
that means that current implementations are limited but has not much
to do with API - i would treat differently API and implementations and
would make clear that i am criticizing implementation and not API!
i am also working on Xerces 2 XNI based implementation of XMLPULL API
that should then should be a reference and complete implementation of
XMLPULL V1 API.
> I've also heard it claimed recently that the parsers aren't doing all
> the name character checking they're supposed to, though I haven't
> confirmed that one for myself. See
> <http://lists.xml.org/archives/xml-dev/200207/msg00502.html>
in MPX1 i have currently support for proposed XML 1.1 character
checking as it seems better reflecting current UNICODE usage patterns
(http://www.w3.org/TR/xml11/#sec2.3)
i would like to ask that when making comments about XMLPULL API
to be precise if it is API that has flaws (this is serious problem!)
or particular implementation that needs fixing (that i think is minor problem).
thanks,
alek
ps. as of my implementation MXP1 it certainly works good for tasks
it was designed (for example SOAP parsing) but i am aware of its
limitations and i both intent to remove them and in longer term
to pass it through OASIS/W3C conformance tests.
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
xmlpull api [Re: [Announce] The CyberNeko Tools for XNI 2002.07.17
Available]
Posted by Aleksander Slominski <as...@cs.indiana.edu>.
Andy Clark wrote:
> Elliotte Rusty Harold wrote:
> >> the only thing that could possibly make XMLPULL API not
> >> 100% compatible with XML 1.0 is when PROCESS DOCDECL feature
> >> [...]
> >
> > That's a very big one. A parser should not be allowed to turn off
> > processing of the internal DTD subset at all. And to make not processing
> > it the default?! That's just wrong.
>
> Well, you gotta look at the intended purpose of these types
> of parsers. If I remember correctly, the XPP work was started
> because of SOAP which subsets XML syntax and doesn't allow a
> DOCTYPE line at all.
that means that those implementations were designed to concentrate
on size (like kXML2) or speed (like MXP1) but there can be many
other implementations ...
> > Worse yet, according to http://www.xmlpull.org/impls.shtml neither of
> > the existing implementations even allows you to set that feature to true.
>
> I think Alexsander has code to use Xerces2 as the driver
> for the push API. So, if used that way then it should be
> able to check the DTD just like Xerces. And when I finish
> my API for the CyberNeko tools, the default impl will be
> driven by Xerces so it should have no problem in that
> regard.
exactly - one thing is API and completely another is implementation.
as long as each implementation is correctly described users can make
informed choices.
> > I've also heard it claimed recently that the parsers aren't doing all
> > the name character checking they're supposed to, though I haven't
>
> No wonder they're so fast. ;) This is one of the big
> checks that implementors would love to remove from their
> inner loop. Xerces, being fully compliant, can't do that
> and suffers some performance hits.
in MXP1 i use lookup table for char values below
and if statement for the rest. i am putting
relevant part of code from MXP1 below and
welcome comments about it (especially if you find anything
wrong with the functions!).
> Just about any XML parser can be written to go fast if
> they don't do all of the work. For example, removing
> character checking, avoiding DTD parsing and processing,
> not implementing XML Schema, etc. But, depending on the
> situation, these are all perfectly acceptable choices.
well - i think that MXP1 do all XML parsing and i am slowly
improving it to the level of non validating parsing - only
remaining incompatibilities i know about is DTD parsing and
add XML 1.0 character set support (i am a bit hesitant about
it as i like XML 1.1 much more ...).
thanks,
alek
ps. here is fragment of MXParser - please comment if you think
that i am missing something important when looking on what is
required in http://www.w3.org/TR/xml11/#sec2.3 (thanks in advance!)
protected static final int LOOKUP_MAX = 0x400;
protected static final char LOOKUP_MAX_CHAR = (char)LOOKUP_MAX;
protected static boolean lookupNameStartChar[] = new boolean[ LOOKUP_MAX ];
protected static boolean lookupNameChar[] = new boolean[ LOOKUP_MAX ];
private static final void setName(char ch)
{ lookupNameChar[ ch ] = true; }
private static final void setNameStart(char ch)
{ lookupNameStartChar[ ch ] = true; setName(ch); }
static {
setNameStart(':');
for (char ch = 'A'; ch <= 'Z'; ++ch) setNameStart(ch);
setNameStart('_');
for (char ch = 'a'; ch <= 'z'; ++ch) setNameStart(ch);
for (char ch = '\u00c0'; ch <= '\u02FF'; ++ch) setNameStart(ch);
for (char ch = '\u0370'; ch <= '\u037d'; ++ch) setNameStart(ch);
for (char ch = '\u037f'; ch < '\u0400'; ++ch) setNameStart(ch);
setName('-');
setName('.');
for (char ch = '0'; ch <= '9'; ++ch) setName(ch);
setName('\u00b7');
for (char ch = '\u0300'; ch <= '\u036f'; ++ch) setName(ch);
}
private final static boolean isNameStartChar(char ch) {
return (ch < LOOKUP_MAX_CHAR && lookupNameStartChar[ ch ])
|| (ch >= LOOKUP_MAX_CHAR && ch <= '\u2027')
|| (ch >= '\u202A' && ch <= '\u218F')
|| (ch >= '\u2800' && ch <= '\uFFEF')
;
}
private final static boolean isNameChar(char ch) {
return (ch < LOOKUP_MAX_CHAR && lookupNameChar[ ch ])
|| (ch >= LOOKUP_MAX_CHAR && ch <= '\u2027')
|| (ch >= '\u202A' && ch <= '\u218F')
|| (ch >= '\u2800' && ch <= '\uFFEF')
;
}
protected boolean isS(char ch) {
return (ch == ' ' || ch == '\n' || ch == '\r' || ch == '\t');
}
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
Re: [Announce] The CyberNeko Tools for XNI 2002.07.17 Available
Posted by Andy Clark <an...@apache.org>.
Elliotte Rusty Harold wrote:
>> the only thing that could possibly make XMLPULL API not
>> 100% compatible with XML 1.0 is when PROCESS DOCDECL feature
>> [...]
>
> That's a very big one. A parser should not be allowed to turn off
> processing of the internal DTD subset at all. And to make not processing
> it the default?! That's just wrong.
Well, you gotta look at the intended purpose of these types
of parsers. If I remember correctly, the XPP work was started
because of SOAP which subsets XML syntax and doesn't allow a
DOCTYPE line at all.
> Worse yet, according to http://www.xmlpull.org/impls.shtml neither of
> the existing implementations even allows you to set that feature to true.
I think Alexsander has code to use Xerces2 as the driver
for the push API. So, if used that way then it should be
able to check the DTD just like Xerces. And when I finish
my API for the CyberNeko tools, the default impl will be
driven by Xerces so it should have no problem in that
regard.
> I've also heard it claimed recently that the parsers aren't doing all
> the name character checking they're supposed to, though I haven't
No wonder they're so fast. ;) This is one of the big
checks that implementors would love to remove from their
inner loop. Xerces, being fully compliant, can't do that
and suffers some performance hits.
Just about any XML parser can be written to go fast if
they don't do all of the work. For example, removing
character checking, avoiding DTD parsing and processing,
not implementing XML Schema, etc. But, depending on the
situation, these are all perfectly acceptable choices.
--
Andy Clark * andyc@apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
Re: [Announce] The CyberNeko Tools for XNI 2002.07.17 Available
Posted by Elliotte Rusty Harold <el...@metalab.unc.edu>.
At 4:52 PM -0400 7/19/02, Aleksander Slominski wrote:
>hmm, citing: 'outright violations with XML 1.0, none of which the maintainers
>seem willing to fix.' - this is pretty strong statement!
>
>i am not aware of any XML 1.0 violations in XMLPULL API.
>the only thing that could possibly make XMLPULL API not
>100% compatible with XML 1.0 is when PROCESS DOCDECL feature
>(http://xmlpull.org/v1/doc/features.html#process-docdecl)
>is set to false and we did only to allow creation of limited XML
>parsers (mainly for J2ME).
>
That's a very big one. A parser should not be allowed to turn off
processing of the internal DTD subset at all. And to make not
processing it the default?! That's just wrong.
Worse yet, according to http://www.xmlpull.org/impls.shtml neither of
the existing implementations even allows you to set that feature to
true.
I've also heard it claimed recently that the parsers aren't doing all
the name character checking they're supposed to, though I haven't
confirmed that one for myself. See
<http://lists.xml.org/archives/xml-dev/200207/msg00502.html>
--
+-----------------------+------------------------+-------------------+
| Elliotte Rusty Harold | elharo@metalab.unc.edu | Writer/Programmer |
+-----------------------+------------------------+-------------------+
| XML in a Nutshell, 2nd Edition (O'Reilly, 2002) |
| http://www.cafeconleche.org/books/xian2/ |
| http://www.amazon.com/exec/obidos/ISBN%3D0596002920/cafeaulaitA/ |
+----------------------------------+---------------------------------+
| Read Cafe au Lait for Java News: http://www.cafeaulait.org/ |
| Read Cafe con Leche for XML News: http://www.cafeconleche.org/ |
+----------------------------------+---------------------------------+
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
Re: [Announce] The CyberNeko Tools for XNI 2002.07.17 Available
Posted by Aleksander Slominski <as...@cs.indiana.edu>.
Elliotte Rusty Harold wrote:
> Please do make it a competitor to XPP et al. Competion is good for
> everyone. Currently the XMLPULL API has a lot of things I consider to
> be bad design and some outright violations with XML 1.0, none of
> which the maintainers seem willing to fix. If nothing better comes
> along, then I'm afraid pull parsing in Java is doomed to irrelevance,
> which would be a shame. Pull parsing is a useful, convenient style of
> processing. However it needs a parser and an API that actually works.
hi,
hmm, citing: 'outright violations with XML 1.0, none of which the maintainers
seem willing to fix.' - this is pretty strong statement!
i am not aware of any XML 1.0 violations in XMLPULL API.
the only thing that could possibly make XMLPULL API not
100% compatible with XML 1.0 is when PROCESS DOCDECL feature
(http://xmlpull.org/v1/doc/features.html#process-docdecl)
is set to false and we did only to allow creation of limited XML
parsers (mainly for J2ME).
so could you explain what are the basis for this statement?
thanks,
alek
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
Re: [Announce] The CyberNeko Tools for XNI 2002.07.17 Available
Posted by Elliotte Rusty Harold <el...@metalab.unc.edu>.
At 12:58 AM +0900 7/18/02, Andy Clark wrote:
>Next, I plan on designing a pull-parser API that builds
>on the Xerces Native Interface. This API is not meant to
>be a competitor to existing pull-parsing projects such
>as XPP. Rather, it is meant to show the flexibility and
>power of the XNI framework to address other parsing needs
>while leveraging the existing library of XNI components.
Please do make it a competitor to XPP et al. Competion is good for
everyone. Currently the XMLPULL API has a lot of things I consider to
be bad design and some outright violations with XML 1.0, none of
which the maintainers seem willing to fix. If nothing better comes
along, then I'm afraid pull parsing in Java is doomed to irrelevance,
which would be a shame. Pull parsing is a useful, convenient style of
processing. However it needs a parser and an API that actually works.
--
+-----------------------+------------------------+-------------------+
| Elliotte Rusty Harold | elharo@metalab.unc.edu | Writer/Programmer |
+-----------------------+------------------------+-------------------+
| XML in a Nutshell, 2nd Edition (O'Reilly, 2002) |
| http://www.cafeconleche.org/books/xian2/ |
| http://www.amazon.com/exec/obidos/ISBN%3D0596002920/cafeaulaitA/ |
+----------------------------------+---------------------------------+
| Read Cafe au Lait for Java News: http://www.cafeaulait.org/ |
| Read Cafe con Leche for XML News: http://www.cafeconleche.org/ |
+----------------------------------+---------------------------------+
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
Re: [Announce] The CyberNeko Tools for XNI 2002.07.17 Available
Posted by Steven Noels <st...@outerthought.org>.
Andy Clark wrote:
> NekoDTD was updated to include stylesheets to convert DTD
> grammars to Relax NG. Also, this release adds a new tool,
> called ManekiNeko, that includes a parser configuration
> built with a Relax NG validator instead of the standard
> DTD/XML Schema validators.
Thanks, Andy. Everyone: be sure to take a look at Andy's work since it
is very interesting. In the Forrest project, we wrapped NekoDTD as a
Cocoon Generator and use it to produce DTD documentation with some XSLT
stylesheets: http://xml.apache.org/forrest/dtd-docs.html
</Steven>
--
Steven Noels http://outerthought.org/
Outerthought - Open Source, Java & XML Competence Support Center
stevenn@outerthought.org stevenn@apache.org
---------------------------------------------------------------------
In case of troubles, e-mail: webmaster@xml.apache.org
To unsubscribe, e-mail: general-unsubscribe@xml.apache.org
For additional commands, e-mail: general-help@xml.apache.org