You are viewing a plain text version of this content. The canonical link for it is here.
Posted to j-users@xerces.apache.org by Xiaoming Liu <li...@lanl.gov> on 2005/01/04 00:14:25 UTC
read byte offset information during xml parsing
hi,
I am looking for a Java XML parser which supports reading byte offset
information during xml parsing, e.g. in '<foo><bar></bar></foo>', the
parser can report '<bar>' starts from byte 5; and '</bar>' starts from
byte 10 .
I went through standard APIs like DOM, SAX, and XMLPull and cannot find
related APIs. In Sax, the nearest interface is org.xml.sax.Locator. I also
checked Xerces XNI and found the nearest class is
org.apache.xerces.xni.XMLLocator. In either class, only line number and
column number are reported.
However, similar functions are provided in other languages, such as the
"XML_GetCurrentByteIndex" of expat parser (C, perl).
so my question is whether there is a Java XML Parser reporting byte offset
information during parsing, and if not, is there any plan to
implement this feature?
many thanks,
Xiaoming
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-j-user-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-j-user-help@xml.apache.org
RE: read byte offset information during xml parsing
Posted by Xiaoming Liu <li...@lanl.gov>.
hi skoya,
I am really thinking about byte offset here. For a file or networked
document, regardless how java IO api wraps character reader, a byte
offset is a fixed value. In other words, if xerces reads a
file by FileInputStream.read(byte[] b), it will have byte offset
information for certain events.
I shall illustrate more about our use case in the reply to Michael's
message.
Xiaoming
On Tue, 4 Jan 2005, Suresh Babu Koya wrote:
> If you are using UTF-16 encoding then probably <bar>
> will not start at byte 5. Do you want the character
> position or the byte position?
>
> Also if you have the BOM Marker for Unicode at the start
> of the Stream then again the byte position varies.
> I think that was one reason why xerces-j was not providing
> that kind API.
>
> /Suresh
>
> >>-----Original Message-----
> >>From: Xiaoming Liu [mailto:liu_x@lanl.gov]
> >>Sent: Tuesday, January 04, 2005 4:44 AM
> >>To: xerces-j-user@xml.apache.org
> >>Subject: read byte offset information during xml parsing
> >>
> >>
> >>hi,
> >>
> >>I am looking for a Java XML parser which supports reading byte offset
> >>information during xml parsing, e.g. in '<foo><bar></bar></foo>', the
> >>parser can report '<bar>' starts from byte 5; and '</bar>' starts from
> >>byte 10 .
> >>
> >>I went through standard APIs like DOM, SAX, and XMLPull and
> >>cannot find
> >>related APIs. In Sax, the nearest interface is
> >>org.xml.sax.Locator. I also
> >>checked Xerces XNI and found the nearest class is
> >>org.apache.xerces.xni.XMLLocator. In either class, only line
> >>number and
> >>column number are reported.
> >>
> >>However, similar functions are provided in other languages,
> >>such as the
> >>"XML_GetCurrentByteIndex" of expat parser (C, perl).
> >>
> >>so my question is whether there is a Java XML Parser
> >>reporting byte offset
> >>information during parsing, and if not, is there any plan to
> >>implement this feature?
> >>
> >>many thanks,
> >>Xiaoming
> >>
> >>
> >>---------------------------------------------------------------------
> >>To unsubscribe, e-mail: xerces-j-user-unsubscribe@xml.apache.org
> >>For additional commands, e-mail: xerces-j-user-help@xml.apache.org
> >>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: xerces-j-user-unsubscribe@xml.apache.org
> For additional commands, e-mail: xerces-j-user-help@xml.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-j-user-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-j-user-help@xml.apache.org
RE: read byte offset information during xml parsing
Posted by Suresh Babu Koya <sk...@in-reality.com>.
If you are using UTF-16 encoding then probably <bar>
will not start at byte 5. Do you want the character
position or the byte position?
Also if you have the BOM Marker for Unicode at the start
of the Stream then again the byte position varies.
I think that was one reason why xerces-j was not providing
that kind API.
/Suresh
>>-----Original Message-----
>>From: Xiaoming Liu [mailto:liu_x@lanl.gov]
>>Sent: Tuesday, January 04, 2005 4:44 AM
>>To: xerces-j-user@xml.apache.org
>>Subject: read byte offset information during xml parsing
>>
>>
>>hi,
>>
>>I am looking for a Java XML parser which supports reading byte offset
>>information during xml parsing, e.g. in '<foo><bar></bar></foo>', the
>>parser can report '<bar>' starts from byte 5; and '</bar>' starts from
>>byte 10 .
>>
>>I went through standard APIs like DOM, SAX, and XMLPull and
>>cannot find
>>related APIs. In Sax, the nearest interface is
>>org.xml.sax.Locator. I also
>>checked Xerces XNI and found the nearest class is
>>org.apache.xerces.xni.XMLLocator. In either class, only line
>>number and
>>column number are reported.
>>
>>However, similar functions are provided in other languages,
>>such as the
>>"XML_GetCurrentByteIndex" of expat parser (C, perl).
>>
>>so my question is whether there is a Java XML Parser
>>reporting byte offset
>>information during parsing, and if not, is there any plan to
>>implement this feature?
>>
>>many thanks,
>>Xiaoming
>>
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: xerces-j-user-unsubscribe@xml.apache.org
>>For additional commands, e-mail: xerces-j-user-help@xml.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-j-user-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-j-user-help@xml.apache.org
Re: read byte offset information during xml parsing
Posted by Michael Glavassevich <mr...@ca.ibm.com>.
All of the fields on DOMLocator are allowed to be -1 or null if the
relevant data is not available. That includes byte offsets, so conforming
implementations are allowed to return -1 if they don't report this
information.
Xiaoming Liu <li...@lanl.gov> wrote on 01/04/2005 03:31:02 PM:
> Thanks, that makes the matter clear.
>
> I do have last question though, since DOM level 3 supports byte offset
in
> DOMLocator [1], does xerces have a plan of supporting DOM level 3?
>
> [1]
> http://www.w3.org/TR/2004/REC-DOM-Level-3-Core-20040407/core.
> html#Interfaces-DOMLocator
>
> Xiaoming
>
>
> On Tue, 4 Jan 2005, Andy Clark wrote:
>
> > Xiaoming Liu wrote:
> > > It seems to me that in order to fast access very large XML files,
byte
> > > offset is an efficient way. Probably it's also doable by character
offset,
> > > however I didn't know a java class providing character-based random
> > > access.
> >
> > Reporting byte offsets is just not possible given a number
> > of factors. The primary factor being that Xerces does not
> > control the decoding of the source bytes to characters.
> >
> > In order to maintain the proper byte location within the
> > stream, the parser would need to know *exactly* how many
> > bytes were read by the underlying input stream. Since we
> > rely on the decoders present in Java, this just isn't
> > possible.
> >
> > I once had the idea of putting a special byte-counting
> > input stream filter between the underlying stream and the
> > reader that converted the bytes to chars. I thought that
> > I could read one char at a time and then look at the
> > current byte offset to see how many bytes were actually
> > used to encode that single character. But this didn't
> > work.
> >
> > It turned out that the underlying decoders were buffering
> > internally. So even if I asked the reader for a single char,
> > the reader would buffer 1K or 2K of data. This fact makes
> > it impossible to do anything using the default readers to
> > report true byte offsets.
> >
> > > By the way, the java-based XP parser does provide a way to locate
> > > byteoffset of starting element event, oddly, it doesn't provide ways
to
> > > locate endElement and other events [1]. Since XP is not actively
> >
> > My guess is that the byte offsets reported by XP are only
> > valid for specific encodings: fixed byte length encodings
> > like 1-byte or 2-byte character encodings (e.g. US-ASCII,
> > ISO-8859-1, UTF-16, UCS4, etc) *and* assuming no Unicode
> > character normalization. Unless, of course, that he does
> > the character conversions himself and can keep track of
> > the true byte offsets.
> >
> > If you still need to report byte offsets with Xerces, I
> > think the only way to do it properly is to pre-normalize
> > your docs to a fixed length encoding and then use the
> > Xerces feature that reports character offsets. Then it's
> > just a matter of multiplying the character offset by the
> > char "width" of that encoding. Make sense?
> >
> > --
> > Andy Clark * andyc@apache.org
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: xerces-j-user-unsubscribe@xml.apache.org
> > For additional commands, e-mail: xerces-j-user-help@xml.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: xerces-j-user-unsubscribe@xml.apache.org
> For additional commands, e-mail: xerces-j-user-help@xml.apache.org
>
Michael Glavassevich
XML Parser Development
IBM Toronto Lab
E-mail: mrglavas@ca.ibm.com
E-mail: mrglavas@apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-j-user-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-j-user-help@xml.apache.org
Re: read byte offset information during xml parsing
Posted by Xiaoming Liu <li...@lanl.gov>.
Thanks, that makes the matter clear.
I do have last question though, since DOM level 3 supports byte offset in
DOMLocator [1], does xerces have a plan of supporting DOM level 3?
[1]
http://www.w3.org/TR/2004/REC-DOM-Level-3-Core-20040407/core.html#Interfaces-DOMLocator
Xiaoming
On Tue, 4 Jan 2005, Andy Clark wrote:
> Xiaoming Liu wrote:
> > It seems to me that in order to fast access very large XML files, byte
> > offset is an efficient way. Probably it's also doable by character offset,
> > however I didn't know a java class providing character-based random
> > access.
>
> Reporting byte offsets is just not possible given a number
> of factors. The primary factor being that Xerces does not
> control the decoding of the source bytes to characters.
>
> In order to maintain the proper byte location within the
> stream, the parser would need to know *exactly* how many
> bytes were read by the underlying input stream. Since we
> rely on the decoders present in Java, this just isn't
> possible.
>
> I once had the idea of putting a special byte-counting
> input stream filter between the underlying stream and the
> reader that converted the bytes to chars. I thought that
> I could read one char at a time and then look at the
> current byte offset to see how many bytes were actually
> used to encode that single character. But this didn't
> work.
>
> It turned out that the underlying decoders were buffering
> internally. So even if I asked the reader for a single char,
> the reader would buffer 1K or 2K of data. This fact makes
> it impossible to do anything using the default readers to
> report true byte offsets.
>
> > By the way, the java-based XP parser does provide a way to locate
> > byteoffset of starting element event, oddly, it doesn't provide ways to
> > locate endElement and other events [1]. Since XP is not actively
>
> My guess is that the byte offsets reported by XP are only
> valid for specific encodings: fixed byte length encodings
> like 1-byte or 2-byte character encodings (e.g. US-ASCII,
> ISO-8859-1, UTF-16, UCS4, etc) *and* assuming no Unicode
> character normalization. Unless, of course, that he does
> the character conversions himself and can keep track of
> the true byte offsets.
>
> If you still need to report byte offsets with Xerces, I
> think the only way to do it properly is to pre-normalize
> your docs to a fixed length encoding and then use the
> Xerces feature that reports character offsets. Then it's
> just a matter of multiplying the character offset by the
> char "width" of that encoding. Make sense?
>
> --
> Andy Clark * andyc@apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: xerces-j-user-unsubscribe@xml.apache.org
> For additional commands, e-mail: xerces-j-user-help@xml.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-j-user-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-j-user-help@xml.apache.org
Re: read byte offset information during xml parsing
Posted by Andy Clark <an...@cyberneko.net>.
Xiaoming Liu wrote:
> It seems to me that in order to fast access very large XML files, byte
> offset is an efficient way. Probably it's also doable by character offset,
> however I didn't know a java class providing character-based random
> access.
Reporting byte offsets is just not possible given a number
of factors. The primary factor being that Xerces does not
control the decoding of the source bytes to characters.
In order to maintain the proper byte location within the
stream, the parser would need to know *exactly* how many
bytes were read by the underlying input stream. Since we
rely on the decoders present in Java, this just isn't
possible.
I once had the idea of putting a special byte-counting
input stream filter between the underlying stream and the
reader that converted the bytes to chars. I thought that
I could read one char at a time and then look at the
current byte offset to see how many bytes were actually
used to encode that single character. But this didn't
work.
It turned out that the underlying decoders were buffering
internally. So even if I asked the reader for a single char,
the reader would buffer 1K or 2K of data. This fact makes
it impossible to do anything using the default readers to
report true byte offsets.
> By the way, the java-based XP parser does provide a way to locate
> byteoffset of starting element event, oddly, it doesn't provide ways to
> locate endElement and other events [1]. Since XP is not actively
My guess is that the byte offsets reported by XP are only
valid for specific encodings: fixed byte length encodings
like 1-byte or 2-byte character encodings (e.g. US-ASCII,
ISO-8859-1, UTF-16, UCS4, etc) *and* assuming no Unicode
character normalization. Unless, of course, that he does
the character conversions himself and can keep track of
the true byte offsets.
If you still need to report byte offsets with Xerces, I
think the only way to do it properly is to pre-normalize
your docs to a fixed length encoding and then use the
Xerces feature that reports character offsets. Then it's
just a matter of multiplying the character offset by the
char "width" of that encoding. Make sense?
--
Andy Clark * andyc@apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-j-user-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-j-user-help@xml.apache.org
Re: read byte offset information during xml parsing
Posted by Xiaoming Liu <li...@lanl.gov>.
Michael,
Thanks a lot for pointing out the reference. I would like to illstrate our
use cases to see if it's a generic requirement other folks may also
think.
We are working on very large XML files containing many records, in order
to random access this very large file, we would like to build index of
byte offset and length of these records, java.io.RandomAccessFile provides
the API to easily seek/read file by bytes, however we cannot generate the
index by off-the-shelf java XML parsers.
It seems to me that in order to fast access very large XML files, byte
offset is an efficient way. Probably it's also doable by character offset,
however I didn't know a java class providing character-based random
access.
By the way, the java-based XP parser does provide a way to locate
byteoffset of starting element event, oddly, it doesn't provide ways to
locate endElement and other events [1]. Since XP is not actively
maintained, I would like to listen opinions of xerces experts. I did
implement a simple parser to continue our work, but it would be nice if
this feature is supported by a well developed xml parser.
[1]http://www.jclark.com/xml/xp/api/packages.html
Xiaoming
On Mon, 3 Jan 2005, Michael Glavassevich wrote:
> Hello Xiaoming,
>
> In general byte offsets aren't available to the parser. The document
> scanners read from a java.io.Reader so the byte to character decoding is
> being done at a lower level. The parser only sees the decoded characters.
> If what you actually want is the character offset, we made some changes to
> XNI last year (they're in CVS) to expose this information in
> org.apache.xerces.xni.XMLLocator. If you're using DOM Level 3, we've also
> made the character offset available to DOMLocator [1]. You'll get this
> functionality from the latest jars [2].
>
> [1]
> http://www.w3.org/TR/2004/REC-DOM-Level-3-Core-20040407/core.html#Interfaces-DOMLocator
> [2] http://brutus.apache.org/gump/public-jars/xml-xerces2/jars/
>
> Xiaoming Liu <li...@lanl.gov> wrote on 01/03/2005 06:14:25 PM:
>
> > hi,
> >
> > I am looking for a Java XML parser which supports reading byte offset
> > information during xml parsing, e.g. in '<foo><bar></bar></foo>', the
> > parser can report '<bar>' starts from byte 5; and '</bar>' starts from
> > byte 10 .
> >
> > I went through standard APIs like DOM, SAX, and XMLPull and cannot find
> > related APIs. In Sax, the nearest interface is org.xml.sax.Locator. I
> also
> > checked Xerces XNI and found the nearest class is
> > org.apache.xerces.xni.XMLLocator. In either class, only line number and
> > column number are reported.
> >
> > However, similar functions are provided in other languages, such as the
> > "XML_GetCurrentByteIndex" of expat parser (C, perl).
> >
> > so my question is whether there is a Java XML Parser reporting byte
> offset
> > information during parsing, and if not, is there any plan to
> > implement this feature?
> >
> > many thanks,
> > Xiaoming
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: xerces-j-user-unsubscribe@xml.apache.org
> > For additional commands, e-mail: xerces-j-user-help@xml.apache.org
> >
>
> Michael Glavassevich
> XML Parser Development
> IBM Toronto Lab
> E-mail: mrglavas@ca.ibm.com
> E-mail: mrglavas@apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: xerces-j-user-unsubscribe@xml.apache.org
> For additional commands, e-mail: xerces-j-user-help@xml.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-j-user-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-j-user-help@xml.apache.org
Re: read byte offset information during xml parsing
Posted by Michael Glavassevich <mr...@ca.ibm.com>.
Hello Xiaoming,
In general byte offsets aren't available to the parser. The document
scanners read from a java.io.Reader so the byte to character decoding is
being done at a lower level. The parser only sees the decoded characters.
If what you actually want is the character offset, we made some changes to
XNI last year (they're in CVS) to expose this information in
org.apache.xerces.xni.XMLLocator. If you're using DOM Level 3, we've also
made the character offset available to DOMLocator [1]. You'll get this
functionality from the latest jars [2].
[1]
http://www.w3.org/TR/2004/REC-DOM-Level-3-Core-20040407/core.html#Interfaces-DOMLocator
[2] http://brutus.apache.org/gump/public-jars/xml-xerces2/jars/
Xiaoming Liu <li...@lanl.gov> wrote on 01/03/2005 06:14:25 PM:
> hi,
>
> I am looking for a Java XML parser which supports reading byte offset
> information during xml parsing, e.g. in '<foo><bar></bar></foo>', the
> parser can report '<bar>' starts from byte 5; and '</bar>' starts from
> byte 10 .
>
> I went through standard APIs like DOM, SAX, and XMLPull and cannot find
> related APIs. In Sax, the nearest interface is org.xml.sax.Locator. I
also
> checked Xerces XNI and found the nearest class is
> org.apache.xerces.xni.XMLLocator. In either class, only line number and
> column number are reported.
>
> However, similar functions are provided in other languages, such as the
> "XML_GetCurrentByteIndex" of expat parser (C, perl).
>
> so my question is whether there is a Java XML Parser reporting byte
offset
> information during parsing, and if not, is there any plan to
> implement this feature?
>
> many thanks,
> Xiaoming
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: xerces-j-user-unsubscribe@xml.apache.org
> For additional commands, e-mail: xerces-j-user-help@xml.apache.org
>
Michael Glavassevich
XML Parser Development
IBM Toronto Lab
E-mail: mrglavas@ca.ibm.com
E-mail: mrglavas@apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-j-user-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-j-user-help@xml.apache.org