You are viewing a plain text version of this content. The canonical link for it is here.
Posted to c-dev@xerces.apache.org by Dominik Stadler <ds...@ftisoft.at> on 2005/06/23 10:10:44 UTC
Split file that contains multiple XML
Hi,
We have a file that contains multiple XML-Messages in the form of:
<FIRSTMESSAGE>...</FIRSTMESSAGE><SECONDMESSAGE>...</SECONDMESSAGE>
I know this looks broken from the beginning, but we cannot change the
application that generates these kind of data, so we need a way to cope
with it.
How would I go about reading/splitting this? I don't think there is
functionality available to do this with Xerces, right?
I thought about using SAX to try to parse the complete text (we should get
an error at the second message) and then read the char/line information
from the errormessage, but this sounds like a hack to me, is there some
other way?
Thanks... Dominik.
---------------------------------------------------------------------
To unsubscribe, e-mail: c-dev-unsubscribe@xerces.apache.org
For additional commands, e-mail: c-dev-help@xerces.apache.org
Re: Split file that contains multiple XML
Posted by "Jeroen N. Witmond" <jn...@xs4all.nl>.
> We know, however, that every XML-Message starts on
> a new line.
In that case, forget that your data is XML and treat it as plain text with
the standard Unix utilities, as sketched below. You are warned that this
code has not been tested.
Jeroen.
File header.xml contains
<?xml version="1.0"?>
<!DOCTYPE STREET_REF SYSTEM "fti://repository/dtd/STREET_REF">
<STREET_REF>
File footer.xml contains
</STREET_REF>
(If I understand correctly, the top element name must be equal to the
first argument of the DOCTYPE.)
Commands to execute are
FILETOREAD=/path/to/your/pseudo-xml.file
grep -v '^<!DOCTYPE' $FILETOREAD | \
grep -v '^<?xml ' | \
cat header.xml - footer.xml | \
your-application
---------------------------------------------------------------------
To unsubscribe, e-mail: c-dev-unsubscribe@xerces.apache.org
For additional commands, e-mail: c-dev-help@xerces.apache.org
Re: Split file that contains multiple XML
Posted by Alex Dickinson <al...@zeugmasystems.com>.
Dominik Stadler wrote:
>On Thu, 23 Jun 2005 10:29:59 +0200, Axel Weiß wrote:
>
>Hi
>
>
>
>>you could try to make your data stream look like legal xml content, by
>>preceding the stream with a xml header and an opening root element. It
>>would then look like
>>
>><?xml version="1.0"?>
>><data-stream>
>><FIRSTMESSAGE>...</FIRSTMESSAGE><SECONDMESSAGE>...</SECONDMESSAGE>
>>
>>This can be parsed with SAX parser, even if the root element never will
>>be closed.
>>
>>
>
>Thanks for your suggestion, I forgot to mention that each XML-Message
>might contain header-information, so I could have something like
>
><!DOCTYPE
>STREET_REF SYSTEM "fti://repository/dtd/STREET_REF">
><FIRSTMESSAGE>...</FIRSTMESSAGE>
><?xml version="1.0" encoding="UTF-8" standalone="no" ?> <!DOCTYPE
>STREET_REF SYSTEM "fti://repository/dtd/STREET_REF">
><SECONDMESSAGE>...</SECONDMESSAGE>
>
>which would not lead to a valid XML-Message, because there are
><?xml...>- and <!DOCTYPE..>-elements somewhere in between...
>
>We know, however, that every XML-Message starts on
>a new line, so we might need to concatenate lines until we have a complete
>XML-Message and then use that... Also kind of ugly and potentially
>time-consuming, but I don't see any other way except to write a small
>specialized XML-Grammar and this still wouldn't be able to parse
>messages in encodings like UTF-16.
>
>Dominik.
>
>
I was in a situation similar to this. I was reading multiple XML
messages from a socket. The protcol required that the XML messages were
to be delimited by a special character sequence. I implemented a
InputSource and a BinInputStream that could determine when the end of
each XML message was reached.
The custom BinInputStream have to implement the following methods:
virtual unsigned int curPos() const;
virtual unsigned int readBytes( XMLByte* const toFill, const unsigned
int maxToRead);
And the custom InputSource has return the custom BinInputStream in the
method:
virtual BinInputStream* makeStream() const;
Hope it helps,
-Alex
---------------------------------------------------------------------
To unsubscribe, e-mail: c-dev-unsubscribe@xerces.apache.org
For additional commands, e-mail: c-dev-help@xerces.apache.org
Re: Split file that contains multiple XML
Posted by Axel Weiß <aw...@informatik.hu-berlin.de>.
Am Donnerstag, 23. Juni 2005 13:52 schrieb Dominik Stadler:
> On Thu, 23 Jun 2005 10:29:59 +0200, Axel Weiß wrote:
>
> Hi
>
> > you could try to make your data stream look like legal xml content,
> > by preceding the stream with a xml header and an opening root
> > element. It would then look like
> >
> > <?xml version="1.0"?>
> > <data-stream>
> > <FIRSTMESSAGE>...</FIRSTMESSAGE><SECONDMESSAGE>...</SECONDMESSAGE>
> >
> > This can be parsed with SAX parser, even if the root element never
> > will be closed.
>
> Thanks for your suggestion, I forgot to mention that each XML-Message
> might contain header-information, so I could have something like
>
> <!DOCTYPE
> STREET_REF SYSTEM "fti://repository/dtd/STREET_REF">
> <FIRSTMESSAGE>...</FIRSTMESSAGE>
> <?xml version="1.0" encoding="UTF-8" standalone="no" ?> <!DOCTYPE
> STREET_REF SYSTEM "fti://repository/dtd/STREET_REF">
> <SECONDMESSAGE>...</SECONDMESSAGE>
>
> which would not lead to a valid XML-Message, because there are
> <?xml...>- and <!DOCTYPE..>-elements somewhere in between...
Dominik,
then I'd suggest to try using the SAX-parser. What you have is a
sequence of valid xml contents. You will the be able to parse each
message, since the SAX parser tells you when it's input has reached the
end, that is the closing root element tag. Your input then points to
the beginning of the new message, and you can parse it with the SAX
parser being reset.
> We know, however, that every XML-Message starts on
> a new line, so we might need to concatenate lines until we have a
> complete XML-Message and then use that... Also kind of ugly and
> potentially time-consuming, but I don't see any other way except to
> write a small specialized XML-Grammar and this still wouldn't be able
> to parse messages in encodings like UTF-16.
I don't think you need to write your own parser. Just divide your input
stream into xml documents (the SAX parser will help you with this). You
should even be able to parse arbitrary encodings following each other.
Cheers,
Axel
--
Humboldt-Universität zu Berlin
Institut für Informatik
Signalverarbeitung und Mustererkennung
Dipl.-Inf. Axel Weiß
Rudower Chaussee 25
12489 Berlin-Adlershof
+49-30-2093-3050
** www.freesp.de **
---------------------------------------------------------------------
To unsubscribe, e-mail: c-dev-unsubscribe@xerces.apache.org
For additional commands, e-mail: c-dev-help@xerces.apache.org
Re: Split file that contains multiple XML
Posted by Dominik Stadler <ds...@thegoldensource.com>.
On Thu, 23 Jun 2005 10:29:59 +0200, Axel Weiß wrote:
Hi
> you could try to make your data stream look like legal xml content, by
> preceding the stream with a xml header and an opening root element. It
> would then look like
>
> <?xml version="1.0"?>
> <data-stream>
> <FIRSTMESSAGE>...</FIRSTMESSAGE><SECONDMESSAGE>...</SECONDMESSAGE>
>
> This can be parsed with SAX parser, even if the root element never will
> be closed.
Thanks for your suggestion, I forgot to mention that each XML-Message
might contain header-information, so I could have something like
<!DOCTYPE
STREET_REF SYSTEM "fti://repository/dtd/STREET_REF">
<FIRSTMESSAGE>...</FIRSTMESSAGE>
<?xml version="1.0" encoding="UTF-8" standalone="no" ?> <!DOCTYPE
STREET_REF SYSTEM "fti://repository/dtd/STREET_REF">
<SECONDMESSAGE>...</SECONDMESSAGE>
which would not lead to a valid XML-Message, because there are
<?xml...>- and <!DOCTYPE..>-elements somewhere in between...
We know, however, that every XML-Message starts on
a new line, so we might need to concatenate lines until we have a complete
XML-Message and then use that... Also kind of ugly and potentially
time-consuming, but I don't see any other way except to write a small
specialized XML-Grammar and this still wouldn't be able to parse
messages in encodings like UTF-16.
Dominik.
---------------------------------------------------------------------
To unsubscribe, e-mail: c-dev-unsubscribe@xerces.apache.org
For additional commands, e-mail: c-dev-help@xerces.apache.org
Re: Split file that contains multiple XML
Posted by Axel Weiß <aw...@informatik.hu-berlin.de>.
Am Donnerstag, 23. Juni 2005 10:10 schrieb Dominik Stadler:
> Hi,
>
> We have a file that contains multiple XML-Messages in the form of:
>
> <FIRSTMESSAGE>...</FIRSTMESSAGE><SECONDMESSAGE>...</SECONDMESSAGE>
>
> I know this looks broken from the beginning, but we cannot change the
> application that generates these kind of data, so we need a way to
> cope with it.
>
> How would I go about reading/splitting this? I don't think there is
> functionality available to do this with Xerces, right?
>
> I thought about using SAX to try to parse the complete text (we
> should get an error at the second message) and then read the
> char/line information from the errormessage, but this sounds like a
> hack to me, is there some other way?
Hi Dominik,
you could try to make your data stream look like legal xml content, by
preceding the stream with a xml header and an opening root element. It
would then look like
<?xml version="1.0"?>
<data-stream>
<FIRSTMESSAGE>...</FIRSTMESSAGE><SECONDMESSAGE>...</SECONDMESSAGE>
This can be parsed with SAX parser, even if the root element never will
be closed.
Cheers,
Axel
--
Humboldt-Universität zu Berlin
Institut für Informatik
Signalverarbeitung und Mustererkennung
Dipl.-Inf. Axel Weiß
Rudower Chaussee 25
12489 Berlin-Adlershof
+49-30-2093-3050
** www.freesp.de **
---------------------------------------------------------------------
To unsubscribe, e-mail: c-dev-unsubscribe@xerces.apache.org
For additional commands, e-mail: c-dev-help@xerces.apache.org