You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@abdera.apache.org by Garrett Rooney <ro...@electricjellyfish.net> on 2006/07/13 00:23:59 UTC
Async Parsing?
Seeing the AsyncWeb proposal on the incubator general list today made
me think about async parsing in Abdera. It looks, at the moment, like
we don't currently provide a way to incrementally parse documents, you
just hand a stream to the parser and let it go. Is there any interest
in providing an API where you push data into the parser? It doesn't
look like our current Axiom based implementation could support that,
but I suppose someone who was sufficiently motivated could implement
another back end.
Anyway, I have no use for something that could do that right now, but
I figured I'd toss the idea against the wall and see if it sounds like
fun to someone.
-garrett
Re: Async Parsing?
Posted by James M Snell <ja...@gmail.com>.
Garrett Rooney wrote:
> [snip]
> This is neat, but it's not really what I'm thinking of.
I know, but it's still neat ;-)
I'll reread your note later tonight and think about the use case a bit more.
- James
Re: Async Parsing?
Posted by Garrett Rooney <ro...@electricjellyfish.net>.
On 7/12/06, James M Snell <ja...@gmail.com> wrote:
> I'm not sure I could reasonably envision any use of nonblocking i/o
> operations in an xml parser. I'm not sure if I've ever seen anyone do
> it before.
Well, I wouldn't expect that you'd put the nonblocking IO inside the
parser, it would be more like you'd be using nonblocking IO to pull
data off the wire and then passing that data off to a SAX style parser
once you get it.
> In any case, I figured you might find this entertaining:
>
> http://www.snellspace.com/wp/?p=381
>
> http://danga.com:8081/atom-stream.xml is a never-ending xml stream.
>
> URL url = new URL("http://danga.com:8081/atom-stream.xml");
> // we only care about the feed title and alternate link,
> // we'll ignore everything else
> ParseFilter filter = new WhiteListParseFilter();
> filter.add(new QName("atomStream"));
> filter.add(Constants.FEED);
> filter.add(Constants.TITLE);
> filter.add(Constants.LINK);
> ParserOptions options = Parser.INSTANCE.getDefaultParserOptions();
> options.setParseFilter(filter);
> Document doc = Parser.INSTANCE.parse(
> url.openStream(),(URI)null,options);
> Element el = doc.getRoot();
> // get the first feed in the stream, then continue to iterate
> // from there, printing the title and alt link to the console
> Feed feed = el.getFirstChild(Constants.FEED);
> while (feed != null) {
> System.out.println(
> feed.getTitle() + "t" + feed.getAlternateLink().getHref());
> Feed next = feed.getNextSibling(Constants.FEED);
> feed.discard();
> feed = next;
> }
>
> There are some memory-creep issues so I wouldn't recommend keeping this
> running forever :-)
This is neat, but it's not really what I'm thinking of. The use case
I was more concerned about would be a crawler that's trying to pull
down a scary amount of data, but doesn't want to devote a thread to
each one, so as it gets data down it hands it off to a parser as it
gets it. Now, practically speaking it's debatable if you'd want to
actually do this, it might make more sense to spool the data off
someplace and then parse the feed after you've got it all, unless of
course you're talking about a never ending atom feed ;-)
-garrett
Re: Async Parsing?
Posted by James M Snell <ja...@gmail.com>.
I'm not sure I could reasonably envision any use of nonblocking i/o
operations in an xml parser. I'm not sure if I've ever seen anyone do
it before.
In any case, I figured you might find this entertaining:
http://www.snellspace.com/wp/?p=381
http://danga.com:8081/atom-stream.xml is a never-ending xml stream.
URL url = new URL("http://danga.com:8081/atom-stream.xml");
// we only care about the feed title and alternate link,
// we'll ignore everything else
ParseFilter filter = new WhiteListParseFilter();
filter.add(new QName("atomStream"));
filter.add(Constants.FEED);
filter.add(Constants.TITLE);
filter.add(Constants.LINK);
ParserOptions options = Parser.INSTANCE.getDefaultParserOptions();
options.setParseFilter(filter);
Document doc = Parser.INSTANCE.parse(
url.openStream(),(URI)null,options);
Element el = doc.getRoot();
// get the first feed in the stream, then continue to iterate
// from there, printing the title and alt link to the console
Feed feed = el.getFirstChild(Constants.FEED);
while (feed != null) {
System.out.println(
feed.getTitle() + "t" + feed.getAlternateLink().getHref());
Feed next = feed.getNextSibling(Constants.FEED);
feed.discard();
feed = next;
}
There are some memory-creep issues so I wouldn't recommend keeping this
running forever :-)
- James
Garrett Rooney wrote:
> On 7/12/06, James M Snell <ja...@gmail.com> wrote:
>> Right now documents are parsed incrementally. When I
>> Parser.INSTANCE.parse(...) what I'm handed back is not yet a fully
>> parsed document. As I go through the various getters, the document is
>> parsed up to whatever point it needs to answer that method call. It
>> could definitely be better, however.
>
> Sure, but if you call getRoot() it's still going to block until it
> reads enough off of the InputStream to parse the root element, right?
> There's no nonblocking interface.
>
> -garrett
>
Re: Async Parsing?
Posted by Garrett Rooney <ro...@electricjellyfish.net>.
On 7/12/06, James M Snell <ja...@gmail.com> wrote:
> Right now documents are parsed incrementally. When I
> Parser.INSTANCE.parse(...) what I'm handed back is not yet a fully
> parsed document. As I go through the various getters, the document is
> parsed up to whatever point it needs to answer that method call. It
> could definitely be better, however.
Sure, but if you call getRoot() it's still going to block until it
reads enough off of the InputStream to parse the root element, right?
There's no nonblocking interface.
-garrett
Re: Async Parsing?
Posted by James M Snell <ja...@gmail.com>.
Right now documents are parsed incrementally. When I
Parser.INSTANCE.parse(...) what I'm handed back is not yet a fully
parsed document. As I go through the various getters, the document is
parsed up to whatever point it needs to answer that method call. It
could definitely be better, however.
- James
Garrett Rooney wrote:
> Seeing the AsyncWeb proposal on the incubator general list today made
> me think about async parsing in Abdera. It looks, at the moment, like
> we don't currently provide a way to incrementally parse documents, you
> just hand a stream to the parser and let it go. Is there any interest
> in providing an API where you push data into the parser? It doesn't
> look like our current Axiom based implementation could support that,
> but I suppose someone who was sufficiently motivated could implement
> another back end.
>
> Anyway, I have no use for something that could do that right now, but
> I figured I'd toss the idea against the wall and see if it sounds like
> fun to someone.
>
> -garrett
>