You are viewing a plain text version of this content. The canonical link for it is here.
Posted to droids-dev@incubator.apache.org by Mingfai <mi...@gmail.com> on 2009/04/02 10:53:33 UTC

HTML outllink extraction

hi,

The default LinkExtractor seems to be quite simple. (too simple) It mainly
uses URI.resolve and only cater the # and javascript scenarios. (from
LinkExtractor.java getURI) Simple usage link resolving a <a
href="test.html"> for new URI("http://www.google.com") will be wrong as it
will return a http://www.google.comtest.html". And there are many case that
the URI.resolve doesn't cater. It seems to me we need to do some works at
this area to make Droids more usable. Does anyone have any experience in out
link extraction?

I'm trying to see how other frameworks handle out link extraction and looked
at:
http://svn.apache.org/viewvc/lucene/nutch/trunk/src/java/org/apache/nutch/parse/OutlinkExtractor.java?view=log
https://archive-crawler.svn.sourceforge.net/svnroot/archive-crawler/trunk/heritrix2/engine/src/main/java/org/archive/extractor/RegexpHTMLLinkExtractor.java
(Heritrix's JavaDoc shows they have given some good thought in handling
different tags and attributes)

What do you think if I add a wiki page that list out some scenarios of
outlink handling (i.e. the requirement)? Or does anyone know if any of the
many Java crawler projects have documentation at this area?

regards,
mingfai

Re: HTML outllink extraction

Posted by Mingfai <mi...@gmail.com>.
your comment in JIRA reminded me of another issue that i have encountered
when I'm working with Droids. For web server uses default index file, e.g.
http://www.apache.org/index.html
http://www.apache.org/

two links refer to the same document. Ideally, the crawler framework should
have some ways to manage this. I don't have a clear suggestion in mind,
however. If i didn't recall it incorrectly, one of the other crawlers
actually did some fake request to test certain server behavior, e.g. whether
a 404 error result as a normal 200 page will error message. Maybe there is a
single configuration (disabled by default). if enabled, when a "/index.*" is
encountered, it will make a request to "/" to test if the content is more or
less the same. If a host is detected as with default index, URI with the
matched index file are assumed to be identical as '/'. This is one of the
approaches we could consider.

Now, we have to code the handling logic in our own handler.

regards,
mingfai





On Thu, Apr 2, 2009 at 8:48 PM, Mingfai <mi...@gmail.com> wrote:

> thx, I looked at DROIDS-8 and DROIDS-11, and it seems it is better to
> create a new issue to keep track of resolve failure cases.
> https://issues.apache.org/jira/browse/DROIDS-45
>
> Tiki/NekoHTML is responsible for parsing and provision of the SAX event
> based parser. The actual URI resolving logic is implemented in Droids as of
> now.  And in fact, in the 2nd last comment in DROIDS-8, you guys think the
> Link Extractor doesn't depend on Tiki and should be moved to core.
>
> I wonder if you agree the default Droids Link Extractor should behave as
> same as a Mozilla browser for basic HTML link, ideally? (and for sure it
> can't handle JavaScript link)
>
> re. JDK bug, I'm using the JDK 1.6.0_12 and got the problem. Maybe no one
> has ever reported the issue or they just don't care about that. My point is,
> it seems we have to workaround the bug rather than hoping Sun Microsystems
> to fix the bug and every body upgrade to the latest revision. :-)
>
> Regards,
> mingfai
>
>
>
> On Thu, Apr 2, 2009 at 8:19 PM, Thorsten Scherler <
> thorsten.scherler.ext@juntadeandalucia.es> wrote:
>
>> On Thu, 2009-04-02 at 19:38 +0800, Mingfai wrote:
>> > let's just look at the specific case first. Maybe I have jumped to the
>> > conclusion that the Link Extraction feature is too simple too soon.
>> >
>> > At line 139 of LinkExtractor.java, it uses URI.resolve(String) to
>> resolve a
>> > URI.
>> >       if (!target.toLowerCase().startsWith("javascript")
>> >           && !target.contains(":/")) {
>> > 139:        return base.getURI().resolve(target.split("#")[0]);
>> >       }
>> >       else if (!target.toLowerCase().startsWith("javascript")) {
>> >         return new URI(target.split("#")[0]);
>> >       }
>> >
>> > When I test the URI API with:
>> >   new URI("http://www.google.com").resolve("index.php")
>> > it resolves the url to "http://www.google.comindex.php"
>> >
>> > if you didn't mean it is a bug with my JDK, then we need to specially
>> append
>> > a "/" prefix
>>
>> Hmm,
>>
>> http://java.sun.com/j2se/1.4.2/docs/api/java/net/URI.html#resolve(java.net.URI)<http://java.sun.com/j2se/1.4.2/docs/api/java/net/URI.html#resolve%28java.net.URI%29>
>> "...
>> 3.Otherwise the new URI's authority component is copied from this URI,
>> and its path is computed as follows:
>>
>>     A. If the given URI's path is absolute then the new URI's path is
>>        taken from the given URI.
>>
>>     B. Otherwise the given URI's path is relative, and so the new URI's
>>        path is computed by resolving the path of the given URI against
>>        the path of this URI. This is done by concatenating all but the
>>        last segment of this URI's path, if any, with the given URI's
>>        path and then normalizing the result as if by invoking the
>>        normalize method.
>> ..."
>>
>> That sounds that new URI("http://www.apache.org").resolve("index.html")
>> should return http://www.apache.org/index.html. Since it reads: "the
>> result as if by invoking the normalize method"
>>
>> http://java.sun.com/j2se/1.4.2/docs/api/java/net/URI.html#normalize()<http://java.sun.com/j2se/1.4.2/docs/api/java/net/URI.html#normalize%28%29>
>>
>>
>> >
>> > And previously, I found another scenario that doesn't work, when there
>> is a
>> > link <a href="?test=true">test</a> under www.google.com/index.php , it
>> > resolves to www.google.com/?test=true rather than
>> > www.google.com/index.php?test=true like in a web browser.
>> >
>> > This makes me feel there are many special scenario that a crawler need
>> to
>> > cater. What do you think? Is it really so simple? My suggest to add a
>> page
>> > is for listing those special scenarios, that sometimes maybe just cause
>> by
>> > non-standard usage.
>>
>> Actually that should normally be handled by the above linked methods.
>> Please comment on issue DROIDS-8/DROIDS-11 if you find that the link
>> extraction is not working as expected.
>>
>> salu2
>>
>> >
>> > regards,
>> > mingfai
>> >
>> >
>> >
>> >
>> > On Thu, Apr 2, 2009 at 7:24 PM, Thorsten Scherler <
>> > thorsten.scherler.ext@juntadeandalucia.es> wrote:
>> >
>> > > On Thu, 2009-04-02 at 18:53 +0800, Mingfai wrote:
>> > > > hi,
>> > > >
>> > > > The default LinkExtractor seems to be quite simple. (too simple) It
>> > > mainly
>> > > > uses URI.resolve and only cater the # and javascript scenarios.
>> (from
>> > > > LinkExtractor.java getURI) Simple usage link resolving a <a
>> > > > href="test.html"> for new URI("http://www.google.com") will be
>> wrong as
>> > > it
>> > > > will return a http://www.google.comtest.html".
>> > >
>> > > Well the link extraction always worked well. The case you just pointed
>> > > out looks like a bug BUT if you mean new URL
>> > > ("http://testServer.com","test.html)) then have a look at
>> > >
>> http://java.sun.com/j2se/1.4.2/docs/api/java/net/URL.html#URL(java.net.URL<http://java.sun.com/j2se/1.4.2/docs/api/java/net/URL.html#URL%28java.net.URL>
>> <
>> http://java.sun.com/j2se/1.4.2/docs/api/java/net/URL.html#URL%28java.net.URL
>> >,
>> > > java.lang.String)
>> > >
>> > > > And there are many case that
>> > > > the URI.resolve doesn't cater. It seems to me we need to do some
>> works at
>> > > > this area to make Droids more usable. Does anyone have any
>> experience in
>> > > out
>> > > > link extraction?
>> > >
>> > > Enhancements are always welcome however the link extraction should
>> work
>> > > fine. At least when I last looked at it was fine. The limitation ATM
>> is
>> > > the extraction of jscript generated links.
>> > >
>> > > >
>> > > > I'm trying to see how other frameworks handle out link extraction
>> and
>> > > looked
>> > > > at:
>> > > >
>> > >
>> http://svn.apache.org/viewvc/lucene/nutch/trunk/src/java/org/apache/nutch/parse/OutlinkExtractor.java?view=log
>> > >
>> > > Funny enough that have been the base of droids outlink extraction in
>> the
>> > > first version I hacked.
>> > >
>> > > >
>> > >
>> https://archive-crawler.svn.sourceforge.net/svnroot/archive-crawler/trunk/heritrix2/engine/src/main/java/org/archive/extractor/RegexpHTMLLinkExtractor.java
>> > > > (Heritrix's JavaDoc shows they have given some good thought in
>> handling
>> > > > different tags and attributes)
>> > > >
>> > > > What do you think if I add a wiki page that list out some scenarios
>> of
>> > > > outlink handling (i.e. the requirement)? Or does anyone know if any
>> of
>> > > the
>> > > > many Java crawler projects have documentation at this area?
>> > >
>> > > If you do not look into jscript/ajax link extraction then there is no
>> > > secret to it. Either go with xpath expression or e.g. for plain text
>> > > with regexp. Please fell free to open a wiki page around the issue.
>> > >
>> > > salu2
>> > >
>> > > >
>> > > > regards,
>> > > > mingfai
>> > > --
>> > > Thorsten Scherler <thorsten.at.apache.org>
>> > > Open Source Java <consulting, training and solutions>
>> > >
>> > > Sociedad Andaluza para el Desarrollo de la Sociedad
>> > > de la Información, S.A.U. (SADESI)
>> > >
>> > >
>> > >
>> > >
>> > >
>> --
>> Thorsten Scherler <thorsten.at.apache.org>
>> Open Source Java <consulting, training and solutions>
>>
>> Sociedad Andaluza para el Desarrollo de la Sociedad
>> de la Información, S.A.U. (SADESI)
>>
>>
>>
>>
>>
>

Re: HTML outllink extraction

Posted by Mingfai <mi...@gmail.com>.
thx, I looked at DROIDS-8 and DROIDS-11, and it seems it is better to create
a new issue to keep track of resolve failure cases.
https://issues.apache.org/jira/browse/DROIDS-45

Tiki/NekoHTML is responsible for parsing and provision of the SAX event
based parser. The actual URI resolving logic is implemented in Droids as of
now.  And in fact, in the 2nd last comment in DROIDS-8, you guys think the
Link Extractor doesn't depend on Tiki and should be moved to core.

I wonder if you agree the default Droids Link Extractor should behave as
same as a Mozilla browser for basic HTML link, ideally? (and for sure it
can't handle JavaScript link)

re. JDK bug, I'm using the JDK 1.6.0_12 and got the problem. Maybe no one
has ever reported the issue or they just don't care about that. My point is,
it seems we have to workaround the bug rather than hoping Sun Microsystems
to fix the bug and every body upgrade to the latest revision. :-)

Regards,
mingfai


On Thu, Apr 2, 2009 at 8:19 PM, Thorsten Scherler <
thorsten.scherler.ext@juntadeandalucia.es> wrote:

> On Thu, 2009-04-02 at 19:38 +0800, Mingfai wrote:
> > let's just look at the specific case first. Maybe I have jumped to the
> > conclusion that the Link Extraction feature is too simple too soon.
> >
> > At line 139 of LinkExtractor.java, it uses URI.resolve(String) to resolve
> a
> > URI.
> >       if (!target.toLowerCase().startsWith("javascript")
> >           && !target.contains(":/")) {
> > 139:        return base.getURI().resolve(target.split("#")[0]);
> >       }
> >       else if (!target.toLowerCase().startsWith("javascript")) {
> >         return new URI(target.split("#")[0]);
> >       }
> >
> > When I test the URI API with:
> >   new URI("http://www.google.com").resolve("index.php")
> > it resolves the url to "http://www.google.comindex.php"
> >
> > if you didn't mean it is a bug with my JDK, then we need to specially
> append
> > a "/" prefix
>
> Hmm,
>
> http://java.sun.com/j2se/1.4.2/docs/api/java/net/URI.html#resolve(java.net.URI)<http://java.sun.com/j2se/1.4.2/docs/api/java/net/URI.html#resolve%28java.net.URI%29>
> "...
> 3.Otherwise the new URI's authority component is copied from this URI,
> and its path is computed as follows:
>
>     A. If the given URI's path is absolute then the new URI's path is
>        taken from the given URI.
>
>     B. Otherwise the given URI's path is relative, and so the new URI's
>        path is computed by resolving the path of the given URI against
>        the path of this URI. This is done by concatenating all but the
>        last segment of this URI's path, if any, with the given URI's
>        path and then normalizing the result as if by invoking the
>        normalize method.
> ..."
>
> That sounds that new URI("http://www.apache.org").resolve("index.html")
> should return http://www.apache.org/index.html. Since it reads: "the
> result as if by invoking the normalize method"
>
> http://java.sun.com/j2se/1.4.2/docs/api/java/net/URI.html#normalize()<http://java.sun.com/j2se/1.4.2/docs/api/java/net/URI.html#normalize%28%29>
>
>
> >
> > And previously, I found another scenario that doesn't work, when there is
> a
> > link <a href="?test=true">test</a> under www.google.com/index.php , it
> > resolves to www.google.com/?test=true rather than
> > www.google.com/index.php?test=true like in a web browser.
> >
> > This makes me feel there are many special scenario that a crawler need to
> > cater. What do you think? Is it really so simple? My suggest to add a
> page
> > is for listing those special scenarios, that sometimes maybe just cause
> by
> > non-standard usage.
>
> Actually that should normally be handled by the above linked methods.
> Please comment on issue DROIDS-8/DROIDS-11 if you find that the link
> extraction is not working as expected.
>
> salu2
>
> >
> > regards,
> > mingfai
> >
> >
> >
> >
> > On Thu, Apr 2, 2009 at 7:24 PM, Thorsten Scherler <
> > thorsten.scherler.ext@juntadeandalucia.es> wrote:
> >
> > > On Thu, 2009-04-02 at 18:53 +0800, Mingfai wrote:
> > > > hi,
> > > >
> > > > The default LinkExtractor seems to be quite simple. (too simple) It
> > > mainly
> > > > uses URI.resolve and only cater the # and javascript scenarios. (from
> > > > LinkExtractor.java getURI) Simple usage link resolving a <a
> > > > href="test.html"> for new URI("http://www.google.com") will be wrong
> as
> > > it
> > > > will return a http://www.google.comtest.html".
> > >
> > > Well the link extraction always worked well. The case you just pointed
> > > out looks like a bug BUT if you mean new URL
> > > ("http://testServer.com","test.html)) then have a look at
> > >
> http://java.sun.com/j2se/1.4.2/docs/api/java/net/URL.html#URL(java.net.URL<http://java.sun.com/j2se/1.4.2/docs/api/java/net/URL.html#URL%28java.net.URL>
> <
> http://java.sun.com/j2se/1.4.2/docs/api/java/net/URL.html#URL%28java.net.URL
> >,
> > > java.lang.String)
> > >
> > > > And there are many case that
> > > > the URI.resolve doesn't cater. It seems to me we need to do some
> works at
> > > > this area to make Droids more usable. Does anyone have any experience
> in
> > > out
> > > > link extraction?
> > >
> > > Enhancements are always welcome however the link extraction should work
> > > fine. At least when I last looked at it was fine. The limitation ATM is
> > > the extraction of jscript generated links.
> > >
> > > >
> > > > I'm trying to see how other frameworks handle out link extraction and
> > > looked
> > > > at:
> > > >
> > >
> http://svn.apache.org/viewvc/lucene/nutch/trunk/src/java/org/apache/nutch/parse/OutlinkExtractor.java?view=log
> > >
> > > Funny enough that have been the base of droids outlink extraction in
> the
> > > first version I hacked.
> > >
> > > >
> > >
> https://archive-crawler.svn.sourceforge.net/svnroot/archive-crawler/trunk/heritrix2/engine/src/main/java/org/archive/extractor/RegexpHTMLLinkExtractor.java
> > > > (Heritrix's JavaDoc shows they have given some good thought in
> handling
> > > > different tags and attributes)
> > > >
> > > > What do you think if I add a wiki page that list out some scenarios
> of
> > > > outlink handling (i.e. the requirement)? Or does anyone know if any
> of
> > > the
> > > > many Java crawler projects have documentation at this area?
> > >
> > > If you do not look into jscript/ajax link extraction then there is no
> > > secret to it. Either go with xpath expression or e.g. for plain text
> > > with regexp. Please fell free to open a wiki page around the issue.
> > >
> > > salu2
> > >
> > > >
> > > > regards,
> > > > mingfai
> > > --
> > > Thorsten Scherler <thorsten.at.apache.org>
> > > Open Source Java <consulting, training and solutions>
> > >
> > > Sociedad Andaluza para el Desarrollo de la Sociedad
> > > de la Información, S.A.U. (SADESI)
> > >
> > >
> > >
> > >
> > >
> --
> Thorsten Scherler <thorsten.at.apache.org>
> Open Source Java <consulting, training and solutions>
>
> Sociedad Andaluza para el Desarrollo de la Sociedad
> de la Información, S.A.U. (SADESI)
>
>
>
>
>

Re: HTML outllink extraction

Posted by Thorsten Scherler <th...@juntadeandalucia.es>.
On Thu, 2009-04-02 at 19:38 +0800, Mingfai wrote:
> let's just look at the specific case first. Maybe I have jumped to the
> conclusion that the Link Extraction feature is too simple too soon.
> 
> At line 139 of LinkExtractor.java, it uses URI.resolve(String) to resolve a
> URI.
>       if (!target.toLowerCase().startsWith("javascript")
>           && !target.contains(":/")) {
> 139:        return base.getURI().resolve(target.split("#")[0]);
>       }
>       else if (!target.toLowerCase().startsWith("javascript")) {
>         return new URI(target.split("#")[0]);
>       }
> 
> When I test the URI API with:
>   new URI("http://www.google.com").resolve("index.php")
> it resolves the url to "http://www.google.comindex.php"
> 
> if you didn't mean it is a bug with my JDK, then we need to specially append
> a "/" prefix

Hmm, 
http://java.sun.com/j2se/1.4.2/docs/api/java/net/URI.html#resolve(java.net.URI)
"...
3.Otherwise the new URI's authority component is copied from this URI,
and its path is computed as follows: 

     A. If the given URI's path is absolute then the new URI's path is
        taken from the given URI. 
        
     B. Otherwise the given URI's path is relative, and so the new URI's
        path is computed by resolving the path of the given URI against
        the path of this URI. This is done by concatenating all but the
        last segment of this URI's path, if any, with the given URI's
        path and then normalizing the result as if by invoking the
        normalize method. 
..."

That sounds that new URI("http://www.apache.org").resolve("index.html")
should return http://www.apache.org/index.html. Since it reads: "the
result as if by invoking the normalize method" 

http://java.sun.com/j2se/1.4.2/docs/api/java/net/URI.html#normalize()

        
> 
> And previously, I found another scenario that doesn't work, when there is a
> link <a href="?test=true">test</a> under www.google.com/index.php , it
> resolves to www.google.com/?test=true rather than
> www.google.com/index.php?test=true like in a web browser.
> 
> This makes me feel there are many special scenario that a crawler need to
> cater. What do you think? Is it really so simple? My suggest to add a page
> is for listing those special scenarios, that sometimes maybe just cause by
> non-standard usage.

Actually that should normally be handled by the above linked methods.
Please comment on issue DROIDS-8/DROIDS-11 if you find that the link
extraction is not working as expected. 

salu2

> 
> regards,
> mingfai
> 
> 
> 
> 
> On Thu, Apr 2, 2009 at 7:24 PM, Thorsten Scherler <
> thorsten.scherler.ext@juntadeandalucia.es> wrote:
> 
> > On Thu, 2009-04-02 at 18:53 +0800, Mingfai wrote:
> > > hi,
> > >
> > > The default LinkExtractor seems to be quite simple. (too simple) It
> > mainly
> > > uses URI.resolve and only cater the # and javascript scenarios. (from
> > > LinkExtractor.java getURI) Simple usage link resolving a <a
> > > href="test.html"> for new URI("http://www.google.com") will be wrong as
> > it
> > > will return a http://www.google.comtest.html".
> >
> > Well the link extraction always worked well. The case you just pointed
> > out looks like a bug BUT if you mean new URL
> > ("http://testServer.com","test.html)) then have a look at
> > http://java.sun.com/j2se/1.4.2/docs/api/java/net/URL.html#URL(java.net.URL<http://java.sun.com/j2se/1.4.2/docs/api/java/net/URL.html#URL%28java.net.URL>,
> > java.lang.String)
> >
> > > And there are many case that
> > > the URI.resolve doesn't cater. It seems to me we need to do some works at
> > > this area to make Droids more usable. Does anyone have any experience in
> > out
> > > link extraction?
> >
> > Enhancements are always welcome however the link extraction should work
> > fine. At least when I last looked at it was fine. The limitation ATM is
> > the extraction of jscript generated links.
> >
> > >
> > > I'm trying to see how other frameworks handle out link extraction and
> > looked
> > > at:
> > >
> > http://svn.apache.org/viewvc/lucene/nutch/trunk/src/java/org/apache/nutch/parse/OutlinkExtractor.java?view=log
> >
> > Funny enough that have been the base of droids outlink extraction in the
> > first version I hacked.
> >
> > >
> > https://archive-crawler.svn.sourceforge.net/svnroot/archive-crawler/trunk/heritrix2/engine/src/main/java/org/archive/extractor/RegexpHTMLLinkExtractor.java
> > > (Heritrix's JavaDoc shows they have given some good thought in handling
> > > different tags and attributes)
> > >
> > > What do you think if I add a wiki page that list out some scenarios of
> > > outlink handling (i.e. the requirement)? Or does anyone know if any of
> > the
> > > many Java crawler projects have documentation at this area?
> >
> > If you do not look into jscript/ajax link extraction then there is no
> > secret to it. Either go with xpath expression or e.g. for plain text
> > with regexp. Please fell free to open a wiki page around the issue.
> >
> > salu2
> >
> > >
> > > regards,
> > > mingfai
> > --
> > Thorsten Scherler <thorsten.at.apache.org>
> > Open Source Java <consulting, training and solutions>
> >
> > Sociedad Andaluza para el Desarrollo de la Sociedad
> > de la Información, S.A.U. (SADESI)
> >
> >
> >
> >
> >
-- 
Thorsten Scherler <thorsten.at.apache.org>
Open Source Java <consulting, training and solutions>

Sociedad Andaluza para el Desarrollo de la Sociedad 
de la Información, S.A.U. (SADESI)





Re: HTML outllink extraction

Posted by Mingfai <mi...@gmail.com>.
let's just look at the specific case first. Maybe I have jumped to the
conclusion that the Link Extraction feature is too simple too soon.

At line 139 of LinkExtractor.java, it uses URI.resolve(String) to resolve a
URI.
      if (!target.toLowerCase().startsWith("javascript")
          && !target.contains(":/")) {
139:        return base.getURI().resolve(target.split("#")[0]);
      }
      else if (!target.toLowerCase().startsWith("javascript")) {
        return new URI(target.split("#")[0]);
      }

When I test the URI API with:
  new URI("http://www.google.com").resolve("index.php")
it resolves the url to "http://www.google.comindex.php"

if you didn't mean it is a bug with my JDK, then we need to specially append
a "/" prefix

And previously, I found another scenario that doesn't work, when there is a
link <a href="?test=true">test</a> under www.google.com/index.php , it
resolves to www.google.com/?test=true rather than
www.google.com/index.php?test=true like in a web browser.

This makes me feel there are many special scenario that a crawler need to
cater. What do you think? Is it really so simple? My suggest to add a page
is for listing those special scenarios, that sometimes maybe just cause by
non-standard usage.

regards,
mingfai




On Thu, Apr 2, 2009 at 7:24 PM, Thorsten Scherler <
thorsten.scherler.ext@juntadeandalucia.es> wrote:

> On Thu, 2009-04-02 at 18:53 +0800, Mingfai wrote:
> > hi,
> >
> > The default LinkExtractor seems to be quite simple. (too simple) It
> mainly
> > uses URI.resolve and only cater the # and javascript scenarios. (from
> > LinkExtractor.java getURI) Simple usage link resolving a <a
> > href="test.html"> for new URI("http://www.google.com") will be wrong as
> it
> > will return a http://www.google.comtest.html".
>
> Well the link extraction always worked well. The case you just pointed
> out looks like a bug BUT if you mean new URL
> ("http://testServer.com","test.html)) then have a look at
> http://java.sun.com/j2se/1.4.2/docs/api/java/net/URL.html#URL(java.net.URL<http://java.sun.com/j2se/1.4.2/docs/api/java/net/URL.html#URL%28java.net.URL>,
> java.lang.String)
>
> > And there are many case that
> > the URI.resolve doesn't cater. It seems to me we need to do some works at
> > this area to make Droids more usable. Does anyone have any experience in
> out
> > link extraction?
>
> Enhancements are always welcome however the link extraction should work
> fine. At least when I last looked at it was fine. The limitation ATM is
> the extraction of jscript generated links.
>
> >
> > I'm trying to see how other frameworks handle out link extraction and
> looked
> > at:
> >
> http://svn.apache.org/viewvc/lucene/nutch/trunk/src/java/org/apache/nutch/parse/OutlinkExtractor.java?view=log
>
> Funny enough that have been the base of droids outlink extraction in the
> first version I hacked.
>
> >
> https://archive-crawler.svn.sourceforge.net/svnroot/archive-crawler/trunk/heritrix2/engine/src/main/java/org/archive/extractor/RegexpHTMLLinkExtractor.java
> > (Heritrix's JavaDoc shows they have given some good thought in handling
> > different tags and attributes)
> >
> > What do you think if I add a wiki page that list out some scenarios of
> > outlink handling (i.e. the requirement)? Or does anyone know if any of
> the
> > many Java crawler projects have documentation at this area?
>
> If you do not look into jscript/ajax link extraction then there is no
> secret to it. Either go with xpath expression or e.g. for plain text
> with regexp. Please fell free to open a wiki page around the issue.
>
> salu2
>
> >
> > regards,
> > mingfai
> --
> Thorsten Scherler <thorsten.at.apache.org>
> Open Source Java <consulting, training and solutions>
>
> Sociedad Andaluza para el Desarrollo de la Sociedad
> de la Información, S.A.U. (SADESI)
>
>
>
>
>

Re: HTML outllink extraction

Posted by Thorsten Scherler <th...@juntadeandalucia.es>.
On Thu, 2009-04-02 at 18:53 +0800, Mingfai wrote:
> hi,
> 
> The default LinkExtractor seems to be quite simple. (too simple) It mainly
> uses URI.resolve and only cater the # and javascript scenarios. (from
> LinkExtractor.java getURI) Simple usage link resolving a <a
> href="test.html"> for new URI("http://www.google.com") will be wrong as it
> will return a http://www.google.comtest.html".

Well the link extraction always worked well. The case you just pointed
out looks like a bug BUT if you mean new URL
("http://testServer.com","test.html)) then have a look at
http://java.sun.com/j2se/1.4.2/docs/api/java/net/URL.html#URL(java.net.URL, java.lang.String)

> And there are many case that
> the URI.resolve doesn't cater. It seems to me we need to do some works at
> this area to make Droids more usable. Does anyone have any experience in out
> link extraction?

Enhancements are always welcome however the link extraction should work
fine. At least when I last looked at it was fine. The limitation ATM is
the extraction of jscript generated links.

> 
> I'm trying to see how other frameworks handle out link extraction and looked
> at:
> http://svn.apache.org/viewvc/lucene/nutch/trunk/src/java/org/apache/nutch/parse/OutlinkExtractor.java?view=log

Funny enough that have been the base of droids outlink extraction in the
first version I hacked.

> https://archive-crawler.svn.sourceforge.net/svnroot/archive-crawler/trunk/heritrix2/engine/src/main/java/org/archive/extractor/RegexpHTMLLinkExtractor.java
> (Heritrix's JavaDoc shows they have given some good thought in handling
> different tags and attributes)
> 
> What do you think if I add a wiki page that list out some scenarios of
> outlink handling (i.e. the requirement)? Or does anyone know if any of the
> many Java crawler projects have documentation at this area?

If you do not look into jscript/ajax link extraction then there is no
secret to it. Either go with xpath expression or e.g. for plain text
with regexp. Please fell free to open a wiki page around the issue.

salu2

> 
> regards,
> mingfai
-- 
Thorsten Scherler <thorsten.at.apache.org>
Open Source Java <consulting, training and solutions>

Sociedad Andaluza para el Desarrollo de la Sociedad 
de la Información, S.A.U. (SADESI)