NullPointer exception in HTMLParser

[Piotr Kosiorowski <pk...@gmail.com>]

Hello,
I started a bigger fetch using latest HTMLParser code and got some 
NullPointer exceptions from HTMLParser class (line 201)

	if (metaTags.getRefresh()) {
	      status.setMinorCode(ParseStatus.SUCCESS_REDIRECT);
201:           status.setMessage(metaTags.getRefreshHref().toString());
          }

Example of problematic URL: 
http://calgary.foundlocally.com/Travel/Attr-CityWalks.htm
The problem is with meta tags:
Meta tags for http://calgary.foundlocally.com/Travel/Attr-CityWalks.htm: 
base=null, noCache=false, noFollow=false, noIndex=false, refresh=true, 
refreshHref=null

So it simply has empty refreshHref so some check before invoking 
toString() is needed. I am not sure what should be done here right now 
- but probably this case should not be handled as redirect.
Regards
Piotr

Re: NullPointer exception in HTMLParser

[Jérôme Charron <je...@gmail.com>]

> Whether the rest of the system uses this information, I'd say it's up to
> the other parts of the system an administrative policies. In my opinion,
> the values should be reported as such.

In the patches related to adapative fetch (NUTCH-61) I use this model
> when calculating next re-fetch time - there is a default FetchSchedule,
> which simply increments the time by 30 days (so the end result is as
> today), and there is an adaptive schedule which adjusts refetch interval
> and time based on whether the content has changed or not. However, in
> each case the lastModified time is noted and reported "as is" - so that
> other, more specialized parts, of the system can make more informed
> decisions.


That's ok for me Andrzej. Tks for the explanation.
 
Best regards

Jerome

-- 
http://motrech.free.fr/
http://frutch.free.fr/

Re: NullPointer exception in HTMLParser

[Andrzej Bialecki <ab...@getopt.org>]

Jérôme Charron wrote:
> Piotr,
> looking at the code, and at the meta refresh tag, it seems there's a bug in 
> the HTMLMetaProcessor.

Yes, it's a bug.

> But, does this code make sense for a search engine (refreshing the same Url 
> is usefull for a browser, but not really for a fetcher).

Now, that's a deeply philosophical question... ;-) But seriously, I can 
easily imagine a scenario when a refresh of 30 seconds, with the same 
url, brings each time a new useful content.

Whether the rest of the system uses this information, I'd say it's up to 
the other parts of the system an administrative policies. In my opinion, 
the values should be reported as such.

In the patches related to adapative fetch (NUTCH-61) I use this model 
when calculating next re-fetch time - there is a default FetchSchedule, 
which simply increments the time by 30 days (so the end result is as 
today), and there is an adaptive schedule which adjusts refetch interval 
and time based on whether the content has changed or not. However, in 
each case the lastModified time is noted and reported "as is" - so that 
other, more specialized parts, of the system can make more informed 
decisions.

-- 
Best regards,
Andrzej Bialecki     <><
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com

Re: NullPointer exception in HTMLParser

[Jérôme Charron <je...@gmail.com>]

> So it simply has empty refreshHref so some check before invoking
> toString() is needed. I am not sure what should be done here right now
> - but probably this case should not be handled as redirect.

Piotr,
looking at the code, and at the meta refresh tag, it seems there's a bug in 
the HTMLMetaProcessor.
At line 115, is checked that there's only the refresh time specified (no 
url), so that the time is setted to the content.
But in such a case (no url specified), the refreshURL should be the current 
page (not setted in our case). The code should become:

if (idx == -1) { // just the refresh time
time = content;
refreshUrl = currURL;
} else time = content.substring(0, idx);

But, does this code make sense for a search engine (refreshing the same Url 
is usefull for a browser, but not really for a fetcher).
So, perhaps, the best solution is to setRefresh(false) in such a case. The 
code will become someting like:

String time = null;
if (idx != -1) { // check not only the refresh time
time = content.substring(0, idx);
try {
metaTags.setRefreshTime(Integer.parseInt(time));
// Try now the retrieve the refresh URL (cut/paste the original code at 
line125)
metaTags.setRefresh(true);
} catch (Exception e) {
;
}

Sorry, no time today to provide a full patch .... just some ideas.

Regards

Jerome

-- 
http://motrech.free.fr/
http://frutch.free.fr/