Should URL normalization iterate?

[Doug Cook <na...@candiru.com>]

Hi,

I've run across a few patterns in URLs where applying a normalization puts
the URL in a form matching another normalization pattern (or even the same
one). But that pattern won't get executed because the patterns are applied
only once.

Should normalization iterate until no patterns match (with, perhaps, some
limit to the number of iterations to prevent loops from pattern mistakes)?

It's a minor problem; it doesn't seem to affect too many URLs for things
like session ID removal, since finding two session IDs in the same URL is
rare (but does happen -- that's how I noticed this). I could imagine it
being much more significant, however, if other Nutch users out there are
using "broader" normalization patterns.

Any philosophical/practical objections? (it's early, I've only had 1 coffee,
and I've probably missed something obvious!) 

I'll file an issue and add it to my queue of things to do if people think
its a good idea.

-Doug
-- 
View this message in context: http://www.nabble.com/Should-URL-normalization-iterate--tf2190244.html#a6059957
Sent from the Nutch - Dev forum at Nabble.com.

Re: Should URL normalization iterate?

[Doug Cook <na...@candiru.com>]

Just fixed this (and made some other improvements as well). See:

http://issues.apache.org/jira/browse/NUTCH-410

Hope this is useful; feedback welcome.

-Doug


Neal Richter-3 wrote:
> 
> Doug,
> 
> I think it sounds like a good idea.  It eliminates the need to order the
> rules precisely...
> 
> We don't iterate them in HtDig and it's been on my todo list for a while
> as
> well.
> 
> I would iterate until no matches, some max iteration number, or the URL is
> obviously junk.
> 
> For the max iteration number I would use the number of rewrite rules you
> have.  So if you have 10 rules, you iterate on all 10 rules 10 times. 
> That
> will cover the case where your rules 'chain' in a 10 step sequence.  Sure
> it's an edge case to do that, but I can see rule sets where you construct
> 3-step chains (like swapping strings or something).
> 
> Thanks
> 
> Neal
> 
> On 8/30/06, Doug Cook <na...@candiru.com> wrote:
>>
>>
>> Hi,
>>
>> I've run across a few patterns in URLs where applying a normalization
>> puts
>> the URL in a form matching another normalization pattern (or even the
>> same
>> one). But that pattern won't get executed because the patterns are
>> applied
>> only once.
>>
>> Should normalization iterate until no patterns match (with, perhaps, some
>> limit to the number of iterations to prevent loops from pattern
>> mistakes)?
>>
>> It's a minor problem; it doesn't seem to affect too many URLs for things
>> like session ID removal, since finding two session IDs in the same URL is
>> rare (but does happen -- that's how I noticed this). I could imagine it
>> being much more significant, however, if other Nutch users out there are
>> using "broader" normalization patterns.
>>
>> Any philosophical/practical objections? (it's early, I've only had 1
>> coffee,
>> and I've probably missed something obvious!)
>>
>> I'll file an issue and add it to my queue of things to do if people think
>> its a good idea.
>>
>> -Doug
>> --
>> View this message in context:
>> http://www.nabble.com/Should-URL-normalization-iterate--tf2190244.html#a6059957
>> Sent from the Nutch - Dev forum at Nabble.com.
>>
>>
> 
> 

-- 
View this message in context: http://www.nabble.com/Should-URL-normalization-iterate--tf2190244.html#a7606490
Sent from the Nutch - Dev mailing list archive at Nabble.com.

Re: Should URL normalization iterate?

[Neal Richter <nr...@gmail.com>]

Doug,

I think it sounds like a good idea.  It eliminates the need to order the
rules precisely...

We don't iterate them in HtDig and it's been on my todo list for a while as
well.

I would iterate until no matches, some max iteration number, or the URL is
obviously junk.

For the max iteration number I would use the number of rewrite rules you
have.  So if you have 10 rules, you iterate on all 10 rules 10 times.  That
will cover the case where your rules 'chain' in a 10 step sequence.  Sure
it's an edge case to do that, but I can see rule sets where you construct
3-step chains (like swapping strings or something).

Thanks

Neal

On 8/30/06, Doug Cook <na...@candiru.com> wrote:
>
>
> Hi,
>
> I've run across a few patterns in URLs where applying a normalization puts
> the URL in a form matching another normalization pattern (or even the same
> one). But that pattern won't get executed because the patterns are applied
> only once.
>
> Should normalization iterate until no patterns match (with, perhaps, some
> limit to the number of iterations to prevent loops from pattern mistakes)?
>
> It's a minor problem; it doesn't seem to affect too many URLs for things
> like session ID removal, since finding two session IDs in the same URL is
> rare (but does happen -- that's how I noticed this). I could imagine it
> being much more significant, however, if other Nutch users out there are
> using "broader" normalization patterns.
>
> Any philosophical/practical objections? (it's early, I've only had 1
> coffee,
> and I've probably missed something obvious!)
>
> I'll file an issue and add it to my queue of things to do if people think
> its a good idea.
>
> -Doug
> --
> View this message in context:
> http://www.nabble.com/Should-URL-normalization-iterate--tf2190244.html#a6059957
> Sent from the Nutch - Dev forum at Nabble.com.
>
>