You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by "shubham.gupta" <sh...@orkash.com> on 2016/10/03 04:35:51 UTC
90% of URL rejected by filtering (Nutch 2.3.1)
Hey
When the inject job is run 90% of my seedurls get rejected. Therefore,
very few urls get crawled and does not give proper outputs.
my regex-urlfilter properties are as follows:
# skip file: ftp: and mailto: urls
-^(file|ftp|mailto):
# skip image and other suffixes we can't yet parse
# for a more extensive coverage use the urlfilter-suffix plugin
-\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|CSS|sit|SIT|eps|EPS|wmf|WMF|zip|ZIP|ppt|pdf|PPT|mpg|MPG|xls|XLS|gz|GZ|rpm|RPM|tgz|TGZ|mov|MOV|exe|EXE|jpeg|JPEG|bmp|BMP|js|JS)$
# skip URLs containing certain characters as probable queries, etc.
#-[?*!@=]
# skip URLs with slash-delimited segment that repeats 3+ times, to break
loops
-.*(/[^/]+)/[^/]+\1/[^/]+\1/
# accept anything else
-^(http://up.anv.bz)
+.
# skip URLs longer than 512 characters
-^.{513,}$
--
Shubham Gupta
Re: 90% of URL rejected by filtering (Nutch 2.3.1)
Posted by "shubham.gupta" <sh...@orkash.com>.
is there any way to find out the url filtered?
Also the line -^.{513,}$ was inserted as the update job was failing
consistently due to MongoDb exception : key too large to index.
Thanks and Regards,
Shubham Gupta
On Wednesday 05 October 2016 01:50 PM, Sachin Shaju wrote:
> For the time being you can comment out this line -^.{513,}$ and check.
>
> Regards,
> Sachin Shaju
>
> sachin.s@mstack.com
> +919539887554
>
> On Wed, Oct 5, 2016 at 11:41 AM, shubham.gupta <sh...@orkash.com>
> wrote:
>
>> my current regex-urlfilter properties are as follows:
>>
>> # skip file: ftp: and mailto: urls
>> #-^(file|ftp|mailto):
>>
>> # skip image and other suffixes we can't yet parse
>> # for a more extensive coverage use the urlfilter-suffix plugin
>> #-\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|CSS|sit|SIT|eps|EPS|
>> wmf|WMF|zip|ZIP|ppt|pdf|PPT|mpg|MPG|xls|XLS|gz|GZ|rpm|RPM|
>> tgz|TGZ|mov|MOV|exe|EXE|jpeg|JPEG|bmp|BMP|js|JS)$
>>
>> # skip URLs containing certain characters as probable queries, etc.
>> #-[?*!@=]
>>
>> # skip URLs with slash-delimited segment that repeats 3+ times, to break
>> loops
>> #-.*(/[^/]+)/[^/]+\1/[^/]+\1/
>>
>> # accept anything else
>> -^(http://up.anv.bz)
>> +.
>>
>> # skip URLs longer than 512 characters
>> -^.{513,}$
>>
>> Thanks and Regards,
>> Shubham Gupta
>>
>> On Wednesday 05 October 2016 11:29 AM, Sachin Shaju wrote:
>>
>>> my regex-urlfilter properties are as follows:
>>>>>>> # skip file: ftp: and mailto: urls
>>>>>>> -^(file|ftp|mailto):
>>>>>>>
>>>>>>> # skip image and other suffixes we can't yet parse
>>>>>>> # for a more extensive coverage use the urlfilter-suffix plugin
>>>>>>> -\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|CSS|sit|SIT|eps|EPS|
>>>>>>> wmf|WMF|zip|ZIP|ppt|pdf|PPT|mpg|MPG|xls|XLS|gz|GZ|rpm|RPM|
>>>>>>> tgz|TGZ|mov|MOV|exe|EXE|jpeg|JPEG|bmp|BMP|js|JS)$
>>>>>>>
>>>>>>> # skip URLs containing certain characters as probable queries, etc.
>>>>>>> #-[?*!@=]
>>>>>>>
>>>>>>> # skip URLs with slash-delimited segment that repeats 3+ times, to
>>> break
>>>>>>> loops
>>>>>>> -.*(/[^/]+)/[^/]+\1/[^/]+\1/
>>>>>>>
>>>>>>> # accept anything else
>>>>>>> -^(http://up.anv.bz)
>>>>>>> +.
>>>>>>>
>>>>>>> # skip URLs longer than 512 characters
>>>>>>> -^.{513,}$
>>
Re: 90% of URL rejected by filtering (Nutch 2.3.1)
Posted by Sachin Shaju <sa...@mstack.com>.
For the time being you can comment out this line -^.{513,}$ and check.
Regards,
Sachin Shaju
sachin.s@mstack.com
+919539887554
On Wed, Oct 5, 2016 at 11:41 AM, shubham.gupta <sh...@orkash.com>
wrote:
> my current regex-urlfilter properties are as follows:
>
> # skip file: ftp: and mailto: urls
> #-^(file|ftp|mailto):
>
> # skip image and other suffixes we can't yet parse
> # for a more extensive coverage use the urlfilter-suffix plugin
> #-\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|CSS|sit|SIT|eps|EPS|
> wmf|WMF|zip|ZIP|ppt|pdf|PPT|mpg|MPG|xls|XLS|gz|GZ|rpm|RPM|
> tgz|TGZ|mov|MOV|exe|EXE|jpeg|JPEG|bmp|BMP|js|JS)$
>
> # skip URLs containing certain characters as probable queries, etc.
> #-[?*!@=]
>
> # skip URLs with slash-delimited segment that repeats 3+ times, to break
> loops
> #-.*(/[^/]+)/[^/]+\1/[^/]+\1/
>
> # accept anything else
> -^(http://up.anv.bz)
> +.
>
> # skip URLs longer than 512 characters
> -^.{513,}$
>
> Thanks and Regards,
> Shubham Gupta
>
> On Wednesday 05 October 2016 11:29 AM, Sachin Shaju wrote:
>
>> my regex-urlfilter properties are as follows:
>> >>>>
>> >>>># skip file: ftp: and mailto: urls
>> >>>>-^(file|ftp|mailto):
>> >>>>
>> >>>># skip image and other suffixes we can't yet parse
>> >>>># for a more extensive coverage use the urlfilter-suffix plugin
>> >>>>-\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|CSS|sit|SIT|eps|EPS|
>> >>>>wmf|WMF|zip|ZIP|ppt|pdf|PPT|mpg|MPG|xls|XLS|gz|GZ|rpm|RPM|
>> >>>>tgz|TGZ|mov|MOV|exe|EXE|jpeg|JPEG|bmp|BMP|js|JS)$
>> >>>>
>> >>>># skip URLs containing certain characters as probable queries, etc.
>> >>>>#-[?*!@=]
>> >>>>
>> >>>># skip URLs with slash-delimited segment that repeats 3+ times, to
>> break
>> >>>>loops
>> >>>>-.*(/[^/]+)/[^/]+\1/[^/]+\1/
>> >>>>
>> >>>># accept anything else
>> >>>>-^(http://up.anv.bz)
>> >>>>+.
>> >>>>
>> >>>># skip URLs longer than 512 characters
>> >>>>-^.{513,}$
>>
>
>
--
The information contained in this electronic message and any attachments to
this message are intended for the exclusive use of the addressee(s) and may
contain proprietary, confidential or privileged information. If you are not
the intended recipient, you should not disseminate, distribute or copy this
e-mail. Please notify the sender immediately and destroy all copies of this
message and any attachments.
WARNING: Computer viruses can be transmitted via email. The recipient
should check this email and any attachments for the presence of viruses.
The company accepts no liability for any damage caused by any virus
transmitted by this email.
www.mStack.com
Re: 90% of URL rejected by filtering (Nutch 2.3.1)
Posted by "shubham.gupta" <sh...@orkash.com>.
my current regex-urlfilter properties are as follows:
# skip file: ftp: and mailto: urls
#-^(file|ftp|mailto):
# skip image and other suffixes we can't yet parse
# for a more extensive coverage use the urlfilter-suffix plugin
#-\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|CSS|sit|SIT|eps|EPS|
wmf|WMF|zip|ZIP|ppt|pdf|PPT|mpg|MPG|xls|XLS|gz|GZ|rpm|RPM|
tgz|TGZ|mov|MOV|exe|EXE|jpeg|JPEG|bmp|BMP|js|JS)$
# skip URLs containing certain characters as probable queries, etc.
#-[?*!@=]
# skip URLs with slash-delimited segment that repeats 3+ times, to break
loops
#-.*(/[^/]+)/[^/]+\1/[^/]+\1/
# accept anything else
-^(http://up.anv.bz)
+.
# skip URLs longer than 512 characters
-^.{513,}$
Thanks and Regards,
Shubham Gupta
On Wednesday 05 October 2016 11:29 AM, Sachin Shaju wrote:
> my regex-urlfilter properties are as follows:
> >>>>
> >>>># skip file: ftp: and mailto: urls
> >>>>-^(file|ftp|mailto):
> >>>>
> >>>># skip image and other suffixes we can't yet parse
> >>>># for a more extensive coverage use the urlfilter-suffix plugin
> >>>>-\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|CSS|sit|SIT|eps|EPS|
> >>>>wmf|WMF|zip|ZIP|ppt|pdf|PPT|mpg|MPG|xls|XLS|gz|GZ|rpm|RPM|
> >>>>tgz|TGZ|mov|MOV|exe|EXE|jpeg|JPEG|bmp|BMP|js|JS)$
> >>>>
> >>>># skip URLs containing certain characters as probable queries, etc.
> >>>>#-[?*!@=]
> >>>>
> >>>># skip URLs with slash-delimited segment that repeats 3+ times, to break
> >>>>loops
> >>>>-.*(/[^/]+)/[^/]+\1/[^/]+\1/
> >>>>
> >>>># accept anything else
> >>>>-^(http://up.anv.bz)
> >>>>+.
> >>>>
> >>>># skip URLs longer than 512 characters
> >>>>-^.{513,}$
Re: 90% of URL rejected by filtering (Nutch 2.3.1)
Posted by Sachin Shaju <sa...@mstack.com>.
Hi,
Can you share your current regex-urlfilter file ?
Regards,
Sachin Shaju
sachin.s@mstack.com
+919539887554
On Wed, Oct 5, 2016 at 11:19 AM, shubham.gupta <sh...@orkash.com>
wrote:
> The problem is not yet solved.
>
> Thanks and Regards
> Shubham Gupta
>
> On Monday 03 October 2016 11:12 AM, shubham.gupta wrote:
>
>> After doing this 3 less URLs have been rejected.
>>
>> Thanks and Regards,
>> Shubham Gupta
>>
>> On Monday 03 October 2016 10:28 AM, Sachin Shaju wrote:
>>
>>> You may check by commenting all regex filters in url-filter file and can
>>> try +. to see whether it gives the same output.
>>>
>>> Regards,
>>> Sachin Shaju
>>>
>>> sachin.s@mstack.com
>>>
>>> On Mon, Oct 3, 2016 at 10:05 AM, shubham.gupta <shubham.gupta@orkash.com
>>> >
>>> wrote:
>>>
>>> Hey
>>>>
>>>> When the inject job is run 90% of my seedurls get rejected. Therefore,
>>>> very few urls get crawled and does not give proper outputs.
>>>>
>>>> my regex-urlfilter properties are as follows:
>>>>
>>>> # skip file: ftp: and mailto: urls
>>>> -^(file|ftp|mailto):
>>>>
>>>> # skip image and other suffixes we can't yet parse
>>>> # for a more extensive coverage use the urlfilter-suffix plugin
>>>> -\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|CSS|sit|SIT|eps|EPS|
>>>> wmf|WMF|zip|ZIP|ppt|pdf|PPT|mpg|MPG|xls|XLS|gz|GZ|rpm|RPM|
>>>> tgz|TGZ|mov|MOV|exe|EXE|jpeg|JPEG|bmp|BMP|js|JS)$
>>>>
>>>> # skip URLs containing certain characters as probable queries, etc.
>>>> #-[?*!@=]
>>>>
>>>> # skip URLs with slash-delimited segment that repeats 3+ times, to break
>>>> loops
>>>> -.*(/[^/]+)/[^/]+\1/[^/]+\1/
>>>>
>>>> # accept anything else
>>>> -^(http://up.anv.bz)
>>>> +.
>>>>
>>>> # skip URLs longer than 512 characters
>>>> -^.{513,}$
>>>>
>>>> --
>>>>
>>>> Shubham Gupta
>>>>
>>>>
>>>>
>>
>
--
The information contained in this electronic message and any attachments to
this message are intended for the exclusive use of the addressee(s) and may
contain proprietary, confidential or privileged information. If you are not
the intended recipient, you should not disseminate, distribute or copy this
e-mail. Please notify the sender immediately and destroy all copies of this
message and any attachments.
WARNING: Computer viruses can be transmitted via email. The recipient
should check this email and any attachments for the presence of viruses.
The company accepts no liability for any damage caused by any virus
transmitted by this email.
www.mStack.com
Re: 90% of URL rejected by filtering (Nutch 2.3.1)
Posted by "shubham.gupta" <sh...@orkash.com>.
The problem is not yet solved.
Thanks and Regards
Shubham Gupta
On Monday 03 October 2016 11:12 AM, shubham.gupta wrote:
> After doing this 3 less URLs have been rejected.
>
> Thanks and Regards,
> Shubham Gupta
>
> On Monday 03 October 2016 10:28 AM, Sachin Shaju wrote:
>> You may check by commenting all regex filters in url-filter file and can
>> try +. to see whether it gives the same output.
>>
>> Regards,
>> Sachin Shaju
>>
>> sachin.s@mstack.com
>>
>> On Mon, Oct 3, 2016 at 10:05 AM, shubham.gupta
>> <sh...@orkash.com>
>> wrote:
>>
>>> Hey
>>>
>>> When the inject job is run 90% of my seedurls get rejected. Therefore,
>>> very few urls get crawled and does not give proper outputs.
>>>
>>> my regex-urlfilter properties are as follows:
>>>
>>> # skip file: ftp: and mailto: urls
>>> -^(file|ftp|mailto):
>>>
>>> # skip image and other suffixes we can't yet parse
>>> # for a more extensive coverage use the urlfilter-suffix plugin
>>> -\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|CSS|sit|SIT|eps|EPS|
>>> wmf|WMF|zip|ZIP|ppt|pdf|PPT|mpg|MPG|xls|XLS|gz|GZ|rpm|RPM|
>>> tgz|TGZ|mov|MOV|exe|EXE|jpeg|JPEG|bmp|BMP|js|JS)$
>>>
>>> # skip URLs containing certain characters as probable queries, etc.
>>> #-[?*!@=]
>>>
>>> # skip URLs with slash-delimited segment that repeats 3+ times, to
>>> break
>>> loops
>>> -.*(/[^/]+)/[^/]+\1/[^/]+\1/
>>>
>>> # accept anything else
>>> -^(http://up.anv.bz)
>>> +.
>>>
>>> # skip URLs longer than 512 characters
>>> -^.{513,}$
>>>
>>> --
>>>
>>> Shubham Gupta
>>>
>>>
>
Re: 90% of URL rejected by filtering (Nutch 2.3.1)
Posted by "shubham.gupta" <sh...@orkash.com>.
After doing this 3 less URLs have been rejected.
Thanks and Regards,
Shubham Gupta
On Monday 03 October 2016 10:28 AM, Sachin Shaju wrote:
> You may check by commenting all regex filters in url-filter file and can
> try +. to see whether it gives the same output.
>
> Regards,
> Sachin Shaju
>
> sachin.s@mstack.com
>
> On Mon, Oct 3, 2016 at 10:05 AM, shubham.gupta <sh...@orkash.com>
> wrote:
>
>> Hey
>>
>> When the inject job is run 90% of my seedurls get rejected. Therefore,
>> very few urls get crawled and does not give proper outputs.
>>
>> my regex-urlfilter properties are as follows:
>>
>> # skip file: ftp: and mailto: urls
>> -^(file|ftp|mailto):
>>
>> # skip image and other suffixes we can't yet parse
>> # for a more extensive coverage use the urlfilter-suffix plugin
>> -\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|CSS|sit|SIT|eps|EPS|
>> wmf|WMF|zip|ZIP|ppt|pdf|PPT|mpg|MPG|xls|XLS|gz|GZ|rpm|RPM|
>> tgz|TGZ|mov|MOV|exe|EXE|jpeg|JPEG|bmp|BMP|js|JS)$
>>
>> # skip URLs containing certain characters as probable queries, etc.
>> #-[?*!@=]
>>
>> # skip URLs with slash-delimited segment that repeats 3+ times, to break
>> loops
>> -.*(/[^/]+)/[^/]+\1/[^/]+\1/
>>
>> # accept anything else
>> -^(http://up.anv.bz)
>> +.
>>
>> # skip URLs longer than 512 characters
>> -^.{513,}$
>>
>> --
>>
>> Shubham Gupta
>>
>>
Re: 90% of URL rejected by filtering (Nutch 2.3.1)
Posted by Sachin Shaju <sa...@mstack.com>.
You may check by commenting all regex filters in url-filter file and can
try +. to see whether it gives the same output.
Regards,
Sachin Shaju
sachin.s@mstack.com
On Mon, Oct 3, 2016 at 10:05 AM, shubham.gupta <sh...@orkash.com>
wrote:
> Hey
>
> When the inject job is run 90% of my seedurls get rejected. Therefore,
> very few urls get crawled and does not give proper outputs.
>
> my regex-urlfilter properties are as follows:
>
> # skip file: ftp: and mailto: urls
> -^(file|ftp|mailto):
>
> # skip image and other suffixes we can't yet parse
> # for a more extensive coverage use the urlfilter-suffix plugin
> -\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|CSS|sit|SIT|eps|EPS|
> wmf|WMF|zip|ZIP|ppt|pdf|PPT|mpg|MPG|xls|XLS|gz|GZ|rpm|RPM|
> tgz|TGZ|mov|MOV|exe|EXE|jpeg|JPEG|bmp|BMP|js|JS)$
>
> # skip URLs containing certain characters as probable queries, etc.
> #-[?*!@=]
>
> # skip URLs with slash-delimited segment that repeats 3+ times, to break
> loops
> -.*(/[^/]+)/[^/]+\1/[^/]+\1/
>
> # accept anything else
> -^(http://up.anv.bz)
> +.
>
> # skip URLs longer than 512 characters
> -^.{513,}$
>
> --
>
> Shubham Gupta
>
>
--
The information contained in this electronic message and any attachments to
this message are intended for the exclusive use of the addressee(s) and may
contain proprietary, confidential or privileged information. If you are not
the intended recipient, you should not disseminate, distribute or copy this
e-mail. Please notify the sender immediately and destroy all copies of this
message and any attachments.
WARNING: Computer viruses can be transmitted via email. The recipient
should check this email and any attachments for the presence of viruses.
The company accepts no liability for any damage caused by any virus
transmitted by this email.
www.mStack.com