You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by "McGibbney, Lewis John" <Le...@gcu.ac.uk> on 2011/02/20 20:17:13 UTC
Problems crawling specific site
Hello list,
Whilst using Nutch-1.2 on ubuntu 10.04 and undertaking a crawl either using crawl command or separate commands I can never seem to crawl the following site
www.scotland.gov.uk
My logs files mention that it is fetching the site but then never seems to develop anything further. I have URL in my urls seed file and in crawl-urlfilter.txt as an accepted host. Further to this, what I have noticed is that the URL above is always redirected to www.scotland.gov.uk/Home within my browser. I have tried including this also, but to date has proved less than fruitful.
Can someone do me a favour and see if they can get any results as I am stumped here.
Thank you Lewis
Glasgow Caledonian University is a registered Scottish charity, number SC021474
Winner: Times Higher Education’s Widening Participation Initiative of the Year 2009 and Herald Society’s Education Initiative of the Year 2009.
http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,6219,en.html
Winner: Times Higher Education’s Outstanding Support for Early Career Researchers of the Year 2010, GCU as a lead with Universities Scotland partners.
http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,15691,en.html
Re: Problems crawling specific site
Posted by Estrada Groups <es...@gmail.com>.
This may seem silly buy have you tried wget on the site to see what is getting blocked? It should also honor robots.txt.
Adam
Sent from my iPhone
On Feb 21, 2011, at 8:51 AM, "McGibbney, Lewis John" <Le...@gcu.ac.uk> wrote:
> I'm using Nutch-1.2. Took your advice and extracted clean distribution from tar.gz, only configuration changes I made was *MYSITE.COM/ to *scotland.gov.uk/ as well as add seed url to a crawl directory.
>
> Output from my inject command is as follows
>
> 2011-02-21 13:32:18,531 INFO crawl.Injector - Injector: starting at 2011-02-21 13:32:18
> 2011-02-21 13:32:18,532 INFO crawl.Injector - Injector: crawlDb: crawl/crawldb
> 2011-02-21 13:32:18,532 INFO crawl.Injector - Injector: urlDir: urls
> 2011-02-21 13:32:18,533 INFO crawl.Injector - Injector: Converting injected urls to crawl db entries.
>
> however the proceeding generate command results in the following
>
> 2011-02-21 13:33:43,710 INFO crawl.AbstractFetchSchedule - defaultInterval=2592000
> 2011-02-21 13:33:43,710 INFO crawl.AbstractFetchSchedule - maxInterval=7776000
> 2011-02-21 13:33:43,878 WARN crawl.Generator - Generator: 0 records selected for fetching, exiting ...
>
> I have tried chopping and changing the seed url as well as the domain name and the generate results are identical. To test that my commands are acurate, I changed my seed URL to another domain along with the crawl-urlfilter.txt domain. This resulted in a normal inject and generate stage.
>
> From my thinking this can only mean that this is a problem specific to this particular domain. Did nyone pick up anything fishy about the robots.txt file I posted?
>
> Lewis
>
> -----------------------------------------------------------------------------------------------------------------------------------------
>
>
> What version of nutch are you using? And in my experience, it's always better to try and add a single site to start your testing than all of them at once. I had to rebuild my machine yesterday otherwise I would have tried it too. try with a fresh install of nutch and only change MYSITE.COM to Scotland.gov.uk or whatever the site you want to crawl is in the crawl-urlfilter.txt file. By chance, are you able to get segments or does it just create a single one then die? If it's working correctly you will see it pulling pages through the terminal.
>
>
> Adam
>
>
>
> Glasgow Caledonian University is a registered Scottish charity, number SC021474
>
> Winner: Times Higher Education’s Widening Participation Initiative of the Year 2009 and Herald Society’s Education Initiative of the Year 2009.
> http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,6219,en.html
>
> Winner: Times Higher Education’s Outstanding Support for Early Career Researchers of the Year 2010, GCU as a lead with Universities Scotland partners.
> http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,15691,en.html
RE: Problems crawling specific site
Posted by "McGibbney, Lewis John" <Le...@gcu.ac.uk>.
I'm using Nutch-1.2. Took your advice and extracted clean distribution from tar.gz, only configuration changes I made was *MYSITE.COM/ to *scotland.gov.uk/ as well as add seed url to a crawl directory.
Output from my inject command is as follows
2011-02-21 13:32:18,531 INFO crawl.Injector - Injector: starting at 2011-02-21 13:32:18
2011-02-21 13:32:18,532 INFO crawl.Injector - Injector: crawlDb: crawl/crawldb
2011-02-21 13:32:18,532 INFO crawl.Injector - Injector: urlDir: urls
2011-02-21 13:32:18,533 INFO crawl.Injector - Injector: Converting injected urls to crawl db entries.
however the proceeding generate command results in the following
2011-02-21 13:33:43,710 INFO crawl.AbstractFetchSchedule - defaultInterval=2592000
2011-02-21 13:33:43,710 INFO crawl.AbstractFetchSchedule - maxInterval=7776000
2011-02-21 13:33:43,878 WARN crawl.Generator - Generator: 0 records selected for fetching, exiting ...
I have tried chopping and changing the seed url as well as the domain name and the generate results are identical. To test that my commands are acurate, I changed my seed URL to another domain along with the crawl-urlfilter.txt domain. This resulted in a normal inject and generate stage.
From my thinking this can only mean that this is a problem specific to this particular domain. Did nyone pick up anything fishy about the robots.txt file I posted?
Lewis
-----------------------------------------------------------------------------------------------------------------------------------------
What version of nutch are you using? And in my experience, it's always better to try and add a single site to start your testing than all of them at once. I had to rebuild my machine yesterday otherwise I would have tried it too. try with a fresh install of nutch and only change MYSITE.COM to Scotland.gov.uk or whatever the site you want to crawl is in the crawl-urlfilter.txt file. By chance, are you able to get segments or does it just create a single one then die? If it's working correctly you will see it pulling pages through the terminal.
Adam
Glasgow Caledonian University is a registered Scottish charity, number SC021474
Winner: Times Higher Education’s Widening Participation Initiative of the Year 2009 and Herald Society’s Education Initiative of the Year 2009.
http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,6219,en.html
Winner: Times Higher Education’s Outstanding Support for Early Career Researchers of the Year 2010, GCU as a lead with Universities Scotland partners.
http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,15691,en.html
Re: Problems crawling specific site
Posted by al...@aim.com.
I was trying to index a domain with redirect and noticed that only at depth 4 nutch started to fetch links from the domain's front page. Maybe you needed to increase depth.
-----Original Message-----
From: Estrada Groups <es...@gmail.com>
To: user <us...@nutch.apache.org>
Sent: Mon, Feb 21, 2011 7:07 am
Subject: Re: Problems crawling specific site
What version of nutch are you using? And in my experience, it's always better to
try and add a single site to start your testing than all of them at once. I had
to rebuild my machine yesterday otherwise I would have tried it too. try with a
fresh install of nutch and only change MYSITE.COM to Scotland.gov.uk or whatever
the site you want to crawl is in the crawl-urlfilter.txt file. By chance, are
you able to get segments or does it just create a single one then die? If it's
working correctly you will see it pulling pages through the terminal.
Adam
On Feb 21, 2011, at 6:08 AM, "McGibbney, Lewis John" <Le...@gcu.ac.uk>
wrote:
> Hi Adam,
>
> Crawl line is as follows
>
> bin/nutch crawl urls -depth 10 -topN 50 -solr http://localhost:8080/mywebapp
>
> If I am executing individual stages of the crawl process manually, my commands
are consistent with Nutch Tutorial.
>
> crawl-urlfilter is as follows
> # skip file:, ftp:, & mailto: urls
> -^(file|ftp|mailto):
>
> # skip image and other suffixes we can't yet parse
> -\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|sit|eps|wmf|zip|ppt|mpg|xls|gz|rpm|tgz|mov|MOV|exe|jpeg|JPEG|bmp|BMP)$
>
> # skip URLs containing certain characters as probable queries, etc.
> -[?*!@=]
>
> # skip URLs with slash-delimited segment that repeats 3+ times, to break loops
> -.*(/[^/]+)/[^/]+\1/[^/]+\1/
>
> # accept hosts in the following domains
> +^http://([a-z0-9]*\.)*south-ayrshire.gov.uk/
> +^http://([a-z0-9]*\.)*sabsm.co.uk/
> +^http://([a-z0-9]*\.)*scotland.gov.uk/Home <<< note the same happens if I
remove the /Home
>
> # skip everything else
> -.
>
> Further to this, robots.txt for the specific site reads as follows
>
> # /robots.txt file
> User-agent: *
> Disallow: /_private
> Disallow: /_test
> Disallow: /_dsforums
> Disallow: /_gsi
> Disallow: /_includes
> Disallow: /_temp
> Disallow: /what
> Disallow: /_interviews
> Disallow: /deleted
> Disallow: /News/Releases/1997
> Disallow: /News/Releases/1998
> Disallow: /News/Releases/1998
> Disallow: /News/Releases/2000
> Disallow: /News/Releases/2001
> Disallow: /News/Releases/2002
> Disallow: /News/Releases/2003
> Disallow: /News/Releases/2004
> Disallow: /News/Releases/2005
> Disallow: /News/Releases/2006
>
> I have run a fresh crawl this morning, and again this site is not crawled any
further than the first fetch stage, so at least I know that Nutch is
acknowledging its presence. It just doesn't seem to fetch any pages from the
domain.
>
> Lewis
>
> ________________________________________
> From: estrada.adam@gmail.com [estrada.adam@gmail.com] On Behalf Of Adam
Estrada [estrada.adam.groups@gmail.com]
> Sent: 21 February 2011 00:53
> To: user@nutch.apache.org
> Subject: Re: Problems crawling specific site
>
> Can you post the full command line you ran and what you have entered
> in the crawl-urlfilter.txt file?
>
> Thanks,
> Adam
>
> On Sun, Feb 20, 2011 at 2:17 PM, McGibbney, Lewis John
> <Le...@gcu.ac.uk> wrote:
>> Hello list,
>>
>> Whilst using Nutch-1.2 on ubuntu 10.04 and undertaking a crawl either using
crawl command or separate commands I can never seem to crawl the following site
>>
>> www.scotland.gov.uk
>>
>> My logs files mention that it is fetching the site but then never seems to
develop anything further. I have URL in my urls seed file and in
crawl-urlfilter.txt as an accepted host. Further to this, what I have noticed is
that the URL above is always redirected to www.scotland.gov.uk/Home within my
browser. I have tried including this also, but to date has proved less than
fruitful.
>>
>> Can someone do me a favour and see if they can get any results as I am
stumped here.
>>
>> Thank you Lewis
>>
>> Glasgow Caledonian University is a registered Scottish charity, number
SC021474
>>
>> Winner: Times Higher Education’s Widening Participation Initiative of the
Year 2009 and Herald Society’s Education Initiative of the Year 2009.
>> http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,6219,en.html
>>
>> Winner: Times Higher Education’s Outstanding Support for Early Career
Researchers of the Year 2010, GCU as a lead with Universities Scotland partners.
>> http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,15691,en.html
>>
> Email has been scanned for viruses by Altman Technologies' email management
service - www.altman.co.uk/emailsystems
>
> Glasgow Caledonian University is a registered Scottish charity, number
SC021474
>
> Winner: Times Higher Education’s Widening Participation Initiative of the Year
2009 and Herald Society’s Education Initiative of the Year 2009.
> http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,6219,en.html
>
> Winner: Times Higher Education’s Outstanding Support for Early Career
Researchers of the Year 2010, GCU as a lead with Universities Scotland partners.
> http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,15691,en.html
Re: Problems crawling specific site
Posted by Estrada Groups <es...@gmail.com>.
What version of nutch are you using? And in my experience, it's always better to try and add a single site to start your testing than all of them at once. I had to rebuild my machine yesterday otherwise I would have tried it too. try with a fresh install of nutch and only change MYSITE.COM to Scotland.gov.uk or whatever the site you want to crawl is in the crawl-urlfilter.txt file. By chance, are you able to get segments or does it just create a single one then die? If it's working correctly you will see it pulling pages through the terminal.
Adam
On Feb 21, 2011, at 6:08 AM, "McGibbney, Lewis John" <Le...@gcu.ac.uk> wrote:
> Hi Adam,
>
> Crawl line is as follows
>
> bin/nutch crawl urls -depth 10 -topN 50 -solr http://localhost:8080/mywebapp
>
> If I am executing individual stages of the crawl process manually, my commands are consistent with Nutch Tutorial.
>
> crawl-urlfilter is as follows
> # skip file:, ftp:, & mailto: urls
> -^(file|ftp|mailto):
>
> # skip image and other suffixes we can't yet parse
> -\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|sit|eps|wmf|zip|ppt|mpg|xls|gz|rpm|tgz|mov|MOV|exe|jpeg|JPEG|bmp|BMP)$
>
> # skip URLs containing certain characters as probable queries, etc.
> -[?*!@=]
>
> # skip URLs with slash-delimited segment that repeats 3+ times, to break loops
> -.*(/[^/]+)/[^/]+\1/[^/]+\1/
>
> # accept hosts in the following domains
> +^http://([a-z0-9]*\.)*south-ayrshire.gov.uk/
> +^http://([a-z0-9]*\.)*sabsm.co.uk/
> +^http://([a-z0-9]*\.)*scotland.gov.uk/Home <<< note the same happens if I remove the /Home
>
> # skip everything else
> -.
>
> Further to this, robots.txt for the specific site reads as follows
>
> # /robots.txt file
> User-agent: *
> Disallow: /_private
> Disallow: /_test
> Disallow: /_dsforums
> Disallow: /_gsi
> Disallow: /_includes
> Disallow: /_temp
> Disallow: /what
> Disallow: /_interviews
> Disallow: /deleted
> Disallow: /News/Releases/1997
> Disallow: /News/Releases/1998
> Disallow: /News/Releases/1998
> Disallow: /News/Releases/2000
> Disallow: /News/Releases/2001
> Disallow: /News/Releases/2002
> Disallow: /News/Releases/2003
> Disallow: /News/Releases/2004
> Disallow: /News/Releases/2005
> Disallow: /News/Releases/2006
>
> I have run a fresh crawl this morning, and again this site is not crawled any further than the first fetch stage, so at least I know that Nutch is acknowledging its presence. It just doesn't seem to fetch any pages from the domain.
>
> Lewis
>
> ________________________________________
> From: estrada.adam@gmail.com [estrada.adam@gmail.com] On Behalf Of Adam Estrada [estrada.adam.groups@gmail.com]
> Sent: 21 February 2011 00:53
> To: user@nutch.apache.org
> Subject: Re: Problems crawling specific site
>
> Can you post the full command line you ran and what you have entered
> in the crawl-urlfilter.txt file?
>
> Thanks,
> Adam
>
> On Sun, Feb 20, 2011 at 2:17 PM, McGibbney, Lewis John
> <Le...@gcu.ac.uk> wrote:
>> Hello list,
>>
>> Whilst using Nutch-1.2 on ubuntu 10.04 and undertaking a crawl either using crawl command or separate commands I can never seem to crawl the following site
>>
>> www.scotland.gov.uk
>>
>> My logs files mention that it is fetching the site but then never seems to develop anything further. I have URL in my urls seed file and in crawl-urlfilter.txt as an accepted host. Further to this, what I have noticed is that the URL above is always redirected to www.scotland.gov.uk/Home within my browser. I have tried including this also, but to date has proved less than fruitful.
>>
>> Can someone do me a favour and see if they can get any results as I am stumped here.
>>
>> Thank you Lewis
>>
>> Glasgow Caledonian University is a registered Scottish charity, number SC021474
>>
>> Winner: Times Higher Education’s Widening Participation Initiative of the Year 2009 and Herald Society’s Education Initiative of the Year 2009.
>> http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,6219,en.html
>>
>> Winner: Times Higher Education’s Outstanding Support for Early Career Researchers of the Year 2010, GCU as a lead with Universities Scotland partners.
>> http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,15691,en.html
>>
> Email has been scanned for viruses by Altman Technologies' email management service - www.altman.co.uk/emailsystems
>
> Glasgow Caledonian University is a registered Scottish charity, number SC021474
>
> Winner: Times Higher Education’s Widening Participation Initiative of the Year 2009 and Herald Society’s Education Initiative of the Year 2009.
> http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,6219,en.html
>
> Winner: Times Higher Education’s Outstanding Support for Early Career Researchers of the Year 2010, GCU as a lead with Universities Scotland partners.
> http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,15691,en.html
RE: Problems crawling specific site
Posted by "McGibbney, Lewis John" <Le...@gcu.ac.uk>.
Hi Adam,
Crawl line is as follows
bin/nutch crawl urls -depth 10 -topN 50 -solr http://localhost:8080/mywebapp
If I am executing individual stages of the crawl process manually, my commands are consistent with Nutch Tutorial.
crawl-urlfilter is as follows
# skip file:, ftp:, & mailto: urls
-^(file|ftp|mailto):
# skip image and other suffixes we can't yet parse
-\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|sit|eps|wmf|zip|ppt|mpg|xls|gz|rpm|tgz|mov|MOV|exe|jpeg|JPEG|bmp|BMP)$
# skip URLs containing certain characters as probable queries, etc.
-[?*!@=]
# skip URLs with slash-delimited segment that repeats 3+ times, to break loops
-.*(/[^/]+)/[^/]+\1/[^/]+\1/
# accept hosts in the following domains
+^http://([a-z0-9]*\.)*south-ayrshire.gov.uk/
+^http://([a-z0-9]*\.)*sabsm.co.uk/
+^http://([a-z0-9]*\.)*scotland.gov.uk/Home <<< note the same happens if I remove the /Home
# skip everything else
-.
Further to this, robots.txt for the specific site reads as follows
# /robots.txt file
User-agent: *
Disallow: /_private
Disallow: /_test
Disallow: /_dsforums
Disallow: /_gsi
Disallow: /_includes
Disallow: /_temp
Disallow: /what
Disallow: /_interviews
Disallow: /deleted
Disallow: /News/Releases/1997
Disallow: /News/Releases/1998
Disallow: /News/Releases/1998
Disallow: /News/Releases/2000
Disallow: /News/Releases/2001
Disallow: /News/Releases/2002
Disallow: /News/Releases/2003
Disallow: /News/Releases/2004
Disallow: /News/Releases/2005
Disallow: /News/Releases/2006
I have run a fresh crawl this morning, and again this site is not crawled any further than the first fetch stage, so at least I know that Nutch is acknowledging its presence. It just doesn't seem to fetch any pages from the domain.
Lewis
________________________________________
From: estrada.adam@gmail.com [estrada.adam@gmail.com] On Behalf Of Adam Estrada [estrada.adam.groups@gmail.com]
Sent: 21 February 2011 00:53
To: user@nutch.apache.org
Subject: Re: Problems crawling specific site
Can you post the full command line you ran and what you have entered
in the crawl-urlfilter.txt file?
Thanks,
Adam
On Sun, Feb 20, 2011 at 2:17 PM, McGibbney, Lewis John
<Le...@gcu.ac.uk> wrote:
> Hello list,
>
> Whilst using Nutch-1.2 on ubuntu 10.04 and undertaking a crawl either using crawl command or separate commands I can never seem to crawl the following site
>
> www.scotland.gov.uk
>
> My logs files mention that it is fetching the site but then never seems to develop anything further. I have URL in my urls seed file and in crawl-urlfilter.txt as an accepted host. Further to this, what I have noticed is that the URL above is always redirected to www.scotland.gov.uk/Home within my browser. I have tried including this also, but to date has proved less than fruitful.
>
> Can someone do me a favour and see if they can get any results as I am stumped here.
>
> Thank you Lewis
>
> Glasgow Caledonian University is a registered Scottish charity, number SC021474
>
> Winner: Times Higher Education’s Widening Participation Initiative of the Year 2009 and Herald Society’s Education Initiative of the Year 2009.
> http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,6219,en.html
>
> Winner: Times Higher Education’s Outstanding Support for Early Career Researchers of the Year 2010, GCU as a lead with Universities Scotland partners.
> http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,15691,en.html
>
Email has been scanned for viruses by Altman Technologies' email management service - www.altman.co.uk/emailsystems
Glasgow Caledonian University is a registered Scottish charity, number SC021474
Winner: Times Higher Education’s Widening Participation Initiative of the Year 2009 and Herald Society’s Education Initiative of the Year 2009.
http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,6219,en.html
Winner: Times Higher Education’s Outstanding Support for Early Career Researchers of the Year 2010, GCU as a lead with Universities Scotland partners.
http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,15691,en.html
RE: Problems crawling specific site
Posted by "McGibbney, Lewis John" <Le...@gcu.ac.uk>.
Hi Arjun,
Please see my other post this morning which answers this question.
Thanks Lewis
________________________________________
From: Arjun Kumar Reddy [charjunkumar.reddy@iiitb.net]
Sent: 21 February 2011 05:05
To: user@nutch.apache.org
Cc: Adam Estrada
Subject: Re: Problems crawling specific site
Have you specified the protocol in the urlfile?
It's like have you given
http://www.scotland.gov.uk<http://www.scotland.gov.uk/Home>/
or just www.scotland.gov.uk/ <http://www.scotland.gov.uk/Home>
Thanks and regards,*
*Arjun Kumar Reddy,
On Mon, Feb 21, 2011 at 6:23 AM, Adam Estrada <estrada.adam.groups@gmail.com
> wrote:
> Can you post the full command line you ran and what you have entered
> in the crawl-urlfilter.txt file?
>
> Thanks,
> Adam
>
> On Sun, Feb 20, 2011 at 2:17 PM, McGibbney, Lewis John
> <Le...@gcu.ac.uk> wrote:
> > Hello list,
> >
> > Whilst using Nutch-1.2 on ubuntu 10.04 and undertaking a crawl either
> using crawl command or separate commands I can never seem to crawl the
> following site
> >
> > www.scotland.gov.uk
> >
> > My logs files mention that it is fetching the site but then never seems
> to develop anything further. I have URL in my urls seed file and in
> crawl-urlfilter.txt as an accepted host. Further to this, what I have
> noticed is that the URL above is always redirected to
> www.scotland.gov.uk/Home within my browser. I have tried including this
> also, but to date has proved less than fruitful.
> >
> > Can someone do me a favour and see if they can get any results as I am
> stumped here.
> >
> > Thank you Lewis
> >
> > Glasgow Caledonian University is a registered Scottish charity, number
> SC021474
> >
> > Winner: Times Higher Education’s Widening Participation Initiative of the
> Year 2009 and Herald Society’s Education Initiative of the Year 2009.
> >
> http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,6219,en.html
> >
> > Winner: Times Higher Education’s Outstanding Support for Early Career
> Researchers of the Year 2010, GCU as a lead with Universities Scotland
> partners.
> >
> http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,15691,en.html
> >
>
Email has been scanned for viruses by Altman Technologies' email management service - www.altman.co.uk/emailsystems
Glasgow Caledonian University is a registered Scottish charity, number SC021474
Winner: Times Higher Education’s Widening Participation Initiative of the Year 2009 and Herald Society’s Education Initiative of the Year 2009.
http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,6219,en.html
Winner: Times Higher Education’s Outstanding Support for Early Career Researchers of the Year 2010, GCU as a lead with Universities Scotland partners.
http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,15691,en.html
Re: Problems crawling specific site
Posted by Arjun Kumar Reddy <ch...@iiitb.net>.
Have you specified the protocol in the urlfile?
It's like have you given
http://www.scotland.gov.uk<http://www.scotland.gov.uk/Home>/
or just www.scotland.gov.uk/ <http://www.scotland.gov.uk/Home>
Thanks and regards,*
*Arjun Kumar Reddy,
On Mon, Feb 21, 2011 at 6:23 AM, Adam Estrada <estrada.adam.groups@gmail.com
> wrote:
> Can you post the full command line you ran and what you have entered
> in the crawl-urlfilter.txt file?
>
> Thanks,
> Adam
>
> On Sun, Feb 20, 2011 at 2:17 PM, McGibbney, Lewis John
> <Le...@gcu.ac.uk> wrote:
> > Hello list,
> >
> > Whilst using Nutch-1.2 on ubuntu 10.04 and undertaking a crawl either
> using crawl command or separate commands I can never seem to crawl the
> following site
> >
> > www.scotland.gov.uk
> >
> > My logs files mention that it is fetching the site but then never seems
> to develop anything further. I have URL in my urls seed file and in
> crawl-urlfilter.txt as an accepted host. Further to this, what I have
> noticed is that the URL above is always redirected to
> www.scotland.gov.uk/Home within my browser. I have tried including this
> also, but to date has proved less than fruitful.
> >
> > Can someone do me a favour and see if they can get any results as I am
> stumped here.
> >
> > Thank you Lewis
> >
> > Glasgow Caledonian University is a registered Scottish charity, number
> SC021474
> >
> > Winner: Times Higher Education’s Widening Participation Initiative of the
> Year 2009 and Herald Society’s Education Initiative of the Year 2009.
> >
> http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,6219,en.html
> >
> > Winner: Times Higher Education’s Outstanding Support for Early Career
> Researchers of the Year 2010, GCU as a lead with Universities Scotland
> partners.
> >
> http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,15691,en.html
> >
>
Re: Problems crawling specific site
Posted by Adam Estrada <es...@gmail.com>.
Can you post the full command line you ran and what you have entered
in the crawl-urlfilter.txt file?
Thanks,
Adam
On Sun, Feb 20, 2011 at 2:17 PM, McGibbney, Lewis John
<Le...@gcu.ac.uk> wrote:
> Hello list,
>
> Whilst using Nutch-1.2 on ubuntu 10.04 and undertaking a crawl either using crawl command or separate commands I can never seem to crawl the following site
>
> www.scotland.gov.uk
>
> My logs files mention that it is fetching the site but then never seems to develop anything further. I have URL in my urls seed file and in crawl-urlfilter.txt as an accepted host. Further to this, what I have noticed is that the URL above is always redirected to www.scotland.gov.uk/Home within my browser. I have tried including this also, but to date has proved less than fruitful.
>
> Can someone do me a favour and see if they can get any results as I am stumped here.
>
> Thank you Lewis
>
> Glasgow Caledonian University is a registered Scottish charity, number SC021474
>
> Winner: Times Higher Education’s Widening Participation Initiative of the Year 2009 and Herald Society’s Education Initiative of the Year 2009.
> http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,6219,en.html
>
> Winner: Times Higher Education’s Outstanding Support for Early Career Researchers of the Year 2010, GCU as a lead with Universities Scotland partners.
> http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,15691,en.html
>