You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Meenakshi Kanaujia <me...@gmail.com> on 2011/05/23 06:30:05 UTC

Fetch list of urls

Hi,

I have crawled site using Nutch.
Is this possible to fetch the list of urls depth wise from crawlDB.

Thanks,
Meenakshi

RE: Fetch list of urls

Posted by "McGibbney, Lewis John" <Le...@gcu.ac.uk>.
What you describe sounds like a complete update and revamp prior to transition to new CMS. Although what you have described might aid this transition by finding the depth of each web page in your site, I would comment that a more holistic approach needs to be considered possibly taking into account the overall structure of your site.

To answer your question, I would say the best way to do what you require is to crawl at a specific depth then dump the crawldb with readdb -dump. This way you would be able to see the current urls Nutch has fetched at your requested depth.

Lewis

________________________________________
From: Meenakshi Kanaujia [meenakshi.kanaujia@gmail.com]
Sent: 24 May 2011 04:17
To: user@nutch.apache.org
Subject: Re: Fetch list of urls

Hi Lewis,

Actually I am crawlling an intranet site.
We need to migrate intranet site from one CMS to another.
Crawling is done successfully.
Now I have to decide the migration order of the crawled web pages.
For doing that I was just thinking a way to find the list of url at each
depth.
Suppose at the time of crawling I have specified the depth 5, then I want to
fetch the list of url at depth 1,2,3.... and so on.
Can this be possible using Nutch api?

Thanks,
Meenakshi

On Mon, May 23, 2011 at 4:02 PM, McGibbney, Lewis John <
Lewis.McGibbney@gcu.ac.uk> wrote:

> Hi Meenakshi,
>
> Can you expand any on this? This is very vague.
>
> Lewis
>
> ________________________________________
> From: Meenakshi Kanaujia [meenakshi.kanaujia@gmail.com]
> Sent: 23 May 2011 05:30
> To: user@nutch.apache.org
> Subject: Fetch list of urls
>
> Hi,
>
> I have crawled site using Nutch.
> Is this possible to fetch the list of urls depth wise from crawlDB.
>
> Thanks,
> Meenakshi
>
> Email has been scanned for viruses by Altman Technologies' email management
> service - www.altman.co.uk/emailsystems
>
> Glasgow Caledonian University is a registered Scottish charity, number
> SC021474
>
> Winner: Times Higher Education’s Widening Participation Initiative of the
> Year 2009 and Herald Society’s Education Initiative of the Year 2009.
>
> http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,6219,en.html
>
> Winner: Times Higher Education’s Outstanding Support for Early Career
> Researchers of the Year 2010, GCU as a lead with Universities Scotland
> partners.
>
> http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,15691,en.html
>
Email has been scanned for viruses by Altman Technologies' email management service - www.altman.co.uk/emailsystems

Glasgow Caledonian University is a registered Scottish charity, number SC021474

Winner: Times Higher Education’s Widening Participation Initiative of the Year 2009 and Herald Society’s Education Initiative of the Year 2009.
http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,6219,en.html

Winner: Times Higher Education’s Outstanding Support for Early Career Researchers of the Year 2010, GCU as a lead with Universities Scotland partners.
http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,15691,en.html

Re: Fetch list of urls

Posted by Julien Nioche <li...@gmail.com>.
Hi Meenakshi,

This can be done by implementing a custom ScoringFilter which would
propagate the depth from one page to its outlinks via their metadata. The
info about the depth would then be in the crawldb and you could have a
custom mapreduce job to filter the URLs based on the value of the metadata.
Note that this requires recrawling

HTH

Julien

On 24 May 2011 04:17, Meenakshi Kanaujia <me...@gmail.com>wrote:

> Hi Lewis,
>
> Actually I am crawlling an intranet site.
> We need to migrate intranet site from one CMS to another.
> Crawling is done successfully.
> Now I have to decide the migration order of the crawled web pages.
> For doing that I was just thinking a way to find the list of url at each
> depth.
> Suppose at the time of crawling I have specified the depth 5, then I want
> to
> fetch the list of url at depth 1,2,3.... and so on.
> Can this be possible using Nutch api?
>
> Thanks,
> Meenakshi
>
> On Mon, May 23, 2011 at 4:02 PM, McGibbney, Lewis John <
> Lewis.McGibbney@gcu.ac.uk> wrote:
>
> > Hi Meenakshi,
> >
> > Can you expand any on this? This is very vague.
> >
> > Lewis
> >
> > ________________________________________
> > From: Meenakshi Kanaujia [meenakshi.kanaujia@gmail.com]
> > Sent: 23 May 2011 05:30
> > To: user@nutch.apache.org
> > Subject: Fetch list of urls
> >
> > Hi,
> >
> > I have crawled site using Nutch.
> > Is this possible to fetch the list of urls depth wise from crawlDB.
> >
> > Thanks,
> > Meenakshi
> >
> > Email has been scanned for viruses by Altman Technologies' email
> management
> > service - www.altman.co.uk/emailsystems
> >
> > Glasgow Caledonian University is a registered Scottish charity, number
> > SC021474
> >
> > Winner: Times Higher Education’s Widening Participation Initiative of the
> > Year 2009 and Herald Society’s Education Initiative of the Year 2009.
> >
> >
> http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,6219,en.html
> >
> > Winner: Times Higher Education’s Outstanding Support for Early Career
> > Researchers of the Year 2010, GCU as a lead with Universities Scotland
> > partners.
> >
> >
> http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,15691,en.html
> >
>



-- 
*
*Open Source Solutions for Text Engineering

http://digitalpebble.blogspot.com/
http://www.digitalpebble.com

Re: Fetch list of urls

Posted by Meenakshi Kanaujia <me...@gmail.com>.
Hi Lewis,

Actually I am crawlling an intranet site.
We need to migrate intranet site from one CMS to another.
Crawling is done successfully.
Now I have to decide the migration order of the crawled web pages.
For doing that I was just thinking a way to find the list of url at each
depth.
Suppose at the time of crawling I have specified the depth 5, then I want to
fetch the list of url at depth 1,2,3.... and so on.
Can this be possible using Nutch api?

Thanks,
Meenakshi

On Mon, May 23, 2011 at 4:02 PM, McGibbney, Lewis John <
Lewis.McGibbney@gcu.ac.uk> wrote:

> Hi Meenakshi,
>
> Can you expand any on this? This is very vague.
>
> Lewis
>
> ________________________________________
> From: Meenakshi Kanaujia [meenakshi.kanaujia@gmail.com]
> Sent: 23 May 2011 05:30
> To: user@nutch.apache.org
> Subject: Fetch list of urls
>
> Hi,
>
> I have crawled site using Nutch.
> Is this possible to fetch the list of urls depth wise from crawlDB.
>
> Thanks,
> Meenakshi
>
> Email has been scanned for viruses by Altman Technologies' email management
> service - www.altman.co.uk/emailsystems
>
> Glasgow Caledonian University is a registered Scottish charity, number
> SC021474
>
> Winner: Times Higher Education’s Widening Participation Initiative of the
> Year 2009 and Herald Society’s Education Initiative of the Year 2009.
>
> http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,6219,en.html
>
> Winner: Times Higher Education’s Outstanding Support for Early Career
> Researchers of the Year 2010, GCU as a lead with Universities Scotland
> partners.
>
> http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,15691,en.html
>

RE: Fetch list of urls

Posted by "McGibbney, Lewis John" <Le...@gcu.ac.uk>.
Hi Meenakshi,

Can you expand any on this? This is very vague.

Lewis

________________________________________
From: Meenakshi Kanaujia [meenakshi.kanaujia@gmail.com]
Sent: 23 May 2011 05:30
To: user@nutch.apache.org
Subject: Fetch list of urls

Hi,

I have crawled site using Nutch.
Is this possible to fetch the list of urls depth wise from crawlDB.

Thanks,
Meenakshi

Email has been scanned for viruses by Altman Technologies' email management service - www.altman.co.uk/emailsystems

Glasgow Caledonian University is a registered Scottish charity, number SC021474

Winner: Times Higher Education’s Widening Participation Initiative of the Year 2009 and Herald Society’s Education Initiative of the Year 2009.
http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,6219,en.html

Winner: Times Higher Education’s Outstanding Support for Early Career Researchers of the Year 2010, GCU as a lead with Universities Scotland partners.
http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,15691,en.html