You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Bing Li <lb...@gmail.com> on 2010/12/15 05:25:47 UTC
Get Crawled Data in Java or C# Collections
Hi, all,
I am a new Nutch user. Before knowing Nutch, I designed a crawler myself.
However, the quality is not good. So I decide to try Nutch.
However, after reading some materials about Nutch, I notice that Nutch puts
all of crawled pages into persistent Lucene indexes. In my project, I hope I
could get crawled data in memory. So I can manipulate them in Java or C#
collections. I don't want to retrieve the indexes crawled by Nutch.
Could you give me a solution to that? Thanks so much!
Best regards,
Li Bing
Re: Get Crawled Data in Java or C# Collections
Posted by xiao yang <ya...@gmail.com>.
Hi, Bing,
Nutch puts all the crawled pages in HDFS or local FS, in "segments" directory.
It provide APIs to retrieve the page content. You can find them in the
Web app part of Nutch. The "cache" of search results is read through
those APIs.
To process the content while crawling, you can try to write a Nutch
Plug-in. You can find a tutorial on the official site of Nutch.
Or you can try Nutch 2.0. It's under developing. you can check out
from SVN. It puts crawled data in HBase or other Database Systems,
which is easier for you to manipulate.
Thanks!
Xiao
On Wed, Dec 15, 2010 at 12:25 PM, Bing Li <lb...@gmail.com> wrote:
> Hi, all,
>
> I am a new Nutch user. Before knowing Nutch, I designed a crawler myself.
> However, the quality is not good. So I decide to try Nutch.
>
> However, after reading some materials about Nutch, I notice that Nutch puts
> all of crawled pages into persistent Lucene indexes. In my project, I hope I
> could get crawled data in memory. So I can manipulate them in Java or C#
> collections. I don't want to retrieve the indexes crawled by Nutch.
>
> Could you give me a solution to that? Thanks so much!
>
> Best regards,
> Li Bing
>
Re: Get Crawled Data in Java or C# Collections
Posted by Anurag <an...@gmail.com>.
Thanks Li
On Thu, Dec 16, 2010 at 3:44 AM, Bing Li [via Lucene] <
ml-node+2094945-181876348-146354@n3.nabble.com<ml...@n3.nabble.com>
> wrote:
> Hi, Kumar,
>
> To design a crawler is not an easy job. It depends on your goals. The most
> complicated one is to crawl the entire Web.
>
> http://www.amazon.com/HTTP-Programming-Recipes-Java-Bots/dp/0977320669
>
> This book might give you a hand.
>
> Thanks,
> LB
>
> On Thu, Dec 16, 2010 at 12:28 AM, Anurag <[hidden email]<http://user/SendEmail.jtp?type=node&node=2094945&i=0>>
> wrote:
>
> >
> > Can you tell how u designed crawler ? Is it by by writing code like this
> > CrawlDb.java<
> > http://www.docjar.com/html/api/org/apache/nutch/crawl/CrawlDb.java.html>
>
> > ?
> >
> > Actually wring your own Crawler is important stuff, I want to know.
> >
> > Thanks
> >
> > On Wed, Dec 15, 2010 at 9:56 AM, Bing Li [via Lucene] <
> > [hidden email] <http://user/SendEmail.jtp?type=node&node=2094945&i=1><[hidden
> email] <http://user/SendEmail.jtp?type=node&node=2094945&i=2>>
> > <[hidden email] <http://user/SendEmail.jtp?type=node&node=2094945&i=3><[hidden
> email] <http://user/SendEmail.jtp?type=node&node=2094945&i=4>>
> > >
> > > wrote:
> >
> > > Hi, all,
> > >
> > > I am a new Nutch user. Before knowing Nutch, I designed a crawler
> myself.
> > > However, the quality is not good. So I decide to try Nutch.
> > >
> > > However, after reading some materials about Nutch, I notice that Nutch
> > puts
> > >
> > > all of crawled pages into persistent Lucene indexes. In my project, I
> > hope
> > > I
> > > could get crawled data in memory. So I can manipulate them in Java or
> C#
> > > collections. I don't want to retrieve the indexes crawled by Nutch.
> > >
> > > Could you give me a solution to that? Thanks so much!
> > >
> > > Best regards,
> > > Li Bing
> > >
> > >
> > > ------------------------------
> > > View message @
> > >
> >
> http://lucene.472066.n3.nabble.com/Get-Crawled-Data-in-Java-or-C-Collections-tp2089972p2089972.html<http://lucene.472066.n3.nabble.com/Get-Crawled-Data-in-Java-or-C-Collections-tp2089972p2089972.html?by-user=t>
> > > To start a new topic under Nutch - User, email
> > > [hidden email] <http://user/SendEmail.jtp?type=node&node=2094945&i=5><[hidden
> email] <http://user/SendEmail.jtp?type=node&node=2094945&i=6>>
> > <[hidden email] <http://user/SendEmail.jtp?type=node&node=2094945&i=7><[hidden
> email] <http://user/SendEmail.jtp?type=node&node=2094945&i=8>>
> > >
> > > To unsubscribe from Nutch - User, click here<
> >
> http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=603147&code=YW51cmFnLml0LmpvbGx5QGdtYWlsLmNvbXw2MDMxNDd8LTIwOTgzNDQxOTY=<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=603147&code=YW51cmFnLml0LmpvbGx5QGdtYWlsLmNvbXw2MDMxNDd8LTIwOTgzNDQxOTY=&by-user=t>
> > >.
> > >
> > >
> >
> >
> >
> > --
> > Kumar Anurag
> >
> >
> > -----
> > Kumar Anurag
> >
> > --
> > View this message in context:
> >
> http://lucene.472066.n3.nabble.com/Get-Crawled-Data-in-Java-or-C-Collections-tp2089972p2092990.html<http://lucene.472066.n3.nabble.com/Get-Crawled-Data-in-Java-or-C-Collections-tp2089972p2092990.html?by-user=t>
> > Sent from the Nutch - User mailing list archive at Nabble.com.
> >
>
>
> ------------------------------
> View message @
> http://lucene.472066.n3.nabble.com/Get-Crawled-Data-in-Java-or-C-Collections-tp2089972p2094945.html
>
> To start a new topic under Nutch - User, email
> ml-node+603147-156023097-146354@n3.nabble.com<ml...@n3.nabble.com>
> To unsubscribe from Nutch - User, click here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=603147&code=YW51cmFnLml0LmpvbGx5QGdtYWlsLmNvbXw2MDMxNDd8LTIwOTgzNDQxOTY=>.
>
>
--
Kumar Anurag
-----
Kumar Anurag
--
View this message in context: http://lucene.472066.n3.nabble.com/Get-Crawled-Data-in-Java-or-C-Collections-tp2089972p2097015.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: Get Crawled Data in Java or C# Collections
Posted by Bing Li <lb...@gmail.com>.
Hi, Kumar,
To design a crawler is not an easy job. It depends on your goals. The most
complicated one is to crawl the entire Web.
http://www.amazon.com/HTTP-Programming-Recipes-Java-Bots/dp/0977320669
This book might give you a hand.
Thanks,
LB
On Thu, Dec 16, 2010 at 12:28 AM, Anurag <an...@gmail.com> wrote:
>
> Can you tell how u designed crawler ? Is it by by writing code like this
> CrawlDb.java<
> http://www.docjar.com/html/api/org/apache/nutch/crawl/CrawlDb.java.html>
> ?
>
> Actually wring your own Crawler is important stuff, I want to know.
>
> Thanks
>
> On Wed, Dec 15, 2010 at 9:56 AM, Bing Li [via Lucene] <
> ml-node+2089972-1635484590-146354@n3.nabble.com<ml...@n3.nabble.com>
> <ml...@n3.nabble.com>
> >
> > wrote:
>
> > Hi, all,
> >
> > I am a new Nutch user. Before knowing Nutch, I designed a crawler myself.
> > However, the quality is not good. So I decide to try Nutch.
> >
> > However, after reading some materials about Nutch, I notice that Nutch
> puts
> >
> > all of crawled pages into persistent Lucene indexes. In my project, I
> hope
> > I
> > could get crawled data in memory. So I can manipulate them in Java or C#
> > collections. I don't want to retrieve the indexes crawled by Nutch.
> >
> > Could you give me a solution to that? Thanks so much!
> >
> > Best regards,
> > Li Bing
> >
> >
> > ------------------------------
> > View message @
> >
> http://lucene.472066.n3.nabble.com/Get-Crawled-Data-in-Java-or-C-Collections-tp2089972p2089972.html
> > To start a new topic under Nutch - User, email
> > ml-node+603147-156023097-146354@n3.nabble.com<ml...@n3.nabble.com>
> <ml...@n3.nabble.com>
> >
> > To unsubscribe from Nutch - User, click here<
> http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=603147&code=YW51cmFnLml0LmpvbGx5QGdtYWlsLmNvbXw2MDMxNDd8LTIwOTgzNDQxOTY=
> >.
> >
> >
>
>
>
> --
> Kumar Anurag
>
>
> -----
> Kumar Anurag
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Get-Crawled-Data-in-Java-or-C-Collections-tp2089972p2092990.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>
Re: Get Crawled Data in Java or C# Collections
Posted by Anurag <an...@gmail.com>.
Can you tell how u designed crawler ? Is it by by writing code like this
CrawlDb.java<http://www.docjar.com/html/api/org/apache/nutch/crawl/CrawlDb.java.html>
?
Actually wring your own Crawler is important stuff, I want to know.
Thanks
On Wed, Dec 15, 2010 at 9:56 AM, Bing Li [via Lucene] <
ml-node+2089972-1635484590-146354@n3.nabble.com<ml...@n3.nabble.com>
> wrote:
> Hi, all,
>
> I am a new Nutch user. Before knowing Nutch, I designed a crawler myself.
> However, the quality is not good. So I decide to try Nutch.
>
> However, after reading some materials about Nutch, I notice that Nutch puts
>
> all of crawled pages into persistent Lucene indexes. In my project, I hope
> I
> could get crawled data in memory. So I can manipulate them in Java or C#
> collections. I don't want to retrieve the indexes crawled by Nutch.
>
> Could you give me a solution to that? Thanks so much!
>
> Best regards,
> Li Bing
>
>
> ------------------------------
> View message @
> http://lucene.472066.n3.nabble.com/Get-Crawled-Data-in-Java-or-C-Collections-tp2089972p2089972.html
> To start a new topic under Nutch - User, email
> ml-node+603147-156023097-146354@n3.nabble.com<ml...@n3.nabble.com>
> To unsubscribe from Nutch - User, click here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=603147&code=YW51cmFnLml0LmpvbGx5QGdtYWlsLmNvbXw2MDMxNDd8LTIwOTgzNDQxOTY=>.
>
>
--
Kumar Anurag
-----
Kumar Anurag
--
View this message in context: http://lucene.472066.n3.nabble.com/Get-Crawled-Data-in-Java-or-C-Collections-tp2089972p2092990.html
Sent from the Nutch - User mailing list archive at Nabble.com.