You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Marcus Herou <ma...@tailsweep.com> on 2008/06/17 19:39:51 UTC
Nutch + HBase
Hi.
Anyone tried to implement HBase as storage for:
* CrawlDB
* LinkDB
* Fetched and parsed url data
It would certainly be cool I think to be able to search in all these three
db's. Currently it is a little bit hard to use the data crawled without
actually indexing it.
Kindly
//Marcus
--
Marcus Herou CTO and co-founder Tailsweep AB
+46702561312
marcus.herou@tailsweep.com
http://www.tailsweep.com/
http://blogg.tailsweep.com/
Re: Nutch + HBase
Posted by Marcus Herou <ma...@tailsweep.com>.
Hi.
I have (of course) created my own crawling arch but really want to reuse
many of the good plugins which is written out there but I feel that the data
which Nutch fetches and parses is not in any way optimal for integrating
with other apps.
/M
On Thu, Jun 19, 2008 at 8:49 AM, Isabel Drost <ap...@isabel-drost.de>
wrote:
> On Wednesday 18 June 2008, Dennis Kubes wrote:
> > was to
> > may the Nutch architecture itself more flexible. This is what I have
> > been terming Nutch 2 and what I have been currently working on.
>
> Sounds interesting. Can you point me to any further information or
> discussion
> threads on Nutch 2?
>
> Isabel
>
> --
> While it may be true that a watched pot never boils, the one you don'tkeep
> an
> eye on can make an awful mess of your stove. -- Edward Stevenson
> |\ _,,,---,,_ Web: <http://www.isabel-drost.de>
> /,`.-'`' -. ;-;;,_
> |,4- ) )-,_..;\ ( `'-'
> '---''(_/--' `-'\_) (fL) IM: <xm...@spaceboyz.net>
>
--
Marcus Herou CTO and co-founder Tailsweep AB
+46702561312
marcus.herou@tailsweep.com
http://www.tailsweep.com/
http://blogg.tailsweep.com/
Re: Nutch + HBase
Posted by Isabel Drost <ap...@isabel-drost.de>.
On Wednesday 18 June 2008, Dennis Kubes wrote:
> was to
> may the Nutch architecture itself more flexible. This is what I have
> been terming Nutch 2 and what I have been currently working on.
Sounds interesting. Can you point me to any further information or discussion
threads on Nutch 2?
Isabel
--
While it may be true that a watched pot never boils, the one you don'tkeep an
eye on can make an awful mess of your stove. -- Edward Stevenson
|\ _,,,---,,_ Web: <http://www.isabel-drost.de>
/,`.-'`' -. ;-;;,_
|,4- ) )-,_..;\ ( `'-'
'---''(_/--' `-'\_) (fL) IM: <xm...@spaceboyz.net>
Re: Nutch + HBase
Posted by Dennis Kubes <ku...@apache.org>.
We have discussed it but not implemented it. A previous step before
implementing interfaces to use HBase for current Nutch databases was to
may the Nutch architecture itself more flexible. This is what I have
been terming Nutch 2 and what I have been currently working on.
Dennis
Marcus Herou wrote:
> Hi.
>
> Anyone tried to implement HBase as storage for:
>
> * CrawlDB
> * LinkDB
> * Fetched and parsed url data
>
> It would certainly be cool I think to be able to search in all these three
> db's. Currently it is a little bit hard to use the data crawled without
> actually indexing it.
>
> Kindly
>
> //Marcus
>
>
>