You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by lewis john mcgibbney <le...@gmail.com> on 2011/07/06 00:44:06 UTC

Crawling relation database

Hi,

I'm curious to hear if anyone has information for configuring Nutch to crawl
a RDB such as MySQL. In my hypothetical example there are N number of
databases residing in various distributed geographical locations, to make a
worst case scenario, say that they are NOT all the same type, and I wish to
use Nutch trunk 2.0 to push the results to some other structured data store
which I can then connect to to serve search results.

Does anyone have any information such as an overview of database crawling
and serving using Nutch? I have been unsuccesful obtaining info on the Web
as query results are ambiguous and usually refer to crawldb or linkdb.

If I can get this it would be a real nice entry for inclusion in our wiki.

Thanks for any suggestions or info.

-- 
*Lewis*

Re: Crawling relation database

Posted by lewis john mcgibbney <le...@gmail.com>.
thanks to you both

On Tue, Jul 5, 2011 at 4:35 PM, Markus Jelsma <ma...@openindex.io>wrote:

> H,
>
> About geographical search: Solr will do this for you. Built-in for 3.x+ and
> using third-party plugins for 1.4.x. Both provide different features. In
> Solr
> it's you'd not base similarity on geographical data but use spatial data to
> boost textual similar documents instead, or filter.
>
> This keeps text similarity intact and offers spatial features on top.
>
> You'll get more feedback on the Solr list indeed :)
>
> Cheers
>
> > Thanks for this Markus, it had occured to me that DIH was a very
> plausable
> > solution to progress with. I think you have just confirmed due to the
> > flexibility it offers amongst other attributes.
> >
> > I'm looking at creating a context aware web application which would use
> > geographical search to obtain results based on location. This is required
> > as the data will contain (amongst others) fields with integer values
> which
> > vary dependent upon a building location cost index. Similarity is
> directly
> > linked through geographical location factor. I wanted to have the data
> > stored within the n number of distributed RDB's available in a cloud
> > environment which could be searched as oppose to the non-trivial task of
> > searching across a fragmented distrubuted number of DB's.
> >
> > As you mention, it does make more sense to save documents in a doc (or
> > column) oriented DB.
> >
> > Essentially, using the DIH tool would remove requirement for Nutch?
> >
> > I think to progress with this, I'm best moving the thread to Solr-user@if
> > I have further questions.
> >
> > Thank you
> >
> > On Tue, Jul 5, 2011 at 3:53 PM, Markus Jelsma
> <ma...@openindex.io>wrote:
> > > Hi Lewis,
> > >
> > > It sounds to me you'd be better of using Solr's very advanced
> > > DataImportHandler [1]. It can (delta) import data from your RDBMS' and
> > > offers
> > > much flexibility on how to transform entities.
> > >
> > > Besides crawling you also mentions you'd like to push results (of what)
> > > to another structured data store. But why would you want that?
> Handling,
> > > processing and serving search results is done by Solr (and ES in the
> > > future)
> > > and since our entities are flat (just a document) it makes more sense
> to
> > > me to
> > > save documents in a document (or column) oriented DB.
> > >
> > > [1] :http://wiki.apache.org/solr/DataImportHandler
> > >
> > > Cheers,
> > >
> > > > Hi,
> > > >
> > > > I'm curious to hear if anyone has information for configuring Nutch
> to
> > > > crawl a RDB such as MySQL. In my hypothetical example there are N
> > > > number of databases residing in various distributed geographical
> > > > locations, to make a worst case scenario, say that they are NOT all
> > > > the same type, and
> > >
> > > I
> > >
> > > > wish to use Nutch trunk 2.0 to push the results to some other
> > > > structured data store which I can then connect to to serve search
> > > > results.
> > > >
> > > > Does anyone have any information such as an overview of database
> > > > crawling and serving using Nutch? I have been unsuccesful obtaining
> > > > info on the
> > >
> > > Web
> > >
> > > > as query results are ambiguous and usually refer to crawldb or
> linkdb.
> > > >
> > > > If I can get this it would be a real nice entry for inclusion in our
> > >
> > > wiki.
> > >
> > > > Thanks for any suggestions or info.
>



-- 
*Lewis*

Re: Crawling relation database

Posted by Ken Krugler <kk...@transpac.com>.
Another approach for large DBs is to use Sqoop to pull the DB records into Hadoop, then create Solr indexes from those records.

In our situation we're using Cascading, so we just hook up our cascading.solr tap.

For small(er) DBs, or if you need incremental updates, then Solr's DIH is more appropriate.

-- Ken

On Jul 5, 2011, at 4:35pm, Markus Jelsma wrote:

> H,
> 
> About geographical search: Solr will do this for you. Built-in for 3.x+ and 
> using third-party plugins for 1.4.x. Both provide different features. In Solr 
> it's you'd not base similarity on geographical data but use spatial data to 
> boost textual similar documents instead, or filter.
> 
> This keeps text similarity intact and offers spatial features on top.
> 
> You'll get more feedback on the Solr list indeed :)
> 
> Cheers
> 
>> Thanks for this Markus, it had occured to me that DIH was a very plausable
>> solution to progress with. I think you have just confirmed due to the
>> flexibility it offers amongst other attributes.
>> 
>> I'm looking at creating a context aware web application which would use
>> geographical search to obtain results based on location. This is required
>> as the data will contain (amongst others) fields with integer values which
>> vary dependent upon a building location cost index. Similarity is directly
>> linked through geographical location factor. I wanted to have the data
>> stored within the n number of distributed RDB's available in a cloud
>> environment which could be searched as oppose to the non-trivial task of
>> searching across a fragmented distrubuted number of DB's.
>> 
>> As you mention, it does make more sense to save documents in a doc (or
>> column) oriented DB.
>> 
>> Essentially, using the DIH tool would remove requirement for Nutch?
>> 
>> I think to progress with this, I'm best moving the thread to Solr-user@ if
>> I have further questions.
>> 
>> Thank you
>> 
>> On Tue, Jul 5, 2011 at 3:53 PM, Markus Jelsma 
> <ma...@openindex.io>wrote:
>>> Hi Lewis,
>>> 
>>> It sounds to me you'd be better of using Solr's very advanced
>>> DataImportHandler [1]. It can (delta) import data from your RDBMS' and
>>> offers
>>> much flexibility on how to transform entities.
>>> 
>>> Besides crawling you also mentions you'd like to push results (of what)
>>> to another structured data store. But why would you want that? Handling,
>>> processing and serving search results is done by Solr (and ES in the
>>> future)
>>> and since our entities are flat (just a document) it makes more sense to
>>> me to
>>> save documents in a document (or column) oriented DB.
>>> 
>>> [1] :http://wiki.apache.org/solr/DataImportHandler
>>> 
>>> Cheers,
>>> 
>>>> Hi,
>>>> 
>>>> I'm curious to hear if anyone has information for configuring Nutch to
>>>> crawl a RDB such as MySQL. In my hypothetical example there are N
>>>> number of databases residing in various distributed geographical
>>>> locations, to make a worst case scenario, say that they are NOT all
>>>> the same type, and
>>> 
>>> I
>>> 
>>>> wish to use Nutch trunk 2.0 to push the results to some other
>>>> structured data store which I can then connect to to serve search
>>>> results.
>>>> 
>>>> Does anyone have any information such as an overview of database
>>>> crawling and serving using Nutch? I have been unsuccesful obtaining
>>>> info on the
>>> 
>>> Web
>>> 
>>>> as query results are ambiguous and usually refer to crawldb or linkdb.
>>>> 
>>>> If I can get this it would be a real nice entry for inclusion in our
>>> 
>>> wiki.
>>> 
>>>> Thanks for any suggestions or info.

--------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
custom data mining solutions







Re: Crawling relation database

Posted by Markus Jelsma <ma...@openindex.io>.
H,

About geographical search: Solr will do this for you. Built-in for 3.x+ and 
using third-party plugins for 1.4.x. Both provide different features. In Solr 
it's you'd not base similarity on geographical data but use spatial data to 
boost textual similar documents instead, or filter.

This keeps text similarity intact and offers spatial features on top.

You'll get more feedback on the Solr list indeed :)

Cheers

> Thanks for this Markus, it had occured to me that DIH was a very plausable
> solution to progress with. I think you have just confirmed due to the
> flexibility it offers amongst other attributes.
> 
> I'm looking at creating a context aware web application which would use
> geographical search to obtain results based on location. This is required
> as the data will contain (amongst others) fields with integer values which
> vary dependent upon a building location cost index. Similarity is directly
> linked through geographical location factor. I wanted to have the data
> stored within the n number of distributed RDB's available in a cloud
> environment which could be searched as oppose to the non-trivial task of
> searching across a fragmented distrubuted number of DB's.
> 
> As you mention, it does make more sense to save documents in a doc (or
> column) oriented DB.
> 
> Essentially, using the DIH tool would remove requirement for Nutch?
> 
> I think to progress with this, I'm best moving the thread to Solr-user@ if
> I have further questions.
> 
> Thank you
> 
> On Tue, Jul 5, 2011 at 3:53 PM, Markus Jelsma 
<ma...@openindex.io>wrote:
> > Hi Lewis,
> > 
> > It sounds to me you'd be better of using Solr's very advanced
> > DataImportHandler [1]. It can (delta) import data from your RDBMS' and
> > offers
> > much flexibility on how to transform entities.
> > 
> > Besides crawling you also mentions you'd like to push results (of what)
> > to another structured data store. But why would you want that? Handling,
> > processing and serving search results is done by Solr (and ES in the
> > future)
> > and since our entities are flat (just a document) it makes more sense to
> > me to
> > save documents in a document (or column) oriented DB.
> > 
> > [1] :http://wiki.apache.org/solr/DataImportHandler
> > 
> > Cheers,
> > 
> > > Hi,
> > > 
> > > I'm curious to hear if anyone has information for configuring Nutch to
> > > crawl a RDB such as MySQL. In my hypothetical example there are N
> > > number of databases residing in various distributed geographical
> > > locations, to make a worst case scenario, say that they are NOT all
> > > the same type, and
> > 
> > I
> > 
> > > wish to use Nutch trunk 2.0 to push the results to some other
> > > structured data store which I can then connect to to serve search
> > > results.
> > > 
> > > Does anyone have any information such as an overview of database
> > > crawling and serving using Nutch? I have been unsuccesful obtaining
> > > info on the
> > 
> > Web
> > 
> > > as query results are ambiguous and usually refer to crawldb or linkdb.
> > > 
> > > If I can get this it would be a real nice entry for inclusion in our
> > 
> > wiki.
> > 
> > > Thanks for any suggestions or info.

Re: Crawling relation database

Posted by Kirby Bohling <ki...@gmail.com>.
By the by, this is the best source of open geospatial data I've found
is OpenStreetMap [1].  I'm unaware of any Java bindings for it
(honestly I haven't looked that hard), but it appears to have pretty
excellent data, and the built in search seems better than an other
open source implementation I've seen.

I've got a version of JGeoCoder[2] that I've hacked up to make it all
Java and work with a modern version of Maven.  Hopefully that will
make it easier to use, it appears to have been left for dead by the
auther.  Not sure if that helps or not, but I spent a bit of time
tracking down open sources.  Internationally, I can't find any decent
detailed data.  In the US and Canada, there's plenty of open data, I'd
start with the Tiger Data and OpenStreetMap.

Once I get permission from my boss, I hope to open source my
modifications to JGeoCoder (I'd also like to move the backend search
to Lucene/Solr), make sure there is a Java API for the OpenStreetMap
data.

Not sure if this helps you at all with your project, but it was
information I wish I'd known when I started working on some geospatial
related search at work.

Kirby

[1] http://www.openstreetmap.org/
[2] http://jgeocoder.sourceforge.net/project-info.html




On Tue, Jul 5, 2011 at 6:19 PM, lewis john mcgibbney
<le...@gmail.com> wrote:
> Thanks for this Markus, it had occured to me that DIH was a very plausable
> solution to progress with. I think you have just confirmed due to the
> flexibility it offers amongst other attributes.
>
> I'm looking at creating a context aware web application which would use
> geographical search to obtain results based on location. This is required as
> the data will contain (amongst others) fields with integer values which vary
> dependent upon a building location cost index. Similarity is directly linked
> through geographical location factor. I wanted to have the data stored
> within the n number of distributed RDB's available in a cloud environment
> which could be searched as oppose to the non-trivial task of searching
> across a fragmented distrubuted number of DB's.
>
> As you mention, it does make more sense to save documents in a doc (or
> column) oriented DB.
>
> Essentially, using the DIH tool would remove requirement for Nutch?
>
> I think to progress with this, I'm best moving the thread to Solr-user@ if I
> have further questions.
>
> Thank you
> On Tue, Jul 5, 2011 at 3:53 PM, Markus Jelsma <ma...@openindex.io>wrote:
>
>> Hi Lewis,
>>
>> It sounds to me you'd be better of using Solr's very advanced
>> DataImportHandler [1]. It can (delta) import data from your RDBMS' and
>> offers
>> much flexibility on how to transform entities.
>>
>> Besides crawling you also mentions you'd like to push results (of what) to
>> another structured data store. But why would you want that? Handling,
>> processing and serving search results is done by Solr (and ES in the
>> future)
>> and since our entities are flat (just a document) it makes more sense to me
>> to
>> save documents in a document (or column) oriented DB.
>>
>> [1] :http://wiki.apache.org/solr/DataImportHandler
>>
>> Cheers,
>>
>> > Hi,
>> >
>> > I'm curious to hear if anyone has information for configuring Nutch to
>> > crawl a RDB such as MySQL. In my hypothetical example there are N number
>> > of databases residing in various distributed geographical locations, to
>> > make a worst case scenario, say that they are NOT all the same type, and
>> I
>> > wish to use Nutch trunk 2.0 to push the results to some other structured
>> > data store which I can then connect to to serve search results.
>> >
>> > Does anyone have any information such as an overview of database crawling
>> > and serving using Nutch? I have been unsuccesful obtaining info on the
>> Web
>> > as query results are ambiguous and usually refer to crawldb or linkdb.
>> >
>> > If I can get this it would be a real nice entry for inclusion in our
>> wiki.
>> >
>> > Thanks for any suggestions or info.
>>
>
>
>
> --
> *Lewis*
>

Re: Crawling relation database

Posted by lewis john mcgibbney <le...@gmail.com>.
Thanks for this Markus, it had occured to me that DIH was a very plausable
solution to progress with. I think you have just confirmed due to the
flexibility it offers amongst other attributes.

I'm looking at creating a context aware web application which would use
geographical search to obtain results based on location. This is required as
the data will contain (amongst others) fields with integer values which vary
dependent upon a building location cost index. Similarity is directly linked
through geographical location factor. I wanted to have the data stored
within the n number of distributed RDB's available in a cloud environment
which could be searched as oppose to the non-trivial task of searching
across a fragmented distrubuted number of DB's.

As you mention, it does make more sense to save documents in a doc (or
column) oriented DB.

Essentially, using the DIH tool would remove requirement for Nutch?

I think to progress with this, I'm best moving the thread to Solr-user@ if I
have further questions.

Thank you
On Tue, Jul 5, 2011 at 3:53 PM, Markus Jelsma <ma...@openindex.io>wrote:

> Hi Lewis,
>
> It sounds to me you'd be better of using Solr's very advanced
> DataImportHandler [1]. It can (delta) import data from your RDBMS' and
> offers
> much flexibility on how to transform entities.
>
> Besides crawling you also mentions you'd like to push results (of what) to
> another structured data store. But why would you want that? Handling,
> processing and serving search results is done by Solr (and ES in the
> future)
> and since our entities are flat (just a document) it makes more sense to me
> to
> save documents in a document (or column) oriented DB.
>
> [1] :http://wiki.apache.org/solr/DataImportHandler
>
> Cheers,
>
> > Hi,
> >
> > I'm curious to hear if anyone has information for configuring Nutch to
> > crawl a RDB such as MySQL. In my hypothetical example there are N number
> > of databases residing in various distributed geographical locations, to
> > make a worst case scenario, say that they are NOT all the same type, and
> I
> > wish to use Nutch trunk 2.0 to push the results to some other structured
> > data store which I can then connect to to serve search results.
> >
> > Does anyone have any information such as an overview of database crawling
> > and serving using Nutch? I have been unsuccesful obtaining info on the
> Web
> > as query results are ambiguous and usually refer to crawldb or linkdb.
> >
> > If I can get this it would be a real nice entry for inclusion in our
> wiki.
> >
> > Thanks for any suggestions or info.
>



-- 
*Lewis*

Re: Crawling relation database

Posted by Markus Jelsma <ma...@openindex.io>.
Hi Lewis,

It sounds to me you'd be better of using Solr's very advanced 
DataImportHandler [1]. It can (delta) import data from your RDBMS' and offers 
much flexibility on how to transform entities.

Besides crawling you also mentions you'd like to push results (of what) to 
another structured data store. But why would you want that? Handling, 
processing and serving search results is done by Solr (and ES in the future) 
and since our entities are flat (just a document) it makes more sense to me to 
save documents in a document (or column) oriented DB.

[1] :http://wiki.apache.org/solr/DataImportHandler

Cheers,

> Hi,
> 
> I'm curious to hear if anyone has information for configuring Nutch to
> crawl a RDB such as MySQL. In my hypothetical example there are N number
> of databases residing in various distributed geographical locations, to
> make a worst case scenario, say that they are NOT all the same type, and I
> wish to use Nutch trunk 2.0 to push the results to some other structured
> data store which I can then connect to to serve search results.
> 
> Does anyone have any information such as an overview of database crawling
> and serving using Nutch? I have been unsuccesful obtaining info on the Web
> as query results are ambiguous and usually refer to crawldb or linkdb.
> 
> If I can get this it would be a real nice entry for inclusion in our wiki.
> 
> Thanks for any suggestions or info.

Re: Crawling relation database

Posted by Kirby Bohling <ki...@gmail.com>.
I was tasked with doing something like this.  We didn't do it, but my
thought process was to use the Java JDBC URL pathed out to the
database.

The two ideas we had were the following:

1. jdbc://dbserver.domain.com/tableName/schemaName/tableName
2. jdbc://dbserver.domain.com/${SQLQuery}

Then write the protocol handler for this, which given a URL of form
one, generate outlinks, one per row, where the URL is derived from the
primary key in the first case.  In the second case, you'd have to
handle that by assembling the appropriate primary keys, we didn't
finish that.

That is at least if you are searching for 'rows'.  We primarily wanted
to integrate Nutch into a Federated Search system, where it could
search both the public web, and a local document store.  We didn't get
too concerned about in/out links or the like.  Although there is
nothing to stop you from doing that, and following the foreign keys
and treating them like links.  The problem is that will likely lead
you to nasty scoring skew.

So if there were an internal row that had text which matched the core
values, it was imported and would likely be found.  The big problem
with that is ensuring that lookup tables get applied (in a
denormalized database, the author's name likely isn't attached to a
document, it is likely referenced via a foreign key, so searching for
those became problematic, as 'John Doe' and '1234' where associated in
the author table, and 1234 and the full text of the document were
associated, 'John Doe' and the full text weren't).  That's where the
second idea came from, so we'd enumerate queries which were
representative of what we thought the results from querying the
database should be.

Kirby

On Tue, Jul 5, 2011 at 5:44 PM, lewis john mcgibbney
<le...@gmail.com> wrote:
> Hi,
>
> I'm curious to hear if anyone has information for configuring Nutch to crawl
> a RDB such as MySQL. In my hypothetical example there are N number of
> databases residing in various distributed geographical locations, to make a
> worst case scenario, say that they are NOT all the same type, and I wish to
> use Nutch trunk 2.0 to push the results to some other structured data store
> which I can then connect to to serve search results.
>
> Does anyone have any information such as an overview of database crawling
> and serving using Nutch? I have been unsuccesful obtaining info on the Web
> as query results are ambiguous and usually refer to crawldb or linkdb.
>
> If I can get this it would be a real nice entry for inclusion in our wiki.
>
> Thanks for any suggestions or info.
>
> --
> *Lewis*
>