You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by j....@thomsonreuters.com on 2012/10/09 12:17:14 UTC
Keeping History/Archive with Nutch 2.x
Hi
Rather than a wide crawl of the web keeping track of the current state of sites (as I understand Nutch is currently optimized for) I am interested in keeping copies of a more modest number of sites over time as they change. In other words keeping copies of both the old webpages and the new pages as they change. My overly optimistic wishful thinking is that I could get close enough to this by simply adding the signature (TextProfileSignature in particular) to the current id key. Any thoughts as to if this is feasible and if so where in the codebase I should start looking in order to do that? I am aware Heritrix specializes in archiving but I would really like to stick with Nutch if possible unless it absolutely doesn't make sense.
Thanks
James
RE: Keeping History/Archive with Nutch 2.x
Posted by j....@thomsonreuters.com.
Thank Julien and Ferdy, appreciated. I will look into a custom map reduce job for MySQL first as I don't really have the search size yet to justify HBase. Depending on how difficult that turns out to be I may try HBase.
-----Original Message-----
From: Julien Nioche [mailto:lists.digitalpebble@gmail.com]
Sent: Wednesday, October 10, 2012 5:46 AM
To: user@nutch.apache.org
Subject: Re: Keeping History/Archive with Nutch 2.x
Good point Ferdy, thanks!
On 9 October 2012 18:10, Ferdy Galema <fe...@kalooga.com> wrote:
> Hi,
>
> HBase with multiple versions is certainly an option, however the current
> HBaseStore implementation is implemented with a single version in mind. (I
> have not really tested what happens with multiple versions, I guess you get
> unexpected/undefined results). The exception to this case would be to
> setting specific column families for multiple values (for example 'content'
> and put it into a separate column family). Storing new content would
> overwrite the old ones. You have to have an external process or implemented
> tool to retrieve earlier versions from the store. For information like maps
> (inlinks, outlinks, metadata) the results with multiple versions are a lot
> more confusing. There is still some work to do.
>
> In short, yes HBase would work but you definitely would have to hack a
> custom HBaseStore if you want to perfectly keep track of snapshots.
>
> Ferdy.
>
> On Tue, Oct 9, 2012 at 3:30 PM, Julien Nioche <
> lists.digitalpebble@gmail.com
> > wrote:
>
> > Hi James
> >
> > You could have a custom map reduce job to copy the documents with a
> custom
> > ID as you just described. Another option would be to use Nutch 2 + HBase
> > and set a large value of versions (
> > http://hbase.apache.org/book/schema.versions.html) in the HBase schema.
> >
> > Julien
> >
> > On 9 October 2012 11:17, <j....@thomsonreuters.com> wrote:
> >
> > > Hi
> > >
> > > Rather than a wide crawl of the web keeping track of the current state
> of
> > > sites (as I understand Nutch is currently optimized for) I am
> interested
> > in
> > > keeping copies of a more modest number of sites over time as they
> change.
> > > In other words keeping copies of both the old webpages and the new
> pages
> > as
> > > they change. My overly optimistic wishful thinking is that I could get
> > > close enough to this by simply adding the signature
> (TextProfileSignature
> > > in particular) to the current id key. Any thoughts as to if this is
> > > feasible and if so where in the codebase I should start looking in
> order
> > to
> > > do that? I am aware Heritrix specializes in archiving but I would
> really
> > > like to stick with Nutch if possible unless it absolutely doesn't make
> > > sense.
> > >
> > > Thanks
> > >
> > > James
> > >
> >
> >
> >
> > --
> > *
> > *Open Source Solutions for Text Engineering
> >
> > http://digitalpebble.blogspot.com/
> > http://www.digitalpebble.com
> > http://twitter.com/digitalpebble
> >
>
--
*
*Open Source Solutions for Text Engineering
http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
http://twitter.com/digitalpebble
Re: Keeping History/Archive with Nutch 2.x
Posted by Jorge Luis Betancourt Gonzalez <jl...@uci.cu>.
If I want to keep a cache of the websites crawled, something similar to the Google cached view, this way to go with the HBase would be the best option or storing in a filesystem?
----- Mensaje original -----
De: "Julien Nioche" <li...@gmail.com>
Para: user@nutch.apache.org
Enviados: Martes, 9 de Octubre 2012 16:45:59
Asunto: Re: Keeping History/Archive with Nutch 2.x
Good point Ferdy, thanks!
On 9 October 2012 18:10, Ferdy Galema <fe...@kalooga.com> wrote:
> Hi,
>
> HBase with multiple versions is certainly an option, however the current
> HBaseStore implementation is implemented with a single version in mind. (I
> have not really tested what happens with multiple versions, I guess you get
> unexpected/undefined results). The exception to this case would be to
> setting specific column families for multiple values (for example 'content'
> and put it into a separate column family). Storing new content would
> overwrite the old ones. You have to have an external process or implemented
> tool to retrieve earlier versions from the store. For information like maps
> (inlinks, outlinks, metadata) the results with multiple versions are a lot
> more confusing. There is still some work to do.
>
> In short, yes HBase would work but you definitely would have to hack a
> custom HBaseStore if you want to perfectly keep track of snapshots.
>
> Ferdy.
>
> On Tue, Oct 9, 2012 at 3:30 PM, Julien Nioche <
> lists.digitalpebble@gmail.com
> > wrote:
>
> > Hi James
> >
> > You could have a custom map reduce job to copy the documents with a
> custom
> > ID as you just described. Another option would be to use Nutch 2 + HBase
> > and set a large value of versions (
> > http://hbase.apache.org/book/schema.versions.html) in the HBase schema.
> >
> > Julien
> >
> > On 9 October 2012 11:17, <j....@thomsonreuters.com> wrote:
> >
> > > Hi
> > >
> > > Rather than a wide crawl of the web keeping track of the current state
> of
> > > sites (as I understand Nutch is currently optimized for) I am
> interested
> > in
> > > keeping copies of a more modest number of sites over time as they
> change.
> > > In other words keeping copies of both the old webpages and the new
> pages
> > as
> > > they change. My overly optimistic wishful thinking is that I could get
> > > close enough to this by simply adding the signature
> (TextProfileSignature
> > > in particular) to the current id key. Any thoughts as to if this is
> > > feasible and if so where in the codebase I should start looking in
> order
> > to
> > > do that? I am aware Heritrix specializes in archiving but I would
> really
> > > like to stick with Nutch if possible unless it absolutely doesn't make
> > > sense.
> > >
> > > Thanks
> > >
> > > James
> > >
> >
> >
> >
> > --
> > *
> > *Open Source Solutions for Text Engineering
> >
> > http://digitalpebble.blogspot.com/
> > http://www.digitalpebble.com
> > http://twitter.com/digitalpebble
> >
>
--
*
*Open Source Solutions for Text Engineering
http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
http://twitter.com/digitalpebble
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
Re: Keeping History/Archive with Nutch 2.x
Posted by Julien Nioche <li...@gmail.com>.
Good point Ferdy, thanks!
On 9 October 2012 18:10, Ferdy Galema <fe...@kalooga.com> wrote:
> Hi,
>
> HBase with multiple versions is certainly an option, however the current
> HBaseStore implementation is implemented with a single version in mind. (I
> have not really tested what happens with multiple versions, I guess you get
> unexpected/undefined results). The exception to this case would be to
> setting specific column families for multiple values (for example 'content'
> and put it into a separate column family). Storing new content would
> overwrite the old ones. You have to have an external process or implemented
> tool to retrieve earlier versions from the store. For information like maps
> (inlinks, outlinks, metadata) the results with multiple versions are a lot
> more confusing. There is still some work to do.
>
> In short, yes HBase would work but you definitely would have to hack a
> custom HBaseStore if you want to perfectly keep track of snapshots.
>
> Ferdy.
>
> On Tue, Oct 9, 2012 at 3:30 PM, Julien Nioche <
> lists.digitalpebble@gmail.com
> > wrote:
>
> > Hi James
> >
> > You could have a custom map reduce job to copy the documents with a
> custom
> > ID as you just described. Another option would be to use Nutch 2 + HBase
> > and set a large value of versions (
> > http://hbase.apache.org/book/schema.versions.html) in the HBase schema.
> >
> > Julien
> >
> > On 9 October 2012 11:17, <j....@thomsonreuters.com> wrote:
> >
> > > Hi
> > >
> > > Rather than a wide crawl of the web keeping track of the current state
> of
> > > sites (as I understand Nutch is currently optimized for) I am
> interested
> > in
> > > keeping copies of a more modest number of sites over time as they
> change.
> > > In other words keeping copies of both the old webpages and the new
> pages
> > as
> > > they change. My overly optimistic wishful thinking is that I could get
> > > close enough to this by simply adding the signature
> (TextProfileSignature
> > > in particular) to the current id key. Any thoughts as to if this is
> > > feasible and if so where in the codebase I should start looking in
> order
> > to
> > > do that? I am aware Heritrix specializes in archiving but I would
> really
> > > like to stick with Nutch if possible unless it absolutely doesn't make
> > > sense.
> > >
> > > Thanks
> > >
> > > James
> > >
> >
> >
> >
> > --
> > *
> > *Open Source Solutions for Text Engineering
> >
> > http://digitalpebble.blogspot.com/
> > http://www.digitalpebble.com
> > http://twitter.com/digitalpebble
> >
>
--
*
*Open Source Solutions for Text Engineering
http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
http://twitter.com/digitalpebble
Re: Keeping History/Archive with Nutch 2.x
Posted by Ferdy Galema <fe...@kalooga.com>.
Hi,
HBase with multiple versions is certainly an option, however the current
HBaseStore implementation is implemented with a single version in mind. (I
have not really tested what happens with multiple versions, I guess you get
unexpected/undefined results). The exception to this case would be to
setting specific column families for multiple values (for example 'content'
and put it into a separate column family). Storing new content would
overwrite the old ones. You have to have an external process or implemented
tool to retrieve earlier versions from the store. For information like maps
(inlinks, outlinks, metadata) the results with multiple versions are a lot
more confusing. There is still some work to do.
In short, yes HBase would work but you definitely would have to hack a
custom HBaseStore if you want to perfectly keep track of snapshots.
Ferdy.
On Tue, Oct 9, 2012 at 3:30 PM, Julien Nioche <lists.digitalpebble@gmail.com
> wrote:
> Hi James
>
> You could have a custom map reduce job to copy the documents with a custom
> ID as you just described. Another option would be to use Nutch 2 + HBase
> and set a large value of versions (
> http://hbase.apache.org/book/schema.versions.html) in the HBase schema.
>
> Julien
>
> On 9 October 2012 11:17, <j....@thomsonreuters.com> wrote:
>
> > Hi
> >
> > Rather than a wide crawl of the web keeping track of the current state of
> > sites (as I understand Nutch is currently optimized for) I am interested
> in
> > keeping copies of a more modest number of sites over time as they change.
> > In other words keeping copies of both the old webpages and the new pages
> as
> > they change. My overly optimistic wishful thinking is that I could get
> > close enough to this by simply adding the signature (TextProfileSignature
> > in particular) to the current id key. Any thoughts as to if this is
> > feasible and if so where in the codebase I should start looking in order
> to
> > do that? I am aware Heritrix specializes in archiving but I would really
> > like to stick with Nutch if possible unless it absolutely doesn't make
> > sense.
> >
> > Thanks
> >
> > James
> >
>
>
>
> --
> *
> *Open Source Solutions for Text Engineering
>
> http://digitalpebble.blogspot.com/
> http://www.digitalpebble.com
> http://twitter.com/digitalpebble
>
Re: Keeping History/Archive with Nutch 2.x
Posted by Julien Nioche <li...@gmail.com>.
Hi James
You could have a custom map reduce job to copy the documents with a custom
ID as you just described. Another option would be to use Nutch 2 + HBase
and set a large value of versions (
http://hbase.apache.org/book/schema.versions.html) in the HBase schema.
Julien
On 9 October 2012 11:17, <j....@thomsonreuters.com> wrote:
> Hi
>
> Rather than a wide crawl of the web keeping track of the current state of
> sites (as I understand Nutch is currently optimized for) I am interested in
> keeping copies of a more modest number of sites over time as they change.
> In other words keeping copies of both the old webpages and the new pages as
> they change. My overly optimistic wishful thinking is that I could get
> close enough to this by simply adding the signature (TextProfileSignature
> in particular) to the current id key. Any thoughts as to if this is
> feasible and if so where in the codebase I should start looking in order to
> do that? I am aware Heritrix specializes in archiving but I would really
> like to stick with Nutch if possible unless it absolutely doesn't make
> sense.
>
> Thanks
>
> James
>
--
*
*Open Source Solutions for Text Engineering
http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
http://twitter.com/digitalpebble
RE: Keeping History/Archive with Nutch 2.x
Posted by j....@thomsonreuters.com.
I agree that doing it at the Solr level is the most straight forward easy way. However, if possible I would like to do it at the webpage table level. That way I would have the original data and I would be able to reindex data at a later date and retroactively apply any improvements to the indexing. I am fairly certain I will screw up my solr index/want to make changes to it at some point.
________________________________________
From: Dave Stuart [david.stuart@progressivealliance.co.uk]
Sent: Tuesday, October 09, 2012 7:22 PM
To: user@nutch.apache.org
Subject: Re: Keeping History/Archive with Nutch 2.x
Are you pushing it into a search index of some sort?
As I mostly push things into Solr I would modify the key to take signature into account.
On 9 Oct 2012, at 11:17, <j....@thomsonreuters.com> wrote:
> Hi
>
> Rather than a wide crawl of the web keeping track of the current state of sites (as I understand Nutch is currently optimized for) I am interested in keeping copies of a more modest number of sites over time as they change. In other words keeping copies of both the old webpages and the new pages as they change. My overly optimistic wishful thinking is that I could get close enough to this by simply adding the signature (TextProfileSignature in particular) to the current id key. Any thoughts as to if this is feasible and if so where in the codebase I should start looking in order to do that? I am aware Heritrix specializes in archiving but I would really like to stick with Nutch if possible unless it absolutely doesn't make sense.
>
> Thanks
>
> James
Re: Keeping History/Archive with Nutch 2.x
Posted by Dave Stuart <da...@progressivealliance.co.uk>.
Are you pushing it into a search index of some sort?
As I mostly push things into Solr I would modify the key to take signature into account.
On 9 Oct 2012, at 11:17, <j....@thomsonreuters.com> wrote:
> Hi
>
> Rather than a wide crawl of the web keeping track of the current state of sites (as I understand Nutch is currently optimized for) I am interested in keeping copies of a more modest number of sites over time as they change. In other words keeping copies of both the old webpages and the new pages as they change. My overly optimistic wishful thinking is that I could get close enough to this by simply adding the signature (TextProfileSignature in particular) to the current id key. Any thoughts as to if this is feasible and if so where in the codebase I should start looking in order to do that? I am aware Heritrix specializes in archiving but I would really like to stick with Nutch if possible unless it absolutely doesn't make sense.
>
> Thanks
>
> James