You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Katrina Riehl <ka...@continuum.io> on 2015/04/08 15:50:30 UTC
Adding field to Nutch / Solr
Hello,
I am new to using Nutch. I'm developing an application that crawls
websites, and then indexes information about those websites into a Solr
instance. The problem is, it's putting all the crawled documents into the
same Solr collection.
Is there a way for me to add a field specifying which crawl the index came
from? Is there a command line option I can add when I start the crawl?
Thank you so much for your help.
--
Katrina Riehl
Continuum Analytics
katrina.riehl@continuum.io
RE: Adding field to Nutch / Solr
Posted by Iain Lopata <il...@hotmail.com>.
If you are planning to have the crawl name specified at the command line, one approach would be to modify your script to use sed to modify nutch-site.xml before the crawl steps begins.
-----Original Message-----
From: Katrina Riehl [mailto:katrina.riehl@continuum.io]
Sent: Wednesday, April 8, 2015 10:00 AM
To: user@nutch.apache.org
Subject: Re: Adding field to Nutch / Solr
My understanding is that I can use the index-state plugin if the information doesn't change... remains static.... but, there will be a different crawl_name every time I run a new crawl. I'd like to take that new crawl name and add it into solr somehow.
Is my understanding correct? Or is there a way to override that field on a per-crawl basis?
Thanks
On Wed, Apr 8, 2015 at 9:54 AM, Iain Lopata <il...@hotmail.com> wrote:
> Katrina,
>
> If I am understanding you correctly, you could do this with the
> index-static plugin which is configured with the following property:
>
> <property>
> <name>index.static</name>
> <value> fieldname:fieldcontent </value>
> <description>
> A simple plugin called at indexing that adds fields with static data.
> You can specify a list of fieldname:fieldcontent per nutch job.
> It can be useful when collections can't be created by urlpatterns,
> like in subcollection, but on a job-basis.
> </description>
> </property>
>
> Use crawlname as your fieldname and use a different config directory
> for each of your crawls with an appropriate value for fieldcontent set in each.
>
> Iain
>
> -----Original Message-----
> From: Katrina Riehl [mailto:katrina.riehl@continuum.io]
> Sent: Wednesday, April 8, 2015 9:41 AM
> To: user@nutch.apache.org
> Subject: Re: Adding field to Nutch / Solr
>
> Right, I can create multiple collections no problem... but, what I'd
> really love is to put them into the same collection, just adding a
> field like "crawl_name" to the index.
>
> Any way I can do that?
>
> Thanks!
>
>
> On Wed, Apr 8, 2015 at 9:15 AM, Iain Lopata <il...@hotmail.com> wrote:
>
> > Katrina,
> >
> > When you specify the solr instance as the third parameter to
> > bin/crawl try specifying the collection name in the path e.g.
> > http://localhost:8080/solr/collection1
> >
> > Iain
> >
> > -----Original Message-----
> > From: Katrina Riehl [mailto:katrina.riehl@continuum.io]
> > Sent: Wednesday, April 8, 2015 8:51 AM
> > To: user@nutch.apache.org
> > Subject: Adding field to Nutch / Solr
> >
> > Hello,
> >
> > I am new to using Nutch. I'm developing an application that crawls
> > websites, and then indexes information about those websites into a
> > Solr instance. The problem is, it's putting all the crawled
> > documents into the same Solr collection.
> >
> > Is there a way for me to add a field specifying which crawl the
> > index came from? Is there a command line option I can add when I
> > start the
> crawl?
> >
> > Thank you so much for your help.
> >
> > --
> > Katrina Riehl
> > Continuum Analytics
> > katrina.riehl@continuum.io
> >
> >
>
>
Re: Adding field to Nutch / Solr
Posted by Katrina Riehl <ka...@continuum.io>.
My understanding is that I can use the index-state plugin if the
information doesn't change... remains static.... but, there will be a
different crawl_name every time I run a new crawl. I'd like to take that
new crawl name and add it into solr somehow.
Is my understanding correct? Or is there a way to override that field on a
per-crawl basis?
Thanks
On Wed, Apr 8, 2015 at 9:54 AM, Iain Lopata <il...@hotmail.com> wrote:
> Katrina,
>
> If I am understanding you correctly, you could do this with the
> index-static plugin which is configured with the following property:
>
> <property>
> <name>index.static</name>
> <value> fieldname:fieldcontent </value>
> <description>
> A simple plugin called at indexing that adds fields with static data.
> You can specify a list of fieldname:fieldcontent per nutch job.
> It can be useful when collections can't be created by urlpatterns,
> like in subcollection, but on a job-basis.
> </description>
> </property>
>
> Use crawlname as your fieldname and use a different config directory for
> each of your crawls with an appropriate value for fieldcontent set in each.
>
> Iain
>
> -----Original Message-----
> From: Katrina Riehl [mailto:katrina.riehl@continuum.io]
> Sent: Wednesday, April 8, 2015 9:41 AM
> To: user@nutch.apache.org
> Subject: Re: Adding field to Nutch / Solr
>
> Right, I can create multiple collections no problem... but, what I'd
> really love is to put them into the same collection, just adding a field
> like "crawl_name" to the index.
>
> Any way I can do that?
>
> Thanks!
>
>
> On Wed, Apr 8, 2015 at 9:15 AM, Iain Lopata <il...@hotmail.com> wrote:
>
> > Katrina,
> >
> > When you specify the solr instance as the third parameter to bin/crawl
> > try specifying the collection name in the path e.g.
> > http://localhost:8080/solr/collection1
> >
> > Iain
> >
> > -----Original Message-----
> > From: Katrina Riehl [mailto:katrina.riehl@continuum.io]
> > Sent: Wednesday, April 8, 2015 8:51 AM
> > To: user@nutch.apache.org
> > Subject: Adding field to Nutch / Solr
> >
> > Hello,
> >
> > I am new to using Nutch. I'm developing an application that crawls
> > websites, and then indexes information about those websites into a
> > Solr instance. The problem is, it's putting all the crawled documents
> > into the same Solr collection.
> >
> > Is there a way for me to add a field specifying which crawl the index
> > came from? Is there a command line option I can add when I start the
> crawl?
> >
> > Thank you so much for your help.
> >
> > --
> > Katrina Riehl
> > Continuum Analytics
> > katrina.riehl@continuum.io
> >
> >
>
>
RE: Adding field to Nutch / Solr
Posted by Iain Lopata <il...@hotmail.com>.
Katrina,
If I am understanding you correctly, you could do this with the index-static plugin which is configured with the following property:
<property>
<name>index.static</name>
<value> fieldname:fieldcontent </value>
<description>
A simple plugin called at indexing that adds fields with static data.
You can specify a list of fieldname:fieldcontent per nutch job.
It can be useful when collections can't be created by urlpatterns,
like in subcollection, but on a job-basis.
</description>
</property>
Use crawlname as your fieldname and use a different config directory for each of your crawls with an appropriate value for fieldcontent set in each.
Iain
-----Original Message-----
From: Katrina Riehl [mailto:katrina.riehl@continuum.io]
Sent: Wednesday, April 8, 2015 9:41 AM
To: user@nutch.apache.org
Subject: Re: Adding field to Nutch / Solr
Right, I can create multiple collections no problem... but, what I'd really love is to put them into the same collection, just adding a field like "crawl_name" to the index.
Any way I can do that?
Thanks!
On Wed, Apr 8, 2015 at 9:15 AM, Iain Lopata <il...@hotmail.com> wrote:
> Katrina,
>
> When you specify the solr instance as the third parameter to bin/crawl
> try specifying the collection name in the path e.g.
> http://localhost:8080/solr/collection1
>
> Iain
>
> -----Original Message-----
> From: Katrina Riehl [mailto:katrina.riehl@continuum.io]
> Sent: Wednesday, April 8, 2015 8:51 AM
> To: user@nutch.apache.org
> Subject: Adding field to Nutch / Solr
>
> Hello,
>
> I am new to using Nutch. I'm developing an application that crawls
> websites, and then indexes information about those websites into a
> Solr instance. The problem is, it's putting all the crawled documents
> into the same Solr collection.
>
> Is there a way for me to add a field specifying which crawl the index
> came from? Is there a command line option I can add when I start the crawl?
>
> Thank you so much for your help.
>
> --
> Katrina Riehl
> Continuum Analytics
> katrina.riehl@continuum.io
>
>
Re: Adding field to Nutch / Solr
Posted by Katrina Riehl <ka...@continuum.io>.
Right, I can create multiple collections no problem... but, what I'd really
love is to put them into the same collection, just adding a field like
"crawl_name" to the index.
Any way I can do that?
Thanks!
On Wed, Apr 8, 2015 at 9:15 AM, Iain Lopata <il...@hotmail.com> wrote:
> Katrina,
>
> When you specify the solr instance as the third parameter to bin/crawl
> try specifying the collection name in the path e.g.
> http://localhost:8080/solr/collection1
>
> Iain
>
> -----Original Message-----
> From: Katrina Riehl [mailto:katrina.riehl@continuum.io]
> Sent: Wednesday, April 8, 2015 8:51 AM
> To: user@nutch.apache.org
> Subject: Adding field to Nutch / Solr
>
> Hello,
>
> I am new to using Nutch. I'm developing an application that crawls
> websites, and then indexes information about those websites into a Solr
> instance. The problem is, it's putting all the crawled documents into the
> same Solr collection.
>
> Is there a way for me to add a field specifying which crawl the index came
> from? Is there a command line option I can add when I start the crawl?
>
> Thank you so much for your help.
>
> --
> Katrina Riehl
> Continuum Analytics
> katrina.riehl@continuum.io
>
>
RE: Adding field to Nutch / Solr
Posted by Iain Lopata <il...@hotmail.com>.
Katrina,
When you specify the solr instance as the third parameter to bin/crawl try specifying the collection name in the path e.g. http://localhost:8080/solr/collection1
Iain
-----Original Message-----
From: Katrina Riehl [mailto:katrina.riehl@continuum.io]
Sent: Wednesday, April 8, 2015 8:51 AM
To: user@nutch.apache.org
Subject: Adding field to Nutch / Solr
Hello,
I am new to using Nutch. I'm developing an application that crawls websites, and then indexes information about those websites into a Solr instance. The problem is, it's putting all the crawled documents into the same Solr collection.
Is there a way for me to add a field specifying which crawl the index came from? Is there a command line option I can add when I start the crawl?
Thank you so much for your help.
--
Katrina Riehl
Continuum Analytics
katrina.riehl@continuum.io