You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@nutch.apache.org by tripiy <po...@gmail.com> on 2014/03/12 12:17:21 UTC

How do I customize Nutch to cater to existing SOLR schema

Hi,

I am working with a SOLR instance with a schema defined by the CMS system
that are using. Additionally i want to crawl and index content from some
external sites using Nutch. The biggest issue i am facing is that my
uniquekey is different from that provided by Nutch (i want to us "_uniqueid"
instead of default "id"). Though there are other custom fields which i can
manage using the solr copyfield however i'm not able to get around the
uniquekey which cannot be set as destination for a copyfield.

Is there a way to do this customization on Nutch thru config or the Nutch
code needs to be modified to accommodate custom SOLR schema.

thanx



--
View this message in context: http://lucene.472066.n3.nabble.com/How-do-I-customize-Nutch-to-cater-to-existing-SOLR-schema-tp4123062.html
Sent from the Nutch - Dev mailing list archive at Nabble.com.

Re: How do I customize Nutch to cater to existing SOLR schema

Posted by Lajos <la...@protulae.com>.

Hi tripiy,

I had raised this issue a few months ago, as I had the exact same 
problem. It cannot be solved by configuration, because of the way that 
the MappingReader puts fields in Maps.

I solved this by implementing a custom plugin that further would support 
some basic transformations, ala Solr DIH.

I proposed this as a contribution, but the only feedback comment didn't 
get the use case. Given you have the exact same issue, I'd say it is a 
useful contribution.

I'll see how/what I can do to get the code available. If a committer can 
comment/provide guidelines, I'd appreciate it.

Thanks,

Lajos

On 12/03/2014 12:17, tripiy wrote:
> Hi,
>
> I am working with a SOLR instance with a schema defined by the CMS system
> that are using. Additionally i want to crawl and index content from some
> external sites using Nutch. The biggest issue i am facing is that my
> uniquekey is different from that provided by Nutch (i want to us "_uniqueid"
> instead of default "id"). Though there are other custom fields which i can
> manage using the solr copyfield however i'm not able to get around the
> uniquekey which cannot be set as destination for a copyfield.
>
> Is there a way to do this customization on Nutch thru config or the Nutch
> code needs to be modified to accommodate custom SOLR schema.
>
> thanx
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/How-do-I-customize-Nutch-to-cater-to-existing-SOLR-schema-tp4123062.html
> Sent from the Nutch - Dev mailing list archive at Nabble.com.
>

Re: How do I customize Nutch to cater to existing SOLR schema

Posted by Lajos <la...@protulae.com>.

Hi tripiy,

Good to hear.

I've been under the weather lately and haven't been able to turn my 
attention to on this ... hope to be better by the weekend.

Cheers,

Lajos


On 17/03/2014 11:42, tripiy wrote:
> Hi Lajos,
>
> Appreciate ur help in providing the patch which would definitely improve the
> usability of the product.
>
> For now we have resolved the unique field issue using the following changes
> to solrindex-mapping.xml:
> <field dest="_uniqueid" source="url"/>
>   <copyField source="url" dest="_uniqueid"/>
>
> For other the Nutch fields we have added the corresponding fields in SOLR
> schema which are copied into the respective target CMS Schema fields.
>
> Will wait for ur patch to get a more robust and flexible solution.
>
> thanx
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/How-do-I-customize-Nutch-to-cater-to-existing-SOLR-schema-tp4123062p4124742.html
> Sent from the Nutch - Dev mailing list archive at Nabble.com.
>

Re: How do I customize Nutch to cater to existing SOLR schema

Posted by tripiy <po...@gmail.com>.

Hi Lajos,

Appreciate ur help in providing the patch which would definitely improve the
usability of the product.

For now we have resolved the unique field issue using the following changes
to solrindex-mapping.xml:
<field dest="_uniqueid" source="url"/> 
 <copyField source="url" dest="_uniqueid"/>

For other the Nutch fields we have added the corresponding fields in SOLR
schema which are copied into the respective target CMS Schema fields.

Will wait for ur patch to get a more robust and flexible solution.

thanx




--
View this message in context: http://lucene.472066.n3.nabble.com/How-do-I-customize-Nutch-to-cater-to-existing-SOLR-schema-tp4123062p4124742.html
Sent from the Nutch - Dev mailing list archive at Nabble.com.