How to ask Nutch to get value of extra fields in IndexerJob/IndexerMapper?

[jefferyyuan <yu...@gmail.com>]

I am trying to save raw content and other information such as outlinks to
Solr. 
I know Nutch2 doesn't do this, So I am trying to make some code change in
Nutch2(a nutch IndexingFilter plugin later). 

Checked Nutch2 code, seems we can add extra fields we need into
Collection<WebPage.Field> FIELDS, Nutch2 uses Gora to run a query and add
the value into Webpage instance. 

If I add FIELDS.add(WebPage.Field.CONTENT)(just for a test); in
DbUpdaterJob, it works, in DbUpdateMapper I can see the value of content
field in Webpage instance.. 

But if I can FIELDS.add(WebPage.Field.CONTENT) into IndexerJob, it doesn't
work, still no content in webpage instance of IndexerMapper.map method. 
Could anyone provide me suggestion how to read value of extra fields from
underlying datastore in IndexerJob/IndexerMapper? 
Thanks very much.



--
View this message in context: http://lucene.472066.n3.nabble.com/How-to-ask-Nutch-to-get-value-of-extra-fields-in-IndexerJob-IndexerMapper-tp4083376.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Re: How to ask Nutch to get value of extra fields in IndexerJob/IndexerMapper?

[Lewis John Mcgibbney <le...@gmail.com>]

Hi Jeffery,
Sorry about length of time to respond.
Did you get a solution here?
I wonder if this has to do with your crawlId?
I would definately say that an indexing plugin is the way to go here. Put
the outlinks to avro map in WebPage then get them and add them to your doc.

On Thursday, August 8, 2013, jefferyyuan <yu...@gmail.com> wrote:
> I am trying to save raw content and other information such as outlinks to
> Solr.
> I know Nutch2 doesn't do this, So I am trying to make some code change in
> Nutch2(a nutch IndexingFilter plugin later).
>
> Checked Nutch2 code, seems we can add extra fields we need into
> Collection<WebPage.Field> FIELDS, Nutch2 uses Gora to run a query and add
> the value into Webpage instance.
>
> If I add FIELDS.add(WebPage.Field.CONTENT)(just for a test); in
> DbUpdaterJob, it works, in DbUpdateMapper I can see the value of content
> field in Webpage instance..
>
> But if I can FIELDS.add(WebPage.Field.CONTENT) into IndexerJob, it doesn't
> work, still no content in webpage instance of IndexerMapper.map method.
> Could anyone provide me suggestion how to read value of extra fields from
> underlying datastore in IndexerJob/IndexerMapper?
> Thanks very much.
>
>
>
> --
> View this message in context:
http://lucene.472066.n3.nabble.com/How-to-ask-Nutch-to-get-value-of-extra-fields-in-IndexerJob-IndexerMapper-tp4083376.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>

-- 
*Lewis*