I think I found a bug --> multiple_values_encountered_for_non_multiValued_field_title

[kaveh minooie <ka...@plutoz.com>]

so I've been getting this error 
"multiple_values_encountered_for_non_multiValued_field_title" every once 
in a while when I am trying to run solrindex. I can now say that this is 
being caused by index-more plug in (MoreIndexingFilter.java)

private NutchDocument resetTitle(NutchDocument doc, ParseData data, 
String url) {
	    String contentDisposition = data.getMeta(Metadata.CONTENT_DISPOSITION);
	    if (contentDisposition == null)
	      return doc;
	
	    for (int i=0; i<patterns.length; i++) {
	      Matcher matcher = patterns[i].matcher(contentDisposition);
	      if (matcher.find()) {
	        doc.add("title", matcher.group(1));
	        break;
	      }
	    }
	   return doc;
	  }


the problem here is that in my case this function is not reseting but it 
is just adding a new title. it seems that the original idea was that if 
CONTENT_DISPOSITION exist then the document will not have a title set 
from other plug ins (namely index-basic). unfortunately this seems not 
to be always the case as you can see by running this command:

bin/nutch indexchecker http://www.2modern.com/site/gift-registry.html

what i do get (the part that is relevant) is:

	
tstamp :	Tue Feb 21 13:18:13 PST 2012
type :	text/html
type :	text
type :	html
date :	Tue Feb 21 13:18:13 PST 2012
url :	http://www.2modern.com/site/gift-registry.html
content :	2Modern Gift Registry      Modern Furniture & Lighting items 
in cart 0 checkout Returning 2Modern cu
user_ranking :	25.0
title :	2Modern Gift Registry
title :	gift-registry.html
plutoz_ranking :	10.0
categories :	Furniture Home
contentLength :	12924

and as you can see there are 2 titles. I think it would be very easy to 
fix that. just check to see if a title exist already before setting the 
name of the file as title:

if (contentDisposition == null || null != doc.getField("title"))
	      return doc;


  or if the substitution must happen in presence of CONTENT_DISPOSITION, 
at least remove the old one:

if (matcher.find()) {
         doc.remove("title");
	doc.add("title", matcher.group(1));
         break;
  }


  now that being said, the real problem here is why NutchDocument 
doesn't observe the schema.xml file and alway assumes that all fields 
are multi value?

public void add(String name, Object value) {
53	    NutchField field = fields.get(name);
54	    if (field == null) {
55	      field = new NutchField(value);
56	      fields.put(name, field);
57	    } else {
58	----> field.add(value);  <---
59	    }
60	  }

-- 
Kaveh Minooie

www.plutoz.com

Re: I think I found a bug --> multiple_values_encountered_for_non_multiValued_field_title

[Lewis John Mcgibbney <le...@gmail.com>]

Hi Kaveh,

Based on the example you provide there is no doubt that there needs to be
something done here to address multivalued titles, I think that the small
patch you provide addresses this.



On Wed, Feb 22, 2012 at 12:48 AM, kaveh minooie <ka...@plutoz.com> wrote:

>
>
>  now that being said, the real problem here is why NutchDocument doesn't
> observe the schema.xml file and alway assumes that all fields are multi
> value?
>

I would like to hear from guys more experienced with schema semantics, how
a multiValued field option for _every_ field would affect desired indexing
and subsequently searching within Solr! I am sorry, but admit that I don't
know enough about this particular element to comment in any more detail.

Lewis