You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Albert Vila <av...@imente.com> on 2007/08/06 14:12:14 UTC

Multiple fields vs one field

Hi all

  My data looks like:

	Document 1
		code, title, content, type, language, date, ...
	Document 2
		code, title, content, type, language, date, ...
	...
	Document n
		code, title, content, type, language, date, ...

  Now all document types share the same fields, but in a future we  
need to add more document types with specific fields. I allways sort  
documents by date. I have 200.000 new documents each day and 130  
million documents. The janaury index size is 4.2Gb (the data size is  
about 10Gb).

  I was wondering how to index the new document types.
	Option 1: One index for each document type. Each index will have its  
fields.
		Problem: I will have to perfom a search for each index, and sort  
results by date.
	Option 2: One big index containing all fields. A field could be  
empty if the field is not applicable for that document type.
	Option 3: One big index containing all common fields, and adding and  
extra field named metadata. Inside this field I will add all specific  
fields (field1:value1 field2:value2).

Comments, pros and contras will be appreciated. I don't know exactly  
the diference between option 2 and option 3.

Thanks

Albert

  

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Multiple fields vs one field

Posted by Albert Vila <av...@imente.com>.
Thank you very much, that is right :)

El 06/08/2007, a las 14:23, Chhabra, Kapil escribió:

> Hey Albert,
> Just to remind you, that the fields in Lucene are per document and not
> per index. This means that you can have documents in an index which  
> have
> different fields altogether.
> So, in effect, you can all your document types to your existing index.
> And guess what, you don't need to change anything for this.
>
> All the best.
>
> Regards,
> kapilChhabra
>
> -----Original Message-----
> From: Albert Vila [mailto:avp@imente.com]
> Sent: Monday, August 06, 2007 5:42 PM
> To: java-user@lucene.apache.org
> Subject: Multiple fields vs one field
>
> Hi all
>
>   My data looks like:
>
> 	Document 1
> 		code, title, content, type, language, date, ...
> 	Document 2
> 		code, title, content, type, language, date, ...
> 	...
> 	Document n
> 		code, title, content, type, language, date, ...
>
>   Now all document types share the same fields, but in a future we
> need to add more document types with specific fields. I allways sort
> documents by date. I have 200.000 new documents each day and 130
> million documents. The janaury index size is 4.2Gb (the data size is
> about 10Gb).
>
>   I was wondering how to index the new document types.
> 	Option 1: One index for each document type. Each index will have
> its
> fields.
> 		Problem: I will have to perfom a search for each index,
> and sort
> results by date.
> 	Option 2: One big index containing all fields. A field could be
>
> empty if the field is not applicable for that document type.
> 	Option 3: One big index containing all common fields, and adding
> and
> extra field named metadata. Inside this field I will add all specific
> fields (field1:value1 field2:value2).
>
> Comments, pros and contras will be appreciated. I don't know exactly
> the diference between option 2 and option 3.
>
> Thanks
>
> Albert
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

RE: Multiple fields vs one field

Posted by "Chhabra, Kapil" <kc...@akamai.com>.
Hey Albert,
Just to remind you, that the fields in Lucene are per document and not
per index. This means that you can have documents in an index which have
different fields altogether.
So, in effect, you can all your document types to your existing index.
And guess what, you don't need to change anything for this.

All the best.

Regards,
kapilChhabra

-----Original Message-----
From: Albert Vila [mailto:avp@imente.com] 
Sent: Monday, August 06, 2007 5:42 PM
To: java-user@lucene.apache.org
Subject: Multiple fields vs one field

Hi all

  My data looks like:

	Document 1
		code, title, content, type, language, date, ...
	Document 2
		code, title, content, type, language, date, ...
	...
	Document n
		code, title, content, type, language, date, ...

  Now all document types share the same fields, but in a future we  
need to add more document types with specific fields. I allways sort  
documents by date. I have 200.000 new documents each day and 130  
million documents. The janaury index size is 4.2Gb (the data size is  
about 10Gb).

  I was wondering how to index the new document types.
	Option 1: One index for each document type. Each index will have
its  
fields.
		Problem: I will have to perfom a search for each index,
and sort  
results by date.
	Option 2: One big index containing all fields. A field could be

empty if the field is not applicable for that document type.
	Option 3: One big index containing all common fields, and adding
and  
extra field named metadata. Inside this field I will add all specific  
fields (field1:value1 field2:value2).

Comments, pros and contras will be appreciated. I don't know exactly  
the diference between option 2 and option 3.

Thanks

Albert

  

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org