Is it OK to have very big number of fields in solr/lucene ?

[Lisheng Zhang <lz...@gmail.com>]

In our application there are many complicated filter conditions, very often
those conditions are special to each user (like whether or not a doc is
important or already read by a user ..), two possible solutions to
implement those filters in lucene:

1/ create many fields
2/ create many collections (for each user, for example)

Here the number could be as big as 10G. I would prefer the 1st solution
(many fields), but if lucene cache all existing field info, memory could be
a problem ?

Thanks very much for helps, Lisheng

Re: Is it OK to have very big number of fields in solr/lucene ?

[Jack Krupansky <ja...@basetechnology.com>]

Solr scales based on number of documents, not fields or collections. Dozens 
of fields or collections is perfectly fine. Hundreds of fields or 
collections CAN work, but you have to be extra diligent and use more 
powerful hardware. Millions and even billions of DOCUMENTS is fine - that's 
the primary way that Solr scales (that and shards.)

Dynamic fields are fine too, but dozens or hundreds for a single document 
are recommended limits. Different documents can have different dynamic 
fields, so the total field count could be thousands, although, again, you 
may have to be extra diligent and use more powerful hardware.

Architect your application and model your data around the strengths of Solr 
(and Lucene.) And also look at your queries first, to make sure they will 
make sense.

-- Jack Krupansky

-----Original Message----- 
From: Lisheng Zhang
Sent: Friday, August 8, 2014 5:25 PM
To: solr-user@lucene.apache.org
Subject: Is it OK to have very big number of fields in solr/lucene ?

In our application there are many complicated filter conditions, very often
those conditions are special to each user (like whether or not a doc is
important or already read by a user ..), two possible solutions to
implement those filters in lucene:

1/ create many fields
2/ create many collections (for each user, for example)

Here the number could be as big as 10G. I would prefer the 1st solution
(many fields), but if lucene cache all existing field info, memory could be
a problem ?

Thanks very much for helps, Lisheng 

Re: Is it OK to have very big number of fields in solr/lucene ?

[Toke Eskildsen <te...@statsbiblioteket.dk>]

On Mon, 2014-08-11 at 09:44 +0200, Lisheng Zhang wrote:

[...]

> But tough part is that we may need to sort files according to such flag,
> for each user (I should have mentioned in last mail). My solution is to add
> many fields to file document, like
> 
> user1_status, user2_status, user3_status // value can be important or
> unimportant
> 
> then I can sort files according to each user's accessment.

Flip assessment and user name:

status:important_user1
status:unimportant_user2
status:customflag_user1
status:anothercustomflag_user2

That way sorting will work. Unfortunately there will be no re-use of the
filter cache between users, so you might want to disable filter caching.

If you also need to be able to filter on which documents a user has
assessed, then add a second field for that

assessed_by:user1
assessed_by:user2

> My concern is that with too many fields it would not scale well ?

I have successfully experimented with 10.000 fields, but I doubt that it
will work with millions.

- Toke Eskildsen, State and University Library, Denmark

Re: Is it OK to have very big number of fields in solr/lucene ?

[Lisheng Zhang <lz...@gmail.com>]

Thanks for helps!

The use case is that there is a file to which different users have
different accessment, if only for filtering I can create one field, like
status_field, with value like

user1_important user2_important user3_unimportant ...

then using a filter to get all files which is important to user1, like
status_field:user1_important

But tough part is that we may need to sort files according to such flag,
for each user (I should have mentioned in last mail). My solution is to add
many fields to file document, like

user1_status, user2_status, user3_status // value can be important or
unimportant

then I can sort files according to each user's accessment.

My concern is that with too many fields it would not scale well ?

Best regards, Lisheng




On Sat, Aug 9, 2014 at 6:18 AM, Toke Eskildsen <te...@statsbiblioteket.dk>
wrote:

> Lisheng Zhang [lz0522y2k@gmail.com] wrote:
> > In our application there are many complicated filter conditions, very
> often
> > those conditions are special to each user (like whether or not a doc is
> > important or already read by a user ..), two possible solutions to
> > implement those filters in lucene:
>
> > 1/ create many fields
> > 2/ create many collections (for each user, for example)
>
> 3) Define a few fields or even just a single field where users can provide
> their special tags.
>
> misc_field:important
> misc_field:read
> misc_field:personal_letters
> misc_field:todo
> misc_field:very_important
>
> Chances are that there will be a lot of overlap of terms between the users
> (if not, please describe in more detail what the user-specific things are),
> so that the filter caches can be re-used between them.
>
> - Toke Eskildsen
>

RE: Is it OK to have very big number of fields in solr/lucene ?

[Toke Eskildsen <te...@statsbiblioteket.dk>]

Lisheng Zhang [lz0522y2k@gmail.com] wrote:
> In our application there are many complicated filter conditions, very often
> those conditions are special to each user (like whether or not a doc is
> important or already read by a user ..), two possible solutions to
> implement those filters in lucene:

> 1/ create many fields
> 2/ create many collections (for each user, for example)

3) Define a few fields or even just a single field where users can provide their special tags.

misc_field:important
misc_field:read
misc_field:personal_letters
misc_field:todo
misc_field:very_important

Chances are that there will be a lot of overlap of terms between the users (if not, please describe in more detail what the user-specific things are), so that the filter caches can be re-used between them.

- Toke Eskildsen