You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Michael Wechner <mi...@wyona.com> on 2008/11/10 20:52:54 UTC

performance/scalability issues re filtering of protected search results

Hi

We have about 1 mio documents and growing within a hierarchical order (3 
to 20 deep) and about 3000 people accessing these nodes, whereas some 
people have access to certain branches and other people to other 
branches and some branches are shared. The access control of these nodes 
is changing every day and also contains shortcuts  which allows people 
to glimpse into parts of branches which they otherwise do not have 
access to.

Currently we have one index for all nodes, which is ok 
peformance/scalability wise, but before displaying the results we need 
to filter based on the access privileges each user has, which is very 
bad peformance wise, because it might be that the first 10K hits are all 
protected re this user and hence it can take a very long time that one 
finally finds a result that the user is actually allowed to see.

We were thinking about introducing an index for each user which only 
contains the documents a user is actually is allowed to see, but this 
doesn't scale well either if the user number is growing.

Any hints how other people are approaching such a situation would be 
very much appreciated.

Thanks

Michael

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: performance/scalability issues re filtering of protected search results

Posted by Michael Wechner <mi...@wyona.com>.
Erick Erickson schrieb:
> This has been discussed more than a few times, I suggest you take
> a look at the searchable archive for things like privileges, access
> privileges, etc. You'll find lots of information faster that way...
>   
You mean Erik Hatcher's answer re SecurityFilter
http://archives.devshed.com/forums/apache-92/lucene-vs-sql-database-1416862.html

or Eugene Dzhurinsky's post where he is also asking re pre-filtering

http://mail-archives.apache.org/mod_mbox/lucene-java-user/200510.mbox/%3c20051005080100.GD457@jdevoff.zssm.zp.ua%3e
http://mail-archives.apache.org/mod_mbox/lucene-java-user/200510.mbox/%3C20051005113829.GA59420@jdevoff.zssm.zp.ua%3E
http://mail-archives.apache.org/mod_mbox/lucene-java-user/200510.mbox/%3c20051005125619.GC59420@jdevoff.zssm.zp.ua%3e

and Erik keeps suggesting to use a filter

http://mail-archives.apache.org/mod_mbox/lucene-java-user/200510.mbox/%3c20051005101406125.00000003632@huixp%3e

whereas Hui also points out that filters are a problem re 
peformance/scalability

http://mail-archives.apache.org/mod_mbox/lucene-java-user/200510.mbox/%3c20051005101406125.00000003632@huixp%3e

I actually searched quite a bit before my post, but didn't find an 
answer which was really answering the performance/scalability issues.
But if it was answered before and somebody knows about it, then I would 
very much appreciate any concrete URLs/pointers.

Thanks

Michael
> Best
> Erick
>
> On Mon, Nov 10, 2008 at 2:52 PM, Michael Wechner
> <mi...@wyona.com>wrote:
>
>   
>> Hi
>>
>> We have about 1 mio documents and growing within a hierarchical order (3 to
>> 20 deep) and about 3000 people accessing these nodes, whereas some people
>> have access to certain branches and other people to other branches and some
>> branches are shared. The access control of these nodes is changing every day
>> and also contains shortcuts  which allows people to glimpse into parts of
>> branches which they otherwise do not have access to.
>>
>> Currently we have one index for all nodes, which is ok
>> peformance/scalability wise, but before displaying the results we need to
>> filter based on the access privileges each user has, which is very bad
>> peformance wise, because it might be that the first 10K hits are all
>> protected re this user and hence it can take a very long time that one
>> finally finds a result that the user is actually allowed to see.
>>
>> We were thinking about introducing an index for each user which only
>> contains the documents a user is actually is allowed to see, but this
>> doesn't scale well either if the user number is growing.
>>
>> Any hints how other people are approaching such a situation would be very
>> much appreciated.
>>
>> Thanks
>>
>> Michael
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>     
>
>   


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: performance/scalability issues re filtering of protected search results

Posted by Erick Erickson <er...@gmail.com>.
This has been discussed more than a few times, I suggest you take
a look at the searchable archive for things like privileges, access
privileges, etc. You'll find lots of information faster that way...

Best
Erick

On Mon, Nov 10, 2008 at 2:52 PM, Michael Wechner
<mi...@wyona.com>wrote:

> Hi
>
> We have about 1 mio documents and growing within a hierarchical order (3 to
> 20 deep) and about 3000 people accessing these nodes, whereas some people
> have access to certain branches and other people to other branches and some
> branches are shared. The access control of these nodes is changing every day
> and also contains shortcuts  which allows people to glimpse into parts of
> branches which they otherwise do not have access to.
>
> Currently we have one index for all nodes, which is ok
> peformance/scalability wise, but before displaying the results we need to
> filter based on the access privileges each user has, which is very bad
> peformance wise, because it might be that the first 10K hits are all
> protected re this user and hence it can take a very long time that one
> finally finds a result that the user is actually allowed to see.
>
> We were thinking about introducing an index for each user which only
> contains the documents a user is actually is allowed to see, but this
> doesn't scale well either if the user number is growing.
>
> Any hints how other people are approaching such a situation would be very
> much appreciated.
>
> Thanks
>
> Michael
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>