You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Rok Rejc <ro...@gmail.com> on 2011/01/20 21:44:17 UTC
Document level security
Hi all,
I have an index containing a couple of million documents.
Documents are grouped into "groups", each group contains from 1000-20000
documents.
The problem:
Each group has defined permission settings. It can be viewed by public,
viewed by registred users, or viewed by a list of users (each group has her
own list of users).
Said differently: I need a document security.
What I read from the other threads it is not recommended to store
permissions in the index. I have already all the permissions in the
database, but I don't "know" how to connect the database and the index.
I can query the database to get the groups in which the user is and after
that do the OR query, but I am afraid that this list can be too big (100
OR's could also exceeds maximum HTTP GET query string length).
What are the other options? Should I write a custom collector which will
query (and cache) the database for permissions?
Any ideas are appreciated...
Many thanks, Rok
Re: Document level security
Posted by Dennis Gearon <ge...@sbcglobal.net>.
Would you do that with 1000's of users? How expensive in processor time is it?
Have you ever benchmarked it?
Dennis Gearon
Signature Warning
----------------
It is always a good idea to learn from your own mistakes. It is usually a better
idea to learn from others’ mistakes, so you do not have to make them yourself.
from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'
EARTH has a Right To Life,
otherwise we all die.
----- Original Message ----
From: Grijesh <pi...@gmail.com>
To: solr-user@lucene.apache.org
Sent: Thu, January 20, 2011 11:05:33 PM
Subject: Re: Document level security
Hi Rok,
I have used about 250000 ids with OR Operator and its working fine for
me.Just Have to Increase the MaxBoolClouse parameter and also have to
configure max header size on Servlet container to enable for big query
requests.
-----
Thanx:
Grijesh
--
View this message in context:
http://lucene.472066.n3.nabble.com/Document-level-security-tp2298066p2300117.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Document level security
Posted by Lance Norskog <go...@gmail.com>.
Most security systems don't authenticate by user, they authenticate by
roles. Each user has one or more roles. In one large enterprise there
are 7000 roles (including printer access), each user may have 300, and
each document may have 20-50.
It works well to add roles as a multivalued field in the document,
then search for the roles that a user presents. A document that has
any role presented by the user will be visible to the user. It's a
little slow, but faster than the above approaches.
Some side problems: spellchecking & autosuggestions should not show
terms that are not in a document available to the user. If a secret
document is named 'Project Xanadu', you can't autocomplete 'project
xanadu' to the user and then say, "no documents found for this
search".
On Thu, Jan 20, 2011 at 11:05 PM, Grijesh <pi...@gmail.com> wrote:
>
> Hi Rok,
>
> I have used about 250000 ids with OR Operator and its working fine for
> me.Just Have to Increase the MaxBoolClouse parameter and also have to
> configure max header size on Servlet container to enable for big query
> requests.
>
> -----
> Thanx:
> Grijesh
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Document-level-security-tp2298066p2300117.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
--
Lance Norskog
goksron@gmail.com
Re: Document level security
Posted by Grijesh <pi...@gmail.com>.
Hi Rok,
I have used about 250000 ids with OR Operator and its working fine for
me.Just Have to Increase the MaxBoolClouse parameter and also have to
configure max header size on Servlet container to enable for big query
requests.
-----
Thanx:
Grijesh
--
View this message in context: http://lucene.472066.n3.nabble.com/Document-level-security-tp2298066p2300117.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Document level security
Posted by Dennis Gearon <ge...@sbcglobal.net>.
I'm not sure how you COULD do searching without having the permissions in the
documents. I mentally use the model of unix filesystems, as a starter. Simple,
but powerful. If I needed a separate table for permissions, or index, I'd have
to do queries, with GINORMOUS amounts of OR statements.
I see it flowing like:
User U
Has Access to Documents DS (40,000,000 out of 100,000,000 of them),
Now get these (list of 40x10^06) documents.
How do you see it Peter?
Dennis Gearon
----- Original Message ----
From: Peter Sturge <pe...@gmail.com>
To: solr-user@lucene.apache.org
Sent: Thu, January 20, 2011 3:16:59 PM
Subject: Re: Document level security
Hi,
One of the things about Document Security is that it never involves
just one thing. There are a lot of things to consider, and
unfortunately, they're generally non-trivial.
Deciding how to store/hold/retrieve permissions is certainly one of
those things, and you're right, you should avoid attaching permissions
to document data in the index, because if you want to change
permissions (and you will want to change them at some point), it can
be a cumbersome job, particularly if it involves millions of
documents, replication, shards etc. It's also generally a good idea
not to tie your schema to permission fields.
Another big consideration is authentication - how can you be sure the
request is coming from the user you think it is? Is there a
certificate involved? Has the user authenticated to the container? If
so, how do you get to this? and so on...
For permissions storage, there are two realistic approaches to consider:
1. Write a SearchComponent that handles permission requests. This
typically involves storing/reading permissions in/from a file,
database or separate index (see SOLR-1872)
2. Use an LCF module to retrieve permissions from the original
documents themselves (see SOLR-1834)
Hope this helps,
Peter
On Thu, Jan 20, 2011 at 8:44 PM, Rok Rejc <ro...@gmail.com> wrote:
> Hi all,
>
> I have an index containing a couple of million documents.
> Documents are grouped into "groups", each group contains from 1000-20000
> documents.
>
> The problem:
> Each group has defined permission settings. It can be viewed by public,
> viewed by registred users, or viewed by a list of users (each group has her
> own list of users).
> Said differently: I need a document security.
>
> What I read from the other threads it is not recommended to store
> permissions in the index. I have already all the permissions in the
> database, but I don't "know" how to connect the database and the index.
> I can query the database to get the groups in which the user is and after
> that do the OR query, but I am afraid that this list can be too big (100
> OR's could also exceeds maximum HTTP GET query string length).
>
> What are the other options? Should I write a custom collector which will
> query (and cache) the database for permissions?
>
> Any ideas are appreciated...
>
> Many thanks, Rok
>
Re: Document level security
Posted by Peter Sturge <pe...@gmail.com>.
Hi,
One of the things about Document Security is that it never involves
just one thing. There are a lot of things to consider, and
unfortunately, they're generally non-trivial.
Deciding how to store/hold/retrieve permissions is certainly one of
those things, and you're right, you should avoid attaching permissions
to document data in the index, because if you want to change
permissions (and you will want to change them at some point), it can
be a cumbersome job, particularly if it involves millions of
documents, replication, shards etc. It's also generally a good idea
not to tie your schema to permission fields.
Another big consideration is authentication - how can you be sure the
request is coming from the user you think it is? Is there a
certificate involved? Has the user authenticated to the container? If
so, how do you get to this? and so on...
For permissions storage, there are two realistic approaches to consider:
1. Write a SearchComponent that handles permission requests. This
typically involves storing/reading permissions in/from a file,
database or separate index (see SOLR-1872)
2. Use an LCF module to retrieve permissions from the original
documents themselves (see SOLR-1834)
Hope this helps,
Peter
On Thu, Jan 20, 2011 at 8:44 PM, Rok Rejc <ro...@gmail.com> wrote:
> Hi all,
>
> I have an index containing a couple of million documents.
> Documents are grouped into "groups", each group contains from 1000-20000
> documents.
>
> The problem:
> Each group has defined permission settings. It can be viewed by public,
> viewed by registred users, or viewed by a list of users (each group has her
> own list of users).
> Said differently: I need a document security.
>
> What I read from the other threads it is not recommended to store
> permissions in the index. I have already all the permissions in the
> database, but I don't "know" how to connect the database and the index.
> I can query the database to get the groups in which the user is and after
> that do the OR query, but I am afraid that this list can be too big (100
> OR's could also exceeds maximum HTTP GET query string length).
>
> What are the other options? Should I write a custom collector which will
> query (and cache) the database for permissions?
>
> Any ideas are appreciated...
>
> Many thanks, Rok
>