You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Joel Bernstein <jo...@gmail.com> on 2020/08/10 18:34:16 UTC
hybrid document routing
I have a situation where I'd like to have the standard compositeId router
in place for a collection. But, I'd like certain documents (ACL documents)
to be duplicated on each shard in the collection. To achieve the level of
access control performance and scalability I'm looking for I need the ACL
records to be in the same core as the main documents.
I put together a prototype where the compositeId router accepted implicit
routing parameters and it worked in my testing. Before I open a ticket
suggesting this approach I wonder what other people thought the best
approach would be to accomplish this goal.
Re: hybrid document routing
Posted by David Smiley <ds...@apache.org>.
Cool!
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
On Tue, Aug 11, 2020 at 9:55 AM Joel Bernstein <jo...@gmail.com> wrote:
> SOLR-14728 supports sub-second performance on joins with more than 1
> million values from the from index. Nice for access control.
>
>
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
>
> On Tue, Aug 11, 2020 at 9:49 AM Joel Bernstein <jo...@gmail.com> wrote:
>
>> This ticket will shed some light:
>>
>> https://issues.apache.org/jira/browse/SOLR-14728
>>
>>
>> I think I'm planning using a different approach to distribute tha ACL's
>> to all shards.
>>
>>
>>
>>
>> Joel Bernstein
>> http://joelsolr.blogspot.com/
>>
>>
>> On Tue, Aug 11, 2020 at 1:18 AM Gus Heck <gu...@gmail.com> wrote:
>>
>>> Sounds like complex ACLs based on group memberships that use graph
>>> queries ? that would require local ACL's...
>>>
>>> On Mon, Aug 10, 2020 at 5:56 PM Ishan Chattopadhyaya <
>>> ichattopadhyaya@gmail.com> wrote:
>>>
>>>> This seems like an XY problem. Would it be possible to describe the
>>>> original problem that led you to this solution (in the prototype)? Also, do
>>>> you think folks at solr-users@ list would have more ideas related to
>>>> this usecase and cross posting there would help?
>>>>
>>>> On Tue, 11 Aug, 2020, 1:43 am David Smiley, <ds...@apache.org> wrote:
>>>>
>>>>> Are you sure you need the docs in the same shard when maybe you could
>>>>> assume a core exists on each node and then do a query-time join?
>>>>>
>>>>> ~ David Smiley
>>>>> Apache Lucene/Solr Search Developer
>>>>> http://www.linkedin.com/in/davidwsmiley
>>>>>
>>>>>
>>>>> On Mon, Aug 10, 2020 at 2:34 PM Joel Bernstein <jo...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I have a situation where I'd like to have the standard compositeId
>>>>>> router in place for a collection. But, I'd like certain documents (ACL
>>>>>> documents) to be duplicated on each shard in the collection. To achieve the
>>>>>> level of access control performance and scalability I'm looking for I need
>>>>>> the ACL records to be in the same core as the main documents.
>>>>>>
>>>>>> I put together a prototype where the compositeId router accepted
>>>>>> implicit routing parameters and it worked in my testing. Before I open a
>>>>>> ticket suggesting this approach I wonder what other people thought the best
>>>>>> approach would be to accomplish this goal.
>>>>>>
>>>>>>
>>>>>>
>>>
>>> --
>>> http://www.needhamsoftware.com (work)
>>> http://www.the111shift.com (play)
>>>
>>
Re: hybrid document routing
Posted by Joel Bernstein <jo...@gmail.com>.
SOLR-14728 supports sub-second performance on joins with more than 1
million values from the from index. Nice for access control.
Joel Bernstein
http://joelsolr.blogspot.com/
On Tue, Aug 11, 2020 at 9:49 AM Joel Bernstein <jo...@gmail.com> wrote:
> This ticket will shed some light:
>
> https://issues.apache.org/jira/browse/SOLR-14728
>
>
> I think I'm planning using a different approach to distribute tha ACL's to
> all shards.
>
>
>
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
>
> On Tue, Aug 11, 2020 at 1:18 AM Gus Heck <gu...@gmail.com> wrote:
>
>> Sounds like complex ACLs based on group memberships that use graph
>> queries ? that would require local ACL's...
>>
>> On Mon, Aug 10, 2020 at 5:56 PM Ishan Chattopadhyaya <
>> ichattopadhyaya@gmail.com> wrote:
>>
>>> This seems like an XY problem. Would it be possible to describe the
>>> original problem that led you to this solution (in the prototype)? Also, do
>>> you think folks at solr-users@ list would have more ideas related to
>>> this usecase and cross posting there would help?
>>>
>>> On Tue, 11 Aug, 2020, 1:43 am David Smiley, <ds...@apache.org> wrote:
>>>
>>>> Are you sure you need the docs in the same shard when maybe you could
>>>> assume a core exists on each node and then do a query-time join?
>>>>
>>>> ~ David Smiley
>>>> Apache Lucene/Solr Search Developer
>>>> http://www.linkedin.com/in/davidwsmiley
>>>>
>>>>
>>>> On Mon, Aug 10, 2020 at 2:34 PM Joel Bernstein <jo...@gmail.com>
>>>> wrote:
>>>>
>>>>> I have a situation where I'd like to have the standard compositeId
>>>>> router in place for a collection. But, I'd like certain documents (ACL
>>>>> documents) to be duplicated on each shard in the collection. To achieve the
>>>>> level of access control performance and scalability I'm looking for I need
>>>>> the ACL records to be in the same core as the main documents.
>>>>>
>>>>> I put together a prototype where the compositeId router accepted
>>>>> implicit routing parameters and it worked in my testing. Before I open a
>>>>> ticket suggesting this approach I wonder what other people thought the best
>>>>> approach would be to accomplish this goal.
>>>>>
>>>>>
>>>>>
>>
>> --
>> http://www.needhamsoftware.com (work)
>> http://www.the111shift.com (play)
>>
>
Re: hybrid document routing
Posted by Joel Bernstein <jo...@gmail.com>.
This ticket will shed some light:
https://issues.apache.org/jira/browse/SOLR-14728
I think I'm planning using a different approach to distribute tha ACL's to
all shards.
Joel Bernstein
http://joelsolr.blogspot.com/
On Tue, Aug 11, 2020 at 1:18 AM Gus Heck <gu...@gmail.com> wrote:
> Sounds like complex ACLs based on group memberships that use graph queries
> ? that would require local ACL's...
>
> On Mon, Aug 10, 2020 at 5:56 PM Ishan Chattopadhyaya <
> ichattopadhyaya@gmail.com> wrote:
>
>> This seems like an XY problem. Would it be possible to describe the
>> original problem that led you to this solution (in the prototype)? Also, do
>> you think folks at solr-users@ list would have more ideas related to
>> this usecase and cross posting there would help?
>>
>> On Tue, 11 Aug, 2020, 1:43 am David Smiley, <ds...@apache.org> wrote:
>>
>>> Are you sure you need the docs in the same shard when maybe you could
>>> assume a core exists on each node and then do a query-time join?
>>>
>>> ~ David Smiley
>>> Apache Lucene/Solr Search Developer
>>> http://www.linkedin.com/in/davidwsmiley
>>>
>>>
>>> On Mon, Aug 10, 2020 at 2:34 PM Joel Bernstein <jo...@gmail.com>
>>> wrote:
>>>
>>>> I have a situation where I'd like to have the standard compositeId
>>>> router in place for a collection. But, I'd like certain documents (ACL
>>>> documents) to be duplicated on each shard in the collection. To achieve the
>>>> level of access control performance and scalability I'm looking for I need
>>>> the ACL records to be in the same core as the main documents.
>>>>
>>>> I put together a prototype where the compositeId router accepted
>>>> implicit routing parameters and it worked in my testing. Before I open a
>>>> ticket suggesting this approach I wonder what other people thought the best
>>>> approach would be to accomplish this goal.
>>>>
>>>>
>>>>
>
> --
> http://www.needhamsoftware.com (work)
> http://www.the111shift.com (play)
>
Re: hybrid document routing
Posted by Gus Heck <gu...@gmail.com>.
Sounds like complex ACLs based on group memberships that use graph queries
? that would require local ACL's...
On Mon, Aug 10, 2020 at 5:56 PM Ishan Chattopadhyaya <
ichattopadhyaya@gmail.com> wrote:
> This seems like an XY problem. Would it be possible to describe the
> original problem that led you to this solution (in the prototype)? Also, do
> you think folks at solr-users@ list would have more ideas related to this
> usecase and cross posting there would help?
>
> On Tue, 11 Aug, 2020, 1:43 am David Smiley, <ds...@apache.org> wrote:
>
>> Are you sure you need the docs in the same shard when maybe you could
>> assume a core exists on each node and then do a query-time join?
>>
>> ~ David Smiley
>> Apache Lucene/Solr Search Developer
>> http://www.linkedin.com/in/davidwsmiley
>>
>>
>> On Mon, Aug 10, 2020 at 2:34 PM Joel Bernstein <jo...@gmail.com>
>> wrote:
>>
>>> I have a situation where I'd like to have the standard compositeId
>>> router in place for a collection. But, I'd like certain documents (ACL
>>> documents) to be duplicated on each shard in the collection. To achieve the
>>> level of access control performance and scalability I'm looking for I need
>>> the ACL records to be in the same core as the main documents.
>>>
>>> I put together a prototype where the compositeId router accepted
>>> implicit routing parameters and it worked in my testing. Before I open a
>>> ticket suggesting this approach I wonder what other people thought the best
>>> approach would be to accomplish this goal.
>>>
>>>
>>>
--
http://www.needhamsoftware.com (work)
http://www.the111shift.com (play)
Re: hybrid document routing
Posted by Ishan Chattopadhyaya <ic...@gmail.com>.
This seems like an XY problem. Would it be possible to describe the
original problem that led you to this solution (in the prototype)? Also, do
you think folks at solr-users@ list would have more ideas related to this
usecase and cross posting there would help?
On Tue, 11 Aug, 2020, 1:43 am David Smiley, <ds...@apache.org> wrote:
> Are you sure you need the docs in the same shard when maybe you could
> assume a core exists on each node and then do a query-time join?
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Mon, Aug 10, 2020 at 2:34 PM Joel Bernstein <jo...@gmail.com> wrote:
>
>> I have a situation where I'd like to have the standard compositeId router
>> in place for a collection. But, I'd like certain documents (ACL documents)
>> to be duplicated on each shard in the collection. To achieve the level of
>> access control performance and scalability I'm looking for I need the ACL
>> records to be in the same core as the main documents.
>>
>> I put together a prototype where the compositeId router accepted implicit
>> routing parameters and it worked in my testing. Before I open a ticket
>> suggesting this approach I wonder what other people thought the best
>> approach would be to accomplish this goal.
>>
>>
>>
Re: hybrid document routing
Posted by David Smiley <ds...@apache.org>.
Are you sure you need the docs in the same shard when maybe you could
assume a core exists on each node and then do a query-time join?
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
On Mon, Aug 10, 2020 at 2:34 PM Joel Bernstein <jo...@gmail.com> wrote:
> I have a situation where I'd like to have the standard compositeId router
> in place for a collection. But, I'd like certain documents (ACL documents)
> to be duplicated on each shard in the collection. To achieve the level of
> access control performance and scalability I'm looking for I need the ACL
> records to be in the same core as the main documents.
>
> I put together a prototype where the compositeId router accepted implicit
> routing parameters and it worked in my testing. Before I open a ticket
> suggesting this approach I wonder what other people thought the best
> approach would be to accomplish this goal.
>
>
>