Disable caching

[Jamie Johnson <je...@gmail.com>]

I see that if Solr is in realtime mode that caching is disable within the
SolrIndexSearcher that is created in SolrCore, but is there anyway to
disable caching without being in realtime mode?  Currently I'm implementing
a NoOp cache that implements SolrCache but returns null for everything and
doesn't return anything on the get requests, but it would be nice to not
need to do this by being able to disable caching in general.  Is this
possible?

-Jamie

Re: Disable caching

[Jamie Johnson <je...@gmail.com>]

I ran into another issue that I am having issue running to ground.  My
implementation on Solr 4.x worked as I expected but trying to migrate this
to Solr 5.x it looks like some of the faceting is delegated to
DocValuesFacets which ultimately caches things at a field level in the
FieldCache.DEFAULT cache.  I don't see anyway to override this cache or
augment the key in anyway, am I missing an extension point here?  Is there
another approach I should be taking in this case?

On Wed, Aug 19, 2015 at 9:08 AM, Jamie Johnson <je...@gmail.com> wrote:

> This was my original thought.  We already have the thread local so should
> be straight fwd to just wrap the Field name and use that as the key.  Again
> thanks, I really appreciate the feedback
> On Aug 19, 2015 8:12 AM, "Yonik Seeley" <ys...@gmail.com> wrote:
>
>> On Tue, Aug 18, 2015 at 10:58 PM, Jamie Johnson <je...@gmail.com>
>> wrote:
>> > Hmm...so I think I have things setup correctly, I have a custom
>> > QParserPlugin building a custom query that wraps the query built from
>> the
>> > base parser and stores the user who is executing the query.  I've added
>> the
>> > username to the hashCode and equals checks so I think everything is
>> setup
>> > properly.  I ran a quick test and it definitely looks like my items are
>> > being cached now per user, which is really great.
>> >
>> > The outage that I'm running into now is the FieldValueCache doesn't take
>> > into account the query, so the FieldValueCache is built for user a and
>> then
>> > reused for user b, which is an issue for me.  In short I'm back to my
>> > NoOpCache for FieldValues.  It's great that I'm in a better spot for the
>> > others, but is there anything that can be done with FieldValues to take
>> > into account the requesting user?
>>
>> I guess a cache implementation that gets the user through a thread
>> local and either wraps the original key with an object containing the
>> user, or delegates to a per-user cache underneath.
>>
>> -Yonik
>>
>

Re: Disable caching

[Jamie Johnson <je...@gmail.com>]

This was my original thought.  We already have the thread local so should
be straight fwd to just wrap the Field name and use that as the key.  Again
thanks, I really appreciate the feedback
On Aug 19, 2015 8:12 AM, "Yonik Seeley" <ys...@gmail.com> wrote:

> On Tue, Aug 18, 2015 at 10:58 PM, Jamie Johnson <je...@gmail.com> wrote:
> > Hmm...so I think I have things setup correctly, I have a custom
> > QParserPlugin building a custom query that wraps the query built from the
> > base parser and stores the user who is executing the query.  I've added
> the
> > username to the hashCode and equals checks so I think everything is setup
> > properly.  I ran a quick test and it definitely looks like my items are
> > being cached now per user, which is really great.
> >
> > The outage that I'm running into now is the FieldValueCache doesn't take
> > into account the query, so the FieldValueCache is built for user a and
> then
> > reused for user b, which is an issue for me.  In short I'm back to my
> > NoOpCache for FieldValues.  It's great that I'm in a better spot for the
> > others, but is there anything that can be done with FieldValues to take
> > into account the requesting user?
>
> I guess a cache implementation that gets the user through a thread
> local and either wraps the original key with an object containing the
> user, or delegates to a per-user cache underneath.
>
> -Yonik
>

Re: Disable caching

[Yonik Seeley <ys...@gmail.com>]

On Tue, Aug 18, 2015 at 10:58 PM, Jamie Johnson <je...@gmail.com> wrote:
> Hmm...so I think I have things setup correctly, I have a custom
> QParserPlugin building a custom query that wraps the query built from the
> base parser and stores the user who is executing the query.  I've added the
> username to the hashCode and equals checks so I think everything is setup
> properly.  I ran a quick test and it definitely looks like my items are
> being cached now per user, which is really great.
>
> The outage that I'm running into now is the FieldValueCache doesn't take
> into account the query, so the FieldValueCache is built for user a and then
> reused for user b, which is an issue for me.  In short I'm back to my
> NoOpCache for FieldValues.  It's great that I'm in a better spot for the
> others, but is there anything that can be done with FieldValues to take
> into account the requesting user?

I guess a cache implementation that gets the user through a thread
local and either wraps the original key with an object containing the
user, or delegates to a per-user cache underneath.

-Yonik

Re: Disable caching

[Jamie Johnson <je...@gmail.com>]

Hmm...so I think I have things setup correctly, I have a custom
QParserPlugin building a custom query that wraps the query built from the
base parser and stores the user who is executing the query.  I've added the
username to the hashCode and equals checks so I think everything is setup
properly.  I ran a quick test and it definitely looks like my items are
being cached now per user, which is really great.

The outage that I'm running into now is the FieldValueCache doesn't take
into account the query, so the FieldValueCache is built for user a and then
reused for user b, which is an issue for me.  In short I'm back to my
NoOpCache for FieldValues.  It's great that I'm in a better spot for the
others, but is there anything that can be done with FieldValues to take
into account the requesting user?

On Tue, Aug 18, 2015 at 9:59 PM, Yonik Seeley <ys...@gmail.com> wrote:

> On Tue, Aug 18, 2015 at 9:51 PM, Jamie Johnson <je...@gmail.com> wrote:
> > Thanks, I'll try to delve into this.  We are currently using the parent
> > query parser, within we could use {!secure} I think.  Ultimately I would
> > want the solr qparser to actually do the work of parsing and I'd just
> wrap
> > that.
>
> Right... look at something like BoostQParserPlugin
> it should be trivial to wrap any other type of query.
>
> baseParser = subQuery(localParams.get(QueryParsing.V), null);
> Query q = baseParser.getQuery();
>
> q={!secure}my_normal_query
> OR
> q={!secure v=$qq)&qq=my_normal_query
> OR
> q={!secure}{!parent ...}
> OR
> q={!secure v=$qq}&qq={!parent. ..}
>
> -Yonik
>
>
> >  Are there any examples that I could look at for this?  It's not
> > clear to me what to do in the qparser once I have the user auths though.
> > Again thanks, this is really good stuff.
> > On Aug 18, 2015 8:54 PM, "Yonik Seeley" <ys...@gmail.com> wrote:
> >
> >> On Tue, Aug 18, 2015 at 8:38 PM, Jamie Johnson <je...@gmail.com>
> wrote:
> >> > I really like this idea in concept.  My query would literally be just
> a
> >> > wrapper at that point, what would be the appropriate place to do this?
> >>
> >> It depends on how much you are trying to make everything transparent
> >> (that there is security) or not.
> >>
> >> First approach is explicitly changing the query types (you obviously
> >> need to make sure that only trusted code can run queries against solr
> >> for this method):
> >> q=foo:bar&fq=inStock:true
> >> q={!secure id=user}foo:bar&fq={!secure id=user}inStock:true
> >>   you could even make the {!secure} qparser look for global security
> >> params so you don't need to repeat them.
> >> q={!secure}foo:bar&fq={!secure}inStock:true&security_id=user
> >>
> >> Second approach would prob involve a search component, probably that
> >> runs after the query component, that would handle wrapping any queries
> >> or filters in the prepare() phase.  This would be slightly more
> >> difficult since it would require ensuring that none of the solr code /
> >> features you use re-grab the "q" or "fq" parameters re-parse without
> >> the opportunity for you to wrap them again.
> >>
> >> > What would I need to do to the query to make it behave with the cache.
> >>
> >> Probably not much... record the credentials in the wrapper and use in
> >> the hashCode / equals.
> >>
> >> -Yonik
> >>
> >>
> >> > Again thanks for the idea, I think this could be a simple way to use
> the
> >> > caches.
> >> >
> >> > On Tue, Aug 18, 2015 at 8:31 PM, Yonik Seeley <ys...@gmail.com>
> wrote:
> >> >
> >> >> On Tue, Aug 18, 2015 at 8:19 PM, Jamie Johnson <je...@gmail.com>
> >> wrote:
> >> >> > when you say a security filter, are you asking if I can express my
> >> >> security
> >> >> > constraint as a query?  If that is the case then the answer is
> no.  At
> >> >> this
> >> >> > point I have a requirement to secure Terms (a nightmare I know).
> >> >>
> >> >> Heh - ok, I figured as much.
> >> >>
> >> >> So... you could also wrap the main query and any filter queries in a
> >> >> custom security query that would contain the user, and thus still be
> >> >> able to use filter and query caches unmodified. I know... that's only
> >> >> a small part of the problem though.
> >> >>
> >> >> -Yonik
> >> >>
> >>
>

Re: Disable caching

[Yonik Seeley <ys...@gmail.com>]

On Tue, Aug 18, 2015 at 9:51 PM, Jamie Johnson <je...@gmail.com> wrote:
> Thanks, I'll try to delve into this.  We are currently using the parent
> query parser, within we could use {!secure} I think.  Ultimately I would
> want the solr qparser to actually do the work of parsing and I'd just wrap
> that.

Right... look at something like BoostQParserPlugin
it should be trivial to wrap any other type of query.

baseParser = subQuery(localParams.get(QueryParsing.V), null);
Query q = baseParser.getQuery();

q={!secure}my_normal_query
OR
q={!secure v=$qq)&qq=my_normal_query
OR
q={!secure}{!parent ...}
OR
q={!secure v=$qq}&qq={!parent. ..}

-Yonik


>  Are there any examples that I could look at for this?  It's not
> clear to me what to do in the qparser once I have the user auths though.
> Again thanks, this is really good stuff.
> On Aug 18, 2015 8:54 PM, "Yonik Seeley" <ys...@gmail.com> wrote:
>
>> On Tue, Aug 18, 2015 at 8:38 PM, Jamie Johnson <je...@gmail.com> wrote:
>> > I really like this idea in concept.  My query would literally be just a
>> > wrapper at that point, what would be the appropriate place to do this?
>>
>> It depends on how much you are trying to make everything transparent
>> (that there is security) or not.
>>
>> First approach is explicitly changing the query types (you obviously
>> need to make sure that only trusted code can run queries against solr
>> for this method):
>> q=foo:bar&fq=inStock:true
>> q={!secure id=user}foo:bar&fq={!secure id=user}inStock:true
>>   you could even make the {!secure} qparser look for global security
>> params so you don't need to repeat them.
>> q={!secure}foo:bar&fq={!secure}inStock:true&security_id=user
>>
>> Second approach would prob involve a search component, probably that
>> runs after the query component, that would handle wrapping any queries
>> or filters in the prepare() phase.  This would be slightly more
>> difficult since it would require ensuring that none of the solr code /
>> features you use re-grab the "q" or "fq" parameters re-parse without
>> the opportunity for you to wrap them again.
>>
>> > What would I need to do to the query to make it behave with the cache.
>>
>> Probably not much... record the credentials in the wrapper and use in
>> the hashCode / equals.
>>
>> -Yonik
>>
>>
>> > Again thanks for the idea, I think this could be a simple way to use the
>> > caches.
>> >
>> > On Tue, Aug 18, 2015 at 8:31 PM, Yonik Seeley <ys...@gmail.com> wrote:
>> >
>> >> On Tue, Aug 18, 2015 at 8:19 PM, Jamie Johnson <je...@gmail.com>
>> wrote:
>> >> > when you say a security filter, are you asking if I can express my
>> >> security
>> >> > constraint as a query?  If that is the case then the answer is no.  At
>> >> this
>> >> > point I have a requirement to secure Terms (a nightmare I know).
>> >>
>> >> Heh - ok, I figured as much.
>> >>
>> >> So... you could also wrap the main query and any filter queries in a
>> >> custom security query that would contain the user, and thus still be
>> >> able to use filter and query caches unmodified. I know... that's only
>> >> a small part of the problem though.
>> >>
>> >> -Yonik
>> >>
>>

Re: Disable caching

[Jamie Johnson <je...@gmail.com>]

Thanks, I'll try to delve into this.  We are currently using the parent
query parser, within we could use {!secure} I think.  Ultimately I would
want the solr qparser to actually do the work of parsing and I'd just wrap
that.  Are there any examples that I could look at for this?  It's not
clear to me what to do in the qparser once I have the user auths though.
Again thanks, this is really good stuff.
On Aug 18, 2015 8:54 PM, "Yonik Seeley" <ys...@gmail.com> wrote:

> On Tue, Aug 18, 2015 at 8:38 PM, Jamie Johnson <je...@gmail.com> wrote:
> > I really like this idea in concept.  My query would literally be just a
> > wrapper at that point, what would be the appropriate place to do this?
>
> It depends on how much you are trying to make everything transparent
> (that there is security) or not.
>
> First approach is explicitly changing the query types (you obviously
> need to make sure that only trusted code can run queries against solr
> for this method):
> q=foo:bar&fq=inStock:true
> q={!secure id=user}foo:bar&fq={!secure id=user}inStock:true
>   you could even make the {!secure} qparser look for global security
> params so you don't need to repeat them.
> q={!secure}foo:bar&fq={!secure}inStock:true&security_id=user
>
> Second approach would prob involve a search component, probably that
> runs after the query component, that would handle wrapping any queries
> or filters in the prepare() phase.  This would be slightly more
> difficult since it would require ensuring that none of the solr code /
> features you use re-grab the "q" or "fq" parameters re-parse without
> the opportunity for you to wrap them again.
>
> > What would I need to do to the query to make it behave with the cache.
>
> Probably not much... record the credentials in the wrapper and use in
> the hashCode / equals.
>
> -Yonik
>
>
> > Again thanks for the idea, I think this could be a simple way to use the
> > caches.
> >
> > On Tue, Aug 18, 2015 at 8:31 PM, Yonik Seeley <ys...@gmail.com> wrote:
> >
> >> On Tue, Aug 18, 2015 at 8:19 PM, Jamie Johnson <je...@gmail.com>
> wrote:
> >> > when you say a security filter, are you asking if I can express my
> >> security
> >> > constraint as a query?  If that is the case then the answer is no.  At
> >> this
> >> > point I have a requirement to secure Terms (a nightmare I know).
> >>
> >> Heh - ok, I figured as much.
> >>
> >> So... you could also wrap the main query and any filter queries in a
> >> custom security query that would contain the user, and thus still be
> >> able to use filter and query caches unmodified. I know... that's only
> >> a small part of the problem though.
> >>
> >> -Yonik
> >>
>

Re: Disable caching

[Yonik Seeley <ys...@gmail.com>]

On Tue, Aug 18, 2015 at 8:38 PM, Jamie Johnson <je...@gmail.com> wrote:
> I really like this idea in concept.  My query would literally be just a
> wrapper at that point, what would be the appropriate place to do this?

It depends on how much you are trying to make everything transparent
(that there is security) or not.

First approach is explicitly changing the query types (you obviously
need to make sure that only trusted code can run queries against solr
for this method):
q=foo:bar&fq=inStock:true
q={!secure id=user}foo:bar&fq={!secure id=user}inStock:true
  you could even make the {!secure} qparser look for global security
params so you don't need to repeat them.
q={!secure}foo:bar&fq={!secure}inStock:true&security_id=user

Second approach would prob involve a search component, probably that
runs after the query component, that would handle wrapping any queries
or filters in the prepare() phase.  This would be slightly more
difficult since it would require ensuring that none of the solr code /
features you use re-grab the "q" or "fq" parameters re-parse without
the opportunity for you to wrap them again.

> What would I need to do to the query to make it behave with the cache.

Probably not much... record the credentials in the wrapper and use in
the hashCode / equals.

-Yonik


> Again thanks for the idea, I think this could be a simple way to use the
> caches.
>
> On Tue, Aug 18, 2015 at 8:31 PM, Yonik Seeley <ys...@gmail.com> wrote:
>
>> On Tue, Aug 18, 2015 at 8:19 PM, Jamie Johnson <je...@gmail.com> wrote:
>> > when you say a security filter, are you asking if I can express my
>> security
>> > constraint as a query?  If that is the case then the answer is no.  At
>> this
>> > point I have a requirement to secure Terms (a nightmare I know).
>>
>> Heh - ok, I figured as much.
>>
>> So... you could also wrap the main query and any filter queries in a
>> custom security query that would contain the user, and thus still be
>> able to use filter and query caches unmodified. I know... that's only
>> a small part of the problem though.
>>
>> -Yonik
>>

Re: Disable caching

[Jamie Johnson <je...@gmail.com>]

I really like this idea in concept.  My query would literally be just a
wrapper at that point, what would be the appropriate place to do this?
What would I need to do to the query to make it behave with the cache.

Again thanks for the idea, I think this could be a simple way to use the
caches.

On Tue, Aug 18, 2015 at 8:31 PM, Yonik Seeley <ys...@gmail.com> wrote:

> On Tue, Aug 18, 2015 at 8:19 PM, Jamie Johnson <je...@gmail.com> wrote:
> > when you say a security filter, are you asking if I can express my
> security
> > constraint as a query?  If that is the case then the answer is no.  At
> this
> > point I have a requirement to secure Terms (a nightmare I know).
>
> Heh - ok, I figured as much.
>
> So... you could also wrap the main query and any filter queries in a
> custom security query that would contain the user, and thus still be
> able to use filter and query caches unmodified. I know... that's only
> a small part of the problem though.
>
> -Yonik
>

Re: Disable caching

[Yonik Seeley <ys...@gmail.com>]

On Tue, Aug 18, 2015 at 8:19 PM, Jamie Johnson <je...@gmail.com> wrote:
> when you say a security filter, are you asking if I can express my security
> constraint as a query?  If that is the case then the answer is no.  At this
> point I have a requirement to secure Terms (a nightmare I know).

Heh - ok, I figured as much.

So... you could also wrap the main query and any filter queries in a
custom security query that would contain the user, and thus still be
able to use filter and query caches unmodified. I know... that's only
a small part of the problem though.

-Yonik

Re: Disable caching

[Jamie Johnson <je...@gmail.com>]

when you say a security filter, are you asking if I can express my security
constraint as a query?  If that is the case then the answer is no.  At this
point I have a requirement to secure Terms (a nightmare I know).  Our
fallback is to aggregate the authorizations to a document level and secure
the document which I think we wouldn't have to do anything to the caches
but our customer has pushed back on this in the past.

On Tue, Aug 18, 2015 at 7:46 PM, Yonik Seeley <ys...@gmail.com> wrote:

> On Tue, Aug 18, 2015 at 7:11 PM, Jamie Johnson <je...@gmail.com> wrote:
> > Yes, my use case is security.  Basically I am executing queries with
> > certain auths and when they are executed multiple times with differing
> > auths I'm getting cached results.
>
> If it's just simple stuff like top N docs returned, can't you just use
> a security filter?
>
> The queryResult cache uses both the main query and a list of filters
> (and the sort order) for the cache key.
>
> -Yonik
>

Re: Disable caching

[Yonik Seeley <ys...@gmail.com>]

On Tue, Aug 18, 2015 at 7:11 PM, Jamie Johnson <je...@gmail.com> wrote:
> Yes, my use case is security.  Basically I am executing queries with
> certain auths and when they are executed multiple times with differing
> auths I'm getting cached results.

If it's just simple stuff like top N docs returned, can't you just use
a security filter?

The queryResult cache uses both the main query and a list of filters
(and the sort order) for the cache key.

-Yonik

Re: Disable caching

[Jamie Johnson <je...@gmail.com>]

Yes, my use case is security.  Basically I am executing queries with
certain auths and when they are executed multiple times with differing
auths I'm getting cached results.  One option is to have another
implementation that has a number of caches based on the auths, something
that I suspect we will at some point go (unless there is a better solution
;).  I'd be happy to look at other options so all suggestions are
appreciated.

On Tue, Aug 18, 2015 at 6:56 PM, Yonik Seeley <ys...@gmail.com> wrote:

> You can comment out (some) of the caches.
>
> There are some caches like field caches that are more at the lucene
> level and can't be disabled.
>
> Can I ask what you are trying to prevent from being cached and why?
> Different caches are for different things, so it would seem to be an
> odd usecase to disable them all.  Security?
>
> -Yonik
>
>
> On Tue, Aug 18, 2015 at 6:52 PM, Jamie Johnson <je...@gmail.com> wrote:
> > I see that if Solr is in realtime mode that caching is disable within the
> > SolrIndexSearcher that is created in SolrCore, but is there anyway to
> > disable caching without being in realtime mode?  Currently I'm
> implementing
> > a NoOp cache that implements SolrCache but returns null for everything
> and
> > doesn't return anything on the get requests, but it would be nice to not
> > need to do this by being able to disable caching in general.  Is this
> > possible?
> >
> > -Jamie
>

Re: Disable caching

[Yonik Seeley <ys...@gmail.com>]

You can comment out (some) of the caches.

There are some caches like field caches that are more at the lucene
level and can't be disabled.

Can I ask what you are trying to prevent from being cached and why?
Different caches are for different things, so it would seem to be an
odd usecase to disable them all.  Security?

-Yonik


On Tue, Aug 18, 2015 at 6:52 PM, Jamie Johnson <je...@gmail.com> wrote:
> I see that if Solr is in realtime mode that caching is disable within the
> SolrIndexSearcher that is created in SolrCore, but is there anyway to
> disable caching without being in realtime mode?  Currently I'm implementing
> a NoOp cache that implements SolrCache but returns null for everything and
> doesn't return anything on the get requests, but it would be nice to not
> need to do this by being able to disable caching in general.  Is this
> possible?
>
> -Jamie