You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by William Bell <bi...@gmail.com> on 2013/06/21 07:48:10 UTC
Varnish
Who is using varnish in front of SOLR?
Anyone have any configs that work with the cache control headers of SOLR?
--
Bill Bell
billnbell@gmail.com
cell 720-256-8076
Re: Varnish
Posted by Lance Norskog <go...@gmail.com>.
Solr HTTP caching also support "e-tags". These are unique keys for the
output of a query. If you send a query twice, and the index has not
changed, the return will be the same. The e-tag is generated from the
query string and the index generation number.
If Varnish supports e-tags, you can keep some queries cached longer than
your timeout.
Lance
On 06/29/2013 05:51 PM, William Bell wrote:
> On a large website, by putting 1 varnish in front of all 4 SOLR boxes we
> were able to trim 25% off the load time (TTFB) of the page.
>
> Our hit ratio was between 55 and 75%. We gave varnish 24GB of RAM, and was
> not able to fill it under full load with a 10 minute cache timeout.
>
> We get about 2.4M SOLR calls every 15 to 20 minutes.
>
> One varnish was able to handle it with almost no lingering connections, and
> load average of < 1.
>
> Varnish is very optimized and worth trying.
>
>
>
> On Sat, Jun 29, 2013 at 6:47 PM, William Bell <bi...@gmail.com> wrote:
>
>> OK.
>>
>> Here is the answer for us. Here is a sample default.vcl. We are validating
>> the LastModified ( if (!beresp.http.last-modified) )
>> is changing when the core is indexed and the version changes of the index.
>>
>> This does 10 minutes caching and a 1hr grace period (if solr is down, it
>> will deliver results up to 1 hr).
>>
>> This uses the URL for caching.
>>
>> You can also do:
>>
>> http://localhost?PURGEME
>>
>> To clear varnish if your IP is in the ACL list.
>>
>>
>> backend server1 {
>> .host = "XXX.domain.com";
>> .port = "8983";
>> .probe = {
>> .url = "/solr/pingall/select/?q=*%3A*";
>> .interval = 5s;
>> .timeout = 1s;
>> .window = 5;
>> .threshold = 3;
>> }
>> }
>> backend server2{
>> .host = "XXX1.domain.com";
>> .port = "8983";
>> .probe = {
>> .url = "/solr/pingall/select/?q=*%3A*";
>> .interval = 5s;
>> .timeout = 1s;
>> .window = 5;
>> .threshold = 3;
>> }
>> }
>> backend server3{
>> .host = "XXX2.domain.com";
>> .port = "8983";
>> .probe = {
>> .url = "/solr/pingall/select/?q=*%3A*";
>> .interval = 5s;
>> .timeout = 1s;
>> .window = 5;
>> .threshold = 3;
>> }
>> }
>> backend server4{
>> .host = "XXX3.domain.com";
>> .port = "8983";
>> .probe = {
>> .url = "/solr/pingall/select/?q=*%3A*";
>> .interval = 5s;
>> .timeout = 1s;
>> .window = 5;
>> .threshold = 3;
>> }
>> }
>>
>> director default round-robin {
>> {
>> .backend = server1;
>> }
>> {
>> .backend = server2;
>> }
>> {
>> .backend = server3;
>> }
>> {
>> .backend = server4;
>> }
>> }
>>
>> acl purge {
>> "localhost";
>> "10.0.1.0"/24;
>> "10.0.3.0"/24;
>> }
>>
>>
>> sub vcl_recv {
>> if (req.url ~ "\?PURGEME$") {
>> if (!client.ip ~ purge) {
>> error 405 "Not allowed. " + client.ip;
>> }
>> ban("req.url ~ /");
>> error 200 "Cached Cleared";
>> }
>> remove req.http.Cookie;
>> if (req.backend.healthy) {
>> set req.grace = 15s;
>> } else {
>> set req.grace = 1h;
>> }
>> return (lookup);
>> }
>>
>> sub vcl_fetch {
>> set beresp.grace = 1h;
>> if (!beresp.http.last-modified) {
>> set beresp.ttl = 600s;
>> }
>> if (beresp.ttl < 600s) {
>> set beresp.ttl = 600s;
>> }
>> unset beresp.http.Set-Cookie;
>> }
>>
>> sub vcl_deliver {
>> if (obj.hits > 0) {
>> set resp.http.X-Cache = "HIT";
>> } else {
>> set resp.http.X-Cache = "MISS";
>> }
>> }
>>
>> sub vcl_hash {
>> hash_data(req.url);
>> return (hash);
>> }
>>
>>
>>
>>
>>
>>
>> On Tue, Jun 25, 2013 at 4:44 PM, Learner <bb...@gmail.com> wrote:
>>
>>> Check this link..
>>> http://lucene.472066.n3.nabble.com/SolrJ-HTTP-caching-td490063.html
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://lucene.472066.n3.nabble.com/Varnish-tp4072057p4073205.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>
>>
>> --
>> Bill Bell
>> billnbell@gmail.com
>> cell 720-256-8076
>>
>
>
Re: Varnish
Posted by William Bell <bi...@gmail.com>.
On a large website, by putting 1 varnish in front of all 4 SOLR boxes we
were able to trim 25% off the load time (TTFB) of the page.
Our hit ratio was between 55 and 75%. We gave varnish 24GB of RAM, and was
not able to fill it under full load with a 10 minute cache timeout.
We get about 2.4M SOLR calls every 15 to 20 minutes.
One varnish was able to handle it with almost no lingering connections, and
load average of < 1.
Varnish is very optimized and worth trying.
On Sat, Jun 29, 2013 at 6:47 PM, William Bell <bi...@gmail.com> wrote:
> OK.
>
> Here is the answer for us. Here is a sample default.vcl. We are validating
> the LastModified ( if (!beresp.http.last-modified) )
> is changing when the core is indexed and the version changes of the index.
>
> This does 10 minutes caching and a 1hr grace period (if solr is down, it
> will deliver results up to 1 hr).
>
> This uses the URL for caching.
>
> You can also do:
>
> http://localhost?PURGEME
>
> To clear varnish if your IP is in the ACL list.
>
>
> backend server1 {
> .host = "XXX.domain.com";
> .port = "8983";
> .probe = {
> .url = "/solr/pingall/select/?q=*%3A*";
> .interval = 5s;
> .timeout = 1s;
> .window = 5;
> .threshold = 3;
> }
> }
> backend server2{
> .host = "XXX1.domain.com";
> .port = "8983";
> .probe = {
> .url = "/solr/pingall/select/?q=*%3A*";
> .interval = 5s;
> .timeout = 1s;
> .window = 5;
> .threshold = 3;
> }
> }
> backend server3{
> .host = "XXX2.domain.com";
> .port = "8983";
> .probe = {
> .url = "/solr/pingall/select/?q=*%3A*";
> .interval = 5s;
> .timeout = 1s;
> .window = 5;
> .threshold = 3;
> }
> }
> backend server4{
> .host = "XXX3.domain.com";
> .port = "8983";
> .probe = {
> .url = "/solr/pingall/select/?q=*%3A*";
> .interval = 5s;
> .timeout = 1s;
> .window = 5;
> .threshold = 3;
> }
> }
>
> director default round-robin {
> {
> .backend = server1;
> }
> {
> .backend = server2;
> }
> {
> .backend = server3;
> }
> {
> .backend = server4;
> }
> }
>
> acl purge {
> "localhost";
> "10.0.1.0"/24;
> "10.0.3.0"/24;
> }
>
>
> sub vcl_recv {
> if (req.url ~ "\?PURGEME$") {
> if (!client.ip ~ purge) {
> error 405 "Not allowed. " + client.ip;
> }
> ban("req.url ~ /");
> error 200 "Cached Cleared";
> }
> remove req.http.Cookie;
> if (req.backend.healthy) {
> set req.grace = 15s;
> } else {
> set req.grace = 1h;
> }
> return (lookup);
> }
>
> sub vcl_fetch {
> set beresp.grace = 1h;
> if (!beresp.http.last-modified) {
> set beresp.ttl = 600s;
> }
> if (beresp.ttl < 600s) {
> set beresp.ttl = 600s;
> }
> unset beresp.http.Set-Cookie;
> }
>
> sub vcl_deliver {
> if (obj.hits > 0) {
> set resp.http.X-Cache = "HIT";
> } else {
> set resp.http.X-Cache = "MISS";
> }
> }
>
> sub vcl_hash {
> hash_data(req.url);
> return (hash);
> }
>
>
>
>
>
>
> On Tue, Jun 25, 2013 at 4:44 PM, Learner <bb...@gmail.com> wrote:
>
>> Check this link..
>> http://lucene.472066.n3.nabble.com/SolrJ-HTTP-caching-td490063.html
>>
>>
>>
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/Varnish-tp4072057p4073205.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>
>
>
> --
> Bill Bell
> billnbell@gmail.com
> cell 720-256-8076
>
--
Bill Bell
billnbell@gmail.com
cell 720-256-8076
Re: Varnish
Posted by William Bell <bi...@gmail.com>.
OK.
Here is the answer for us. Here is a sample default.vcl. We are validating
the LastModified ( if (!beresp.http.last-modified) )
is changing when the core is indexed and the version changes of the index.
This does 10 minutes caching and a 1hr grace period (if solr is down, it
will deliver results up to 1 hr).
This uses the URL for caching.
You can also do:
http://localhost?PURGEME
To clear varnish if your IP is in the ACL list.
backend server1 {
.host = "XXX.domain.com";
.port = "8983";
.probe = {
.url = "/solr/pingall/select/?q=*%3A*";
.interval = 5s;
.timeout = 1s;
.window = 5;
.threshold = 3;
}
}
backend server2{
.host = "XXX1.domain.com";
.port = "8983";
.probe = {
.url = "/solr/pingall/select/?q=*%3A*";
.interval = 5s;
.timeout = 1s;
.window = 5;
.threshold = 3;
}
}
backend server3{
.host = "XXX2.domain.com";
.port = "8983";
.probe = {
.url = "/solr/pingall/select/?q=*%3A*";
.interval = 5s;
.timeout = 1s;
.window = 5;
.threshold = 3;
}
}
backend server4{
.host = "XXX3.domain.com";
.port = "8983";
.probe = {
.url = "/solr/pingall/select/?q=*%3A*";
.interval = 5s;
.timeout = 1s;
.window = 5;
.threshold = 3;
}
}
director default round-robin {
{
.backend = server1;
}
{
.backend = server2;
}
{
.backend = server3;
}
{
.backend = server4;
}
}
acl purge {
"localhost";
"10.0.1.0"/24;
"10.0.3.0"/24;
}
sub vcl_recv {
if (req.url ~ "\?PURGEME$") {
if (!client.ip ~ purge) {
error 405 "Not allowed. " + client.ip;
}
ban("req.url ~ /");
error 200 "Cached Cleared";
}
remove req.http.Cookie;
if (req.backend.healthy) {
set req.grace = 15s;
} else {
set req.grace = 1h;
}
return (lookup);
}
sub vcl_fetch {
set beresp.grace = 1h;
if (!beresp.http.last-modified) {
set beresp.ttl = 600s;
}
if (beresp.ttl < 600s) {
set beresp.ttl = 600s;
}
unset beresp.http.Set-Cookie;
}
sub vcl_deliver {
if (obj.hits > 0) {
set resp.http.X-Cache = "HIT";
} else {
set resp.http.X-Cache = "MISS";
}
}
sub vcl_hash {
hash_data(req.url);
return (hash);
}
On Tue, Jun 25, 2013 at 4:44 PM, Learner <bb...@gmail.com> wrote:
> Check this link..
> http://lucene.472066.n3.nabble.com/SolrJ-HTTP-caching-td490063.html
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Varnish-tp4072057p4073205.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
--
Bill Bell
billnbell@gmail.com
cell 720-256-8076
Re: Varnish
Posted by Learner <bb...@gmail.com>.
Check this link..
http://lucene.472066.n3.nabble.com/SolrJ-HTTP-caching-td490063.html
--
View this message in context: http://lucene.472066.n3.nabble.com/Varnish-tp4072057p4073205.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Varnish
Posted by William Bell <bi...@gmail.com>.
Yes. A lot of people are using it in front of SOLR to speed up and reduce
the Garbase Collections.
On Fri, Jun 21, 2013 at 9:44 AM, Jack Park <ja...@topicquests.org> wrote:
> I presume you mean https://www.varnish-cache.org/
> That's the first I'd heard of it.
>
> Thanks
> Jack
>
> On Thu, Jun 20, 2013 at 10:48 PM, William Bell <bi...@gmail.com>
> wrote:
> > Who is using varnish in front of SOLR?
> >
> > Anyone have any configs that work with the cache control headers of SOLR?
> >
> > --
> > Bill Bell
> > billnbell@gmail.com
> > cell 720-256-8076
>
--
Bill Bell
billnbell@gmail.com
cell 720-256-8076
Re: Varnish
Posted by Jack Park <ja...@topicquests.org>.
I presume you mean https://www.varnish-cache.org/
That's the first I'd heard of it.
Thanks
Jack
On Thu, Jun 20, 2013 at 10:48 PM, William Bell <bi...@gmail.com> wrote:
> Who is using varnish in front of SOLR?
>
> Anyone have any configs that work with the cache control headers of SOLR?
>
> --
> Bill Bell
> billnbell@gmail.com
> cell 720-256-8076