You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by David Hastings <ha...@gmail.com> on 2017/03/31 19:55:17 UTC
Phrase Fields performance
Hey all.
I ran into an issue recently. I have a rather large index and in my
application I had defined values for the "pf" parameter, but I had
commented it out years ago not really knowing why I did it. Obviously the
point of the pf helps rank fields higher if they are in close proximity,
and this is good if you do a text search but want a boost on the "title"
for example.
So I un-commented out the line, to enable it to go against 6 important
fields. Afterwards through monitoring performance I noticed that my
searches were taking roughly 50% to 100% (2x!) longer, and it started at
the exact time I committed that change, 1:40 pm, qtimes below in a 15
minute average cycle with the start time listed.
What im assuming is happening behind the scenes is solr is doing an
auxiliary search on the defined fields as
field1:"word word word"~n+x
field2:"word word word"~n+x
field3:"word word word"~n+x
where n is the number of words and x is whatever arbitrary number.
obviously this adds a LOT to the search on the back end if I understand
the functionality correctly.
is there any recommendation to get roughly my old speeds back without a
full re-index to a new "metadata" type field BUT still keep the features of
the pf parameter? Also using a qf parameter.
set up:
solr 5.x
400gb+ index
16mm docs, 24mm with deletes.
2TB SSD, 24 cores, 148gb ram, 50gb heap size, about 30,000 searches a day.
Thanks!
-Dave
no pf:
| 2017-03-29 13:00:03 | 355.1961 |
| 2017-03-29 13:15:05 | 277.5479 |
| 2017-03-29 13:30:03 | 235.2263 |
| 2017-03-29 13:45:01 | 398.5000 |
| 2017-03-29 14:00:00 | 371.3048 |
| 2017-03-29 14:15:05 | 373.9901 |
| 2017-03-29 14:30:00 | 401.2474 |
| 2017-03-29 14:45:01 | 358.0344 |
| 2017-03-29 15:00:00 | 404.0234 |
changed pf at 140pm:
| 2017-03-30 13:00:00 | 270.4377 |
| 2017-03-30 13:15:00 | 397.9880 |
| 2017-03-30 13:30:00 | 474.8985 |
| 2017-03-30 13:45:02 | 1082.0114 |
| 2017-03-30 14:00:03 | 821.8152 |
| 2017-03-30 14:15:04 | 1034.8388 |
| 2017-03-30 14:30:00 | 1405.7792 |
| 2017-03-30 14:45:01 | 1200.1488 |
| 2017-03-30 15:00:00 | 694.4559 |
Re: Phrase Fields performance
Posted by Erick Erickson <er...@gmail.com>.
bq: ...and reducing the boost values to much smaller numbers...
not sure why that would matter for performance, muliplying is
multiplying, although reducing the boost on the default field might
have added up to a _lot_ of math ops.
Or is the boosting just a way to change the ranking to something you
can live with and not really a comment on performance?
I suspect the big difference is reducing the number of fields in "qf"
and my guess would be that the two fields omitted are larger text
fields.
FWIW,
Erick
On Tue, Apr 4, 2017 at 8:36 AM, David Hastings
<ha...@gmail.com> wrote:
> FYI, think i managed to get the results back and the speeds that i desired
> back reducing the number of fields in the qf/pf values from 6 to 4, also
> making sure to not boost the default field, and reducing the boost values
> to much smaller numbers but still significant enough to boost properly, so
> went from around .3 seconds pre qf/pf, above 1 sec after agressive
> settings, and now back down to around half a second with modified values,
> which I can live with. also if anyone else like myself stores qtimes in a
> table this is a good 15 minute rolling average sql query you may or may not
> find useful:
>
>
> SELECT when_done as timestamp, AVG( qtime ), count(id) FROM qtimes WHERE
> `when_done` >= '2017-03-23 09:00:00' AND `when_done` <= '2017-03-23
> 13:00:00' GROUP BY year(when_done),month(when_done),day(when_done),( 4 *
> HOUR( when_done ) + FLOOR( MINUTE( when_done ) / 15 )) ORDER BY
> `qtimes`.`when_done` ASC;
>
>
>
>
>
> pre qf/pf values:
> | timestamp | AVG( qtime ) | count(id) |
> +---------------------+--------------+-----------+
> | 2017-03-23 09:00:00 | 322.0585 | 581 |
> | 2017-03-23 09:15:01 | 243.9634 | 628 |
> | 2017-03-23 09:30:00 | 347.1856 | 652 |
> | 2017-03-23 09:45:03 | 407.3195 | 673 |
> | 2017-03-23 10:00:02 | 307.1313 | 678 |
> | 2017-03-23 10:15:00 | 266.9802 | 759 |
> | 2017-03-23 10:30:01 | 288.1789 | 833 |
> | 2017-03-23 10:45:01 | 275.0880 | 852 |
> | 2017-03-23 11:00:02 | 417.0151 | 861 |
> | 2017-03-23 11:15:01 | 267.1153 | 945 |
> | 2017-03-23 11:30:00 | 387.1656 | 803 |
> | 2017-03-23 11:45:00 | 268.5137 | 837 |
> | 2017-03-23 12:00:00 | 294.5911 | 807 |
> | 2017-03-23 12:15:00 | 411.8617 | 752 |
> | 2017-03-23 12:30:00 | 478.3566 | 788 |
> | 2017-03-23 12:45:01 | 262.2294 | 680 |
>
>
>
> after pf/qf values but too agressive:
>
> | timestamp | AVG( qtime ) | count(id) |
> +---------------------+--------------+-----------+
> | 2017-04-03 09:00:04 | 1002.1900 | 600 |
> | 2017-04-03 09:15:04 | 873.2367 | 659 |
> | 2017-04-03 09:30:00 | 1013.9041 | 563 |
> | 2017-04-03 09:45:01 | 1256.8596 | 591 |
> | 2017-04-03 10:00:08 | 1092.8582 | 663 |
> | 2017-04-03 10:15:00 | 1322.4262 | 671 |
> | 2017-04-03 10:30:06 | 848.1130 | 770 |
> | 2017-04-03 10:45:00 | 1039.3202 | 887 |
> | 2017-04-03 11:00:00 | 1144.9216 | 536 |
> | 2017-04-03 11:15:02 | 620.8999 | 719 |
> | 2017-04-03 11:30:03 | 999.7113 | 665 |
> | 2017-04-03 11:45:00 | 1144.1348 | 564 |
> | 2017-04-03 12:00:01 | 1317.7461 | 453 |
> | 2017-04-03 12:15:02 | 1413.5864 | 573 |
> | 2017-04-03 12:30:02 | 746.9422 | 623 |
> | 2017-04-03 12:45:00 | 1088.4789 | 568 |
>
>
> and finally modified pf/qf values changed at exactly 1046 am today:
>
>
> +---------------------+--------------+-----------+
> | timestamp | AVG( qtime ) | count(id) |
> +---------------------+--------------+-----------+
> | 2017-04-04 09:00:00 | 1079.3983 | 605 |
> | 2017-04-04 09:15:04 | 1190.4540 | 544 |
> | 2017-04-04 09:30:00 | 1459.6425 | 621 |
> | 2017-04-04 09:45:00 | 2074.2777 | 677 |
> | 2017-04-04 10:00:01 | 1555.0798 | 664 |
> | 2017-04-04 10:15:00 | 1313.1793 | 697 |
> | 2017-04-04 10:30:00 | 1042.4969 | 809 |
> | 2017-04-04 10:45:00 | 773.2043 | 695 |
> | 2017-04-04 11:00:00 | 526.7830 | 788 |
> | 2017-04-04 11:15:01 | 470.1969 | 711 |
> | 2017-04-04 11:30:02 | 642.1838 | 136 |
>
>
>
>
> On Sat, Apr 1, 2017 at 11:13 AM, Dave <ha...@gmail.com> wrote:
>
>> Maybe commongrams could help this but it boils down to
>> speed/quality/cheap. Choose two. Thanks
>>
>> > On Apr 1, 2017, at 10:28 AM, Shawn Heisey <ap...@elyograg.org> wrote:
>> >
>> >> On 3/31/2017 1:55 PM, David Hastings wrote:
>> >> So I un-commented out the line, to enable it to go against 6 important
>> >> fields. Afterwards through monitoring performance I noticed that my
>> >> searches were taking roughly 50% to 100% (2x!) longer, and it started
>> >> at the exact time I committed that change, 1:40 pm, qtimes below in a
>> >> 15 minute average cycle with the start time listed.
>> >
>> > That is fully expected. Using both pf and qf basically has Solr doing
>> > the exact same queries twice, once as specified on fields in qf, then
>> > again as a phrase query on fields in pf. If you add pf2 and/or pf3, you
>> > can expect further speed drops.
>> >
>> > If you're sorting by relevancy, using pf with higher boosts than qf
>> > generally will make your results better, but it comes at a cost in
>> > performance.
>> >
>> > Thanks,
>> > Shawn
>> >
>>
Re: Phrase Fields performance
Posted by David Hastings <ha...@gmail.com>.
FYI, think i managed to get the results back and the speeds that i desired
back reducing the number of fields in the qf/pf values from 6 to 4, also
making sure to not boost the default field, and reducing the boost values
to much smaller numbers but still significant enough to boost properly, so
went from around .3 seconds pre qf/pf, above 1 sec after agressive
settings, and now back down to around half a second with modified values,
which I can live with. also if anyone else like myself stores qtimes in a
table this is a good 15 minute rolling average sql query you may or may not
find useful:
SELECT when_done as timestamp, AVG( qtime ), count(id) FROM qtimes WHERE
`when_done` >= '2017-03-23 09:00:00' AND `when_done` <= '2017-03-23
13:00:00' GROUP BY year(when_done),month(when_done),day(when_done),( 4 *
HOUR( when_done ) + FLOOR( MINUTE( when_done ) / 15 )) ORDER BY
`qtimes`.`when_done` ASC;
pre qf/pf values:
| timestamp | AVG( qtime ) | count(id) |
+---------------------+--------------+-----------+
| 2017-03-23 09:00:00 | 322.0585 | 581 |
| 2017-03-23 09:15:01 | 243.9634 | 628 |
| 2017-03-23 09:30:00 | 347.1856 | 652 |
| 2017-03-23 09:45:03 | 407.3195 | 673 |
| 2017-03-23 10:00:02 | 307.1313 | 678 |
| 2017-03-23 10:15:00 | 266.9802 | 759 |
| 2017-03-23 10:30:01 | 288.1789 | 833 |
| 2017-03-23 10:45:01 | 275.0880 | 852 |
| 2017-03-23 11:00:02 | 417.0151 | 861 |
| 2017-03-23 11:15:01 | 267.1153 | 945 |
| 2017-03-23 11:30:00 | 387.1656 | 803 |
| 2017-03-23 11:45:00 | 268.5137 | 837 |
| 2017-03-23 12:00:00 | 294.5911 | 807 |
| 2017-03-23 12:15:00 | 411.8617 | 752 |
| 2017-03-23 12:30:00 | 478.3566 | 788 |
| 2017-03-23 12:45:01 | 262.2294 | 680 |
after pf/qf values but too agressive:
| timestamp | AVG( qtime ) | count(id) |
+---------------------+--------------+-----------+
| 2017-04-03 09:00:04 | 1002.1900 | 600 |
| 2017-04-03 09:15:04 | 873.2367 | 659 |
| 2017-04-03 09:30:00 | 1013.9041 | 563 |
| 2017-04-03 09:45:01 | 1256.8596 | 591 |
| 2017-04-03 10:00:08 | 1092.8582 | 663 |
| 2017-04-03 10:15:00 | 1322.4262 | 671 |
| 2017-04-03 10:30:06 | 848.1130 | 770 |
| 2017-04-03 10:45:00 | 1039.3202 | 887 |
| 2017-04-03 11:00:00 | 1144.9216 | 536 |
| 2017-04-03 11:15:02 | 620.8999 | 719 |
| 2017-04-03 11:30:03 | 999.7113 | 665 |
| 2017-04-03 11:45:00 | 1144.1348 | 564 |
| 2017-04-03 12:00:01 | 1317.7461 | 453 |
| 2017-04-03 12:15:02 | 1413.5864 | 573 |
| 2017-04-03 12:30:02 | 746.9422 | 623 |
| 2017-04-03 12:45:00 | 1088.4789 | 568 |
and finally modified pf/qf values changed at exactly 1046 am today:
+---------------------+--------------+-----------+
| timestamp | AVG( qtime ) | count(id) |
+---------------------+--------------+-----------+
| 2017-04-04 09:00:00 | 1079.3983 | 605 |
| 2017-04-04 09:15:04 | 1190.4540 | 544 |
| 2017-04-04 09:30:00 | 1459.6425 | 621 |
| 2017-04-04 09:45:00 | 2074.2777 | 677 |
| 2017-04-04 10:00:01 | 1555.0798 | 664 |
| 2017-04-04 10:15:00 | 1313.1793 | 697 |
| 2017-04-04 10:30:00 | 1042.4969 | 809 |
| 2017-04-04 10:45:00 | 773.2043 | 695 |
| 2017-04-04 11:00:00 | 526.7830 | 788 |
| 2017-04-04 11:15:01 | 470.1969 | 711 |
| 2017-04-04 11:30:02 | 642.1838 | 136 |
On Sat, Apr 1, 2017 at 11:13 AM, Dave <ha...@gmail.com> wrote:
> Maybe commongrams could help this but it boils down to
> speed/quality/cheap. Choose two. Thanks
>
> > On Apr 1, 2017, at 10:28 AM, Shawn Heisey <ap...@elyograg.org> wrote:
> >
> >> On 3/31/2017 1:55 PM, David Hastings wrote:
> >> So I un-commented out the line, to enable it to go against 6 important
> >> fields. Afterwards through monitoring performance I noticed that my
> >> searches were taking roughly 50% to 100% (2x!) longer, and it started
> >> at the exact time I committed that change, 1:40 pm, qtimes below in a
> >> 15 minute average cycle with the start time listed.
> >
> > That is fully expected. Using both pf and qf basically has Solr doing
> > the exact same queries twice, once as specified on fields in qf, then
> > again as a phrase query on fields in pf. If you add pf2 and/or pf3, you
> > can expect further speed drops.
> >
> > If you're sorting by relevancy, using pf with higher boosts than qf
> > generally will make your results better, but it comes at a cost in
> > performance.
> >
> > Thanks,
> > Shawn
> >
>
Re: Phrase Fields performance
Posted by Dave <ha...@gmail.com>.
Maybe commongrams could help this but it boils down to speed/quality/cheap. Choose two. Thanks
> On Apr 1, 2017, at 10:28 AM, Shawn Heisey <ap...@elyograg.org> wrote:
>
>> On 3/31/2017 1:55 PM, David Hastings wrote:
>> So I un-commented out the line, to enable it to go against 6 important
>> fields. Afterwards through monitoring performance I noticed that my
>> searches were taking roughly 50% to 100% (2x!) longer, and it started
>> at the exact time I committed that change, 1:40 pm, qtimes below in a
>> 15 minute average cycle with the start time listed.
>
> That is fully expected. Using both pf and qf basically has Solr doing
> the exact same queries twice, once as specified on fields in qf, then
> again as a phrase query on fields in pf. If you add pf2 and/or pf3, you
> can expect further speed drops.
>
> If you're sorting by relevancy, using pf with higher boosts than qf
> generally will make your results better, but it comes at a cost in
> performance.
>
> Thanks,
> Shawn
>
Re: Phrase Fields performance
Posted by Shawn Heisey <ap...@elyograg.org>.
On 3/31/2017 1:55 PM, David Hastings wrote:
> So I un-commented out the line, to enable it to go against 6 important
> fields. Afterwards through monitoring performance I noticed that my
> searches were taking roughly 50% to 100% (2x!) longer, and it started
> at the exact time I committed that change, 1:40 pm, qtimes below in a
> 15 minute average cycle with the start time listed.
That is fully expected. Using both pf and qf basically has Solr doing
the exact same queries twice, once as specified on fields in qf, then
again as a phrase query on fields in pf. If you add pf2 and/or pf3, you
can expect further speed drops.
If you're sorting by relevancy, using pf with higher boosts than qf
generally will make your results better, but it comes at a cost in
performance.
Thanks,
Shawn