You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Lee Carroll <le...@googlemail.com> on 2013/01/09 12:08:03 UTC
performance improvements on ip look up query
Hi
We are doing a lat/lon look up query using ip address.
We have a 6.5 million document core of the following structure
start ip block
end ip block
location id
location_lat_lon
the field defs are
<types>
<fieldType name="string" class="solr.StrField" sortMissingLast="true"
omitNorms="true"/>
<fieldType name="tlong" class="solr.TrieLongField" precisionStep="8"
omitNorms="true" positionIncrementGap="0"/>
<fieldType name="tfloat" class="solr.TrieFloatField" precisionStep="8"
omitNorms="true" positionIncrementGap="0"/>
<fieldType name="tdouble" class="solr.TrieDoubleField" precisionStep="8"
omitNorms="true" positionIncrementGap="0"/>
<fieldType name="location" class="solr.LatLonType" subFieldSuffix="
_coordinate"/>
</types>
<fields>
<field name="startIp" type="string" indexed="true" stored="false" required="
true"/>
<field name="startIpNum" type="tlong" indexed="true" stored="false" required
="true"/>
<field name="endIpNum" type="tlong" indexed="true" stored="false" required="
true"/>
<field name="locId" type="string" indexed="true" stored="true" required="
true"/>
<field name="countryCode" type="string" indexed="true" stored="true"
required="false"/>
<field name="cityName" type="string" indexed="false" stored="true" required
="false"/>
<field name="latLon" type="location" indexed="true" stored="true" required="
true"/>
<field name="latitude" type="string" indexed="false" stored="true" required
="true"/>
<field name="longitude" type="string" indexed="false" stored="true" required
="true"/>
<dynamicField name="*_coordinate" type="tdouble" indexed="true" stored="
false"/>
</fields>
the query at the moment is simply a range query
q=startIpNum:[* TO 180891652]AND endIpNum:[180891652 TO *]
we are seeing a full query cache with a low hit rate 0.2 and a high
eviction rate which makes sense given the use of ip address in the query.
query time mean is 120.
Is their a better way of structuring the core for this usecase ?
I suspect our heap memory settings are conservative 1g but will need to
convince our sys admins to change this (they are not ringing any resource
alarm bells) just the query is a little slow
Re: performance improvements on ip look up query
Posted by Lee Carroll <le...@googlemail.com>.
Hi Otis
The cache was modest 4096 with a hit rate of 0.23 after a 24hr period.
We doubled it and the hit rate went to 0.25. Our interpretation is ip is
pretty much a cache busting value ? and
cache size is not at play here.
the q param is just startIpNum:[* TO 180891652]AND endIpNum:[180891652 TO
*] so again our
interpretation is its got little reuse
Could we re-formulate the query to be more per-formant ?
On 9 January 2013 12:56, Otis Gospodnetic <ot...@gmail.com>wrote:
> Hi,
>
> Maybe your cache is too small? How big is it and does the hit rate change
> if you make it bigger?
>
> Do any parts of the query repeat a lot? Maybe there is room for fq.
>
> Otis
> Solr & ElasticSearch Support
> http://sematext.com/
> On Jan 9, 2013 6:08 AM, "Lee Carroll" <le...@googlemail.com>
> wrote:
>
> > Hi
> >
> > We are doing a lat/lon look up query using ip address.
> >
> > We have a 6.5 million document core of the following structure
> > start ip block
> > end ip block
> > location id
> > location_lat_lon
> >
> > the field defs are
> > <types>
> > <fieldType name="string" class="solr.StrField" sortMissingLast="true"
> > omitNorms="true"/>
> > <fieldType name="tlong" class="solr.TrieLongField" precisionStep="8"
> > omitNorms="true" positionIncrementGap="0"/>
> > <fieldType name="tfloat" class="solr.TrieFloatField" precisionStep="8"
> > omitNorms="true" positionIncrementGap="0"/>
> > <fieldType name="tdouble" class="solr.TrieDoubleField" precisionStep="8"
> > omitNorms="true" positionIncrementGap="0"/>
> > <fieldType name="location" class="solr.LatLonType" subFieldSuffix="
> > _coordinate"/>
> > </types>
> > <fields>
> > <field name="startIp" type="string" indexed="true" stored="false"
> > required="
> > true"/>
> > <field name="startIpNum" type="tlong" indexed="true" stored="false"
> > required
> > ="true"/>
> > <field name="endIpNum" type="tlong" indexed="true" stored="false"
> > required="
> > true"/>
> > <field name="locId" type="string" indexed="true" stored="true" required="
> > true"/>
> > <field name="countryCode" type="string" indexed="true" stored="true"
> > required="false"/>
> > <field name="cityName" type="string" indexed="false" stored="true"
> required
> > ="false"/>
> > <field name="latLon" type="location" indexed="true" stored="true"
> > required="
> > true"/>
> > <field name="latitude" type="string" indexed="false" stored="true"
> required
> > ="true"/>
> > <field name="longitude" type="string" indexed="false" stored="true"
> > required
> > ="true"/>
> > <dynamicField name="*_coordinate" type="tdouble" indexed="true" stored="
> > false"/>
> > </fields>
> >
> > the query at the moment is simply a range query
> >
> > q=startIpNum:[* TO 180891652]AND endIpNum:[180891652 TO *]
> >
> > we are seeing a full query cache with a low hit rate 0.2 and a high
> > eviction rate which makes sense given the use of ip address in the query.
> >
> > query time mean is 120.
> >
> > Is their a better way of structuring the core for this usecase ?
> > I suspect our heap memory settings are conservative 1g but will need to
> > convince our sys admins to change this (they are not ringing any resource
> > alarm bells) just the query is a little slow
> >
>
Re: performance improvements on ip look up query
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi,
Maybe your cache is too small? How big is it and does the hit rate change
if you make it bigger?
Do any parts of the query repeat a lot? Maybe there is room for fq.
Otis
Solr & ElasticSearch Support
http://sematext.com/
On Jan 9, 2013 6:08 AM, "Lee Carroll" <le...@googlemail.com> wrote:
> Hi
>
> We are doing a lat/lon look up query using ip address.
>
> We have a 6.5 million document core of the following structure
> start ip block
> end ip block
> location id
> location_lat_lon
>
> the field defs are
> <types>
> <fieldType name="string" class="solr.StrField" sortMissingLast="true"
> omitNorms="true"/>
> <fieldType name="tlong" class="solr.TrieLongField" precisionStep="8"
> omitNorms="true" positionIncrementGap="0"/>
> <fieldType name="tfloat" class="solr.TrieFloatField" precisionStep="8"
> omitNorms="true" positionIncrementGap="0"/>
> <fieldType name="tdouble" class="solr.TrieDoubleField" precisionStep="8"
> omitNorms="true" positionIncrementGap="0"/>
> <fieldType name="location" class="solr.LatLonType" subFieldSuffix="
> _coordinate"/>
> </types>
> <fields>
> <field name="startIp" type="string" indexed="true" stored="false"
> required="
> true"/>
> <field name="startIpNum" type="tlong" indexed="true" stored="false"
> required
> ="true"/>
> <field name="endIpNum" type="tlong" indexed="true" stored="false"
> required="
> true"/>
> <field name="locId" type="string" indexed="true" stored="true" required="
> true"/>
> <field name="countryCode" type="string" indexed="true" stored="true"
> required="false"/>
> <field name="cityName" type="string" indexed="false" stored="true" required
> ="false"/>
> <field name="latLon" type="location" indexed="true" stored="true"
> required="
> true"/>
> <field name="latitude" type="string" indexed="false" stored="true" required
> ="true"/>
> <field name="longitude" type="string" indexed="false" stored="true"
> required
> ="true"/>
> <dynamicField name="*_coordinate" type="tdouble" indexed="true" stored="
> false"/>
> </fields>
>
> the query at the moment is simply a range query
>
> q=startIpNum:[* TO 180891652]AND endIpNum:[180891652 TO *]
>
> we are seeing a full query cache with a low hit rate 0.2 and a high
> eviction rate which makes sense given the use of ip address in the query.
>
> query time mean is 120.
>
> Is their a better way of structuring the core for this usecase ?
> I suspect our heap memory settings are conservative 1g but will need to
> convince our sys admins to change this (they are not ringing any resource
> alarm bells) just the query is a little slow
>