You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Peter Keegan <pe...@gmail.com> on 2006/10/27 21:29:20 UTC

Announcement: Lucene powering Monster job search index (Beta)

I am pleased to announce the launch of Monster's new job search Beta web
site, powered by Lucene, at: http://jobsearch.beta.monster.com (notice the
Lucene logo at the bottom of the page!).

The jobs index is implemented with Java Lucene 2.0 on 64-bit Windows (AMD
and Intel processors)

Here are some of the new features:

1. 'Improve your search by'...

The job search results page allows you to browse and 'drill down' through
the results by job category, status, type and salary. The number of matching
jobs in each facet is displayed. There will likely be many more facets to
browse by in the future.

This feature is currently implemented with a custom HitCollector and the
DocSet class from Solr.

2. 'More like this'

Find more jobs like the one you see by clicking on the 'MORE LIKE THIS'
link, which is visible when you hover the mouse over the job title.

This feature is implemented with Lucene's term vectors and the
'MoreLikeThis' contribution class. If you are in 'detailed view', the term
vectors from the job description are used. In 'brief' view, the job title's
term vectors are used.

3. 'Related Titles'

When you do a 'keywords' search, click on a 'related titles' link to filter
you search by similar job titles.

This feature is implemented via a separate Lucene.Net index.

4. Sort by 'Miles'

Find jobs close to you via zip code/radius search. In the search results
page, click on the 'Miles' column to sort the results by distance from your
zip code/radius.

This custom sorting feature is implemented via Lucene's
'SortComparatorSource' interface.

5. Search by date, salary, distance.

Find jobs posted in the last day (or 2,3, etc) or by salary range or
distance.

Numeric range search is one of Lucene's weak points (performance-wise) so we
have implemented this with a custom HitCollector and an extension to the
Lucene index files that stores the numeric field values for all documents.

It is important to point out that this has all been implemented with the
stock Lucene 2.0 library. No code changes were made to the Lucene core.

If you have any feedback regarding the UI, please use the link on the web
page ("send us your feedback"). You can hit me with any other
questions/comments.

Peter

Re: Announcement: Lucene powering Monster job search index (Beta)

Posted by Peter Keegan <pe...@gmail.com>.
Note: this is a reply to a posting to java-dev  --Peter

Eric,

> Now that it is live, is performance pretty good?

Performance is outstanding. Each server can easily handle well over 100 qps
on an index of over 800K documents. There are several servers (4 dual core
(8 CPU) Opteron) supporting the query load and we have backup servers for
disaster recovery. For a few hours one day, all job search query traffic for
the entire site was being handled by a single server - with no noticable
latency!

>Are you using dotLucene or a webservice tier and java?

We are using Java Lucene on dedicated servers.


>How did you implement your bounding box for the searching? It sounds like
you do this outside of lucene and return a custom hitcollector.

The 'bounding box' is merely the conjunction of 2 numeric range searches.
It's really not that hard to do - I think there has been discussion of this
elsewhere in this group. We use (not 'return') a custom HitCollector to
exclude hits that aren't in the bounding box. I tried to explain this in a
reply earlier today, but if I failed let me know.

> Why not use a rangequery or functionquery for the basic bounding before
sorting

Basically, 'RangeQuery' doesn't offer sufficient performance. We have
implemented our own 'numeric value' search 'next to Lucene' (I think I like
this better than 'outside of Lucene' ;-)).  FunctionQuery could be used if
you wanted the jobs sorted by a combination of keywords and distance. Our
users (apparently) expect the jobs to be sorted strictly by distance on a
radius search.

================================================================

Peter

Hello Peter,

Now that the monster lucene search is live, is performance pretty good? Are
you still running it on a single 8 core server? Can you give me a rough idea
on the number of queries you can handle/second and the number of docs in the
index? Are you using dotLucene or a webservice tier and java?

How did you implement your bounding box for the searching? It sounds like
you do this outside of lucene and return a custom hitcollector. Why not use
a rangequery or functionquery for the basic bounding before sorting?

Thanks,
Eric

Re: Announcement: Lucene powering Monster job search index (Beta)

Posted by pd...@yahoo.com.
Peter,

Can you please share some of the details of the lucene index extensions you implemented for numeric range searches.

Thanks

Sent from my Verizon Wireless BlackBerry  

-----Original Message-----
From: "Peter Keegan" <pe...@gmail.com>
Date: Fri, 27 Oct 2006 15:29:20 
To:java-user@lucene.apache.org, lucene-net-user@incubator.apache.org
Subject: Announcement: Lucene powering Monster job search index (Beta)

I am pleased to announce the launch of Monster's new job search Beta web
site, powered by Lucene, at: http://jobsearch.beta.monster.com (notice the
Lucene logo at the bottom of the page!).

The jobs index is implemented with Java Lucene 2.0 on 64-bit Windows (AMD
and Intel processors)

Here are some of the new features:

1. 'Improve your search by'...

The job search results page allows you to browse and 'drill down' through
the results by job category, status, type and salary. The number of matching
jobs in each facet is displayed. There will likely be many more facets to
browse by in the future.

This feature is currently implemented with a custom HitCollector and the
DocSet class from Solr.

2. 'More like this'

Find more jobs like the one you see by clicking on the 'MORE LIKE THIS'
link, which is visible when you hover the mouse over the job title.

This feature is implemented with Lucene's term vectors and the
'MoreLikeThis' contribution class. If you are in 'detailed view', the term
vectors from the job description are used. In 'brief' view, the job title's
term vectors are used.

3. 'Related Titles'

When you do a 'keywords' search, click on a 'related titles' link to filter
you search by similar job titles.

This feature is implemented via a separate Lucene.Net index.

4. Sort by 'Miles'

Find jobs close to you via zip code/radius search. In the search results
page, click on the 'Miles' column to sort the results by distance from your
zip code/radius.

This custom sorting feature is implemented via Lucene's
'SortComparatorSource' interface.

5. Search by date, salary, distance.

Find jobs posted in the last day (or 2,3, etc) or by salary range or
distance.

Numeric range search is one of Lucene's weak points (performance-wise) so we
have implemented this with a custom HitCollector and an extension to the
Lucene index files that stores the numeric field values for all documents.

It is important to point out that this has all been implemented with the
stock Lucene 2.0 library. No code changes were made to the Lucene core.

If you have any feedback regarding the UI, please use the link on the web
page ("send us your feedback"). You can hit me with any other
questions/comments.

Peter


Re: Announcement: Lucene powering Monster job search index (Beta)

Posted by Peter Keegan <pe...@gmail.com>.
Note: this is a reply to a posting to java-dev  --Peter

Eric,

> Now that it is live, is performance pretty good?

Performance is outstanding. Each server can easily handle well over 100 qps
on an index of over 800K documents. There are several servers (4 dual core
(8 CPU) Opteron) supporting the query load and we have backup servers for
disaster recovery. For a few hours one day, all job search query traffic for
the entire site was being handled by a single server - with no noticable
latency!

>Are you using dotLucene or a webservice tier and java?

We are using Java Lucene on dedicated servers.


>How did you implement your bounding box for the searching? It sounds like
you do this outside of lucene and return a custom hitcollector.

The 'bounding box' is merely the conjunction of 2 numeric range searches.
It's really not that hard to do - I think there has been discussion of this
elsewhere in this group. We use (not 'return') a custom HitCollector to
exclude hits that aren't in the bounding box. I tried to explain this in a
reply earlier today, but if I failed let me know.

> Why not use a rangequery or functionquery for the basic bounding before
sorting

Basically, 'RangeQuery' doesn't offer sufficient performance. We have
implemented our own 'numeric value' search 'next to Lucene' (I think I like
this better than 'outside of Lucene' ;-)).  FunctionQuery could be used if
you wanted the jobs sorted by a combination of keywords and distance. Our
users (apparently) expect the jobs to be sorted strictly by distance on a
radius search.

================================================================

Peter

Hello Peter,

Now that the monster lucene search is live, is performance pretty good? Are
you still running it on a single 8 core server? Can you give me a rough idea
on the number of queries you can handle/second and the number of docs in the
index? Are you using dotLucene or a webservice tier and java?

How did you implement your bounding box for the searching? It sounds like
you do this outside of lucene and return a custom hitcollector. Why not use
a rangequery or functionquery for the basic bounding before sorting?

Thanks,
Eric

Re: Announcement: Lucene powering Monster job search index (Beta)

Posted by Peter Keegan <pe...@gmail.com>.
Paramasivam,

Take a look at Solr, in particular the DocSetHitCollector class. The
collector simply sets a bit in a BitSet, or saves the docIds in an array
(for low hit counts). Solr's BitSet was optimized (by Yonik, I believe) to
be faster than Java's BitSet, so this HitCollector is very fast. This is
essentially what we are doing for counting.

Peter

On 11/2/06, Paramasivam Srinivasan <sr...@indicus.net> wrote:
>
> Hi Peter
>
> When I use the CustomHitCollector, it affect the application performance.
> Also how you accomplish the grouping the results with out affecting
> performance. Also If possible give some code snippet for custome
> hitcollector.
>
> TIA
>
> Sri
>
> "Peter Keegan" <pe...@gmail.com> wrote in message
> news:e994873a0610300831p1229ad0k69de63cba052bd22@mail.gmail.com...
> > Joe,
> >
> > Fields with numeric values are stored in a separate file as binary
> values
> > in
> > an internal format. Lucene is unaware of this file and unaware of the
> > range
> > expression in the query. The range expression is parsed outside of
> Lucene
> > and used in a custom HitCollector to filter out documents that aren't in
> > the
> > requested range(s). A goal was to do this without having to modify
> Lucene.
> > Our scheme is pretty efficient, but not very general purpose in its
> > current
> > form, though.
> >
> > Peter
> >
> >
> > On 10/30/06, Joe Shaw <jo...@novell.com> wrote:
> >>
> >> Hi Peter,
> >>
> >> On Fri, 2006-10-27 at 15:29 -0400, Peter Keegan wrote:
> >> > Numeric range search is one of Lucene's weak points
> (performance-wise)
> >> so we
> >> > have implemented this with a custom HitCollector and an extension to
> >> > the
> >> > Lucene index files that stores the numeric field values for all
> >> documents.
> >> >
> >> > It is important to point out that this has all been implemented with
> >> > the
> >> > stock Lucene 2.0 library. No code changes were made to the Lucene
> core.
> >>
> >> Can you give some technical details on the extension to the Lucene
> index
> >> files?  How did you do it without making any changes to the Lucene
> core?
> >>
> >> Thanks,
> >> Joe
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
> >
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Re: Announcement: Lucene powering Monster job search index (Beta)

Posted by Paramasivam Srinivasan <sr...@indicus.net>.
Hi Peter

When I use the CustomHitCollector, it affect the application performance. 
Also how you accomplish the grouping the results with out affecting 
performance. Also If possible give some code snippet for custome 
hitcollector.

TIA

Sri

"Peter Keegan" <pe...@gmail.com> wrote in message 
news:e994873a0610300831p1229ad0k69de63cba052bd22@mail.gmail.com...
> Joe,
>
> Fields with numeric values are stored in a separate file as binary values 
> in
> an internal format. Lucene is unaware of this file and unaware of the 
> range
> expression in the query. The range expression is parsed outside of Lucene
> and used in a custom HitCollector to filter out documents that aren't in 
> the
> requested range(s). A goal was to do this without having to modify Lucene.
> Our scheme is pretty efficient, but not very general purpose in its 
> current
> form, though.
>
> Peter
>
>
> On 10/30/06, Joe Shaw <jo...@novell.com> wrote:
>>
>> Hi Peter,
>>
>> On Fri, 2006-10-27 at 15:29 -0400, Peter Keegan wrote:
>> > Numeric range search is one of Lucene's weak points (performance-wise)
>> so we
>> > have implemented this with a custom HitCollector and an extension to 
>> > the
>> > Lucene index files that stores the numeric field values for all
>> documents.
>> >
>> > It is important to point out that this has all been implemented with 
>> > the
>> > stock Lucene 2.0 library. No code changes were made to the Lucene core.
>>
>> Can you give some technical details on the extension to the Lucene index
>> files?  How did you do it without making any changes to the Lucene core?
>>
>> Thanks,
>> Joe
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
> 




---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Announcement: Lucene powering Monster job search index (Beta)

Posted by Peter Keegan <pe...@gmail.com>.
Joe,

Fields with numeric values are stored in a separate file as binary values in
an internal format. Lucene is unaware of this file and unaware of the range
expression in the query. The range expression is parsed outside of Lucene
and used in a custom HitCollector to filter out documents that aren't in the
requested range(s). A goal was to do this without having to modify Lucene.
Our scheme is pretty efficient, but not very general purpose in its current
form, though.

Peter


On 10/30/06, Joe Shaw <jo...@novell.com> wrote:
>
> Hi Peter,
>
> On Fri, 2006-10-27 at 15:29 -0400, Peter Keegan wrote:
> > Numeric range search is one of Lucene's weak points (performance-wise)
> so we
> > have implemented this with a custom HitCollector and an extension to the
> > Lucene index files that stores the numeric field values for all
> documents.
> >
> > It is important to point out that this has all been implemented with the
> > stock Lucene 2.0 library. No code changes were made to the Lucene core.
>
> Can you give some technical details on the extension to the Lucene index
> files?  How did you do it without making any changes to the Lucene core?
>
> Thanks,
> Joe
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Re: Announcement: Lucene powering Monster job search index (Beta)

Posted by Joe Shaw <jo...@novell.com>.
Hi Peter,

On Fri, 2006-10-27 at 15:29 -0400, Peter Keegan wrote:
> Numeric range search is one of Lucene's weak points (performance-wise) so we
> have implemented this with a custom HitCollector and an extension to the
> Lucene index files that stores the numeric field values for all documents.
> 
> It is important to point out that this has all been implemented with the
> stock Lucene 2.0 library. No code changes were made to the Lucene core.

Can you give some technical details on the extension to the Lucene index
files?  How did you do it without making any changes to the Lucene core?

Thanks,
Joe


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Announcement: Lucene powering Monster job search index (Beta)

Posted by Peter Keegan <pe...@gmail.com>.
KEGan,

>When you search by "4. Sort by Miles", I suppose the sorting by relevance
>(of the search keyword) is lost? Since this is implemented using a custom
>SortComparatorSource.

Sorting by miles becomes the primary sort key, score and date become
secondary sort fields (in the case of ties).

>Also, I suppose, if FunctionQuery were used, we can make "job distance by
>miles" part of the relavancy of the search results?

Yes, this is my understanding of the power of FunctionQuery.

Peter

On 10/30/06, KEGan <kh...@gmail.com> wrote:
>
> Peter,
>
> Congratulation on the beta launch :)
>
> If you dont mind, I would like to ask you more on the feature "4. Sort by
> Miles".
>
> When you search by "4. Sort by Miles", I suppose the sorting by relevance
> (of the search keyword) is lost? Since this is implemented using a custom
> SortComparatorSource.
>
> Also, I suppose, if FunctionQuery were used, we can make "job distance by
> miles" part of the relavancy of the search results?
>
> Could you comment or confirm my assertion ? Thanks :)
>
>
> On 10/28/06, Peter Keegan <pe...@gmail.com> wrote:
> >
> > On 10/27/06, Chris Lu <ch...@gmail.com> wrote:
> > >
> > > Hi, Peter,
> > >
> > > Really great job!
> >
> >
> > Thanks. (I'll tell the team)
> >
> > I am interested to know how you implemented "4. Sort by 'Miles'". For
> > > example, if starting from a zip code, how to match items within 20
> > > miles?
> >
> >
> > I can tell you how we use Lucene to accomplish this.
> > At indexing time, each job's location is indexed as a special field. How
> > you
> > represent the location is up to you. Each time a new index is built the
> > location data for all documents in the index are fetched via TermEnum
> and
> > TermDocs. This is practical because the searcher refresh is done at
> > predictable times. At query time, a custom SortComparatorSource is
> > created,
> > using the 'reference' location (the zip/radius). The 'compare' method
> > performs the calculation to compare the 2 documents' location values
> > (saved
> > from above) to the reference location.
> >
> > I believe this can also be accomplished with Solr's FunctionQuery, but I
> > haven't tried that yet.
> >
> > Peter
> >
> > --
> > > Chris Lu
> > > -------------------------
> > > Instant Full-Text Search On Any Database/Application
> > > site: http://www.dbsight.net
> > > demo: http://search.dbsight.com
> > >
> > > On 10/27/06, Peter Keegan <pe...@gmail.com> wrote:
> > > > I am pleased to announce the launch of Monster's new job search Beta
> > web
> > > > site, powered by Lucene, at: http://jobsearch.beta.monster.com(notice
> > > the
> > > > Lucene logo at the bottom of the page!).
> > > >
> > > > The jobs index is implemented with Java Lucene 2.0 on 64-bit Windows
> > > (AMD
> > > > and Intel processors)
> > > >
> > > > Here are some of the new features:
> > > >
> > > > 1. 'Improve your search by'...
> > > >
> > > > The job search results page allows you to browse and 'drill down'
> > > through
> > > > the results by job category, status, type and salary. The number of
> > > matching
> > > > jobs in each facet is displayed. There will likely be many more
> facets
> > > to
> > > > browse by in the future.
> > > >
> > > > This feature is currently implemented with a custom HitCollector and
> > the
> > > > DocSet class from Solr.
> > > >
> > > > 2. 'More like this'
> > > >
> > > > Find more jobs like the one you see by clicking on the 'MORE LIKE
> > THIS'
> > > > link, which is visible when you hover the mouse over the job title.
> > > >
> > > > This feature is implemented with Lucene's term vectors and the
> > > > 'MoreLikeThis' contribution class. If you are in 'detailed view',
> the
> > > term
> > > > vectors from the job description are used. In 'brief' view, the job
> > > title's
> > > > term vectors are used.
> > > >
> > > > 3. 'Related Titles'
> > > >
> > > > When you do a 'keywords' search, click on a 'related titles' link to
> > > filter
> > > > you search by similar job titles.
> > > >
> > > > This feature is implemented via a separate Lucene.Net index.
> > > >
> > > > 4. Sort by 'Miles'
> > > >
> > > > Find jobs close to you via zip code/radius search. In the search
> > results
> > > > page, click on the 'Miles' column to sort the results by distance
> from
> > > your
> > > > zip code/radius.
> > > >
> > > > This custom sorting feature is implemented via Lucene's
> > > > 'SortComparatorSource' interface.
> > > >
> > > > 5. Search by date, salary, distance.
> > > >
> > > > Find jobs posted in the last day (or 2,3, etc) or by salary range or
> > > > distance.
> > > >
> > > > Numeric range search is one of Lucene's weak points
> (performance-wise)
> > > so we
> > > > have implemented this with a custom HitCollector and an extension to
> > the
> > > > Lucene index files that stores the numeric field values for all
> > > documents.
> > > >
> > > > It is important to point out that this has all been implemented with
> > the
> > > > stock Lucene 2.0 library. No code changes were made to the Lucene
> > core.
> > > >
> > > > If you have any feedback regarding the UI, please use the link on
> the
> > > web
> > > > page ("send us your feedback"). You can hit me with any other
> > > > questions/comments.
> > > >
> > > > Peter
> > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
> >
> >
>
>

Re: Announcement: Lucene powering Monster job search index (Beta)

Posted by KEGan <kh...@gmail.com>.
Peter,

Congratulation on the beta launch :)

If you dont mind, I would like to ask you more on the feature "4. Sort by
Miles".

When you search by "4. Sort by Miles", I suppose the sorting by relevance
(of the search keyword) is lost? Since this is implemented using a custom
SortComparatorSource.

Also, I suppose, if FunctionQuery were used, we can make "job distance by
miles" part of the relavancy of the search results?

Could you comment or confirm my assertion ? Thanks :)


On 10/28/06, Peter Keegan <pe...@gmail.com> wrote:
>
> On 10/27/06, Chris Lu <ch...@gmail.com> wrote:
> >
> > Hi, Peter,
> >
> > Really great job!
>
>
> Thanks. (I'll tell the team)
>
> I am interested to know how you implemented "4. Sort by 'Miles'". For
> > example, if starting from a zip code, how to match items within 20
> > miles?
>
>
> I can tell you how we use Lucene to accomplish this.
> At indexing time, each job's location is indexed as a special field. How
> you
> represent the location is up to you. Each time a new index is built the
> location data for all documents in the index are fetched via TermEnum and
> TermDocs. This is practical because the searcher refresh is done at
> predictable times. At query time, a custom SortComparatorSource is
> created,
> using the 'reference' location (the zip/radius). The 'compare' method
> performs the calculation to compare the 2 documents' location values
> (saved
> from above) to the reference location.
>
> I believe this can also be accomplished with Solr's FunctionQuery, but I
> haven't tried that yet.
>
> Peter
>
> --
> > Chris Lu
> > -------------------------
> > Instant Full-Text Search On Any Database/Application
> > site: http://www.dbsight.net
> > demo: http://search.dbsight.com
> >
> > On 10/27/06, Peter Keegan <pe...@gmail.com> wrote:
> > > I am pleased to announce the launch of Monster's new job search Beta
> web
> > > site, powered by Lucene, at: http://jobsearch.beta.monster.com (notice
> > the
> > > Lucene logo at the bottom of the page!).
> > >
> > > The jobs index is implemented with Java Lucene 2.0 on 64-bit Windows
> > (AMD
> > > and Intel processors)
> > >
> > > Here are some of the new features:
> > >
> > > 1. 'Improve your search by'...
> > >
> > > The job search results page allows you to browse and 'drill down'
> > through
> > > the results by job category, status, type and salary. The number of
> > matching
> > > jobs in each facet is displayed. There will likely be many more facets
> > to
> > > browse by in the future.
> > >
> > > This feature is currently implemented with a custom HitCollector and
> the
> > > DocSet class from Solr.
> > >
> > > 2. 'More like this'
> > >
> > > Find more jobs like the one you see by clicking on the 'MORE LIKE
> THIS'
> > > link, which is visible when you hover the mouse over the job title.
> > >
> > > This feature is implemented with Lucene's term vectors and the
> > > 'MoreLikeThis' contribution class. If you are in 'detailed view', the
> > term
> > > vectors from the job description are used. In 'brief' view, the job
> > title's
> > > term vectors are used.
> > >
> > > 3. 'Related Titles'
> > >
> > > When you do a 'keywords' search, click on a 'related titles' link to
> > filter
> > > you search by similar job titles.
> > >
> > > This feature is implemented via a separate Lucene.Net index.
> > >
> > > 4. Sort by 'Miles'
> > >
> > > Find jobs close to you via zip code/radius search. In the search
> results
> > > page, click on the 'Miles' column to sort the results by distance from
> > your
> > > zip code/radius.
> > >
> > > This custom sorting feature is implemented via Lucene's
> > > 'SortComparatorSource' interface.
> > >
> > > 5. Search by date, salary, distance.
> > >
> > > Find jobs posted in the last day (or 2,3, etc) or by salary range or
> > > distance.
> > >
> > > Numeric range search is one of Lucene's weak points (performance-wise)
> > so we
> > > have implemented this with a custom HitCollector and an extension to
> the
> > > Lucene index files that stores the numeric field values for all
> > documents.
> > >
> > > It is important to point out that this has all been implemented with
> the
> > > stock Lucene 2.0 library. No code changes were made to the Lucene
> core.
> > >
> > > If you have any feedback regarding the UI, please use the link on the
> > web
> > > page ("send us your feedback"). You can hit me with any other
> > > questions/comments.
> > >
> > > Peter
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
>

Re: Announcement: Lucene powering Monster job search index (Beta)

Posted by Peter Keegan <pe...@gmail.com>.
Alex,

I like your suggestion (I've found myself wondering what the last search
was, too), and I've forwarded it to the UI developer.

Thanks,
Peter


On 10/29/06, Alexandru Popescu <th...@gmail.com> wrote:
>
> Peter it looks impressive. Congrats! A small suggestion, though, after
> performing a search the filtering criteria is not displayed anywhere.
> I guess it would make sense to write it in a read-only form somewhere
> on the result pages:
>
> Jobs 1-50 of 7896 matches to Jobs 1-50 of 7896 matching criteria (a
> small hidden stuff showing the criteria).
>
> ./alex
> --
> .w( the_mindstorm )p.
>
>
> On 10/29/06, Otis Gospodnetic <ot...@yahoo.com> wrote:
> > Hi,
> >
> > --- Peter Keegan <pe...@gmail.com> wrote:
> >
> > > On 10/27/06, Chris Lu <ch...@gmail.com> wrote:
> > > >
> > > > Hi, Peter,
> > > >
> > > > Really great job!
> > >
> > >
> > > Thanks. (I'll tell the team)
> >
> > If it's not a secret, can you tell us a bit more about what's behind
> > the search in terms of hardware, and how much pounding that hardware
> > takes in terms of QPS?  People always ask about this stuff.
> >
> > Thanks,
> > Otis
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Re: Announcement: Lucene powering Monster job search index (Beta)

Posted by Alexandru Popescu <th...@gmail.com>.
Peter it looks impressive. Congrats! A small suggestion, though, after
performing a search the filtering criteria is not displayed anywhere.
I guess it would make sense to write it in a read-only form somewhere
on the result pages:

Jobs 1-50 of 7896 matches to Jobs 1-50 of 7896 matching criteria (a
small hidden stuff showing the criteria).

./alex
--
.w( the_mindstorm )p.


On 10/29/06, Otis Gospodnetic <ot...@yahoo.com> wrote:
> Hi,
>
> --- Peter Keegan <pe...@gmail.com> wrote:
>
> > On 10/27/06, Chris Lu <ch...@gmail.com> wrote:
> > >
> > > Hi, Peter,
> > >
> > > Really great job!
> >
> >
> > Thanks. (I'll tell the team)
>
> If it's not a secret, can you tell us a bit more about what's behind
> the search in terms of hardware, and how much pounding that hardware
> takes in terms of QPS?  People always ask about this stuff.
>
> Thanks,
> Otis
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Announcement: Lucene powering Monster job search index (Beta)

Posted by Peter Keegan <pe...@gmail.com>.
Otis,

The Lucene components for this beta are running on 4 dual core AMD Opteron (
2.6GHZ) processors, for a total of 8 CPUs. It has 32GB RAM, although 16GB
would probably suffice. The query rate is currently quite low probably
because of the low visibility of the beta page. We haven't measured QPS
rates for this configuration, yet, but if you look at some of my previous
posts, you'll see some QPS data on somewhat similar hardware. I think that
actual rates will be lower, though, because the complexity of the queries,
counting, sorting, etc have increased.

Peter

On 10/28/06, Otis Gospodnetic <ot...@yahoo.com> wrote:
>
> Hi,
>
> --- Peter Keegan <pe...@gmail.com> wrote:
>
> > On 10/27/06, Chris Lu <ch...@gmail.com> wrote:
> > >
> > > Hi, Peter,
> > >
> > > Really great job!
> >
> >
> > Thanks. (I'll tell the team)
>
> If it's not a secret, can you tell us a bit more about what's behind
> the search in terms of hardware, and how much pounding that hardware
> takes in terms of QPS?  People always ask about this stuff.
>
> Thanks,
> Otis
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Re: Announcement: Lucene powering Monster job search index (Beta)

Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hi,

--- Peter Keegan <pe...@gmail.com> wrote:

> On 10/27/06, Chris Lu <ch...@gmail.com> wrote:
> >
> > Hi, Peter,
> >
> > Really great job!
> 
> 
> Thanks. (I'll tell the team)

If it's not a secret, can you tell us a bit more about what's behind
the search in terms of hardware, and how much pounding that hardware
takes in terms of QPS?  People always ask about this stuff.

Thanks,
Otis


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Announcement: Lucene powering Monster job search index (Beta)

Posted by Peter Keegan <pe...@gmail.com>.
On 10/27/06, Chris Lu <ch...@gmail.com> wrote:
>
> Hi, Peter,
>
> Really great job!


Thanks. (I'll tell the team)

I am interested to know how you implemented "4. Sort by 'Miles'". For
> example, if starting from a zip code, how to match items within 20
> miles?


I can tell you how we use Lucene to accomplish this.
At indexing time, each job's location is indexed as a special field. How you
represent the location is up to you. Each time a new index is built the
location data for all documents in the index are fetched via TermEnum and
TermDocs. This is practical because the searcher refresh is done at
predictable times. At query time, a custom SortComparatorSource is created,
using the 'reference' location (the zip/radius). The 'compare' method
performs the calculation to compare the 2 documents' location values (saved
from above) to the reference location.

I believe this can also be accomplished with Solr's FunctionQuery, but I
haven't tried that yet.

Peter

--
> Chris Lu
> -------------------------
> Instant Full-Text Search On Any Database/Application
> site: http://www.dbsight.net
> demo: http://search.dbsight.com
>
> On 10/27/06, Peter Keegan <pe...@gmail.com> wrote:
> > I am pleased to announce the launch of Monster's new job search Beta web
> > site, powered by Lucene, at: http://jobsearch.beta.monster.com (notice
> the
> > Lucene logo at the bottom of the page!).
> >
> > The jobs index is implemented with Java Lucene 2.0 on 64-bit Windows
> (AMD
> > and Intel processors)
> >
> > Here are some of the new features:
> >
> > 1. 'Improve your search by'...
> >
> > The job search results page allows you to browse and 'drill down'
> through
> > the results by job category, status, type and salary. The number of
> matching
> > jobs in each facet is displayed. There will likely be many more facets
> to
> > browse by in the future.
> >
> > This feature is currently implemented with a custom HitCollector and the
> > DocSet class from Solr.
> >
> > 2. 'More like this'
> >
> > Find more jobs like the one you see by clicking on the 'MORE LIKE THIS'
> > link, which is visible when you hover the mouse over the job title.
> >
> > This feature is implemented with Lucene's term vectors and the
> > 'MoreLikeThis' contribution class. If you are in 'detailed view', the
> term
> > vectors from the job description are used. In 'brief' view, the job
> title's
> > term vectors are used.
> >
> > 3. 'Related Titles'
> >
> > When you do a 'keywords' search, click on a 'related titles' link to
> filter
> > you search by similar job titles.
> >
> > This feature is implemented via a separate Lucene.Net index.
> >
> > 4. Sort by 'Miles'
> >
> > Find jobs close to you via zip code/radius search. In the search results
> > page, click on the 'Miles' column to sort the results by distance from
> your
> > zip code/radius.
> >
> > This custom sorting feature is implemented via Lucene's
> > 'SortComparatorSource' interface.
> >
> > 5. Search by date, salary, distance.
> >
> > Find jobs posted in the last day (or 2,3, etc) or by salary range or
> > distance.
> >
> > Numeric range search is one of Lucene's weak points (performance-wise)
> so we
> > have implemented this with a custom HitCollector and an extension to the
> > Lucene index files that stores the numeric field values for all
> documents.
> >
> > It is important to point out that this has all been implemented with the
> > stock Lucene 2.0 library. No code changes were made to the Lucene core.
> >
> > If you have any feedback regarding the UI, please use the link on the
> web
> > page ("send us your feedback"). You can hit me with any other
> > questions/comments.
> >
> > Peter
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Re: Announcement: Lucene powering Monster job search index (Beta)

Posted by Chris Lu <ch...@gmail.com>.
Hi, Peter,

Really great job!

I am interested to know how you implemented "4. Sort by 'Miles'". For
example, if starting from a zip code, how to match items within 20
miles?

-- 
Chris Lu
-------------------------
Instant Full-Text Search On Any Database/Application
site: http://www.dbsight.net
demo: http://search.dbsight.com

On 10/27/06, Peter Keegan <pe...@gmail.com> wrote:
> I am pleased to announce the launch of Monster's new job search Beta web
> site, powered by Lucene, at: http://jobsearch.beta.monster.com (notice the
> Lucene logo at the bottom of the page!).
>
> The jobs index is implemented with Java Lucene 2.0 on 64-bit Windows (AMD
> and Intel processors)
>
> Here are some of the new features:
>
> 1. 'Improve your search by'...
>
> The job search results page allows you to browse and 'drill down' through
> the results by job category, status, type and salary. The number of matching
> jobs in each facet is displayed. There will likely be many more facets to
> browse by in the future.
>
> This feature is currently implemented with a custom HitCollector and the
> DocSet class from Solr.
>
> 2. 'More like this'
>
> Find more jobs like the one you see by clicking on the 'MORE LIKE THIS'
> link, which is visible when you hover the mouse over the job title.
>
> This feature is implemented with Lucene's term vectors and the
> 'MoreLikeThis' contribution class. If you are in 'detailed view', the term
> vectors from the job description are used. In 'brief' view, the job title's
> term vectors are used.
>
> 3. 'Related Titles'
>
> When you do a 'keywords' search, click on a 'related titles' link to filter
> you search by similar job titles.
>
> This feature is implemented via a separate Lucene.Net index.
>
> 4. Sort by 'Miles'
>
> Find jobs close to you via zip code/radius search. In the search results
> page, click on the 'Miles' column to sort the results by distance from your
> zip code/radius.
>
> This custom sorting feature is implemented via Lucene's
> 'SortComparatorSource' interface.
>
> 5. Search by date, salary, distance.
>
> Find jobs posted in the last day (or 2,3, etc) or by salary range or
> distance.
>
> Numeric range search is one of Lucene's weak points (performance-wise) so we
> have implemented this with a custom HitCollector and an extension to the
> Lucene index files that stores the numeric field values for all documents.
>
> It is important to point out that this has all been implemented with the
> stock Lucene 2.0 library. No code changes were made to the Lucene core.
>
> If you have any feedback regarding the UI, please use the link on the web
> page ("send us your feedback"). You can hit me with any other
> questions/comments.
>
> Peter
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org