You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Charlie Jackson <Ch...@cision.com> on 2008/10/24 22:18:46 UTC
Entity extraction?
During a recent sales pitch to my company by FAST, they mentioned entity
extraction. I'd never heard of it before, but they described it as
basically recognizing people/places/things in documents being indexed
and then being able to do faceting on this data at query time. Does
anything like this already exist in SOLR? If not, I'm not opposed to
developing it myself, but I could use some pointers on where to start.
Thanks,
- Charlie
Re: Entity extraction?
Posted by Grant Ingersoll <gs...@apache.org>.
On Oct 27, 2008, at 8:53 PM, Ryan McKinley wrote:
>
> On Oct 27, 2008, at 6:10 PM, Grant Ingersoll wrote:
>
>> Warning: shameless plug: Tom Morton and I have a chapter on NER
>> and OpenNLP (and Solr, for that matter) in our book "Taming
>> Text" (Manning) and the code will be open once we have a place to
>> put it (hopefully soon). In fact, you'll see us doing a lot of
>> this kind of stuff w/ Solr and it should all be coming back to Solr/
>> Lucene/Mahout at some point (for instance, see https://issues.apache.org/jira/browse/SOLR-769
>> , as I'm sure FAST told you they can do clustering, too!)
>> --end shameless plug ---
>>
>
> thats great!
>
> I just got the MEAP copy, it looks really good
> http://www.manning.com/ingersoll/
Thanks!
>
>
>
>> As for Mahout, NER is a classification problem, and there are some
>> tools in Mahout to do classification, but nothing specifically
>> targeted at NER at the moment. Mahout, like Nutch, also takes
>> advantage of Hadoop for scaling. The combination of Mahout in Solr
>> makes a lot of sense, IMO.
>>
>
> Perhaps this is more appropriate to ask on the mahout list, but...
> when you say "Mahout, like Nutch, also takes advantage of Hadoop for
> scaling", does that mean that much of Mahout requires hadoop? Is it
> possible to do smaller scale problems on a simple setup and only
> invoke hadoop when required?
Yes, probably better asked on Mahout, but to answer your question,
yes, most of the implementations require Hadoop so far, but it is not
a strict requirement. That being said, it is fairly easy to run them
on a simple setup (i.e. single node).
Re: Entity extraction?
Posted by Ryan McKinley <ry...@gmail.com>.
On Oct 27, 2008, at 6:10 PM, Grant Ingersoll wrote:
> Warning: shameless plug: Tom Morton and I have a chapter on NER and
> OpenNLP (and Solr, for that matter) in our book "Taming
> Text" (Manning) and the code will be open once we have a place to
> put it (hopefully soon). In fact, you'll see us doing a lot of this
> kind of stuff w/ Solr and it should all be coming back to Solr/
> Lucene/Mahout at some point (for instance, see https://issues.apache.org/jira/browse/SOLR-769
> , as I'm sure FAST told you they can do clustering, too!)
> --end shameless plug ---
>
thats great!
I just got the MEAP copy, it looks really good
http://www.manning.com/ingersoll/
> As for Mahout, NER is a classification problem, and there are some
> tools in Mahout to do classification, but nothing specifically
> targeted at NER at the moment. Mahout, like Nutch, also takes
> advantage of Hadoop for scaling. The combination of Mahout in Solr
> makes a lot of sense, IMO.
>
Perhaps this is more appropriate to ask on the mahout list, but...
when you say "Mahout, like Nutch, also takes advantage of Hadoop for
scaling", does that mean that much of Mahout requires hadoop? Is it
possible to do smaller scale problems on a simple setup and only
invoke hadoop when required?
ryan
Re: Entity extraction?
Posted by Grant Ingersoll <gs...@apache.org>.
Warning: shameless plug: Tom Morton and I have a chapter on NER and
OpenNLP (and Solr, for that matter) in our book "Taming
Text" (Manning) and the code will be open once we have a place to put
it (hopefully soon). In fact, you'll see us doing a lot of this kind
of stuff w/ Solr and it should all be coming back to Solr/Lucene/
Mahout at some point (for instance, see https://issues.apache.org/jira/browse/SOLR-769
, as I'm sure FAST told you they can do clustering, too!)
--end shameless plug ---
As for Mahout, NER is a classification problem, and there are some
tools in Mahout to do classification, but nothing specifically
targeted at NER at the moment. Mahout, like Nutch, also takes
advantage of Hadoop for scaling. The combination of Mahout in Solr
makes a lot of sense, IMO.
On Oct 25, 2008, at 11:25 PM, Vaijanath N. Rao wrote:
> Hi,
>
> One can use the OpenNLP Max entropy library and create there own
> named-entity extraction.
> I had used it in one of the projects which I did with Solr.
>
> It is easy to integrate most of the NLP libraries with Solr. Though
> we had named-entity extraction embedded in our crawler which would
> populate a field called entities in the database, which we would
> ingest in Solr as yet another field.
>
> --Thanks and Regards
> Vaijanath N. Rao
>
> Julien Nioche wrote:
>> Hi,
>>
>> Open Source NLP platforms like GATE (http://gate.ac.uk) or Apache
>> UIMA are
>> typically used for these types of tasks. GATE in particular comes
>> with an
>> application called ANNIE which does Named Entity Recognition.
>> OpenCalais
>> does that as well and should be easy to embed, but it can't be
>> tuned to do
>> more specific things unlike UIMA or GATE based applications.
>>
>> Depending on the architecture you have in mind it could be worth
>> investigating Nutch and add the NER as a custom plugin; NLP being
>> often a
>> CPU intensive task you could leverage the scalability of Hadoop in
>> Nutch.
>> There is a patch which allows to delegate the indexing to SOLR. As
>> someone
>> else already said these named entities could then be used as facets.
>>
>> HTH
>>
>> Julien
>>
>
--------------------------
Grant Ingersoll
Lucene Boot Camp Training Nov. 3-4, 2008, ApacheCon US New Orleans.
http://www.lucenebootcamp.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ
Re: Entity extraction?
Posted by "Vaijanath N. Rao" <va...@gmail.com>.
Hi,
One can use the OpenNLP Max entropy library and create there own
named-entity extraction.
I had used it in one of the projects which I did with Solr.
It is easy to integrate most of the NLP libraries with Solr. Though we
had named-entity extraction embedded in our crawler which would populate
a field called entities in the database, which we would ingest in Solr
as yet another field.
--Thanks and Regards
Vaijanath N. Rao
Julien Nioche wrote:
> Hi,
>
> Open Source NLP platforms like GATE (http://gate.ac.uk) or Apache UIMA are
> typically used for these types of tasks. GATE in particular comes with an
> application called ANNIE which does Named Entity Recognition. OpenCalais
> does that as well and should be easy to embed, but it can't be tuned to do
> more specific things unlike UIMA or GATE based applications.
>
> Depending on the architecture you have in mind it could be worth
> investigating Nutch and add the NER as a custom plugin; NLP being often a
> CPU intensive task you could leverage the scalability of Hadoop in Nutch.
> There is a patch which allows to delegate the indexing to SOLR. As someone
> else already said these named entities could then be used as facets.
>
> HTH
>
> Julien
>
Re: Entity extraction?
Posted by Julien Nioche <li...@gmail.com>.
Hi,
Open Source NLP platforms like GATE (http://gate.ac.uk) or Apache UIMA are
typically used for these types of tasks. GATE in particular comes with an
application called ANNIE which does Named Entity Recognition. OpenCalais
does that as well and should be easy to embed, but it can't be tuned to do
more specific things unlike UIMA or GATE based applications.
Depending on the architecture you have in mind it could be worth
investigating Nutch and add the NER as a custom plugin; NLP being often a
CPU intensive task you could leverage the scalability of Hadoop in Nutch.
There is a patch which allows to delegate the indexing to SOLR. As someone
else already said these named entities could then be used as facets.
HTH
Julien
--
DigitalPebble Ltd
http://www.digitalpebble.com
2008/10/24 Rogerio Pereira <ro...@gmail.com>
> I agree Ryan and I would like see a completly integration between solr,
> nutch, tika and mahout in the future.
>
> 2008/10/24 Ryan McKinley <ry...@gmail.com>
>
> > This is not something solr does currently...
> >
> > It sounds like something that should be added to Mahout:
> > http://lucene.apache.org/mahout/
> >
> >
> >
> > On Oct 24, 2008, at 4:18 PM, Charlie Jackson wrote:
> >
> > During a recent sales pitch to my company by FAST, they mentioned entity
> >> extraction. I'd never heard of it before, but they described it as
> >> basically recognizing people/places/things in documents being indexed
> >> and then being able to do faceting on this data at query time. Does
> >> anything like this already exist in SOLR? If not, I'm not opposed to
> >> developing it myself, but I could use some pointers on where to start.
> >>
> >>
> >>
> >> Thanks,
> >>
> >> - Charlie
> >>
> >>
> >
>
>
> --
> Regards,
>
> Rogério (_rogerio_)
>
> [Blog: http://faces.eti.br] [Sandbox: http://bmobile.dyndns.org]
> [Twitter:
> http://twitter.com/ararog]
>
> "Faça a diferença! Ajude o seu país a crescer, não retenha conhecimento,
> distribua e aprenda mais."
> (http://faces.eti.br/2006/10/30/conhecimento-e-amadurecimento)
>
Re: Entity extraction?
Posted by Rogerio Pereira <ro...@gmail.com>.
I agree Ryan and I would like see a completly integration between solr,
nutch, tika and mahout in the future.
2008/10/24 Ryan McKinley <ry...@gmail.com>
> This is not something solr does currently...
>
> It sounds like something that should be added to Mahout:
> http://lucene.apache.org/mahout/
>
>
>
> On Oct 24, 2008, at 4:18 PM, Charlie Jackson wrote:
>
> During a recent sales pitch to my company by FAST, they mentioned entity
>> extraction. I'd never heard of it before, but they described it as
>> basically recognizing people/places/things in documents being indexed
>> and then being able to do faceting on this data at query time. Does
>> anything like this already exist in SOLR? If not, I'm not opposed to
>> developing it myself, but I could use some pointers on where to start.
>>
>>
>>
>> Thanks,
>>
>> - Charlie
>>
>>
>
--
Regards,
Rogério (_rogerio_)
[Blog: http://faces.eti.br] [Sandbox: http://bmobile.dyndns.org] [Twitter:
http://twitter.com/ararog]
"Faça a diferença! Ajude o seu país a crescer, não retenha conhecimento,
distribua e aprenda mais."
(http://faces.eti.br/2006/10/30/conhecimento-e-amadurecimento)
Re: Entity extraction?
Posted by Ryan McKinley <ry...@gmail.com>.
This is not something solr does currently...
It sounds like something that should be added to Mahout:
http://lucene.apache.org/mahout/
On Oct 24, 2008, at 4:18 PM, Charlie Jackson wrote:
> During a recent sales pitch to my company by FAST, they mentioned
> entity
> extraction. I'd never heard of it before, but they described it as
> basically recognizing people/places/things in documents being indexed
> and then being able to do faceting on this data at query time. Does
> anything like this already exist in SOLR? If not, I'm not opposed to
> developing it myself, but I could use some pointers on where to start.
>
>
>
> Thanks,
>
> - Charlie
>
RE: Entity extraction?
Posted by Charlie Jackson <Ch...@cision.com>.
Thanks for the replies, guys, that gives me a good place to start looking.
- Charlie
-----Original Message-----
From: Rogerio Pereira [mailto:rogerio.araujo@gmail.com]
Sent: Friday, October 24, 2008 5:14 PM
To: solr-user@lucene.apache.org
Subject: Re: Entity extraction?
You can find more about this topic in this book availabe at amazon:
http://www.amazon.com/Building-Search-Applications-Lucene-Lingpipe/dp/0615204252/
2008/10/24 Rafael Rossini <ra...@gmail.com>
> Solr can do a simple facet seach like FAST, but the entity extraction
> demands other tecnologies. I do not know how FAST does it but at the
> company
> I´m working on (www.cortex-intelligence.com), we use a mix of statistical
> and language-specific tasks to recognize and categorize entities in the
> text. Ling Pipe is another tool (free) that does that too. In case you
> would
> like to see a simple demo: http://www.cortex-intelligence.com/tech/
>
> Rossini
>
>
> On Fri, Oct 24, 2008 at 6:18 PM, Charlie Jackson <
> Charlie.Jackson@cision.com
> > wrote:
>
> > During a recent sales pitch to my company by FAST, they mentioned entity
> > extraction. I'd never heard of it before, but they described it as
> > basically recognizing people/places/things in documents being indexed
> > and then being able to do faceting on this data at query time. Does
> > anything like this already exist in SOLR? If not, I'm not opposed to
> > developing it myself, but I could use some pointers on where to start.
> >
> >
> >
> > Thanks,
> >
> > - Charlie
> >
> >
>
--
Regards,
Rogério (_rogerio_)
[Blog: http://faces.eti.br] [Sandbox: http://bmobile.dyndns.org] [Twitter:
http://twitter.com/ararog]
"Faça a diferença! Ajude o seu país a crescer, não retenha conhecimento,
distribua e aprenda mais."
(http://faces.eti.br/2006/10/30/conhecimento-e-amadurecimento)
Re: Entity extraction?
Posted by Rogerio Pereira <ro...@gmail.com>.
You can find more about this topic in this book availabe at amazon:
http://www.amazon.com/Building-Search-Applications-Lucene-Lingpipe/dp/0615204252/
2008/10/24 Rafael Rossini <ra...@gmail.com>
> Solr can do a simple facet seach like FAST, but the entity extraction
> demands other tecnologies. I do not know how FAST does it but at the
> company
> I´m working on (www.cortex-intelligence.com), we use a mix of statistical
> and language-specific tasks to recognize and categorize entities in the
> text. Ling Pipe is another tool (free) that does that too. In case you
> would
> like to see a simple demo: http://www.cortex-intelligence.com/tech/
>
> Rossini
>
>
> On Fri, Oct 24, 2008 at 6:18 PM, Charlie Jackson <
> Charlie.Jackson@cision.com
> > wrote:
>
> > During a recent sales pitch to my company by FAST, they mentioned entity
> > extraction. I'd never heard of it before, but they described it as
> > basically recognizing people/places/things in documents being indexed
> > and then being able to do faceting on this data at query time. Does
> > anything like this already exist in SOLR? If not, I'm not opposed to
> > developing it myself, but I could use some pointers on where to start.
> >
> >
> >
> > Thanks,
> >
> > - Charlie
> >
> >
>
--
Regards,
Rogério (_rogerio_)
[Blog: http://faces.eti.br] [Sandbox: http://bmobile.dyndns.org] [Twitter:
http://twitter.com/ararog]
"Faça a diferença! Ajude o seu país a crescer, não retenha conhecimento,
distribua e aprenda mais."
(http://faces.eti.br/2006/10/30/conhecimento-e-amadurecimento)
Re: Entity extraction?
Posted by Rafael Rossini <ra...@gmail.com>.
Solr can do a simple facet seach like FAST, but the entity extraction
demands other tecnologies. I do not know how FAST does it but at the company
I´m working on (www.cortex-intelligence.com), we use a mix of statistical
and language-specific tasks to recognize and categorize entities in the
text. Ling Pipe is another tool (free) that does that too. In case you would
like to see a simple demo: http://www.cortex-intelligence.com/tech/
Rossini
On Fri, Oct 24, 2008 at 6:18 PM, Charlie Jackson <Charlie.Jackson@cision.com
> wrote:
> During a recent sales pitch to my company by FAST, they mentioned entity
> extraction. I'd never heard of it before, but they described it as
> basically recognizing people/places/things in documents being indexed
> and then being able to do faceting on this data at query time. Does
> anything like this already exist in SOLR? If not, I'm not opposed to
> developing it myself, but I could use some pointers on where to start.
>
>
>
> Thanks,
>
> - Charlie
>
>