You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by marotosg <ma...@gmail.com> on 2017/03/26 16:05:32 UTC
Classify document using bag of words
Hi,
I have a very simple use case where I would need to classify a document
using a bag of words. Basically if a field within the document contains any
of the words on my bag then I use a new field to assign a category to the
document.
Is this something achievable on Solr?
I was thinking on using Lucene Document
classificationhttps://wiki.apache.org/solr/SolrClassification.
From what I understand I need to feed already the category on some
documents. New documents would be classified.
Is there anything else I can't find?
Thanks a lot.
--
View this message in context: http://lucene.472066.n3.nabble.com/Classify-document-using-bag-of-words-tp4326865.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Classify document using bag of words
Posted by John Blythe <jo...@curvolabs.com>.
Glad to hear it!
On Sun, Mar 26, 2017 at 3:49 PM Sergio García Maroto <ma...@gmail.com>
wrote:
> Sorry it actually works. Thanks a lot.
>
> On 26 March 2017 at 21:45, Sergio García Maroto <ma...@gmail.com>
> wrote:
>
> > Hi John.
> > thanks for that.
> >
> > That's actually a good option but I would need the category text on the
> > field so I can facet on the field and get every category and the number.
> >
> > On 26 March 2017 at 18:27, John Blythe <jo...@curvolabs.com> wrote:
> >
> >> You could use keepwords to filter out any other words besides your bag
> and
> >> then have a synonym filter that translates the remaining word(s) to a
> >> corresponding category/classification
> >>
> >> On Sun, Mar 26, 2017 at 12:05 PM marotosg <ma...@gmail.com> wrote:
> >>
> >> > Hi,
> >> >
> >> > I have a very simple use case where I would need to classify a
> document
> >> > using a bag of words. Basically if a field within the document
> contains
> >> any
> >> > of the words on my bag then I use a new field to assign a category to
> >> the
> >> > document.
> >> >
> >> > Is this something achievable on Solr?
> >> >
> >> > I was thinking on using Lucene Document
> >> > classificationhttps://wiki.apache.org/solr/SolrClassification.
> >> > From what I understand I need to feed already the category on some
> >> > documents. New documents would be classified.
> >> >
> >> > Is there anything else I can't find?
> >> >
> >> > Thanks a lot.
> >> >
> >> >
> >> >
> >> > --
> >> > View this message in context:
> >> > http://lucene.472066.n3.nabble.com/Classify-document-using-
> >> bag-of-words-tp4326865.html
> >> > Sent from the Solr - User mailing list archive at Nabble.com.
> >> >
> >> --
> >> --
> >> *John Blythe*
> >> Product Manager & Lead Developer
> >>
> >> 251.605.3071 | john@curvolabs.com
> >> www.curvolabs.com
> >>
> >> 58 Adams Ave
> >> Evansville, IN 47713
> >>
> >
> >
>
--
--
*John Blythe*
Product Manager & Lead Developer
251.605.3071 | john@curvolabs.com
www.curvolabs.com
58 Adams Ave
Evansville, IN 47713
Re: Classify document using bag of words
Posted by Sergio García Maroto <ma...@gmail.com>.
Sorry it actually works. Thanks a lot.
On 26 March 2017 at 21:45, Sergio García Maroto <ma...@gmail.com> wrote:
> Hi John.
> thanks for that.
>
> That's actually a good option but I would need the category text on the
> field so I can facet on the field and get every category and the number.
>
> On 26 March 2017 at 18:27, John Blythe <jo...@curvolabs.com> wrote:
>
>> You could use keepwords to filter out any other words besides your bag and
>> then have a synonym filter that translates the remaining word(s) to a
>> corresponding category/classification
>>
>> On Sun, Mar 26, 2017 at 12:05 PM marotosg <ma...@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > I have a very simple use case where I would need to classify a document
>> > using a bag of words. Basically if a field within the document contains
>> any
>> > of the words on my bag then I use a new field to assign a category to
>> the
>> > document.
>> >
>> > Is this something achievable on Solr?
>> >
>> > I was thinking on using Lucene Document
>> > classificationhttps://wiki.apache.org/solr/SolrClassification.
>> > From what I understand I need to feed already the category on some
>> > documents. New documents would be classified.
>> >
>> > Is there anything else I can't find?
>> >
>> > Thanks a lot.
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> > http://lucene.472066.n3.nabble.com/Classify-document-using-
>> bag-of-words-tp4326865.html
>> > Sent from the Solr - User mailing list archive at Nabble.com.
>> >
>> --
>> --
>> *John Blythe*
>> Product Manager & Lead Developer
>>
>> 251.605.3071 | john@curvolabs.com
>> www.curvolabs.com
>>
>> 58 Adams Ave
>> Evansville, IN 47713
>>
>
>
Re: Classify document using bag of words
Posted by Sergio García Maroto <ma...@gmail.com>.
Hi John.
thanks for that.
That's actually a good option but I would need the category text on the
field so I can facet on the field and get every category and the number.
On 26 March 2017 at 18:27, John Blythe <jo...@curvolabs.com> wrote:
> You could use keepwords to filter out any other words besides your bag and
> then have a synonym filter that translates the remaining word(s) to a
> corresponding category/classification
>
> On Sun, Mar 26, 2017 at 12:05 PM marotosg <ma...@gmail.com> wrote:
>
> > Hi,
> >
> > I have a very simple use case where I would need to classify a document
> > using a bag of words. Basically if a field within the document contains
> any
> > of the words on my bag then I use a new field to assign a category to the
> > document.
> >
> > Is this something achievable on Solr?
> >
> > I was thinking on using Lucene Document
> > classificationhttps://wiki.apache.org/solr/SolrClassification.
> > From what I understand I need to feed already the category on some
> > documents. New documents would be classified.
> >
> > Is there anything else I can't find?
> >
> > Thanks a lot.
> >
> >
> >
> > --
> > View this message in context:
> > http://lucene.472066.n3.nabble.com/Classify-document-
> using-bag-of-words-tp4326865.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
> >
> --
> --
> *John Blythe*
> Product Manager & Lead Developer
>
> 251.605.3071 | john@curvolabs.com
> www.curvolabs.com
>
> 58 Adams Ave
> Evansville, IN 47713
>
Re: Classify document using bag of words
Posted by John Blythe <jo...@curvolabs.com>.
You could use keepwords to filter out any other words besides your bag and
then have a synonym filter that translates the remaining word(s) to a
corresponding category/classification
On Sun, Mar 26, 2017 at 12:05 PM marotosg <ma...@gmail.com> wrote:
> Hi,
>
> I have a very simple use case where I would need to classify a document
> using a bag of words. Basically if a field within the document contains any
> of the words on my bag then I use a new field to assign a category to the
> document.
>
> Is this something achievable on Solr?
>
> I was thinking on using Lucene Document
> classificationhttps://wiki.apache.org/solr/SolrClassification.
> From what I understand I need to feed already the category on some
> documents. New documents would be classified.
>
> Is there anything else I can't find?
>
> Thanks a lot.
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Classify-document-using-bag-of-words-tp4326865.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
--
--
*John Blythe*
Product Manager & Lead Developer
251.605.3071 | john@curvolabs.com
www.curvolabs.com
58 Adams Ave
Evansville, IN 47713
Re: Classify document using bag of words
Posted by "alessandro.benedetti" <a....@sease.io>.
Hi marotosg,
john's suggestion will definitely work ( I recommend you a copyfield for
that analysis).
What happens in your use case if a word is in common for more than one bag
of word ( if possible at all in your use case)?
Do you expect to get back all the classes ? scored in some way ?
In that case you may need a different approach, and the Solr Document
Classification should help.
At the moment the only available integration is the indexing time one (
which means you don't have control on human validation, Solr is going to
assign the class( or classes) and you just decide the output field.
Documentation was not very up to date, I just updated it [1]
In case you like a different approach ( including human validation), there a
Jira issue for a request handler approach, that could be called by your
indexing application and ask for human feedback before the document is sent
to solr, a contribution is welcome ! [2] .
Cheers
[1] https://wiki.apache.org/solr/SolrClassification
[2] https://issues.apache.org/jira/browse/SOLR-7738
-----
---------------
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director
Sease Ltd. - www.sease.io
--
View this message in context: http://lucene.472066.n3.nabble.com/Classify-document-using-bag-of-words-tp4326865p4326988.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Classify document using bag of words
Posted by Koji Sekiguchi <ko...@rondhuit.com>.
Hi,
I'm not sure that it can help you but I'd like to show you the link of an article
which I wrote about document classification years ago:
Comparing Document Classification Functions of Lucene and Mahout
http://soleami.com/blog/comparing-document-classification-functions-of-lucene-and-mahout.html
Thanks!
--
koji
On 2017/03/27 1:05, marotosg wrote:
> Hi,
>
> I have a very simple use case where I would need to classify a document
> using a bag of words. Basically if a field within the document contains any
> of the words on my bag then I use a new field to assign a category to the
> document.
>
> Is this something achievable on Solr?
>
> I was thinking on using Lucene Document
> classificationhttps://wiki.apache.org/solr/SolrClassification.
>>From what I understand I need to feed already the category on some
> documents. New documents would be classified.
>
> Is there anything else I can't find?
>
> Thanks a lot.
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Classify-document-using-bag-of-words-tp4326865.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>