[jira] [Comment Edited] (SOLR-8492) Add LogisticRegressionQuery and LogitStream

["Alessandro Benedetti (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-8492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15226032#comment-15226032 ] 

Alessandro Benedetti edited comment on SOLR-8492 at 4/5/16 10:20 AM:
---------------------------------------------------------------------

Guys, 
great work !
I have been following the discussion and it is very interesting.
Can you give an estimation of the maturity of this patch ?
I didn't have the chance to take a look to the code yet, but it seems it is almost mature to be contributed !
Am i right ?
This would open nice scenarios for classification(as Joel previously pointed out in a separate issue), a score predictor and possibly re-ranking  in Solr ( as we are de facto allowing to build a model that would be able to assign the score for the document) .

It comes naturally to me to think to add support for categorical features ( as I think at the moment only ordinal features are supported (double/integer)).
Would say using one-hot encoding or other approaches for feature-decomposition !
Well done again !
As soon as I have some time I will take a look to the code !
Cheers



was (Author: alessandro.benedetti):
Guys, 
great work !
I have been following the discussion and it is very interesting.
Can you give an estimation of the maturity of this patch ?
I didn't have the chance to take a look to the code yet, but it seems it is almost mature to be contributed !
Am i right ?
This would open nice scenarios for classification, a score predictor and possibly re-ranking  in Solr ( as we are de facto allowing to build a model that would be able to assign the score for the document) .

It comes naturally to me to think to add support for categorical features ( as I think at the moment only ordinal features are supported (double/integer)).
Would say using one-hot encoding or other approaches for feature-decomposition !
Well done again !
As soon as I have some time I will take a look to the code !
Cheers


> Add LogisticRegressionQuery and LogitStream
> -------------------------------------------
>
>                 Key: SOLR-8492
>                 URL: https://issues.apache.org/jira/browse/SOLR-8492
>             Project: Solr
>          Issue Type: New Feature
>            Reporter: Joel Bernstein
>             Fix For: 6.1
>
>         Attachments: SOLR-8492.diff, SOLR-8492.diff, SOLR-8492.patch, SOLR-8492.patch, SOLR-8492.patch, SOLR-8492.patch, SOLR-8492.patch, SOLR-8492.patch, logit.csv
>
>
> This ticket is to add a new query called a LogisticRegressionQuery (LRQ).
> The LRQ extends AnalyticsQuery (http://joelsolr.blogspot.com/2015/12/understanding-solrs-analyticsquery.html) and returns a DelegatingCollector that implements a Stochastic Gradient Descent (SGD) optimizer for Logistic Regression.
> This ticket also adds the LogitStream which leverages Streaming Expressions to provide iteration over the shards. Each call to LogitStream.read() calls down to the shards and executes the LogisticRegressionQuery. The model data is collected from the shards and the weights are averaged and sent back to the shards with the next iteration. Each call to read() returns a Tuple with the averaged weights and error from the shards. With this approach the LogitStream streams the changing model back to the client after each iteration.
> The LogitStream will return the EOF Tuple when it reaches the defined maxIterations. When sent as a Streaming Expression to the Stream handler this provides parallel iterative behavior. This same approach can be used to implement other parallel iterative algorithms.
> The initial patch has  a test which simply tests the mechanics of the iteration. More work will need to be done to ensure the SGD is properly implemented. The distributed approach of the SGD will also need to be reviewed.  
> This implementation is designed for use cases with a small number of features because each feature is it's own discreet field.
> An implementation which supports a higher number of features would be possible by packing features into a byte array and storing as binary DocValues.
> This implementation is designed to support a large sample set. With a large number of shards, a sample set into the billions may be possible.
> sample Streaming Expression Syntax:
> {code}
> logit(collection1, features="a,b,c,d,e,f" outcome="x" maxIterations="80")
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org