You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by rajani m <ra...@gmail.com> on 2023/06/01 14:28:07 UTC
Re: Streaming expr as LTR feature
Thank you for links to the docs, they were super helpful and the expression
syntax also works perfectly. Appreciate it.
Just another question inline to the streaming api. Is it possible to invoke
the stream expression via solr local param standard query syntax such as
{!stream expr=select()}. Any specific handler that can allow stream expr go
through standard solr request and perhaps also returns stream response into
standard solr response?
On Wed, May 31, 2023 at 1:53 PM Joel Bernstein <jo...@gmail.com> wrote:
> The select function is documented here:
>
> https://solr.apache.org/guide/solr/latest/query-guide/transform.html
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
>
> On Wed, May 31, 2023 at 1:51 PM Joel Bernstein <jo...@gmail.com> wrote:
>
> > The array function doesn't operate in the way its being used here. Here
> > are the docs on arrays:
> >
> >
> >
> https://solr.apache.org/guide/solr/latest/query-guide/vector-math.html#arrays
> >
> > Using a multi-valued field you could do something like this where test_fs
> > is a multi-valued float field. The select function evaluates arrays
> > automatically so they can be operated on by math expressions in a
> streaming
> > context.
> >
> > select(search(jdata, fl=test_fs),
> > dotProduct(test_fs, test_fs) as p)
> >
> > Which returns:
> >
> > { "result-set": { "docs": [ { "p": 1.9350813535659377 }, { "p":
> > 2.2449532856850816 }, { "p": 1.7212359783803421 }, { "p":
> 2.761290822044021
> > },
> >
> >
> >
> >
> > Joel Bernstein
> > http://joelsolr.blogspot.com/
> >
> >
> > On Wed, May 31, 2023 at 12:59 PM Alessandro Benedetti <
> > a.benedetti@sease.io> wrote:
> >
> >> Hi,
> >> we are working on contributing the possibility of having
> vector-similarity
> >> features, in Apache Solr Learning To Rank.
> >> We are starting from the Lucene contribution of related function
> queries,
> >> which we are close to merging.
> >> Then we'll do the Solr part.
> >>
> >> What you are trying to do has not been tested, it may work but there's
> no
> >> dedicated design for that so it may be quite clunky and expensive.
> >> And by the way, Images are not visible in the mailing list.
> >>
> >> Cheers
> >> --------------------------
> >> *Alessandro Benedetti*
> >> Director @ Sease Ltd.
> >> *Apache Lucene/Solr Committer*
> >> *Apache Solr PMC Member*
> >>
> >> e-mail: a.benedetti@sease.io
> >>
> >>
> >> *Sease* - Information Retrieval Applied
> >> Consulting | Training | Open Source
> >>
> >> Website: Sease.io <http://sease.io/>
> >> LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter
> >> <https://twitter.com/seaseltd> | Youtube
> >> <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github
> >> <https://github.com/seaseltd>
> >>
> >>
> >> On Wed, 31 May 2023 at 16:05, rajani m <ra...@gmail.com> wrote:
> >>
> >> > Validating the expression to begin with, it doesn't work. Vector math
> >> > supports reading from an array of values so I tried the following
> >> > expression.
> >> >
> >> > dotProduct(array(search(v9,
> >> > q="id:1",
> >> > fl="numeric_field_dfd",
> >> > sort="numeric_field_dfd asc",
> >> > qt="/export")),array(2))
> >> >
> >> >
> >> > where numeric_field_dfd - single valued dynamic field double type. I
> >> > tried the multivalued double type assuming that it converts to an
> array
> >> of
> >> > values but it didn't work, so tried the single value to start with.
> >> >
> >> > output of the expression is an exception -
> >> >
> >> > [image: image.png]
> >> >
> >> >
> >> > The value is not null as seen below, so am I wrong in terms of
> >> expression
> >> > syntax then, any suggestions?
> >> >
> >> >
> >> > [image: image.png]
> >> >
> >> >
> >> >
> >> > On Tue, May 30, 2023 at 4:47 PM rajani m <ra...@gmail.com>
> wrote:
> >> >
> >> >> Hi Solr Users,
> >> >>
> >> >> Does LTR Solr Feature
> >> >> <
> >>
> https://solr.apache.org/guide/8_7/learning-to-rank.html#feature-engineering
> >
> >> support
> >> >> streaming expressions? Steaming expr supports vector math
> >> >> <
> >>
> https://solr.apache.org/guide/7_5/vector-math.html#dot-product-and-cosine-similarity
> >> >,
> >> >> I am trying to configure stream apis vector math as a solr feature
> >> which
> >> >> would fetch a vector from a document field and another from query
> >> param and
> >> >> compute cosine or dot product.
> >> >>
> >> >> For example, a LTR feature definition that would look like below, is
> >> this
> >> >> supported? Does LTR solr feature support parsing streaming api
> >> requests and
> >> >> its somewhat unique response that is not same as standard solr
> >> response?
> >> >>
> >> >>
> >> >> {
> >> >> "name": "vector_sim_score",
> >> >> "class": "org.apache.solr.ltr.feature.SolrFeature",
> >> >> "params": {
> >> >> "q":
> >> "expr=dotProduct(search(collection_name,q="id:$uniq_id",fl="doc_vector",
> >> sort="from asc", qt="/export"), ${query_vector})"
> >> >> },
> >> >> "store": "v1_feature_store"
> >> >> }
> >> >>
> >> >>
> >>
> >
>
Re: Streaming expr as LTR feature
Posted by Joel Bernstein <jo...@gmail.com>.
There currently is not a way to inline a streaming expression into a
traditional Solr search. Part of the reason why is that they have different
performance and design goals. A standard search is designed to scale to a
high qps with sub-second performance. The design of Solr's
traditional search supports that. Streaming expressions were never designed
to perform at scale like that. So it would be hard to integrate streaming
expressions into a system that required high qps and sub-second
performance.
Joel Bernstein
http://joelsolr.blogspot.com/
On Thu, Jun 1, 2023 at 10:28 AM rajani m <ra...@gmail.com> wrote:
> Thank you for links to the docs, they were super helpful and the expression
> syntax also works perfectly. Appreciate it.
> Just another question inline to the streaming api. Is it possible to invoke
> the stream expression via solr local param standard query syntax such as
> {!stream expr=select()}. Any specific handler that can allow stream expr go
> through standard solr request and perhaps also returns stream response into
> standard solr response?
>
> On Wed, May 31, 2023 at 1:53 PM Joel Bernstein <jo...@gmail.com> wrote:
>
> > The select function is documented here:
> >
> > https://solr.apache.org/guide/solr/latest/query-guide/transform.html
> >
> > Joel Bernstein
> > http://joelsolr.blogspot.com/
> >
> >
> > On Wed, May 31, 2023 at 1:51 PM Joel Bernstein <jo...@gmail.com>
> wrote:
> >
> > > The array function doesn't operate in the way its being used here. Here
> > > are the docs on arrays:
> > >
> > >
> > >
> >
> https://solr.apache.org/guide/solr/latest/query-guide/vector-math.html#arrays
> > >
> > > Using a multi-valued field you could do something like this where
> test_fs
> > > is a multi-valued float field. The select function evaluates arrays
> > > automatically so they can be operated on by math expressions in a
> > streaming
> > > context.
> > >
> > > select(search(jdata, fl=test_fs),
> > > dotProduct(test_fs, test_fs) as p)
> > >
> > > Which returns:
> > >
> > > { "result-set": { "docs": [ { "p": 1.9350813535659377 }, { "p":
> > > 2.2449532856850816 }, { "p": 1.7212359783803421 }, { "p":
> > 2.761290822044021
> > > },
> > >
> > >
> > >
> > >
> > > Joel Bernstein
> > > http://joelsolr.blogspot.com/
> > >
> > >
> > > On Wed, May 31, 2023 at 12:59 PM Alessandro Benedetti <
> > > a.benedetti@sease.io> wrote:
> > >
> > >> Hi,
> > >> we are working on contributing the possibility of having
> > vector-similarity
> > >> features, in Apache Solr Learning To Rank.
> > >> We are starting from the Lucene contribution of related function
> > queries,
> > >> which we are close to merging.
> > >> Then we'll do the Solr part.
> > >>
> > >> What you are trying to do has not been tested, it may work but there's
> > no
> > >> dedicated design for that so it may be quite clunky and expensive.
> > >> And by the way, Images are not visible in the mailing list.
> > >>
> > >> Cheers
> > >> --------------------------
> > >> *Alessandro Benedetti*
> > >> Director @ Sease Ltd.
> > >> *Apache Lucene/Solr Committer*
> > >> *Apache Solr PMC Member*
> > >>
> > >> e-mail: a.benedetti@sease.io
> > >>
> > >>
> > >> *Sease* - Information Retrieval Applied
> > >> Consulting | Training | Open Source
> > >>
> > >> Website: Sease.io <http://sease.io/>
> > >> LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter
> > >> <https://twitter.com/seaseltd> | Youtube
> > >> <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github
> > >> <https://github.com/seaseltd>
> > >>
> > >>
> > >> On Wed, 31 May 2023 at 16:05, rajani m <ra...@gmail.com> wrote:
> > >>
> > >> > Validating the expression to begin with, it doesn't work. Vector
> math
> > >> > supports reading from an array of values so I tried the following
> > >> > expression.
> > >> >
> > >> > dotProduct(array(search(v9,
> > >> > q="id:1",
> > >> > fl="numeric_field_dfd",
> > >> > sort="numeric_field_dfd asc",
> > >> > qt="/export")),array(2))
> > >> >
> > >> >
> > >> > where numeric_field_dfd - single valued dynamic field double type.
> I
> > >> > tried the multivalued double type assuming that it converts to an
> > array
> > >> of
> > >> > values but it didn't work, so tried the single value to start with.
> > >> >
> > >> > output of the expression is an exception -
> > >> >
> > >> > [image: image.png]
> > >> >
> > >> >
> > >> > The value is not null as seen below, so am I wrong in terms of
> > >> expression
> > >> > syntax then, any suggestions?
> > >> >
> > >> >
> > >> > [image: image.png]
> > >> >
> > >> >
> > >> >
> > >> > On Tue, May 30, 2023 at 4:47 PM rajani m <ra...@gmail.com>
> > wrote:
> > >> >
> > >> >> Hi Solr Users,
> > >> >>
> > >> >> Does LTR Solr Feature
> > >> >> <
> > >>
> >
> https://solr.apache.org/guide/8_7/learning-to-rank.html#feature-engineering
> > >
> > >> support
> > >> >> streaming expressions? Steaming expr supports vector math
> > >> >> <
> > >>
> >
> https://solr.apache.org/guide/7_5/vector-math.html#dot-product-and-cosine-similarity
> > >> >,
> > >> >> I am trying to configure stream apis vector math as a solr feature
> > >> which
> > >> >> would fetch a vector from a document field and another from query
> > >> param and
> > >> >> compute cosine or dot product.
> > >> >>
> > >> >> For example, a LTR feature definition that would look like below,
> is
> > >> this
> > >> >> supported? Does LTR solr feature support parsing streaming api
> > >> requests and
> > >> >> its somewhat unique response that is not same as standard solr
> > >> response?
> > >> >>
> > >> >>
> > >> >> {
> > >> >> "name": "vector_sim_score",
> > >> >> "class": "org.apache.solr.ltr.feature.SolrFeature",
> > >> >> "params": {
> > >> >> "q":
> > >>
> "expr=dotProduct(search(collection_name,q="id:$uniq_id",fl="doc_vector",
> > >> sort="from asc", qt="/export"), ${query_vector})"
> > >> >> },
> > >> >> "store": "v1_feature_store"
> > >> >> }
> > >> >>
> > >> >>
> > >>
> > >
> >
>