You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Tamás Barta <ba...@gmail.com> on 2016/04/01 16:19:11 UTC
Sorting question
Hi,
I have a problem and I don't know how should I solve it in Solr.
I have products indexed. Every product can be in lists. It is possible that
a product isn't in any list or it is in multiple list.
In a list the products are ordered. I would like to search for products in
a specified list with the correct order.
Earlier I tried to create a field for every list the product is in and
these fields stored the index value of the product in that list.
For example: list_23=1, list_841=8, ...
After that I could do a query for list N: product where list_N exists order
by list_N asc
In this case the index size were extremely large because there are lot of
lists and I guess every document stored a value for every field (it was
long time ago, I don't know if it would be still a problem in current
version of Solr)
Do you have any idea how could I solve this problem?
Thanks,
Tamas
Re: Sorting question
Posted by Tamás Barta <ba...@gmail.com>.
Sorry I don't know what you mean.
If "listpos" field contains multiple values like "list=pos" then is it
possible to order by field value where that field value fits a query?
For example list 1 contains: p1 and p2, list 2 contains p2 and p1 in this
order, then
p1 document has a listpos field with values "1=1" and "2=2"
p2 document has a listpos fiels with values "1=2" and "2=1"
And if list 1 should be displayed then i should say to solr: sort by field
value where listpos:1=*
Because i need products which is in list1, and i want to sort by this
matching term.
2016. ápr. 1. 19:35 ezt írta ( <bi...@gmail.com>):
> Put the match into 2 separate fields and index it. Then sort in Solr by
> the 2 fields is one way
>
> Bill Bell
> Sent from mobile
>
>
> > On Apr 1, 2016, at 11:15 AM, John Bickerstaff <jo...@johnbickerstaff.com>
> wrote:
> >
> > Just to be clear - I don't mean who requests the list (application or
> user)
> > I mean what "rule" determines the ordering of the list?
> >
> > Or, is there even a rule of any kind?
> >
> > In other words, does a user arbitrarily decide that documentA, documentF,
> > and documentW should be on a list of their own? For reasons known only
> to
> > the user?
> >
> > Or - does the ordering of the list depend on some piece of data? (like a
> > date, or a manufacturer, or a price range or any other piece of "hard"
> data)
> >
> > ===
> >
> > To give an example from what I'm working on right now --
> >
> > My subject matter experts have given me a rule that says:
> >
> > *Documents of content_type "bar" should come higher in the results than
> > documents of content_type "foo".*
> >
> > PsuedoCode: If (content_type == bar) then put this doc highest in the
> > results. If (content_type == foo) put those docs after the "bar"
> > content_type docs.
> >
> >
> > This is an example of the ordering being tied to a specific piece of data
> > which I can manipulate in a "sub query" (that's probably the wrong
> term...)
> >
> >
> > This isn't exactly what you're doing, but it's close -- IF you have rules
> > you can express clearly in this way...
> >
> > ---
> >
> > Also, I'm confused a little by your statement that SOLR does the
> filtering
> > and pagination, thus you can't sort the documents after Solr returns
> them...
> >
> > My mental model is that you ask Solr for all the documents that match a
> > certain criteria. Solr returns that "set" of documents and then for your
> > list, you sort those document titles or ID's according to some rule --
> > possibly in the javascript on the web page... But perhaps I'm not
> > understanding your situation well enough...
> >
> > Oh - are you perhaps saying that your ONLY criteria for getting these
> > documents is the list number? That would make sense, although there may
> > still be room for sorting based on some kind of logic / data point
> outside
> > of SOlR. You could get all the documents associated to list #4, and then
> > sort them based on some hard data point they all contain. At the very
> > least, your listpos "array" becomes simpler...
> >
> > What does your query currently look like?
> >
> >> On Fri, Apr 1, 2016 at 10:51 AM, Tamás Barta <ba...@gmail.com>
> wrote:
> >>
> >> Some of the lists are created by users and some are generated by
> >> applications, it doesn't matter.
> >>
> >> It would be fine to solve it in Solr because Solr does the work of
> >> filtering and pagination. If sorting were done outside than I would
> have to
> >> read every document from Solr to sort them. It is not an option, I have
> to
> >> query onle one page.
> >>
> >> I don't understand how to solve it using subqueries.
> >> 2016. ápr. 1. 18:42 ezt írta ("John Bickerstaff" <
> john@johnbickerstaff.com
> >>> ):
> >>
> >>> Specifically, what drives the position in the list? Is it arbitrary or
> >> is
> >>> it driven by some piece of data?
> >>>
> >>> If data-driven - code could do the sorting based on that data...
> >> separate
> >>> from SOLR...
> >>>
> >>> Alternatively, if the data point exists in SOLR, a "sub-query" might be
> >>> used to get the right sort order on the items returned by the "main"
> >>> search... Possibly without having to resort to the clunky-feeling
> >> listpos
> >>> multivalued field...
> >>>
> >>>> On Fri, Apr 1, 2016 at 10:32 AM, Tamás Barta <ba...@gmail.com>
> >>> wrote:
> >>>
> >>>> For example I have to display sellable products which are in list X in
> >>> the
> >>>> correct order.
> >>>>
> >>>> If I add a "status" and "list" (multivalued) fields to every document
> >>>> (products), then I can execute a query: status:sellable AND list:X,
> >>> where X
> >>>> is the ID of the list. The list field contains IDs of the list in
> which
> >>> the
> >>>> product is in.
> >>>>
> >>>> The problem is that I can't sort the result. A product has different
> >>> index
> >>>> for every list.
> >>>>
> >>>> Is it clear now?
> >>>>
> >>>> Earlier I added a "listpos" field with multivalue content, for
> example:
> >>>>
> >>>> 1:23
> >>>> 2:4
> >>>>
> >>>> Which means that this product is in position 23 in list 1 and it is in
> >>>> position 4 in list 2. After that I created a custom comparator which
> >>> parses
> >>>> field values to get index for the specified list and sorts by that
> >> index.
> >>>>
> >>>> But I didn't like that solution much. I wish there would be a better
> >>>> solution. In SolrJ unfortunately I can't find an API to set custom
> >>>> comparator like I did in Lucene. So I don't know how to solve this
> >>> problem
> >>>> in Solr.
> >>>>
> >>>> Thanks,
> >>>> Tamás
> >>>> 2016. ápr. 1. 17:25 ezt írta ("Alessandro Benedetti" <
> >>>> abenedetti@apache.org
> >>>>> ):
> >>>>
> >>>>> I think this is a classic XY Problem , you are trying to solve X with
> >>> Y ,
> >>>>> and you are asking us about Y .
> >>>>> Could you describe us what is your X problem ? What are you trying to
> >>> do
> >>>>> with this ordered lists ?
> >>>>>
> >>>>> If not I would add a field to the product called :
> >>>>> list_position ( or a similar name) of type geo point (x,y) .
> >>>>> X could be your list ID
> >>>>> Y the position.
> >>>>> Then you can play with spatial search, to get what you want.
> >>>>>
> >>>>> But again, let's try to solve X.
> >>>>>
> >>>>> Cheers
> >>
>
Re: Sorting question
Posted by bi...@gmail.com.
Put the match into 2 separate fields and index it. Then sort in Solr by the 2 fields is one way
Bill Bell
Sent from mobile
> On Apr 1, 2016, at 11:15 AM, John Bickerstaff <jo...@johnbickerstaff.com> wrote:
>
> Just to be clear - I don't mean who requests the list (application or user)
> I mean what "rule" determines the ordering of the list?
>
> Or, is there even a rule of any kind?
>
> In other words, does a user arbitrarily decide that documentA, documentF,
> and documentW should be on a list of their own? For reasons known only to
> the user?
>
> Or - does the ordering of the list depend on some piece of data? (like a
> date, or a manufacturer, or a price range or any other piece of "hard" data)
>
> ===
>
> To give an example from what I'm working on right now --
>
> My subject matter experts have given me a rule that says:
>
> *Documents of content_type "bar" should come higher in the results than
> documents of content_type "foo".*
>
> PsuedoCode: If (content_type == bar) then put this doc highest in the
> results. If (content_type == foo) put those docs after the "bar"
> content_type docs.
>
>
> This is an example of the ordering being tied to a specific piece of data
> which I can manipulate in a "sub query" (that's probably the wrong term...)
>
>
> This isn't exactly what you're doing, but it's close -- IF you have rules
> you can express clearly in this way...
>
> ---
>
> Also, I'm confused a little by your statement that SOLR does the filtering
> and pagination, thus you can't sort the documents after Solr returns them...
>
> My mental model is that you ask Solr for all the documents that match a
> certain criteria. Solr returns that "set" of documents and then for your
> list, you sort those document titles or ID's according to some rule --
> possibly in the javascript on the web page... But perhaps I'm not
> understanding your situation well enough...
>
> Oh - are you perhaps saying that your ONLY criteria for getting these
> documents is the list number? That would make sense, although there may
> still be room for sorting based on some kind of logic / data point outside
> of SOlR. You could get all the documents associated to list #4, and then
> sort them based on some hard data point they all contain. At the very
> least, your listpos "array" becomes simpler...
>
> What does your query currently look like?
>
>> On Fri, Apr 1, 2016 at 10:51 AM, Tamás Barta <ba...@gmail.com> wrote:
>>
>> Some of the lists are created by users and some are generated by
>> applications, it doesn't matter.
>>
>> It would be fine to solve it in Solr because Solr does the work of
>> filtering and pagination. If sorting were done outside than I would have to
>> read every document from Solr to sort them. It is not an option, I have to
>> query onle one page.
>>
>> I don't understand how to solve it using subqueries.
>> 2016. ápr. 1. 18:42 ezt írta ("John Bickerstaff" <john@johnbickerstaff.com
>>> ):
>>
>>> Specifically, what drives the position in the list? Is it arbitrary or
>> is
>>> it driven by some piece of data?
>>>
>>> If data-driven - code could do the sorting based on that data...
>> separate
>>> from SOLR...
>>>
>>> Alternatively, if the data point exists in SOLR, a "sub-query" might be
>>> used to get the right sort order on the items returned by the "main"
>>> search... Possibly without having to resort to the clunky-feeling
>> listpos
>>> multivalued field...
>>>
>>>> On Fri, Apr 1, 2016 at 10:32 AM, Tamás Barta <ba...@gmail.com>
>>> wrote:
>>>
>>>> For example I have to display sellable products which are in list X in
>>> the
>>>> correct order.
>>>>
>>>> If I add a "status" and "list" (multivalued) fields to every document
>>>> (products), then I can execute a query: status:sellable AND list:X,
>>> where X
>>>> is the ID of the list. The list field contains IDs of the list in which
>>> the
>>>> product is in.
>>>>
>>>> The problem is that I can't sort the result. A product has different
>>> index
>>>> for every list.
>>>>
>>>> Is it clear now?
>>>>
>>>> Earlier I added a "listpos" field with multivalue content, for example:
>>>>
>>>> 1:23
>>>> 2:4
>>>>
>>>> Which means that this product is in position 23 in list 1 and it is in
>>>> position 4 in list 2. After that I created a custom comparator which
>>> parses
>>>> field values to get index for the specified list and sorts by that
>> index.
>>>>
>>>> But I didn't like that solution much. I wish there would be a better
>>>> solution. In SolrJ unfortunately I can't find an API to set custom
>>>> comparator like I did in Lucene. So I don't know how to solve this
>>> problem
>>>> in Solr.
>>>>
>>>> Thanks,
>>>> Tamás
>>>> 2016. ápr. 1. 17:25 ezt írta ("Alessandro Benedetti" <
>>>> abenedetti@apache.org
>>>>> ):
>>>>
>>>>> I think this is a classic XY Problem , you are trying to solve X with
>>> Y ,
>>>>> and you are asking us about Y .
>>>>> Could you describe us what is your X problem ? What are you trying to
>>> do
>>>>> with this ordered lists ?
>>>>>
>>>>> If not I would add a field to the product called :
>>>>> list_position ( or a similar name) of type geo point (x,y) .
>>>>> X could be your list ID
>>>>> Y the position.
>>>>> Then you can play with spatial search, to get what you want.
>>>>>
>>>>> But again, let's try to solve X.
>>>>>
>>>>> Cheers
>>
Re: Sorting question
Posted by John Bickerstaff <jo...@johnbickerstaff.com>.
Oh - and if you send a copy of your query - please include a human-readable
version of what your intent is...
Something like: Find all the documents that have "blue" in the color field
in addition to searching the title field for the user's search term..."
...Or whatever your intent is for this search.
On Fri, Apr 1, 2016 at 11:15 AM, John Bickerstaff <jo...@johnbickerstaff.com>
wrote:
> Just to be clear - I don't mean who requests the list (application or
> user) I mean what "rule" determines the ordering of the list?
>
> Or, is there even a rule of any kind?
>
> In other words, does a user arbitrarily decide that documentA, documentF,
> and documentW should be on a list of their own? For reasons known only to
> the user?
>
> Or - does the ordering of the list depend on some piece of data? (like a
> date, or a manufacturer, or a price range or any other piece of "hard" data)
>
> ===
>
> To give an example from what I'm working on right now --
>
> My subject matter experts have given me a rule that says:
>
> *Documents of content_type "bar" should come higher in the results than
> documents of content_type "foo".*
>
> PsuedoCode: If (content_type == bar) then put this doc highest in the
> results. If (content_type == foo) put those docs after the "bar"
> content_type docs.
>
>
> This is an example of the ordering being tied to a specific piece of data
> which I can manipulate in a "sub query" (that's probably the wrong term...)
>
>
> This isn't exactly what you're doing, but it's close -- IF you have rules
> you can express clearly in this way...
>
> ---
>
> Also, I'm confused a little by your statement that SOLR does the filtering
> and pagination, thus you can't sort the documents after Solr returns them...
>
> My mental model is that you ask Solr for all the documents that match a
> certain criteria. Solr returns that "set" of documents and then for your
> list, you sort those document titles or ID's according to some rule --
> possibly in the javascript on the web page... But perhaps I'm not
> understanding your situation well enough...
>
> Oh - are you perhaps saying that your ONLY criteria for getting these
> documents is the list number? That would make sense, although there may
> still be room for sorting based on some kind of logic / data point outside
> of SOlR. You could get all the documents associated to list #4, and then
> sort them based on some hard data point they all contain. At the very
> least, your listpos "array" becomes simpler...
>
> What does your query currently look like?
>
> On Fri, Apr 1, 2016 at 10:51 AM, Tamás Barta <ba...@gmail.com> wrote:
>
>> Some of the lists are created by users and some are generated by
>> applications, it doesn't matter.
>>
>> It would be fine to solve it in Solr because Solr does the work of
>> filtering and pagination. If sorting were done outside than I would have
>> to
>> read every document from Solr to sort them. It is not an option, I have to
>> query onle one page.
>>
>> I don't understand how to solve it using subqueries.
>> 2016. ápr. 1. 18:42 ezt írta ("John Bickerstaff" <
>> john@johnbickerstaff.com
>> >):
>>
>> > Specifically, what drives the position in the list? Is it arbitrary or
>> is
>> > it driven by some piece of data?
>> >
>> > If data-driven - code could do the sorting based on that data...
>> separate
>> > from SOLR...
>> >
>> > Alternatively, if the data point exists in SOLR, a "sub-query" might be
>> > used to get the right sort order on the items returned by the "main"
>> > search... Possibly without having to resort to the clunky-feeling
>> listpos
>> > multivalued field...
>> >
>> > On Fri, Apr 1, 2016 at 10:32 AM, Tamás Barta <ba...@gmail.com>
>> wrote:
>> >
>> > > For example I have to display sellable products which are in list X in
>> > the
>> > > correct order.
>> > >
>> > > If I add a "status" and "list" (multivalued) fields to every document
>> > > (products), then I can execute a query: status:sellable AND list:X,
>> > where X
>> > > is the ID of the list. The list field contains IDs of the list in
>> which
>> > the
>> > > product is in.
>> > >
>> > > The problem is that I can't sort the result. A product has different
>> > index
>> > > for every list.
>> > >
>> > > Is it clear now?
>> > >
>> > > Earlier I added a "listpos" field with multivalue content, for
>> example:
>> > >
>> > > 1:23
>> > > 2:4
>> > >
>> > > Which means that this product is in position 23 in list 1 and it is in
>> > > position 4 in list 2. After that I created a custom comparator which
>> > parses
>> > > field values to get index for the specified list and sorts by that
>> index.
>> > >
>> > > But I didn't like that solution much. I wish there would be a better
>> > > solution. In SolrJ unfortunately I can't find an API to set custom
>> > > comparator like I did in Lucene. So I don't know how to solve this
>> > problem
>> > > in Solr.
>> > >
>> > > Thanks,
>> > > Tamás
>> > > 2016. ápr. 1. 17:25 ezt írta ("Alessandro Benedetti" <
>> > > abenedetti@apache.org
>> > > >):
>> > >
>> > > > I think this is a classic XY Problem , you are trying to solve X
>> with
>> > Y ,
>> > > > and you are asking us about Y .
>> > > > Could you describe us what is your X problem ? What are you trying
>> to
>> > do
>> > > > with this ordered lists ?
>> > > >
>> > > > If not I would add a field to the product called :
>> > > > list_position ( or a similar name) of type geo point (x,y) .
>> > > > X could be your list ID
>> > > > Y the position.
>> > > > Then you can play with spatial search, to get what you want.
>> > > >
>> > > > But again, let's try to solve X.
>> > > >
>> > > > Cheers
>> > > >
>> > >
>> >
>>
>
>
Re: Sorting question
Posted by John Bickerstaff <jo...@johnbickerstaff.com>.
Thanks for sharing the solution Tamas -- I was hoping you'd let us know...
On Mon, Apr 4, 2016 at 8:05 AM, Tamás Barta <ba...@gmail.com> wrote:
> Hi,
>
> FYI: the final solution I found is that I created a custom
> "listpos(fieldName, listId)" function and now I can display a sorted list
> via:
>
> fq=listid_s:378
> sort=listpos(listpos_s,378) asc
>
> Regards,
> Tamas
>
> On Fri, Apr 1, 2016 at 8:55 PM, John Bickerstaff <john@johnbickerstaff.com
> >
> wrote:
>
> > Tamas,
> >
> > This feels a bit like a "user favorites" problem.
> >
> > I did a little searching and found this... Don't know if it will help,
> but
> > when I'm looking for stuff like this I find it helps to try to come up
> with
> > generic or different descriptions of my problem and go search those as
> > well...
> >
> >
> >
> http://stackoverflow.com/questions/3931827/solr-merging-results-of-2-cores-into-only-those-results-that-have-a-matching-fie
> >
> > On Fri, Apr 1, 2016 at 12:40 PM, John Bickerstaff <
> > john@johnbickerstaff.com>
> > wrote:
> >
> > > Tamas,
> > >
> > > I'm brainstorming here - not being careful, just throwing out ideas...
> > >
> > > One thing that comes up is a separate document in SOLR - one doc for
> each
> > > list.
> > >
> > > If a user adds a doc to their list, that doc's id gets added to this
> > other
> > > type of document...
> > >
> > > So, a document with the title "List 1" would have a multivalue field of
> > > ID's and the list order number like so:
> > >
> > > ID List Position
> > > _________________
> > > doc1 ID : 1
> > > doc2 ID: 2
> > > doc3 ID: 3
> > >
> > > and so on... The big problem I see with this is keeping it organized
> > > correctly. More code would have to be written to handle this when the
> > user
> > > does any kind of "crud" on the list...
> > >
> > > I'm pretty sure there's a way to write a query that uses that list to
> > > properly order the items returned by your primary search, although I
> > > haven't written such a query yet.
> > >
> > > If you have the luxury of NOT being in production yet with this system,
> > > I'd seriously consider pushing to keep application metadata OUT of your
> > > product information store. This particular problem (of ordering the
> > > results based on arbitrary user choices) might be more easily handled
> > via a
> > > separate step that queries a relational database to handle list order -
> > > once Solr gives you the documents that match the query and the user's
> > list
> > > number...
> > >
> > > Even if you can't use another relational data store - keeping that
> > > metadata out of your individual product documents could be argued to
> be a
> > > good design idea...
> > >
> > > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >
> > > Here's an alternative brainstorm...
> > >
> > > Where does the user data live? What about putting the information
> about
> > > the order of document ID's in the User's lists with the User? Then you
> > can
> > > get all documents that match the search terms and are on List X from
> > Solr -
> > > and then sort them by ID based on the data associated with the User (a
> > list
> > > of ID's, in order)
> > >
> > > There is even a way to write a plugin that will go after external data
> to
> > > help sort Solr documents, although I'm guessing you'd rather avoid
> > that...
> > >
> > >
> > >
> > > On Fri, Apr 1, 2016 at 11:59 AM, John Bickerstaff <
> > > john@johnbickerstaff.com> wrote:
> > >
> > >> OK - I get it. List order is totally arbitrary and cannot be tied to
> an
> > >> hard data point.
> > >>
> > >> I'll have to think - Perhaps billnbell's solution will help, although
> > I'm
> > >> not totally sure I understand that suggestion yet.
> > >>
> > >> At this point, you could get all the documents for List X that match
> the
> > >> search terms. The next problem is sorting. If you have the listpos
> > field
> > >> too, you could use that, and some regex to find the proper order for
> > these
> > >> documents before displaying them (in code I mean) but of course that
> > means
> > >> you need some kind of "interceptor" to deal with this before the
> results
> > >> are displayed.
> > >>
> > >> If I had enough control to do this in code, behind the scenes, I'd
> grab
> > >> that second part of the listops field, put it into a variable on each
> > >> object and then sort by that. Then I'd return the entire list to the
> > UI.
> > >>
> > >> I understand that if you could get SOLR to do it all, that would be
> > >> ideal... There is the possibility of writing some new code and
> > plugging it
> > >> in to Solr, but I'm guessing you don't want to go that far.. As a
> final
> > >> step in the process, with custom code to consume the listpos entry,
> > sorting
> > >> these would be fairly straightforward. I'm not sure how you get away
> > from
> > >> the lispos multivalue field however...
> > >>
> > >> I'll keep thinking...
> > >>
> > >> On Fri, Apr 1, 2016 at 11:26 AM, Tamás Barta <ba...@gmail.com>
> > >> wrote:
> > >>
> > >>> So, the list order is determined by the user. The user creates a
> list,
> > >>> adds
> > >>> products to it and i have to display these list using filters and
> > >>> pagination.
> > >>>
> > >>> Let's assume there is list with 10000 products in it. In the website
> > >>> where
> > >>> i display the list only 50 products are displayed in a page. So if i
> > >>> could
> > >>> query solr to give me products from list X, ordered as user defined,
> > but
> > >>> only products with some criteria (status, amount, ..) from offset and
> > 50
> > >>> rows then it would be perfect and fast. If ordering would be outside
> of
> > >>> solr then i have to retrive almost every 10000 documents from solr (a
> > bit
> > >>> less if filtered) to order them and display the page of 50 products.
> > >>> 2016. ápr. 1. 19:15 ezt írta ("John Bickerstaff" <
> > >>> john@johnbickerstaff.com
> > >>> >):
> > >>>
> > >>> > Just to be clear - I don't mean who requests the list (application
> or
> > >>> user)
> > >>> > I mean what "rule" determines the ordering of the list?
> > >>> >
> > >>> > Or, is there even a rule of any kind?
> > >>> >
> > >>> > In other words, does a user arbitrarily decide that documentA,
> > >>> documentF,
> > >>> > and documentW should be on a list of their own? For reasons known
> > >>> only to
> > >>> > the user?
> > >>> >
> > >>> > Or - does the ordering of the list depend on some piece of data?
> > >>> (like a
> > >>> > date, or a manufacturer, or a price range or any other piece of
> > "hard"
> > >>> > data)
> > >>> >
> > >>> > ===
> > >>> >
> > >>> > To give an example from what I'm working on right now --
> > >>> >
> > >>> > My subject matter experts have given me a rule that says:
> > >>> >
> > >>> > *Documents of content_type "bar" should come higher in the results
> > >>> than
> > >>> > documents of content_type "foo".*
> > >>> >
> > >>> > PsuedoCode: If (content_type == bar) then put this doc highest in
> the
> > >>> > results. If (content_type == foo) put those docs after the "bar"
> > >>> > content_type docs.
> > >>> >
> > >>> >
> > >>> > This is an example of the ordering being tied to a specific piece
> of
> > >>> data
> > >>> > which I can manipulate in a "sub query" (that's probably the wrong
> > >>> > term...)
> > >>> >
> > >>> >
> > >>> > This isn't exactly what you're doing, but it's close -- IF you have
> > >>> rules
> > >>> > you can express clearly in this way...
> > >>> >
> > >>> > ---
> > >>> >
> > >>> > Also, I'm confused a little by your statement that SOLR does the
> > >>> filtering
> > >>> > and pagination, thus you can't sort the documents after Solr
> returns
> > >>> > them...
> > >>> >
> > >>> > My mental model is that you ask Solr for all the documents that
> > match a
> > >>> > certain criteria. Solr returns that "set" of documents and then
> for
> > >>> your
> > >>> > list, you sort those document titles or ID's according to some rule
> > --
> > >>> > possibly in the javascript on the web page... But perhaps I'm not
> > >>> > understanding your situation well enough...
> > >>> >
> > >>> > Oh - are you perhaps saying that your ONLY criteria for getting
> these
> > >>> > documents is the list number? That would make sense, although
> there
> > >>> may
> > >>> > still be room for sorting based on some kind of logic / data point
> > >>> outside
> > >>> > of SOlR. You could get all the documents associated to list #4,
> and
> > >>> then
> > >>> > sort them based on some hard data point they all contain. At the
> > very
> > >>> > least, your listpos "array" becomes simpler...
> > >>> >
> > >>> > What does your query currently look like?
> > >>> >
> > >>> > On Fri, Apr 1, 2016 at 10:51 AM, Tamás Barta <bartatamas@gmail.com
> >
> > >>> wrote:
> > >>> >
> > >>> > > Some of the lists are created by users and some are generated by
> > >>> > > applications, it doesn't matter.
> > >>> > >
> > >>> > > It would be fine to solve it in Solr because Solr does the work
> of
> > >>> > > filtering and pagination. If sorting were done outside than I
> would
> > >>> have
> > >>> > to
> > >>> > > read every document from Solr to sort them. It is not an option,
> I
> > >>> have
> > >>> > to
> > >>> > > query onle one page.
> > >>> > >
> > >>> > > I don't understand how to solve it using subqueries.
> > >>> > > 2016. ápr. 1. 18:42 ezt írta ("John Bickerstaff" <
> > >>> > john@johnbickerstaff.com
> > >>> > > >):
> > >>> > >
> > >>> > > > Specifically, what drives the position in the list? Is it
> > >>> arbitrary or
> > >>> > > is
> > >>> > > > it driven by some piece of data?
> > >>> > > >
> > >>> > > > If data-driven - code could do the sorting based on that
> data...
> > >>> > > separate
> > >>> > > > from SOLR...
> > >>> > > >
> > >>> > > > Alternatively, if the data point exists in SOLR, a "sub-query"
> > >>> might be
> > >>> > > > used to get the right sort order on the items returned by the
> > >>> "main"
> > >>> > > > search... Possibly without having to resort to the
> > clunky-feeling
> > >>> > > listpos
> > >>> > > > multivalued field...
> > >>> > > >
> > >>> > > > On Fri, Apr 1, 2016 at 10:32 AM, Tamás Barta <
> > bartatamas@gmail.com
> > >>> >
> > >>> > > wrote:
> > >>> > > >
> > >>> > > > > For example I have to display sellable products which are in
> > >>> list X
> > >>> > in
> > >>> > > > the
> > >>> > > > > correct order.
> > >>> > > > >
> > >>> > > > > If I add a "status" and "list" (multivalued) fields to every
> > >>> document
> > >>> > > > > (products), then I can execute a query: status:sellable AND
> > >>> list:X,
> > >>> > > > where X
> > >>> > > > > is the ID of the list. The list field contains IDs of the
> list
> > in
> > >>> > which
> > >>> > > > the
> > >>> > > > > product is in.
> > >>> > > > >
> > >>> > > > > The problem is that I can't sort the result. A product has
> > >>> different
> > >>> > > > index
> > >>> > > > > for every list.
> > >>> > > > >
> > >>> > > > > Is it clear now?
> > >>> > > > >
> > >>> > > > > Earlier I added a "listpos" field with multivalue content,
> for
> > >>> > example:
> > >>> > > > >
> > >>> > > > > 1:23
> > >>> > > > > 2:4
> > >>> > > > >
> > >>> > > > > Which means that this product is in position 23 in list 1 and
> > it
> > >>> is
> > >>> > in
> > >>> > > > > position 4 in list 2. After that I created a custom
> comparator
> > >>> which
> > >>> > > > parses
> > >>> > > > > field values to get index for the specified list and sorts by
> > >>> that
> > >>> > > index.
> > >>> > > > >
> > >>> > > > > But I didn't like that solution much. I wish there would be a
> > >>> better
> > >>> > > > > solution. In SolrJ unfortunately I can't find an API to set
> > >>> custom
> > >>> > > > > comparator like I did in Lucene. So I don't know how to solve
> > >>> this
> > >>> > > > problem
> > >>> > > > > in Solr.
> > >>> > > > >
> > >>> > > > > Thanks,
> > >>> > > > > Tamás
> > >>> > > > > 2016. ápr. 1. 17:25 ezt írta ("Alessandro Benedetti" <
> > >>> > > > > abenedetti@apache.org
> > >>> > > > > >):
> > >>> > > > >
> > >>> > > > > > I think this is a classic XY Problem , you are trying to
> > solve
> > >>> X
> > >>> > with
> > >>> > > > Y ,
> > >>> > > > > > and you are asking us about Y .
> > >>> > > > > > Could you describe us what is your X problem ? What are you
> > >>> trying
> > >>> > to
> > >>> > > > do
> > >>> > > > > > with this ordered lists ?
> > >>> > > > > >
> > >>> > > > > > If not I would add a field to the product called :
> > >>> > > > > > list_position ( or a similar name) of type geo point (x,y)
> .
> > >>> > > > > > X could be your list ID
> > >>> > > > > > Y the position.
> > >>> > > > > > Then you can play with spatial search, to get what you
> want.
> > >>> > > > > >
> > >>> > > > > > But again, let's try to solve X.
> > >>> > > > > >
> > >>> > > > > > Cheers
> > >>> > > > > >
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> > >>
> > >>
> > >
> >
>
Re: Sorting question
Posted by Tamás Barta <ba...@gmail.com>.
Hi,
FYI: the final solution I found is that I created a custom
"listpos(fieldName, listId)" function and now I can display a sorted list
via:
fq=listid_s:378
sort=listpos(listpos_s,378) asc
Regards,
Tamas
On Fri, Apr 1, 2016 at 8:55 PM, John Bickerstaff <jo...@johnbickerstaff.com>
wrote:
> Tamas,
>
> This feels a bit like a "user favorites" problem.
>
> I did a little searching and found this... Don't know if it will help, but
> when I'm looking for stuff like this I find it helps to try to come up with
> generic or different descriptions of my problem and go search those as
> well...
>
>
> http://stackoverflow.com/questions/3931827/solr-merging-results-of-2-cores-into-only-those-results-that-have-a-matching-fie
>
> On Fri, Apr 1, 2016 at 12:40 PM, John Bickerstaff <
> john@johnbickerstaff.com>
> wrote:
>
> > Tamas,
> >
> > I'm brainstorming here - not being careful, just throwing out ideas...
> >
> > One thing that comes up is a separate document in SOLR - one doc for each
> > list.
> >
> > If a user adds a doc to their list, that doc's id gets added to this
> other
> > type of document...
> >
> > So, a document with the title "List 1" would have a multivalue field of
> > ID's and the list order number like so:
> >
> > ID List Position
> > _________________
> > doc1 ID : 1
> > doc2 ID: 2
> > doc3 ID: 3
> >
> > and so on... The big problem I see with this is keeping it organized
> > correctly. More code would have to be written to handle this when the
> user
> > does any kind of "crud" on the list...
> >
> > I'm pretty sure there's a way to write a query that uses that list to
> > properly order the items returned by your primary search, although I
> > haven't written such a query yet.
> >
> > If you have the luxury of NOT being in production yet with this system,
> > I'd seriously consider pushing to keep application metadata OUT of your
> > product information store. This particular problem (of ordering the
> > results based on arbitrary user choices) might be more easily handled
> via a
> > separate step that queries a relational database to handle list order -
> > once Solr gives you the documents that match the query and the user's
> list
> > number...
> >
> > Even if you can't use another relational data store - keeping that
> > metadata out of your individual product documents could be argued to be a
> > good design idea...
> >
> > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >
> > Here's an alternative brainstorm...
> >
> > Where does the user data live? What about putting the information about
> > the order of document ID's in the User's lists with the User? Then you
> can
> > get all documents that match the search terms and are on List X from
> Solr -
> > and then sort them by ID based on the data associated with the User (a
> list
> > of ID's, in order)
> >
> > There is even a way to write a plugin that will go after external data to
> > help sort Solr documents, although I'm guessing you'd rather avoid
> that...
> >
> >
> >
> > On Fri, Apr 1, 2016 at 11:59 AM, John Bickerstaff <
> > john@johnbickerstaff.com> wrote:
> >
> >> OK - I get it. List order is totally arbitrary and cannot be tied to an
> >> hard data point.
> >>
> >> I'll have to think - Perhaps billnbell's solution will help, although
> I'm
> >> not totally sure I understand that suggestion yet.
> >>
> >> At this point, you could get all the documents for List X that match the
> >> search terms. The next problem is sorting. If you have the listpos
> field
> >> too, you could use that, and some regex to find the proper order for
> these
> >> documents before displaying them (in code I mean) but of course that
> means
> >> you need some kind of "interceptor" to deal with this before the results
> >> are displayed.
> >>
> >> If I had enough control to do this in code, behind the scenes, I'd grab
> >> that second part of the listops field, put it into a variable on each
> >> object and then sort by that. Then I'd return the entire list to the
> UI.
> >>
> >> I understand that if you could get SOLR to do it all, that would be
> >> ideal... There is the possibility of writing some new code and
> plugging it
> >> in to Solr, but I'm guessing you don't want to go that far.. As a final
> >> step in the process, with custom code to consume the listpos entry,
> sorting
> >> these would be fairly straightforward. I'm not sure how you get away
> from
> >> the lispos multivalue field however...
> >>
> >> I'll keep thinking...
> >>
> >> On Fri, Apr 1, 2016 at 11:26 AM, Tamás Barta <ba...@gmail.com>
> >> wrote:
> >>
> >>> So, the list order is determined by the user. The user creates a list,
> >>> adds
> >>> products to it and i have to display these list using filters and
> >>> pagination.
> >>>
> >>> Let's assume there is list with 10000 products in it. In the website
> >>> where
> >>> i display the list only 50 products are displayed in a page. So if i
> >>> could
> >>> query solr to give me products from list X, ordered as user defined,
> but
> >>> only products with some criteria (status, amount, ..) from offset and
> 50
> >>> rows then it would be perfect and fast. If ordering would be outside of
> >>> solr then i have to retrive almost every 10000 documents from solr (a
> bit
> >>> less if filtered) to order them and display the page of 50 products.
> >>> 2016. ápr. 1. 19:15 ezt írta ("John Bickerstaff" <
> >>> john@johnbickerstaff.com
> >>> >):
> >>>
> >>> > Just to be clear - I don't mean who requests the list (application or
> >>> user)
> >>> > I mean what "rule" determines the ordering of the list?
> >>> >
> >>> > Or, is there even a rule of any kind?
> >>> >
> >>> > In other words, does a user arbitrarily decide that documentA,
> >>> documentF,
> >>> > and documentW should be on a list of their own? For reasons known
> >>> only to
> >>> > the user?
> >>> >
> >>> > Or - does the ordering of the list depend on some piece of data?
> >>> (like a
> >>> > date, or a manufacturer, or a price range or any other piece of
> "hard"
> >>> > data)
> >>> >
> >>> > ===
> >>> >
> >>> > To give an example from what I'm working on right now --
> >>> >
> >>> > My subject matter experts have given me a rule that says:
> >>> >
> >>> > *Documents of content_type "bar" should come higher in the results
> >>> than
> >>> > documents of content_type "foo".*
> >>> >
> >>> > PsuedoCode: If (content_type == bar) then put this doc highest in the
> >>> > results. If (content_type == foo) put those docs after the "bar"
> >>> > content_type docs.
> >>> >
> >>> >
> >>> > This is an example of the ordering being tied to a specific piece of
> >>> data
> >>> > which I can manipulate in a "sub query" (that's probably the wrong
> >>> > term...)
> >>> >
> >>> >
> >>> > This isn't exactly what you're doing, but it's close -- IF you have
> >>> rules
> >>> > you can express clearly in this way...
> >>> >
> >>> > ---
> >>> >
> >>> > Also, I'm confused a little by your statement that SOLR does the
> >>> filtering
> >>> > and pagination, thus you can't sort the documents after Solr returns
> >>> > them...
> >>> >
> >>> > My mental model is that you ask Solr for all the documents that
> match a
> >>> > certain criteria. Solr returns that "set" of documents and then for
> >>> your
> >>> > list, you sort those document titles or ID's according to some rule
> --
> >>> > possibly in the javascript on the web page... But perhaps I'm not
> >>> > understanding your situation well enough...
> >>> >
> >>> > Oh - are you perhaps saying that your ONLY criteria for getting these
> >>> > documents is the list number? That would make sense, although there
> >>> may
> >>> > still be room for sorting based on some kind of logic / data point
> >>> outside
> >>> > of SOlR. You could get all the documents associated to list #4, and
> >>> then
> >>> > sort them based on some hard data point they all contain. At the
> very
> >>> > least, your listpos "array" becomes simpler...
> >>> >
> >>> > What does your query currently look like?
> >>> >
> >>> > On Fri, Apr 1, 2016 at 10:51 AM, Tamás Barta <ba...@gmail.com>
> >>> wrote:
> >>> >
> >>> > > Some of the lists are created by users and some are generated by
> >>> > > applications, it doesn't matter.
> >>> > >
> >>> > > It would be fine to solve it in Solr because Solr does the work of
> >>> > > filtering and pagination. If sorting were done outside than I would
> >>> have
> >>> > to
> >>> > > read every document from Solr to sort them. It is not an option, I
> >>> have
> >>> > to
> >>> > > query onle one page.
> >>> > >
> >>> > > I don't understand how to solve it using subqueries.
> >>> > > 2016. ápr. 1. 18:42 ezt írta ("John Bickerstaff" <
> >>> > john@johnbickerstaff.com
> >>> > > >):
> >>> > >
> >>> > > > Specifically, what drives the position in the list? Is it
> >>> arbitrary or
> >>> > > is
> >>> > > > it driven by some piece of data?
> >>> > > >
> >>> > > > If data-driven - code could do the sorting based on that data...
> >>> > > separate
> >>> > > > from SOLR...
> >>> > > >
> >>> > > > Alternatively, if the data point exists in SOLR, a "sub-query"
> >>> might be
> >>> > > > used to get the right sort order on the items returned by the
> >>> "main"
> >>> > > > search... Possibly without having to resort to the
> clunky-feeling
> >>> > > listpos
> >>> > > > multivalued field...
> >>> > > >
> >>> > > > On Fri, Apr 1, 2016 at 10:32 AM, Tamás Barta <
> bartatamas@gmail.com
> >>> >
> >>> > > wrote:
> >>> > > >
> >>> > > > > For example I have to display sellable products which are in
> >>> list X
> >>> > in
> >>> > > > the
> >>> > > > > correct order.
> >>> > > > >
> >>> > > > > If I add a "status" and "list" (multivalued) fields to every
> >>> document
> >>> > > > > (products), then I can execute a query: status:sellable AND
> >>> list:X,
> >>> > > > where X
> >>> > > > > is the ID of the list. The list field contains IDs of the list
> in
> >>> > which
> >>> > > > the
> >>> > > > > product is in.
> >>> > > > >
> >>> > > > > The problem is that I can't sort the result. A product has
> >>> different
> >>> > > > index
> >>> > > > > for every list.
> >>> > > > >
> >>> > > > > Is it clear now?
> >>> > > > >
> >>> > > > > Earlier I added a "listpos" field with multivalue content, for
> >>> > example:
> >>> > > > >
> >>> > > > > 1:23
> >>> > > > > 2:4
> >>> > > > >
> >>> > > > > Which means that this product is in position 23 in list 1 and
> it
> >>> is
> >>> > in
> >>> > > > > position 4 in list 2. After that I created a custom comparator
> >>> which
> >>> > > > parses
> >>> > > > > field values to get index for the specified list and sorts by
> >>> that
> >>> > > index.
> >>> > > > >
> >>> > > > > But I didn't like that solution much. I wish there would be a
> >>> better
> >>> > > > > solution. In SolrJ unfortunately I can't find an API to set
> >>> custom
> >>> > > > > comparator like I did in Lucene. So I don't know how to solve
> >>> this
> >>> > > > problem
> >>> > > > > in Solr.
> >>> > > > >
> >>> > > > > Thanks,
> >>> > > > > Tamás
> >>> > > > > 2016. ápr. 1. 17:25 ezt írta ("Alessandro Benedetti" <
> >>> > > > > abenedetti@apache.org
> >>> > > > > >):
> >>> > > > >
> >>> > > > > > I think this is a classic XY Problem , you are trying to
> solve
> >>> X
> >>> > with
> >>> > > > Y ,
> >>> > > > > > and you are asking us about Y .
> >>> > > > > > Could you describe us what is your X problem ? What are you
> >>> trying
> >>> > to
> >>> > > > do
> >>> > > > > > with this ordered lists ?
> >>> > > > > >
> >>> > > > > > If not I would add a field to the product called :
> >>> > > > > > list_position ( or a similar name) of type geo point (x,y) .
> >>> > > > > > X could be your list ID
> >>> > > > > > Y the position.
> >>> > > > > > Then you can play with spatial search, to get what you want.
> >>> > > > > >
> >>> > > > > > But again, let's try to solve X.
> >>> > > > > >
> >>> > > > > > Cheers
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> >>
> >>
> >
>
Re: Sorting question
Posted by John Bickerstaff <jo...@johnbickerstaff.com>.
Tamas,
This feels a bit like a "user favorites" problem.
I did a little searching and found this... Don't know if it will help, but
when I'm looking for stuff like this I find it helps to try to come up with
generic or different descriptions of my problem and go search those as
well...
http://stackoverflow.com/questions/3931827/solr-merging-results-of-2-cores-into-only-those-results-that-have-a-matching-fie
On Fri, Apr 1, 2016 at 12:40 PM, John Bickerstaff <jo...@johnbickerstaff.com>
wrote:
> Tamas,
>
> I'm brainstorming here - not being careful, just throwing out ideas...
>
> One thing that comes up is a separate document in SOLR - one doc for each
> list.
>
> If a user adds a doc to their list, that doc's id gets added to this other
> type of document...
>
> So, a document with the title "List 1" would have a multivalue field of
> ID's and the list order number like so:
>
> ID List Position
> _________________
> doc1 ID : 1
> doc2 ID: 2
> doc3 ID: 3
>
> and so on... The big problem I see with this is keeping it organized
> correctly. More code would have to be written to handle this when the user
> does any kind of "crud" on the list...
>
> I'm pretty sure there's a way to write a query that uses that list to
> properly order the items returned by your primary search, although I
> haven't written such a query yet.
>
> If you have the luxury of NOT being in production yet with this system,
> I'd seriously consider pushing to keep application metadata OUT of your
> product information store. This particular problem (of ordering the
> results based on arbitrary user choices) might be more easily handled via a
> separate step that queries a relational database to handle list order -
> once Solr gives you the documents that match the query and the user's list
> number...
>
> Even if you can't use another relational data store - keeping that
> metadata out of your individual product documents could be argued to be a
> good design idea...
>
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
> Here's an alternative brainstorm...
>
> Where does the user data live? What about putting the information about
> the order of document ID's in the User's lists with the User? Then you can
> get all documents that match the search terms and are on List X from Solr -
> and then sort them by ID based on the data associated with the User (a list
> of ID's, in order)
>
> There is even a way to write a plugin that will go after external data to
> help sort Solr documents, although I'm guessing you'd rather avoid that...
>
>
>
> On Fri, Apr 1, 2016 at 11:59 AM, John Bickerstaff <
> john@johnbickerstaff.com> wrote:
>
>> OK - I get it. List order is totally arbitrary and cannot be tied to an
>> hard data point.
>>
>> I'll have to think - Perhaps billnbell's solution will help, although I'm
>> not totally sure I understand that suggestion yet.
>>
>> At this point, you could get all the documents for List X that match the
>> search terms. The next problem is sorting. If you have the listpos field
>> too, you could use that, and some regex to find the proper order for these
>> documents before displaying them (in code I mean) but of course that means
>> you need some kind of "interceptor" to deal with this before the results
>> are displayed.
>>
>> If I had enough control to do this in code, behind the scenes, I'd grab
>> that second part of the listops field, put it into a variable on each
>> object and then sort by that. Then I'd return the entire list to the UI.
>>
>> I understand that if you could get SOLR to do it all, that would be
>> ideal... There is the possibility of writing some new code and plugging it
>> in to Solr, but I'm guessing you don't want to go that far.. As a final
>> step in the process, with custom code to consume the listpos entry, sorting
>> these would be fairly straightforward. I'm not sure how you get away from
>> the lispos multivalue field however...
>>
>> I'll keep thinking...
>>
>> On Fri, Apr 1, 2016 at 11:26 AM, Tamás Barta <ba...@gmail.com>
>> wrote:
>>
>>> So, the list order is determined by the user. The user creates a list,
>>> adds
>>> products to it and i have to display these list using filters and
>>> pagination.
>>>
>>> Let's assume there is list with 10000 products in it. In the website
>>> where
>>> i display the list only 50 products are displayed in a page. So if i
>>> could
>>> query solr to give me products from list X, ordered as user defined, but
>>> only products with some criteria (status, amount, ..) from offset and 50
>>> rows then it would be perfect and fast. If ordering would be outside of
>>> solr then i have to retrive almost every 10000 documents from solr (a bit
>>> less if filtered) to order them and display the page of 50 products.
>>> 2016. ápr. 1. 19:15 ezt írta ("John Bickerstaff" <
>>> john@johnbickerstaff.com
>>> >):
>>>
>>> > Just to be clear - I don't mean who requests the list (application or
>>> user)
>>> > I mean what "rule" determines the ordering of the list?
>>> >
>>> > Or, is there even a rule of any kind?
>>> >
>>> > In other words, does a user arbitrarily decide that documentA,
>>> documentF,
>>> > and documentW should be on a list of their own? For reasons known
>>> only to
>>> > the user?
>>> >
>>> > Or - does the ordering of the list depend on some piece of data?
>>> (like a
>>> > date, or a manufacturer, or a price range or any other piece of "hard"
>>> > data)
>>> >
>>> > ===
>>> >
>>> > To give an example from what I'm working on right now --
>>> >
>>> > My subject matter experts have given me a rule that says:
>>> >
>>> > *Documents of content_type "bar" should come higher in the results
>>> than
>>> > documents of content_type "foo".*
>>> >
>>> > PsuedoCode: If (content_type == bar) then put this doc highest in the
>>> > results. If (content_type == foo) put those docs after the "bar"
>>> > content_type docs.
>>> >
>>> >
>>> > This is an example of the ordering being tied to a specific piece of
>>> data
>>> > which I can manipulate in a "sub query" (that's probably the wrong
>>> > term...)
>>> >
>>> >
>>> > This isn't exactly what you're doing, but it's close -- IF you have
>>> rules
>>> > you can express clearly in this way...
>>> >
>>> > ---
>>> >
>>> > Also, I'm confused a little by your statement that SOLR does the
>>> filtering
>>> > and pagination, thus you can't sort the documents after Solr returns
>>> > them...
>>> >
>>> > My mental model is that you ask Solr for all the documents that match a
>>> > certain criteria. Solr returns that "set" of documents and then for
>>> your
>>> > list, you sort those document titles or ID's according to some rule --
>>> > possibly in the javascript on the web page... But perhaps I'm not
>>> > understanding your situation well enough...
>>> >
>>> > Oh - are you perhaps saying that your ONLY criteria for getting these
>>> > documents is the list number? That would make sense, although there
>>> may
>>> > still be room for sorting based on some kind of logic / data point
>>> outside
>>> > of SOlR. You could get all the documents associated to list #4, and
>>> then
>>> > sort them based on some hard data point they all contain. At the very
>>> > least, your listpos "array" becomes simpler...
>>> >
>>> > What does your query currently look like?
>>> >
>>> > On Fri, Apr 1, 2016 at 10:51 AM, Tamás Barta <ba...@gmail.com>
>>> wrote:
>>> >
>>> > > Some of the lists are created by users and some are generated by
>>> > > applications, it doesn't matter.
>>> > >
>>> > > It would be fine to solve it in Solr because Solr does the work of
>>> > > filtering and pagination. If sorting were done outside than I would
>>> have
>>> > to
>>> > > read every document from Solr to sort them. It is not an option, I
>>> have
>>> > to
>>> > > query onle one page.
>>> > >
>>> > > I don't understand how to solve it using subqueries.
>>> > > 2016. ápr. 1. 18:42 ezt írta ("John Bickerstaff" <
>>> > john@johnbickerstaff.com
>>> > > >):
>>> > >
>>> > > > Specifically, what drives the position in the list? Is it
>>> arbitrary or
>>> > > is
>>> > > > it driven by some piece of data?
>>> > > >
>>> > > > If data-driven - code could do the sorting based on that data...
>>> > > separate
>>> > > > from SOLR...
>>> > > >
>>> > > > Alternatively, if the data point exists in SOLR, a "sub-query"
>>> might be
>>> > > > used to get the right sort order on the items returned by the
>>> "main"
>>> > > > search... Possibly without having to resort to the clunky-feeling
>>> > > listpos
>>> > > > multivalued field...
>>> > > >
>>> > > > On Fri, Apr 1, 2016 at 10:32 AM, Tamás Barta <bartatamas@gmail.com
>>> >
>>> > > wrote:
>>> > > >
>>> > > > > For example I have to display sellable products which are in
>>> list X
>>> > in
>>> > > > the
>>> > > > > correct order.
>>> > > > >
>>> > > > > If I add a "status" and "list" (multivalued) fields to every
>>> document
>>> > > > > (products), then I can execute a query: status:sellable AND
>>> list:X,
>>> > > > where X
>>> > > > > is the ID of the list. The list field contains IDs of the list in
>>> > which
>>> > > > the
>>> > > > > product is in.
>>> > > > >
>>> > > > > The problem is that I can't sort the result. A product has
>>> different
>>> > > > index
>>> > > > > for every list.
>>> > > > >
>>> > > > > Is it clear now?
>>> > > > >
>>> > > > > Earlier I added a "listpos" field with multivalue content, for
>>> > example:
>>> > > > >
>>> > > > > 1:23
>>> > > > > 2:4
>>> > > > >
>>> > > > > Which means that this product is in position 23 in list 1 and it
>>> is
>>> > in
>>> > > > > position 4 in list 2. After that I created a custom comparator
>>> which
>>> > > > parses
>>> > > > > field values to get index for the specified list and sorts by
>>> that
>>> > > index.
>>> > > > >
>>> > > > > But I didn't like that solution much. I wish there would be a
>>> better
>>> > > > > solution. In SolrJ unfortunately I can't find an API to set
>>> custom
>>> > > > > comparator like I did in Lucene. So I don't know how to solve
>>> this
>>> > > > problem
>>> > > > > in Solr.
>>> > > > >
>>> > > > > Thanks,
>>> > > > > Tamás
>>> > > > > 2016. ápr. 1. 17:25 ezt írta ("Alessandro Benedetti" <
>>> > > > > abenedetti@apache.org
>>> > > > > >):
>>> > > > >
>>> > > > > > I think this is a classic XY Problem , you are trying to solve
>>> X
>>> > with
>>> > > > Y ,
>>> > > > > > and you are asking us about Y .
>>> > > > > > Could you describe us what is your X problem ? What are you
>>> trying
>>> > to
>>> > > > do
>>> > > > > > with this ordered lists ?
>>> > > > > >
>>> > > > > > If not I would add a field to the product called :
>>> > > > > > list_position ( or a similar name) of type geo point (x,y) .
>>> > > > > > X could be your list ID
>>> > > > > > Y the position.
>>> > > > > > Then you can play with spatial search, to get what you want.
>>> > > > > >
>>> > > > > > But again, let's try to solve X.
>>> > > > > >
>>> > > > > > Cheers
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>
>>
>>
>
Re: Sorting question
Posted by John Bickerstaff <jo...@johnbickerstaff.com>.
Tamas,
I'm brainstorming here - not being careful, just throwing out ideas...
One thing that comes up is a separate document in SOLR - one doc for each
list.
If a user adds a doc to their list, that doc's id gets added to this other
type of document...
So, a document with the title "List 1" would have a multivalue field of
ID's and the list order number like so:
ID List Position
_________________
doc1 ID : 1
doc2 ID: 2
doc3 ID: 3
and so on... The big problem I see with this is keeping it organized
correctly. More code would have to be written to handle this when the user
does any kind of "crud" on the list...
I'm pretty sure there's a way to write a query that uses that list to
properly order the items returned by your primary search, although I
haven't written such a query yet.
If you have the luxury of NOT being in production yet with this system, I'd
seriously consider pushing to keep application metadata OUT of your product
information store. This particular problem (of ordering the results based
on arbitrary user choices) might be more easily handled via a separate step
that queries a relational database to handle list order - once Solr gives
you the documents that match the query and the user's list number...
Even if you can't use another relational data store - keeping that metadata
out of your individual product documents could be argued to be a good
design idea...
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Here's an alternative brainstorm...
Where does the user data live? What about putting the information about
the order of document ID's in the User's lists with the User? Then you can
get all documents that match the search terms and are on List X from Solr -
and then sort them by ID based on the data associated with the User (a list
of ID's, in order)
There is even a way to write a plugin that will go after external data to
help sort Solr documents, although I'm guessing you'd rather avoid that...
On Fri, Apr 1, 2016 at 11:59 AM, John Bickerstaff <jo...@johnbickerstaff.com>
wrote:
> OK - I get it. List order is totally arbitrary and cannot be tied to an
> hard data point.
>
> I'll have to think - Perhaps billnbell's solution will help, although I'm
> not totally sure I understand that suggestion yet.
>
> At this point, you could get all the documents for List X that match the
> search terms. The next problem is sorting. If you have the listpos field
> too, you could use that, and some regex to find the proper order for these
> documents before displaying them (in code I mean) but of course that means
> you need some kind of "interceptor" to deal with this before the results
> are displayed.
>
> If I had enough control to do this in code, behind the scenes, I'd grab
> that second part of the listops field, put it into a variable on each
> object and then sort by that. Then I'd return the entire list to the UI.
>
> I understand that if you could get SOLR to do it all, that would be
> ideal... There is the possibility of writing some new code and plugging it
> in to Solr, but I'm guessing you don't want to go that far.. As a final
> step in the process, with custom code to consume the listpos entry, sorting
> these would be fairly straightforward. I'm not sure how you get away from
> the lispos multivalue field however...
>
> I'll keep thinking...
>
> On Fri, Apr 1, 2016 at 11:26 AM, Tamás Barta <ba...@gmail.com> wrote:
>
>> So, the list order is determined by the user. The user creates a list,
>> adds
>> products to it and i have to display these list using filters and
>> pagination.
>>
>> Let's assume there is list with 10000 products in it. In the website where
>> i display the list only 50 products are displayed in a page. So if i could
>> query solr to give me products from list X, ordered as user defined, but
>> only products with some criteria (status, amount, ..) from offset and 50
>> rows then it would be perfect and fast. If ordering would be outside of
>> solr then i have to retrive almost every 10000 documents from solr (a bit
>> less if filtered) to order them and display the page of 50 products.
>> 2016. ápr. 1. 19:15 ezt írta ("John Bickerstaff" <
>> john@johnbickerstaff.com
>> >):
>>
>> > Just to be clear - I don't mean who requests the list (application or
>> user)
>> > I mean what "rule" determines the ordering of the list?
>> >
>> > Or, is there even a rule of any kind?
>> >
>> > In other words, does a user arbitrarily decide that documentA,
>> documentF,
>> > and documentW should be on a list of their own? For reasons known only
>> to
>> > the user?
>> >
>> > Or - does the ordering of the list depend on some piece of data? (like
>> a
>> > date, or a manufacturer, or a price range or any other piece of "hard"
>> > data)
>> >
>> > ===
>> >
>> > To give an example from what I'm working on right now --
>> >
>> > My subject matter experts have given me a rule that says:
>> >
>> > *Documents of content_type "bar" should come higher in the results than
>> > documents of content_type "foo".*
>> >
>> > PsuedoCode: If (content_type == bar) then put this doc highest in the
>> > results. If (content_type == foo) put those docs after the "bar"
>> > content_type docs.
>> >
>> >
>> > This is an example of the ordering being tied to a specific piece of
>> data
>> > which I can manipulate in a "sub query" (that's probably the wrong
>> > term...)
>> >
>> >
>> > This isn't exactly what you're doing, but it's close -- IF you have
>> rules
>> > you can express clearly in this way...
>> >
>> > ---
>> >
>> > Also, I'm confused a little by your statement that SOLR does the
>> filtering
>> > and pagination, thus you can't sort the documents after Solr returns
>> > them...
>> >
>> > My mental model is that you ask Solr for all the documents that match a
>> > certain criteria. Solr returns that "set" of documents and then for
>> your
>> > list, you sort those document titles or ID's according to some rule --
>> > possibly in the javascript on the web page... But perhaps I'm not
>> > understanding your situation well enough...
>> >
>> > Oh - are you perhaps saying that your ONLY criteria for getting these
>> > documents is the list number? That would make sense, although there may
>> > still be room for sorting based on some kind of logic / data point
>> outside
>> > of SOlR. You could get all the documents associated to list #4, and
>> then
>> > sort them based on some hard data point they all contain. At the very
>> > least, your listpos "array" becomes simpler...
>> >
>> > What does your query currently look like?
>> >
>> > On Fri, Apr 1, 2016 at 10:51 AM, Tamás Barta <ba...@gmail.com>
>> wrote:
>> >
>> > > Some of the lists are created by users and some are generated by
>> > > applications, it doesn't matter.
>> > >
>> > > It would be fine to solve it in Solr because Solr does the work of
>> > > filtering and pagination. If sorting were done outside than I would
>> have
>> > to
>> > > read every document from Solr to sort them. It is not an option, I
>> have
>> > to
>> > > query onle one page.
>> > >
>> > > I don't understand how to solve it using subqueries.
>> > > 2016. ápr. 1. 18:42 ezt írta ("John Bickerstaff" <
>> > john@johnbickerstaff.com
>> > > >):
>> > >
>> > > > Specifically, what drives the position in the list? Is it
>> arbitrary or
>> > > is
>> > > > it driven by some piece of data?
>> > > >
>> > > > If data-driven - code could do the sorting based on that data...
>> > > separate
>> > > > from SOLR...
>> > > >
>> > > > Alternatively, if the data point exists in SOLR, a "sub-query"
>> might be
>> > > > used to get the right sort order on the items returned by the "main"
>> > > > search... Possibly without having to resort to the clunky-feeling
>> > > listpos
>> > > > multivalued field...
>> > > >
>> > > > On Fri, Apr 1, 2016 at 10:32 AM, Tamás Barta <ba...@gmail.com>
>> > > wrote:
>> > > >
>> > > > > For example I have to display sellable products which are in list
>> X
>> > in
>> > > > the
>> > > > > correct order.
>> > > > >
>> > > > > If I add a "status" and "list" (multivalued) fields to every
>> document
>> > > > > (products), then I can execute a query: status:sellable AND
>> list:X,
>> > > > where X
>> > > > > is the ID of the list. The list field contains IDs of the list in
>> > which
>> > > > the
>> > > > > product is in.
>> > > > >
>> > > > > The problem is that I can't sort the result. A product has
>> different
>> > > > index
>> > > > > for every list.
>> > > > >
>> > > > > Is it clear now?
>> > > > >
>> > > > > Earlier I added a "listpos" field with multivalue content, for
>> > example:
>> > > > >
>> > > > > 1:23
>> > > > > 2:4
>> > > > >
>> > > > > Which means that this product is in position 23 in list 1 and it
>> is
>> > in
>> > > > > position 4 in list 2. After that I created a custom comparator
>> which
>> > > > parses
>> > > > > field values to get index for the specified list and sorts by that
>> > > index.
>> > > > >
>> > > > > But I didn't like that solution much. I wish there would be a
>> better
>> > > > > solution. In SolrJ unfortunately I can't find an API to set custom
>> > > > > comparator like I did in Lucene. So I don't know how to solve this
>> > > > problem
>> > > > > in Solr.
>> > > > >
>> > > > > Thanks,
>> > > > > Tamás
>> > > > > 2016. ápr. 1. 17:25 ezt írta ("Alessandro Benedetti" <
>> > > > > abenedetti@apache.org
>> > > > > >):
>> > > > >
>> > > > > > I think this is a classic XY Problem , you are trying to solve X
>> > with
>> > > > Y ,
>> > > > > > and you are asking us about Y .
>> > > > > > Could you describe us what is your X problem ? What are you
>> trying
>> > to
>> > > > do
>> > > > > > with this ordered lists ?
>> > > > > >
>> > > > > > If not I would add a field to the product called :
>> > > > > > list_position ( or a similar name) of type geo point (x,y) .
>> > > > > > X could be your list ID
>> > > > > > Y the position.
>> > > > > > Then you can play with spatial search, to get what you want.
>> > > > > >
>> > > > > > But again, let's try to solve X.
>> > > > > >
>> > > > > > Cheers
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
Re: Sorting question
Posted by John Bickerstaff <jo...@johnbickerstaff.com>.
OK - I get it. List order is totally arbitrary and cannot be tied to an
hard data point.
I'll have to think - Perhaps billnbell's solution will help, although I'm
not totally sure I understand that suggestion yet.
At this point, you could get all the documents for List X that match the
search terms. The next problem is sorting. If you have the listpos field
too, you could use that, and some regex to find the proper order for these
documents before displaying them (in code I mean) but of course that means
you need some kind of "interceptor" to deal with this before the results
are displayed.
If I had enough control to do this in code, behind the scenes, I'd grab
that second part of the listops field, put it into a variable on each
object and then sort by that. Then I'd return the entire list to the UI.
I understand that if you could get SOLR to do it all, that would be
ideal... There is the possibility of writing some new code and plugging it
in to Solr, but I'm guessing you don't want to go that far.. As a final
step in the process, with custom code to consume the listpos entry, sorting
these would be fairly straightforward. I'm not sure how you get away from
the lispos multivalue field however...
I'll keep thinking...
On Fri, Apr 1, 2016 at 11:26 AM, Tamás Barta <ba...@gmail.com> wrote:
> So, the list order is determined by the user. The user creates a list, adds
> products to it and i have to display these list using filters and
> pagination.
>
> Let's assume there is list with 10000 products in it. In the website where
> i display the list only 50 products are displayed in a page. So if i could
> query solr to give me products from list X, ordered as user defined, but
> only products with some criteria (status, amount, ..) from offset and 50
> rows then it would be perfect and fast. If ordering would be outside of
> solr then i have to retrive almost every 10000 documents from solr (a bit
> less if filtered) to order them and display the page of 50 products.
> 2016. ápr. 1. 19:15 ezt írta ("John Bickerstaff" <john@johnbickerstaff.com
> >):
>
> > Just to be clear - I don't mean who requests the list (application or
> user)
> > I mean what "rule" determines the ordering of the list?
> >
> > Or, is there even a rule of any kind?
> >
> > In other words, does a user arbitrarily decide that documentA, documentF,
> > and documentW should be on a list of their own? For reasons known only
> to
> > the user?
> >
> > Or - does the ordering of the list depend on some piece of data? (like a
> > date, or a manufacturer, or a price range or any other piece of "hard"
> > data)
> >
> > ===
> >
> > To give an example from what I'm working on right now --
> >
> > My subject matter experts have given me a rule that says:
> >
> > *Documents of content_type "bar" should come higher in the results than
> > documents of content_type "foo".*
> >
> > PsuedoCode: If (content_type == bar) then put this doc highest in the
> > results. If (content_type == foo) put those docs after the "bar"
> > content_type docs.
> >
> >
> > This is an example of the ordering being tied to a specific piece of data
> > which I can manipulate in a "sub query" (that's probably the wrong
> > term...)
> >
> >
> > This isn't exactly what you're doing, but it's close -- IF you have rules
> > you can express clearly in this way...
> >
> > ---
> >
> > Also, I'm confused a little by your statement that SOLR does the
> filtering
> > and pagination, thus you can't sort the documents after Solr returns
> > them...
> >
> > My mental model is that you ask Solr for all the documents that match a
> > certain criteria. Solr returns that "set" of documents and then for your
> > list, you sort those document titles or ID's according to some rule --
> > possibly in the javascript on the web page... But perhaps I'm not
> > understanding your situation well enough...
> >
> > Oh - are you perhaps saying that your ONLY criteria for getting these
> > documents is the list number? That would make sense, although there may
> > still be room for sorting based on some kind of logic / data point
> outside
> > of SOlR. You could get all the documents associated to list #4, and then
> > sort them based on some hard data point they all contain. At the very
> > least, your listpos "array" becomes simpler...
> >
> > What does your query currently look like?
> >
> > On Fri, Apr 1, 2016 at 10:51 AM, Tamás Barta <ba...@gmail.com>
> wrote:
> >
> > > Some of the lists are created by users and some are generated by
> > > applications, it doesn't matter.
> > >
> > > It would be fine to solve it in Solr because Solr does the work of
> > > filtering and pagination. If sorting were done outside than I would
> have
> > to
> > > read every document from Solr to sort them. It is not an option, I have
> > to
> > > query onle one page.
> > >
> > > I don't understand how to solve it using subqueries.
> > > 2016. ápr. 1. 18:42 ezt írta ("John Bickerstaff" <
> > john@johnbickerstaff.com
> > > >):
> > >
> > > > Specifically, what drives the position in the list? Is it arbitrary
> or
> > > is
> > > > it driven by some piece of data?
> > > >
> > > > If data-driven - code could do the sorting based on that data...
> > > separate
> > > > from SOLR...
> > > >
> > > > Alternatively, if the data point exists in SOLR, a "sub-query" might
> be
> > > > used to get the right sort order on the items returned by the "main"
> > > > search... Possibly without having to resort to the clunky-feeling
> > > listpos
> > > > multivalued field...
> > > >
> > > > On Fri, Apr 1, 2016 at 10:32 AM, Tamás Barta <ba...@gmail.com>
> > > wrote:
> > > >
> > > > > For example I have to display sellable products which are in list X
> > in
> > > > the
> > > > > correct order.
> > > > >
> > > > > If I add a "status" and "list" (multivalued) fields to every
> document
> > > > > (products), then I can execute a query: status:sellable AND list:X,
> > > > where X
> > > > > is the ID of the list. The list field contains IDs of the list in
> > which
> > > > the
> > > > > product is in.
> > > > >
> > > > > The problem is that I can't sort the result. A product has
> different
> > > > index
> > > > > for every list.
> > > > >
> > > > > Is it clear now?
> > > > >
> > > > > Earlier I added a "listpos" field with multivalue content, for
> > example:
> > > > >
> > > > > 1:23
> > > > > 2:4
> > > > >
> > > > > Which means that this product is in position 23 in list 1 and it is
> > in
> > > > > position 4 in list 2. After that I created a custom comparator
> which
> > > > parses
> > > > > field values to get index for the specified list and sorts by that
> > > index.
> > > > >
> > > > > But I didn't like that solution much. I wish there would be a
> better
> > > > > solution. In SolrJ unfortunately I can't find an API to set custom
> > > > > comparator like I did in Lucene. So I don't know how to solve this
> > > > problem
> > > > > in Solr.
> > > > >
> > > > > Thanks,
> > > > > Tamás
> > > > > 2016. ápr. 1. 17:25 ezt írta ("Alessandro Benedetti" <
> > > > > abenedetti@apache.org
> > > > > >):
> > > > >
> > > > > > I think this is a classic XY Problem , you are trying to solve X
> > with
> > > > Y ,
> > > > > > and you are asking us about Y .
> > > > > > Could you describe us what is your X problem ? What are you
> trying
> > to
> > > > do
> > > > > > with this ordered lists ?
> > > > > >
> > > > > > If not I would add a field to the product called :
> > > > > > list_position ( or a similar name) of type geo point (x,y) .
> > > > > > X could be your list ID
> > > > > > Y the position.
> > > > > > Then you can play with spatial search, to get what you want.
> > > > > >
> > > > > > But again, let's try to solve X.
> > > > > >
> > > > > > Cheers
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Sorting question
Posted by Tamás Barta <ba...@gmail.com>.
So, the list order is determined by the user. The user creates a list, adds
products to it and i have to display these list using filters and
pagination.
Let's assume there is list with 10000 products in it. In the website where
i display the list only 50 products are displayed in a page. So if i could
query solr to give me products from list X, ordered as user defined, but
only products with some criteria (status, amount, ..) from offset and 50
rows then it would be perfect and fast. If ordering would be outside of
solr then i have to retrive almost every 10000 documents from solr (a bit
less if filtered) to order them and display the page of 50 products.
2016. ápr. 1. 19:15 ezt írta ("John Bickerstaff" <john@johnbickerstaff.com
>):
> Just to be clear - I don't mean who requests the list (application or user)
> I mean what "rule" determines the ordering of the list?
>
> Or, is there even a rule of any kind?
>
> In other words, does a user arbitrarily decide that documentA, documentF,
> and documentW should be on a list of their own? For reasons known only to
> the user?
>
> Or - does the ordering of the list depend on some piece of data? (like a
> date, or a manufacturer, or a price range or any other piece of "hard"
> data)
>
> ===
>
> To give an example from what I'm working on right now --
>
> My subject matter experts have given me a rule that says:
>
> *Documents of content_type "bar" should come higher in the results than
> documents of content_type "foo".*
>
> PsuedoCode: If (content_type == bar) then put this doc highest in the
> results. If (content_type == foo) put those docs after the "bar"
> content_type docs.
>
>
> This is an example of the ordering being tied to a specific piece of data
> which I can manipulate in a "sub query" (that's probably the wrong
> term...)
>
>
> This isn't exactly what you're doing, but it's close -- IF you have rules
> you can express clearly in this way...
>
> ---
>
> Also, I'm confused a little by your statement that SOLR does the filtering
> and pagination, thus you can't sort the documents after Solr returns
> them...
>
> My mental model is that you ask Solr for all the documents that match a
> certain criteria. Solr returns that "set" of documents and then for your
> list, you sort those document titles or ID's according to some rule --
> possibly in the javascript on the web page... But perhaps I'm not
> understanding your situation well enough...
>
> Oh - are you perhaps saying that your ONLY criteria for getting these
> documents is the list number? That would make sense, although there may
> still be room for sorting based on some kind of logic / data point outside
> of SOlR. You could get all the documents associated to list #4, and then
> sort them based on some hard data point they all contain. At the very
> least, your listpos "array" becomes simpler...
>
> What does your query currently look like?
>
> On Fri, Apr 1, 2016 at 10:51 AM, Tamás Barta <ba...@gmail.com> wrote:
>
> > Some of the lists are created by users and some are generated by
> > applications, it doesn't matter.
> >
> > It would be fine to solve it in Solr because Solr does the work of
> > filtering and pagination. If sorting were done outside than I would have
> to
> > read every document from Solr to sort them. It is not an option, I have
> to
> > query onle one page.
> >
> > I don't understand how to solve it using subqueries.
> > 2016. ápr. 1. 18:42 ezt írta ("John Bickerstaff" <
> john@johnbickerstaff.com
> > >):
> >
> > > Specifically, what drives the position in the list? Is it arbitrary or
> > is
> > > it driven by some piece of data?
> > >
> > > If data-driven - code could do the sorting based on that data...
> > separate
> > > from SOLR...
> > >
> > > Alternatively, if the data point exists in SOLR, a "sub-query" might be
> > > used to get the right sort order on the items returned by the "main"
> > > search... Possibly without having to resort to the clunky-feeling
> > listpos
> > > multivalued field...
> > >
> > > On Fri, Apr 1, 2016 at 10:32 AM, Tamás Barta <ba...@gmail.com>
> > wrote:
> > >
> > > > For example I have to display sellable products which are in list X
> in
> > > the
> > > > correct order.
> > > >
> > > > If I add a "status" and "list" (multivalued) fields to every document
> > > > (products), then I can execute a query: status:sellable AND list:X,
> > > where X
> > > > is the ID of the list. The list field contains IDs of the list in
> which
> > > the
> > > > product is in.
> > > >
> > > > The problem is that I can't sort the result. A product has different
> > > index
> > > > for every list.
> > > >
> > > > Is it clear now?
> > > >
> > > > Earlier I added a "listpos" field with multivalue content, for
> example:
> > > >
> > > > 1:23
> > > > 2:4
> > > >
> > > > Which means that this product is in position 23 in list 1 and it is
> in
> > > > position 4 in list 2. After that I created a custom comparator which
> > > parses
> > > > field values to get index for the specified list and sorts by that
> > index.
> > > >
> > > > But I didn't like that solution much. I wish there would be a better
> > > > solution. In SolrJ unfortunately I can't find an API to set custom
> > > > comparator like I did in Lucene. So I don't know how to solve this
> > > problem
> > > > in Solr.
> > > >
> > > > Thanks,
> > > > Tamás
> > > > 2016. ápr. 1. 17:25 ezt írta ("Alessandro Benedetti" <
> > > > abenedetti@apache.org
> > > > >):
> > > >
> > > > > I think this is a classic XY Problem , you are trying to solve X
> with
> > > Y ,
> > > > > and you are asking us about Y .
> > > > > Could you describe us what is your X problem ? What are you trying
> to
> > > do
> > > > > with this ordered lists ?
> > > > >
> > > > > If not I would add a field to the product called :
> > > > > list_position ( or a similar name) of type geo point (x,y) .
> > > > > X could be your list ID
> > > > > Y the position.
> > > > > Then you can play with spatial search, to get what you want.
> > > > >
> > > > > But again, let's try to solve X.
> > > > >
> > > > > Cheers
> > > > >
> > > >
> > >
> >
>
Re: Sorting question
Posted by John Bickerstaff <jo...@johnbickerstaff.com>.
Just to be clear - I don't mean who requests the list (application or user)
I mean what "rule" determines the ordering of the list?
Or, is there even a rule of any kind?
In other words, does a user arbitrarily decide that documentA, documentF,
and documentW should be on a list of their own? For reasons known only to
the user?
Or - does the ordering of the list depend on some piece of data? (like a
date, or a manufacturer, or a price range or any other piece of "hard" data)
===
To give an example from what I'm working on right now --
My subject matter experts have given me a rule that says:
*Documents of content_type "bar" should come higher in the results than
documents of content_type "foo".*
PsuedoCode: If (content_type == bar) then put this doc highest in the
results. If (content_type == foo) put those docs after the "bar"
content_type docs.
This is an example of the ordering being tied to a specific piece of data
which I can manipulate in a "sub query" (that's probably the wrong term...)
This isn't exactly what you're doing, but it's close -- IF you have rules
you can express clearly in this way...
---
Also, I'm confused a little by your statement that SOLR does the filtering
and pagination, thus you can't sort the documents after Solr returns them...
My mental model is that you ask Solr for all the documents that match a
certain criteria. Solr returns that "set" of documents and then for your
list, you sort those document titles or ID's according to some rule --
possibly in the javascript on the web page... But perhaps I'm not
understanding your situation well enough...
Oh - are you perhaps saying that your ONLY criteria for getting these
documents is the list number? That would make sense, although there may
still be room for sorting based on some kind of logic / data point outside
of SOlR. You could get all the documents associated to list #4, and then
sort them based on some hard data point they all contain. At the very
least, your listpos "array" becomes simpler...
What does your query currently look like?
On Fri, Apr 1, 2016 at 10:51 AM, Tamás Barta <ba...@gmail.com> wrote:
> Some of the lists are created by users and some are generated by
> applications, it doesn't matter.
>
> It would be fine to solve it in Solr because Solr does the work of
> filtering and pagination. If sorting were done outside than I would have to
> read every document from Solr to sort them. It is not an option, I have to
> query onle one page.
>
> I don't understand how to solve it using subqueries.
> 2016. ápr. 1. 18:42 ezt írta ("John Bickerstaff" <john@johnbickerstaff.com
> >):
>
> > Specifically, what drives the position in the list? Is it arbitrary or
> is
> > it driven by some piece of data?
> >
> > If data-driven - code could do the sorting based on that data...
> separate
> > from SOLR...
> >
> > Alternatively, if the data point exists in SOLR, a "sub-query" might be
> > used to get the right sort order on the items returned by the "main"
> > search... Possibly without having to resort to the clunky-feeling
> listpos
> > multivalued field...
> >
> > On Fri, Apr 1, 2016 at 10:32 AM, Tamás Barta <ba...@gmail.com>
> wrote:
> >
> > > For example I have to display sellable products which are in list X in
> > the
> > > correct order.
> > >
> > > If I add a "status" and "list" (multivalued) fields to every document
> > > (products), then I can execute a query: status:sellable AND list:X,
> > where X
> > > is the ID of the list. The list field contains IDs of the list in which
> > the
> > > product is in.
> > >
> > > The problem is that I can't sort the result. A product has different
> > index
> > > for every list.
> > >
> > > Is it clear now?
> > >
> > > Earlier I added a "listpos" field with multivalue content, for example:
> > >
> > > 1:23
> > > 2:4
> > >
> > > Which means that this product is in position 23 in list 1 and it is in
> > > position 4 in list 2. After that I created a custom comparator which
> > parses
> > > field values to get index for the specified list and sorts by that
> index.
> > >
> > > But I didn't like that solution much. I wish there would be a better
> > > solution. In SolrJ unfortunately I can't find an API to set custom
> > > comparator like I did in Lucene. So I don't know how to solve this
> > problem
> > > in Solr.
> > >
> > > Thanks,
> > > Tamás
> > > 2016. ápr. 1. 17:25 ezt írta ("Alessandro Benedetti" <
> > > abenedetti@apache.org
> > > >):
> > >
> > > > I think this is a classic XY Problem , you are trying to solve X with
> > Y ,
> > > > and you are asking us about Y .
> > > > Could you describe us what is your X problem ? What are you trying to
> > do
> > > > with this ordered lists ?
> > > >
> > > > If not I would add a field to the product called :
> > > > list_position ( or a similar name) of type geo point (x,y) .
> > > > X could be your list ID
> > > > Y the position.
> > > > Then you can play with spatial search, to get what you want.
> > > >
> > > > But again, let's try to solve X.
> > > >
> > > > Cheers
> > > >
> > >
> >
>
Re: Sorting question
Posted by Tamás Barta <ba...@gmail.com>.
Some of the lists are created by users and some are generated by
applications, it doesn't matter.
It would be fine to solve it in Solr because Solr does the work of
filtering and pagination. If sorting were done outside than I would have to
read every document from Solr to sort them. It is not an option, I have to
query onle one page.
I don't understand how to solve it using subqueries.
2016. ápr. 1. 18:42 ezt írta ("John Bickerstaff" <john@johnbickerstaff.com
>):
> Specifically, what drives the position in the list? Is it arbitrary or is
> it driven by some piece of data?
>
> If data-driven - code could do the sorting based on that data... separate
> from SOLR...
>
> Alternatively, if the data point exists in SOLR, a "sub-query" might be
> used to get the right sort order on the items returned by the "main"
> search... Possibly without having to resort to the clunky-feeling listpos
> multivalued field...
>
> On Fri, Apr 1, 2016 at 10:32 AM, Tamás Barta <ba...@gmail.com> wrote:
>
> > For example I have to display sellable products which are in list X in
> the
> > correct order.
> >
> > If I add a "status" and "list" (multivalued) fields to every document
> > (products), then I can execute a query: status:sellable AND list:X,
> where X
> > is the ID of the list. The list field contains IDs of the list in which
> the
> > product is in.
> >
> > The problem is that I can't sort the result. A product has different
> index
> > for every list.
> >
> > Is it clear now?
> >
> > Earlier I added a "listpos" field with multivalue content, for example:
> >
> > 1:23
> > 2:4
> >
> > Which means that this product is in position 23 in list 1 and it is in
> > position 4 in list 2. After that I created a custom comparator which
> parses
> > field values to get index for the specified list and sorts by that index.
> >
> > But I didn't like that solution much. I wish there would be a better
> > solution. In SolrJ unfortunately I can't find an API to set custom
> > comparator like I did in Lucene. So I don't know how to solve this
> problem
> > in Solr.
> >
> > Thanks,
> > Tamás
> > 2016. ápr. 1. 17:25 ezt írta ("Alessandro Benedetti" <
> > abenedetti@apache.org
> > >):
> >
> > > I think this is a classic XY Problem , you are trying to solve X with
> Y ,
> > > and you are asking us about Y .
> > > Could you describe us what is your X problem ? What are you trying to
> do
> > > with this ordered lists ?
> > >
> > > If not I would add a field to the product called :
> > > list_position ( or a similar name) of type geo point (x,y) .
> > > X could be your list ID
> > > Y the position.
> > > Then you can play with spatial search, to get what you want.
> > >
> > > But again, let's try to solve X.
> > >
> > > Cheers
> > >
> >
>
Re: Sorting question
Posted by John Bickerstaff <jo...@johnbickerstaff.com>.
Specifically, what drives the position in the list? Is it arbitrary or is
it driven by some piece of data?
If data-driven - code could do the sorting based on that data... separate
from SOLR...
Alternatively, if the data point exists in SOLR, a "sub-query" might be
used to get the right sort order on the items returned by the "main"
search... Possibly without having to resort to the clunky-feeling listpos
multivalued field...
On Fri, Apr 1, 2016 at 10:32 AM, Tamás Barta <ba...@gmail.com> wrote:
> For example I have to display sellable products which are in list X in the
> correct order.
>
> If I add a "status" and "list" (multivalued) fields to every document
> (products), then I can execute a query: status:sellable AND list:X, where X
> is the ID of the list. The list field contains IDs of the list in which the
> product is in.
>
> The problem is that I can't sort the result. A product has different index
> for every list.
>
> Is it clear now?
>
> Earlier I added a "listpos" field with multivalue content, for example:
>
> 1:23
> 2:4
>
> Which means that this product is in position 23 in list 1 and it is in
> position 4 in list 2. After that I created a custom comparator which parses
> field values to get index for the specified list and sorts by that index.
>
> But I didn't like that solution much. I wish there would be a better
> solution. In SolrJ unfortunately I can't find an API to set custom
> comparator like I did in Lucene. So I don't know how to solve this problem
> in Solr.
>
> Thanks,
> Tamás
> 2016. ápr. 1. 17:25 ezt írta ("Alessandro Benedetti" <
> abenedetti@apache.org
> >):
>
> > I think this is a classic XY Problem , you are trying to solve X with Y ,
> > and you are asking us about Y .
> > Could you describe us what is your X problem ? What are you trying to do
> > with this ordered lists ?
> >
> > If not I would add a field to the product called :
> > list_position ( or a similar name) of type geo point (x,y) .
> > X could be your list ID
> > Y the position.
> > Then you can play with spatial search, to get what you want.
> >
> > But again, let's try to solve X.
> >
> > Cheers
> >
>
Re: Sorting question
Posted by Tamás Barta <ba...@gmail.com>.
For example I have to display sellable products which are in list X in the
correct order.
If I add a "status" and "list" (multivalued) fields to every document
(products), then I can execute a query: status:sellable AND list:X, where X
is the ID of the list. The list field contains IDs of the list in which the
product is in.
The problem is that I can't sort the result. A product has different index
for every list.
Is it clear now?
Earlier I added a "listpos" field with multivalue content, for example:
1:23
2:4
Which means that this product is in position 23 in list 1 and it is in
position 4 in list 2. After that I created a custom comparator which parses
field values to get index for the specified list and sorts by that index.
But I didn't like that solution much. I wish there would be a better
solution. In SolrJ unfortunately I can't find an API to set custom
comparator like I did in Lucene. So I don't know how to solve this problem
in Solr.
Thanks,
Tamás
2016. ápr. 1. 17:25 ezt írta ("Alessandro Benedetti" <abenedetti@apache.org
>):
> I think this is a classic XY Problem , you are trying to solve X with Y ,
> and you are asking us about Y .
> Could you describe us what is your X problem ? What are you trying to do
> with this ordered lists ?
>
> If not I would add a field to the product called :
> list_position ( or a similar name) of type geo point (x,y) .
> X could be your list ID
> Y the position.
> Then you can play with spatial search, to get what you want.
>
> But again, let's try to solve X.
>
> Cheers
>
Re: Sorting question
Posted by Alessandro Benedetti <ab...@apache.org>.
I think this is a classic XY Problem , you are trying to solve X with Y ,
and you are asking us about Y .
Could you describe us what is your X problem ? What are you trying to do
with this ordered lists ?
If not I would add a field to the product called :
list_position ( or a similar name) of type geo point (x,y) .
X could be your list ID
Y the position.
Then you can play with spatial search, to get what you want.
But again, let's try to solve X.
Cheers
Re: Sorting question
Posted by Binoy Dalal <bi...@gmail.com>.
I don't think I understand your problem properly. Are you trying to
pre-sort the products?
On Fri, 1 Apr 2016, 19:49 Tamás Barta, <ba...@gmail.com> wrote:
> Hi,
>
> I have a problem and I don't know how should I solve it in Solr.
>
> I have products indexed. Every product can be in lists. It is possible that
> a product isn't in any list or it is in multiple list.
> In a list the products are ordered. I would like to search for products in
> a specified list with the correct order.
>
> Earlier I tried to create a field for every list the product is in and
> these fields stored the index value of the product in that list.
> For example: list_23=1, list_841=8, ...
>
> After that I could do a query for list N: product where list_N exists order
> by list_N asc
> In this case the index size were extremely large because there are lot of
> lists and I guess every document stored a value for every field (it was
> long time ago, I don't know if it would be still a problem in current
> version of Solr)
>
> Do you have any idea how could I solve this problem?
>
> Thanks,
> Tamas
>
--
Regards,
Binoy Dalal