You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by lboutros <bo...@gmail.com> on 2014/09/18 17:24:36 UTC

AND operator in multi valued fields

Dear all,

let's say you have two multivalued fields with two different complex
analyzers in a quite complex schema.

I would like to match specific combinations of values in these fields.

For instance :

Field1 : Value1, Value2
Field2 : Value3, Value4

I would like to match this document with a query like this one :

+Field1:Value1 +Field2:Value3

But not with this one :

+Field1:Value1 +Field2:Value4

I tried to check the PayloadNearQuery class but this class cannot use two
different fields (due to the SpanNearQuery inheritance).

Is there an easy way to do that ?

Ludovic.



-----
Jouve
France.
--
View this message in context: http://lucene.472066.n3.nabble.com/AND-operator-in-multi-valued-fields-tp4159715.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: AND operator in multi valued fields

Posted by lboutros <bo...@gmail.com>.
That's excellent Mikhail !

Thanks so much.

I have to use it in my custom query parser now.

Ludovic.



-----
Jouve
France.
--
View this message in context: http://lucene.472066.n3.nabble.com/AND-operator-in-multi-valued-fields-tp4159715p4160668.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: AND operator in multi valued fields

Posted by Mikhail Khludnev <mk...@griddynamics.com>.
On Fri, Sep 19, 2014 at 12:45 PM, lboutros <bo...@gmail.com> wrote:

> What do you think about developing a new SpanQuery class that allows "cross
> field" queries ?
>

indeed
http://lucene.apache.org/core/3_0_3/api/all/org/apache/lucene/search/spans/FieldMaskingSpanQuery.html


-- 
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics

<http://www.griddynamics.com>
<mk...@griddynamics.com>

Re: AND operator in multi valued fields

Posted by Alexandre Rafalovitch <ar...@gmail.com>.
Well, if it works, open source it. Could even become an official
contribution. You are not the only one asking for this kind of
features. Though your use case does seem to be a bit further out than
most.

Regards,
   Alex.
Personal: http://www.outerthoughts.com/ and @arafalov
Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
Solr popularizers community: https://www.linkedin.com/groups?gid=6713853


On 19 September 2014 09:34, lboutros <bo...@gmail.com> wrote:
> I 've just finished a first implementation of a CrossFieldSpanNearQuery and
> it just works perfectly :D
>
> I can now play with position increments and slops to get "exact" results
> within two multi valued fields.
>
> And for the 1st proposal, my user queries can be bigger than 10k with lots
> of different blocks and operators ;)
>
> Now I have to use this query in my custom query parser.
>
> Thx a lot Alex,
>
> Ludovic.
>
>
>
> -----
> Jouve
> France.
> --
> View this message in context: http://lucene.472066.n3.nabble.com/AND-operator-in-multi-valued-fields-tp4159715p4159942.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: AND operator in multi valued fields

Posted by lboutros <bo...@gmail.com>.
I 've just finished a first implementation of a CrossFieldSpanNearQuery and
it just works perfectly :D

I can now play with position increments and slops to get "exact" results
within two multi valued fields.

And for the 1st proposal, my user queries can be bigger than 10k with lots
of different blocks and operators ;)

Now I have to use this query in my custom query parser.

Thx a lot Alex,

Ludovic.



-----
Jouve
France.
--
View this message in context: http://lucene.472066.n3.nabble.com/AND-operator-in-multi-valued-fields-tp4159715p4159942.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: AND operator in multi valued fields

Posted by Alexandre Rafalovitch <ar...@gmail.com>.
I do not think the queries have access to multiple fields at once. Did you
check the API?

But I am not sure why 1 would be so hard. You know what event field names
are, so you just need to copy their conditions into subquery. You could
probably even do that in custom search component and not even touch the
client code.

Regards,
     Alex
On 19/09/2014 4:45 am, "lboutros" <bo...@gmail.com> wrote:

> Thx Alex for your answer.
>
> 1) This could be tricky, because the application users write very complex
> combined queries with main document fields and event fields too. A custom
> parser does the abstraction. I think that could be very tricky to extract
> event part of a complex query in order to filter on children events.
>
> 2) I did not thought about this solution ! Thanks, I will check the
> feasibility.
>
> 3) You are right. I think that denormalizing will have too much impact on
> performances during document updates and on the index size too.
>
> 4) You are right. Not an option here.
>
> I have a 5th proposal :
>
> What do you think about developing a new SpanQuery class that allows "cross
> field" queries ?
>
> Ludovic
>
>
>
>
> -----
> Jouve
> France.
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/AND-operator-in-multi-valued-fields-tp4159715p4159883.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: AND operator in multi valued fields

Posted by lboutros <bo...@gmail.com>.
Thx Alex for your answer.

1) This could be tricky, because the application users write very complex
combined queries with main document fields and event fields too. A custom
parser does the abstraction. I think that could be very tricky to extract
event part of a complex query in order to filter on children events.

2) I did not thought about this solution ! Thanks, I will check the
feasibility.

3) You are right. I think that denormalizing will have too much impact on
performances during document updates and on the index size too.

4) You are right. Not an option here.

I have a 5th proposal :

What do you think about developing a new SpanQuery class that allows "cross
field" queries ? 

Ludovic




-----
Jouve
France.
--
View this message in context: http://lucene.472066.n3.nabble.com/AND-operator-in-multi-valued-fields-tp4159715p4159883.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: AND operator in multi valued fields

Posted by Alexandre Rafalovitch <ar...@gmail.com>.
Well, I can think of four ways, increasingly complicated.

1) You could have both parent record with unzipped events and also
child events as individual documents. Then, you do filtering based on
children and highlighting based on parent documents.

2) The other way is to have a custom post filter that looks at the
matches and discards the ones that have different offset (by using
very large positionIncrementGap to create clear group boundaries). But
I don't know whether you can access the match token offsets in the
post filter, so this is more of a thought experiment.

3) You could also duplicate main field contents and be the "document"
one per event. If most of the fields are indexed, it's ok and no real
duplication. But you may need to store fields for highlighter and
those are not de-duplicated internally, as far as I know.

4) You could create zipped pairs of values in a dedicated field and
search that as near-queries. But than you do have to have the same
analyzer for all members. Sounds like this may not be an option for
you.

Regards,
   Alex.
Personal: http://www.outerthoughts.com/ and @arafalov
Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
Solr popularizers community: https://www.linkedin.com/groups?gid=6713853


On 18 September 2014 16:41, lboutros <bo...@gmail.com> wrote:
> Thx Alex.
>
> We have main documents in the index. (more than 100 complex fields).
>
> Each document can have events attached.
>
> An event contains 4 fields with 3 different analyzers.
>
> We need more than just filtering on them (highlighting on documents and
> events at the same time for instance).
> That means that nested documents cannot be used.
>
> These events are indexed as additional multi valued fields in each
> documents.
> They are searched like any other field.
>
> The issue here is that the operator 'AND' between event fields can match
> false positives.
>
> We do not know the position during search. We just want to respect the event
> integrity in the search. So you are right, we just want them to be parallel
> within their tokenized groups ?
>
> The first idea was to index the event in only one field and use
> proximity/phrase search in order to prevent false positives.
>
> But that means that we need to index dates, ids and text in one unique
> field.
>
> Do you think this could be a better/easier approach ?
>
> Ludovic
>
>
>
> -----
> Jouve
> France.
> --
> View this message in context: http://lucene.472066.n3.nabble.com/AND-operator-in-multi-valued-fields-tp4159715p4159797.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: AND operator in multi valued fields

Posted by lboutros <bo...@gmail.com>.
Thx Alex.

We have main documents in the index. (more than 100 complex fields).

Each document can have events attached.

An event contains 4 fields with 3 different analyzers.

We need more than just filtering on them (highlighting on documents and
events at the same time for instance).
That means that nested documents cannot be used.

These events are indexed as additional multi valued fields in each
documents.
They are searched like any other field.

The issue here is that the operator 'AND' between event fields can match
false positives.

We do not know the position during search. We just want to respect the event
integrity in the search. So you are right, we just want them to be parallel
within their tokenized groups ? 

The first idea was to index the event in only one field and use
proximity/phrase search in order to prevent false positives.

But that means that we need to index dates, ids and text in one unique
field.

Do you think this could be a better/easier approach ?

Ludovic



-----
Jouve
France.
--
View this message in context: http://lucene.472066.n3.nabble.com/AND-operator-in-multi-valued-fields-tp4159715p4159797.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: AND operator in multi valued fields

Posted by Alexandre Rafalovitch <ar...@gmail.com>.
Do you know the position when you are doing the search? Or just that
they need to be parallel within their tokenized groups?

Regards,
   Alex.
P.s. It may help if you explain a business level issue here. There
might be a completely different approach to that as well.
Personal: http://www.outerthoughts.com/ and @arafalov
Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
Solr popularizers community: https://www.linkedin.com/groups?gid=6713853


On 18 September 2014 11:42, lboutros <bo...@gmail.com> wrote:
> Alexandre Rafalovitch wrote
>> Are you saying you want to match 1st value with 1st value (like positional
>> constraints?).
>
> That's exactly what I would like to do. :)
>
>
>
>
> -----
> Jouve
> France.
> --
> View this message in context: http://lucene.472066.n3.nabble.com/AND-operator-in-multi-valued-fields-tp4159715p4159728.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: AND operator in multi valued fields

Posted by lboutros <bo...@gmail.com>.
Alexandre Rafalovitch wrote
> Are you saying you want to match 1st value with 1st value (like positional
> constraints?).

That's exactly what I would like to do. :)




-----
Jouve
France.
--
View this message in context: http://lucene.472066.n3.nabble.com/AND-operator-in-multi-valued-fields-tp4159715p4159728.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: AND operator in multi valued fields

Posted by Alexandre Rafalovitch <ar...@gmail.com>.
Both queries seem valid. The values are there and you asking to match
them. They both should match.

Can you explain how query 2 is actually different from query 1? Are
you saying you want to match 1st value with 1st value (like positional
constraints?).

Regards,
   Alex.
Personal: http://www.outerthoughts.com/ and @arafalov
Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
Solr popularizers community: https://www.linkedin.com/groups?gid=6713853


On 18 September 2014 11:24, lboutros <bo...@gmail.com> wrote:
> Dear all,
>
> let's say you have two multivalued fields with two different complex
> analyzers in a quite complex schema.
>
> I would like to match specific combinations of values in these fields.
>
> For instance :
>
> Field1 : Value1, Value2
> Field2 : Value3, Value4
>
> I would like to match this document with a query like this one :
>
> +Field1:Value1 +Field2:Value3
>
> But not with this one :
>
> +Field1:Value1 +Field2:Value4
>
> I tried to check the PayloadNearQuery class but this class cannot use two
> different fields (due to the SpanNearQuery inheritance).
>
> Is there an easy way to do that ?
>
> Ludovic.
>
>
>
> -----
> Jouve
> France.
> --
> View this message in context: http://lucene.472066.n3.nabble.com/AND-operator-in-multi-valued-fields-tp4159715.html
> Sent from the Solr - User mailing list archive at Nabble.com.