You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Yoni Amir <Yo...@actimize.com> on 2012/08/26 10:31:04 UTC

hl.fl is ignored in solr 4 beta?

I have a setup for /select handler which, regarding highlighting, looks roughly like this:

	<requestHandler name="/select" class="solr.SearchHandler" default="true">
		<lst name="defaults">
			<str name="defType">edismax</str>
			<str name="qf">all_text</str>
			...
			<str name="fl">id, module, identifier, type, category, ... , score</str>
			...
			<str name="hl">on</str>
			<str name="hl.fl">identifier type owner_name *_custom_txt ...</str>
			<bool name="hl.requireFieldMatch">false</bool>
			<bool name="hl.usePhraseHighlighter">true</bool>
			<bool name="hl.highlightMultiTerm">true</bool>
			<int name="hl.snippets">10</int>
			<bool name="hl.mergeContiguous">true</bool>
			<int name="hl.maxAnalyzedChars">-1</int>
			
		</lst>
	</requestHandler>

As you can see, I search only on one field "all_text". I use hl.requireFieldMatch=false in order to highlight other fields. However, I limit the number of other fields using hl.fl parameter. However, it is not working. Solr returns highlights for a number of other fields which are not in the list (one of those is the field all_text itself, but other fields get highlight too).

Is this a known bug?

Thanks,
Yoni

RE: hl.fl is ignored in solr 4 beta?

Posted by Yoni Amir <Yo...@actimize.com>.
Thanks for your help. Using the regex trick actually worked and this is the direction we are taking now, but I think I'll open an enhancement request as well. I'll try to see if we can improve that code locally for our product first.
Thanks,
Yoni

-----Original Message-----
From: Jack Krupansky [mailto:jack@basetechnology.com] 
Sent: Monday, August 27, 2012 3:19 PM
To: solr-user@lucene.apache.org
Subject: Re: hl.fl is ignored in solr 4 beta?

Highlighting does work on dynamic fields. I verified that there is no bug there.

It is glob that does not work the way you used it that is the issue. Whether that is a sloppy doc description that needs to get cleaned up or a bug in the code is an open question. I would suggest that the doc needs to be updated and maybe then you have an "improvement" request. So, it is really two issues, I think. But, feel free to open a bug for "Glob does not work as expected for hl.fl field list".

I wouldn't suggest digging into that code, but if you really want to, see the "getHighlightFields" method here:
http://svn.apache.org/viewvc/lucene/dev/branches/branch_4x/solr/core/src/java/org/apache/solr/highlight/SolrHighlighter.java?view=markup

-- Jack Krupansky

-----Original Message-----
From: Yoni Amir
Sent: Monday, August 27, 2012 3:31 AM
To: solr-user@lucene.apache.org
Subject: RE: hl.fl is ignored in solr 4 beta?

Thanks for this information. Can you please send me to the right place in the code, I'll check it out.

Regardless, it sounds like a bug to me, highlighting should work on dynamic fields too. Should I open a bug for this?

Thanks,
Yoni

-----Original Message-----
From: Jack Krupansky [mailto:jack@basetechnology.com]
Sent: Sunday, August 26, 2012 8:11 PM
To: solr-user@lucene.apache.org
Subject: Re: hl.fl is ignored in solr 4 beta?

I am unable to reproduce this scenario of all fields being highlighted. In fact, from looking at the code, I don't see a way that could happen.
Further, the code, both 4.0-BETA and 3.6.1 as well as trunk, does NOT support "glob" patterns within a comma/space-delimited list of fields. Only a single glob pattern is supported, such as "*" or "*_custom_txt".

Actually, the code has an undocumented feature that almost full regular expressions are supported. If your single hl.fl parameter contains at least one "*", all the code does is take your hl.fl value and replace all "*" with ".*". So, you might be able to do what you want as follows:

<str name="hl.fl">identifier|type owner_name|*_custom_txt|...</str>

Where "..." is a list of field names separated by vertical bars. But, realize that this is an undocumented feature, at least for now.

Actually, it turns out that the vertical bar can also be used for many parameters where space and comma are supported as delimiters. But for hl.fl, glob is only supported using the vertical bar as a list delimiter.

-- Jack Krupansky

-----Original Message-----
From: Yoni Amir
Sent: Sunday, August 26, 2012 5:07 AM
To: solr-user@lucene.apache.org
Subject: RE: hl.fl is ignored in solr 4 beta?

I managed to narrow it down to the presence of dynamic field "*_custom_txt"
in the hl.fl list. If the list contains only regular fields the highlighting works fine.
However, I also want to highlight some dynamic fields.

Is this a bug?

Thanks,
Yoni

-----Original Message-----
From: Yoni Amir [mailto:Yoni.Amir@actimize.com]
Sent: Sunday, August 26, 2012 11:31 AM
To: solr-user@lucene.apache.org
Subject: hl.fl is ignored in solr 4 beta?

I have a setup for /select handler which, regarding highlighting, looks roughly like this:

<requestHandler name="/select" class="solr.SearchHandler" default="true"> <lst name="defaults"> <str name="defType">edismax</str> <str name="qf">all_text</str> ...
<str name="fl">id, module, identifier, type, category, ... , score</str> ...
<str name="hl">on</str>
<str name="hl.fl">identifier type owner_name *_custom_txt ...</str> <bool name="hl.requireFieldMatch">false</bool>
<bool name="hl.usePhraseHighlighter">true</bool>
<bool name="hl.highlightMultiTerm">true</bool>
<int name="hl.snippets">10</int>
<bool name="hl.mergeContiguous">true</bool>
<int name="hl.maxAnalyzedChars">-1</int>

</lst>
</requestHandler>

As you can see, I search only on one field "all_text". I use hl.requireFieldMatch=false in order to highlight other fields. However, I limit the number of other fields using hl.fl parameter. However, it is not working. Solr returns highlights for a number of other fields which are not in the list (one of those is the field all_text itself, but other fields get highlight too).

Is this a known bug?

Thanks,
Yoni

Re: hl.fl is ignored in solr 4 beta?

Posted by Jack Krupansky <ja...@basetechnology.com>.
Highlighting does work on dynamic fields. I verified that there is no bug 
there.

It is glob that does not work the way you used it that is the issue. Whether 
that is a sloppy doc description that needs to get cleaned up or a bug in 
the code is an open question. I would suggest that the doc needs to be 
updated and maybe then you have an "improvement" request. So, it is really 
two issues, I think. But, feel free to open a bug for "Glob does not work as 
expected for hl.fl field list".

I wouldn't suggest digging into that code, but if you really want to, see 
the "getHighlightFields" method here:
http://svn.apache.org/viewvc/lucene/dev/branches/branch_4x/solr/core/src/java/org/apache/solr/highlight/SolrHighlighter.java?view=markup

-- Jack Krupansky

-----Original Message----- 
From: Yoni Amir
Sent: Monday, August 27, 2012 3:31 AM
To: solr-user@lucene.apache.org
Subject: RE: hl.fl is ignored in solr 4 beta?

Thanks for this information. Can you please send me to the right place in 
the code, I'll check it out.

Regardless, it sounds like a bug to me, highlighting should work on dynamic 
fields too. Should I open a bug for this?

Thanks,
Yoni

-----Original Message-----
From: Jack Krupansky [mailto:jack@basetechnology.com]
Sent: Sunday, August 26, 2012 8:11 PM
To: solr-user@lucene.apache.org
Subject: Re: hl.fl is ignored in solr 4 beta?

I am unable to reproduce this scenario of all fields being highlighted. In 
fact, from looking at the code, I don't see a way that could happen.
Further, the code, both 4.0-BETA and 3.6.1 as well as trunk, does NOT 
support "glob" patterns within a comma/space-delimited list of fields. Only 
a single glob pattern is supported, such as "*" or "*_custom_txt".

Actually, the code has an undocumented feature that almost full regular 
expressions are supported. If your single hl.fl parameter contains at least 
one "*", all the code does is take your hl.fl value and replace all "*" with 
".*". So, you might be able to do what you want as follows:

<str name="hl.fl">identifier|type owner_name|*_custom_txt|...</str>

Where "..." is a list of field names separated by vertical bars. But, 
realize that this is an undocumented feature, at least for now.

Actually, it turns out that the vertical bar can also be used for many 
parameters where space and comma are supported as delimiters. But for hl.fl, 
glob is only supported using the vertical bar as a list delimiter.

-- Jack Krupansky

-----Original Message-----
From: Yoni Amir
Sent: Sunday, August 26, 2012 5:07 AM
To: solr-user@lucene.apache.org
Subject: RE: hl.fl is ignored in solr 4 beta?

I managed to narrow it down to the presence of dynamic field "*_custom_txt"
in the hl.fl list. If the list contains only regular fields the highlighting 
works fine.
However, I also want to highlight some dynamic fields.

Is this a bug?

Thanks,
Yoni

-----Original Message-----
From: Yoni Amir [mailto:Yoni.Amir@actimize.com]
Sent: Sunday, August 26, 2012 11:31 AM
To: solr-user@lucene.apache.org
Subject: hl.fl is ignored in solr 4 beta?

I have a setup for /select handler which, regarding highlighting, looks 
roughly like this:

<requestHandler name="/select" class="solr.SearchHandler" default="true"> 
<lst name="defaults"> <str name="defType">edismax</str> <str 
name="qf">all_text</str> ...
<str name="fl">id, module, identifier, type, category, ... , score</str> ...
<str name="hl">on</str>
<str name="hl.fl">identifier type owner_name *_custom_txt ...</str> <bool 
name="hl.requireFieldMatch">false</bool>
<bool name="hl.usePhraseHighlighter">true</bool>
<bool name="hl.highlightMultiTerm">true</bool>
<int name="hl.snippets">10</int>
<bool name="hl.mergeContiguous">true</bool>
<int name="hl.maxAnalyzedChars">-1</int>

</lst>
</requestHandler>

As you can see, I search only on one field "all_text". I use 
hl.requireFieldMatch=false in order to highlight other fields. However, I 
limit the number of other fields using hl.fl parameter. However, it is not 
working. Solr returns highlights for a number of other fields which are not 
in the list (one of those is the field all_text itself, but other fields get 
highlight too).

Is this a known bug?

Thanks,
Yoni

RE: hl.fl is ignored in solr 4 beta?

Posted by Yoni Amir <Yo...@actimize.com>.
Thanks for this information. Can you please send me to the right place in the code, I'll check it out.

Regardless, it sounds like a bug to me, highlighting should work on dynamic fields too. Should I open a bug for this?

Thanks,
Yoni

-----Original Message-----
From: Jack Krupansky [mailto:jack@basetechnology.com] 
Sent: Sunday, August 26, 2012 8:11 PM
To: solr-user@lucene.apache.org
Subject: Re: hl.fl is ignored in solr 4 beta?

I am unable to reproduce this scenario of all fields being highlighted. In fact, from looking at the code, I don't see a way that could happen. 
Further, the code, both 4.0-BETA and 3.6.1 as well as trunk, does NOT support "glob" patterns within a comma/space-delimited list of fields. Only a single glob pattern is supported, such as "*" or "*_custom_txt".

Actually, the code has an undocumented feature that almost full regular expressions are supported. If your single hl.fl parameter contains at least one "*", all the code does is take your hl.fl value and replace all "*" with ".*". So, you might be able to do what you want as follows:

<str name="hl.fl">identifier|type owner_name|*_custom_txt|...</str>

Where "..." is a list of field names separated by vertical bars. But, realize that this is an undocumented feature, at least for now.

Actually, it turns out that the vertical bar can also be used for many parameters where space and comma are supported as delimiters. But for hl.fl, glob is only supported using the vertical bar as a list delimiter.

-- Jack Krupansky

-----Original Message-----
From: Yoni Amir
Sent: Sunday, August 26, 2012 5:07 AM
To: solr-user@lucene.apache.org
Subject: RE: hl.fl is ignored in solr 4 beta?

I managed to narrow it down to the presence of dynamic field "*_custom_txt" 
in the hl.fl list. If the list contains only regular fields the highlighting works fine.
However, I also want to highlight some dynamic fields.

Is this a bug?

Thanks,
Yoni

-----Original Message-----
From: Yoni Amir [mailto:Yoni.Amir@actimize.com]
Sent: Sunday, August 26, 2012 11:31 AM
To: solr-user@lucene.apache.org
Subject: hl.fl is ignored in solr 4 beta?

I have a setup for /select handler which, regarding highlighting, looks roughly like this:

<requestHandler name="/select" class="solr.SearchHandler" default="true"> <lst name="defaults"> <str name="defType">edismax</str> <str name="qf">all_text</str> ...
<str name="fl">id, module, identifier, type, category, ... , score</str> ...
<str name="hl">on</str>
<str name="hl.fl">identifier type owner_name *_custom_txt ...</str> <bool name="hl.requireFieldMatch">false</bool>
<bool name="hl.usePhraseHighlighter">true</bool>
<bool name="hl.highlightMultiTerm">true</bool>
<int name="hl.snippets">10</int>
<bool name="hl.mergeContiguous">true</bool>
<int name="hl.maxAnalyzedChars">-1</int>

</lst>
</requestHandler>

As you can see, I search only on one field "all_text". I use hl.requireFieldMatch=false in order to highlight other fields. However, I limit the number of other fields using hl.fl parameter. However, it is not working. Solr returns highlights for a number of other fields which are not in the list (one of those is the field all_text itself, but other fields get highlight too).

Is this a known bug?

Thanks,
Yoni

Re: hl.fl is ignored in solr 4 beta?

Posted by Jack Krupansky <ja...@basetechnology.com>.
I am unable to reproduce this scenario of all fields being highlighted. In 
fact, from looking at the code, I don't see a way that could happen. 
Further, the code, both 4.0-BETA and 3.6.1 as well as trunk, does NOT 
support "glob" patterns within a comma/space-delimited list of fields. Only 
a single glob pattern is supported, such as "*" or "*_custom_txt".

Actually, the code has an undocumented feature that almost full regular 
expressions are supported. If your single hl.fl parameter contains at least 
one "*", all the code does is take your hl.fl value and replace all "*" with 
".*". So, you might be able to do what you want as follows:

<str name="hl.fl">identifier|type owner_name|*_custom_txt|...</str>

Where "..." is a list of field names separated by vertical bars. But, 
realize that this is an undocumented feature, at least for now.

Actually, it turns out that the vertical bar can also be used for many 
parameters where space and comma are supported as delimiters. But for hl.fl, 
glob is only supported using the vertical bar as a list delimiter.

-- Jack Krupansky

-----Original Message----- 
From: Yoni Amir
Sent: Sunday, August 26, 2012 5:07 AM
To: solr-user@lucene.apache.org
Subject: RE: hl.fl is ignored in solr 4 beta?

I managed to narrow it down to the presence of dynamic field "*_custom_txt" 
in the hl.fl list. If the list contains only regular fields the highlighting 
works fine.
However, I also want to highlight some dynamic fields.

Is this a bug?

Thanks,
Yoni

-----Original Message-----
From: Yoni Amir [mailto:Yoni.Amir@actimize.com]
Sent: Sunday, August 26, 2012 11:31 AM
To: solr-user@lucene.apache.org
Subject: hl.fl is ignored in solr 4 beta?

I have a setup for /select handler which, regarding highlighting, looks 
roughly like this:

<requestHandler name="/select" class="solr.SearchHandler" default="true">
<lst name="defaults">
<str name="defType">edismax</str>
<str name="qf">all_text</str>
...
<str name="fl">id, module, identifier, type, category, ... , score</str>
...
<str name="hl">on</str>
<str name="hl.fl">identifier type owner_name *_custom_txt ...</str>
<bool name="hl.requireFieldMatch">false</bool>
<bool name="hl.usePhraseHighlighter">true</bool>
<bool name="hl.highlightMultiTerm">true</bool>
<int name="hl.snippets">10</int>
<bool name="hl.mergeContiguous">true</bool>
<int name="hl.maxAnalyzedChars">-1</int>

</lst>
</requestHandler>

As you can see, I search only on one field "all_text". I use 
hl.requireFieldMatch=false in order to highlight other fields. However, I 
limit the number of other fields using hl.fl parameter. However, it is not 
working. Solr returns highlights for a number of other fields which are not 
in the list (one of those is the field all_text itself, but other fields get 
highlight too).

Is this a known bug?

Thanks,
Yoni

RE: hl.fl is ignored in solr 4 beta?

Posted by Yoni Amir <Yo...@actimize.com>.
I managed to narrow it down to the presence of dynamic field "*_custom_txt" in the hl.fl list. If the list contains only regular fields the highlighting works fine.
However, I also want to highlight some dynamic fields.

Is this a bug?

Thanks,
Yoni

-----Original Message-----
From: Yoni Amir [mailto:Yoni.Amir@actimize.com] 
Sent: Sunday, August 26, 2012 11:31 AM
To: solr-user@lucene.apache.org
Subject: hl.fl is ignored in solr 4 beta?

I have a setup for /select handler which, regarding highlighting, looks roughly like this:

	<requestHandler name="/select" class="solr.SearchHandler" default="true">
		<lst name="defaults">
			<str name="defType">edismax</str>
			<str name="qf">all_text</str>
			...
			<str name="fl">id, module, identifier, type, category, ... , score</str>
			...
			<str name="hl">on</str>
			<str name="hl.fl">identifier type owner_name *_custom_txt ...</str>
			<bool name="hl.requireFieldMatch">false</bool>
			<bool name="hl.usePhraseHighlighter">true</bool>
			<bool name="hl.highlightMultiTerm">true</bool>
			<int name="hl.snippets">10</int>
			<bool name="hl.mergeContiguous">true</bool>
			<int name="hl.maxAnalyzedChars">-1</int>
			
		</lst>
	</requestHandler>

As you can see, I search only on one field "all_text". I use hl.requireFieldMatch=false in order to highlight other fields. However, I limit the number of other fields using hl.fl parameter. However, it is not working. Solr returns highlights for a number of other fields which are not in the list (one of those is the field all_text itself, but other fields get highlight too).

Is this a known bug?

Thanks,
Yoni