You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "stack@archive.org (JIRA)" <ji...@apache.org> on 2006/04/28 22:37:37 UTC
[jira] Created: (NUTCH-257) Summary#toString always Entity encodes
-- problem for OpenSearchServlet#description field
Summary#toString always Entity encodes -- problem for OpenSearchServlet#description field
-----------------------------------------------------------------------------------------
Key: NUTCH-257
URL: http://issues.apache.org/jira/browse/NUTCH-257
Project: Nutch
Type: Bug
Components: searcher
Versions: 0.8-dev
Reporter: stack@archive.org
Priority: Minor
All search result data we display in search results has to be explicitly Entity.encoded outputing in search.jsp ( title, url, etc.) except Summaries. Its already Entity.encoded. This is fine when outputing HTML but it gets in the way when outputing otherwise -- as xml for example. I'd suggest we not make any presumption about how search results are used.
The problem becomes especially acute when the text language is other than english.
Here is an example of what a Czech description field in an OpenSearchServlet hit record looks like:
<description><span class="ellipsis"> ... </span>V&#283;deck&aacute; knihovna v Olomouci Bezru&#269;ova 2, Olomouc 9, 779 11, &#268;esk&aacute; republika &nbsp; tel. +420-585223441 &nbsp; fax +420-585225774 http://www.<span class="highlight">vkol</span>.cz/ &nbsp;&nbsp; mailto:info@<span class="highlight">vkol</span>.cz Otev&#345;eno : &nbsp; po-p&aacute; &nbsp; 8 30 -19 00 &nbsp;&nbsp;&nbsp; so &nbsp; 9 00 -13 00 &nbsp;&nbsp;&nbsp; ne &nbsp; zav&#345;eno V katalogu s &uacute;pln&yacute;m &#269;asov&yacute;m<span class="ellipsis"> ... </span>03 Organizace 20/12 Odkazy 19/04 Hledej 23/03 &nbsp; 23/03 &nbsp; Po&#269;et p&#345;&iacute;stup&#367; od 1.9.1998. Statistiky . [ ] &nbsp; [ Nahoru ] <span class="highlight">VKOL</span></description>
Here is same description field with Entity.encoding disabled:
<description><span class="ellipsis"> ... </span>tisky statistiky knihovny WWW serveru středověké rukopisy studovny CD-ROM historických fondů hlavní Internet Německé knihovny vázaných novin SVKOL viz <span class="highlight">VKOL</span> šatna T telefonní čísla knihovny zaměstnanců U V vazba věcný popis vedení knihovny vedoucí oddělení video <span class="highlight">VKOL</span> volný výběr výpůjčka výroční zpráva výstavy W webmaster WWW odkazy X Y Z - Ž zamluvení knihy zahraniční periodika zpracování fondu<span class="highlight">VKOL</span> - hledej Hledej [ <span class="highlight">VKOL</span> ] [ Novinky ] [ Katalog ] [ Služby ] [ Aktivity ] [ Průvodce ] [ Dokumenty ] [ Regionální fce ] [ Organizace ] [ Odkazy ] [ Hledej ] [ ] [ ] Obsah full-textové vyhledávání, 19/04/2003 rejstřík vybraných<span class="ellipsis"> ... </span></description>
Notice how the Czech characters in the first are all numerically encoded: i.e. #NNN;.
I'd suggest that Summary#toString() become Summary#toEntityEncodedString() and that toString return raw aggregation of Fragments. Would likely require adding methods to the HitSummarizer interface so can ask for either raw text or entity encoded with addition to NutchBean so can ask for either. Or, better I'd suggest is that Summarizer never return Entity.encoded text. Let that happen in search.jsp (I can make patch to do the latter if its amenable).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Resolved: (NUTCH-257) Summary#toString always Entity encodes
-- problem for OpenSearchServlet#description field
Posted by "Jerome Charron (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-257?page=all ]
Jerome Charron resolved NUTCH-257:
----------------------------------
Fix Version: 0.8-dev
Resolution: Fixed
Assign To: Jerome Charron
Fixed in revision #405565 - http://svn.apache.org/viewcvs?view=rev&rev=405565
> Summary#toString always Entity encodes -- problem for OpenSearchServlet#description field
> -----------------------------------------------------------------------------------------
>
> Key: NUTCH-257
> URL: http://issues.apache.org/jira/browse/NUTCH-257
> Project: Nutch
> Type: Bug
> Components: searcher
> Versions: 0.8-dev
> Reporter: stack@archive.org
> Assignee: Jerome Charron
> Priority: Minor
> Fix For: 0.8-dev
>
> All search result data we display in search results has to be explicitly Entity.encoded outputing in search.jsp ( title, url, etc.) except Summaries. Its already Entity.encoded. This is fine when outputing HTML but it gets in the way when outputing otherwise -- as xml for example. I'd suggest we not make any presumption about how search results are used.
> The problem becomes especially acute when the text language is other than english.
> Here is an example of what a Czech description field in an OpenSearchServlet hit record looks like:
> <description><span class="ellipsis"> ... </span>V&#283;deck&aacute; knihovna v Olomouci Bezru&#269;ova 2, Olomouc 9, 779 11, &#268;esk&aacute; republika &nbsp; tel. +420-585223441 &nbsp; fax +420-585225774 http://www.<span class="highlight">vkol</span>.cz/ &nbsp;&nbsp; mailto:info@<span class="highlight">vkol</span>.cz Otev&#345;eno : &nbsp; po-p&aacute; &nbsp; 8 30 -19 00 &nbsp;&nbsp;&nbsp; so &nbsp; 9 00 -13 00 &nbsp;&nbsp;&nbsp; ne &nbsp; zav&#345;eno V katalogu s &uacute;pln&yacute;m &#269;asov&yacute;m<span class="ellipsis"> ... </span>03 Organizace 20/12 Odkazy 19/04 Hledej 23/03 &nbsp; 23/03 &nbsp; Po&#269;et p&#345;&iacute;stup&#367; od 1.9.1998. Statistiky . [ ] &nbsp; [ Nahoru ] <span class="highlight">VKOL</span></description>
> Here is same description field with Entity.encoding disabled:
> <description><span class="ellipsis"> ... </span>tisky statistiky knihovny WWW serveru st?edov?ké rukopisy studovny CD-ROM historických fond? hlavní Internet N?mecké knihovny vázaných novin SVKOL viz <span class="highlight">VKOL</span> ?atna T telefonní ?ísla knihovny zam?stnanc? U V vazba v?cný popis vedení knihovny vedoucí odd?lení video <span class="highlight">VKOL</span> volný výb?r výp?j?ka výro?ní zpráva výstavy W webmaster WWW odkazy X Y Z - ? zamluvení knihy zahrani?ní periodika zpracování fondu<span class="highlight">VKOL</span> - hledej Hledej [ <span class="highlight">VKOL</span> ] [ Novinky ] [ Katalog ] [ Slu?by ] [ Aktivity ] [ Pr?vodce ] [ Dokumenty ] [ Regionální fce ] [ Organizace ] [ Odkazy ] [ Hledej ] [ ] [ ] Obsah full-textové vyhledávání, 19/04/2003 rejst?ík vybraných<span class="ellipsis"> ... </span></description>
> Notice how the Czech characters in the first are all numerically encoded: i.e. #NNN;.
> I'd suggest that Summary#toString() become Summary#toEntityEncodedString() and that toString return raw aggregation of Fragments. Would likely require adding methods to the HitSummarizer interface so can ask for either raw text or entity encoded with addition to NutchBean so can ask for either. Or, better I'd suggest is that Summarizer never return Entity.encoded text. Let that happen in search.jsp (I can make patch to do the latter if its amenable).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (NUTCH-257) Summary#toString always Entity
encodes -- problem for OpenSearchServlet#description field
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-257?page=comments#action_12376989 ]
Doug Cutting commented on NUTCH-257:
------------------------------------
I'd vote to never have Summary#toString() perform entity encoding, to fix search.jsp to encode things itself, and *not* to add a new Summary#toEntityEncodedString() method.
> Summary#toString always Entity encodes -- problem for OpenSearchServlet#description field
> -----------------------------------------------------------------------------------------
>
> Key: NUTCH-257
> URL: http://issues.apache.org/jira/browse/NUTCH-257
> Project: Nutch
> Type: Bug
> Components: searcher
> Versions: 0.8-dev
> Reporter: stack@archive.org
> Priority: Minor
>
> All search result data we display in search results has to be explicitly Entity.encoded outputing in search.jsp ( title, url, etc.) except Summaries. Its already Entity.encoded. This is fine when outputing HTML but it gets in the way when outputing otherwise -- as xml for example. I'd suggest we not make any presumption about how search results are used.
> The problem becomes especially acute when the text language is other than english.
> Here is an example of what a Czech description field in an OpenSearchServlet hit record looks like:
> <description><span class="ellipsis"> ... </span>V&#283;deck&aacute; knihovna v Olomouci Bezru&#269;ova 2, Olomouc 9, 779 11, &#268;esk&aacute; republika &nbsp; tel. +420-585223441 &nbsp; fax +420-585225774 http://www.<span class="highlight">vkol</span>.cz/ &nbsp;&nbsp; mailto:info@<span class="highlight">vkol</span>.cz Otev&#345;eno : &nbsp; po-p&aacute; &nbsp; 8 30 -19 00 &nbsp;&nbsp;&nbsp; so &nbsp; 9 00 -13 00 &nbsp;&nbsp;&nbsp; ne &nbsp; zav&#345;eno V katalogu s &uacute;pln&yacute;m &#269;asov&yacute;m<span class="ellipsis"> ... </span>03 Organizace 20/12 Odkazy 19/04 Hledej 23/03 &nbsp; 23/03 &nbsp; Po&#269;et p&#345;&iacute;stup&#367; od 1.9.1998. Statistiky . [ ] &nbsp; [ Nahoru ] <span class="highlight">VKOL</span></description>
> Here is same description field with Entity.encoding disabled:
> <description><span class="ellipsis"> ... </span>tisky statistiky knihovny WWW serveru st?edov?ké rukopisy studovny CD-ROM historických fond? hlavní Internet N?mecké knihovny vázaných novin SVKOL viz <span class="highlight">VKOL</span> ?atna T telefonní ?ísla knihovny zam?stnanc? U V vazba v?cný popis vedení knihovny vedoucí odd?lení video <span class="highlight">VKOL</span> volný výb?r výp?j?ka výro?ní zpráva výstavy W webmaster WWW odkazy X Y Z - ? zamluvení knihy zahrani?ní periodika zpracování fondu<span class="highlight">VKOL</span> - hledej Hledej [ <span class="highlight">VKOL</span> ] [ Novinky ] [ Katalog ] [ Slu?by ] [ Aktivity ] [ Pr?vodce ] [ Dokumenty ] [ Regionální fce ] [ Organizace ] [ Odkazy ] [ Hledej ] [ ] [ ] Obsah full-textové vyhledávání, 19/04/2003 rejst?ík vybraných<span class="ellipsis"> ... </span></description>
> Notice how the Czech characters in the first are all numerically encoded: i.e. #NNN;.
> I'd suggest that Summary#toString() become Summary#toEntityEncodedString() and that toString return raw aggregation of Fragments. Would likely require adding methods to the HitSummarizer interface so can ask for either raw text or entity encoded with addition to NutchBean so can ask for either. Or, better I'd suggest is that Summarizer never return Entity.encoded text. Let that happen in search.jsp (I can make patch to do the latter if its amenable).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (NUTCH-257) Summary#toString always Entity
encodes -- problem for OpenSearchServlet#description field
Posted by "stack@archive.org (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-257?page=comments#action_12376997 ]
stack@archive.org commented on NUTCH-257:
-----------------------------------------
I took a closer look. Turns out Summary is inherently all about rendering HTML (See the different Summary.Fragment subclasses -- one for ellipsis, another for hightlight. In each of these, the to String wraps the fragment in some HTML 'span' markup).
What about changing HitSummarizer#getSummary to return Summary instead of String or String []. If the rendering context requires HTML, ask Summary to compose the HTML to output (Summary#toHtmlString()?). If, xml, get plain-text version of summary (Summary#toString())?
> Summary#toString always Entity encodes -- problem for OpenSearchServlet#description field
> -----------------------------------------------------------------------------------------
>
> Key: NUTCH-257
> URL: http://issues.apache.org/jira/browse/NUTCH-257
> Project: Nutch
> Type: Bug
> Components: searcher
> Versions: 0.8-dev
> Reporter: stack@archive.org
> Priority: Minor
>
> All search result data we display in search results has to be explicitly Entity.encoded outputing in search.jsp ( title, url, etc.) except Summaries. Its already Entity.encoded. This is fine when outputing HTML but it gets in the way when outputing otherwise -- as xml for example. I'd suggest we not make any presumption about how search results are used.
> The problem becomes especially acute when the text language is other than english.
> Here is an example of what a Czech description field in an OpenSearchServlet hit record looks like:
> <description><span class="ellipsis"> ... </span>V&#283;deck&aacute; knihovna v Olomouci Bezru&#269;ova 2, Olomouc 9, 779 11, &#268;esk&aacute; republika &nbsp; tel. +420-585223441 &nbsp; fax +420-585225774 http://www.<span class="highlight">vkol</span>.cz/ &nbsp;&nbsp; mailto:info@<span class="highlight">vkol</span>.cz Otev&#345;eno : &nbsp; po-p&aacute; &nbsp; 8 30 -19 00 &nbsp;&nbsp;&nbsp; so &nbsp; 9 00 -13 00 &nbsp;&nbsp;&nbsp; ne &nbsp; zav&#345;eno V katalogu s &uacute;pln&yacute;m &#269;asov&yacute;m<span class="ellipsis"> ... </span>03 Organizace 20/12 Odkazy 19/04 Hledej 23/03 &nbsp; 23/03 &nbsp; Po&#269;et p&#345;&iacute;stup&#367; od 1.9.1998. Statistiky . [ ] &nbsp; [ Nahoru ] <span class="highlight">VKOL</span></description>
> Here is same description field with Entity.encoding disabled:
> <description><span class="ellipsis"> ... </span>tisky statistiky knihovny WWW serveru st?edov?ké rukopisy studovny CD-ROM historických fond? hlavní Internet N?mecké knihovny vázaných novin SVKOL viz <span class="highlight">VKOL</span> ?atna T telefonní ?ísla knihovny zam?stnanc? U V vazba v?cný popis vedení knihovny vedoucí odd?lení video <span class="highlight">VKOL</span> volný výb?r výp?j?ka výro?ní zpráva výstavy W webmaster WWW odkazy X Y Z - ? zamluvení knihy zahrani?ní periodika zpracování fondu<span class="highlight">VKOL</span> - hledej Hledej [ <span class="highlight">VKOL</span> ] [ Novinky ] [ Katalog ] [ Slu?by ] [ Aktivity ] [ Pr?vodce ] [ Dokumenty ] [ Regionální fce ] [ Organizace ] [ Odkazy ] [ Hledej ] [ ] [ ] Obsah full-textové vyhledávání, 19/04/2003 rejst?ík vybraných<span class="ellipsis"> ... </span></description>
> Notice how the Czech characters in the first are all numerically encoded: i.e. #NNN;.
> I'd suggest that Summary#toString() become Summary#toEntityEncodedString() and that toString return raw aggregation of Fragments. Would likely require adding methods to the HitSummarizer interface so can ask for either raw text or entity encoded with addition to NutchBean so can ask for either. Or, better I'd suggest is that Summarizer never return Entity.encoded text. Let that happen in search.jsp (I can make patch to do the latter if its amenable).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira