search.jsp (and opensearchservlet) and query (utf-8) encoding

[stack <st...@archive.org>]

Looks like I have to do following to 'queryString' in search.jsp to 
search for terms that have non-ascii characters in them (e.g. japanese 
utf-8 text):
   
1.  String queryString = request.getParameter("query");
2.  if (queryString == null) {
3.    queryString = "";
4.  }
5.   queryString = new String(queryString.getBytes("ISO-8859-1"), "UTF-8");
6.   String htmlQueryString = Entities.encode(queryString);

Line 5. in the above is the added line.

Are others finding that they have to do this?  Seems like I shouldn't 
have to but other users of our nutch-based application report having to 
make this change running with different versions of tomcat (I've tried 
with 5.0.28 and 5.5.9).  Making the change would be consistent with 
(old) advice given here: http://www.jguru.com/faq/view.jsp?EID=391295.

Same change is seemingly needed in opensearchservlet.java.

Thanks in advance.
Yours,
St.Ack

Re: search.jsp (and opensearchservlet) and query (utf-8) encoding

[Dawid Weiss <da...@cs.put.poznan.pl>]

 > 5.   queryString = new String(queryString.getBytes("ISO-8859-1"), 
"UTF-8");

This will work, but is I'd say a dirty solution :) Add a configuration 
option to Tomcat forcing the container to treat URL-encoded arguments as 
UTF-8. In Tomcat 5 it means you need to add:

UseBodyEncodingForURI="true"

attribute to the connector and call 
request.setCharacterEncoding("UTF-8") in your servlet/ JSP page before 
you start reading any input (or parameters).

Dawid