You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Tim Gilbert <TI...@morningstar.com> on 2010/05/20 17:25:48 UTC
Non-English query via Solr Example Admin corrupts text
Hi guys/gals,
I am using apache-solr-1.4.0.war deployed to glassfishv3 on my development machine which is Ubuntu 9.10 64-bit. I am using Solrj 1.4 using the CommonsHttpSolrServer connection to that Solr instance (http://localhost:8080/apache-solr-1.4.0) during my development. To simplify things however, I have found that I can duplicate my issue directly from Solr example admin page so for ease of confirmation, I will use the Solr Example Admin page for this example:
I deployed the apache-solr-1.4.0/dist/apache-solr-1.4.0.war file to my glassfishv3 application server. It deploys successfully. I access http://localhost:8080/apache-solr-1.4.0/admin/form.jsp and enter into "Solr/Lucene Statement" textarea this word:
numéro (Note the é)
When I check the server.log file, I see this:
INFO: [] webapp=/apache-solr-1.4.0 path=/select params={indent=on&version=2.2&q=numéro&fq=&start=0&rows=10&fl=*,score&qt=standard&wt=standard&explainOther=&hl.fl=} hits=0 status=0 QTime=16
As well, the output from the Admin system is with the same incorrect decoding.
In my SolrJ using application, I have a test case which queries for "numéro" and succeeds if I use Embedded and fails if I use CommonsHttpSolrServer... I don't want to use embedded for a number of reasons including that its not recommended (http://wiki.apache.org/solr/EmbeddedSolr)
I am sorry if you'd dealt with this issue in the past, I've spent a few hours googling for solr utf-8 query and glassfishv3 utf-8 uri plus other permutations/combinations but there were seemingly endless amounts of chaff that I couldn't find anything useful after scouring it for a few hours. I can't decide whether it's a glassfish issue or not so I am not sure where to direct my energy. Any tips or advice are appreciated!
Thanks in advance,
Tim Gilbert
RE: Non-English query via Solr Example Admin corrupts text
Posted by Chris Hostetter <ho...@fucit.org>.
This should be fixed now -- please update the Jira issue if you have any
other problems creating an account.
: Hmmm... yes, there definitely seems to be a problem with creating new wiki
: accounts on wiki.apache.org -- i've opened an issue with INFRA...
:
: https://issues.apache.org/jira/browse/INFRA-2726
-Hoss
RE: Non-English query via Solr Example Admin corrupts text
Posted by Chris Hostetter <ho...@fucit.org>.
: I wanted to improve the documentation in the solr wiki by adding in my
: findings. However, when I try to log in and create a new account, I
: receive this error message:
:
: You are not allowed to do newaccount on this page. Login and try again.
:
: Does anyone know how I can get permission to add a page to the
: documentation?
Hmmm... yes, there definitely seems to be a problem with creating new wiki
accounts on wiki.apache.org -- i've opened an issue with INFRA...
https://issues.apache.org/jira/browse/INFRA-2726
-Hoss
RE: Non-English query via Solr Example Admin corrupts text
Posted by Tim Gilbert <TI...@morningstar.com>.
I wanted to improve the documentation in the solr wiki by adding in my
findings. However, when I try to log in and create a new account, I
receive this error message:
You are not allowed to do newaccount on this page. Login and try again.
Does anyone know how I can get permission to add a page to the
documentation?
Tim
-----Original Message-----
From: Chris Hostetter [mailto:hossman_lucene@fucit.org]
Sent: Thursday, May 20, 2010 3:21 PM
To: solr-user@lucene.apache.org
Subject: RE: Non-English query via Solr Example Admin corrupts text
: Starting with glassfishv3 (I think) UTF-8 is the default for URI. You
: can see this by going to the admin site, clicking on Network Config |
: Network Listeners | then select the listener. Select the tab "HTTP"
and
: about half way down, you will see URI Encoding: UTF-8.
:
: HOWEVER, that doesn't appear to be correct because following
Abdelhamid
...
I know nothing about glassfish, but according to that forum URL i
mentioned before, the URI Encoding option in glassfish explicitly (and
evidently
contenciously) does not apply to hte query args -- only the path, hence
the two different config options mentioned in the FAQ...
: http://forums.java.net/jive/thread.jspa?threadID=38020
...
:
http://wiki.glassfish.java.net/Wiki.jsp?page=FaqHttpRequestParameterEnco
ding
-Hoss
RE: Non-English query via Solr Example Admin corrupts text
Posted by Chris Hostetter <ho...@fucit.org>.
: Starting with glassfishv3 (I think) UTF-8 is the default for URI. You
: can see this by going to the admin site, clicking on Network Config |
: Network Listeners | then select the listener. Select the tab "HTTP" and
: about half way down, you will see URI Encoding: UTF-8.
:
: HOWEVER, that doesn't appear to be correct because following Abdelhamid
...
I know nothing about glassfish, but according to that forum URL i
mentioned before, the URI Encoding option in glassfish explicitly (and evidently
contenciously) does not apply to hte query args -- only the path, hence
the two different config options mentioned in the FAQ...
: http://forums.java.net/jive/thread.jspa?threadID=38020
...
: http://wiki.glassfish.java.net/Wiki.jsp?page=FaqHttpRequestParameterEncoding
-Hoss
RE: Non-English query via Solr Example Admin corrupts text
Posted by Tim Gilbert <TI...@morningstar.com>.
Chris,
You are the best. Switching to POST solved the problem. I hadn't noticed that option earlier but after finding: https://issues.apache.org/jira/browse/SOLR-612 I found the option in the code.
Thank you, you just made my day.
Secondly, in an effort to narrow down whether this was a glassfish issue or not, here is what I found.
Starting with glassfishv3 (I think) UTF-8 is the default for URI. You can see this by going to the admin site, clicking on Network Config | Network Listeners | then select the listener. Select the tab "HTTP" and about half way down, you will see URI Encoding: UTF-8.
HOWEVER, that doesn't appear to be correct because following Abdelhamid Abid's advice, I deployed Solr to Tomcat, then followed the direction here:
http://wiki.apache.org/solr/SolrTomcat to force tomcat to UTF-8 for URI. Then I deployed Solr to tomcat, and using CommonsHttpSolrServer, connected to that tomcat served instance. It worked- first time.
So, it appears that there is a problem with glassfishv3 and UTF-8 URI's for at least the apache-solr-1.4.0.war. I wonder if I added that sun-web.xml file into the war to force UTF-8 it might work... not sure. However, the workaround is to change the method to POST as Chris suggested. You can do that in Solrj here:
server.query(solrQuery, METHOD.POST);
and it works as you'd expect.
Thanks for the advice/tips,
Tim
-----Original Message-----
From: Chris Hostetter [mailto:hossman_lucene@fucit.org]
Sent: Thursday, May 20, 2010 2:41 PM
To: solr-user@lucene.apache.org
Subject: Re: Non-English query via Solr Example Admin corrupts text
: I am using apache-solr-1.4.0.war deployed to glassfishv3 on my
...
: INFO: [] webapp=/apache-solr-1.4.0 path=/select
: params={indent=on&version=2.2&q=numéro&fq=&start=0&rows=10&fl=*,score&qt=standard&wt=standard&explainOther=&hl.fl=}
: hits=0 status=0 QTime=16
...
: In my SolrJ using application, I have a test case which queries for
: "numéro" and succeeds if I use Embedded and fails if I use
: CommonsHttpSolrServer... I don't want to use embedded for a number of
...
: I am sorry if you'd dealt with this issue in the past, I've spent a few
: hours googling for solr utf-8 query and glassfishv3 utf-8 uri plus other
: permutations/combinations but there were seemingly endless amounts of
: chaff that I couldn't find anything useful after scouring it for a few
: hours. I can't decide whether it's a glassfish issue or not so I am not
: sure where to direct my energy. Any tips or advice are appreciated!
I suspect if you switched to using POST instead of GET your problem would
go away -- this stems from amiguity in the way HTTP servers/browsers deal
with encoding UTF8 in URLs. a quick search for "glassfish url encoding"
turns up this thread...
http://forums.java.net/jive/thread.jspa?threadID=38020
which refreneces...
http://wiki.glassfish.java.net/Wiki.jsp?page=FaqHttpRequestParameterEncoding
...it looks like you want to modify the "default-charset attribute of the
<parameter-encoding>"
-Hoss
Re: Non-English query via Solr Example Admin corrupts text
Posted by Dennis Gearon <ge...@sbcglobal.net>.
<rant_by_HTTP_Verb_Nazi>
Using POST totally violates the access model for an entity in the HTTP Verb model.
Basically:
GET=READ
POST=CREATE
PUT=MODIFY
DELETE=(drum roll please)DELETE
Granted, the whole web uses POST for modify, but let's not make the situation worse by using it for everything.
</rant_by_HTTP_Verb_Nazi>
Dennis Gearon
Signature Warning
----------------
EARTH has a Right To Life,
otherwise we all die.
Read 'Hot, Flat, and Crowded'
Laugh at http://www.yert.com/film.php
--- On Thu, 5/20/10, Chris Hostetter <ho...@fucit.org> wrote:
> From: Chris Hostetter <ho...@fucit.org>
> Subject: Re: Non-English query via Solr Example Admin corrupts text
> To: solr-user@lucene.apache.org
> Date: Thursday, May 20, 2010, 11:40 AM
>
> : I am using apache-solr-1.4.0.war deployed to glassfishv3
> on my
> ...
> : INFO: [] webapp=/apache-solr-1.4.0 path=/select
> :
> params={indent=on&version=2.2&q=numéro&fq=&start=0&rows=10&fl=*,score&qt=standard&wt=standard&explainOther=&hl.fl=}
>
> : hits=0 status=0 QTime=16
> ...
> : In my SolrJ using application, I have a test case which
> queries for
> : "numéro" and succeeds if I use Embedded and fails if I
> use
> : CommonsHttpSolrServer... I don't want to use embedded for
> a number of
> ...
> : I am sorry if you'd dealt with this issue in the past,
> I've spent a few
> : hours googling for solr utf-8 query and glassfishv3 utf-8
> uri plus other
> : permutations/combinations but there were seemingly
> endless amounts of
> : chaff that I couldn't find anything useful after scouring
> it for a few
> : hours. I can't decide whether it's a glassfish
> issue or not so I am not
> : sure where to direct my energy. Any tips or advice
> are appreciated!
>
> I suspect if you switched to using POST instead of GET your
> problem would
> go away -- this stems from amiguity in the way HTTP
> servers/browsers deal
> with encoding UTF8 in URLs. a quick search for
> "glassfish url encoding"
> turns up this thread...
>
> http://forums.java.net/jive/thread.jspa?threadID=38020
>
> which refreneces...
>
> http://wiki.glassfish.java.net/Wiki.jsp?page=FaqHttpRequestParameterEncoding
>
> ...it looks like you want to modify the "default-charset
> attribute of the
> <parameter-encoding>"
>
>
> -Hoss
>
Re: Non-English query via Solr Example Admin corrupts text
Posted by Chris Hostetter <ho...@fucit.org>.
: I am using apache-solr-1.4.0.war deployed to glassfishv3 on my
...
: INFO: [] webapp=/apache-solr-1.4.0 path=/select
: params={indent=on&version=2.2&q=numéro&fq=&start=0&rows=10&fl=*,score&qt=standard&wt=standard&explainOther=&hl.fl=}
: hits=0 status=0 QTime=16
...
: In my SolrJ using application, I have a test case which queries for
: "numéro" and succeeds if I use Embedded and fails if I use
: CommonsHttpSolrServer... I don't want to use embedded for a number of
...
: I am sorry if you'd dealt with this issue in the past, I've spent a few
: hours googling for solr utf-8 query and glassfishv3 utf-8 uri plus other
: permutations/combinations but there were seemingly endless amounts of
: chaff that I couldn't find anything useful after scouring it for a few
: hours. I can't decide whether it's a glassfish issue or not so I am not
: sure where to direct my energy. Any tips or advice are appreciated!
I suspect if you switched to using POST instead of GET your problem would
go away -- this stems from amiguity in the way HTTP servers/browsers deal
with encoding UTF8 in URLs. a quick search for "glassfish url encoding"
turns up this thread...
http://forums.java.net/jive/thread.jspa?threadID=38020
which refreneces...
http://wiki.glassfish.java.net/Wiki.jsp?page=FaqHttpRequestParameterEncoding
...it looks like you want to modify the "default-charset attribute of the
<parameter-encoding>"
-Hoss