You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Roger Håkansson (Created JIRA)" <ji...@apache.org> on 2012/04/18 20:58:41 UTC
[jira] [Created] (SOLR-3375) Charset problem using HttpSolrServer
instead of CommonsHttpSolrServer
Charset problem using HttpSolrServer instead of CommonsHttpSolrServer
---------------------------------------------------------------------
Key: SOLR-3375
URL: https://issues.apache.org/jira/browse/SOLR-3375
Project: Solr
Issue Type: Bug
Components: clients - java
Affects Versions: 3.6, 4.0, 3.6.1
Reporter: Roger Håkansson
I've written an application which sends PDF files to Solr for indexing, but I also need to index some meta-data which isn't contained inside the PDF.
I recently upgraded to 3.6.0 and when recompiling my app, I got some deprecated messages which mainly was to switch from CommonsHttpSolrServer to HttpSolrServer.
The problem I've noticed since doing this, is that all extra fields which I add is sent to the Solr server as ASCII only, i.e UTF-8/ISO-8859-1 doesn't matter, anything above char 127 is sent as '?'. This was not the behaviour of CommonsHttpSolrServer.
I've tracked it down to a line (271 in 3.6.0) in HttpSolrServer.java which is:
entity.addPart(name, new StringBody(value));
The problem is that StringBody(String text) maps to
StringBody(text, "text/plain", null);
and in
StringBody(String text, String mimeType, Charset charset)
we have this piece of code:
if (charset == null) {
charset = Charset.forName("US-ASCII");
}
this.content = text.getBytes(charset.name());
this.charset = charset;
So unless charset is set everything is converted to US-ASCII.
On the other hand, in CommonsHttpSolrServer.java (line 310 in 3.6.0) there is this line
parts.add(new StringPart(p, v, "UTF-8"));
which adds everything as UTF-8.
The simple solution would be to change the faulty line in HttpSolrServer.java to
entity.addPart(name, new StringBody(value,Charset.forName("UTF-8")));
However, this doesn't work either since my tests have shown that neither Jetty or Tomcat recognizes the strings as UTF-8 but interprets them as 8-bit (8859-1 I guess).
So changing HttpSolrServer.java to
entity.addPart(name, new StringBody(value,Charset.forName("ISO-8859-1")));
actually gives me the same result as using CommonsHttpSolrServer.
But my investigations have shown that there is a difference in how Commons-HttpClient and HttpClient-4.x works.
Commons-HttpClient sends all parameters as regular POST parameters but URLEncoded (/update/extract?param1=value¶m2=value2) while
HttpClient-4.x sends them as multipart/form-data messages and I think that the problem is that each multipart-message should have its own charset parameter.
I.e HttpClient-4.x sends
-----------------------------------------------------------------------------------
--jNljZ3jE1sHG529HrzSjZWYEad-6Wu
Content-Disposition: form-data; name="literal.string_txt"
åäö
-----------------------------------------------------------------------------------
But it should probably send something like this
-----------------------------------------------------------------------------------
--jNljZ3jE1sHG529HrzSjZWYEad-6Wu
Content-Disposition: form-data; name="literal.string_txt"
Content-Type: text/plain; charset=utf-8
åäö
-----------------------------------------------------------------------------------
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3375) Charset problem using HttpSolrServer
instead of CommonsHttpSolrServer
Posted by "Roger Håkansson (Commented JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257400#comment-13257400 ]
Roger Håkansson commented on SOLR-3375:
---------------------------------------
I've downloaded HttpSolrServer.java from trunk and recompiled the 3.6 tree and verified that the fix solves the problem.
> Charset problem using HttpSolrServer instead of CommonsHttpSolrServer
> ---------------------------------------------------------------------
>
> Key: SOLR-3375
> URL: https://issues.apache.org/jira/browse/SOLR-3375
> Project: Solr
> Issue Type: Bug
> Components: clients - java
> Affects Versions: 3.6
> Reporter: Roger Håkansson
> Assignee: Sami Siren
> Fix For: 3.6.1
>
> Attachments: SolrTest.java, commonshttpsolrserver-dump.txt, httpsolrserver-dump.txt
>
>
> I've written an application which sends PDF files to Solr for indexing, but I also need to index some meta-data which isn't contained inside the PDF.
> I recently upgraded to 3.6.0 and when recompiling my app, I got some deprecated messages which mainly was to switch from CommonsHttpSolrServer to HttpSolrServer.
> The problem I've noticed since doing this, is that all extra fields which I add is sent to the Solr server as ASCII only, i.e UTF-8/ISO-8859-1 doesn't matter, anything above char 127 is sent as '?'. This was not the behaviour of CommonsHttpSolrServer.
> I've tracked it down to a line (271 in 3.6.0) in HttpSolrServer.java which is:
> entity.addPart(name, new StringBody(value));
> The problem is that StringBody(String text) maps to
> StringBody(text, "text/plain", null);
> and in
> StringBody(String text, String mimeType, Charset charset)
> we have this piece of code:
> if (charset == null) {
> charset = Charset.forName("US-ASCII");
> }
> this.content = text.getBytes(charset.name());
> this.charset = charset;
> So unless charset is set everything is converted to US-ASCII.
> On the other hand, in CommonsHttpSolrServer.java (line 310 in 3.6.0) there is this line
> parts.add(new StringPart(p, v, "UTF-8"));
> which adds everything as UTF-8.
> The simple solution would be to change the faulty line in HttpSolrServer.java to
> entity.addPart(name, new StringBody(value,Charset.forName("UTF-8")));
> However, this doesn't work either since my tests have shown that neither Jetty or Tomcat recognizes the strings as UTF-8 but interprets them as 8-bit (8859-1 I guess).
> So changing HttpSolrServer.java to
> entity.addPart(name, new StringBody(value,Charset.forName("ISO-8859-1")));
> actually gives me the same result as using CommonsHttpSolrServer.
> But my investigations have shown that there is a difference in how Commons-HttpClient and HttpClient-4.x works.
> Commons-HttpClient sends all parameters as regular POST parameters but URLEncoded (/update/extract?param1=value¶m2=value2) while
> HttpClient-4.x sends them as multipart/form-data messages and I think that the problem is that each multipart-message should have its own charset parameter.
> I.e HttpClient-4.x sends
> -----------------------------------------------------------------------------------
> --jNljZ3jE1sHG529HrzSjZWYEad-6Wu
> Content-Disposition: form-data; name="literal.string_txt"
> åäö
> -----------------------------------------------------------------------------------
> But it should probably send something like this
> -----------------------------------------------------------------------------------
> --jNljZ3jE1sHG529HrzSjZWYEad-6Wu
> Content-Disposition: form-data; name="literal.string_txt"
> Content-Type: text/plain; charset=utf-8
> åäö
> -----------------------------------------------------------------------------------
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3375) Charset problem using HttpSolrServer
instead of CommonsHttpSolrServer
Posted by "Oleg Kalnichevski (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258424#comment-13258424 ]
Oleg Kalnichevski commented on SOLR-3375:
-----------------------------------------
@Roger
> But it also involves getting the guys developing HttpClient to make a change
HttpClient supports two modes for multipart MIME messages: strict and browser compatible. The code snippet you have pasted above is executed in the compatibility mode only. Common browsers include a Content-Type field in body parts that represent a file upload.
Oleg
> Charset problem using HttpSolrServer instead of CommonsHttpSolrServer
> ---------------------------------------------------------------------
>
> Key: SOLR-3375
> URL: https://issues.apache.org/jira/browse/SOLR-3375
> Project: Solr
> Issue Type: Bug
> Components: clients - java
> Affects Versions: 3.6
> Reporter: Roger Håkansson
> Assignee: Sami Siren
> Fix For: 3.6.1
>
> Attachments: SolrTest.java, commonshttpsolrserver-dump.txt, httpsolrserver-dump.txt
>
>
> I've written an application which sends PDF files to Solr for indexing, but I also need to index some meta-data which isn't contained inside the PDF.
> I recently upgraded to 3.6.0 and when recompiling my app, I got some deprecated messages which mainly was to switch from CommonsHttpSolrServer to HttpSolrServer.
> The problem I've noticed since doing this, is that all extra fields which I add is sent to the Solr server as ASCII only, i.e UTF-8/ISO-8859-1 doesn't matter, anything above char 127 is sent as '?'. This was not the behaviour of CommonsHttpSolrServer.
> I've tracked it down to a line (271 in 3.6.0) in HttpSolrServer.java which is:
> entity.addPart(name, new StringBody(value));
> The problem is that StringBody(String text) maps to
> StringBody(text, "text/plain", null);
> and in
> StringBody(String text, String mimeType, Charset charset)
> we have this piece of code:
> if (charset == null) {
> charset = Charset.forName("US-ASCII");
> }
> this.content = text.getBytes(charset.name());
> this.charset = charset;
> So unless charset is set everything is converted to US-ASCII.
> On the other hand, in CommonsHttpSolrServer.java (line 310 in 3.6.0) there is this line
> parts.add(new StringPart(p, v, "UTF-8"));
> which adds everything as UTF-8.
> The simple solution would be to change the faulty line in HttpSolrServer.java to
> entity.addPart(name, new StringBody(value,Charset.forName("UTF-8")));
> However, this doesn't work either since my tests have shown that neither Jetty or Tomcat recognizes the strings as UTF-8 but interprets them as 8-bit (8859-1 I guess).
> So changing HttpSolrServer.java to
> entity.addPart(name, new StringBody(value,Charset.forName("ISO-8859-1")));
> actually gives me the same result as using CommonsHttpSolrServer.
> But my investigations have shown that there is a difference in how Commons-HttpClient and HttpClient-4.x works.
> Commons-HttpClient sends all parameters as regular POST parameters but URLEncoded (/update/extract?param1=value¶m2=value2) while
> HttpClient-4.x sends them as multipart/form-data messages and I think that the problem is that each multipart-message should have its own charset parameter.
> I.e HttpClient-4.x sends
> -----------------------------------------------------------------------------------
> --jNljZ3jE1sHG529HrzSjZWYEad-6Wu
> Content-Disposition: form-data; name="literal.string_txt"
> åäö
> -----------------------------------------------------------------------------------
> But it should probably send something like this
> -----------------------------------------------------------------------------------
> --jNljZ3jE1sHG529HrzSjZWYEad-6Wu
> Content-Disposition: form-data; name="literal.string_txt"
> Content-Type: text/plain; charset=utf-8
> åäö
> -----------------------------------------------------------------------------------
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-3375) Charset problem using HttpSolrServer
instead of CommonsHttpSolrServer
Posted by "Sami Siren (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sami Siren updated SOLR-3375:
-----------------------------
Affects Version/s: (was: 3.6.1)
(was: 4.0)
Fix Version/s: 3.6.1
Thanks Roger for the detailed report. I already fixed some bugs in trunk that I introduced in SOLR-2020 and I believe that this problem should be fixed there (r1327635).
I will leave this issue open so that if there will be a 3.6.1 release this fix must be backported. In the meanwhile on 3.x the only workaround is to use the CommonsHttpSolrServer.
> Charset problem using HttpSolrServer instead of CommonsHttpSolrServer
> ---------------------------------------------------------------------
>
> Key: SOLR-3375
> URL: https://issues.apache.org/jira/browse/SOLR-3375
> Project: Solr
> Issue Type: Bug
> Components: clients - java
> Affects Versions: 3.6
> Reporter: Roger Håkansson
> Fix For: 3.6.1
>
> Attachments: SolrTest.java, commonshttpsolrserver-dump.txt, httpsolrserver-dump.txt
>
>
> I've written an application which sends PDF files to Solr for indexing, but I also need to index some meta-data which isn't contained inside the PDF.
> I recently upgraded to 3.6.0 and when recompiling my app, I got some deprecated messages which mainly was to switch from CommonsHttpSolrServer to HttpSolrServer.
> The problem I've noticed since doing this, is that all extra fields which I add is sent to the Solr server as ASCII only, i.e UTF-8/ISO-8859-1 doesn't matter, anything above char 127 is sent as '?'. This was not the behaviour of CommonsHttpSolrServer.
> I've tracked it down to a line (271 in 3.6.0) in HttpSolrServer.java which is:
> entity.addPart(name, new StringBody(value));
> The problem is that StringBody(String text) maps to
> StringBody(text, "text/plain", null);
> and in
> StringBody(String text, String mimeType, Charset charset)
> we have this piece of code:
> if (charset == null) {
> charset = Charset.forName("US-ASCII");
> }
> this.content = text.getBytes(charset.name());
> this.charset = charset;
> So unless charset is set everything is converted to US-ASCII.
> On the other hand, in CommonsHttpSolrServer.java (line 310 in 3.6.0) there is this line
> parts.add(new StringPart(p, v, "UTF-8"));
> which adds everything as UTF-8.
> The simple solution would be to change the faulty line in HttpSolrServer.java to
> entity.addPart(name, new StringBody(value,Charset.forName("UTF-8")));
> However, this doesn't work either since my tests have shown that neither Jetty or Tomcat recognizes the strings as UTF-8 but interprets them as 8-bit (8859-1 I guess).
> So changing HttpSolrServer.java to
> entity.addPart(name, new StringBody(value,Charset.forName("ISO-8859-1")));
> actually gives me the same result as using CommonsHttpSolrServer.
> But my investigations have shown that there is a difference in how Commons-HttpClient and HttpClient-4.x works.
> Commons-HttpClient sends all parameters as regular POST parameters but URLEncoded (/update/extract?param1=value¶m2=value2) while
> HttpClient-4.x sends them as multipart/form-data messages and I think that the problem is that each multipart-message should have its own charset parameter.
> I.e HttpClient-4.x sends
> -----------------------------------------------------------------------------------
> --jNljZ3jE1sHG529HrzSjZWYEad-6Wu
> Content-Disposition: form-data; name="literal.string_txt"
> åäö
> -----------------------------------------------------------------------------------
> But it should probably send something like this
> -----------------------------------------------------------------------------------
> --jNljZ3jE1sHG529HrzSjZWYEad-6Wu
> Content-Disposition: form-data; name="literal.string_txt"
> Content-Type: text/plain; charset=utf-8
> åäö
> -----------------------------------------------------------------------------------
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Resolved] (SOLR-3375) Charset problem using HttpSolrServer
instead of CommonsHttpSolrServer
Posted by "Sami Siren (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sami Siren resolved SOLR-3375.
------------------------------
Resolution: Fixed
the fix is now committed to 3.6 branch
> Charset problem using HttpSolrServer instead of CommonsHttpSolrServer
> ---------------------------------------------------------------------
>
> Key: SOLR-3375
> URL: https://issues.apache.org/jira/browse/SOLR-3375
> Project: Solr
> Issue Type: Bug
> Components: clients - java
> Affects Versions: 3.6
> Reporter: Roger Håkansson
> Assignee: Sami Siren
> Fix For: 3.6.1
>
> Attachments: SolrTest.java, commonshttpsolrserver-dump.txt, httpsolrserver-dump.txt
>
>
> I've written an application which sends PDF files to Solr for indexing, but I also need to index some meta-data which isn't contained inside the PDF.
> I recently upgraded to 3.6.0 and when recompiling my app, I got some deprecated messages which mainly was to switch from CommonsHttpSolrServer to HttpSolrServer.
> The problem I've noticed since doing this, is that all extra fields which I add is sent to the Solr server as ASCII only, i.e UTF-8/ISO-8859-1 doesn't matter, anything above char 127 is sent as '?'. This was not the behaviour of CommonsHttpSolrServer.
> I've tracked it down to a line (271 in 3.6.0) in HttpSolrServer.java which is:
> entity.addPart(name, new StringBody(value));
> The problem is that StringBody(String text) maps to
> StringBody(text, "text/plain", null);
> and in
> StringBody(String text, String mimeType, Charset charset)
> we have this piece of code:
> if (charset == null) {
> charset = Charset.forName("US-ASCII");
> }
> this.content = text.getBytes(charset.name());
> this.charset = charset;
> So unless charset is set everything is converted to US-ASCII.
> On the other hand, in CommonsHttpSolrServer.java (line 310 in 3.6.0) there is this line
> parts.add(new StringPart(p, v, "UTF-8"));
> which adds everything as UTF-8.
> The simple solution would be to change the faulty line in HttpSolrServer.java to
> entity.addPart(name, new StringBody(value,Charset.forName("UTF-8")));
> However, this doesn't work either since my tests have shown that neither Jetty or Tomcat recognizes the strings as UTF-8 but interprets them as 8-bit (8859-1 I guess).
> So changing HttpSolrServer.java to
> entity.addPart(name, new StringBody(value,Charset.forName("ISO-8859-1")));
> actually gives me the same result as using CommonsHttpSolrServer.
> But my investigations have shown that there is a difference in how Commons-HttpClient and HttpClient-4.x works.
> Commons-HttpClient sends all parameters as regular POST parameters but URLEncoded (/update/extract?param1=value¶m2=value2) while
> HttpClient-4.x sends them as multipart/form-data messages and I think that the problem is that each multipart-message should have its own charset parameter.
> I.e HttpClient-4.x sends
> -----------------------------------------------------------------------------------
> --jNljZ3jE1sHG529HrzSjZWYEad-6Wu
> Content-Disposition: form-data; name="literal.string_txt"
> åäö
> -----------------------------------------------------------------------------------
> But it should probably send something like this
> -----------------------------------------------------------------------------------
> --jNljZ3jE1sHG529HrzSjZWYEad-6Wu
> Content-Disposition: form-data; name="literal.string_txt"
> Content-Type: text/plain; charset=utf-8
> åäö
> -----------------------------------------------------------------------------------
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-3375) Charset problem using HttpSolrServer
instead of CommonsHttpSolrServer
Posted by "Roger Håkansson (Updated JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Roger Håkansson updated SOLR-3375:
----------------------------------
Attachment: SolrTest.java
Test program to show the problem.
Pass a URL to a Solr server as first arg and a PDF file as second.
Then search for id 1234567890 and 1234567891 and see the difference in string_txt/string2_txt between the documents
> Charset problem using HttpSolrServer instead of CommonsHttpSolrServer
> ---------------------------------------------------------------------
>
> Key: SOLR-3375
> URL: https://issues.apache.org/jira/browse/SOLR-3375
> Project: Solr
> Issue Type: Bug
> Components: clients - java
> Affects Versions: 3.6, 4.0, 3.6.1
> Reporter: Roger Håkansson
> Attachments: SolrTest.java
>
>
> I've written an application which sends PDF files to Solr for indexing, but I also need to index some meta-data which isn't contained inside the PDF.
> I recently upgraded to 3.6.0 and when recompiling my app, I got some deprecated messages which mainly was to switch from CommonsHttpSolrServer to HttpSolrServer.
> The problem I've noticed since doing this, is that all extra fields which I add is sent to the Solr server as ASCII only, i.e UTF-8/ISO-8859-1 doesn't matter, anything above char 127 is sent as '?'. This was not the behaviour of CommonsHttpSolrServer.
> I've tracked it down to a line (271 in 3.6.0) in HttpSolrServer.java which is:
> entity.addPart(name, new StringBody(value));
> The problem is that StringBody(String text) maps to
> StringBody(text, "text/plain", null);
> and in
> StringBody(String text, String mimeType, Charset charset)
> we have this piece of code:
> if (charset == null) {
> charset = Charset.forName("US-ASCII");
> }
> this.content = text.getBytes(charset.name());
> this.charset = charset;
> So unless charset is set everything is converted to US-ASCII.
> On the other hand, in CommonsHttpSolrServer.java (line 310 in 3.6.0) there is this line
> parts.add(new StringPart(p, v, "UTF-8"));
> which adds everything as UTF-8.
> The simple solution would be to change the faulty line in HttpSolrServer.java to
> entity.addPart(name, new StringBody(value,Charset.forName("UTF-8")));
> However, this doesn't work either since my tests have shown that neither Jetty or Tomcat recognizes the strings as UTF-8 but interprets them as 8-bit (8859-1 I guess).
> So changing HttpSolrServer.java to
> entity.addPart(name, new StringBody(value,Charset.forName("ISO-8859-1")));
> actually gives me the same result as using CommonsHttpSolrServer.
> But my investigations have shown that there is a difference in how Commons-HttpClient and HttpClient-4.x works.
> Commons-HttpClient sends all parameters as regular POST parameters but URLEncoded (/update/extract?param1=value¶m2=value2) while
> HttpClient-4.x sends them as multipart/form-data messages and I think that the problem is that each multipart-message should have its own charset parameter.
> I.e HttpClient-4.x sends
> -----------------------------------------------------------------------------------
> --jNljZ3jE1sHG529HrzSjZWYEad-6Wu
> Content-Disposition: form-data; name="literal.string_txt"
> åäö
> -----------------------------------------------------------------------------------
> But it should probably send something like this
> -----------------------------------------------------------------------------------
> --jNljZ3jE1sHG529HrzSjZWYEad-6Wu
> Content-Disposition: form-data; name="literal.string_txt"
> Content-Type: text/plain; charset=utf-8
> åäö
> -----------------------------------------------------------------------------------
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Assigned] (SOLR-3375) Charset problem using HttpSolrServer
instead of CommonsHttpSolrServer
Posted by "Sami Siren (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sami Siren reassigned SOLR-3375:
--------------------------------
Assignee: Sami Siren
> Charset problem using HttpSolrServer instead of CommonsHttpSolrServer
> ---------------------------------------------------------------------
>
> Key: SOLR-3375
> URL: https://issues.apache.org/jira/browse/SOLR-3375
> Project: Solr
> Issue Type: Bug
> Components: clients - java
> Affects Versions: 3.6
> Reporter: Roger Håkansson
> Assignee: Sami Siren
> Fix For: 3.6.1
>
> Attachments: SolrTest.java, commonshttpsolrserver-dump.txt, httpsolrserver-dump.txt
>
>
> I've written an application which sends PDF files to Solr for indexing, but I also need to index some meta-data which isn't contained inside the PDF.
> I recently upgraded to 3.6.0 and when recompiling my app, I got some deprecated messages which mainly was to switch from CommonsHttpSolrServer to HttpSolrServer.
> The problem I've noticed since doing this, is that all extra fields which I add is sent to the Solr server as ASCII only, i.e UTF-8/ISO-8859-1 doesn't matter, anything above char 127 is sent as '?'. This was not the behaviour of CommonsHttpSolrServer.
> I've tracked it down to a line (271 in 3.6.0) in HttpSolrServer.java which is:
> entity.addPart(name, new StringBody(value));
> The problem is that StringBody(String text) maps to
> StringBody(text, "text/plain", null);
> and in
> StringBody(String text, String mimeType, Charset charset)
> we have this piece of code:
> if (charset == null) {
> charset = Charset.forName("US-ASCII");
> }
> this.content = text.getBytes(charset.name());
> this.charset = charset;
> So unless charset is set everything is converted to US-ASCII.
> On the other hand, in CommonsHttpSolrServer.java (line 310 in 3.6.0) there is this line
> parts.add(new StringPart(p, v, "UTF-8"));
> which adds everything as UTF-8.
> The simple solution would be to change the faulty line in HttpSolrServer.java to
> entity.addPart(name, new StringBody(value,Charset.forName("UTF-8")));
> However, this doesn't work either since my tests have shown that neither Jetty or Tomcat recognizes the strings as UTF-8 but interprets them as 8-bit (8859-1 I guess).
> So changing HttpSolrServer.java to
> entity.addPart(name, new StringBody(value,Charset.forName("ISO-8859-1")));
> actually gives me the same result as using CommonsHttpSolrServer.
> But my investigations have shown that there is a difference in how Commons-HttpClient and HttpClient-4.x works.
> Commons-HttpClient sends all parameters as regular POST parameters but URLEncoded (/update/extract?param1=value¶m2=value2) while
> HttpClient-4.x sends them as multipart/form-data messages and I think that the problem is that each multipart-message should have its own charset parameter.
> I.e HttpClient-4.x sends
> -----------------------------------------------------------------------------------
> --jNljZ3jE1sHG529HrzSjZWYEad-6Wu
> Content-Disposition: form-data; name="literal.string_txt"
> åäö
> -----------------------------------------------------------------------------------
> But it should probably send something like this
> -----------------------------------------------------------------------------------
> --jNljZ3jE1sHG529HrzSjZWYEad-6Wu
> Content-Disposition: form-data; name="literal.string_txt"
> Content-Type: text/plain; charset=utf-8
> åäö
> -----------------------------------------------------------------------------------
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3375) Charset problem using HttpSolrServer
instead of CommonsHttpSolrServer
Posted by "Roger Håkansson (Commented JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257080#comment-13257080 ]
Roger Håkansson commented on SOLR-3375:
---------------------------------------
After having to go through a ton of code back an forth, I've come to this conclusion.
First, the reason for the initial problem is that CommonsHttpSolrServer will make the client send an ContentStreamUpdateRequest as a POST with all parameters in the URL plus the file data. HttpSolrServer on the other hand sends everything as different parts in a multipart-post, one part for each parameter.
Regarding fixing HttpSolrServer, I've tested the two solutions I previously described and both seems to work but might have totally different implications.
First solution is to change HttpSolrServer.java so
{code}
entity.addPart(name, new StringBody(value));
{code}
is changed to
{code}
entity.addPart(name, new StringBody(value, "text/plain", Charset.forName("ISO-8859-1")));
{code}
What implications this might have I'm not sure, it might be wrong according to some standard to assume 8859-1 and it doesn't solve this problem universally. But both the dist-Jetty and my Tomcat(7.0.22) work with this fix.
Second solution is a more generic fix.
This involves the same change as the previous, except the charset is "UTF-8".
{code}
entity.addPart(name, new StringBody(value, "text/plain", Charset.forName("UTF-8")));
{code}
But it also involves getting the guys developing HttpClient to make a change.
Currently their code looks like this
{code:title=HttpMultiPart.java}
String filename = part.getBody().getFilename();
if (filename != null) {
MinimalField ct = part.getHeader().getField(MIME.CONTENT_TYPE);
writeField(ct, this.charset, out);
}
{code}
If they would change their code to not only add Content-Type when there is a filename, but always do it.
Together with the fix in HttpSolrServer.java that would make sure that UTF-8 encoded strings always would be sent to the server.
But this requires them to make a change...
A third option would be to get HttpClient to post just like Commons-HttpClient did, i.e no multipart posting, but what that might require in terms of work I have no idea
> Charset problem using HttpSolrServer instead of CommonsHttpSolrServer
> ---------------------------------------------------------------------
>
> Key: SOLR-3375
> URL: https://issues.apache.org/jira/browse/SOLR-3375
> Project: Solr
> Issue Type: Bug
> Components: clients - java
> Affects Versions: 3.6, 4.0, 3.6.1
> Reporter: Roger Håkansson
> Attachments: SolrTest.java
>
>
> I've written an application which sends PDF files to Solr for indexing, but I also need to index some meta-data which isn't contained inside the PDF.
> I recently upgraded to 3.6.0 and when recompiling my app, I got some deprecated messages which mainly was to switch from CommonsHttpSolrServer to HttpSolrServer.
> The problem I've noticed since doing this, is that all extra fields which I add is sent to the Solr server as ASCII only, i.e UTF-8/ISO-8859-1 doesn't matter, anything above char 127 is sent as '?'. This was not the behaviour of CommonsHttpSolrServer.
> I've tracked it down to a line (271 in 3.6.0) in HttpSolrServer.java which is:
> entity.addPart(name, new StringBody(value));
> The problem is that StringBody(String text) maps to
> StringBody(text, "text/plain", null);
> and in
> StringBody(String text, String mimeType, Charset charset)
> we have this piece of code:
> if (charset == null) {
> charset = Charset.forName("US-ASCII");
> }
> this.content = text.getBytes(charset.name());
> this.charset = charset;
> So unless charset is set everything is converted to US-ASCII.
> On the other hand, in CommonsHttpSolrServer.java (line 310 in 3.6.0) there is this line
> parts.add(new StringPart(p, v, "UTF-8"));
> which adds everything as UTF-8.
> The simple solution would be to change the faulty line in HttpSolrServer.java to
> entity.addPart(name, new StringBody(value,Charset.forName("UTF-8")));
> However, this doesn't work either since my tests have shown that neither Jetty or Tomcat recognizes the strings as UTF-8 but interprets them as 8-bit (8859-1 I guess).
> So changing HttpSolrServer.java to
> entity.addPart(name, new StringBody(value,Charset.forName("ISO-8859-1")));
> actually gives me the same result as using CommonsHttpSolrServer.
> But my investigations have shown that there is a difference in how Commons-HttpClient and HttpClient-4.x works.
> Commons-HttpClient sends all parameters as regular POST parameters but URLEncoded (/update/extract?param1=value¶m2=value2) while
> HttpClient-4.x sends them as multipart/form-data messages and I think that the problem is that each multipart-message should have its own charset parameter.
> I.e HttpClient-4.x sends
> -----------------------------------------------------------------------------------
> --jNljZ3jE1sHG529HrzSjZWYEad-6Wu
> Content-Disposition: form-data; name="literal.string_txt"
> åäö
> -----------------------------------------------------------------------------------
> But it should probably send something like this
> -----------------------------------------------------------------------------------
> --jNljZ3jE1sHG529HrzSjZWYEad-6Wu
> Content-Disposition: form-data; name="literal.string_txt"
> Content-Type: text/plain; charset=utf-8
> åäö
> -----------------------------------------------------------------------------------
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-3375) Charset problem using HttpSolrServer
instead of CommonsHttpSolrServer
Posted by "Roger Håkansson (Updated JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Roger Håkansson updated SOLR-3375:
----------------------------------
Attachment: httpsolrserver-dump.txt
commonshttpsolrserver-dump.txt
Uploaded network dumps which shows difference between CommonsHttpSolrServer and HttpSolrServer
> Charset problem using HttpSolrServer instead of CommonsHttpSolrServer
> ---------------------------------------------------------------------
>
> Key: SOLR-3375
> URL: https://issues.apache.org/jira/browse/SOLR-3375
> Project: Solr
> Issue Type: Bug
> Components: clients - java
> Affects Versions: 3.6, 4.0, 3.6.1
> Reporter: Roger Håkansson
> Attachments: SolrTest.java, commonshttpsolrserver-dump.txt, httpsolrserver-dump.txt
>
>
> I've written an application which sends PDF files to Solr for indexing, but I also need to index some meta-data which isn't contained inside the PDF.
> I recently upgraded to 3.6.0 and when recompiling my app, I got some deprecated messages which mainly was to switch from CommonsHttpSolrServer to HttpSolrServer.
> The problem I've noticed since doing this, is that all extra fields which I add is sent to the Solr server as ASCII only, i.e UTF-8/ISO-8859-1 doesn't matter, anything above char 127 is sent as '?'. This was not the behaviour of CommonsHttpSolrServer.
> I've tracked it down to a line (271 in 3.6.0) in HttpSolrServer.java which is:
> entity.addPart(name, new StringBody(value));
> The problem is that StringBody(String text) maps to
> StringBody(text, "text/plain", null);
> and in
> StringBody(String text, String mimeType, Charset charset)
> we have this piece of code:
> if (charset == null) {
> charset = Charset.forName("US-ASCII");
> }
> this.content = text.getBytes(charset.name());
> this.charset = charset;
> So unless charset is set everything is converted to US-ASCII.
> On the other hand, in CommonsHttpSolrServer.java (line 310 in 3.6.0) there is this line
> parts.add(new StringPart(p, v, "UTF-8"));
> which adds everything as UTF-8.
> The simple solution would be to change the faulty line in HttpSolrServer.java to
> entity.addPart(name, new StringBody(value,Charset.forName("UTF-8")));
> However, this doesn't work either since my tests have shown that neither Jetty or Tomcat recognizes the strings as UTF-8 but interprets them as 8-bit (8859-1 I guess).
> So changing HttpSolrServer.java to
> entity.addPart(name, new StringBody(value,Charset.forName("ISO-8859-1")));
> actually gives me the same result as using CommonsHttpSolrServer.
> But my investigations have shown that there is a difference in how Commons-HttpClient and HttpClient-4.x works.
> Commons-HttpClient sends all parameters as regular POST parameters but URLEncoded (/update/extract?param1=value¶m2=value2) while
> HttpClient-4.x sends them as multipart/form-data messages and I think that the problem is that each multipart-message should have its own charset parameter.
> I.e HttpClient-4.x sends
> -----------------------------------------------------------------------------------
> --jNljZ3jE1sHG529HrzSjZWYEad-6Wu
> Content-Disposition: form-data; name="literal.string_txt"
> åäö
> -----------------------------------------------------------------------------------
> But it should probably send something like this
> -----------------------------------------------------------------------------------
> --jNljZ3jE1sHG529HrzSjZWYEad-6Wu
> Content-Disposition: form-data; name="literal.string_txt"
> Content-Type: text/plain; charset=utf-8
> åäö
> -----------------------------------------------------------------------------------
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org