[jira] [Commented] (NUTCH-2198) Indexing binary content by index-html causes Solr Exception

["Sebastian Nagel (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NUTCH-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15090625#comment-15090625 ] 

Sebastian Nagel commented on NUTCH-2198:
----------------------------------------

Tried to reproduce the Solr exception by indexing on of the JPEGs shown in the log snippet (ciencia11.jpg).
* the Solr exception is not caused by this image (or Solr 4.10.4 is safe)
* however, the indexed rawcontent is modified. E.g., the 4 leading bytes are stripped:
{noformat}
% od -tcx1 ciencia11.jpg | head -2
0000000 377 330 377 341  \v   /   E   x   i   f  \0  \0   M   M  \0   *
         ff  d8  ff  e1  0b  2f  45  78  69  66  00  00  4d  4d  00  2a
{noformat}
vs.
{noformat}
% curl -s 'http://localhost:8983/solr/collection1/select?q=url%3A%22http%3A%2F%2Flocalhost%2Fnutch%2Ftest%2Fciencia11.jpg%22&wt=json&indent=true'
{
  "responseHeader":{
    "status":0,
    "QTime":0,
    "params":{
      "q":"url:\"http://localhost/nutch/test/ciencia11.jpg\"",
      "indent":"true",
      "wt":"json"}},
  "response":{"numFound":1,"start":0,"docs":[
      {
        "tstamp":"1970-01-01T00:00:00Z",
        "rawcontent":"#11;/Exif#0;#0;MM#0;*#0;#0;#0;#8;#0; ...
{noformat}

We need a different mechanism to index HTML or binary content -- as binary field, converting it to Base64, etc. Forcing a string conversion by a platform-dependent charset and then stripping some (but not all!) binary characters away is surely no proper solution.

> Indexing binary content by index-html causes Solr Exception
> -----------------------------------------------------------
>
>                 Key: NUTCH-2198
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2198
>             Project: Nutch
>          Issue Type: Bug
>          Components: indexer
>    Affects Versions: 2.3.1
>            Reporter: Sebastian Nagel
>             Fix For: 2.4
>
>
> (reported by [~kalanya] in NUTCH-2168)
> If raw binary is indexed using the plugin index-html this may cause an exception in Solr:
> {noformat}
> 2016-01-05 12:28:00,152 INFO html.HtmlIndexingFilter - Html indexing for: http://ujiapps.uji.es/com/investigacio/img/ciencia11.jpg
> 2016-01-05 12:28:00,163 INFO html.HtmlIndexingFilter - Html indexing for: http://ujiapps.uji.es/serveis/cd/bib/reservori/2015/e-llibres/
> 2016-01-05 12:28:00,164 INFO solr.SolrIndexWriter - Adding 250 documents
> 2016-01-05 12:28:00,531 INFO solr.SolrIndexWriter - Adding 250 documents
> 2016-01-05 12:28:00,842 WARN mapred.LocalJobRunner - job_local1207147570_0001
> java.lang.Exception: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: [was class java.io.CharConversionException] Invalid UTF-8 character 0xfffe at char #137317, byte #139263)
> {noformat}
> The index-html plugin tries to treat any raw content as readable content converting it to a String based on the platform-dependent charset (cf. [Scanner API docs|http://docs.oracle.com/javase/7/docs/api/java/util/Scanner.html]):
> {code:title=HtmlIndexingFilter.java}
>             Scanner scanner = new Scanner(arrayInputStream);
>             scanner.useDelimiter("\\Z");//To read all scanner content in one String
>             String data = "";
>             if (scanner.hasNext()) {
>                 data = scanner.next();
>             }
>             doc.add("rawcontent", StringUtil.cleanField(data));
> {code}
> The field "rawcontent" is of type "string":
> {code:xml|title=conf/schema.xml}
>     <!-- fields for index-html plugin
>          Note: although raw document content may be binary,
>                index-html adds a String to the index field -->
>     <field name="rawcontent" type="string" stored="true" indexed="false"/>
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)