IndexableBinaryStringTools (was FieldCache)

[Mathias Walter <ma...@gmx.net>]

Hi,

> > [...] I tried to use IndexableBinaryStringTools to re-encode my 11 byte
> > array. The size was increased to 7 characters (= 14 bytes)
> > which is still a gain of more than 50 percent compared to the UTF8
> > encoding. BTW: I found no sample how to use the
> > IndexableBinaryStringTools class except in the unit tests.
> 
> IndexableBinaryStringTools will eventually be deprecated and then dropped, in favor of native
> indexable/searchable binary terms.  More work is required before these are possible, though.
> 
> Well-maintained unit tests are not a bad way to describe functionality...

Sure, but there is no unit test for Solr.

> > I assume that the char[] returned form IndexableBinaryStringTools.encode
> > is encoded in UTF-8 again and then stored. At some point
> > the information is lost and cannot be recovered.
> 
> Can you give an example?  This should not happen.

It's hard to give an example output, because the binary string representation contains unprintiple characters. I'll try to explain what I'm doing.

My character array returned by IndexableBinaryStringTools.encode looks like following:

char[] encoded = new char[] {0, 8508, 3392, 64, 0, 8, 0, 0};

Then I add it to a SolrInputDocument:

SolrInputDocument doc = new SolrInputDocument();
doc.addField("id", new String(encoded));

If I now print the SolrInputDocument using System.out.println(doc), the String representation of the character array is correct.

Then I add it to a RAMDirectory:

ArrayList<SolrInputDocument> docs = new ArrayList<SolrInputDocument>();
docs.add(doc);
solrServer.add(docs);
solrServer.commit();

... and immediately retrieve it like follows:

SolrQuery query = new SolrQuery();
query.setQuery("*:*");
QueryResponse rsp = solrServer.query(query);
SolrDocumentList docList = rsp.getResults();
System.out.println(docList);

Now the string representation of the SolrDocuments ID looks different than that of the SolrInputDocument.

If I do not create a new string in doc.addField, just the string representation of the array address will be added the the SolrInputDocument.

BTW: I've tested it with EmbeddedSolrServer and Solr/Lucene trunk.

Why has the string representation changed? From the changed string I cannot decode the correct ID.

--
Kind regards,
Mathias

RE: IndexableBinaryStringTools (was FieldCache)

[Steven A Rowe <sa...@syr.edu>]

On 11/13/2010 at 2:04 PM, Yonik Seeley wrote:
n Sat, Nov 13, 2010 at 1:50 PM, Steven A Rowe <sa...@syr.edu> wrote:
> > Looks to me like the returned value is in a Solr-internal form of XML
> > character escaping: \u0000 is represented as "#0;" and \u0008 is
> > represented as "#8;".  (The escaping code is in
> > solr/src/java/org/apache/common/util/XML.java.)
> 
> Yep, there is no legal way to represent some unicode code points in XML.

Right - the real fix here (as you pointed out on #lucene) is to not use XML transports.

> > You can get the value back in its original binary form by unescaping the
> > /#[0-9]+;/ format.  Here is a test illustrating this fix that I added to
> > SolrExampleTests, then ran from SolrExampleEmbeddedTest:
> 
> The problem here is that one might then unescape what was meant to be
> a literal "#8;"
> One could come up with a full escaping mechanism over XML I suppose...
> but I'm not sure it would be worth it.

s/illustrating this fix/exposing this dirty hack/ :)

Steve

Re: IndexableBinaryStringTools (was FieldCache)

[Yonik Seeley <yo...@lucidimagination.com>]

On Sat, Nov 13, 2010 at 1:50 PM, Steven A Rowe <sa...@syr.edu> wrote:
> Looks to me like the returned value is in a Solr-internal form of XML character escaping: \u0000 is represented as "#0;" and \u0008 is represented as "#8;".  (The escaping code is in solr/src/java/org/apache/common/util/XML.java.)

Yep, there is no legal way to represent some unicode code points in XML.

> You can get the value back in its original binary form by unescaping the /#[0-9]+;/ format.  Here is a test illustrating this fix that I added to SolrExampleTests, then ran from SolrExampleEmbeddedTest:

The problem here is that one might then unescape what was meant to be
a literal "#8;"
One could come up with a full escaping mechanism over XML I suppose...
but I'm not sure it would be worth it.

-Yonik
http://www.lucidimagination.com

RE: IndexableBinaryStringTools (was FieldCache)

[Steven A Rowe <sa...@syr.edu>]

Hi Mathias,

> > > I assume that the char[] returned form
> > > IndexableBinaryStringTools.encode is encoded in UTF-8 again
> > > and then stored. At some point the information is lost and
> > > cannot be recovered.
> >
> > Can you give an example?  This should not happen.
> 
> My character array returned by IndexableBinaryStringTools.encode looks
> like following:
> 
> char[] encoded = new char[] {0, 8508, 3392, 64, 0, 8, 0, 0};
[...]
> BTW: I've tested it with EmbeddedSolrServer and Solr/Lucene trunk.
> 
> Why has the string representation changed? From the changed string I
> cannot decode the correct ID.

Looks to me like the returned value is in a Solr-internal form of XML character escaping: \u0000 is represented as "#0;" and \u0008 is represented as "#8;".  (The escaping code is in solr/src/java/org/apache/common/util/XML.java.)  

You can get the value back in its original binary form by unescaping the /#[0-9]+;/ format.  Here is a test illustrating this fix that I added to SolrExampleTests, then ran from SolrExampleEmbeddedTest:

==============
  @Test
  public void testIndexableBinary() throws Exception {
    // Empty the database...
    server.deleteByQuery( "*:*" );// delete everything!
    server.commit();
    assertNumFound( "*:*", 0 ); // make sure it got in
 
    byte[] binary = new byte[] 
      { (byte)0, (byte)0, (byte)0x84, (byte)0xF0, (byte)0x6A, (byte)0, 
        (byte)4, (byte)0, (byte)0,    (byte)0,    (byte)2,    (byte)0 };
    int encodedLen = IndexableBinaryStringTools.getEncodedLength
      (binary, 0, binary.length);
    char encoded[] = new char[encodedLen];
    IndexableBinaryStringTools.encode
      (binary, 0, binary.length, encoded, 0, encoded.length);
    final String encodedString = new String(encoded);
    log.info("Encoded: " + stringToIntSequence(encodedString));
    // Expected encoded: {         0, 8508, 3392,   64,    0,    8,    0,    0 }
    String expectedEncoded = "\u0000\u213C\u0D40\u0040\u0000\u0008\u0000\u0000";
    assertEquals(stringToIntSequence(expectedEncoded),
                 stringToIntSequence(encodedString));
      
    SolrInputDocument doc = new SolrInputDocument();
    doc.addField("id", encodedString);
    server.add(doc);
    server.commit();
        
    SolrQuery query = new SolrQuery();
    query.setQuery("*:*");
    QueryResponse rsp = server.query(query);
    SolrDocument retrievedDoc = rsp.getResults().get(0);
    String retrievedEncoded = (String)retrievedDoc.getFieldValue("id");
    String unescapedRetrievedEncoded = unescapeSolrXMLEscaping(retrievedEncoded);
    assertEquals(stringToIntSequence(encodedString), 
                 stringToIntSequence(unescapedRetrievedEncoded));
  }
    
  String stringToIntSequence(String str) {
    StringBuilder builder = new StringBuilder();
    for (int chnum = 0 ; chnum < str.length() ; ++chnum) {
      if (chnum > 0) {
        builder.append(", ");
      }
      builder.append((int)str.charAt(chnum))
        .append(" (").append(str.charAt(chnum)).append(")");
    }
    return builder.toString();
  }
  String unescapeSolrXMLEscaping(String escaped) {
    StringBuffer unescaped = new StringBuffer();
    Matcher matcher = Pattern.compile("#(\\d+);").matcher(escaped);
    while (matcher.find()) {
      String replacement = String.format
        ("%c",(char)Integer.parseInt(matcher.group(1)));
      matcher.appendReplacement(unescaped, replacement); 
    }
    matcher.appendTail(unescaped);
    return unescaped.toString();
  }
==============

Steve