[jira] Issue Comment Edited: (SOLR-486) Support binary formats for QueryresponseWriter

["Noble Paul (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12605126#action_12605126 ] 

noble.paul edited comment on SOLR-486 at 6/16/08 9:04 AM:
----------------------------------------------------------

 If we take a look at the data that is written down by NamedListCodec there are a lot of "names" which are repeated.  If we could avoid the repetitions we can achieve better optimization. 
Can we have another type EXTERN_STRING 
The NamedListCodec maintains a Map<String,Integer>  of EXTERN_STRING vs index as it is written out. When the same string is written it checks up in the List whether it already has a reference.

While decoding all the EXTERN_STRING values are copied into a List <String>. When an EXTERN_STRING with an index comes it is copied from the List.

{code:title=NamedListCodec.java}
  private int stringsCount  =  0;
  private Map<String,Integer> stringsMap;
  private List<String > stringsList;
  public void writeExternString(String s) throws IOException {
    if(s == null) {
      writeTag(NULL) ;
      return;
    }
    Integer idx = stringsMap == null ? null : stringsMap.get(s);
    if(idx == null) idx =0;
    writeTag(EXTERN_STRING,idx);
    if(idx == 0){
      writeStr(s);
      if(stringsMap == null) stringsMap = new HashMap<String, Integer>();
      stringsMap.put(s,++stringsCount);
    }

  }
  public String  readExternString(FastInputStream fis) throws IOException {
    int idx = readSize(fis);
    if (idx != 0) {// idx != 0 is the index of the extern string
      return stringsList.get(idx-1);
    } else {// idx == 0 means it has a string value
      String s = (String) readVal(fis);
      if(stringsList == null ) stringsList = new ArrayList<String>();
      stringsList.add(s);
      return s;
    }
  }
{code}

      was (Author: noble.paul):
     If we take a look at the data that is written down by NamedListCodec there are a lot of "names" which are repeated.  If we could avoid the repetitions we can achieve better optimization. 
Can we have another type EXTERN_STRING 
The NamedListCodec maintains a Map<String,Integer>  of EXTERN_STRING vs index as it is written out. When the same string is written it checks up in the List whether it already has a reference.

While decoding all the EXTERN_STRING values are copied into a List <String>. When an EXTERN_STRING with an index comes it is copied from the List.

{code:title=NamedListCodec.java}
private int stringsCount  =  -1;
  private Map<String,Integer> stringsMap;
  private List<String > stringsList;
  public void writeExternString(String s) throws IOException {
    writeTag(EXTERN_STRING);
    if(s == null) {
      writeTag(NULL) ;
      return;
    }
    if(stringsMap.containsKey(s)){
      writeInt(stringsMap.get(s));
    } else {
      writeStr(s);
      stringsCount++;
      if(stringsMap == null) stringsMap = new HashMap<String, Integer>();
      stringsMap.put(s,stringsCount);
    }

  }
  public String  readExternString(FastInputStream fis) throws IOException {
    Object o = readVal(fis);
    if(o == null) return null;
    if (o instanceof String) {
      String s = (String) o;
      if(stringsList == null ) stringsList = new ArrayList<String>();
      stringsList.add(s);
      return s;
    } else {// this must be an integer
      int index = (Integer)o;
      return stringsList.get(index);
    }
  }

{code}
  
> Support binary formats for QueryresponseWriter
> ----------------------------------------------
>
>                 Key: SOLR-486
>                 URL: https://issues.apache.org/jira/browse/SOLR-486
>             Project: Solr
>          Issue Type: Improvement
>          Components: clients - java, search
>            Reporter: Noble Paul
>            Assignee: Yonik Seeley
>             Fix For: 1.3
>
>         Attachments: SOLR-486.patch, solr-486.patch, SOLR-486.patch, SOLR-486.patch, SOLR-486.patch, SOLR-486.patch, SOLR-486.patch, SOLR-486.patch, SOLR-486.patch, SOLR-486.patch
>
>
> QueryResponse writer only allows text data to be written.
> So it is not possible to implement a binary protocol . Create another interface which has a method 
> write(OutputStream os, SolrQueryRequest request, SolrQueryResponse response)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.