A couple newbie questions

[Stuart Smith <st...@yahoo.com>]

Hello!
  I just started using Solr. My general use case is pushing a lot of data from Hbase to solr via an M/R job using Solrj. I have lots of questions, but the ones I'd like to start with are:

(1)
I noticed this:
http://lucene.472066.n3.nabble.com/what-happens-to-docsPending-if-stop-solr-before-commit-td2781493.html

Would seem to indicate that pending documents are commited on restart. This is great! I also noticed, that while there is a lag on start up if I have documents pending - it's only a few minutes or so. But if I issue a commit for the same number of files, the server stays blocked for 20 min or so. It almost seems like it would be a faster to add all my documents and restart the server, rather than issuing a commit. Am I doing something strange? Is this a valid conclusion?

(2)
I'm also getting a lot of errors about invalid UTF-8:

SEVERE: org.apache.solr.common.SolrException: Invalid UTF-8 character 0xffff at char #2380289, byte #2378666)
	at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:79)
	at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:55)
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
	at org.apache.solr.core.SolrCore.execute(SolrCore.java:1360)
	at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)

It could be that the values I have in some of my document fields is indeed invalid. My question is what does this mean when I'm submitting a batch of documents (specifically I'm using Solrj's StreamingUpdateSolrServer w/ a BinaryRequestWriter) - do I:

- lose the whole batch that has the bad document?
- lose the document?
- lose the one field?

I wish it was the third, hope it's the second, and I'm afraid it's the first...

Ooo.. and I guess a third question - I'm having trouble finding a document that describes the overall design/functionality of Solr, something that would help me reason about stuff like "what happens to pending documents when the server restarts" or "does a commit in one indexing thread commit previously added documents from another indexing thread". Both of those I've answered to my satisfaction by looking over the Solr logs & mailing lists, but I'm wondering if there's some documentation I missed somehow..
For example, something like this:
http://hadoop.apache.org/common/docs/current/hdfs_design.html
http://hbase.apache.org/book.html#architecture

Thanks!

Take care,
  -stu