You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Jörg Agatz <jo...@googlemail.com> on 2010/10/01 13:20:09 UTC
Problem with Indexing
Hallo,
i tryed to index a lot of XML. 700.000 to 800.000 but i becom a Error: and a
Java heap Space error..
i doo this:
find -maxdepth 1 -mindepth 1 -name '*_SEARCH*' -exec java -jar
/opt/solr/apache-solr-1.4.0/example/exampledocs/post.jar '{}' +
and i get this:
...
...
...
...
SimplePostTool: POSTing file e3a02194-cd21-11df-b405-000c29e17baa_SEARCH.xml
SimplePostTool: POSTing file e448d816-cd21-11df-8bf5-000c29e17baa_SEARCH.xml
SimplePostTool: POSTing file e4a5f00a-cd21-11df-8530-000c29e17baa_SEARCH.xml
SimplePostTool: POSTing file e503f452-cd21-11df-ac74-000c29e17baa_SEARCH.xml
SimplePostTool: FATAL: Solr returned an error:
Illegal_character_CTRLCHAR_code_27__at_rowcol_unknownsource_22110
Or i get somethink likethis:
....
...
...
...
...
SimplePostTool: POSTing file a501b15c-cc98-11df-a262-000c29e17baa_SEARCH.xml
SimplePostTool: POSTing file a6cef6de-cc98-11df-84f2-000c29e17baa_SEARCH.xml
SimplePostTool: POSTing file a7ae877c-cc98-11df-b4e4-000c29e17baa_SEARCH.xml
SimplePostTool: POSTing file a7ee32dc-cc98-11df-8b09-000c29e17baa_SEARCH.xml
SimplePostTool: POSTing file a954f8c2-cc98-11df-81f5-000c29e17baa_SEARCH.xml
SimplePostTool: POSTing file a9babbda-cc98-11df-bc9d-000c29e17baa_SEARCH.xml
SimplePostTool: FATAL: Solr returned an error:
Java_heap_space__javalangOutOfMemoryError_Java_heap_space_
SimplePostTool: version 1.2
SimplePostTool: WARNING: Make sure your XML documents are encoded in UTF-8,
other encodings are not currently supported
SimplePostTool: POSTing files to http://localhost:8983/solr/update..
SimplePostTool: POSTing file 5c4a2eb0-cc9f-11df-bbd1-000c29e17baa_SEARCH.xml
SimplePostTool: POSTing file 5caf02d6-cc9f-11df-92cc-000c29e17baa_SEARCH.xml
SimplePostTool: POSTing file 5d58b81c-cc9f-11df-958a-000c29e17baa_SEARCH.xml
SimplePostTool: POSTing file 5e5b2998-cc9f-11df-9a3b-000c29e17baa_SEARCH.xml
...
...
...
Maby you have an Idie for me?
Re: Problem with Indexing
Posted by Gora Mohanty <go...@mimirtech.com>.
On Tue, Oct 12, 2010 at 5:57 PM, Jörg Agatz <jo...@googlemail.com> wrote:
> java -Xmx1024m -jar start.jar
Hmm, could you try increasing the RAM, though this out to be enough?
Maybe I missed this earlier, but what is the size of the data being indexed?
Regards,
Gora
Re: Problem with Indexing
Posted by Jörg Agatz <jo...@googlemail.com>.
java -Xmx1024m -jar start.jar
Re: Problem with Indexing
Posted by Gora Mohanty <go...@mimirtech.com>.
On Mon, Oct 11, 2010 at 1:27 PM, Jörg Agatz <jo...@googlemail.com> wrote:
> ok, i have try it.. and now iget this error:
>
> POSTing file e067f59c-d046-11df-b552-000c29e17baa_SEARCH.xml
> SimplePostTool: FATAL: Solr returned an error:
> this_writer_hit_an_OutOfMemoryError_cannot_flush__javalangIllegalStateException
[...]
Not sure in this particular case, but this looks like Solr is running out of
memory. How much RAM do you have allocated in the Java container
that Solr is running in?
Regards,
Gora
Re: Problem with Indexing
Posted by Jörg Agatz <jo...@googlemail.com>.
ok, i have try it.. and now iget this error:
POSTing file e067f59c-d046-11df-b552-000c29e17baa_SEARCH.xml
SimplePostTool: FATAL: Solr returned an error:
this_writer_hit_an_OutOfMemoryError_cannot_flush__javalangIllegalStateException_this_writer_hit_an_OutOfMemoryError_cannot_flush__at_orgapacheluceneindexIndexWriterdoFlushInternalIndexWriterjava4204__at_orgapacheluceneindexIndexWriterdoFlushIndexWriterjava4192__at_orgapacheluceneindexIndexWriterflushIndexWriterjava4183__at_orgapacheluceneindexIndexWriterupdateDocumentIndexWriterjava2647__at_orgapacheluceneindexIndexWriterupdateDocumentIndexWriterjava2601__at_orgapachesolrupdateDirectUpdateHandler2addDocDirectUpdateHandler2java241__at_orgapachesolrupdateprocessorRunUpdateProcessorprocessAddRunUpdateProcessorFactoryjava61__at_orgapachesolrhandlerXMLLoaderprocessUpdateXMLLoaderjava139__at_orgapachesolrhandlerXMLLoaderloadXMLLoaderjava69__at_orgapachesolrhandlerContentStreamHandlerBasehandleRequestBodyContentStreamHandlerBasejava54__at_orgapachesolrhandlerRequestHandlerBasehandleRequestRequestHandlerBasejava131__at_orgapachesolrcoreSolrCoreexecuteSolrCorejava1316__at_orgapachesolrservletSolrDispatchFilterexecuteSolrDispatchFilterjava338__at_orgapachesolrservletSolrDispatchFilterdoFilterSolrDispatchFilterjava241__at_orgmortbayjettyservletServletHandler$CachedChaindoFilterServletHandlerjava1089__at_orgmortbayjettyservletServletHandlerhandleServletHandlerjava365__at_orgmortbayjettysecuritySecurityHandlerhandleSecurityHandlerjava216__at_orgmortbayjettyservletSessionHandlerhandleSessionHandlerjava181__at_orgmortbayjettyhandlerContextHandlerhandleContextHandlerjava712__at_orgmortbayjettywebappWebAppContexthandleWebAppContextjava405__at_orgmortbayjettyhandlerContextHandlerCollectionhandleContextHandlerCollectionjava211__at_orgmortbayjettyhandlerHandlerCollectionhandleHandlerCollectionjava114__at_orgmortbayjettyhandlerHandlerWrapperhandleHandlerWrapperjava139__at_orgmortbayjettyServerhandleServerjava285__at_
i dont know, how i can index a lot of xml (fast)
Re: Problem with Indexing
Posted by Erick Erickson <er...@gmail.com>.
How much memory are you giving the JVM? Try adding the -Xmx parameter to
your
java invocation. e.g. -Xmx512M
HTH
Erick
On Fri, Oct 1, 2010 at 7:20 AM, Jörg Agatz <jo...@googlemail.com>wrote:
> Hallo,
>
> i tryed to index a lot of XML. 700.000 to 800.000 but i becom a Error: and
> a
> Java heap Space error..
>
> i doo this:
>
> find -maxdepth 1 -mindepth 1 -name '*_SEARCH*' -exec java -jar
> /opt/solr/apache-solr-1.4.0/example/exampledocs/post.jar '{}' +
>
> and i get this:
>
> ...
> ...
> ...
> ...
> SimplePostTool: POSTing file
> e3a02194-cd21-11df-b405-000c29e17baa_SEARCH.xml
> SimplePostTool: POSTing file
> e448d816-cd21-11df-8bf5-000c29e17baa_SEARCH.xml
> SimplePostTool: POSTing file
> e4a5f00a-cd21-11df-8530-000c29e17baa_SEARCH.xml
> SimplePostTool: POSTing file
> e503f452-cd21-11df-ac74-000c29e17baa_SEARCH.xml
> SimplePostTool: FATAL: Solr returned an error:
> Illegal_character_CTRLCHAR_code_27__at_rowcol_unknownsource_22110
>
>
> Or i get somethink likethis:
>
> ....
> ...
> ...
> ...
> ...
> SimplePostTool: POSTing file
> a501b15c-cc98-11df-a262-000c29e17baa_SEARCH.xml
> SimplePostTool: POSTing file
> a6cef6de-cc98-11df-84f2-000c29e17baa_SEARCH.xml
> SimplePostTool: POSTing file
> a7ae877c-cc98-11df-b4e4-000c29e17baa_SEARCH.xml
> SimplePostTool: POSTing file
> a7ee32dc-cc98-11df-8b09-000c29e17baa_SEARCH.xml
> SimplePostTool: POSTing file
> a954f8c2-cc98-11df-81f5-000c29e17baa_SEARCH.xml
> SimplePostTool: POSTing file
> a9babbda-cc98-11df-bc9d-000c29e17baa_SEARCH.xml
> SimplePostTool: FATAL: Solr returned an error:
> Java_heap_space__javalangOutOfMemoryError_Java_heap_space_
> SimplePostTool: version 1.2
> SimplePostTool: WARNING: Make sure your XML documents are encoded in UTF-8,
> other encodings are not currently supported
> SimplePostTool: POSTing files to http://localhost:8983/solr/update..
> SimplePostTool: POSTing file
> 5c4a2eb0-cc9f-11df-bbd1-000c29e17baa_SEARCH.xml
> SimplePostTool: POSTing file
> 5caf02d6-cc9f-11df-92cc-000c29e17baa_SEARCH.xml
> SimplePostTool: POSTing file
> 5d58b81c-cc9f-11df-958a-000c29e17baa_SEARCH.xml
> SimplePostTool: POSTing file
> 5e5b2998-cc9f-11df-9a3b-000c29e17baa_SEARCH.xml
> ...
> ...
> ...
>
>
> Maby you have an Idie for me?
>