Re: Resume Solr indexing CSV after exception

[David George <da...@gmail.com>]

I modified TikaEntityProcessor to ignore these exceptions.:

If the Tika Entity processor encounters an exception it will stop indexing.
I had to make two fixes to TikaEntityProcessor to work around this problem.

>From the Solr SVN trunk edit the file:

~/src/solr-svn/trunk/solr/contrib/dataimporthandler/src/extras/main/java/org/apache/solr/handler/dataimport/TikaEntityProcessor.jar

First of all if a file is not found on the disk we want to continue
indexing. At the top of nextRow() add

File f = new File (context.getResolvedEntityAttribute(URL));
if (! f.exists()) {
  return null;
}

Secondly if the document parser throws an error, for example certain PDF
revisions can cause the PDFBox parser to barf, we will trap the exception
and continue:

try {
  tikaParser.parse(is, contentHandler, metadata , new ParseContext());
} catch (Exception e) {
  return null;
} finally {
  IOUtils.closeQuietly(is);
}

We will also close IOUtils in the finally section which is not done in the
original code. Build and deploy the extras.jar in the solr-instance/lib
directory. 

see also: http://www.abcseo.com/tech/search/solr-and-liferay-integration
-- 
View this message in context: http://lucene.472066.n3.nabble.com/Resume-Solr-indexing-CSV-after-exception-tp878801p888143.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Resume Solr indexing CSV after exception

[Brad Greenlee <br...@footle.org>]

Why not just use the onError attribute on entity? The default is to abort,
but you can also specify "skip" to skip the current document, or "continue"
to continue as if the error never happened. See
http://wiki.apache.org/solr/DataImportHandler#Schema_for_the_data_config

Brad

On Fri, Jun 11, 2010 at 3:00 AM, David George <da...@gmail.com>wrote:

>
> I modified TikaEntityProcessor to ignore these exceptions.:
>
> If the Tika Entity processor encounters an exception it will stop indexing.
> I had to make two fixes to TikaEntityProcessor to work around this problem.
>
> From the Solr SVN trunk edit the file:
>
>
> ~/src/solr-svn/trunk/solr/contrib/dataimporthandler/src/extras/main/java/org/apache/solr/handler/dataimport/TikaEntityProcessor.jar
>
> First of all if a file is not found on the disk we want to continue
> indexing. At the top of nextRow() add
>
> File f = new File (context.getResolvedEntityAttribute(URL));
> if (! f.exists()) {
>  return null;
> }
>
> Secondly if the document parser throws an error, for example certain PDF
> revisions can cause the PDFBox parser to barf, we will trap the exception
> and continue:
>
> try {
>  tikaParser.parse(is, contentHandler, metadata , new ParseContext());
> } catch (Exception e) {
>  return null;
> } finally {
>  IOUtils.closeQuietly(is);
> }
>
> We will also close IOUtils in the finally section which is not done in the
> original code. Build and deploy the extras.jar in the solr-instance/lib
> directory.
>
> see also: http://www.abcseo.com/tech/search/solr-and-liferay-integration
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Resume-Solr-indexing-CSV-after-exception-tp878801p888143.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: Resume Solr indexing CSV after exception

[Lance Norskog <go...@gmail.com>]

No. They are talking about a new feature in the DataImportHandler for
reading CSV files.

On Tue, Aug 31, 2010 at 1:55 PM, romiawasthy <ro...@gmail.com> wrote:
>
> How do I use this feature, is there some parameter that I need to specify in
> the update request?
>
> curl
> http://localhost:8983/solr/update/csv?stream.file=exampledocs/books.csv&stream.contentType=text/plain;charset=utf-8
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Resume-Solr-indexing-CSV-after-exception-tp878801p1396875.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>



-- 
Lance Norskog
goksron@gmail.com

Re: Resume Solr indexing CSV after exception

[romiawasthy <ro...@gmail.com>]

How do I use this feature, is there some parameter that I need to specify in
the update request?

curl
http://localhost:8983/solr/update/csv?stream.file=exampledocs/books.csv&stream.contentType=text/plain;charset=utf-8

-- 
View this message in context: http://lucene.472066.n3.nabble.com/Resume-Solr-indexing-CSV-after-exception-tp878801p1396875.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Resume Solr indexing CSV after exception

[David George <da...@gmail.com>]

cool I will try that.
-- 
View this message in context: http://lucene.472066.n3.nabble.com/Resume-Solr-indexing-CSV-after-exception-tp878801p888605.html
Sent from the Solr - User mailing list archive at Nabble.com.