You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Vijaya Narayana Reddy Bhoomi Reddy <vi...@whishworks.com> on 2015/04/14 16:57:23 UTC
Indexing PDF and MS Office files
Hi,
I am trying to index PDF and Microsoft Office files (.doc, .docx, .ppt,
.pptx, .xlx, and .xlx) files into Solr. I am facing the following issues.
Request to please let me know what is going wrong with the indexing
process.
I am using solr 4.10.2 and using the default example server configuration
that comes with Solr distribution.
PDF Files - Indexing as such works fine, but when I query using *.* in the
Solr Query console, metadata information is displayed properly. However,
the PDF content field is empty. This is happening for all PDF files I have
tried. I have tried with some proprietary files, PDF eBooks etc. Whatever
be the PDF file, content is not being displayed.
MS Office files - For some office files, everything works perfect and the
extracted content is visible in the query console. However, for others, I
see the below error message during the indexing process.
*Exception in thread "main"
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
org.apache.tika.exception.TikaException: Unexpected RuntimeException from
org.apache.tika.parser.microsoft.OfficeParser*
I am using SolrJ to index the documents and below is the code snippet
related to indexing. Please let me know where the issue is occurring.
static String solrServerURL = "
http://localhost:8983/solr";
static SolrServer solrServer = new HttpSolrServer(solrServerURL);
static ContentStreamUpdateRequest indexingReq = new
ContentStreamUpdateRequest("/update/extract");
indexingReq.addFile(file, fileType);
indexingReq.setParam("literal.id", literalId);
indexingReq.setParam("uprefix", "attr_");
indexingReq.setParam("fmap.content", "content");
indexingReq.setParam("literal.fileurl", fileURL);
indexingReq.setAction(AbstractUpdateRequest.ACTION.COMMIT, true, true);
solrServer.request(indexingReq);
Thanks & Regards
Vijay
--
The contents of this e-mail are confidential and for the exclusive use of
the intended recipient. If you receive this e-mail in error please delete
it from your system immediately and notify us either by e-mail or
telephone. You should not copy, forward or otherwise disclose the content
of the e-mail. The views expressed in this communication may not
necessarily be the view held by WHISHWORKS.
Re: Indexing PDF and MS Office files
Posted by Andrea Gazzarini <a....@gmail.com>.
Hi Vijay,
Please paste an extract of your schema, where the "content" field (the
field where the PDF text shoudl be) and its type are declared.
For the other issue, please paste the whole stacktrace because
org.apache.tika.parser.microsoft.OfficeParser*
says nothing. The complete stacktrace (or at least another three / four
lines) should contain some other detail.
Best,
Andrea
On 04/14/2015 04:57 PM, Vijaya Narayana Reddy Bhoomi Reddy wrote:
> Hi,
>
> I am trying to index PDF and Microsoft Office files (.doc, .docx, .ppt,
> .pptx, .xlx, and .xlx) files into Solr. I am facing the following issues.
> Request to please let me know what is going wrong with the indexing
> process.
>
> I am using solr 4.10.2 and using the default example server configuration
> that comes with Solr distribution.
>
> PDF Files - Indexing as such works fine, but when I query using *.* in the
> Solr Query console, metadata information is displayed properly. However,
> the PDF content field is empty. This is happening for all PDF files I have
> tried. I have tried with some proprietary files, PDF eBooks etc. Whatever
> be the PDF file, content is not being displayed.
>
> MS Office files - For some office files, everything works perfect and the
> extracted content is visible in the query console. However, for others, I
> see the below error message during the indexing process.
>
> *Exception in thread "main"
> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
> org.apache.tika.exception.TikaException: Unexpected RuntimeException from
> org.apache.tika.parser.microsoft.OfficeParser*
>
>
> I am using SolrJ to index the documents and below is the code snippet
> related to indexing. Please let me know where the issue is occurring.
>
> static String solrServerURL = "
> http://localhost:8983/solr";
> static SolrServer solrServer = new HttpSolrServer(solrServerURL);
> static ContentStreamUpdateRequest indexingReq = new
>
> ContentStreamUpdateRequest("/update/extract");
>
> indexingReq.addFile(file, fileType);
> indexingReq.setParam("literal.id", literalId);
> indexingReq.setParam("uprefix", "attr_");
> indexingReq.setParam("fmap.content", "content");
> indexingReq.setParam("literal.fileurl", fileURL);
> indexingReq.setAction(AbstractUpdateRequest.ACTION.COMMIT, true, true);
> solrServer.request(indexingReq);
>
> Thanks & Regards
> Vijay
>
Re: Indexing PDF and MS Office files
Posted by Andrea Gazzarini <a....@gmail.com>.
Hi,
solrconfig.xml (especially if you didn't touch it) should be good. What
about the schema? Are you using the one that comes with the download
bundle, too?
I don't see the stacktrace..did you forget to paste it?
Best,
Andrea
On 04/14/2015 06:06 PM, Vijaya Narayana Reddy Bhoomi Reddy wrote:
> Hi,
>
> Here are the solr-config xml and the error log from Solr logs for your
> reference. As mentioned earlier, I didnt make any changes to the
> solr-config.xml as I am using the xml file out of the box one that
> came with the default installation.
>
> Please let me know your thoughts on why these issues are occuring.
>
> Thanks & Regards
> Vijay
>
>
>
>
> *Vijay Bhoomireddy*, Big Data Architect
>
> 1000 Great West Road, Brentford, London, TW8 9DW
>
> *T:+44 20 3475 7980*
> *M:**+44 7481 298 360*
> *W: *ww <http://www.whishworks.com/>w.whishworks.com
> <http://www.whishworks.com/>
>
> <https://www.linkedin.com/company/whishworks><http://www.whishworks.com/blog/><https://twitter.com/WHISHWORKS><https://www.facebook.com/whishworksit>
>
>
> On 14 April 2015 at 15:57, Vijaya Narayana Reddy Bhoomi Reddy
> <vijaya.bhoomireddy@whishworks.com
> <ma...@whishworks.com>> wrote:
>
> Hi,
>
> I am trying to index PDF and Microsoft Office files (.doc, .docx,
> .ppt, .pptx, .xlx, and .xlx) files into Solr. I am facing the
> following issues. Request to please let me know what is going
> wrong with the indexing process.
>
> I am using solr 4.10.2 and using the default example server
> configuration that comes with Solr distribution.
>
> PDF Files - Indexing as such works fine, but when I query using
> *.* in the Solr Query console, metadata information is displayed
> properly. However, the PDF content field is empty. This is
> happening for all PDF files I have tried. I have tried with some
> proprietary files, PDF eBooks etc. Whatever be the PDF file,
> content is not being displayed.
>
> MS Office files - For some office files, everything works perfect
> and the extracted content is visible in the query console.
> However, for others, I see the below error message during the
> indexing process.
>
> *Exception in thread "main"
> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
> org.apache.tika.exception.TikaException: Unexpected
> RuntimeException from org.apache.tika.parser.microsoft.OfficeParser*
> *
> *
>
> I am using SolrJ to index the documents and below is the code
> snippet related to indexing. Please let me know where the issue is
> occurring.
>
> static String solrServerURL =
> "http://localhost:8983/solr";
> static SolrServer solrServer = new HttpSolrServer(solrServerURL);
> static ContentStreamUpdateRequest
> indexingReq = new ContentStreamUpdateRequest("/update/extract");
>
> indexingReq.addFile(file, fileType);
> indexingReq.setParam("literal.id <http://literal.id>", literalId);
> indexingReq.setParam("uprefix", "attr_");
> indexingReq.setParam("fmap.content", "content");
> indexingReq.setParam("literal.fileurl", fileURL);
> indexingReq.setAction(AbstractUpdateRequest.ACTION.COMMIT, true,
> true);
> solrServer.request(indexingReq);
>
> Thanks & Regards
> Vijay
>
>
>
>
> The contents of this e-mail are confidential and for the exclusive use
> of the intended recipient. If you receive this e-mail in error please
> delete it from your system immediately and notify us either by e-mail
> or telephone. You should not copy, forward or otherwise disclose the
> content of the e-mail. The views expressed in this communication may
> not necessarily be the view held by WHISHWORKS.
Re: Indexing PDF and MS Office files
Posted by Andrea Gazzarini <a....@gmail.com>.
It seems something like https://issues.apache.org/jira/browse/TIKA-1251.
I see you're using Solr 4.10.2 which uses Tika 1.5 and that issue seems
to be fixed in Tika 1.6.
I agree with Erik: you should try with another version of Tika.
Best,
Andrea
On 04/14/2015 06:44 PM, Vijaya Narayana Reddy Bhoomi Reddy wrote:
> Andrea,
>
> Yes, I am using the stock schema.xml that comes with the example server of
> Solr-4.10.2 Hence not sure why the PDF content is not getting extracted and
> put into the content field in the index.
>
> Please find the log information for the Parsing error below.
>
>
> org.apache.solr.common.SolrException; org.apache.solr.common.SolrException:
> org.apache.tika.exception.TikaException: Unexpected RuntimeException from
> org.apache.tika.parser.microsoft.OfficeParser@138b0c5
> at
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:225)
> at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> at
> org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(RequestHandlers.java:246)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1967)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at org.eclipse.jetty.server.Server.handle(Server.java:368)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
> at
> org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
> at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:953)
> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
> at
> org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
> at
> org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.tika.exception.TikaException: Unexpected
> RuntimeException from org.apache.tika.parser.microsoft.OfficeParser@138b0c5
> at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:244)
> at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242)
> at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:120)
> at
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:219)
> ... 32 more
> Caused by: java.lang.IllegalArgumentException: This paragraph is not the
> first one in the table
> at org.apache.poi.hwpf.usermodel.Range.getTable(Range.java:932)
> at
> org.apache.tika.parser.microsoft.WordExtractor.handleParagraph(WordExtractor.java:188)
> at
> org.apache.tika.parser.microsoft.WordExtractor.handleHeaderFooter(WordExtractor.java:172)
> at
> org.apache.tika.parser.microsoft.WordExtractor.parse(WordExtractor.java:98)
> at
> org.apache.tika.parser.microsoft.OfficeParser.parse(OfficeParser.java:199)
> at
> org.apache.tika.parser.microsoft.OfficeParser.parse(OfficeParser.java:167)
> at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242)
> ... 35 more
>
> ERROR - 2015-04-14 14:51:21.151; org.apache.solr.common.SolrException;
> null:org.apache.solr.common.SolrException:
> org.apache.tika.exception.TikaException: Unexpected RuntimeException from
> org.apache.tika.parser.microsoft.OfficeParser@138b0c5
> at
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:225)
> at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> at
> org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(RequestHandlers.java:246)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1967)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at org.eclipse.jetty.server.Server.handle(Server.java:368)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
> at
> org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
> at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:953)
> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
> at
> org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
> at
> org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.tika.exception.TikaException: Unexpected
> RuntimeException from org.apache.tika.parser.microsoft.OfficeParser@138b0c5
> at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:244)
> at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242)
> at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:120)
> at
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:219)
> ... 32 more
> Caused by: java.lang.IllegalArgumentException: This paragraph is not the
> first one in the table
> at org.apache.poi.hwpf.usermodel.Range.getTable(Range.java:932)
> at
> org.apache.tika.parser.microsoft.WordExtractor.handleParagraph(WordExtractor.java:188)
> at
> org.apache.tika.parser.microsoft.WordExtractor.handleHeaderFooter(WordExtractor.java:172)
> at
> org.apache.tika.parser.microsoft.WordExtractor.parse(WordExtractor.java:98)
> at
> org.apache.tika.parser.microsoft.OfficeParser.parse(OfficeParser.java:199)
> at
> org.apache.tika.parser.microsoft.OfficeParser.parse(OfficeParser.java:167)
> at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242)
> ... 35 more
>
>
> Thanks & Regards
> Vijay
>
>
> On 14 April 2015 at 17:06, Vijaya Narayana Reddy Bhoomi Reddy <
> vijaya.bhoomireddy@whishworks.com> wrote:
>
>> Hi,
>>
>> Here are the solr-config xml and the error log from Solr logs for your
>> reference. As mentioned earlier, I didnt make any changes to the
>> solr-config.xml as I am using the xml file out of the box one that came
>> with the default installation.
>>
>> Please let me know your thoughts on why these issues are occuring.
>>
>> Thanks & Regards
>> Vijay
>>
>>
>> *Vijay Bhoomireddy*, Big Data Architect
>>
>> 1000 Great West Road, Brentford, London, TW8 9DW
>> *T: +44 20 3475 7980*
>> *M: **+44 7481 298 360*
>> *W: *ww <http://www.whishworks.com/>w.whishworks.com
>> <http://www.whishworks.com/>
>>
>> <https://www.linkedin.com/company/whishworks>
>> <http://www.whishworks.com/blog/> <https://twitter.com/WHISHWORKS>
>> <https://www.facebook.com/whishworksit>
>>
>> On 14 April 2015 at 15:57, Vijaya Narayana Reddy Bhoomi Reddy <
>> vijaya.bhoomireddy@whishworks.com> wrote:
>>
>>> Hi,
>>>
>>> I am trying to index PDF and Microsoft Office files (.doc, .docx, .ppt,
>>> .pptx, .xlx, and .xlx) files into Solr. I am facing the following issues.
>>> Request to please let me know what is going wrong with the indexing
>>> process.
>>>
>>> I am using solr 4.10.2 and using the default example server configuration
>>> that comes with Solr distribution.
>>>
>>> PDF Files - Indexing as such works fine, but when I query using *.* in
>>> the Solr Query console, metadata information is displayed properly.
>>> However, the PDF content field is empty. This is happening for all PDF
>>> files I have tried. I have tried with some proprietary files, PDF eBooks
>>> etc. Whatever be the PDF file, content is not being displayed.
>>>
>>> MS Office files - For some office files, everything works perfect and
>>> the extracted content is visible in the query console. However, for others,
>>> I see the below error message during the indexing process.
>>>
>>> *Exception in thread "main"
>>> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
>>> org.apache.tika.exception.TikaException: Unexpected RuntimeException from
>>> org.apache.tika.parser.microsoft.OfficeParser*
>>>
>>>
>>> I am using SolrJ to index the documents and below is the code snippet
>>> related to indexing. Please let me know where the issue is occurring.
>>>
>>> static String solrServerURL = "
>>> http://localhost:8983/solr";
>>> static SolrServer solrServer = new HttpSolrServer(solrServerURL);
>>> static ContentStreamUpdateRequest indexingReq =
>>> new
>>> ContentStreamUpdateRequest("/update/extract");
>>>
>>> indexingReq.addFile(file, fileType);
>>> indexingReq.setParam("literal.id", literalId);
>>> indexingReq.setParam("uprefix", "attr_");
>>> indexingReq.setParam("fmap.content", "content");
>>> indexingReq.setParam("literal.fileurl", fileURL);
>>> indexingReq.setAction(AbstractUpdateRequest.ACTION.COMMIT, true, true);
>>> solrServer.request(indexingReq);
>>>
>>> Thanks & Regards
>>> Vijay
>>>
>>>
>>>
Re: Indexing PDF and MS Office files
Posted by Erick Erickson <er...@gmail.com>.
looks like this is just a file that Tika can't handle, based on this line:
bq: org.apache.tika.exception.TikaException: Unexpected
RuntimeException from org.apache.tika.parser.microsoft.OfficeParser
You might be able to get some joy from parsing this from Java and see
if a more recent Tika would fix it. Here's some sample code:
http://lucidworks.com/blog/indexing-with-solrj/
Best,
Erick
On Tue, Apr 14, 2015 at 9:44 AM, Vijaya Narayana Reddy Bhoomi Reddy
<vi...@whishworks.com> wrote:
> Andrea,
>
> Yes, I am using the stock schema.xml that comes with the example server of
> Solr-4.10.2 Hence not sure why the PDF content is not getting extracted and
> put into the content field in the index.
>
> Please find the log information for the Parsing error below.
>
>
> org.apache.solr.common.SolrException; org.apache.solr.common.SolrException:
> org.apache.tika.exception.TikaException: Unexpected RuntimeException from
> org.apache.tika.parser.microsoft.OfficeParser@138b0c5
> at
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:225)
> at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> at
> org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(RequestHandlers.java:246)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1967)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at org.eclipse.jetty.server.Server.handle(Server.java:368)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
> at
> org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
> at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:953)
> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
> at
> org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
> at
> org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.tika.exception.TikaException: Unexpected
> RuntimeException from org.apache.tika.parser.microsoft.OfficeParser@138b0c5
> at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:244)
> at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242)
> at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:120)
> at
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:219)
> ... 32 more
> Caused by: java.lang.IllegalArgumentException: This paragraph is not the
> first one in the table
> at org.apache.poi.hwpf.usermodel.Range.getTable(Range.java:932)
> at
> org.apache.tika.parser.microsoft.WordExtractor.handleParagraph(WordExtractor.java:188)
> at
> org.apache.tika.parser.microsoft.WordExtractor.handleHeaderFooter(WordExtractor.java:172)
> at
> org.apache.tika.parser.microsoft.WordExtractor.parse(WordExtractor.java:98)
> at
> org.apache.tika.parser.microsoft.OfficeParser.parse(OfficeParser.java:199)
> at
> org.apache.tika.parser.microsoft.OfficeParser.parse(OfficeParser.java:167)
> at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242)
> ... 35 more
>
> ERROR - 2015-04-14 14:51:21.151; org.apache.solr.common.SolrException;
> null:org.apache.solr.common.SolrException:
> org.apache.tika.exception.TikaException: Unexpected RuntimeException from
> org.apache.tika.parser.microsoft.OfficeParser@138b0c5
> at
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:225)
> at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> at
> org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(RequestHandlers.java:246)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1967)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at org.eclipse.jetty.server.Server.handle(Server.java:368)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
> at
> org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
> at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:953)
> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
> at
> org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
> at
> org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.tika.exception.TikaException: Unexpected
> RuntimeException from org.apache.tika.parser.microsoft.OfficeParser@138b0c5
> at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:244)
> at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242)
> at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:120)
> at
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:219)
> ... 32 more
> Caused by: java.lang.IllegalArgumentException: This paragraph is not the
> first one in the table
> at org.apache.poi.hwpf.usermodel.Range.getTable(Range.java:932)
> at
> org.apache.tika.parser.microsoft.WordExtractor.handleParagraph(WordExtractor.java:188)
> at
> org.apache.tika.parser.microsoft.WordExtractor.handleHeaderFooter(WordExtractor.java:172)
> at
> org.apache.tika.parser.microsoft.WordExtractor.parse(WordExtractor.java:98)
> at
> org.apache.tika.parser.microsoft.OfficeParser.parse(OfficeParser.java:199)
> at
> org.apache.tika.parser.microsoft.OfficeParser.parse(OfficeParser.java:167)
> at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242)
> ... 35 more
>
>
> Thanks & Regards
> Vijay
>
>
> On 14 April 2015 at 17:06, Vijaya Narayana Reddy Bhoomi Reddy <
> vijaya.bhoomireddy@whishworks.com> wrote:
>
>> Hi,
>>
>> Here are the solr-config xml and the error log from Solr logs for your
>> reference. As mentioned earlier, I didnt make any changes to the
>> solr-config.xml as I am using the xml file out of the box one that came
>> with the default installation.
>>
>> Please let me know your thoughts on why these issues are occuring.
>>
>> Thanks & Regards
>> Vijay
>>
>>
>> *Vijay Bhoomireddy*, Big Data Architect
>>
>> 1000 Great West Road, Brentford, London, TW8 9DW
>> *T: +44 20 3475 7980*
>> *M: **+44 7481 298 360*
>> *W: *ww <http://www.whishworks.com/>w.whishworks.com
>> <http://www.whishworks.com/>
>>
>> <https://www.linkedin.com/company/whishworks>
>> <http://www.whishworks.com/blog/> <https://twitter.com/WHISHWORKS>
>> <https://www.facebook.com/whishworksit>
>>
>> On 14 April 2015 at 15:57, Vijaya Narayana Reddy Bhoomi Reddy <
>> vijaya.bhoomireddy@whishworks.com> wrote:
>>
>>> Hi,
>>>
>>> I am trying to index PDF and Microsoft Office files (.doc, .docx, .ppt,
>>> .pptx, .xlx, and .xlx) files into Solr. I am facing the following issues.
>>> Request to please let me know what is going wrong with the indexing
>>> process.
>>>
>>> I am using solr 4.10.2 and using the default example server configuration
>>> that comes with Solr distribution.
>>>
>>> PDF Files - Indexing as such works fine, but when I query using *.* in
>>> the Solr Query console, metadata information is displayed properly.
>>> However, the PDF content field is empty. This is happening for all PDF
>>> files I have tried. I have tried with some proprietary files, PDF eBooks
>>> etc. Whatever be the PDF file, content is not being displayed.
>>>
>>> MS Office files - For some office files, everything works perfect and
>>> the extracted content is visible in the query console. However, for others,
>>> I see the below error message during the indexing process.
>>>
>>> *Exception in thread "main"
>>> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
>>> org.apache.tika.exception.TikaException: Unexpected RuntimeException from
>>> org.apache.tika.parser.microsoft.OfficeParser*
>>>
>>>
>>> I am using SolrJ to index the documents and below is the code snippet
>>> related to indexing. Please let me know where the issue is occurring.
>>>
>>> static String solrServerURL = "
>>> http://localhost:8983/solr";
>>> static SolrServer solrServer = new HttpSolrServer(solrServerURL);
>>> static ContentStreamUpdateRequest indexingReq =
>>> new
>>> ContentStreamUpdateRequest("/update/extract");
>>>
>>> indexingReq.addFile(file, fileType);
>>> indexingReq.setParam("literal.id", literalId);
>>> indexingReq.setParam("uprefix", "attr_");
>>> indexingReq.setParam("fmap.content", "content");
>>> indexingReq.setParam("literal.fileurl", fileURL);
>>> indexingReq.setAction(AbstractUpdateRequest.ACTION.COMMIT, true, true);
>>> solrServer.request(indexingReq);
>>>
>>> Thanks & Regards
>>> Vijay
>>>
>>>
>>>
>>
>
> --
> The contents of this e-mail are confidential and for the exclusive use of
> the intended recipient. If you receive this e-mail in error please delete
> it from your system immediately and notify us either by e-mail or
> telephone. You should not copy, forward or otherwise disclose the content
> of the e-mail. The views expressed in this communication may not
> necessarily be the view held by WHISHWORKS.
Re: Indexing PDF and MS Office files
Posted by Vijaya Narayana Reddy Bhoomi Reddy <vi...@whishworks.com>.
Andrea,
Yes, I am using the stock schema.xml that comes with the example server of
Solr-4.10.2 Hence not sure why the PDF content is not getting extracted and
put into the content field in the index.
Please find the log information for the Parsing error below.
org.apache.solr.common.SolrException; org.apache.solr.common.SolrException:
org.apache.tika.exception.TikaException: Unexpected RuntimeException from
org.apache.tika.parser.microsoft.OfficeParser@138b0c5
at
org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:225)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at
org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(RequestHandlers.java:246)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1967)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368)
at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at
org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:953)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
at
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.tika.exception.TikaException: Unexpected
RuntimeException from org.apache.tika.parser.microsoft.OfficeParser@138b0c5
at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:244)
at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242)
at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:120)
at
org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:219)
... 32 more
Caused by: java.lang.IllegalArgumentException: This paragraph is not the
first one in the table
at org.apache.poi.hwpf.usermodel.Range.getTable(Range.java:932)
at
org.apache.tika.parser.microsoft.WordExtractor.handleParagraph(WordExtractor.java:188)
at
org.apache.tika.parser.microsoft.WordExtractor.handleHeaderFooter(WordExtractor.java:172)
at
org.apache.tika.parser.microsoft.WordExtractor.parse(WordExtractor.java:98)
at
org.apache.tika.parser.microsoft.OfficeParser.parse(OfficeParser.java:199)
at
org.apache.tika.parser.microsoft.OfficeParser.parse(OfficeParser.java:167)
at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242)
... 35 more
ERROR - 2015-04-14 14:51:21.151; org.apache.solr.common.SolrException;
null:org.apache.solr.common.SolrException:
org.apache.tika.exception.TikaException: Unexpected RuntimeException from
org.apache.tika.parser.microsoft.OfficeParser@138b0c5
at
org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:225)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at
org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(RequestHandlers.java:246)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1967)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368)
at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at
org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:953)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
at
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.tika.exception.TikaException: Unexpected
RuntimeException from org.apache.tika.parser.microsoft.OfficeParser@138b0c5
at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:244)
at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242)
at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:120)
at
org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:219)
... 32 more
Caused by: java.lang.IllegalArgumentException: This paragraph is not the
first one in the table
at org.apache.poi.hwpf.usermodel.Range.getTable(Range.java:932)
at
org.apache.tika.parser.microsoft.WordExtractor.handleParagraph(WordExtractor.java:188)
at
org.apache.tika.parser.microsoft.WordExtractor.handleHeaderFooter(WordExtractor.java:172)
at
org.apache.tika.parser.microsoft.WordExtractor.parse(WordExtractor.java:98)
at
org.apache.tika.parser.microsoft.OfficeParser.parse(OfficeParser.java:199)
at
org.apache.tika.parser.microsoft.OfficeParser.parse(OfficeParser.java:167)
at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242)
... 35 more
Thanks & Regards
Vijay
On 14 April 2015 at 17:06, Vijaya Narayana Reddy Bhoomi Reddy <
vijaya.bhoomireddy@whishworks.com> wrote:
> Hi,
>
> Here are the solr-config xml and the error log from Solr logs for your
> reference. As mentioned earlier, I didnt make any changes to the
> solr-config.xml as I am using the xml file out of the box one that came
> with the default installation.
>
> Please let me know your thoughts on why these issues are occuring.
>
> Thanks & Regards
> Vijay
>
>
> *Vijay Bhoomireddy*, Big Data Architect
>
> 1000 Great West Road, Brentford, London, TW8 9DW
> *T: +44 20 3475 7980*
> *M: **+44 7481 298 360*
> *W: *ww <http://www.whishworks.com/>w.whishworks.com
> <http://www.whishworks.com/>
>
> <https://www.linkedin.com/company/whishworks>
> <http://www.whishworks.com/blog/> <https://twitter.com/WHISHWORKS>
> <https://www.facebook.com/whishworksit>
>
> On 14 April 2015 at 15:57, Vijaya Narayana Reddy Bhoomi Reddy <
> vijaya.bhoomireddy@whishworks.com> wrote:
>
>> Hi,
>>
>> I am trying to index PDF and Microsoft Office files (.doc, .docx, .ppt,
>> .pptx, .xlx, and .xlx) files into Solr. I am facing the following issues.
>> Request to please let me know what is going wrong with the indexing
>> process.
>>
>> I am using solr 4.10.2 and using the default example server configuration
>> that comes with Solr distribution.
>>
>> PDF Files - Indexing as such works fine, but when I query using *.* in
>> the Solr Query console, metadata information is displayed properly.
>> However, the PDF content field is empty. This is happening for all PDF
>> files I have tried. I have tried with some proprietary files, PDF eBooks
>> etc. Whatever be the PDF file, content is not being displayed.
>>
>> MS Office files - For some office files, everything works perfect and
>> the extracted content is visible in the query console. However, for others,
>> I see the below error message during the indexing process.
>>
>> *Exception in thread "main"
>> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
>> org.apache.tika.exception.TikaException: Unexpected RuntimeException from
>> org.apache.tika.parser.microsoft.OfficeParser*
>>
>>
>> I am using SolrJ to index the documents and below is the code snippet
>> related to indexing. Please let me know where the issue is occurring.
>>
>> static String solrServerURL = "
>> http://localhost:8983/solr";
>> static SolrServer solrServer = new HttpSolrServer(solrServerURL);
>> static ContentStreamUpdateRequest indexingReq =
>> new
>> ContentStreamUpdateRequest("/update/extract");
>>
>> indexingReq.addFile(file, fileType);
>> indexingReq.setParam("literal.id", literalId);
>> indexingReq.setParam("uprefix", "attr_");
>> indexingReq.setParam("fmap.content", "content");
>> indexingReq.setParam("literal.fileurl", fileURL);
>> indexingReq.setAction(AbstractUpdateRequest.ACTION.COMMIT, true, true);
>> solrServer.request(indexingReq);
>>
>> Thanks & Regards
>> Vijay
>>
>>
>>
>
--
The contents of this e-mail are confidential and for the exclusive use of
the intended recipient. If you receive this e-mail in error please delete
it from your system immediately and notify us either by e-mail or
telephone. You should not copy, forward or otherwise disclose the content
of the e-mail. The views expressed in this communication may not
necessarily be the view held by WHISHWORKS.
Re: Indexing PDF and MS Office files
Posted by Vijaya Narayana Reddy Bhoomi Reddy <vi...@whishworks.com>.
Hi,
Here are the solr-config xml and the error log from Solr logs for your
reference. As mentioned earlier, I didnt make any changes to the
solr-config.xml as I am using the xml file out of the box one that came
with the default installation.
Please let me know your thoughts on why these issues are occuring.
Thanks & Regards
Vijay
*Vijay Bhoomireddy*, Big Data Architect
1000 Great West Road, Brentford, London, TW8 9DW
*T: +44 20 3475 7980*
*M: **+44 7481 298 360*
*W: *ww <http://www.whishworks.com/>w.whishworks.com
<http://www.whishworks.com/>
<https://www.linkedin.com/company/whishworks>
<http://www.whishworks.com/blog/> <https://twitter.com/WHISHWORKS>
<https://www.facebook.com/whishworksit>
On 14 April 2015 at 15:57, Vijaya Narayana Reddy Bhoomi Reddy <
vijaya.bhoomireddy@whishworks.com> wrote:
> Hi,
>
> I am trying to index PDF and Microsoft Office files (.doc, .docx, .ppt,
> .pptx, .xlx, and .xlx) files into Solr. I am facing the following issues.
> Request to please let me know what is going wrong with the indexing
> process.
>
> I am using solr 4.10.2 and using the default example server configuration
> that comes with Solr distribution.
>
> PDF Files - Indexing as such works fine, but when I query using *.* in the
> Solr Query console, metadata information is displayed properly. However,
> the PDF content field is empty. This is happening for all PDF files I have
> tried. I have tried with some proprietary files, PDF eBooks etc. Whatever
> be the PDF file, content is not being displayed.
>
> MS Office files - For some office files, everything works perfect and the
> extracted content is visible in the query console. However, for others, I
> see the below error message during the indexing process.
>
> *Exception in thread "main"
> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
> org.apache.tika.exception.TikaException: Unexpected RuntimeException from
> org.apache.tika.parser.microsoft.OfficeParser*
>
>
> I am using SolrJ to index the documents and below is the code snippet
> related to indexing. Please let me know where the issue is occurring.
>
> static String solrServerURL = "
> http://localhost:8983/solr";
> static SolrServer solrServer = new HttpSolrServer(solrServerURL);
> static ContentStreamUpdateRequest indexingReq =
> new
> ContentStreamUpdateRequest("/update/extract");
>
> indexingReq.addFile(file, fileType);
> indexingReq.setParam("literal.id", literalId);
> indexingReq.setParam("uprefix", "attr_");
> indexingReq.setParam("fmap.content", "content");
> indexingReq.setParam("literal.fileurl", fileURL);
> indexingReq.setAction(AbstractUpdateRequest.ACTION.COMMIT, true, true);
> solrServer.request(indexingReq);
>
> Thanks & Regards
> Vijay
>
>
>
--
The contents of this e-mail are confidential and for the exclusive use of
the intended recipient. If you receive this e-mail in error please delete
it from your system immediately and notify us either by e-mail or
telephone. You should not copy, forward or otherwise disclose the content
of the e-mail. The views expressed in this communication may not
necessarily be the view held by WHISHWORKS.
Re: Indexing PDF and MS Office files
Posted by Charlie Hull <ch...@flax.co.uk>.
On 16/04/2015 12:53, Siegfried Goeschl wrote:
> Hi Vijay,
>
> I know the this road too well :-)
>
> For PDF you can fallback to other tools for text extraction
>
> * ps2ascii.ps
> * XPDF's pdftotext CLI utility (more comfortable than Ghostscript)
> * some other tools exists as well (pdflib)
Here's some file extractors we built a while ago:
https://github.com/flaxsearch/flaxcode/tree/master/flax_filters
You might find them useful: they use a number of external programs
including pdf2text and headless Open Office.
Cheers
Charlie
>
> If you start command line tools from your JVM please have a look at
> commons-exec :-)
>
> Cheers,
>
> Siegfried Goeschl
>
> PS: one more thing - please, tell your management that you will never
> ever successfully all real-world PDFs and cater for that fact in your
> requirements :-)
>
> On 16.04.15 13:10, Vijaya Narayana Reddy Bhoomi Reddy wrote:
>> Erick,
>>
>> I tried indexing both ways - SolrJ / Tika's AutoParser and as well as
>> SolrCell's ExtractRequestHandler. Majority of the PDF and Word documents
>> are getting parsed properly and indexed into Solr. However, a minority of
>> them keep failing wither PDFParser or OfficeParser error.
>>
>> Not sure if this behaviour can be modified so that all the documents
>> can be
>> indexed. The business requirement we have is to index all the documents.
>> However, if a small percentage of them fails, not sure what other ways
>> exist to index them.
>>
>> Any help please?
>>
>>
>> Thanks & Regards
>> Vijay
>>
>>
>>
>> On 15 April 2015 at 15:20, Erick Erickson <er...@gmail.com>
>> wrote:
>>
>>> There's quite a discussion here:
>>> https://issues.apache.org/jira/browse/SOLR-7137
>>>
>>> But, I personally am not a huge fan of pushing all the work on to
>>> Solr, in
>>> a
>>> production environment the Solr server is responsible for indexing,
>>> parsing the
>>> docs through Tika, perhaps searching etc. This doesn't scale all that
>>> well.
>>>
>>> So an alternative is to use SolrJ with Tika, which is totally
>>> independent
>>> of
>>> what version of Tika is on the Solr server. Here's an example.
>>>
>>> http://lucidworks.com/blog/indexing-with-solrj/
>>>
>>> Best,
>>> Erick
>>>
>>> On Wed, Apr 15, 2015 at 4:46 AM, Vijaya Narayana Reddy Bhoomi Reddy
>>> <vi...@whishworks.com> wrote:
>>>> Thanks everyone for the responses. Now I am able to index PDF documents
>>>> successfully. I have implemented manual extraction using Tika's
>>> AutoParser
>>>> and PDF functionality is working fine. However, the error with some MS
>>>> office word documents still persist.
>>>>
>>>> The error message is "java.lang.IllegalArgumentException: This
>>>> paragraph
>>> is
>>>> not the first one in the table" which will eventually result in
>>> "Unexpected
>>>> RuntimeException from org.apache.tika.parser.microsoft.OfficeParser"
>>>>
>>>> Upon some reading, it looks like its a bug with Tika 1.5 and seems to
>>> have
>>>> been fixed with Tika 1.6 (
>>> https://issues.apache.org/jira/browse/TIKA-1251 ).
>>>> I am new to Solr / Tika and hence wondering whether I can change the
>>>> Tika
>>>> library alone to v1.6 without impacting any of the libraries within
>>>> Solr
>>>> 4.10.2? Please let me know your response and how to get away with this
>>>> issue.
>>>>
>>>> Many thanks in advance.
>>>>
>>>> Thanks & Regards
>>>> Vijay
>>>>
>>>>
>>>> On 15 April 2015 at 05:14, Shyam R <sh...@gmail.com> wrote:
>>>>
>>>>> Vijay,
>>>>>
>>>>> You could try different excel files with different formats to rule out
>>> the
>>>>> issue is with TIKA version being used.
>>>>>
>>>>> Thanks
>>>>> Murthy
>>>>>
>>>>> On Wed, Apr 15, 2015 at 9:35 AM, Terry Rhodes <tr...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Perhaps the PDF is protected and the content can not be extracted?
>>>>>>
>>>>>> i have an unverified suspicion that the tika shipped with solr 4.10.2
>>> may
>>>>>> not support some/all office 2013 document formats.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 4/14/2015 8:18 PM, Jack Krupansky wrote:
>>>>>>
>>>>>>> Try doing a manual extraction request directly to Solr (not via
>>> SolrJ)
>>>>> and
>>>>>>> use the extractOnly option to see if the content is actually
>>> extracted.
>>>>>>>
>>>>>>> See:
>>>>>>> https://cwiki.apache.org/confluence/display/solr/
>>>>>>> Uploading+Data+with+Solr+Cell+using+Apache+Tika
>>>>>>>
>>>>>>> Also, some PDF files actually have the content as a bitmap image, so
>>> no
>>>>>>> text is extracted.
>>>>>>>
>>>>>>>
>>>>>>> -- Jack Krupansky
>>>>>>>
>>>>>>> On Tue, Apr 14, 2015 at 10:57 AM, Vijaya Narayana Reddy Bhoomi Reddy
>>> <
>>>>>>> vijaya.bhoomireddy@whishworks.com> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I am trying to index PDF and Microsoft Office files (.doc, .docx,
>>> .ppt,
>>>>>>>> .pptx, .xlx, and .xlx) files into Solr. I am facing the following
>>>>> issues.
>>>>>>>> Request to please let me know what is going wrong with the indexing
>>>>>>>> process.
>>>>>>>>
>>>>>>>> I am using solr 4.10.2 and using the default example server
>>>>> configuration
>>>>>>>> that comes with Solr distribution.
>>>>>>>>
>>>>>>>> PDF Files - Indexing as such works fine, but when I query using *.*
>>> in
>>>>>>>> the
>>>>>>>> Solr Query console, metadata information is displayed properly.
>>>>> However,
>>>>>>>> the PDF content field is empty. This is happening for all PDF files
>>> I
>>>>>>>> have
>>>>>>>> tried. I have tried with some proprietary files, PDF eBooks etc.
>>>>> Whatever
>>>>>>>> be the PDF file, content is not being displayed.
>>>>>>>>
>>>>>>>> MS Office files - For some office files, everything works perfect
>>> and
>>>>>>>> the
>>>>>>>> extracted content is visible in the query console. However, for
>>>>> others, I
>>>>>>>> see the below error message during the indexing process.
>>>>>>>>
>>>>>>>> *Exception in thread "main"
>>>>>>>>
>>> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
>>>>>>>> org.apache.tika.exception.TikaException: Unexpected
>>>>>>>> RuntimeException
>>>>>>>> from
>>>>>>>> org.apache.tika.parser.microsoft.OfficeParser*
>>>>>>>>
>>>>>>>>
>>>>>>>> I am using SolrJ to index the documents and below is the code
>>> snippet
>>>>>>>> related to indexing. Please let me know where the issue is
>>> occurring.
>>>>>>>>
>>>>>>>> static String solrServerURL = "
>>>>>>>> http://localhost:8983/solr";
>>>>>>>> static SolrServer solrServer = new HttpSolrServer(solrServerURL);
>>>>>>>> static ContentStreamUpdateRequest
>>> indexingReq
>>>>> =
>>>>>>>> new
>>>>>>>>
>>>>>>>> ContentStreamUpdateRequest("/update/extract");
>>>>>>>>
>>>>>>>> indexingReq.addFile(file, fileType);
>>>>>>>> indexingReq.setParam("literal.id", literalId);
>>>>>>>> indexingReq.setParam("uprefix", "attr_");
>>>>>>>> indexingReq.setParam("fmap.content", "content");
>>>>>>>> indexingReq.setParam("literal.fileurl", fileURL);
>>>>>>>> indexingReq.setAction(AbstractUpdateRequest.ACTION.COMMIT, true,
>>> true);
>>>>>>>> solrServer.request(indexingReq);
>>>>>>>>
>>>>>>>> Thanks & Regards
>>>>>>>> Vijay
>>>>>>>>
>>>>>>>> --
>>>>>>>> The contents of this e-mail are confidential and for the exclusive
>>> use
>>>>> of
>>>>>>>> the intended recipient. If you receive this e-mail in error please
>>>>> delete
>>>>>>>> it from your system immediately and notify us either by e-mail or
>>>>>>>> telephone. You should not copy, forward or otherwise disclose the
>>>>> content
>>>>>>>> of the e-mail. The views expressed in this communication may not
>>>>>>>> necessarily be the view held by WHISHWORKS.
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Ph: 9845704792
>>>>>
>>>>
>>>> --
>>>> The contents of this e-mail are confidential and for the exclusive
>>>> use of
>>>> the intended recipient. If you receive this e-mail in error please
>>>> delete
>>>> it from your system immediately and notify us either by e-mail or
>>>> telephone. You should not copy, forward or otherwise disclose the
>>>> content
>>>> of the e-mail. The views expressed in this communication may not
>>>> necessarily be the view held by WHISHWORKS.
>>>
>>
>
--
Charlie Hull
Flax - Open Source Enterprise Search
tel/fax: +44 (0)8700 118334
mobile: +44 (0)7767 825828
web: www.flax.co.uk
RE: Indexing PDF and MS Office files
Posted by "Davis, Daniel (NIH/NLM) [C]" <da...@nih.gov>.
If you use pdftotext with a simple fork/exec per document, you will get about 5 MB/s throughput on a single AMD x86_64. Much of that is because of the fork/exec. I suggest that you use HTML output and UTF-8 encoding for the PDF, because that way you can get title/keywords and such as http meta keywords.
If you have the appetite for something truly great, try:
- Socket server listening for parsing requests
- pass off accept() sockets to pre-forked children
- in the children, use vfork, rather than fork
- tmpfs for outputted HTML documents
- Tempting to implement using mod_perl and httpd, at least to me.
-----Original Message-----
From: Siegfried Goeschl [mailto:sgoeschl@gmx.at]
Sent: Thursday, April 16, 2015 7:53 AM
To: solr-user@lucene.apache.org
Subject: Re: Indexing PDF and MS Office files
Hi Vijay,
I know the this road too well :-)
For PDF you can fallback to other tools for text extraction
* ps2ascii.ps
* XPDF's pdftotext CLI utility (more comfortable than Ghostscript)
* some other tools exists as well (pdflib)
If you start command line tools from your JVM please have a look at commons-exec :-)
Cheers,
Siegfried Goeschl
PS: one more thing - please, tell your management that you will never ever successfully all real-world PDFs and cater for that fact in your requirements :-)
On 16.04.15 13:10, Vijaya Narayana Reddy Bhoomi Reddy wrote:
> Erick,
>
> I tried indexing both ways - SolrJ / Tika's AutoParser and as well as
> SolrCell's ExtractRequestHandler. Majority of the PDF and Word
> documents are getting parsed properly and indexed into Solr. However,
> a minority of them keep failing wither PDFParser or OfficeParser error.
>
> Not sure if this behaviour can be modified so that all the documents
> can be indexed. The business requirement we have is to index all the documents.
> However, if a small percentage of them fails, not sure what other ways
> exist to index them.
>
> Any help please?
>
>
> Thanks & Regards
> Vijay
>
>
>
> On 15 April 2015 at 15:20, Erick Erickson <er...@gmail.com> wrote:
>
>> There's quite a discussion here:
>> https://issues.apache.org/jira/browse/SOLR-7137
>>
>> But, I personally am not a huge fan of pushing all the work on to
>> Solr, in a production environment the Solr server is responsible for
>> indexing, parsing the docs through Tika, perhaps searching etc. This
>> doesn't scale all that well.
>>
>> So an alternative is to use SolrJ with Tika, which is totally
>> independent of what version of Tika is on the Solr server. Here's an
>> example.
>>
>> http://lucidworks.com/blog/indexing-with-solrj/
>>
>> Best,
>> Erick
>>
>> On Wed, Apr 15, 2015 at 4:46 AM, Vijaya Narayana Reddy Bhoomi Reddy
>> <vi...@whishworks.com> wrote:
>>> Thanks everyone for the responses. Now I am able to index PDF
>>> documents successfully. I have implemented manual extraction using
>>> Tika's
>> AutoParser
>>> and PDF functionality is working fine. However, the error with some
>>> MS office word documents still persist.
>>>
>>> The error message is "java.lang.IllegalArgumentException: This
>>> paragraph
>> is
>>> not the first one in the table" which will eventually result in
>> "Unexpected
>>> RuntimeException from org.apache.tika.parser.microsoft.OfficeParser"
>>>
>>> Upon some reading, it looks like its a bug with Tika 1.5 and seems
>>> to
>> have
>>> been fixed with Tika 1.6 (
>> https://issues.apache.org/jira/browse/TIKA-1251 ).
>>> I am new to Solr / Tika and hence wondering whether I can change the
>>> Tika library alone to v1.6 without impacting any of the libraries
>>> within Solr 4.10.2? Please let me know your response and how to get
>>> away with this issue.
>>>
>>> Many thanks in advance.
>>>
>>> Thanks & Regards
>>> Vijay
>>>
>>>
>>> On 15 April 2015 at 05:14, Shyam R <sh...@gmail.com> wrote:
>>>
>>>> Vijay,
>>>>
>>>> You could try different excel files with different formats to rule
>>>> out
>> the
>>>> issue is with TIKA version being used.
>>>>
>>>> Thanks
>>>> Murthy
>>>>
>>>> On Wed, Apr 15, 2015 at 9:35 AM, Terry Rhodes
>>>> <tr...@gmail.com>
>>>> wrote:
>>>>
>>>>> Perhaps the PDF is protected and the content can not be extracted?
>>>>>
>>>>> i have an unverified suspicion that the tika shipped with solr
>>>>> 4.10.2
>> may
>>>>> not support some/all office 2013 document formats.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 4/14/2015 8:18 PM, Jack Krupansky wrote:
>>>>>
>>>>>> Try doing a manual extraction request directly to Solr (not via
>> SolrJ)
>>>> and
>>>>>> use the extractOnly option to see if the content is actually
>> extracted.
>>>>>>
>>>>>> See:
>>>>>> https://cwiki.apache.org/confluence/display/solr/
>>>>>> Uploading+Data+with+Solr+Cell+using+Apache+Tika
>>>>>>
>>>>>> Also, some PDF files actually have the content as a bitmap image,
>>>>>> so
>> no
>>>>>> text is extracted.
>>>>>>
>>>>>>
>>>>>> -- Jack Krupansky
>>>>>>
>>>>>> On Tue, Apr 14, 2015 at 10:57 AM, Vijaya Narayana Reddy Bhoomi
>>>>>> Reddy
>> <
>>>>>> vijaya.bhoomireddy@whishworks.com> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>>
>>>>>>> I am trying to index PDF and Microsoft Office files (.doc,
>>>>>>> .docx,
>> .ppt,
>>>>>>> .pptx, .xlx, and .xlx) files into Solr. I am facing the
>>>>>>> following
>>>> issues.
>>>>>>> Request to please let me know what is going wrong with the
>>>>>>> indexing process.
>>>>>>>
>>>>>>> I am using solr 4.10.2 and using the default example server
>>>> configuration
>>>>>>> that comes with Solr distribution.
>>>>>>>
>>>>>>> PDF Files - Indexing as such works fine, but when I query using
>>>>>>> *.*
>> in
>>>>>>> the
>>>>>>> Solr Query console, metadata information is displayed properly.
>>>> However,
>>>>>>> the PDF content field is empty. This is happening for all PDF
>>>>>>> files
>> I
>>>>>>> have
>>>>>>> tried. I have tried with some proprietary files, PDF eBooks etc.
>>>> Whatever
>>>>>>> be the PDF file, content is not being displayed.
>>>>>>>
>>>>>>> MS Office files - For some office files, everything works
>>>>>>> perfect
>> and
>>>>>>> the
>>>>>>> extracted content is visible in the query console. However, for
>>>> others, I
>>>>>>> see the below error message during the indexing process.
>>>>>>>
>>>>>>> *Exception in thread "main"
>>>>>>>
>> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
>>>>>>> org.apache.tika.exception.TikaException: Unexpected
>>>>>>> RuntimeException from
>>>>>>> org.apache.tika.parser.microsoft.OfficeParser*
>>>>>>>
>>>>>>>
>>>>>>> I am using SolrJ to index the documents and below is the code
>> snippet
>>>>>>> related to indexing. Please let me know where the issue is
>> occurring.
>>>>>>>
>>>>>>> static String solrServerURL = "
>>>>>>> http://localhost:8983/solr";
>>>>>>> static SolrServer solrServer = new HttpSolrServer(solrServerURL);
>>>>>>> static ContentStreamUpdateRequest
>> indexingReq
>>>> =
>>>>>>> new
>>>>>>>
>>>>>>> ContentStreamUpdateRequest("/update/extract");
>>>>>>>
>>>>>>> indexingReq.addFile(file, fileType);
>>>>>>> indexingReq.setParam("literal.id", literalId);
>>>>>>> indexingReq.setParam("uprefix", "attr_");
>>>>>>> indexingReq.setParam("fmap.content", "content");
>>>>>>> indexingReq.setParam("literal.fileurl", fileURL);
>>>>>>> indexingReq.setAction(AbstractUpdateRequest.ACTION.COMMIT, true,
>> true);
>>>>>>> solrServer.request(indexingReq);
>>>>>>>
>>>>>>> Thanks & Regards
>>>>>>> Vijay
>>>>>>>
>>>>>>> --
>>>>>>> The contents of this e-mail are confidential and for the
>>>>>>> exclusive
>> use
>>>> of
>>>>>>> the intended recipient. If you receive this e-mail in error
>>>>>>> please
>>>> delete
>>>>>>> it from your system immediately and notify us either by e-mail
>>>>>>> or telephone. You should not copy, forward or otherwise disclose
>>>>>>> the
>>>> content
>>>>>>> of the e-mail. The views expressed in this communication may not
>>>>>>> necessarily be the view held by WHISHWORKS.
>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Ph: 9845704792
>>>>
>>>
>>> --
>>> The contents of this e-mail are confidential and for the exclusive
>>> use of the intended recipient. If you receive this e-mail in error
>>> please delete it from your system immediately and notify us either
>>> by e-mail or telephone. You should not copy, forward or otherwise
>>> disclose the content of the e-mail. The views expressed in this
>>> communication may not necessarily be the view held by WHISHWORKS.
>>
>
Re: Indexing PDF and MS Office files
Posted by Walter Underwood <wu...@wunderwood.org>.
Turning PDF back into a structured document is like trying to turn hamburger back into a cow.
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
On Apr 16, 2015, at 4:55 AM, Allison, Timothy B. <ta...@mitre.org> wrote:
> +1
>
> :)
>
>> PS: one more thing - please, tell your management that you will never
>> ever successfully all real-world PDFs and cater for that fact in your
>> requirements :-)
>
RE: Indexing PDF and MS Office files
Posted by "Davis, Daniel (NIH/NLM) [C]" <da...@nih.gov>.
Indeed. Another solution is to purchase ABBYY or Nuance as a server, and have them do that work.
You will even get OCR. Both offer a Linux SDK.
-----Original Message-----
From: Allison, Timothy B. [mailto:tallison@mitre.org]
Sent: Thursday, April 16, 2015 7:56 AM
To: solr-user@lucene.apache.org
Subject: RE: Indexing PDF and MS Office files
+1
:)
>PS: one more thing - please, tell your management that you will never
>ever successfully all real-world PDFs and cater for that fact in your
>requirements :-)
RE: Indexing PDF and MS Office files
Posted by "Allison, Timothy B." <ta...@mitre.org>.
+1
:)
>PS: one more thing - please, tell your management that you will never
>ever successfully all real-world PDFs and cater for that fact in your
>requirements :-)
Re: Indexing PDF and MS Office files
Posted by Siegfried Goeschl <sg...@gmx.at>.
Hi Vijay,
I know the this road too well :-)
For PDF you can fallback to other tools for text extraction
* ps2ascii.ps
* XPDF's pdftotext CLI utility (more comfortable than Ghostscript)
* some other tools exists as well (pdflib)
If you start command line tools from your JVM please have a look at
commons-exec :-)
Cheers,
Siegfried Goeschl
PS: one more thing - please, tell your management that you will never
ever successfully all real-world PDFs and cater for that fact in your
requirements :-)
On 16.04.15 13:10, Vijaya Narayana Reddy Bhoomi Reddy wrote:
> Erick,
>
> I tried indexing both ways - SolrJ / Tika's AutoParser and as well as
> SolrCell's ExtractRequestHandler. Majority of the PDF and Word documents
> are getting parsed properly and indexed into Solr. However, a minority of
> them keep failing wither PDFParser or OfficeParser error.
>
> Not sure if this behaviour can be modified so that all the documents can be
> indexed. The business requirement we have is to index all the documents.
> However, if a small percentage of them fails, not sure what other ways
> exist to index them.
>
> Any help please?
>
>
> Thanks & Regards
> Vijay
>
>
>
> On 15 April 2015 at 15:20, Erick Erickson <er...@gmail.com> wrote:
>
>> There's quite a discussion here:
>> https://issues.apache.org/jira/browse/SOLR-7137
>>
>> But, I personally am not a huge fan of pushing all the work on to Solr, in
>> a
>> production environment the Solr server is responsible for indexing,
>> parsing the
>> docs through Tika, perhaps searching etc. This doesn't scale all that well.
>>
>> So an alternative is to use SolrJ with Tika, which is totally independent
>> of
>> what version of Tika is on the Solr server. Here's an example.
>>
>> http://lucidworks.com/blog/indexing-with-solrj/
>>
>> Best,
>> Erick
>>
>> On Wed, Apr 15, 2015 at 4:46 AM, Vijaya Narayana Reddy Bhoomi Reddy
>> <vi...@whishworks.com> wrote:
>>> Thanks everyone for the responses. Now I am able to index PDF documents
>>> successfully. I have implemented manual extraction using Tika's
>> AutoParser
>>> and PDF functionality is working fine. However, the error with some MS
>>> office word documents still persist.
>>>
>>> The error message is "java.lang.IllegalArgumentException: This paragraph
>> is
>>> not the first one in the table" which will eventually result in
>> "Unexpected
>>> RuntimeException from org.apache.tika.parser.microsoft.OfficeParser"
>>>
>>> Upon some reading, it looks like its a bug with Tika 1.5 and seems to
>> have
>>> been fixed with Tika 1.6 (
>> https://issues.apache.org/jira/browse/TIKA-1251 ).
>>> I am new to Solr / Tika and hence wondering whether I can change the Tika
>>> library alone to v1.6 without impacting any of the libraries within Solr
>>> 4.10.2? Please let me know your response and how to get away with this
>>> issue.
>>>
>>> Many thanks in advance.
>>>
>>> Thanks & Regards
>>> Vijay
>>>
>>>
>>> On 15 April 2015 at 05:14, Shyam R <sh...@gmail.com> wrote:
>>>
>>>> Vijay,
>>>>
>>>> You could try different excel files with different formats to rule out
>> the
>>>> issue is with TIKA version being used.
>>>>
>>>> Thanks
>>>> Murthy
>>>>
>>>> On Wed, Apr 15, 2015 at 9:35 AM, Terry Rhodes <tr...@gmail.com>
>>>> wrote:
>>>>
>>>>> Perhaps the PDF is protected and the content can not be extracted?
>>>>>
>>>>> i have an unverified suspicion that the tika shipped with solr 4.10.2
>> may
>>>>> not support some/all office 2013 document formats.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 4/14/2015 8:18 PM, Jack Krupansky wrote:
>>>>>
>>>>>> Try doing a manual extraction request directly to Solr (not via
>> SolrJ)
>>>> and
>>>>>> use the extractOnly option to see if the content is actually
>> extracted.
>>>>>>
>>>>>> See:
>>>>>> https://cwiki.apache.org/confluence/display/solr/
>>>>>> Uploading+Data+with+Solr+Cell+using+Apache+Tika
>>>>>>
>>>>>> Also, some PDF files actually have the content as a bitmap image, so
>> no
>>>>>> text is extracted.
>>>>>>
>>>>>>
>>>>>> -- Jack Krupansky
>>>>>>
>>>>>> On Tue, Apr 14, 2015 at 10:57 AM, Vijaya Narayana Reddy Bhoomi Reddy
>> <
>>>>>> vijaya.bhoomireddy@whishworks.com> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>>
>>>>>>> I am trying to index PDF and Microsoft Office files (.doc, .docx,
>> .ppt,
>>>>>>> .pptx, .xlx, and .xlx) files into Solr. I am facing the following
>>>> issues.
>>>>>>> Request to please let me know what is going wrong with the indexing
>>>>>>> process.
>>>>>>>
>>>>>>> I am using solr 4.10.2 and using the default example server
>>>> configuration
>>>>>>> that comes with Solr distribution.
>>>>>>>
>>>>>>> PDF Files - Indexing as such works fine, but when I query using *.*
>> in
>>>>>>> the
>>>>>>> Solr Query console, metadata information is displayed properly.
>>>> However,
>>>>>>> the PDF content field is empty. This is happening for all PDF files
>> I
>>>>>>> have
>>>>>>> tried. I have tried with some proprietary files, PDF eBooks etc.
>>>> Whatever
>>>>>>> be the PDF file, content is not being displayed.
>>>>>>>
>>>>>>> MS Office files - For some office files, everything works perfect
>> and
>>>>>>> the
>>>>>>> extracted content is visible in the query console. However, for
>>>> others, I
>>>>>>> see the below error message during the indexing process.
>>>>>>>
>>>>>>> *Exception in thread "main"
>>>>>>>
>> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
>>>>>>> org.apache.tika.exception.TikaException: Unexpected RuntimeException
>>>>>>> from
>>>>>>> org.apache.tika.parser.microsoft.OfficeParser*
>>>>>>>
>>>>>>>
>>>>>>> I am using SolrJ to index the documents and below is the code
>> snippet
>>>>>>> related to indexing. Please let me know where the issue is
>> occurring.
>>>>>>>
>>>>>>> static String solrServerURL = "
>>>>>>> http://localhost:8983/solr";
>>>>>>> static SolrServer solrServer = new HttpSolrServer(solrServerURL);
>>>>>>> static ContentStreamUpdateRequest
>> indexingReq
>>>> =
>>>>>>> new
>>>>>>>
>>>>>>> ContentStreamUpdateRequest("/update/extract");
>>>>>>>
>>>>>>> indexingReq.addFile(file, fileType);
>>>>>>> indexingReq.setParam("literal.id", literalId);
>>>>>>> indexingReq.setParam("uprefix", "attr_");
>>>>>>> indexingReq.setParam("fmap.content", "content");
>>>>>>> indexingReq.setParam("literal.fileurl", fileURL);
>>>>>>> indexingReq.setAction(AbstractUpdateRequest.ACTION.COMMIT, true,
>> true);
>>>>>>> solrServer.request(indexingReq);
>>>>>>>
>>>>>>> Thanks & Regards
>>>>>>> Vijay
>>>>>>>
>>>>>>> --
>>>>>>> The contents of this e-mail are confidential and for the exclusive
>> use
>>>> of
>>>>>>> the intended recipient. If you receive this e-mail in error please
>>>> delete
>>>>>>> it from your system immediately and notify us either by e-mail or
>>>>>>> telephone. You should not copy, forward or otherwise disclose the
>>>> content
>>>>>>> of the e-mail. The views expressed in this communication may not
>>>>>>> necessarily be the view held by WHISHWORKS.
>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Ph: 9845704792
>>>>
>>>
>>> --
>>> The contents of this e-mail are confidential and for the exclusive use of
>>> the intended recipient. If you receive this e-mail in error please delete
>>> it from your system immediately and notify us either by e-mail or
>>> telephone. You should not copy, forward or otherwise disclose the content
>>> of the e-mail. The views expressed in this communication may not
>>> necessarily be the view held by WHISHWORKS.
>>
>
RE: Indexing PDF and MS Office files
Posted by "Allison, Timothy B." <ta...@mitre.org>.
Let's move this to the Tika users' list.
I'm aware that [1] is quite common in govdocs1, and it might (?) be the source of your problem with MSWord files.
If you can share a stack trace, we'll be better able to diagnose.
Best,
Tim
[1]
org.apache.poi.hslf.exceptions.HSLFException: java.util.zip.ZipException: unknown compression method
at org.apache.poi.hslf.blip.WMF.getData(WMF.java:65)
at o.a.t.parser.microsoft.HSLFExtractor.handleSlideEmbeddedPictures(HSLFExtractor.java:264)
-----Original Message-----
From: Vijaya Narayana Reddy Bhoomi Reddy [mailto:vijaya.bhoomireddy@whishworks.com]
Sent: Thursday, April 16, 2015 9:17 AM
To: solr-user@lucene.apache.org
Subject: Re: Indexing PDF and MS Office files
For MS Word documents, one common pattern for all failed documents I
noticed is that all of them contain embedded images (like scanned signature
images embedded into the documents. These documents are much like some
letterheads where someone scanned the signature image and then embedded
into the document along with the text) with in the documents.
For other documents which completed successfully, no images were present.
Just wondering if these are causing the issue.
Thanks & Regards
Vijay
On 16 April 2015 at 12:58, Vijaya Narayana Reddy Bhoomi Reddy <
vijaya.bhoomireddy@whishworks.com> wrote:
> Thanks Tim.
>
> I shall raise a Jira with the stack trace information.
>
> Thanks & Regards
> Vijay
>
>
> On 16 April 2015 at 12:54, Allison, Timothy B. <ta...@mitre.org> wrote:
>
>> This sounds like a Tika issue, let's move discussion to that list.
>>
>> If you are still having problems after you upgrade to Tika 1.8, please at
>> least submit the stack traces (if you can) to the Tika jira. We may be
>> able to find a document that triggers that stack trace in govdocs1 or the
>> slice of CommonCrawl that Julien Nioche contributed to our eval effort.
>>
>> Tika is not perfect and it will fail on some files, but we are always
>> working to improve it.
>>
>> Best,
>>
>> Tim
>>
>> -----Original Message-----
>> From: Vijaya Narayana Reddy Bhoomi Reddy [mailto:
>> vijaya.bhoomireddy@whishworks.com]
>> Sent: Thursday, April 16, 2015 7:44 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Indexing PDF and MS Office files
>>
>> Thanks Allison.
>>
>> I tried with the mentioned changes. But still no luck. I am using the code
>> from lucidworks site provided by Erick and now included the changes
>> mentioned by you. But still the issue persists with a small percentage of
>> documents (both PDF and MS Office documents) failing. Unfortunately, these
>> documents are proprietary and client-confidential and hence I am not sure
>> whether they can be uploaded into Jira.
>>
>> These files normally open in Adobe Reader and MS Office tools.
>>
>> Thanks & Regards
>> Vijay
>>
>>
>> On 16 April 2015 at 12:33, Allison, Timothy B. <ta...@mitre.org>
>> wrote:
>>
>> > I entirely agree with Erick -- it is best to isolate Tika in its own jvm
>> > if you can -- bad things can happen if you don't [1] [2].
>> >
>> > Erick's blog on SolrJ is fantastic. If you want to have Tika parse
>> > embedded documents/attachments, make sure to set the parser in the
>> > ParseContext before parsing:
>> >
>> > ParseContext context = new ParseContext();
>> > //add this line:
>> > context.set(Parser.class, _autoParser)
>> > InputStream input = new FileInputStream(file);
>> >
>> > Tika 1.8 is soon to be released. If that doesn't fix your problems,
>> > please submit stacktraces (and docs, if possible) to the Tika jira, and
>> > we'll try to make the fixes.
>> >
>> > Cheers,
>> >
>> > Tim
>> >
>> > [1]
>> >
>> http://events.linuxfoundation.org/sites/events/files/slides/1s_and_0s_1.pdf
>> > [2]
>> >
>> http://events.linuxfoundation.org/sites/events/files/slides/TikaEval_ACNA15_allison_herceg_v2.pdf
>> > -----Original Message-----
>> > From: Vijaya Narayana Reddy Bhoomi Reddy [mailto:
>> > vijaya.bhoomireddy@whishworks.com]
>> > Sent: Thursday, April 16, 2015 7:10 AM
>> > To: solr-user@lucene.apache.org
>> > Subject: Re: Indexing PDF and MS Office files
>> >
>> > Erick,
>> >
>> > I tried indexing both ways - SolrJ / Tika's AutoParser and as well as
>> > SolrCell's ExtractRequestHandler. Majority of the PDF and Word documents
>> > are getting parsed properly and indexed into Solr. However, a minority
>> of
>> > them keep failing wither PDFParser or OfficeParser error.
>> >
>> > Not sure if this behaviour can be modified so that all the documents
>> can be
>> > indexed. The business requirement we have is to index all the documents.
>> > However, if a small percentage of them fails, not sure what other ways
>> > exist to index them.
>> >
>> > Any help please?
>> >
>> >
>> > Thanks & Regards
>> > Vijay
>> >
>> >
>> >
>> > On 15 April 2015 at 15:20, Erick Erickson <er...@gmail.com>
>> wrote:
>> >
>> > > There's quite a discussion here:
>> > > https://issues.apache.org/jira/browse/SOLR-7137
>> > >
>> > > But, I personally am not a huge fan of pushing all the work on to
>> Solr,
>> > in
>> > > a
>> > > production environment the Solr server is responsible for indexing,
>> > > parsing the
>> > > docs through Tika, perhaps searching etc. This doesn't scale all that
>> > well.
>> > >
>> > > So an alternative is to use SolrJ with Tika, which is totally
>> independent
>> > > of
>> > > what version of Tika is on the Solr server. Here's an example.
>> > >
>> > > http://lucidworks.com/blog/indexing-with-solrj/
>> > >
>> > > Best,
>> > > Erick
>> > >
>> > > On Wed, Apr 15, 2015 at 4:46 AM, Vijaya Narayana Reddy Bhoomi Reddy
>> > > <vi...@whishworks.com> wrote:
>> > > > Thanks everyone for the responses. Now I am able to index PDF
>> documents
>> > > > successfully. I have implemented manual extraction using Tika's
>> > > AutoParser
>> > > > and PDF functionality is working fine. However, the error with
>> some MS
>> > > > office word documents still persist.
>> > > >
>> > > > The error message is "java.lang.IllegalArgumentException: This
>> > paragraph
>> > > is
>> > > > not the first one in the table" which will eventually result in
>> > > "Unexpected
>> > > > RuntimeException from org.apache.tika.parser.microsoft.OfficeParser"
>> > > >
>> > > > Upon some reading, it looks like its a bug with Tika 1.5 and seems
>> to
>> > > have
>> > > > been fixed with Tika 1.6 (
>> > > https://issues.apache.org/jira/browse/TIKA-1251 ).
>> > > > I am new to Solr / Tika and hence wondering whether I can change the
>> > Tika
>> > > > library alone to v1.6 without impacting any of the libraries within
>> > Solr
>> > > > 4.10.2? Please let me know your response and how to get away with
>> this
>> > > > issue.
>> > > >
>> > > > Many thanks in advance.
>> > > >
>> > > > Thanks & Regards
>> > > > Vijay
>> > > >
>> > > >
>> > > > On 15 April 2015 at 05:14, Shyam R <sh...@gmail.com> wrote:
>> > > >
>> > > >> Vijay,
>> > > >>
>> > > >> You could try different excel files with different formats to rule
>> out
>> > > the
>> > > >> issue is with TIKA version being used.
>> > > >>
>> > > >> Thanks
>> > > >> Murthy
>> > > >>
>> > > >> On Wed, Apr 15, 2015 at 9:35 AM, Terry Rhodes <
>> trhodes314@gmail.com>
>> > > >> wrote:
>> > > >>
>> > > >> > Perhaps the PDF is protected and the content can not be
>> extracted?
>> > > >> >
>> > > >> > i have an unverified suspicion that the tika shipped with solr
>> > 4.10.2
>> > > may
>> > > >> > not support some/all office 2013 document formats.
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >> > On 4/14/2015 8:18 PM, Jack Krupansky wrote:
>> > > >> >
>> > > >> >> Try doing a manual extraction request directly to Solr (not via
>> > > SolrJ)
>> > > >> and
>> > > >> >> use the extractOnly option to see if the content is actually
>> > > extracted.
>> > > >> >>
>> > > >> >> See:
>> > > >> >> https://cwiki.apache.org/confluence/display/solr/
>> > > >> >> Uploading+Data+with+Solr+Cell+using+Apache+Tika
>> > > >> >>
>> > > >> >> Also, some PDF files actually have the content as a bitmap
>> image,
>> > so
>> > > no
>> > > >> >> text is extracted.
>> > > >> >>
>> > > >> >>
>> > > >> >> -- Jack Krupansky
>> > > >> >>
>> > > >> >> On Tue, Apr 14, 2015 at 10:57 AM, Vijaya Narayana Reddy Bhoomi
>> > Reddy
>> > > <
>> > > >> >> vijaya.bhoomireddy@whishworks.com> wrote:
>> > > >> >>
>> > > >> >> Hi,
>> > > >> >>>
>> > > >> >>> I am trying to index PDF and Microsoft Office files (.doc,
>> .docx,
>> > > .ppt,
>> > > >> >>> .pptx, .xlx, and .xlx) files into Solr. I am facing the
>> following
>> > > >> issues.
>> > > >> >>> Request to please let me know what is going wrong with the
>> > indexing
>> > > >> >>> process.
>> > > >> >>>
>> > > >> >>> I am using solr 4.10.2 and using the default example server
>> > > >> configuration
>> > > >> >>> that comes with Solr distribution.
>> > > >> >>>
>> > > >> >>> PDF Files - Indexing as such works fine, but when I query using
>> > *.*
>> > > in
>> > > >> >>> the
>> > > >> >>> Solr Query console, metadata information is displayed properly.
>> > > >> However,
>> > > >> >>> the PDF content field is empty. This is happening for all PDF
>> > files
>> > > I
>> > > >> >>> have
>> > > >> >>> tried. I have tried with some proprietary files, PDF eBooks
>> etc.
>> > > >> Whatever
>> > > >> >>> be the PDF file, content is not being displayed.
>> > > >> >>>
>> > > >> >>> MS Office files - For some office files, everything works
>> perfect
>> > > and
>> > > >> >>> the
>> > > >> >>> extracted content is visible in the query console. However, for
>> > > >> others, I
>> > > >> >>> see the below error message during the indexing process.
>> > > >> >>>
>> > > >> >>> *Exception in thread "main"
>> > > >> >>>
>> > > org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
>> > > >> >>> org.apache.tika.exception.TikaException: Unexpected
>> > RuntimeException
>> > > >> >>> from
>> > > >> >>> org.apache.tika.parser.microsoft.OfficeParser*
>> > > >> >>>
>> > > >> >>>
>> > > >> >>> I am using SolrJ to index the documents and below is the code
>> > > snippet
>> > > >> >>> related to indexing. Please let me know where the issue is
>> > > occurring.
>> > > >> >>>
>> > > >> >>> static String solrServerURL = "
>> > > >> >>> http://localhost:8983/solr";
>> > > >> >>> static SolrServer solrServer = new
>> HttpSolrServer(solrServerURL);
>> > > >> >>> static ContentStreamUpdateRequest
>> > > indexingReq
>> > > >> =
>> > > >> >>> new
>> > > >> >>>
>> > > >> >>> ContentStreamUpdateRequest("/update/extract");
>> > > >> >>>
>> > > >> >>> indexingReq.addFile(file, fileType);
>> > > >> >>> indexingReq.setParam("literal.id", literalId);
>> > > >> >>> indexingReq.setParam("uprefix", "attr_");
>> > > >> >>> indexingReq.setParam("fmap.content", "content");
>> > > >> >>> indexingReq.setParam("literal.fileurl", fileURL);
>> > > >> >>> indexingReq.setAction(AbstractUpdateRequest.ACTION.COMMIT,
>> true,
>> > > true);
>> > > >> >>> solrServer.request(indexingReq);
>> > > >> >>>
>> > > >> >>> Thanks & Regards
>> > > >> >>> Vijay
>> > > >> >>>
>> > > >> >>> --
>> > > >> >>> The contents of this e-mail are confidential and for the
>> exclusive
>> > > use
>> > > >> of
>> > > >> >>> the intended recipient. If you receive this e-mail in error
>> please
>> > > >> delete
>> > > >> >>> it from your system immediately and notify us either by e-mail
>> or
>> > > >> >>> telephone. You should not copy, forward or otherwise disclose
>> the
>> > > >> content
>> > > >> >>> of the e-mail. The views expressed in this communication may
>> not
>> > > >> >>> necessarily be the view held by WHISHWORKS.
>> > > >> >>>
>> > > >> >>>
>> > > >> >
>> > > >>
>> > > >>
>> > > >> --
>> > > >> Ph: 9845704792
>> > > >>
>> > > >
>> > > > --
>> > > > The contents of this e-mail are confidential and for the exclusive
>> use
>> > of
>> > > > the intended recipient. If you receive this e-mail in error please
>> > delete
>> > > > it from your system immediately and notify us either by e-mail or
>> > > > telephone. You should not copy, forward or otherwise disclose the
>> > content
>> > > > of the e-mail. The views expressed in this communication may not
>> > > > necessarily be the view held by WHISHWORKS.
>> > >
>> >
>> > --
>> > The contents of this e-mail are confidential and for the exclusive use
>> of
>> > the intended recipient. If you receive this e-mail in error please
>> delete
>> > it from your system immediately and notify us either by e-mail or
>> > telephone. You should not copy, forward or otherwise disclose the
>> content
>> > of the e-mail. The views expressed in this communication may not
>> > necessarily be the view held by WHISHWORKS.
>> >
>>
>> --
>> The contents of this e-mail are confidential and for the exclusive use of
>> the intended recipient. If you receive this e-mail in error please delete
>> it from your system immediately and notify us either by e-mail or
>> telephone. You should not copy, forward or otherwise disclose the content
>> of the e-mail. The views expressed in this communication may not
>> necessarily be the view held by WHISHWORKS.
>>
>
>
--
The contents of this e-mail are confidential and for the exclusive use of
the intended recipient. If you receive this e-mail in error please delete
it from your system immediately and notify us either by e-mail or
telephone. You should not copy, forward or otherwise disclose the content
of the e-mail. The views expressed in this communication may not
necessarily be the view held by WHISHWORKS.
Re: Indexing PDF and MS Office files
Posted by Vijaya Narayana Reddy Bhoomi Reddy <vi...@whishworks.com>.
For MS Word documents, one common pattern for all failed documents I
noticed is that all of them contain embedded images (like scanned signature
images embedded into the documents. These documents are much like some
letterheads where someone scanned the signature image and then embedded
into the document along with the text) with in the documents.
For other documents which completed successfully, no images were present.
Just wondering if these are causing the issue.
Thanks & Regards
Vijay
On 16 April 2015 at 12:58, Vijaya Narayana Reddy Bhoomi Reddy <
vijaya.bhoomireddy@whishworks.com> wrote:
> Thanks Tim.
>
> I shall raise a Jira with the stack trace information.
>
> Thanks & Regards
> Vijay
>
>
> On 16 April 2015 at 12:54, Allison, Timothy B. <ta...@mitre.org> wrote:
>
>> This sounds like a Tika issue, let's move discussion to that list.
>>
>> If you are still having problems after you upgrade to Tika 1.8, please at
>> least submit the stack traces (if you can) to the Tika jira. We may be
>> able to find a document that triggers that stack trace in govdocs1 or the
>> slice of CommonCrawl that Julien Nioche contributed to our eval effort.
>>
>> Tika is not perfect and it will fail on some files, but we are always
>> working to improve it.
>>
>> Best,
>>
>> Tim
>>
>> -----Original Message-----
>> From: Vijaya Narayana Reddy Bhoomi Reddy [mailto:
>> vijaya.bhoomireddy@whishworks.com]
>> Sent: Thursday, April 16, 2015 7:44 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Indexing PDF and MS Office files
>>
>> Thanks Allison.
>>
>> I tried with the mentioned changes. But still no luck. I am using the code
>> from lucidworks site provided by Erick and now included the changes
>> mentioned by you. But still the issue persists with a small percentage of
>> documents (both PDF and MS Office documents) failing. Unfortunately, these
>> documents are proprietary and client-confidential and hence I am not sure
>> whether they can be uploaded into Jira.
>>
>> These files normally open in Adobe Reader and MS Office tools.
>>
>> Thanks & Regards
>> Vijay
>>
>>
>> On 16 April 2015 at 12:33, Allison, Timothy B. <ta...@mitre.org>
>> wrote:
>>
>> > I entirely agree with Erick -- it is best to isolate Tika in its own jvm
>> > if you can -- bad things can happen if you don't [1] [2].
>> >
>> > Erick's blog on SolrJ is fantastic. If you want to have Tika parse
>> > embedded documents/attachments, make sure to set the parser in the
>> > ParseContext before parsing:
>> >
>> > ParseContext context = new ParseContext();
>> > //add this line:
>> > context.set(Parser.class, _autoParser)
>> > InputStream input = new FileInputStream(file);
>> >
>> > Tika 1.8 is soon to be released. If that doesn't fix your problems,
>> > please submit stacktraces (and docs, if possible) to the Tika jira, and
>> > we'll try to make the fixes.
>> >
>> > Cheers,
>> >
>> > Tim
>> >
>> > [1]
>> >
>> http://events.linuxfoundation.org/sites/events/files/slides/1s_and_0s_1.pdf
>> > [2]
>> >
>> http://events.linuxfoundation.org/sites/events/files/slides/TikaEval_ACNA15_allison_herceg_v2.pdf
>> > -----Original Message-----
>> > From: Vijaya Narayana Reddy Bhoomi Reddy [mailto:
>> > vijaya.bhoomireddy@whishworks.com]
>> > Sent: Thursday, April 16, 2015 7:10 AM
>> > To: solr-user@lucene.apache.org
>> > Subject: Re: Indexing PDF and MS Office files
>> >
>> > Erick,
>> >
>> > I tried indexing both ways - SolrJ / Tika's AutoParser and as well as
>> > SolrCell's ExtractRequestHandler. Majority of the PDF and Word documents
>> > are getting parsed properly and indexed into Solr. However, a minority
>> of
>> > them keep failing wither PDFParser or OfficeParser error.
>> >
>> > Not sure if this behaviour can be modified so that all the documents
>> can be
>> > indexed. The business requirement we have is to index all the documents.
>> > However, if a small percentage of them fails, not sure what other ways
>> > exist to index them.
>> >
>> > Any help please?
>> >
>> >
>> > Thanks & Regards
>> > Vijay
>> >
>> >
>> >
>> > On 15 April 2015 at 15:20, Erick Erickson <er...@gmail.com>
>> wrote:
>> >
>> > > There's quite a discussion here:
>> > > https://issues.apache.org/jira/browse/SOLR-7137
>> > >
>> > > But, I personally am not a huge fan of pushing all the work on to
>> Solr,
>> > in
>> > > a
>> > > production environment the Solr server is responsible for indexing,
>> > > parsing the
>> > > docs through Tika, perhaps searching etc. This doesn't scale all that
>> > well.
>> > >
>> > > So an alternative is to use SolrJ with Tika, which is totally
>> independent
>> > > of
>> > > what version of Tika is on the Solr server. Here's an example.
>> > >
>> > > http://lucidworks.com/blog/indexing-with-solrj/
>> > >
>> > > Best,
>> > > Erick
>> > >
>> > > On Wed, Apr 15, 2015 at 4:46 AM, Vijaya Narayana Reddy Bhoomi Reddy
>> > > <vi...@whishworks.com> wrote:
>> > > > Thanks everyone for the responses. Now I am able to index PDF
>> documents
>> > > > successfully. I have implemented manual extraction using Tika's
>> > > AutoParser
>> > > > and PDF functionality is working fine. However, the error with
>> some MS
>> > > > office word documents still persist.
>> > > >
>> > > > The error message is "java.lang.IllegalArgumentException: This
>> > paragraph
>> > > is
>> > > > not the first one in the table" which will eventually result in
>> > > "Unexpected
>> > > > RuntimeException from org.apache.tika.parser.microsoft.OfficeParser"
>> > > >
>> > > > Upon some reading, it looks like its a bug with Tika 1.5 and seems
>> to
>> > > have
>> > > > been fixed with Tika 1.6 (
>> > > https://issues.apache.org/jira/browse/TIKA-1251 ).
>> > > > I am new to Solr / Tika and hence wondering whether I can change the
>> > Tika
>> > > > library alone to v1.6 without impacting any of the libraries within
>> > Solr
>> > > > 4.10.2? Please let me know your response and how to get away with
>> this
>> > > > issue.
>> > > >
>> > > > Many thanks in advance.
>> > > >
>> > > > Thanks & Regards
>> > > > Vijay
>> > > >
>> > > >
>> > > > On 15 April 2015 at 05:14, Shyam R <sh...@gmail.com> wrote:
>> > > >
>> > > >> Vijay,
>> > > >>
>> > > >> You could try different excel files with different formats to rule
>> out
>> > > the
>> > > >> issue is with TIKA version being used.
>> > > >>
>> > > >> Thanks
>> > > >> Murthy
>> > > >>
>> > > >> On Wed, Apr 15, 2015 at 9:35 AM, Terry Rhodes <
>> trhodes314@gmail.com>
>> > > >> wrote:
>> > > >>
>> > > >> > Perhaps the PDF is protected and the content can not be
>> extracted?
>> > > >> >
>> > > >> > i have an unverified suspicion that the tika shipped with solr
>> > 4.10.2
>> > > may
>> > > >> > not support some/all office 2013 document formats.
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >> > On 4/14/2015 8:18 PM, Jack Krupansky wrote:
>> > > >> >
>> > > >> >> Try doing a manual extraction request directly to Solr (not via
>> > > SolrJ)
>> > > >> and
>> > > >> >> use the extractOnly option to see if the content is actually
>> > > extracted.
>> > > >> >>
>> > > >> >> See:
>> > > >> >> https://cwiki.apache.org/confluence/display/solr/
>> > > >> >> Uploading+Data+with+Solr+Cell+using+Apache+Tika
>> > > >> >>
>> > > >> >> Also, some PDF files actually have the content as a bitmap
>> image,
>> > so
>> > > no
>> > > >> >> text is extracted.
>> > > >> >>
>> > > >> >>
>> > > >> >> -- Jack Krupansky
>> > > >> >>
>> > > >> >> On Tue, Apr 14, 2015 at 10:57 AM, Vijaya Narayana Reddy Bhoomi
>> > Reddy
>> > > <
>> > > >> >> vijaya.bhoomireddy@whishworks.com> wrote:
>> > > >> >>
>> > > >> >> Hi,
>> > > >> >>>
>> > > >> >>> I am trying to index PDF and Microsoft Office files (.doc,
>> .docx,
>> > > .ppt,
>> > > >> >>> .pptx, .xlx, and .xlx) files into Solr. I am facing the
>> following
>> > > >> issues.
>> > > >> >>> Request to please let me know what is going wrong with the
>> > indexing
>> > > >> >>> process.
>> > > >> >>>
>> > > >> >>> I am using solr 4.10.2 and using the default example server
>> > > >> configuration
>> > > >> >>> that comes with Solr distribution.
>> > > >> >>>
>> > > >> >>> PDF Files - Indexing as such works fine, but when I query using
>> > *.*
>> > > in
>> > > >> >>> the
>> > > >> >>> Solr Query console, metadata information is displayed properly.
>> > > >> However,
>> > > >> >>> the PDF content field is empty. This is happening for all PDF
>> > files
>> > > I
>> > > >> >>> have
>> > > >> >>> tried. I have tried with some proprietary files, PDF eBooks
>> etc.
>> > > >> Whatever
>> > > >> >>> be the PDF file, content is not being displayed.
>> > > >> >>>
>> > > >> >>> MS Office files - For some office files, everything works
>> perfect
>> > > and
>> > > >> >>> the
>> > > >> >>> extracted content is visible in the query console. However, for
>> > > >> others, I
>> > > >> >>> see the below error message during the indexing process.
>> > > >> >>>
>> > > >> >>> *Exception in thread "main"
>> > > >> >>>
>> > > org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
>> > > >> >>> org.apache.tika.exception.TikaException: Unexpected
>> > RuntimeException
>> > > >> >>> from
>> > > >> >>> org.apache.tika.parser.microsoft.OfficeParser*
>> > > >> >>>
>> > > >> >>>
>> > > >> >>> I am using SolrJ to index the documents and below is the code
>> > > snippet
>> > > >> >>> related to indexing. Please let me know where the issue is
>> > > occurring.
>> > > >> >>>
>> > > >> >>> static String solrServerURL = "
>> > > >> >>> http://localhost:8983/solr";
>> > > >> >>> static SolrServer solrServer = new
>> HttpSolrServer(solrServerURL);
>> > > >> >>> static ContentStreamUpdateRequest
>> > > indexingReq
>> > > >> =
>> > > >> >>> new
>> > > >> >>>
>> > > >> >>> ContentStreamUpdateRequest("/update/extract");
>> > > >> >>>
>> > > >> >>> indexingReq.addFile(file, fileType);
>> > > >> >>> indexingReq.setParam("literal.id", literalId);
>> > > >> >>> indexingReq.setParam("uprefix", "attr_");
>> > > >> >>> indexingReq.setParam("fmap.content", "content");
>> > > >> >>> indexingReq.setParam("literal.fileurl", fileURL);
>> > > >> >>> indexingReq.setAction(AbstractUpdateRequest.ACTION.COMMIT,
>> true,
>> > > true);
>> > > >> >>> solrServer.request(indexingReq);
>> > > >> >>>
>> > > >> >>> Thanks & Regards
>> > > >> >>> Vijay
>> > > >> >>>
>> > > >> >>> --
>> > > >> >>> The contents of this e-mail are confidential and for the
>> exclusive
>> > > use
>> > > >> of
>> > > >> >>> the intended recipient. If you receive this e-mail in error
>> please
>> > > >> delete
>> > > >> >>> it from your system immediately and notify us either by e-mail
>> or
>> > > >> >>> telephone. You should not copy, forward or otherwise disclose
>> the
>> > > >> content
>> > > >> >>> of the e-mail. The views expressed in this communication may
>> not
>> > > >> >>> necessarily be the view held by WHISHWORKS.
>> > > >> >>>
>> > > >> >>>
>> > > >> >
>> > > >>
>> > > >>
>> > > >> --
>> > > >> Ph: 9845704792
>> > > >>
>> > > >
>> > > > --
>> > > > The contents of this e-mail are confidential and for the exclusive
>> use
>> > of
>> > > > the intended recipient. If you receive this e-mail in error please
>> > delete
>> > > > it from your system immediately and notify us either by e-mail or
>> > > > telephone. You should not copy, forward or otherwise disclose the
>> > content
>> > > > of the e-mail. The views expressed in this communication may not
>> > > > necessarily be the view held by WHISHWORKS.
>> > >
>> >
>> > --
>> > The contents of this e-mail are confidential and for the exclusive use
>> of
>> > the intended recipient. If you receive this e-mail in error please
>> delete
>> > it from your system immediately and notify us either by e-mail or
>> > telephone. You should not copy, forward or otherwise disclose the
>> content
>> > of the e-mail. The views expressed in this communication may not
>> > necessarily be the view held by WHISHWORKS.
>> >
>>
>> --
>> The contents of this e-mail are confidential and for the exclusive use of
>> the intended recipient. If you receive this e-mail in error please delete
>> it from your system immediately and notify us either by e-mail or
>> telephone. You should not copy, forward or otherwise disclose the content
>> of the e-mail. The views expressed in this communication may not
>> necessarily be the view held by WHISHWORKS.
>>
>
>
--
The contents of this e-mail are confidential and for the exclusive use of
the intended recipient. If you receive this e-mail in error please delete
it from your system immediately and notify us either by e-mail or
telephone. You should not copy, forward or otherwise disclose the content
of the e-mail. The views expressed in this communication may not
necessarily be the view held by WHISHWORKS.
Re: Indexing PDF and MS Office files
Posted by Vijaya Narayana Reddy Bhoomi Reddy <vi...@whishworks.com>.
Thanks Tim.
I shall raise a Jira with the stack trace information.
Thanks & Regards
Vijay
On 16 April 2015 at 12:54, Allison, Timothy B. <ta...@mitre.org> wrote:
> This sounds like a Tika issue, let's move discussion to that list.
>
> If you are still having problems after you upgrade to Tika 1.8, please at
> least submit the stack traces (if you can) to the Tika jira. We may be
> able to find a document that triggers that stack trace in govdocs1 or the
> slice of CommonCrawl that Julien Nioche contributed to our eval effort.
>
> Tika is not perfect and it will fail on some files, but we are always
> working to improve it.
>
> Best,
>
> Tim
>
> -----Original Message-----
> From: Vijaya Narayana Reddy Bhoomi Reddy [mailto:
> vijaya.bhoomireddy@whishworks.com]
> Sent: Thursday, April 16, 2015 7:44 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Indexing PDF and MS Office files
>
> Thanks Allison.
>
> I tried with the mentioned changes. But still no luck. I am using the code
> from lucidworks site provided by Erick and now included the changes
> mentioned by you. But still the issue persists with a small percentage of
> documents (both PDF and MS Office documents) failing. Unfortunately, these
> documents are proprietary and client-confidential and hence I am not sure
> whether they can be uploaded into Jira.
>
> These files normally open in Adobe Reader and MS Office tools.
>
> Thanks & Regards
> Vijay
>
>
> On 16 April 2015 at 12:33, Allison, Timothy B. <ta...@mitre.org> wrote:
>
> > I entirely agree with Erick -- it is best to isolate Tika in its own jvm
> > if you can -- bad things can happen if you don't [1] [2].
> >
> > Erick's blog on SolrJ is fantastic. If you want to have Tika parse
> > embedded documents/attachments, make sure to set the parser in the
> > ParseContext before parsing:
> >
> > ParseContext context = new ParseContext();
> > //add this line:
> > context.set(Parser.class, _autoParser)
> > InputStream input = new FileInputStream(file);
> >
> > Tika 1.8 is soon to be released. If that doesn't fix your problems,
> > please submit stacktraces (and docs, if possible) to the Tika jira, and
> > we'll try to make the fixes.
> >
> > Cheers,
> >
> > Tim
> >
> > [1]
> >
> http://events.linuxfoundation.org/sites/events/files/slides/1s_and_0s_1.pdf
> > [2]
> >
> http://events.linuxfoundation.org/sites/events/files/slides/TikaEval_ACNA15_allison_herceg_v2.pdf
> > -----Original Message-----
> > From: Vijaya Narayana Reddy Bhoomi Reddy [mailto:
> > vijaya.bhoomireddy@whishworks.com]
> > Sent: Thursday, April 16, 2015 7:10 AM
> > To: solr-user@lucene.apache.org
> > Subject: Re: Indexing PDF and MS Office files
> >
> > Erick,
> >
> > I tried indexing both ways - SolrJ / Tika's AutoParser and as well as
> > SolrCell's ExtractRequestHandler. Majority of the PDF and Word documents
> > are getting parsed properly and indexed into Solr. However, a minority of
> > them keep failing wither PDFParser or OfficeParser error.
> >
> > Not sure if this behaviour can be modified so that all the documents can
> be
> > indexed. The business requirement we have is to index all the documents.
> > However, if a small percentage of them fails, not sure what other ways
> > exist to index them.
> >
> > Any help please?
> >
> >
> > Thanks & Regards
> > Vijay
> >
> >
> >
> > On 15 April 2015 at 15:20, Erick Erickson <er...@gmail.com>
> wrote:
> >
> > > There's quite a discussion here:
> > > https://issues.apache.org/jira/browse/SOLR-7137
> > >
> > > But, I personally am not a huge fan of pushing all the work on to Solr,
> > in
> > > a
> > > production environment the Solr server is responsible for indexing,
> > > parsing the
> > > docs through Tika, perhaps searching etc. This doesn't scale all that
> > well.
> > >
> > > So an alternative is to use SolrJ with Tika, which is totally
> independent
> > > of
> > > what version of Tika is on the Solr server. Here's an example.
> > >
> > > http://lucidworks.com/blog/indexing-with-solrj/
> > >
> > > Best,
> > > Erick
> > >
> > > On Wed, Apr 15, 2015 at 4:46 AM, Vijaya Narayana Reddy Bhoomi Reddy
> > > <vi...@whishworks.com> wrote:
> > > > Thanks everyone for the responses. Now I am able to index PDF
> documents
> > > > successfully. I have implemented manual extraction using Tika's
> > > AutoParser
> > > > and PDF functionality is working fine. However, the error with some
> MS
> > > > office word documents still persist.
> > > >
> > > > The error message is "java.lang.IllegalArgumentException: This
> > paragraph
> > > is
> > > > not the first one in the table" which will eventually result in
> > > "Unexpected
> > > > RuntimeException from org.apache.tika.parser.microsoft.OfficeParser"
> > > >
> > > > Upon some reading, it looks like its a bug with Tika 1.5 and seems to
> > > have
> > > > been fixed with Tika 1.6 (
> > > https://issues.apache.org/jira/browse/TIKA-1251 ).
> > > > I am new to Solr / Tika and hence wondering whether I can change the
> > Tika
> > > > library alone to v1.6 without impacting any of the libraries within
> > Solr
> > > > 4.10.2? Please let me know your response and how to get away with
> this
> > > > issue.
> > > >
> > > > Many thanks in advance.
> > > >
> > > > Thanks & Regards
> > > > Vijay
> > > >
> > > >
> > > > On 15 April 2015 at 05:14, Shyam R <sh...@gmail.com> wrote:
> > > >
> > > >> Vijay,
> > > >>
> > > >> You could try different excel files with different formats to rule
> out
> > > the
> > > >> issue is with TIKA version being used.
> > > >>
> > > >> Thanks
> > > >> Murthy
> > > >>
> > > >> On Wed, Apr 15, 2015 at 9:35 AM, Terry Rhodes <trhodes314@gmail.com
> >
> > > >> wrote:
> > > >>
> > > >> > Perhaps the PDF is protected and the content can not be extracted?
> > > >> >
> > > >> > i have an unverified suspicion that the tika shipped with solr
> > 4.10.2
> > > may
> > > >> > not support some/all office 2013 document formats.
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > On 4/14/2015 8:18 PM, Jack Krupansky wrote:
> > > >> >
> > > >> >> Try doing a manual extraction request directly to Solr (not via
> > > SolrJ)
> > > >> and
> > > >> >> use the extractOnly option to see if the content is actually
> > > extracted.
> > > >> >>
> > > >> >> See:
> > > >> >> https://cwiki.apache.org/confluence/display/solr/
> > > >> >> Uploading+Data+with+Solr+Cell+using+Apache+Tika
> > > >> >>
> > > >> >> Also, some PDF files actually have the content as a bitmap image,
> > so
> > > no
> > > >> >> text is extracted.
> > > >> >>
> > > >> >>
> > > >> >> -- Jack Krupansky
> > > >> >>
> > > >> >> On Tue, Apr 14, 2015 at 10:57 AM, Vijaya Narayana Reddy Bhoomi
> > Reddy
> > > <
> > > >> >> vijaya.bhoomireddy@whishworks.com> wrote:
> > > >> >>
> > > >> >> Hi,
> > > >> >>>
> > > >> >>> I am trying to index PDF and Microsoft Office files (.doc,
> .docx,
> > > .ppt,
> > > >> >>> .pptx, .xlx, and .xlx) files into Solr. I am facing the
> following
> > > >> issues.
> > > >> >>> Request to please let me know what is going wrong with the
> > indexing
> > > >> >>> process.
> > > >> >>>
> > > >> >>> I am using solr 4.10.2 and using the default example server
> > > >> configuration
> > > >> >>> that comes with Solr distribution.
> > > >> >>>
> > > >> >>> PDF Files - Indexing as such works fine, but when I query using
> > *.*
> > > in
> > > >> >>> the
> > > >> >>> Solr Query console, metadata information is displayed properly.
> > > >> However,
> > > >> >>> the PDF content field is empty. This is happening for all PDF
> > files
> > > I
> > > >> >>> have
> > > >> >>> tried. I have tried with some proprietary files, PDF eBooks etc.
> > > >> Whatever
> > > >> >>> be the PDF file, content is not being displayed.
> > > >> >>>
> > > >> >>> MS Office files - For some office files, everything works
> perfect
> > > and
> > > >> >>> the
> > > >> >>> extracted content is visible in the query console. However, for
> > > >> others, I
> > > >> >>> see the below error message during the indexing process.
> > > >> >>>
> > > >> >>> *Exception in thread "main"
> > > >> >>>
> > > org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
> > > >> >>> org.apache.tika.exception.TikaException: Unexpected
> > RuntimeException
> > > >> >>> from
> > > >> >>> org.apache.tika.parser.microsoft.OfficeParser*
> > > >> >>>
> > > >> >>>
> > > >> >>> I am using SolrJ to index the documents and below is the code
> > > snippet
> > > >> >>> related to indexing. Please let me know where the issue is
> > > occurring.
> > > >> >>>
> > > >> >>> static String solrServerURL = "
> > > >> >>> http://localhost:8983/solr";
> > > >> >>> static SolrServer solrServer = new
> HttpSolrServer(solrServerURL);
> > > >> >>> static ContentStreamUpdateRequest
> > > indexingReq
> > > >> =
> > > >> >>> new
> > > >> >>>
> > > >> >>> ContentStreamUpdateRequest("/update/extract");
> > > >> >>>
> > > >> >>> indexingReq.addFile(file, fileType);
> > > >> >>> indexingReq.setParam("literal.id", literalId);
> > > >> >>> indexingReq.setParam("uprefix", "attr_");
> > > >> >>> indexingReq.setParam("fmap.content", "content");
> > > >> >>> indexingReq.setParam("literal.fileurl", fileURL);
> > > >> >>> indexingReq.setAction(AbstractUpdateRequest.ACTION.COMMIT, true,
> > > true);
> > > >> >>> solrServer.request(indexingReq);
> > > >> >>>
> > > >> >>> Thanks & Regards
> > > >> >>> Vijay
> > > >> >>>
> > > >> >>> --
> > > >> >>> The contents of this e-mail are confidential and for the
> exclusive
> > > use
> > > >> of
> > > >> >>> the intended recipient. If you receive this e-mail in error
> please
> > > >> delete
> > > >> >>> it from your system immediately and notify us either by e-mail
> or
> > > >> >>> telephone. You should not copy, forward or otherwise disclose
> the
> > > >> content
> > > >> >>> of the e-mail. The views expressed in this communication may not
> > > >> >>> necessarily be the view held by WHISHWORKS.
> > > >> >>>
> > > >> >>>
> > > >> >
> > > >>
> > > >>
> > > >> --
> > > >> Ph: 9845704792
> > > >>
> > > >
> > > > --
> > > > The contents of this e-mail are confidential and for the exclusive
> use
> > of
> > > > the intended recipient. If you receive this e-mail in error please
> > delete
> > > > it from your system immediately and notify us either by e-mail or
> > > > telephone. You should not copy, forward or otherwise disclose the
> > content
> > > > of the e-mail. The views expressed in this communication may not
> > > > necessarily be the view held by WHISHWORKS.
> > >
> >
> > --
> > The contents of this e-mail are confidential and for the exclusive use of
> > the intended recipient. If you receive this e-mail in error please delete
> > it from your system immediately and notify us either by e-mail or
> > telephone. You should not copy, forward or otherwise disclose the content
> > of the e-mail. The views expressed in this communication may not
> > necessarily be the view held by WHISHWORKS.
> >
>
> --
> The contents of this e-mail are confidential and for the exclusive use of
> the intended recipient. If you receive this e-mail in error please delete
> it from your system immediately and notify us either by e-mail or
> telephone. You should not copy, forward or otherwise disclose the content
> of the e-mail. The views expressed in this communication may not
> necessarily be the view held by WHISHWORKS.
>
--
The contents of this e-mail are confidential and for the exclusive use of
the intended recipient. If you receive this e-mail in error please delete
it from your system immediately and notify us either by e-mail or
telephone. You should not copy, forward or otherwise disclose the content
of the e-mail. The views expressed in this communication may not
necessarily be the view held by WHISHWORKS.
RE: Indexing PDF and MS Office files
Posted by "Allison, Timothy B." <ta...@mitre.org>.
This sounds like a Tika issue, let's move discussion to that list.
If you are still having problems after you upgrade to Tika 1.8, please at least submit the stack traces (if you can) to the Tika jira. We may be able to find a document that triggers that stack trace in govdocs1 or the slice of CommonCrawl that Julien Nioche contributed to our eval effort.
Tika is not perfect and it will fail on some files, but we are always working to improve it.
Best,
Tim
-----Original Message-----
From: Vijaya Narayana Reddy Bhoomi Reddy [mailto:vijaya.bhoomireddy@whishworks.com]
Sent: Thursday, April 16, 2015 7:44 AM
To: solr-user@lucene.apache.org
Subject: Re: Indexing PDF and MS Office files
Thanks Allison.
I tried with the mentioned changes. But still no luck. I am using the code
from lucidworks site provided by Erick and now included the changes
mentioned by you. But still the issue persists with a small percentage of
documents (both PDF and MS Office documents) failing. Unfortunately, these
documents are proprietary and client-confidential and hence I am not sure
whether they can be uploaded into Jira.
These files normally open in Adobe Reader and MS Office tools.
Thanks & Regards
Vijay
On 16 April 2015 at 12:33, Allison, Timothy B. <ta...@mitre.org> wrote:
> I entirely agree with Erick -- it is best to isolate Tika in its own jvm
> if you can -- bad things can happen if you don't [1] [2].
>
> Erick's blog on SolrJ is fantastic. If you want to have Tika parse
> embedded documents/attachments, make sure to set the parser in the
> ParseContext before parsing:
>
> ParseContext context = new ParseContext();
> //add this line:
> context.set(Parser.class, _autoParser)
> InputStream input = new FileInputStream(file);
>
> Tika 1.8 is soon to be released. If that doesn't fix your problems,
> please submit stacktraces (and docs, if possible) to the Tika jira, and
> we'll try to make the fixes.
>
> Cheers,
>
> Tim
>
> [1]
> http://events.linuxfoundation.org/sites/events/files/slides/1s_and_0s_1.pdf
> [2]
> http://events.linuxfoundation.org/sites/events/files/slides/TikaEval_ACNA15_allison_herceg_v2.pdf
> -----Original Message-----
> From: Vijaya Narayana Reddy Bhoomi Reddy [mailto:
> vijaya.bhoomireddy@whishworks.com]
> Sent: Thursday, April 16, 2015 7:10 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Indexing PDF and MS Office files
>
> Erick,
>
> I tried indexing both ways - SolrJ / Tika's AutoParser and as well as
> SolrCell's ExtractRequestHandler. Majority of the PDF and Word documents
> are getting parsed properly and indexed into Solr. However, a minority of
> them keep failing wither PDFParser or OfficeParser error.
>
> Not sure if this behaviour can be modified so that all the documents can be
> indexed. The business requirement we have is to index all the documents.
> However, if a small percentage of them fails, not sure what other ways
> exist to index them.
>
> Any help please?
>
>
> Thanks & Regards
> Vijay
>
>
>
> On 15 April 2015 at 15:20, Erick Erickson <er...@gmail.com> wrote:
>
> > There's quite a discussion here:
> > https://issues.apache.org/jira/browse/SOLR-7137
> >
> > But, I personally am not a huge fan of pushing all the work on to Solr,
> in
> > a
> > production environment the Solr server is responsible for indexing,
> > parsing the
> > docs through Tika, perhaps searching etc. This doesn't scale all that
> well.
> >
> > So an alternative is to use SolrJ with Tika, which is totally independent
> > of
> > what version of Tika is on the Solr server. Here's an example.
> >
> > http://lucidworks.com/blog/indexing-with-solrj/
> >
> > Best,
> > Erick
> >
> > On Wed, Apr 15, 2015 at 4:46 AM, Vijaya Narayana Reddy Bhoomi Reddy
> > <vi...@whishworks.com> wrote:
> > > Thanks everyone for the responses. Now I am able to index PDF documents
> > > successfully. I have implemented manual extraction using Tika's
> > AutoParser
> > > and PDF functionality is working fine. However, the error with some MS
> > > office word documents still persist.
> > >
> > > The error message is "java.lang.IllegalArgumentException: This
> paragraph
> > is
> > > not the first one in the table" which will eventually result in
> > "Unexpected
> > > RuntimeException from org.apache.tika.parser.microsoft.OfficeParser"
> > >
> > > Upon some reading, it looks like its a bug with Tika 1.5 and seems to
> > have
> > > been fixed with Tika 1.6 (
> > https://issues.apache.org/jira/browse/TIKA-1251 ).
> > > I am new to Solr / Tika and hence wondering whether I can change the
> Tika
> > > library alone to v1.6 without impacting any of the libraries within
> Solr
> > > 4.10.2? Please let me know your response and how to get away with this
> > > issue.
> > >
> > > Many thanks in advance.
> > >
> > > Thanks & Regards
> > > Vijay
> > >
> > >
> > > On 15 April 2015 at 05:14, Shyam R <sh...@gmail.com> wrote:
> > >
> > >> Vijay,
> > >>
> > >> You could try different excel files with different formats to rule out
> > the
> > >> issue is with TIKA version being used.
> > >>
> > >> Thanks
> > >> Murthy
> > >>
> > >> On Wed, Apr 15, 2015 at 9:35 AM, Terry Rhodes <tr...@gmail.com>
> > >> wrote:
> > >>
> > >> > Perhaps the PDF is protected and the content can not be extracted?
> > >> >
> > >> > i have an unverified suspicion that the tika shipped with solr
> 4.10.2
> > may
> > >> > not support some/all office 2013 document formats.
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > On 4/14/2015 8:18 PM, Jack Krupansky wrote:
> > >> >
> > >> >> Try doing a manual extraction request directly to Solr (not via
> > SolrJ)
> > >> and
> > >> >> use the extractOnly option to see if the content is actually
> > extracted.
> > >> >>
> > >> >> See:
> > >> >> https://cwiki.apache.org/confluence/display/solr/
> > >> >> Uploading+Data+with+Solr+Cell+using+Apache+Tika
> > >> >>
> > >> >> Also, some PDF files actually have the content as a bitmap image,
> so
> > no
> > >> >> text is extracted.
> > >> >>
> > >> >>
> > >> >> -- Jack Krupansky
> > >> >>
> > >> >> On Tue, Apr 14, 2015 at 10:57 AM, Vijaya Narayana Reddy Bhoomi
> Reddy
> > <
> > >> >> vijaya.bhoomireddy@whishworks.com> wrote:
> > >> >>
> > >> >> Hi,
> > >> >>>
> > >> >>> I am trying to index PDF and Microsoft Office files (.doc, .docx,
> > .ppt,
> > >> >>> .pptx, .xlx, and .xlx) files into Solr. I am facing the following
> > >> issues.
> > >> >>> Request to please let me know what is going wrong with the
> indexing
> > >> >>> process.
> > >> >>>
> > >> >>> I am using solr 4.10.2 and using the default example server
> > >> configuration
> > >> >>> that comes with Solr distribution.
> > >> >>>
> > >> >>> PDF Files - Indexing as such works fine, but when I query using
> *.*
> > in
> > >> >>> the
> > >> >>> Solr Query console, metadata information is displayed properly.
> > >> However,
> > >> >>> the PDF content field is empty. This is happening for all PDF
> files
> > I
> > >> >>> have
> > >> >>> tried. I have tried with some proprietary files, PDF eBooks etc.
> > >> Whatever
> > >> >>> be the PDF file, content is not being displayed.
> > >> >>>
> > >> >>> MS Office files - For some office files, everything works perfect
> > and
> > >> >>> the
> > >> >>> extracted content is visible in the query console. However, for
> > >> others, I
> > >> >>> see the below error message during the indexing process.
> > >> >>>
> > >> >>> *Exception in thread "main"
> > >> >>>
> > org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
> > >> >>> org.apache.tika.exception.TikaException: Unexpected
> RuntimeException
> > >> >>> from
> > >> >>> org.apache.tika.parser.microsoft.OfficeParser*
> > >> >>>
> > >> >>>
> > >> >>> I am using SolrJ to index the documents and below is the code
> > snippet
> > >> >>> related to indexing. Please let me know where the issue is
> > occurring.
> > >> >>>
> > >> >>> static String solrServerURL = "
> > >> >>> http://localhost:8983/solr";
> > >> >>> static SolrServer solrServer = new HttpSolrServer(solrServerURL);
> > >> >>> static ContentStreamUpdateRequest
> > indexingReq
> > >> =
> > >> >>> new
> > >> >>>
> > >> >>> ContentStreamUpdateRequest("/update/extract");
> > >> >>>
> > >> >>> indexingReq.addFile(file, fileType);
> > >> >>> indexingReq.setParam("literal.id", literalId);
> > >> >>> indexingReq.setParam("uprefix", "attr_");
> > >> >>> indexingReq.setParam("fmap.content", "content");
> > >> >>> indexingReq.setParam("literal.fileurl", fileURL);
> > >> >>> indexingReq.setAction(AbstractUpdateRequest.ACTION.COMMIT, true,
> > true);
> > >> >>> solrServer.request(indexingReq);
> > >> >>>
> > >> >>> Thanks & Regards
> > >> >>> Vijay
> > >> >>>
> > >> >>> --
> > >> >>> The contents of this e-mail are confidential and for the exclusive
> > use
> > >> of
> > >> >>> the intended recipient. If you receive this e-mail in error please
> > >> delete
> > >> >>> it from your system immediately and notify us either by e-mail or
> > >> >>> telephone. You should not copy, forward or otherwise disclose the
> > >> content
> > >> >>> of the e-mail. The views expressed in this communication may not
> > >> >>> necessarily be the view held by WHISHWORKS.
> > >> >>>
> > >> >>>
> > >> >
> > >>
> > >>
> > >> --
> > >> Ph: 9845704792
> > >>
> > >
> > > --
> > > The contents of this e-mail are confidential and for the exclusive use
> of
> > > the intended recipient. If you receive this e-mail in error please
> delete
> > > it from your system immediately and notify us either by e-mail or
> > > telephone. You should not copy, forward or otherwise disclose the
> content
> > > of the e-mail. The views expressed in this communication may not
> > > necessarily be the view held by WHISHWORKS.
> >
>
> --
> The contents of this e-mail are confidential and for the exclusive use of
> the intended recipient. If you receive this e-mail in error please delete
> it from your system immediately and notify us either by e-mail or
> telephone. You should not copy, forward or otherwise disclose the content
> of the e-mail. The views expressed in this communication may not
> necessarily be the view held by WHISHWORKS.
>
--
The contents of this e-mail are confidential and for the exclusive use of
the intended recipient. If you receive this e-mail in error please delete
it from your system immediately and notify us either by e-mail or
telephone. You should not copy, forward or otherwise disclose the content
of the e-mail. The views expressed in this communication may not
necessarily be the view held by WHISHWORKS.
RE: Indexing PDF and MS Office files
Posted by "Allison, Timothy B." <ta...@mitre.org>.
This sounds like a Tika issue, let's move discussion to that list.
If you are still having problems after you upgrade to Tika 1.8, please at least submit the stack traces (if you can) to the Tika jira. We may be able to find a document that triggers that stack trace in govdocs1 or the slice of CommonCrawl that Julien Nioche contributed to our eval effort.
Tika is not perfect and it will fail on some files, but we are always working to improve it.
Best,
Tim
-----Original Message-----
From: Vijaya Narayana Reddy Bhoomi Reddy [mailto:vijaya.bhoomireddy@whishworks.com]
Sent: Thursday, April 16, 2015 7:44 AM
To: solr-user@lucene.apache.org
Subject: Re: Indexing PDF and MS Office files
Thanks Allison.
I tried with the mentioned changes. But still no luck. I am using the code
from lucidworks site provided by Erick and now included the changes
mentioned by you. But still the issue persists with a small percentage of
documents (both PDF and MS Office documents) failing. Unfortunately, these
documents are proprietary and client-confidential and hence I am not sure
whether they can be uploaded into Jira.
These files normally open in Adobe Reader and MS Office tools.
Thanks & Regards
Vijay
On 16 April 2015 at 12:33, Allison, Timothy B. <ta...@mitre.org> wrote:
> I entirely agree with Erick -- it is best to isolate Tika in its own jvm
> if you can -- bad things can happen if you don't [1] [2].
>
> Erick's blog on SolrJ is fantastic. If you want to have Tika parse
> embedded documents/attachments, make sure to set the parser in the
> ParseContext before parsing:
>
> ParseContext context = new ParseContext();
> //add this line:
> context.set(Parser.class, _autoParser)
> InputStream input = new FileInputStream(file);
>
> Tika 1.8 is soon to be released. If that doesn't fix your problems,
> please submit stacktraces (and docs, if possible) to the Tika jira, and
> we'll try to make the fixes.
>
> Cheers,
>
> Tim
>
> [1]
> http://events.linuxfoundation.org/sites/events/files/slides/1s_and_0s_1.pdf
> [2]
> http://events.linuxfoundation.org/sites/events/files/slides/TikaEval_ACNA15_allison_herceg_v2.pdf
> -----Original Message-----
> From: Vijaya Narayana Reddy Bhoomi Reddy [mailto:
> vijaya.bhoomireddy@whishworks.com]
> Sent: Thursday, April 16, 2015 7:10 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Indexing PDF and MS Office files
>
> Erick,
>
> I tried indexing both ways - SolrJ / Tika's AutoParser and as well as
> SolrCell's ExtractRequestHandler. Majority of the PDF and Word documents
> are getting parsed properly and indexed into Solr. However, a minority of
> them keep failing wither PDFParser or OfficeParser error.
>
> Not sure if this behaviour can be modified so that all the documents can be
> indexed. The business requirement we have is to index all the documents.
> However, if a small percentage of them fails, not sure what other ways
> exist to index them.
>
> Any help please?
>
>
> Thanks & Regards
> Vijay
>
>
>
> On 15 April 2015 at 15:20, Erick Erickson <er...@gmail.com> wrote:
>
> > There's quite a discussion here:
> > https://issues.apache.org/jira/browse/SOLR-7137
> >
> > But, I personally am not a huge fan of pushing all the work on to Solr,
> in
> > a
> > production environment the Solr server is responsible for indexing,
> > parsing the
> > docs through Tika, perhaps searching etc. This doesn't scale all that
> well.
> >
> > So an alternative is to use SolrJ with Tika, which is totally independent
> > of
> > what version of Tika is on the Solr server. Here's an example.
> >
> > http://lucidworks.com/blog/indexing-with-solrj/
> >
> > Best,
> > Erick
> >
> > On Wed, Apr 15, 2015 at 4:46 AM, Vijaya Narayana Reddy Bhoomi Reddy
> > <vi...@whishworks.com> wrote:
> > > Thanks everyone for the responses. Now I am able to index PDF documents
> > > successfully. I have implemented manual extraction using Tika's
> > AutoParser
> > > and PDF functionality is working fine. However, the error with some MS
> > > office word documents still persist.
> > >
> > > The error message is "java.lang.IllegalArgumentException: This
> paragraph
> > is
> > > not the first one in the table" which will eventually result in
> > "Unexpected
> > > RuntimeException from org.apache.tika.parser.microsoft.OfficeParser"
> > >
> > > Upon some reading, it looks like its a bug with Tika 1.5 and seems to
> > have
> > > been fixed with Tika 1.6 (
> > https://issues.apache.org/jira/browse/TIKA-1251 ).
> > > I am new to Solr / Tika and hence wondering whether I can change the
> Tika
> > > library alone to v1.6 without impacting any of the libraries within
> Solr
> > > 4.10.2? Please let me know your response and how to get away with this
> > > issue.
> > >
> > > Many thanks in advance.
> > >
> > > Thanks & Regards
> > > Vijay
> > >
> > >
> > > On 15 April 2015 at 05:14, Shyam R <sh...@gmail.com> wrote:
> > >
> > >> Vijay,
> > >>
> > >> You could try different excel files with different formats to rule out
> > the
> > >> issue is with TIKA version being used.
> > >>
> > >> Thanks
> > >> Murthy
> > >>
> > >> On Wed, Apr 15, 2015 at 9:35 AM, Terry Rhodes <tr...@gmail.com>
> > >> wrote:
> > >>
> > >> > Perhaps the PDF is protected and the content can not be extracted?
> > >> >
> > >> > i have an unverified suspicion that the tika shipped with solr
> 4.10.2
> > may
> > >> > not support some/all office 2013 document formats.
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > On 4/14/2015 8:18 PM, Jack Krupansky wrote:
> > >> >
> > >> >> Try doing a manual extraction request directly to Solr (not via
> > SolrJ)
> > >> and
> > >> >> use the extractOnly option to see if the content is actually
> > extracted.
> > >> >>
> > >> >> See:
> > >> >> https://cwiki.apache.org/confluence/display/solr/
> > >> >> Uploading+Data+with+Solr+Cell+using+Apache+Tika
> > >> >>
> > >> >> Also, some PDF files actually have the content as a bitmap image,
> so
> > no
> > >> >> text is extracted.
> > >> >>
> > >> >>
> > >> >> -- Jack Krupansky
> > >> >>
> > >> >> On Tue, Apr 14, 2015 at 10:57 AM, Vijaya Narayana Reddy Bhoomi
> Reddy
> > <
> > >> >> vijaya.bhoomireddy@whishworks.com> wrote:
> > >> >>
> > >> >> Hi,
> > >> >>>
> > >> >>> I am trying to index PDF and Microsoft Office files (.doc, .docx,
> > .ppt,
> > >> >>> .pptx, .xlx, and .xlx) files into Solr. I am facing the following
> > >> issues.
> > >> >>> Request to please let me know what is going wrong with the
> indexing
> > >> >>> process.
> > >> >>>
> > >> >>> I am using solr 4.10.2 and using the default example server
> > >> configuration
> > >> >>> that comes with Solr distribution.
> > >> >>>
> > >> >>> PDF Files - Indexing as such works fine, but when I query using
> *.*
> > in
> > >> >>> the
> > >> >>> Solr Query console, metadata information is displayed properly.
> > >> However,
> > >> >>> the PDF content field is empty. This is happening for all PDF
> files
> > I
> > >> >>> have
> > >> >>> tried. I have tried with some proprietary files, PDF eBooks etc.
> > >> Whatever
> > >> >>> be the PDF file, content is not being displayed.
> > >> >>>
> > >> >>> MS Office files - For some office files, everything works perfect
> > and
> > >> >>> the
> > >> >>> extracted content is visible in the query console. However, for
> > >> others, I
> > >> >>> see the below error message during the indexing process.
> > >> >>>
> > >> >>> *Exception in thread "main"
> > >> >>>
> > org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
> > >> >>> org.apache.tika.exception.TikaException: Unexpected
> RuntimeException
> > >> >>> from
> > >> >>> org.apache.tika.parser.microsoft.OfficeParser*
> > >> >>>
> > >> >>>
> > >> >>> I am using SolrJ to index the documents and below is the code
> > snippet
> > >> >>> related to indexing. Please let me know where the issue is
> > occurring.
> > >> >>>
> > >> >>> static String solrServerURL = "
> > >> >>> http://localhost:8983/solr";
> > >> >>> static SolrServer solrServer = new HttpSolrServer(solrServerURL);
> > >> >>> static ContentStreamUpdateRequest
> > indexingReq
> > >> =
> > >> >>> new
> > >> >>>
> > >> >>> ContentStreamUpdateRequest("/update/extract");
> > >> >>>
> > >> >>> indexingReq.addFile(file, fileType);
> > >> >>> indexingReq.setParam("literal.id", literalId);
> > >> >>> indexingReq.setParam("uprefix", "attr_");
> > >> >>> indexingReq.setParam("fmap.content", "content");
> > >> >>> indexingReq.setParam("literal.fileurl", fileURL);
> > >> >>> indexingReq.setAction(AbstractUpdateRequest.ACTION.COMMIT, true,
> > true);
> > >> >>> solrServer.request(indexingReq);
> > >> >>>
> > >> >>> Thanks & Regards
> > >> >>> Vijay
> > >> >>>
> > >> >>> --
> > >> >>> The contents of this e-mail are confidential and for the exclusive
> > use
> > >> of
> > >> >>> the intended recipient. If you receive this e-mail in error please
> > >> delete
> > >> >>> it from your system immediately and notify us either by e-mail or
> > >> >>> telephone. You should not copy, forward or otherwise disclose the
> > >> content
> > >> >>> of the e-mail. The views expressed in this communication may not
> > >> >>> necessarily be the view held by WHISHWORKS.
> > >> >>>
> > >> >>>
> > >> >
> > >>
> > >>
> > >> --
> > >> Ph: 9845704792
> > >>
> > >
> > > --
> > > The contents of this e-mail are confidential and for the exclusive use
> of
> > > the intended recipient. If you receive this e-mail in error please
> delete
> > > it from your system immediately and notify us either by e-mail or
> > > telephone. You should not copy, forward or otherwise disclose the
> content
> > > of the e-mail. The views expressed in this communication may not
> > > necessarily be the view held by WHISHWORKS.
> >
>
> --
> The contents of this e-mail are confidential and for the exclusive use of
> the intended recipient. If you receive this e-mail in error please delete
> it from your system immediately and notify us either by e-mail or
> telephone. You should not copy, forward or otherwise disclose the content
> of the e-mail. The views expressed in this communication may not
> necessarily be the view held by WHISHWORKS.
>
--
The contents of this e-mail are confidential and for the exclusive use of
the intended recipient. If you receive this e-mail in error please delete
it from your system immediately and notify us either by e-mail or
telephone. You should not copy, forward or otherwise disclose the content
of the e-mail. The views expressed in this communication may not
necessarily be the view held by WHISHWORKS.
Re: Indexing PDF and MS Office files
Posted by Vijaya Narayana Reddy Bhoomi Reddy <vi...@whishworks.com>.
Thanks Allison.
I tried with the mentioned changes. But still no luck. I am using the code
from lucidworks site provided by Erick and now included the changes
mentioned by you. But still the issue persists with a small percentage of
documents (both PDF and MS Office documents) failing. Unfortunately, these
documents are proprietary and client-confidential and hence I am not sure
whether they can be uploaded into Jira.
These files normally open in Adobe Reader and MS Office tools.
Thanks & Regards
Vijay
On 16 April 2015 at 12:33, Allison, Timothy B. <ta...@mitre.org> wrote:
> I entirely agree with Erick -- it is best to isolate Tika in its own jvm
> if you can -- bad things can happen if you don't [1] [2].
>
> Erick's blog on SolrJ is fantastic. If you want to have Tika parse
> embedded documents/attachments, make sure to set the parser in the
> ParseContext before parsing:
>
> ParseContext context = new ParseContext();
> //add this line:
> context.set(Parser.class, _autoParser)
> InputStream input = new FileInputStream(file);
>
> Tika 1.8 is soon to be released. If that doesn't fix your problems,
> please submit stacktraces (and docs, if possible) to the Tika jira, and
> we'll try to make the fixes.
>
> Cheers,
>
> Tim
>
> [1]
> http://events.linuxfoundation.org/sites/events/files/slides/1s_and_0s_1.pdf
> [2]
> http://events.linuxfoundation.org/sites/events/files/slides/TikaEval_ACNA15_allison_herceg_v2.pdf
> -----Original Message-----
> From: Vijaya Narayana Reddy Bhoomi Reddy [mailto:
> vijaya.bhoomireddy@whishworks.com]
> Sent: Thursday, April 16, 2015 7:10 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Indexing PDF and MS Office files
>
> Erick,
>
> I tried indexing both ways - SolrJ / Tika's AutoParser and as well as
> SolrCell's ExtractRequestHandler. Majority of the PDF and Word documents
> are getting parsed properly and indexed into Solr. However, a minority of
> them keep failing wither PDFParser or OfficeParser error.
>
> Not sure if this behaviour can be modified so that all the documents can be
> indexed. The business requirement we have is to index all the documents.
> However, if a small percentage of them fails, not sure what other ways
> exist to index them.
>
> Any help please?
>
>
> Thanks & Regards
> Vijay
>
>
>
> On 15 April 2015 at 15:20, Erick Erickson <er...@gmail.com> wrote:
>
> > There's quite a discussion here:
> > https://issues.apache.org/jira/browse/SOLR-7137
> >
> > But, I personally am not a huge fan of pushing all the work on to Solr,
> in
> > a
> > production environment the Solr server is responsible for indexing,
> > parsing the
> > docs through Tika, perhaps searching etc. This doesn't scale all that
> well.
> >
> > So an alternative is to use SolrJ with Tika, which is totally independent
> > of
> > what version of Tika is on the Solr server. Here's an example.
> >
> > http://lucidworks.com/blog/indexing-with-solrj/
> >
> > Best,
> > Erick
> >
> > On Wed, Apr 15, 2015 at 4:46 AM, Vijaya Narayana Reddy Bhoomi Reddy
> > <vi...@whishworks.com> wrote:
> > > Thanks everyone for the responses. Now I am able to index PDF documents
> > > successfully. I have implemented manual extraction using Tika's
> > AutoParser
> > > and PDF functionality is working fine. However, the error with some MS
> > > office word documents still persist.
> > >
> > > The error message is "java.lang.IllegalArgumentException: This
> paragraph
> > is
> > > not the first one in the table" which will eventually result in
> > "Unexpected
> > > RuntimeException from org.apache.tika.parser.microsoft.OfficeParser"
> > >
> > > Upon some reading, it looks like its a bug with Tika 1.5 and seems to
> > have
> > > been fixed with Tika 1.6 (
> > https://issues.apache.org/jira/browse/TIKA-1251 ).
> > > I am new to Solr / Tika and hence wondering whether I can change the
> Tika
> > > library alone to v1.6 without impacting any of the libraries within
> Solr
> > > 4.10.2? Please let me know your response and how to get away with this
> > > issue.
> > >
> > > Many thanks in advance.
> > >
> > > Thanks & Regards
> > > Vijay
> > >
> > >
> > > On 15 April 2015 at 05:14, Shyam R <sh...@gmail.com> wrote:
> > >
> > >> Vijay,
> > >>
> > >> You could try different excel files with different formats to rule out
> > the
> > >> issue is with TIKA version being used.
> > >>
> > >> Thanks
> > >> Murthy
> > >>
> > >> On Wed, Apr 15, 2015 at 9:35 AM, Terry Rhodes <tr...@gmail.com>
> > >> wrote:
> > >>
> > >> > Perhaps the PDF is protected and the content can not be extracted?
> > >> >
> > >> > i have an unverified suspicion that the tika shipped with solr
> 4.10.2
> > may
> > >> > not support some/all office 2013 document formats.
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > On 4/14/2015 8:18 PM, Jack Krupansky wrote:
> > >> >
> > >> >> Try doing a manual extraction request directly to Solr (not via
> > SolrJ)
> > >> and
> > >> >> use the extractOnly option to see if the content is actually
> > extracted.
> > >> >>
> > >> >> See:
> > >> >> https://cwiki.apache.org/confluence/display/solr/
> > >> >> Uploading+Data+with+Solr+Cell+using+Apache+Tika
> > >> >>
> > >> >> Also, some PDF files actually have the content as a bitmap image,
> so
> > no
> > >> >> text is extracted.
> > >> >>
> > >> >>
> > >> >> -- Jack Krupansky
> > >> >>
> > >> >> On Tue, Apr 14, 2015 at 10:57 AM, Vijaya Narayana Reddy Bhoomi
> Reddy
> > <
> > >> >> vijaya.bhoomireddy@whishworks.com> wrote:
> > >> >>
> > >> >> Hi,
> > >> >>>
> > >> >>> I am trying to index PDF and Microsoft Office files (.doc, .docx,
> > .ppt,
> > >> >>> .pptx, .xlx, and .xlx) files into Solr. I am facing the following
> > >> issues.
> > >> >>> Request to please let me know what is going wrong with the
> indexing
> > >> >>> process.
> > >> >>>
> > >> >>> I am using solr 4.10.2 and using the default example server
> > >> configuration
> > >> >>> that comes with Solr distribution.
> > >> >>>
> > >> >>> PDF Files - Indexing as such works fine, but when I query using
> *.*
> > in
> > >> >>> the
> > >> >>> Solr Query console, metadata information is displayed properly.
> > >> However,
> > >> >>> the PDF content field is empty. This is happening for all PDF
> files
> > I
> > >> >>> have
> > >> >>> tried. I have tried with some proprietary files, PDF eBooks etc.
> > >> Whatever
> > >> >>> be the PDF file, content is not being displayed.
> > >> >>>
> > >> >>> MS Office files - For some office files, everything works perfect
> > and
> > >> >>> the
> > >> >>> extracted content is visible in the query console. However, for
> > >> others, I
> > >> >>> see the below error message during the indexing process.
> > >> >>>
> > >> >>> *Exception in thread "main"
> > >> >>>
> > org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
> > >> >>> org.apache.tika.exception.TikaException: Unexpected
> RuntimeException
> > >> >>> from
> > >> >>> org.apache.tika.parser.microsoft.OfficeParser*
> > >> >>>
> > >> >>>
> > >> >>> I am using SolrJ to index the documents and below is the code
> > snippet
> > >> >>> related to indexing. Please let me know where the issue is
> > occurring.
> > >> >>>
> > >> >>> static String solrServerURL = "
> > >> >>> http://localhost:8983/solr";
> > >> >>> static SolrServer solrServer = new HttpSolrServer(solrServerURL);
> > >> >>> static ContentStreamUpdateRequest
> > indexingReq
> > >> =
> > >> >>> new
> > >> >>>
> > >> >>> ContentStreamUpdateRequest("/update/extract");
> > >> >>>
> > >> >>> indexingReq.addFile(file, fileType);
> > >> >>> indexingReq.setParam("literal.id", literalId);
> > >> >>> indexingReq.setParam("uprefix", "attr_");
> > >> >>> indexingReq.setParam("fmap.content", "content");
> > >> >>> indexingReq.setParam("literal.fileurl", fileURL);
> > >> >>> indexingReq.setAction(AbstractUpdateRequest.ACTION.COMMIT, true,
> > true);
> > >> >>> solrServer.request(indexingReq);
> > >> >>>
> > >> >>> Thanks & Regards
> > >> >>> Vijay
> > >> >>>
> > >> >>> --
> > >> >>> The contents of this e-mail are confidential and for the exclusive
> > use
> > >> of
> > >> >>> the intended recipient. If you receive this e-mail in error please
> > >> delete
> > >> >>> it from your system immediately and notify us either by e-mail or
> > >> >>> telephone. You should not copy, forward or otherwise disclose the
> > >> content
> > >> >>> of the e-mail. The views expressed in this communication may not
> > >> >>> necessarily be the view held by WHISHWORKS.
> > >> >>>
> > >> >>>
> > >> >
> > >>
> > >>
> > >> --
> > >> Ph: 9845704792
> > >>
> > >
> > > --
> > > The contents of this e-mail are confidential and for the exclusive use
> of
> > > the intended recipient. If you receive this e-mail in error please
> delete
> > > it from your system immediately and notify us either by e-mail or
> > > telephone. You should not copy, forward or otherwise disclose the
> content
> > > of the e-mail. The views expressed in this communication may not
> > > necessarily be the view held by WHISHWORKS.
> >
>
> --
> The contents of this e-mail are confidential and for the exclusive use of
> the intended recipient. If you receive this e-mail in error please delete
> it from your system immediately and notify us either by e-mail or
> telephone. You should not copy, forward or otherwise disclose the content
> of the e-mail. The views expressed in this communication may not
> necessarily be the view held by WHISHWORKS.
>
--
The contents of this e-mail are confidential and for the exclusive use of
the intended recipient. If you receive this e-mail in error please delete
it from your system immediately and notify us either by e-mail or
telephone. You should not copy, forward or otherwise disclose the content
of the e-mail. The views expressed in this communication may not
necessarily be the view held by WHISHWORKS.
RE: Indexing PDF and MS Office files
Posted by "Allison, Timothy B." <ta...@mitre.org>.
I entirely agree with Erick -- it is best to isolate Tika in its own jvm if you can -- bad things can happen if you don't [1] [2].
Erick's blog on SolrJ is fantastic. If you want to have Tika parse embedded documents/attachments, make sure to set the parser in the ParseContext before parsing:
ParseContext context = new ParseContext();
//add this line:
context.set(Parser.class, _autoParser)
InputStream input = new FileInputStream(file);
Tika 1.8 is soon to be released. If that doesn't fix your problems, please submit stacktraces (and docs, if possible) to the Tika jira, and we'll try to make the fixes.
Cheers,
Tim
[1] http://events.linuxfoundation.org/sites/events/files/slides/1s_and_0s_1.pdf
[2] http://events.linuxfoundation.org/sites/events/files/slides/TikaEval_ACNA15_allison_herceg_v2.pdf
-----Original Message-----
From: Vijaya Narayana Reddy Bhoomi Reddy [mailto:vijaya.bhoomireddy@whishworks.com]
Sent: Thursday, April 16, 2015 7:10 AM
To: solr-user@lucene.apache.org
Subject: Re: Indexing PDF and MS Office files
Erick,
I tried indexing both ways - SolrJ / Tika's AutoParser and as well as
SolrCell's ExtractRequestHandler. Majority of the PDF and Word documents
are getting parsed properly and indexed into Solr. However, a minority of
them keep failing wither PDFParser or OfficeParser error.
Not sure if this behaviour can be modified so that all the documents can be
indexed. The business requirement we have is to index all the documents.
However, if a small percentage of them fails, not sure what other ways
exist to index them.
Any help please?
Thanks & Regards
Vijay
On 15 April 2015 at 15:20, Erick Erickson <er...@gmail.com> wrote:
> There's quite a discussion here:
> https://issues.apache.org/jira/browse/SOLR-7137
>
> But, I personally am not a huge fan of pushing all the work on to Solr, in
> a
> production environment the Solr server is responsible for indexing,
> parsing the
> docs through Tika, perhaps searching etc. This doesn't scale all that well.
>
> So an alternative is to use SolrJ with Tika, which is totally independent
> of
> what version of Tika is on the Solr server. Here's an example.
>
> http://lucidworks.com/blog/indexing-with-solrj/
>
> Best,
> Erick
>
> On Wed, Apr 15, 2015 at 4:46 AM, Vijaya Narayana Reddy Bhoomi Reddy
> <vi...@whishworks.com> wrote:
> > Thanks everyone for the responses. Now I am able to index PDF documents
> > successfully. I have implemented manual extraction using Tika's
> AutoParser
> > and PDF functionality is working fine. However, the error with some MS
> > office word documents still persist.
> >
> > The error message is "java.lang.IllegalArgumentException: This paragraph
> is
> > not the first one in the table" which will eventually result in
> "Unexpected
> > RuntimeException from org.apache.tika.parser.microsoft.OfficeParser"
> >
> > Upon some reading, it looks like its a bug with Tika 1.5 and seems to
> have
> > been fixed with Tika 1.6 (
> https://issues.apache.org/jira/browse/TIKA-1251 ).
> > I am new to Solr / Tika and hence wondering whether I can change the Tika
> > library alone to v1.6 without impacting any of the libraries within Solr
> > 4.10.2? Please let me know your response and how to get away with this
> > issue.
> >
> > Many thanks in advance.
> >
> > Thanks & Regards
> > Vijay
> >
> >
> > On 15 April 2015 at 05:14, Shyam R <sh...@gmail.com> wrote:
> >
> >> Vijay,
> >>
> >> You could try different excel files with different formats to rule out
> the
> >> issue is with TIKA version being used.
> >>
> >> Thanks
> >> Murthy
> >>
> >> On Wed, Apr 15, 2015 at 9:35 AM, Terry Rhodes <tr...@gmail.com>
> >> wrote:
> >>
> >> > Perhaps the PDF is protected and the content can not be extracted?
> >> >
> >> > i have an unverified suspicion that the tika shipped with solr 4.10.2
> may
> >> > not support some/all office 2013 document formats.
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On 4/14/2015 8:18 PM, Jack Krupansky wrote:
> >> >
> >> >> Try doing a manual extraction request directly to Solr (not via
> SolrJ)
> >> and
> >> >> use the extractOnly option to see if the content is actually
> extracted.
> >> >>
> >> >> See:
> >> >> https://cwiki.apache.org/confluence/display/solr/
> >> >> Uploading+Data+with+Solr+Cell+using+Apache+Tika
> >> >>
> >> >> Also, some PDF files actually have the content as a bitmap image, so
> no
> >> >> text is extracted.
> >> >>
> >> >>
> >> >> -- Jack Krupansky
> >> >>
> >> >> On Tue, Apr 14, 2015 at 10:57 AM, Vijaya Narayana Reddy Bhoomi Reddy
> <
> >> >> vijaya.bhoomireddy@whishworks.com> wrote:
> >> >>
> >> >> Hi,
> >> >>>
> >> >>> I am trying to index PDF and Microsoft Office files (.doc, .docx,
> .ppt,
> >> >>> .pptx, .xlx, and .xlx) files into Solr. I am facing the following
> >> issues.
> >> >>> Request to please let me know what is going wrong with the indexing
> >> >>> process.
> >> >>>
> >> >>> I am using solr 4.10.2 and using the default example server
> >> configuration
> >> >>> that comes with Solr distribution.
> >> >>>
> >> >>> PDF Files - Indexing as such works fine, but when I query using *.*
> in
> >> >>> the
> >> >>> Solr Query console, metadata information is displayed properly.
> >> However,
> >> >>> the PDF content field is empty. This is happening for all PDF files
> I
> >> >>> have
> >> >>> tried. I have tried with some proprietary files, PDF eBooks etc.
> >> Whatever
> >> >>> be the PDF file, content is not being displayed.
> >> >>>
> >> >>> MS Office files - For some office files, everything works perfect
> and
> >> >>> the
> >> >>> extracted content is visible in the query console. However, for
> >> others, I
> >> >>> see the below error message during the indexing process.
> >> >>>
> >> >>> *Exception in thread "main"
> >> >>>
> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
> >> >>> org.apache.tika.exception.TikaException: Unexpected RuntimeException
> >> >>> from
> >> >>> org.apache.tika.parser.microsoft.OfficeParser*
> >> >>>
> >> >>>
> >> >>> I am using SolrJ to index the documents and below is the code
> snippet
> >> >>> related to indexing. Please let me know where the issue is
> occurring.
> >> >>>
> >> >>> static String solrServerURL = "
> >> >>> http://localhost:8983/solr";
> >> >>> static SolrServer solrServer = new HttpSolrServer(solrServerURL);
> >> >>> static ContentStreamUpdateRequest
> indexingReq
> >> =
> >> >>> new
> >> >>>
> >> >>> ContentStreamUpdateRequest("/update/extract");
> >> >>>
> >> >>> indexingReq.addFile(file, fileType);
> >> >>> indexingReq.setParam("literal.id", literalId);
> >> >>> indexingReq.setParam("uprefix", "attr_");
> >> >>> indexingReq.setParam("fmap.content", "content");
> >> >>> indexingReq.setParam("literal.fileurl", fileURL);
> >> >>> indexingReq.setAction(AbstractUpdateRequest.ACTION.COMMIT, true,
> true);
> >> >>> solrServer.request(indexingReq);
> >> >>>
> >> >>> Thanks & Regards
> >> >>> Vijay
> >> >>>
> >> >>> --
> >> >>> The contents of this e-mail are confidential and for the exclusive
> use
> >> of
> >> >>> the intended recipient. If you receive this e-mail in error please
> >> delete
> >> >>> it from your system immediately and notify us either by e-mail or
> >> >>> telephone. You should not copy, forward or otherwise disclose the
> >> content
> >> >>> of the e-mail. The views expressed in this communication may not
> >> >>> necessarily be the view held by WHISHWORKS.
> >> >>>
> >> >>>
> >> >
> >>
> >>
> >> --
> >> Ph: 9845704792
> >>
> >
> > --
> > The contents of this e-mail are confidential and for the exclusive use of
> > the intended recipient. If you receive this e-mail in error please delete
> > it from your system immediately and notify us either by e-mail or
> > telephone. You should not copy, forward or otherwise disclose the content
> > of the e-mail. The views expressed in this communication may not
> > necessarily be the view held by WHISHWORKS.
>
--
The contents of this e-mail are confidential and for the exclusive use of
the intended recipient. If you receive this e-mail in error please delete
it from your system immediately and notify us either by e-mail or
telephone. You should not copy, forward or otherwise disclose the content
of the e-mail. The views expressed in this communication may not
necessarily be the view held by WHISHWORKS.
Re: Indexing PDF and MS Office files
Posted by Vijaya Narayana Reddy Bhoomi Reddy <vi...@whishworks.com>.
Erick,
I tried indexing both ways - SolrJ / Tika's AutoParser and as well as
SolrCell's ExtractRequestHandler. Majority of the PDF and Word documents
are getting parsed properly and indexed into Solr. However, a minority of
them keep failing wither PDFParser or OfficeParser error.
Not sure if this behaviour can be modified so that all the documents can be
indexed. The business requirement we have is to index all the documents.
However, if a small percentage of them fails, not sure what other ways
exist to index them.
Any help please?
Thanks & Regards
Vijay
On 15 April 2015 at 15:20, Erick Erickson <er...@gmail.com> wrote:
> There's quite a discussion here:
> https://issues.apache.org/jira/browse/SOLR-7137
>
> But, I personally am not a huge fan of pushing all the work on to Solr, in
> a
> production environment the Solr server is responsible for indexing,
> parsing the
> docs through Tika, perhaps searching etc. This doesn't scale all that well.
>
> So an alternative is to use SolrJ with Tika, which is totally independent
> of
> what version of Tika is on the Solr server. Here's an example.
>
> http://lucidworks.com/blog/indexing-with-solrj/
>
> Best,
> Erick
>
> On Wed, Apr 15, 2015 at 4:46 AM, Vijaya Narayana Reddy Bhoomi Reddy
> <vi...@whishworks.com> wrote:
> > Thanks everyone for the responses. Now I am able to index PDF documents
> > successfully. I have implemented manual extraction using Tika's
> AutoParser
> > and PDF functionality is working fine. However, the error with some MS
> > office word documents still persist.
> >
> > The error message is "java.lang.IllegalArgumentException: This paragraph
> is
> > not the first one in the table" which will eventually result in
> "Unexpected
> > RuntimeException from org.apache.tika.parser.microsoft.OfficeParser"
> >
> > Upon some reading, it looks like its a bug with Tika 1.5 and seems to
> have
> > been fixed with Tika 1.6 (
> https://issues.apache.org/jira/browse/TIKA-1251 ).
> > I am new to Solr / Tika and hence wondering whether I can change the Tika
> > library alone to v1.6 without impacting any of the libraries within Solr
> > 4.10.2? Please let me know your response and how to get away with this
> > issue.
> >
> > Many thanks in advance.
> >
> > Thanks & Regards
> > Vijay
> >
> >
> > On 15 April 2015 at 05:14, Shyam R <sh...@gmail.com> wrote:
> >
> >> Vijay,
> >>
> >> You could try different excel files with different formats to rule out
> the
> >> issue is with TIKA version being used.
> >>
> >> Thanks
> >> Murthy
> >>
> >> On Wed, Apr 15, 2015 at 9:35 AM, Terry Rhodes <tr...@gmail.com>
> >> wrote:
> >>
> >> > Perhaps the PDF is protected and the content can not be extracted?
> >> >
> >> > i have an unverified suspicion that the tika shipped with solr 4.10.2
> may
> >> > not support some/all office 2013 document formats.
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On 4/14/2015 8:18 PM, Jack Krupansky wrote:
> >> >
> >> >> Try doing a manual extraction request directly to Solr (not via
> SolrJ)
> >> and
> >> >> use the extractOnly option to see if the content is actually
> extracted.
> >> >>
> >> >> See:
> >> >> https://cwiki.apache.org/confluence/display/solr/
> >> >> Uploading+Data+with+Solr+Cell+using+Apache+Tika
> >> >>
> >> >> Also, some PDF files actually have the content as a bitmap image, so
> no
> >> >> text is extracted.
> >> >>
> >> >>
> >> >> -- Jack Krupansky
> >> >>
> >> >> On Tue, Apr 14, 2015 at 10:57 AM, Vijaya Narayana Reddy Bhoomi Reddy
> <
> >> >> vijaya.bhoomireddy@whishworks.com> wrote:
> >> >>
> >> >> Hi,
> >> >>>
> >> >>> I am trying to index PDF and Microsoft Office files (.doc, .docx,
> .ppt,
> >> >>> .pptx, .xlx, and .xlx) files into Solr. I am facing the following
> >> issues.
> >> >>> Request to please let me know what is going wrong with the indexing
> >> >>> process.
> >> >>>
> >> >>> I am using solr 4.10.2 and using the default example server
> >> configuration
> >> >>> that comes with Solr distribution.
> >> >>>
> >> >>> PDF Files - Indexing as such works fine, but when I query using *.*
> in
> >> >>> the
> >> >>> Solr Query console, metadata information is displayed properly.
> >> However,
> >> >>> the PDF content field is empty. This is happening for all PDF files
> I
> >> >>> have
> >> >>> tried. I have tried with some proprietary files, PDF eBooks etc.
> >> Whatever
> >> >>> be the PDF file, content is not being displayed.
> >> >>>
> >> >>> MS Office files - For some office files, everything works perfect
> and
> >> >>> the
> >> >>> extracted content is visible in the query console. However, for
> >> others, I
> >> >>> see the below error message during the indexing process.
> >> >>>
> >> >>> *Exception in thread "main"
> >> >>>
> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
> >> >>> org.apache.tika.exception.TikaException: Unexpected RuntimeException
> >> >>> from
> >> >>> org.apache.tika.parser.microsoft.OfficeParser*
> >> >>>
> >> >>>
> >> >>> I am using SolrJ to index the documents and below is the code
> snippet
> >> >>> related to indexing. Please let me know where the issue is
> occurring.
> >> >>>
> >> >>> static String solrServerURL = "
> >> >>> http://localhost:8983/solr";
> >> >>> static SolrServer solrServer = new HttpSolrServer(solrServerURL);
> >> >>> static ContentStreamUpdateRequest
> indexingReq
> >> =
> >> >>> new
> >> >>>
> >> >>> ContentStreamUpdateRequest("/update/extract");
> >> >>>
> >> >>> indexingReq.addFile(file, fileType);
> >> >>> indexingReq.setParam("literal.id", literalId);
> >> >>> indexingReq.setParam("uprefix", "attr_");
> >> >>> indexingReq.setParam("fmap.content", "content");
> >> >>> indexingReq.setParam("literal.fileurl", fileURL);
> >> >>> indexingReq.setAction(AbstractUpdateRequest.ACTION.COMMIT, true,
> true);
> >> >>> solrServer.request(indexingReq);
> >> >>>
> >> >>> Thanks & Regards
> >> >>> Vijay
> >> >>>
> >> >>> --
> >> >>> The contents of this e-mail are confidential and for the exclusive
> use
> >> of
> >> >>> the intended recipient. If you receive this e-mail in error please
> >> delete
> >> >>> it from your system immediately and notify us either by e-mail or
> >> >>> telephone. You should not copy, forward or otherwise disclose the
> >> content
> >> >>> of the e-mail. The views expressed in this communication may not
> >> >>> necessarily be the view held by WHISHWORKS.
> >> >>>
> >> >>>
> >> >
> >>
> >>
> >> --
> >> Ph: 9845704792
> >>
> >
> > --
> > The contents of this e-mail are confidential and for the exclusive use of
> > the intended recipient. If you receive this e-mail in error please delete
> > it from your system immediately and notify us either by e-mail or
> > telephone. You should not copy, forward or otherwise disclose the content
> > of the e-mail. The views expressed in this communication may not
> > necessarily be the view held by WHISHWORKS.
>
--
The contents of this e-mail are confidential and for the exclusive use of
the intended recipient. If you receive this e-mail in error please delete
it from your system immediately and notify us either by e-mail or
telephone. You should not copy, forward or otherwise disclose the content
of the e-mail. The views expressed in this communication may not
necessarily be the view held by WHISHWORKS.
Re: Indexing PDF and MS Office files
Posted by Erick Erickson <er...@gmail.com>.
There's quite a discussion here: https://issues.apache.org/jira/browse/SOLR-7137
But, I personally am not a huge fan of pushing all the work on to Solr, in a
production environment the Solr server is responsible for indexing, parsing the
docs through Tika, perhaps searching etc. This doesn't scale all that well.
So an alternative is to use SolrJ with Tika, which is totally independent of
what version of Tika is on the Solr server. Here's an example.
http://lucidworks.com/blog/indexing-with-solrj/
Best,
Erick
On Wed, Apr 15, 2015 at 4:46 AM, Vijaya Narayana Reddy Bhoomi Reddy
<vi...@whishworks.com> wrote:
> Thanks everyone for the responses. Now I am able to index PDF documents
> successfully. I have implemented manual extraction using Tika's AutoParser
> and PDF functionality is working fine. However, the error with some MS
> office word documents still persist.
>
> The error message is "java.lang.IllegalArgumentException: This paragraph is
> not the first one in the table" which will eventually result in "Unexpected
> RuntimeException from org.apache.tika.parser.microsoft.OfficeParser"
>
> Upon some reading, it looks like its a bug with Tika 1.5 and seems to have
> been fixed with Tika 1.6 ( https://issues.apache.org/jira/browse/TIKA-1251 ).
> I am new to Solr / Tika and hence wondering whether I can change the Tika
> library alone to v1.6 without impacting any of the libraries within Solr
> 4.10.2? Please let me know your response and how to get away with this
> issue.
>
> Many thanks in advance.
>
> Thanks & Regards
> Vijay
>
>
> On 15 April 2015 at 05:14, Shyam R <sh...@gmail.com> wrote:
>
>> Vijay,
>>
>> You could try different excel files with different formats to rule out the
>> issue is with TIKA version being used.
>>
>> Thanks
>> Murthy
>>
>> On Wed, Apr 15, 2015 at 9:35 AM, Terry Rhodes <tr...@gmail.com>
>> wrote:
>>
>> > Perhaps the PDF is protected and the content can not be extracted?
>> >
>> > i have an unverified suspicion that the tika shipped with solr 4.10.2 may
>> > not support some/all office 2013 document formats.
>> >
>> >
>> >
>> >
>> >
>> > On 4/14/2015 8:18 PM, Jack Krupansky wrote:
>> >
>> >> Try doing a manual extraction request directly to Solr (not via SolrJ)
>> and
>> >> use the extractOnly option to see if the content is actually extracted.
>> >>
>> >> See:
>> >> https://cwiki.apache.org/confluence/display/solr/
>> >> Uploading+Data+with+Solr+Cell+using+Apache+Tika
>> >>
>> >> Also, some PDF files actually have the content as a bitmap image, so no
>> >> text is extracted.
>> >>
>> >>
>> >> -- Jack Krupansky
>> >>
>> >> On Tue, Apr 14, 2015 at 10:57 AM, Vijaya Narayana Reddy Bhoomi Reddy <
>> >> vijaya.bhoomireddy@whishworks.com> wrote:
>> >>
>> >> Hi,
>> >>>
>> >>> I am trying to index PDF and Microsoft Office files (.doc, .docx, .ppt,
>> >>> .pptx, .xlx, and .xlx) files into Solr. I am facing the following
>> issues.
>> >>> Request to please let me know what is going wrong with the indexing
>> >>> process.
>> >>>
>> >>> I am using solr 4.10.2 and using the default example server
>> configuration
>> >>> that comes with Solr distribution.
>> >>>
>> >>> PDF Files - Indexing as such works fine, but when I query using *.* in
>> >>> the
>> >>> Solr Query console, metadata information is displayed properly.
>> However,
>> >>> the PDF content field is empty. This is happening for all PDF files I
>> >>> have
>> >>> tried. I have tried with some proprietary files, PDF eBooks etc.
>> Whatever
>> >>> be the PDF file, content is not being displayed.
>> >>>
>> >>> MS Office files - For some office files, everything works perfect and
>> >>> the
>> >>> extracted content is visible in the query console. However, for
>> others, I
>> >>> see the below error message during the indexing process.
>> >>>
>> >>> *Exception in thread "main"
>> >>> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
>> >>> org.apache.tika.exception.TikaException: Unexpected RuntimeException
>> >>> from
>> >>> org.apache.tika.parser.microsoft.OfficeParser*
>> >>>
>> >>>
>> >>> I am using SolrJ to index the documents and below is the code snippet
>> >>> related to indexing. Please let me know where the issue is occurring.
>> >>>
>> >>> static String solrServerURL = "
>> >>> http://localhost:8983/solr";
>> >>> static SolrServer solrServer = new HttpSolrServer(solrServerURL);
>> >>> static ContentStreamUpdateRequest indexingReq
>> =
>> >>> new
>> >>>
>> >>> ContentStreamUpdateRequest("/update/extract");
>> >>>
>> >>> indexingReq.addFile(file, fileType);
>> >>> indexingReq.setParam("literal.id", literalId);
>> >>> indexingReq.setParam("uprefix", "attr_");
>> >>> indexingReq.setParam("fmap.content", "content");
>> >>> indexingReq.setParam("literal.fileurl", fileURL);
>> >>> indexingReq.setAction(AbstractUpdateRequest.ACTION.COMMIT, true, true);
>> >>> solrServer.request(indexingReq);
>> >>>
>> >>> Thanks & Regards
>> >>> Vijay
>> >>>
>> >>> --
>> >>> The contents of this e-mail are confidential and for the exclusive use
>> of
>> >>> the intended recipient. If you receive this e-mail in error please
>> delete
>> >>> it from your system immediately and notify us either by e-mail or
>> >>> telephone. You should not copy, forward or otherwise disclose the
>> content
>> >>> of the e-mail. The views expressed in this communication may not
>> >>> necessarily be the view held by WHISHWORKS.
>> >>>
>> >>>
>> >
>>
>>
>> --
>> Ph: 9845704792
>>
>
> --
> The contents of this e-mail are confidential and for the exclusive use of
> the intended recipient. If you receive this e-mail in error please delete
> it from your system immediately and notify us either by e-mail or
> telephone. You should not copy, forward or otherwise disclose the content
> of the e-mail. The views expressed in this communication may not
> necessarily be the view held by WHISHWORKS.
Re: Indexing PDF and MS Office files
Posted by Vijaya Narayana Reddy Bhoomi Reddy <vi...@whishworks.com>.
Thanks everyone for the responses. Now I am able to index PDF documents
successfully. I have implemented manual extraction using Tika's AutoParser
and PDF functionality is working fine. However, the error with some MS
office word documents still persist.
The error message is "java.lang.IllegalArgumentException: This paragraph is
not the first one in the table" which will eventually result in "Unexpected
RuntimeException from org.apache.tika.parser.microsoft.OfficeParser"
Upon some reading, it looks like its a bug with Tika 1.5 and seems to have
been fixed with Tika 1.6 ( https://issues.apache.org/jira/browse/TIKA-1251 ).
I am new to Solr / Tika and hence wondering whether I can change the Tika
library alone to v1.6 without impacting any of the libraries within Solr
4.10.2? Please let me know your response and how to get away with this
issue.
Many thanks in advance.
Thanks & Regards
Vijay
On 15 April 2015 at 05:14, Shyam R <sh...@gmail.com> wrote:
> Vijay,
>
> You could try different excel files with different formats to rule out the
> issue is with TIKA version being used.
>
> Thanks
> Murthy
>
> On Wed, Apr 15, 2015 at 9:35 AM, Terry Rhodes <tr...@gmail.com>
> wrote:
>
> > Perhaps the PDF is protected and the content can not be extracted?
> >
> > i have an unverified suspicion that the tika shipped with solr 4.10.2 may
> > not support some/all office 2013 document formats.
> >
> >
> >
> >
> >
> > On 4/14/2015 8:18 PM, Jack Krupansky wrote:
> >
> >> Try doing a manual extraction request directly to Solr (not via SolrJ)
> and
> >> use the extractOnly option to see if the content is actually extracted.
> >>
> >> See:
> >> https://cwiki.apache.org/confluence/display/solr/
> >> Uploading+Data+with+Solr+Cell+using+Apache+Tika
> >>
> >> Also, some PDF files actually have the content as a bitmap image, so no
> >> text is extracted.
> >>
> >>
> >> -- Jack Krupansky
> >>
> >> On Tue, Apr 14, 2015 at 10:57 AM, Vijaya Narayana Reddy Bhoomi Reddy <
> >> vijaya.bhoomireddy@whishworks.com> wrote:
> >>
> >> Hi,
> >>>
> >>> I am trying to index PDF and Microsoft Office files (.doc, .docx, .ppt,
> >>> .pptx, .xlx, and .xlx) files into Solr. I am facing the following
> issues.
> >>> Request to please let me know what is going wrong with the indexing
> >>> process.
> >>>
> >>> I am using solr 4.10.2 and using the default example server
> configuration
> >>> that comes with Solr distribution.
> >>>
> >>> PDF Files - Indexing as such works fine, but when I query using *.* in
> >>> the
> >>> Solr Query console, metadata information is displayed properly.
> However,
> >>> the PDF content field is empty. This is happening for all PDF files I
> >>> have
> >>> tried. I have tried with some proprietary files, PDF eBooks etc.
> Whatever
> >>> be the PDF file, content is not being displayed.
> >>>
> >>> MS Office files - For some office files, everything works perfect and
> >>> the
> >>> extracted content is visible in the query console. However, for
> others, I
> >>> see the below error message during the indexing process.
> >>>
> >>> *Exception in thread "main"
> >>> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
> >>> org.apache.tika.exception.TikaException: Unexpected RuntimeException
> >>> from
> >>> org.apache.tika.parser.microsoft.OfficeParser*
> >>>
> >>>
> >>> I am using SolrJ to index the documents and below is the code snippet
> >>> related to indexing. Please let me know where the issue is occurring.
> >>>
> >>> static String solrServerURL = "
> >>> http://localhost:8983/solr";
> >>> static SolrServer solrServer = new HttpSolrServer(solrServerURL);
> >>> static ContentStreamUpdateRequest indexingReq
> =
> >>> new
> >>>
> >>> ContentStreamUpdateRequest("/update/extract");
> >>>
> >>> indexingReq.addFile(file, fileType);
> >>> indexingReq.setParam("literal.id", literalId);
> >>> indexingReq.setParam("uprefix", "attr_");
> >>> indexingReq.setParam("fmap.content", "content");
> >>> indexingReq.setParam("literal.fileurl", fileURL);
> >>> indexingReq.setAction(AbstractUpdateRequest.ACTION.COMMIT, true, true);
> >>> solrServer.request(indexingReq);
> >>>
> >>> Thanks & Regards
> >>> Vijay
> >>>
> >>> --
> >>> The contents of this e-mail are confidential and for the exclusive use
> of
> >>> the intended recipient. If you receive this e-mail in error please
> delete
> >>> it from your system immediately and notify us either by e-mail or
> >>> telephone. You should not copy, forward or otherwise disclose the
> content
> >>> of the e-mail. The views expressed in this communication may not
> >>> necessarily be the view held by WHISHWORKS.
> >>>
> >>>
> >
>
>
> --
> Ph: 9845704792
>
--
The contents of this e-mail are confidential and for the exclusive use of
the intended recipient. If you receive this e-mail in error please delete
it from your system immediately and notify us either by e-mail or
telephone. You should not copy, forward or otherwise disclose the content
of the e-mail. The views expressed in this communication may not
necessarily be the view held by WHISHWORKS.
Re: Indexing PDF and MS Office files
Posted by Shyam R <sh...@gmail.com>.
Vijay,
You could try different excel files with different formats to rule out the
issue is with TIKA version being used.
Thanks
Murthy
On Wed, Apr 15, 2015 at 9:35 AM, Terry Rhodes <tr...@gmail.com> wrote:
> Perhaps the PDF is protected and the content can not be extracted?
>
> i have an unverified suspicion that the tika shipped with solr 4.10.2 may
> not support some/all office 2013 document formats.
>
>
>
>
>
> On 4/14/2015 8:18 PM, Jack Krupansky wrote:
>
>> Try doing a manual extraction request directly to Solr (not via SolrJ) and
>> use the extractOnly option to see if the content is actually extracted.
>>
>> See:
>> https://cwiki.apache.org/confluence/display/solr/
>> Uploading+Data+with+Solr+Cell+using+Apache+Tika
>>
>> Also, some PDF files actually have the content as a bitmap image, so no
>> text is extracted.
>>
>>
>> -- Jack Krupansky
>>
>> On Tue, Apr 14, 2015 at 10:57 AM, Vijaya Narayana Reddy Bhoomi Reddy <
>> vijaya.bhoomireddy@whishworks.com> wrote:
>>
>> Hi,
>>>
>>> I am trying to index PDF and Microsoft Office files (.doc, .docx, .ppt,
>>> .pptx, .xlx, and .xlx) files into Solr. I am facing the following issues.
>>> Request to please let me know what is going wrong with the indexing
>>> process.
>>>
>>> I am using solr 4.10.2 and using the default example server configuration
>>> that comes with Solr distribution.
>>>
>>> PDF Files - Indexing as such works fine, but when I query using *.* in
>>> the
>>> Solr Query console, metadata information is displayed properly. However,
>>> the PDF content field is empty. This is happening for all PDF files I
>>> have
>>> tried. I have tried with some proprietary files, PDF eBooks etc. Whatever
>>> be the PDF file, content is not being displayed.
>>>
>>> MS Office files - For some office files, everything works perfect and
>>> the
>>> extracted content is visible in the query console. However, for others, I
>>> see the below error message during the indexing process.
>>>
>>> *Exception in thread "main"
>>> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
>>> org.apache.tika.exception.TikaException: Unexpected RuntimeException
>>> from
>>> org.apache.tika.parser.microsoft.OfficeParser*
>>>
>>>
>>> I am using SolrJ to index the documents and below is the code snippet
>>> related to indexing. Please let me know where the issue is occurring.
>>>
>>> static String solrServerURL = "
>>> http://localhost:8983/solr";
>>> static SolrServer solrServer = new HttpSolrServer(solrServerURL);
>>> static ContentStreamUpdateRequest indexingReq =
>>> new
>>>
>>> ContentStreamUpdateRequest("/update/extract");
>>>
>>> indexingReq.addFile(file, fileType);
>>> indexingReq.setParam("literal.id", literalId);
>>> indexingReq.setParam("uprefix", "attr_");
>>> indexingReq.setParam("fmap.content", "content");
>>> indexingReq.setParam("literal.fileurl", fileURL);
>>> indexingReq.setAction(AbstractUpdateRequest.ACTION.COMMIT, true, true);
>>> solrServer.request(indexingReq);
>>>
>>> Thanks & Regards
>>> Vijay
>>>
>>> --
>>> The contents of this e-mail are confidential and for the exclusive use of
>>> the intended recipient. If you receive this e-mail in error please delete
>>> it from your system immediately and notify us either by e-mail or
>>> telephone. You should not copy, forward or otherwise disclose the content
>>> of the e-mail. The views expressed in this communication may not
>>> necessarily be the view held by WHISHWORKS.
>>>
>>>
>
--
Ph: 9845704792
Re: Indexing PDF and MS Office files
Posted by Terry Rhodes <tr...@gmail.com>.
Perhaps the PDF is protected and the content can not be extracted?
i have an unverified suspicion that the tika shipped with solr 4.10.2
may not support some/all office 2013 document formats.
On 4/14/2015 8:18 PM, Jack Krupansky wrote:
> Try doing a manual extraction request directly to Solr (not via SolrJ) and
> use the extractOnly option to see if the content is actually extracted.
>
> See:
> https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+Solr+Cell+using+Apache+Tika
>
> Also, some PDF files actually have the content as a bitmap image, so no
> text is extracted.
>
>
> -- Jack Krupansky
>
> On Tue, Apr 14, 2015 at 10:57 AM, Vijaya Narayana Reddy Bhoomi Reddy <
> vijaya.bhoomireddy@whishworks.com> wrote:
>
>> Hi,
>>
>> I am trying to index PDF and Microsoft Office files (.doc, .docx, .ppt,
>> .pptx, .xlx, and .xlx) files into Solr. I am facing the following issues.
>> Request to please let me know what is going wrong with the indexing
>> process.
>>
>> I am using solr 4.10.2 and using the default example server configuration
>> that comes with Solr distribution.
>>
>> PDF Files - Indexing as such works fine, but when I query using *.* in the
>> Solr Query console, metadata information is displayed properly. However,
>> the PDF content field is empty. This is happening for all PDF files I have
>> tried. I have tried with some proprietary files, PDF eBooks etc. Whatever
>> be the PDF file, content is not being displayed.
>>
>> MS Office files - For some office files, everything works perfect and the
>> extracted content is visible in the query console. However, for others, I
>> see the below error message during the indexing process.
>>
>> *Exception in thread "main"
>> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
>> org.apache.tika.exception.TikaException: Unexpected RuntimeException from
>> org.apache.tika.parser.microsoft.OfficeParser*
>>
>>
>> I am using SolrJ to index the documents and below is the code snippet
>> related to indexing. Please let me know where the issue is occurring.
>>
>> static String solrServerURL = "
>> http://localhost:8983/solr";
>> static SolrServer solrServer = new HttpSolrServer(solrServerURL);
>> static ContentStreamUpdateRequest indexingReq = new
>>
>> ContentStreamUpdateRequest("/update/extract");
>>
>> indexingReq.addFile(file, fileType);
>> indexingReq.setParam("literal.id", literalId);
>> indexingReq.setParam("uprefix", "attr_");
>> indexingReq.setParam("fmap.content", "content");
>> indexingReq.setParam("literal.fileurl", fileURL);
>> indexingReq.setAction(AbstractUpdateRequest.ACTION.COMMIT, true, true);
>> solrServer.request(indexingReq);
>>
>> Thanks & Regards
>> Vijay
>>
>> --
>> The contents of this e-mail are confidential and for the exclusive use of
>> the intended recipient. If you receive this e-mail in error please delete
>> it from your system immediately and notify us either by e-mail or
>> telephone. You should not copy, forward or otherwise disclose the content
>> of the e-mail. The views expressed in this communication may not
>> necessarily be the view held by WHISHWORKS.
>>
Re: Indexing PDF and MS Office files
Posted by Jack Krupansky <ja...@gmail.com>.
Try doing a manual extraction request directly to Solr (not via SolrJ) and
use the extractOnly option to see if the content is actually extracted.
See:
https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+Solr+Cell+using+Apache+Tika
Also, some PDF files actually have the content as a bitmap image, so no
text is extracted.
-- Jack Krupansky
On Tue, Apr 14, 2015 at 10:57 AM, Vijaya Narayana Reddy Bhoomi Reddy <
vijaya.bhoomireddy@whishworks.com> wrote:
> Hi,
>
> I am trying to index PDF and Microsoft Office files (.doc, .docx, .ppt,
> .pptx, .xlx, and .xlx) files into Solr. I am facing the following issues.
> Request to please let me know what is going wrong with the indexing
> process.
>
> I am using solr 4.10.2 and using the default example server configuration
> that comes with Solr distribution.
>
> PDF Files - Indexing as such works fine, but when I query using *.* in the
> Solr Query console, metadata information is displayed properly. However,
> the PDF content field is empty. This is happening for all PDF files I have
> tried. I have tried with some proprietary files, PDF eBooks etc. Whatever
> be the PDF file, content is not being displayed.
>
> MS Office files - For some office files, everything works perfect and the
> extracted content is visible in the query console. However, for others, I
> see the below error message during the indexing process.
>
> *Exception in thread "main"
> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
> org.apache.tika.exception.TikaException: Unexpected RuntimeException from
> org.apache.tika.parser.microsoft.OfficeParser*
>
>
> I am using SolrJ to index the documents and below is the code snippet
> related to indexing. Please let me know where the issue is occurring.
>
> static String solrServerURL = "
> http://localhost:8983/solr";
> static SolrServer solrServer = new HttpSolrServer(solrServerURL);
> static ContentStreamUpdateRequest indexingReq = new
>
> ContentStreamUpdateRequest("/update/extract");
>
> indexingReq.addFile(file, fileType);
> indexingReq.setParam("literal.id", literalId);
> indexingReq.setParam("uprefix", "attr_");
> indexingReq.setParam("fmap.content", "content");
> indexingReq.setParam("literal.fileurl", fileURL);
> indexingReq.setAction(AbstractUpdateRequest.ACTION.COMMIT, true, true);
> solrServer.request(indexingReq);
>
> Thanks & Regards
> Vijay
>
> --
> The contents of this e-mail are confidential and for the exclusive use of
> the intended recipient. If you receive this e-mail in error please delete
> it from your system immediately and notify us either by e-mail or
> telephone. You should not copy, forward or otherwise disclose the content
> of the e-mail. The views expressed in this communication may not
> necessarily be the view held by WHISHWORKS.
>