You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by levtannen <Le...@usdoj.gov> on 2019/03/19 15:03:16 UTC

Upgrading tika

Hello community,
I am using Tika to extract the text content from pdf files before indexing
them. I used version 1.7 and it worked OK except it produced a lot warnings
like "Font not found". Now I am trying to move to the newer version 1.19.1
and I have problems finding all necessary dependencies. First it gave me the
exception that it cannot find the class 
"org/apache/cxf/jaxrs/ext/multipart/ContentDisposition". I have added jars
"cxf-core-3.2.8.jar" and "cxf-rt-fromtend-jaxrs-2.6.3.jar" from the cxf
project. Now it gives me an exception "java.lang.NoClassDefFoundError:
javax/ws/rs/core/MultivaluedMap".
Could anybody suggest me what files do I need to use the latest version of
Tika and where to find them?
Thank you.
 



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Upgrading tika

Posted by Geoffrey Willis <gw...@yahoo.com.INVALID>.
Thanks for the explanation, it makes sense. I have noticed that sometimes a pdf document with spaces in the name can kill Tika and as result Solr so I get your point. Was trying to keep my webApp all Javascript/Typescript so went with the exposed extract/update handler. In my looking around I did see some python wrappers around a Tika service that might be a better solution. I can’t afford to crash the Solr server and it’s been many years since I wrote any java code! I remember we did use Solrj back in the day (I think the Lucene version was 3.5 and the Solr was similar). May have to dust off that code. Anyway, thanks for the thorough explanation. Gonna have to rethink my design.
Geoff

> On Mar 20, 2019, at 12:51 PM, Geoffrey Willis <gw...@yahoo.com.INVALID> wrote:
> 
> Could you expand on that please? I’m currently building a webApp that submits documents to Solr/Tika via the update/extract handler and it’s working fine. What do you mean when you say “You do not want to have your Solr instance processing via Tika”? If that’s a bad design choice please elaborate. 
> Thanks,
> Geoff
> 
> 
>> On Mar 19, 2019, at 5:15 PM, Phil Scadden <P....@gns.cri.nz> wrote:
>> 
>> As per Erick advice, I would strongly recommend that you do anything tika in a  separate solrj programme. You do not want to have your solr instance processing via tika.
>> 
>> -----Original Message-----
>> From: Tannen, Lev (USAEO) [Contractor] <Le...@usdoj.gov.INVALID>
>> Sent: Wednesday, 20 March 2019 08:17
>> To: solr-user@lucene.apache.org
>> Subject: RE: Upgrading tika
>> 
>> Sorry Erick,
>> Please disregard my previous message. Somehow I downloaded the version without those two files. I am going to download the latest version solr 8.0.0 and try it.
>> Best
>> Lev Tannen
>> 
>> -----Original Message-----
>> From: Erick Erickson <er...@gmail.com>
>> Sent: Tuesday, March 19, 2019 2:48 PM
>> To: solr-user <so...@lucene.apache.org>
>> Subject: Re: Upgrading tika
>> 
>> Yes, Solr is distributed with Tika. Look in:
>> ./solr/contrib/extraction/lib
>> 
>> Tika is upgraded when new versions come out, so the underlying files are whatever are current at the time.
>> 
>> The integration is a fairly loose coupling, if you're using some external program (say a SolrJ program) to parse the files, there's no requirement to use the jars distributed with Solr, use whatever suits your fancy. An external program just constructs a SolrDocument to send to Solr. What you use to create that document is irrelevant. See:
>> https://lucidworks.com/2012/02/14/indexing-with-solrj/ for some background.
>> 
>> If you're using the ExtractingRequestHandler, where you just send the semi-structured docs to Solr (PDFs, Word or whatever), then needing to know anything about individual Tika-related jar files is kind of strange.
>> 
>> If your predecessors wrote some custom code that runs as part of Solr, I don't know what to say...
>> 
>> Best,
>> Erick
>> 
>> On Tue, Mar 19, 2019 at 10:47 AM Tannen, Lev (USAEO) [Contractor] <Le...@usdoj.gov.invalid> wrote:
>>> 
>>> Thank you Shawn.
>>> I assumed that tika has been integrated with solr. I the project written before me they used two tika files taken from solr distribution. I am trying to do the same with solr 7.7.1. However this version contains a different set of tika related files. So I am confused. Does  solr does not have integrated tika anymore, or I just cannot recognize them?
>>> 
>>> -----Original Message-----
>>> From: Shawn Heisey <ap...@elyograg.org>
>>> Sent: Tuesday, March 19, 2019 11:11 AM
>>> To: solr-user@lucene.apache.org
>>> Subject: Re: Upgrading tika
>>> 
>>> On 3/19/2019 9:03 AM, levtannen wrote:
>>>> Could anybody suggest me what files do I need to use the latest
>>>> version of Tika and where to find them?
>>> 
>>> This mailing list is solr-user.  Tika is an entirely separate project from Solr within the Apache Foundation.  To get help with Tika, you'll need to ask that project.
>>> 
>>> https://tika.apache.org/mail-lists.html
>>> 
>>> Thanks,
>>> Shawn
>> Notice: This email and any attachments are confidential and may not be used, published or redistributed without the prior written consent of the Institute of Geological and Nuclear Sciences Limited (GNS Science). If received in error please destroy and immediately notify GNS Science. Do not copy or disclose the contents.
> 


RE: Upgrading tika

Posted by Phil Scadden <P....@gns.cri.nz>.
While using the update/extract handler is good for test, tika is a heavyweight with the risk that a bad document would compromise the solr instance and tika even with ordinary docs is a hog.

I wrote code with solrj to do the indexing and run it on completely different machine to the solr instance. It just sends SolrDocuments (created from analysis by tika) to the server as Erick says. It becomes even more important if you are going to incorporate inline OCR into the tika processing (the default). Solr docs gives you the outline for the solrj process. I don’t do inline OCR.

My workflow is something like this.
Find document to add.
If image PDF convert to searchable PDF via OCR  as searchable PDF is more useful document to deliver as result of search.
Submit document to the solrj-based solr indexer.

The core of my indexer is:
             File f = new File(filename);
             ContentHandler textHandler = new BodyContentHandler(Integer.MAX_VALUE);
             Metadata metadata = new Metadata();
             Parser parser = new AutoDetectParser();
             ParseContext context = new ParseContext();
             if (filename.toLowerCase().contains("pdf")) {  // this special setup of pdf processing is only required to switch OCR off
               PDFParserConfig pdfConfig = new PDFParserConfig();
               pdfConfig.setExtractInlineImages(false);
               pdfConfig.setOcrStrategy(PDFParserConfig.OCR_STRATEGY.NO_OCR);
               context.set(PDFParserConfig.class,pdfConfig);
               context.set(Parser.class,parser);
             }
             InputStream input = new FileInputStream(f);
             try {
               parser.parse(input, textHandler, metadata, context);
             } catch (Exception e) {
 // exception handling
             }
             SolrInputDocument up = new SolrInputDocument();
             up.addField("id",f.getCanonicalPath());
//             other addField calls for items extracted from metadata etc.
             up.addField("_text_",content);
             UpdateRequest req = new UpdateRequest();
             req.add(up);
             req.setBasicAuthCredentials("solrAdmin", password);
             UpdateResponse ur =  req.process(solr,"myindex");
             req.commit(solr, "myindex");

-----Original Message-----
From: Geoffrey Willis <gw...@yahoo.com.INVALID>
Sent: Thursday, 21 March 2019 06:52
To: solr-user@lucene.apache.org
Subject: Re: Upgrading tika

Could you expand on that please? I’m currently building a webApp that submits documents to Solr/Tika via the update/extract handler and it’s working fine. What do you mean when you say “You do not want to have your Solr instance processing via Tika”? If that’s a bad design choice please elaborate.
Thanks,
Geoff


> On Mar 19, 2019, at 5:15 PM, Phil Scadden <P....@gns.cri.nz> wrote:
>
> As per Erick advice, I would strongly recommend that you do anything tika in a  separate solrj programme. You do not want to have your solr instance processing via tika.
>
> -----Original Message-----
> From: Tannen, Lev (USAEO) [Contractor] <Le...@usdoj.gov.INVALID>
> Sent: Wednesday, 20 March 2019 08:17
> To: solr-user@lucene.apache.org
> Subject: RE: Upgrading tika
>
> Sorry Erick,
> Please disregard my previous message. Somehow I downloaded the version without those two files. I am going to download the latest version solr 8.0.0 and try it.
> Best
> Lev Tannen
>
> -----Original Message-----
> From: Erick Erickson <er...@gmail.com>
> Sent: Tuesday, March 19, 2019 2:48 PM
> To: solr-user <so...@lucene.apache.org>
> Subject: Re: Upgrading tika
>
> Yes, Solr is distributed with Tika. Look in:
> ./solr/contrib/extraction/lib
>
> Tika is upgraded when new versions come out, so the underlying files are whatever are current at the time.
>
> The integration is a fairly loose coupling, if you're using some external program (say a SolrJ program) to parse the files, there's no requirement to use the jars distributed with Solr, use whatever suits your fancy. An external program just constructs a SolrDocument to send to Solr. What you use to create that document is irrelevant. See:
> https://lucidworks.com/2012/02/14/indexing-with-solrj/ for some background.
>
> If you're using the ExtractingRequestHandler, where you just send the semi-structured docs to Solr (PDFs, Word or whatever), then needing to know anything about individual Tika-related jar files is kind of strange.
>
> If your predecessors wrote some custom code that runs as part of Solr, I don't know what to say...
>
> Best,
> Erick
>
> On Tue, Mar 19, 2019 at 10:47 AM Tannen, Lev (USAEO) [Contractor] <Le...@usdoj.gov.invalid> wrote:
>>
>> Thank you Shawn.
>> I assumed that tika has been integrated with solr. I the project written before me they used two tika files taken from solr distribution. I am trying to do the same with solr 7.7.1. However this version contains a different set of tika related files. So I am confused. Does  solr does not have integrated tika anymore, or I just cannot recognize them?
>>
>> -----Original Message-----
>> From: Shawn Heisey <ap...@elyograg.org>
>> Sent: Tuesday, March 19, 2019 11:11 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Upgrading tika
>>
>> On 3/19/2019 9:03 AM, levtannen wrote:
>>> Could anybody suggest me what files do I need to use the latest
>>> version of Tika and where to find them?
>>
>> This mailing list is solr-user.  Tika is an entirely separate project from Solr within the Apache Foundation.  To get help with Tika, you'll need to ask that project.
>>
>> https://tika.apache.org/mail-lists.html
>>
>> Thanks,
>> Shawn
> Notice: This email and any attachments are confidential and may not be used, published or redistributed without the prior written consent of the Institute of Geological and Nuclear Sciences Limited (GNS Science). If received in error please destroy and immediately notify GNS Science. Do not copy or disclose the contents.

Notice: This email and any attachments are confidential and may not be used, published or redistributed without the prior written consent of the Institute of Geological and Nuclear Sciences Limited (GNS Science). If received in error please destroy and immediately notify GNS Science. Do not copy or disclose the contents.

Re: Upgrading tika

Posted by Geoffrey Willis <gw...@yahoo.com.INVALID>.
Could you expand on that please? I’m currently building a webApp that submits documents to Solr/Tika via the update/extract handler and it’s working fine. What do you mean when you say “You do not want to have your Solr instance processing via Tika”? If that’s a bad design choice please elaborate. 
Thanks,
Geoff


> On Mar 19, 2019, at 5:15 PM, Phil Scadden <P....@gns.cri.nz> wrote:
> 
> As per Erick advice, I would strongly recommend that you do anything tika in a  separate solrj programme. You do not want to have your solr instance processing via tika.
> 
> -----Original Message-----
> From: Tannen, Lev (USAEO) [Contractor] <Le...@usdoj.gov.INVALID>
> Sent: Wednesday, 20 March 2019 08:17
> To: solr-user@lucene.apache.org
> Subject: RE: Upgrading tika
> 
> Sorry Erick,
> Please disregard my previous message. Somehow I downloaded the version without those two files. I am going to download the latest version solr 8.0.0 and try it.
> Best
> Lev Tannen
> 
> -----Original Message-----
> From: Erick Erickson <er...@gmail.com>
> Sent: Tuesday, March 19, 2019 2:48 PM
> To: solr-user <so...@lucene.apache.org>
> Subject: Re: Upgrading tika
> 
> Yes, Solr is distributed with Tika. Look in:
> ./solr/contrib/extraction/lib
> 
> Tika is upgraded when new versions come out, so the underlying files are whatever are current at the time.
> 
> The integration is a fairly loose coupling, if you're using some external program (say a SolrJ program) to parse the files, there's no requirement to use the jars distributed with Solr, use whatever suits your fancy. An external program just constructs a SolrDocument to send to Solr. What you use to create that document is irrelevant. See:
> https://lucidworks.com/2012/02/14/indexing-with-solrj/ for some background.
> 
> If you're using the ExtractingRequestHandler, where you just send the semi-structured docs to Solr (PDFs, Word or whatever), then needing to know anything about individual Tika-related jar files is kind of strange.
> 
> If your predecessors wrote some custom code that runs as part of Solr, I don't know what to say...
> 
> Best,
> Erick
> 
> On Tue, Mar 19, 2019 at 10:47 AM Tannen, Lev (USAEO) [Contractor] <Le...@usdoj.gov.invalid> wrote:
>> 
>> Thank you Shawn.
>> I assumed that tika has been integrated with solr. I the project written before me they used two tika files taken from solr distribution. I am trying to do the same with solr 7.7.1. However this version contains a different set of tika related files. So I am confused. Does  solr does not have integrated tika anymore, or I just cannot recognize them?
>> 
>> -----Original Message-----
>> From: Shawn Heisey <ap...@elyograg.org>
>> Sent: Tuesday, March 19, 2019 11:11 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Upgrading tika
>> 
>> On 3/19/2019 9:03 AM, levtannen wrote:
>>> Could anybody suggest me what files do I need to use the latest
>>> version of Tika and where to find them?
>> 
>> This mailing list is solr-user.  Tika is an entirely separate project from Solr within the Apache Foundation.  To get help with Tika, you'll need to ask that project.
>> 
>> https://tika.apache.org/mail-lists.html
>> 
>> Thanks,
>> Shawn
> Notice: This email and any attachments are confidential and may not be used, published or redistributed without the prior written consent of the Institute of Geological and Nuclear Sciences Limited (GNS Science). If received in error please destroy and immediately notify GNS Science. Do not copy or disclose the contents.


RE: Upgrading tika

Posted by Phil Scadden <P....@gns.cri.nz>.
As per Erick advice, I would strongly recommend that you do anything tika in a  separate solrj programme. You do not want to have your solr instance processing via tika.

-----Original Message-----
From: Tannen, Lev (USAEO) [Contractor] <Le...@usdoj.gov.INVALID>
Sent: Wednesday, 20 March 2019 08:17
To: solr-user@lucene.apache.org
Subject: RE: Upgrading tika

Sorry Erick,
Please disregard my previous message. Somehow I downloaded the version without those two files. I am going to download the latest version solr 8.0.0 and try it.
Best
Lev Tannen

-----Original Message-----
From: Erick Erickson <er...@gmail.com>
Sent: Tuesday, March 19, 2019 2:48 PM
To: solr-user <so...@lucene.apache.org>
Subject: Re: Upgrading tika

Yes, Solr is distributed with Tika. Look in:
./solr/contrib/extraction/lib

Tika is upgraded when new versions come out, so the underlying files are whatever are current at the time.

The integration is a fairly loose coupling, if you're using some external program (say a SolrJ program) to parse the files, there's no requirement to use the jars distributed with Solr, use whatever suits your fancy. An external program just constructs a SolrDocument to send to Solr. What you use to create that document is irrelevant. See:
https://lucidworks.com/2012/02/14/indexing-with-solrj/ for some background.

If you're using the ExtractingRequestHandler, where you just send the semi-structured docs to Solr (PDFs, Word or whatever), then needing to know anything about individual Tika-related jar files is kind of strange.

If your predecessors wrote some custom code that runs as part of Solr, I don't know what to say...

Best,
Erick

On Tue, Mar 19, 2019 at 10:47 AM Tannen, Lev (USAEO) [Contractor] <Le...@usdoj.gov.invalid> wrote:
>
> Thank you Shawn.
> I assumed that tika has been integrated with solr. I the project written before me they used two tika files taken from solr distribution. I am trying to do the same with solr 7.7.1. However this version contains a different set of tika related files. So I am confused. Does  solr does not have integrated tika anymore, or I just cannot recognize them?
>
> -----Original Message-----
> From: Shawn Heisey <ap...@elyograg.org>
> Sent: Tuesday, March 19, 2019 11:11 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Upgrading tika
>
> On 3/19/2019 9:03 AM, levtannen wrote:
> > Could anybody suggest me what files do I need to use the latest
> > version of Tika and where to find them?
>
> This mailing list is solr-user.  Tika is an entirely separate project from Solr within the Apache Foundation.  To get help with Tika, you'll need to ask that project.
>
> https://tika.apache.org/mail-lists.html
>
> Thanks,
> Shawn
Notice: This email and any attachments are confidential and may not be used, published or redistributed without the prior written consent of the Institute of Geological and Nuclear Sciences Limited (GNS Science). If received in error please destroy and immediately notify GNS Science. Do not copy or disclose the contents.

RE: Upgrading tika

Posted by "Tannen, Lev (USAEO) [Contractor]" <Le...@usdoj.gov.INVALID>.
Sorry Erick, 
Please disregard my previous message. Somehow I downloaded the version without those two files. I am going to download the latest version solr 8.0.0 and try it.
Best 
Lev Tannen

-----Original Message-----
From: Erick Erickson <er...@gmail.com> 
Sent: Tuesday, March 19, 2019 2:48 PM
To: solr-user <so...@lucene.apache.org>
Subject: Re: Upgrading tika

Yes, Solr is distributed with Tika. Look in:
./solr/contrib/extraction/lib

Tika is upgraded when new versions come out, so the underlying files are whatever are current at the time.

The integration is a fairly loose coupling, if you're using some external program (say a SolrJ program) to parse the files, there's no requirement to use the jars distributed with Solr, use whatever suits your fancy. An external program just constructs a SolrDocument to send to Solr. What you use to create that document is irrelevant. See:
https://lucidworks.com/2012/02/14/indexing-with-solrj/ for some background.

If you're using the ExtractingRequestHandler, where you just send the semi-structured docs to Solr (PDFs, Word or whatever), then needing to know anything about individual Tika-related jar files is kind of strange.

If your predecessors wrote some custom code that runs as part of Solr, I don't know what to say...

Best,
Erick

On Tue, Mar 19, 2019 at 10:47 AM Tannen, Lev (USAEO) [Contractor] <Le...@usdoj.gov.invalid> wrote:
>
> Thank you Shawn.
> I assumed that tika has been integrated with solr. I the project written before me they used two tika files taken from solr distribution. I am trying to do the same with solr 7.7.1. However this version contains a different set of tika related files. So I am confused. Does  solr does not have integrated tika anymore, or I just cannot recognize them?
>
> -----Original Message-----
> From: Shawn Heisey <ap...@elyograg.org>
> Sent: Tuesday, March 19, 2019 11:11 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Upgrading tika
>
> On 3/19/2019 9:03 AM, levtannen wrote:
> > Could anybody suggest me what files do I need to use the latest 
> > version of Tika and where to find them?
>
> This mailing list is solr-user.  Tika is an entirely separate project from Solr within the Apache Foundation.  To get help with Tika, you'll need to ask that project.
>
> https://tika.apache.org/mail-lists.html
>
> Thanks,
> Shawn

RE: Upgrading tika

Posted by "Tannen, Lev (USAEO) [Contractor]" <Le...@usdoj.gov.INVALID>.
Thank you Shawn and Erick,
 I truly did not want to dive into Tika and Cxf worlds, but it looks I have no choice.

-----Original Message-----
From: Shawn Heisey <ap...@elyograg.org> 
Sent: Wednesday, March 20, 2019 11:09 AM
To: solr-user@lucene.apache.org
Subject: Re: Upgrading tika

On 3/20/2019 8:24 AM, Tannen, Lev (USAEO) [Contractor] wrote:
> I still need your advice. The program I have to fix uses class AutoDetectParser along with Solrj for parsing PDF files before sending the result to the solr server. To do this it linked two tika jar files taken from the solr distribution. Namely: tika-core and tika-parsers. Maybe it used some other tika related files but I have problems to identify them among a lot of other jar files linked. The program worked more or less OK, but it gave too many warnings of kind "Font not found". I had a rumor that this was fixed in the next tika distribution.

Solr does include a subset of Tika - just enough to make the Extracting Request Handler work.

Since you're writing your own program that uses Tika, the dependencies you need could be very different than what Solr needs for its Tika integration.

It is strongly recommended, as Erick mentioned, to never use Solr's Tika integration in production.  Tika has a tendency to crash with some input files, especially PDF, and if it crashes when it is running inside Solr, then Solr will crash too.  No more search engine.

>   I switched from Solr 7.5 to solr 7.7.1 in a hope that will solve that problem. However when I switched I encountered an  another problem:
> java.lang.NoClassDefFoundError: org/apache/cxf/jaxrs/ext/multipart/ContentDisposition.
> Apparently I have not included some necessary jars. Those jars supposed to come from a different project  called cxf, but because they are related to tika I expected them be distributed with solr. However I did not find them in the solr 7.7.1 (in the solr 7.5 as well).

I had never heard of CXF before.  It is not included with Solr.  The Extracting Request Handler must not use the part of Tika that needs CXF, so we don't include it.

> So could you please advise what is the best way to proceed.

If you want to know how to use Tika in your program and what you need for your particular use case, talk to the Tika project.  There is at least one person from the Tika project subscribed, but questions about that project are off-topic on this list.

Thanks,
Shawn

Re: Upgrading tika

Posted by Shawn Heisey <ap...@elyograg.org>.
On 3/20/2019 8:24 AM, Tannen, Lev (USAEO) [Contractor] wrote:
> I still need your advice. The program I have to fix uses class AutoDetectParser along with Solrj for parsing PDF files before sending the result to the solr server. To do this it linked two tika jar files taken from the solr distribution. Namely: tika-core and tika-parsers. Maybe it used some other tika related files but I have problems to identify them among a lot of other jar files linked. The program worked more or less OK, but it gave too many warnings of kind "Font not found". I had a rumor that this was fixed in the next tika distribution.

Solr does include a subset of Tika - just enough to make the Extracting 
Request Handler work.

Since you're writing your own program that uses Tika, the dependencies 
you need could be very different than what Solr needs for its Tika 
integration.

It is strongly recommended, as Erick mentioned, to never use Solr's Tika 
integration in production.  Tika has a tendency to crash with some input 
files, especially PDF, and if it crashes when it is running inside Solr, 
then Solr will crash too.  No more search engine.

>   I switched from Solr 7.5 to solr 7.7.1 in a hope that will solve that problem. However when I switched I encountered an  another problem:
> java.lang.NoClassDefFoundError: org/apache/cxf/jaxrs/ext/multipart/ContentDisposition.
> Apparently I have not included some necessary jars. Those jars supposed to come from a different project  called cxf, but because they are related to tika I expected them be distributed with solr. However I did not find them in the solr 7.7.1 (in the solr 7.5 as well).

I had never heard of CXF before.  It is not included with Solr.  The 
Extracting Request Handler must not use the part of Tika that needs CXF, 
so we don't include it.

> So could you please advise what is the best way to proceed.

If you want to know how to use Tika in your program and what you need 
for your particular use case, talk to the Tika project.  There is at 
least one person from the Tika project subscribed, but questions about 
that project are off-topic on this list.

Thanks,
Shawn

Re: Upgrading tika

Posted by Erick Erickson <er...@gmail.com>.
Well, I’d have to do the same thing, go spelunking in Tika.. When I used it from SolrJ, I just linked to the Tika distro and it “just worked”, but I admit that was a while ago.

Your best bet would probably be the Tika user’s list.

Best,
Erick

> On Mar 20, 2019, at 7:24 AM, Tannen, Lev (USAEO) [Contractor] <Le...@usdoj.gov.INVALID> wrote:
> 
> Hi Erick,
> 
> I still need your advice. The program I have to fix uses class AutoDetectParser along with Solrj for parsing PDF files before sending the result to the solr server. To do this it linked two tika jar files taken from the solr distribution. Namely: tika-core and tika-parsers. Maybe it used some other tika related files but I have problems to identify them among a lot of other jar files linked. The program worked more or less OK, but it gave too many warnings of kind "Font not found". I had a rumor that this was fixed in the next tika distribution.
> I switched from Solr 7.5 to solr 7.7.1 in a hope that will solve that problem. However when I switched I encountered an  another problem: 
> java.lang.NoClassDefFoundError: org/apache/cxf/jaxrs/ext/multipart/ContentDisposition.
> Apparently I have not included some necessary jars. Those jars supposed to come from a different project  called cxf, but because they are related to tika I expected them be distributed with solr. However I did not find them in the solr 7.7.1 (in the solr 7.5 as well). 
> I have found the necessary file in the cxf distribution and included it. It asked for an another file which I included as well. After this I got a message that some temporary resources were not closed. Apparently something is not matched. And now I am stuck. I do not want to start from scratch and search the whole tika and cxf projects for the files I need and I do not want to include all files from those projects especially because I was not able to find a binary distribution. So could you please advise what is the best way to proceed.
> 
> Thank you,
> Lev Tannen
> 
> -----Original Message-----
> From: Erick Erickson <er...@gmail.com> 
> Sent: Tuesday, March 19, 2019 2:48 PM
> To: solr-user <so...@lucene.apache.org>
> Subject: Re: Upgrading tika
> 
> Yes, Solr is distributed with Tika. Look in:
> ./solr/contrib/extraction/lib
> 
> Tika is upgraded when new versions come out, so the underlying files are whatever are current at the time.
> 
> The integration is a fairly loose coupling, if you're using some external program (say a SolrJ program) to parse the files, there's no requirement to use the jars distributed with Solr, use whatever suits your fancy. An external program just constructs a SolrDocument to send to Solr. What you use to create that document is irrelevant. See:
> https://lucidworks.com/2012/02/14/indexing-with-solrj/ for some background.
> 
> If you're using the ExtractingRequestHandler, where you just send the semi-structured docs to Solr (PDFs, Word or whatever), then needing to know anything about individual Tika-related jar files is kind of strange.
> 
> If your predecessors wrote some custom code that runs as part of Solr, I don't know what to say...
> 
> Best,
> Erick
> 
> On Tue, Mar 19, 2019 at 10:47 AM Tannen, Lev (USAEO) [Contractor] <Le...@usdoj.gov.invalid> wrote:
>> 
>> Thank you Shawn.
>> I assumed that tika has been integrated with solr. I the project written before me they used two tika files taken from solr distribution. I am trying to do the same with solr 7.7.1. However this version contains a different set of tika related files. So I am confused. Does  solr does not have integrated tika anymore, or I just cannot recognize them?
>> 
>> -----Original Message-----
>> From: Shawn Heisey <ap...@elyograg.org>
>> Sent: Tuesday, March 19, 2019 11:11 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Upgrading tika
>> 
>> On 3/19/2019 9:03 AM, levtannen wrote:
>>> Could anybody suggest me what files do I need to use the latest 
>>> version of Tika and where to find them?
>> 
>> This mailing list is solr-user.  Tika is an entirely separate project from Solr within the Apache Foundation.  To get help with Tika, you'll need to ask that project.
>> 
>> https://tika.apache.org/mail-lists.html
>> 
>> Thanks,
>> Shawn


RE: Upgrading tika

Posted by "Tannen, Lev (USAEO) [Contractor]" <Le...@usdoj.gov.INVALID>.
Hi Erick,

I still need your advice. The program I have to fix uses class AutoDetectParser along with Solrj for parsing PDF files before sending the result to the solr server. To do this it linked two tika jar files taken from the solr distribution. Namely: tika-core and tika-parsers. Maybe it used some other tika related files but I have problems to identify them among a lot of other jar files linked. The program worked more or less OK, but it gave too many warnings of kind "Font not found". I had a rumor that this was fixed in the next tika distribution.
 I switched from Solr 7.5 to solr 7.7.1 in a hope that will solve that problem. However when I switched I encountered an  another problem: 
java.lang.NoClassDefFoundError: org/apache/cxf/jaxrs/ext/multipart/ContentDisposition.
Apparently I have not included some necessary jars. Those jars supposed to come from a different project  called cxf, but because they are related to tika I expected them be distributed with solr. However I did not find them in the solr 7.7.1 (in the solr 7.5 as well). 
I have found the necessary file in the cxf distribution and included it. It asked for an another file which I included as well. After this I got a message that some temporary resources were not closed. Apparently something is not matched. And now I am stuck. I do not want to start from scratch and search the whole tika and cxf projects for the files I need and I do not want to include all files from those projects especially because I was not able to find a binary distribution. So could you please advise what is the best way to proceed.

Thank you,
Lev Tannen

-----Original Message-----
From: Erick Erickson <er...@gmail.com> 
Sent: Tuesday, March 19, 2019 2:48 PM
To: solr-user <so...@lucene.apache.org>
Subject: Re: Upgrading tika

Yes, Solr is distributed with Tika. Look in:
./solr/contrib/extraction/lib

Tika is upgraded when new versions come out, so the underlying files are whatever are current at the time.

The integration is a fairly loose coupling, if you're using some external program (say a SolrJ program) to parse the files, there's no requirement to use the jars distributed with Solr, use whatever suits your fancy. An external program just constructs a SolrDocument to send to Solr. What you use to create that document is irrelevant. See:
https://lucidworks.com/2012/02/14/indexing-with-solrj/ for some background.

If you're using the ExtractingRequestHandler, where you just send the semi-structured docs to Solr (PDFs, Word or whatever), then needing to know anything about individual Tika-related jar files is kind of strange.

If your predecessors wrote some custom code that runs as part of Solr, I don't know what to say...

Best,
Erick

On Tue, Mar 19, 2019 at 10:47 AM Tannen, Lev (USAEO) [Contractor] <Le...@usdoj.gov.invalid> wrote:
>
> Thank you Shawn.
> I assumed that tika has been integrated with solr. I the project written before me they used two tika files taken from solr distribution. I am trying to do the same with solr 7.7.1. However this version contains a different set of tika related files. So I am confused. Does  solr does not have integrated tika anymore, or I just cannot recognize them?
>
> -----Original Message-----
> From: Shawn Heisey <ap...@elyograg.org>
> Sent: Tuesday, March 19, 2019 11:11 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Upgrading tika
>
> On 3/19/2019 9:03 AM, levtannen wrote:
> > Could anybody suggest me what files do I need to use the latest 
> > version of Tika and where to find them?
>
> This mailing list is solr-user.  Tika is an entirely separate project from Solr within the Apache Foundation.  To get help with Tika, you'll need to ask that project.
>
> https://tika.apache.org/mail-lists.html
>
> Thanks,
> Shawn

RE: Upgrading tika

Posted by "Tannen, Lev (USAEO) [Contractor]" <Le...@usdoj.gov.INVALID>.
Thank you Erick,
The problem is that the solr7.7.1 distribution does not contain files tika-core and tika-parsers in the contrib/extraction/lib folder. It contains only(instead) tika-java7-1.19.1.jar and tika-xmp-1.19.1.jar.
Have I lost some files while downloading?
Best,
Lev Tannen

-----Original Message-----
From: Erick Erickson <er...@gmail.com> 
Sent: Tuesday, March 19, 2019 2:48 PM
To: solr-user <so...@lucene.apache.org>
Subject: Re: Upgrading tika

Yes, Solr is distributed with Tika. Look in:
./solr/contrib/extraction/lib

Tika is upgraded when new versions come out, so the underlying files are whatever are current at the time.

The integration is a fairly loose coupling, if you're using some external program (say a SolrJ program) to parse the files, there's no requirement to use the jars distributed with Solr, use whatever suits your fancy. An external program just constructs a SolrDocument to send to Solr. What you use to create that document is irrelevant. See:
https://lucidworks.com/2012/02/14/indexing-with-solrj/ for some background.

If you're using the ExtractingRequestHandler, where you just send the semi-structured docs to Solr (PDFs, Word or whatever), then needing to know anything about individual Tika-related jar files is kind of strange.

If your predecessors wrote some custom code that runs as part of Solr, I don't know what to say...

Best,
Erick

On Tue, Mar 19, 2019 at 10:47 AM Tannen, Lev (USAEO) [Contractor] <Le...@usdoj.gov.invalid> wrote:
>
> Thank you Shawn.
> I assumed that tika has been integrated with solr. I the project written before me they used two tika files taken from solr distribution. I am trying to do the same with solr 7.7.1. However this version contains a different set of tika related files. So I am confused. Does  solr does not have integrated tika anymore, or I just cannot recognize them?
>
> -----Original Message-----
> From: Shawn Heisey <ap...@elyograg.org>
> Sent: Tuesday, March 19, 2019 11:11 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Upgrading tika
>
> On 3/19/2019 9:03 AM, levtannen wrote:
> > Could anybody suggest me what files do I need to use the latest 
> > version of Tika and where to find them?
>
> This mailing list is solr-user.  Tika is an entirely separate project from Solr within the Apache Foundation.  To get help with Tika, you'll need to ask that project.
>
> https://tika.apache.org/mail-lists.html
>
> Thanks,
> Shawn

Re: Upgrading tika

Posted by Erick Erickson <er...@gmail.com>.
Yes, Solr is distributed with Tika. Look in:
./solr/contrib/extraction/lib

Tika is upgraded when new versions come out, so the underlying files
are whatever are current at the time.

The integration is a fairly loose coupling, if you're using some
external program (say a SolrJ program) to parse the files, there's no
requirement to use the jars distributed with Solr, use whatever suits
your fancy. An external program just constructs a SolrDocument to send
to Solr. What you use to create that document is irrelevant. See:
https://lucidworks.com/2012/02/14/indexing-with-solrj/ for some
background.

If you're using the ExtractingRequestHandler, where you just send the
semi-structured docs to Solr (PDFs, Word or whatever), then needing to
know anything about individual Tika-related jar files is kind of
strange.

If your predecessors wrote some custom code that runs as part of Solr,
I don't know what to say...

Best,
Erick

On Tue, Mar 19, 2019 at 10:47 AM Tannen, Lev (USAEO) [Contractor]
<Le...@usdoj.gov.invalid> wrote:
>
> Thank you Shawn.
> I assumed that tika has been integrated with solr. I the project written before me they used two tika files taken from solr distribution. I am trying to do the same with solr 7.7.1. However this version contains a different set of tika related files. So I am confused. Does  solr does not have integrated tika anymore, or I just cannot recognize them?
>
> -----Original Message-----
> From: Shawn Heisey <ap...@elyograg.org>
> Sent: Tuesday, March 19, 2019 11:11 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Upgrading tika
>
> On 3/19/2019 9:03 AM, levtannen wrote:
> > Could anybody suggest me what files do I need to use the latest
> > version of Tika and where to find them?
>
> This mailing list is solr-user.  Tika is an entirely separate project from Solr within the Apache Foundation.  To get help with Tika, you'll need to ask that project.
>
> https://tika.apache.org/mail-lists.html
>
> Thanks,
> Shawn

RE: Upgrading tika

Posted by "Tannen, Lev (USAEO) [Contractor]" <Le...@usdoj.gov.INVALID>.
Thank you Shawn. 
I assumed that tika has been integrated with solr. I the project written before me they used two tika files taken from solr distribution. I am trying to do the same with solr 7.7.1. However this version contains a different set of tika related files. So I am confused. Does  solr does not have integrated tika anymore, or I just cannot recognize them?

-----Original Message-----
From: Shawn Heisey <ap...@elyograg.org> 
Sent: Tuesday, March 19, 2019 11:11 AM
To: solr-user@lucene.apache.org
Subject: Re: Upgrading tika

On 3/19/2019 9:03 AM, levtannen wrote:
> Could anybody suggest me what files do I need to use the latest 
> version of Tika and where to find them?

This mailing list is solr-user.  Tika is an entirely separate project from Solr within the Apache Foundation.  To get help with Tika, you'll need to ask that project.

https://tika.apache.org/mail-lists.html

Thanks,
Shawn

Re: Upgrading tika

Posted by Shawn Heisey <ap...@elyograg.org>.
On 3/19/2019 9:03 AM, levtannen wrote:
> Could anybody suggest me what files do I need to use the latest version of
> Tika and where to find them?

This mailing list is solr-user.  Tika is an entirely separate project 
from Solr within the Apache Foundation.  To get help with Tika, you'll 
need to ask that project.

https://tika.apache.org/mail-lists.html

Thanks,
Shawn

RE: Upgrading tika

Posted by "Tannen, Lev (USAEO) [Contractor]" <Le...@usdoj.gov.INVALID>.
Thank you Jeremy,
I am not using Maven, but I took both jars from the same distribution of cxf so they supposed to be compatible.

-----Original Message-----
From: Branham, Jeremy (Experis) <jb...@allstate.com> 
Sent: Tuesday, March 19, 2019 11:10 AM
To: solr-user@lucene.apache.org
Subject: Re: Upgrading tika

I’m not positive – But I think you should match the CXF jar versions.

"cxf-core-3.2.8.jar" 

<dependency>
    <groupId>org.apache.cxf</groupId>
    <artifactId>cxf-rt-frontend-jaxrs</artifactId>
    <version>3.2.8</version>
</dependency>

 
Jeremy Branham
jbrcm@allstate.com

On 3/19/19, 10:03 AM, "levtannen" <Le...@usdoj.gov> wrote:

    "cxf-core-3.2.8.jar" and "cxf-rt-fromtend-jaxrs-2.6.3.jar"


Re: Upgrading tika

Posted by "Branham, Jeremy (Experis)" <jb...@allstate.com>.
I’m not positive – But I think you should match the CXF jar versions.

"cxf-core-3.2.8.jar" 

<dependency>
    <groupId>org.apache.cxf</groupId>
    <artifactId>cxf-rt-frontend-jaxrs</artifactId>
    <version>3.2.8</version>
</dependency>

 
Jeremy Branham
jbrcm@allstate.com

On 3/19/19, 10:03 AM, "levtannen" <Le...@usdoj.gov> wrote:

    "cxf-core-3.2.8.jar" and "cxf-rt-fromtend-jaxrs-2.6.3.jar"