You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@tika.apache.org by "didongke@126.com" <di...@126.com> on 2023/04/19 02:07:59 UTC

Tika server extraction failed

Hi,

    I have two files 111.xlsx and 2222.xlsx ( included in the compressed package).

    I  use Tika server (2.3.0 to extract their metadata and text content :
    curl -T 1111.xlsx http://127.0.0.1:12000/meta
    curl -T 222.xlsx http://127.0.0.1:12000/meta

    However, one extraction (1111.xlsx) was successful, while the other (2222.xlsx) extraction failed.
    2222. xlsx only has one more row of data than 1111. xlsx.

    The same is true for extracting text data.
    curl -T 1111.xlsx http://127.0.0.1:12000/tika --header "Accept: text/plain" -H "Content-Disposition: attachment; filename=1111.xlsx"    
    curl -T 2222.xlsx http://127.0.0.1:12000/tika --header "Accept: text/plain" -H "Content-Disposition: attachment; filename=2222.xlsx"   

Please assist !



didongke@126.com

Tika server extracts text content containing filename

Posted by "didongke@126.com" <di...@126.com>.
Hi,

     1.java contains the following content:
      999999999999999999999999

     I use the following command to extract text content from the Tika server:
    curl -T “/home/ddk/1.java” http://127.0.0.1:12000/tika --header "Accept: text/plain" -H "Content-Disposition: attachment; filename=/home/ddk/1.java"

    The returned content is:
    /home/ddk/1.java999999999999999999999999

    I don't understand why it includes the content after filename in the command.

    if I use  filename=xxx.java ,  it returns:
    xxx.java999999999999999999999999

    The Tika server is :  tika-server-standard-2.7.0.jar

Please assist !



didongke@126.com
 

Re: Re: Tika server extraction failed

Posted by "didongke@126.com" <di...@126.com>.
Thank you for your information.



didongke@126.com
 
From: Tim Allison
Date: 2023-04-20 18:58
To: user
Subject: Re: Re: Tika server extraction failed
Tika's detector runs detection on the byte stream.  If the suffix of
the file is a subtype of the detected byte stream, then it applies
that subtype.
 
So, if you tell tika that your text file is an xml file, it will
detect the bytes as text, but then see that xml is a subtype of text
and run the XML parser against the file, leading to a parse exception.
 
If you use the /rmeta endpoint, you'll get better feedback about parse
exceptions.
 
On Thu, Apr 20, 2023 at 2:47 AM didongke@126.com <di...@126.com> wrote:
>
> Hi,
>
> I have encountered a problem before: some files cannot be detected based on their content.
> So I added the file name and solved the problem.
>
> But now I have another problem: adding a file name actually resulted in not being detected.
>
> If that's the case, I need to make two attempts:
> Try using content detection first,
> and then try using file name detection.
>
>
> ________________________________
> didongke@126.com
>
>
> From: Tilman Hausherr
> Date: 2023-04-20 14:32
> To: user
> Subject: Re: Tika server extraction failed
> Hi,
>
> I don't see why this is a problem, and you're mentioning the solution yourself. If you want detection by content, then don't pass the filename.
>
> Tilman
>
> On 20.04.2023 08:19, didongke@126.com wrote:
>
> Hi, Tilman
>
>     I have encountered another problem.
>
>     t1.xml is a simple plain text file, not a standard XML file.
>     When I use Tika Server 2.7.0 to extract file content, the results are as follows:
>
> curl -T t1.xml http://127.0.0.1:12000/tika --header "Accept: text/plain" -H "Content-Disposition: attachment; filename=t1.xml"
> Result: fail (empty)
>
> curl -T t1.xml http://127.0.0.1:12000/tika --header "Accept: text/plain"
> curl -T t1.xml http://127.0.0.1:12000/tika --header "Accept: text/plain" -H "Content-Disposition: attachment; filename=t1.txt"
> curl -T t1.xml http://127.0.0.1:12000/tika --header "Accept: text/plain" -H "Content-Disposition: attachment; filename=t1.docx"
> Result: success
>
>     The file name information affects the extraction result.
>
>
>

Re: Re: Tika server extraction failed

Posted by Tim Allison <ta...@apache.org>.
Tika's detector runs detection on the byte stream.  If the suffix of
the file is a subtype of the detected byte stream, then it applies
that subtype.

So, if you tell tika that your text file is an xml file, it will
detect the bytes as text, but then see that xml is a subtype of text
and run the XML parser against the file, leading to a parse exception.

If you use the /rmeta endpoint, you'll get better feedback about parse
exceptions.

On Thu, Apr 20, 2023 at 2:47 AM didongke@126.com <di...@126.com> wrote:
>
> Hi,
>
> I have encountered a problem before: some files cannot be detected based on their content.
> So I added the file name and solved the problem.
>
> But now I have another problem: adding a file name actually resulted in not being detected.
>
> If that's the case, I need to make two attempts:
> Try using content detection first,
> and then try using file name detection.
>
>
> ________________________________
> didongke@126.com
>
>
> From: Tilman Hausherr
> Date: 2023-04-20 14:32
> To: user
> Subject: Re: Tika server extraction failed
> Hi,
>
> I don't see why this is a problem, and you're mentioning the solution yourself. If you want detection by content, then don't pass the filename.
>
> Tilman
>
> On 20.04.2023 08:19, didongke@126.com wrote:
>
> Hi, Tilman
>
>     I have encountered another problem.
>
>     t1.xml is a simple plain text file, not a standard XML file.
>     When I use Tika Server 2.7.0 to extract file content, the results are as follows:
>
> curl -T t1.xml http://127.0.0.1:12000/tika --header "Accept: text/plain" -H "Content-Disposition: attachment; filename=t1.xml"
> Result: fail (empty)
>
> curl -T t1.xml http://127.0.0.1:12000/tika --header "Accept: text/plain"
> curl -T t1.xml http://127.0.0.1:12000/tika --header "Accept: text/plain" -H "Content-Disposition: attachment; filename=t1.txt"
> curl -T t1.xml http://127.0.0.1:12000/tika --header "Accept: text/plain" -H "Content-Disposition: attachment; filename=t1.docx"
> Result: success
>
>     The file name information affects the extraction result.
>
>
>

Re: Re: Tika server extraction failed

Posted by "didongke@126.com" <di...@126.com>.
Hi,

I have encountered a problem before: some files cannot be detected based on their content.
So I added the file name and solved the problem.

But now I have another problem: adding a file name actually resulted in not being detected.

If that's the case, I need to make two attempts:
Try using content detection first, 
and then try using file name detection.




didongke@126.com
 
From: Tilman Hausherr
Date: 2023-04-20 14:32
To: user
Subject: Re: Tika server extraction failed
Hi,

I don't see why this is a problem, and you're mentioning the solution yourself. If you want detection by content, then don't pass the filename.

Tilman

On 20.04.2023 08:19, didongke@126.com wrote:
Hi, Tilman

    I have encountered another problem.
    
    t1.xml is a simple plain text file, not a standard XML file.
    When I use Tika Server 2.7.0 to extract file content, the results are as follows:

curl -T t1.xml http://127.0.0.1:12000/tika --header "Accept: text/plain" -H "Content-Disposition: attachment; filename=t1.xml"
Result: fail (empty)    

curl -T t1.xml http://127.0.0.1:12000/tika --header "Accept: text/plain"
curl -T t1.xml http://127.0.0.1:12000/tika --header "Accept: text/plain" -H "Content-Disposition: attachment; filename=t1.txt"
curl -T t1.xml http://127.0.0.1:12000/tika --header "Accept: text/plain" -H "Content-Disposition: attachment; filename=t1.docx"
Result: success

    The file name information affects the extraction result.



Re: Tika server extraction failed

Posted by Tilman Hausherr <TH...@t-online.de>.
Hi,

I don't see why this is a problem, and you're mentioning the solution 
yourself. If you want detection by content, then don't pass the filename.

Tilman

On 20.04.2023 08:19, didongke@126.com wrote:
> Hi, Tilman
>
> I have encountered another problem.
>     t1.xml is a simple plain text file, not a standard XML file.
>     When I use Tika Server 2.7.0 to extract file content, the results 
> are as follows:
>
> curl -T t1.xml http://127.0.0.1:12000/tika --header "Accept: 
> text/plain" -H "Content-Disposition: attachment; filename=t1.xml"
> Result: fail (empty)
>
> curl -T t1.xml http://127.0.0.1:12000/tika --header "Accept: text/plain"
> curl -T t1.xml http://127.0.0.1:12000/tika --header "Accept: 
> text/plain" -H "Content-Disposition: attachment; filename=t1.txt"
> curl -T t1.xml http://127.0.0.1:12000/tika --header "Accept: 
> text/plain" -H "Content-Disposition: attachment; filename=t1.docx"
> Result: success
>
>     The file name information affects the extraction result.



Re: Re: Tika server extraction failed

Posted by "didongke@126.com" <di...@126.com>.
Hi, Tilman

    I have encountered another problem.
    
    t1.xml is a simple plain text file, not a standard XML file.
    When I use Tika Server 2.7.0 to extract file content, the results are as follows:

curl -T t1.xml http://127.0.0.1:12000/tika --header "Accept: text/plain" -H "Content-Disposition: attachment; filename=t1.xml"
Result: fail (empty)    

curl -T t1.xml http://127.0.0.1:12000/tika --header "Accept: text/plain"
curl -T t1.xml http://127.0.0.1:12000/tika --header "Accept: text/plain" -H "Content-Disposition: attachment; filename=t1.txt"
curl -T t1.xml http://127.0.0.1:12000/tika --header "Accept: text/plain" -H "Content-Disposition: attachment; filename=t1.docx"
Result: success

    The file name information affects the extraction result.



didongke@126.com
 
From: Tilman Hausherr
Date: 2023-04-20 11:09
To: user
Subject: Re: Tika server extraction failed
Yes, the second file brings this on the console log:
 
Caused by: org.apache.poi.util.RecordFormatException: Tried to allocate 
an array of length 100,000,015, but the maximum length for this record 
type is 100,000,000.
If the file is not corrupt and not large, please open an issue on 
bugzilla to request
increasing the maximum allowable size for this record type.
You can set a higher override value with IOUtils.setByteArrayMaxOverride()
        at org.apache.poi.util.IOUtils.throwRFE(IOUtils.java:599) 
~[tika-server-standard-2.7.1-SNAPSHOT.jar:2.7.1-SNAPSHOT]
        at org.apache.poi.util.IOUtils.checkLength(IOUtils.java:276) 
~[tika-server-standard-2.7.1-SNAPSHOT.jar:2.7.1-SNAPSHOT]
        at org.apache.poi.util.IOUtils.toByteArray(IOUtils.java:230) 
~[tika-server-standard-2.7.1-SNAPSHOT.jar:2.7.1-SNAPSHOT]
        at org.apache.poi.util.IOUtils.toByteArray(IOUtils.java:203) 
~[tika-server-standard-2.7.1-SNAPSHOT.jar:2.7.1-SNAPSHOT]
        at 
org.apache.poi.openxml4j.util.ZipArchiveFakeEntry.<init>(ZipArchiveFakeEntry.java:82) 
~[tika-server-standard-2.7.1-SNAPSHOT.jar:2.7.1-SNAPSHOT]
        at 
org.apache.poi.openxml4j.util.ZipInputStreamZipEntrySource.<init>(ZipInputStreamZipEntrySource.java:98) 
~[tika-server-standard-2.7.1-SNAPSHOT.jar:2.7.1-SNAPSHOT]
        at 
org.apache.poi.openxml4j.opc.ZipPackage.<init>(ZipPackage.java:132) 
~[tika-server-standard-2.7.1-SNAPSHOT.jar:2.7.1-SNAPSHOT]
        at 
org.apache.poi.openxml4j.opc.OPCPackage.open(OPCPackage.java:319) 
~[tika-server-standard-2.7.1-SNAPSHOT.jar:2.7.1-SNAPSHOT]
        at 
org.apache.tika.parser.microsoft.ooxml.OOXMLExtractorFactory.parse(OOXMLExtractorFactory.java:127) 
~[tika-server-standard-2.7.1-SNAPSHOT.jar:2.7.1-SNAPSHOT]
        at 
org.apache.tika.parser.microsoft.ooxml.OOXMLParser.parse(OOXMLParser.java:115) 
~[tika-server-standard-2.7.1-SNAPSHOT.jar:2.7.1-SNAPSHOT]
        at 
org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:298) 
~[tika-server-standard-2.7.1-SNAPSHOT.jar:2.7.1-SNAPSHOT]
        ... 41 more
 
So I googled for the error message and found this:
 
https://stackoverflow.com/a/64221068/535646
 
I then included this into the config.xml file from 
https://cwiki.apache.org/confluence/display/TIKA/TikaServer+in+Tika+2.x 
and then it works, although the meta output now came as xml instead of 
as text, maybe that default config file does change something instead of 
keeping defaults, but that's another story.
 
Tilman
 

Re: Re: Tika server extraction failed

Posted by "didongke@126.com" <di...@126.com>.
The problem has been solved.

Thank you.



didongke@126.com
 
From: Tilman Hausherr
Date: 2023-04-20 11:09
To: user
Subject: Re: Tika server extraction failed
Yes, the second file brings this on the console log:
 
Caused by: org.apache.poi.util.RecordFormatException: Tried to allocate 
an array of length 100,000,015, but the maximum length for this record 
type is 100,000,000.
If the file is not corrupt and not large, please open an issue on 
bugzilla to request
increasing the maximum allowable size for this record type.
You can set a higher override value with IOUtils.setByteArrayMaxOverride()
        at org.apache.poi.util.IOUtils.throwRFE(IOUtils.java:599) 
~[tika-server-standard-2.7.1-SNAPSHOT.jar:2.7.1-SNAPSHOT]
        at org.apache.poi.util.IOUtils.checkLength(IOUtils.java:276) 
~[tika-server-standard-2.7.1-SNAPSHOT.jar:2.7.1-SNAPSHOT]
        at org.apache.poi.util.IOUtils.toByteArray(IOUtils.java:230) 
~[tika-server-standard-2.7.1-SNAPSHOT.jar:2.7.1-SNAPSHOT]
        at org.apache.poi.util.IOUtils.toByteArray(IOUtils.java:203) 
~[tika-server-standard-2.7.1-SNAPSHOT.jar:2.7.1-SNAPSHOT]
        at 
org.apache.poi.openxml4j.util.ZipArchiveFakeEntry.<init>(ZipArchiveFakeEntry.java:82) 
~[tika-server-standard-2.7.1-SNAPSHOT.jar:2.7.1-SNAPSHOT]
        at 
org.apache.poi.openxml4j.util.ZipInputStreamZipEntrySource.<init>(ZipInputStreamZipEntrySource.java:98) 
~[tika-server-standard-2.7.1-SNAPSHOT.jar:2.7.1-SNAPSHOT]
        at 
org.apache.poi.openxml4j.opc.ZipPackage.<init>(ZipPackage.java:132) 
~[tika-server-standard-2.7.1-SNAPSHOT.jar:2.7.1-SNAPSHOT]
        at 
org.apache.poi.openxml4j.opc.OPCPackage.open(OPCPackage.java:319) 
~[tika-server-standard-2.7.1-SNAPSHOT.jar:2.7.1-SNAPSHOT]
        at 
org.apache.tika.parser.microsoft.ooxml.OOXMLExtractorFactory.parse(OOXMLExtractorFactory.java:127) 
~[tika-server-standard-2.7.1-SNAPSHOT.jar:2.7.1-SNAPSHOT]
        at 
org.apache.tika.parser.microsoft.ooxml.OOXMLParser.parse(OOXMLParser.java:115) 
~[tika-server-standard-2.7.1-SNAPSHOT.jar:2.7.1-SNAPSHOT]
        at 
org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:298) 
~[tika-server-standard-2.7.1-SNAPSHOT.jar:2.7.1-SNAPSHOT]
        ... 41 more
 
So I googled for the error message and found this:
 
https://stackoverflow.com/a/64221068/535646
 
I then included this into the config.xml file from 
https://cwiki.apache.org/confluence/display/TIKA/TikaServer+in+Tika+2.x 
and then it works, although the meta output now came as xml instead of 
as text, maybe that default config file does change something instead of 
keeping defaults, but that's another story.
 
Tilman
 

Re: Tika server extraction failed

Posted by Tilman Hausherr <TH...@t-online.de>.
Yes, the second file brings this on the console log:

Caused by: org.apache.poi.util.RecordFormatException: Tried to allocate 
an array of length 100,000,015, but the maximum length for this record 
type is 100,000,000.
If the file is not corrupt and not large, please open an issue on 
bugzilla to request
increasing the maximum allowable size for this record type.
You can set a higher override value with IOUtils.setByteArrayMaxOverride()
         at org.apache.poi.util.IOUtils.throwRFE(IOUtils.java:599) 
~[tika-server-standard-2.7.1-SNAPSHOT.jar:2.7.1-SNAPSHOT]
         at org.apache.poi.util.IOUtils.checkLength(IOUtils.java:276) 
~[tika-server-standard-2.7.1-SNAPSHOT.jar:2.7.1-SNAPSHOT]
         at org.apache.poi.util.IOUtils.toByteArray(IOUtils.java:230) 
~[tika-server-standard-2.7.1-SNAPSHOT.jar:2.7.1-SNAPSHOT]
         at org.apache.poi.util.IOUtils.toByteArray(IOUtils.java:203) 
~[tika-server-standard-2.7.1-SNAPSHOT.jar:2.7.1-SNAPSHOT]
         at 
org.apache.poi.openxml4j.util.ZipArchiveFakeEntry.<init>(ZipArchiveFakeEntry.java:82) 
~[tika-server-standard-2.7.1-SNAPSHOT.jar:2.7.1-SNAPSHOT]
         at 
org.apache.poi.openxml4j.util.ZipInputStreamZipEntrySource.<init>(ZipInputStreamZipEntrySource.java:98) 
~[tika-server-standard-2.7.1-SNAPSHOT.jar:2.7.1-SNAPSHOT]
         at 
org.apache.poi.openxml4j.opc.ZipPackage.<init>(ZipPackage.java:132) 
~[tika-server-standard-2.7.1-SNAPSHOT.jar:2.7.1-SNAPSHOT]
         at 
org.apache.poi.openxml4j.opc.OPCPackage.open(OPCPackage.java:319) 
~[tika-server-standard-2.7.1-SNAPSHOT.jar:2.7.1-SNAPSHOT]
         at 
org.apache.tika.parser.microsoft.ooxml.OOXMLExtractorFactory.parse(OOXMLExtractorFactory.java:127) 
~[tika-server-standard-2.7.1-SNAPSHOT.jar:2.7.1-SNAPSHOT]
         at 
org.apache.tika.parser.microsoft.ooxml.OOXMLParser.parse(OOXMLParser.java:115) 
~[tika-server-standard-2.7.1-SNAPSHOT.jar:2.7.1-SNAPSHOT]
         at 
org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:298) 
~[tika-server-standard-2.7.1-SNAPSHOT.jar:2.7.1-SNAPSHOT]
         ... 41 more

So I googled for the error message and found this:

https://stackoverflow.com/a/64221068/535646

I then included this into the config.xml file from 
https://cwiki.apache.org/confluence/display/TIKA/TikaServer+in+Tika+2.x 
and then it works, although the meta output now came as xml instead of 
as text, maybe that default config file does change something instead of 
keeping defaults, but that's another story.

Tilman



Re: Re: Tika server extraction failed

Posted by "didongke@126.com" <di...@126.com>.
Sorry,I forgot to say: I have tried Tika-app 2.3.0 and can successfully extract text content.

The problem only occurs on Tika server.



didongke@126.com
 
From: Tilman Hausherr
Date: 2023-04-20 02:28
To: user
Subject: Re: Tika server extraction failed
Please update to the current version, 2.7.0. I was able to process both files, however I used tika-app in non interactive mode.

If the problem still happens: what happens if you do the second file first?

Tilman

On 19.04.2023 04:07, didongke@126.com wrote:
Hi,

    I have two files 111.xlsx and 2222.xlsx ( included in the compressed package).

    I  use Tika server (2.3.0 to extract their metadata and text content :
    curl -T 1111.xlsx http://127.0.0.1:12000/meta
    curl -T 222.xlsx http://127.0.0.1:12000/meta

    However, one extraction (1111.xlsx) was successful, while the other (2222.xlsx) extraction failed.
    2222. xlsx only has one more row of data than 1111. xlsx.

    The same is true for extracting text data.
    curl -T 1111.xlsx http://127.0.0.1:12000/tika --header "Accept: text/plain" -H "Content-Disposition: attachment; filename=1111.xlsx"    
    curl -T 2222.xlsx http://127.0.0.1:12000/tika --header "Accept: text/plain" -H "Content-Disposition: attachment; filename=2222.xlsx"   

Please assist !



didongke@126.com


Re: Tika server extraction failed

Posted by Tilman Hausherr <TH...@t-online.de>.
Please update to the current version, 2.7.0. I was able to process both 
files, however I used tika-app in non interactive mode.

If the problem still happens: what happens if you do the second file first?

Tilman

On 19.04.2023 04:07, didongke@126.com wrote:
> Hi,
>
> I have two files 111.xlsx and 2222.xlsx ( included in the compressed 
> package).
>
>     I  use Tika server (2.3.0 to extract their metadata and text 
> content :
> curl -T 1111.xlsx http://127.0.0.1:12000/meta
> curl -T 222.xlsx http://127.0.0.1:12000/meta
>
> However, one extraction (1111.xlsx) was successful, while the other 
> (2222.xlsx) extraction failed.
>     2222. xlsx only has one more row of data than 1111. xlsx.
>
>     The same is true for extracting text data.
> curl -T 1111.xlsx http://127.0.0.1:12000/tika --header "Accept: 
> text/plain" -H "Content-Disposition: attachment; filename=1111.xlsx"
> curl -T 2222.xlsx http://127.0.0.1:12000/tika --header "Accept: 
> text/plain" -H "Content-Disposition: attachment; filename=2222.xlsx"
>
> Please assist !
>
> ------------------------------------------------------------------------
> didongke@126.com