You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tika.apache.org by "Nicolò Mendola (Jira)" <ji...@apache.org> on 2023/07/13 21:20:00 UTC
[jira] [Comment Edited] (TIKA-4064) Update to 2.8.1
[ https://issues.apache.org/jira/browse/TIKA-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17742957#comment-17742957 ]
Nicolò Mendola edited comment on TIKA-4064 at 7/13/23 9:19 PM:
---------------------------------------------------------------
Just out of interest, is there an eta for release 2.8.1 to be published?
Best regards
was (Author: JIRAUSER296595):
Just out of interest, is there an eta for release 2.8.1 to be published?
Best regards
> Update to 2.8.1
> ---------------
>
> Key: TIKA-4064
> URL: https://issues.apache.org/jira/browse/TIKA-4064
> Project: Tika
> Issue Type: Task
> Components: build
> Affects Versions: 2.8.0
> Reporter: Tilman Hausherr
> Priority: Minor
> Fix For: 2.8.1
>
>
> The latest maven versions plugin finds much more outdated stuff than the previous one, so I'll do a few updates.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: Next release?
Posted by Tilman Hausherr <TH...@t-online.de>.
Btw while you're working on that subproject, would it be possible to add
a filter on top of the excel result files? I think this helps everybody
and doesn't have any downsides.
https://stackoverflow.com/a/13703572/535646
Another thing would be to "fix window", but there I couldn't find any
code and the tastes might be different depending on each user.
Tilman
On 21.07.2023 03:48, Tilman Hausherr wrote:
> On 20.07.2023 17:39, Tim Allison wrote:
>> How about this:
>> https://corpora.tika.apache.org/base/reports/tika-2.8.1-pre-rc1-v3.tgz
>
>
> Thank you this looks much better.
>
> Now the the last file in new_exceptions_in_B_details.xlsx , r-cuda.bmp
> . When doing a drag and drop on tika-app, I SOMETIMES get the NPE. If
> I call tika app with the file from the command line, I don't.
>
> Btw why this for a bmp file?
>
> tiff:BitsPerSample: 5 65
> tiff:ImageLength: 384
> tiff:ImageWidth: 12
>
>
>
Re: Next release?
Posted by Tim Allison <ta...@apache.org>.
tika-eval misaligned the attachments on that file :(
https://corpora.tika.apache.org/base/reports/tika-2.8.1-pre-rc1-v4.tgz
This also happened to: govdocs1/912/912801.ppt
https://issues.apache.org/jira/browse/TIKA-4103
On Thu, Jul 20, 2023 at 11:39 PM Tilman Hausherr <TH...@t-online.de>
wrote:
> I wanted to have a closer look at one of the files that claims to have
> less output:
>
> govdocs1/477/477727.ppt MBD0104A5C8.doc
>
> so I ran tika-app 2.8.0 and 2.8.1 snapshot and got the same output. But
> according to the excel table I should have these more in B
> (TOP_10_MORE_IN_B column):
>
> and: 3 | land: 3 | micro: 3 | reuse: 3 | sprinkler: 3 | water: 3 |
> capture: 2 | collect: 2 | considered: 2 | leveled: 2
>
> I can't find "sprinkler" in the text below nor in the WORD file.
>
> <?xml version="1.0" encoding="UTF-8"?><html
> xmlns="http://www.w3.org/1999/xhtml">
> <head>
> <meta name="cp:revision" content="3"/>
> <meta name="meta:word-count" content="322"/>
> <meta name="extended-properties:Application" content="Microsoft Word
> 10.0"/>
> <meta name="meta:last-author" content="Carolyn.Jones"/>
> <meta name="dc:creator" content="NWCC"/>
> <meta name="extended-properties:Company" content="USDA"/>
> <meta name="xmpTPg:NPages" content="1"/>
> <meta name="resourceName" content="MBD0104A5C8.doc"/>
> <meta name="dcterms:created" content="2005-01-06T20:39:00Z"/>
> <meta name="dcterms:modified" content="2005-03-29T22:02:00Z"/>
> <meta name="meta:character-count" content="1836"/>
> <meta name="extended-properties:Template" content="Normal.dot"/>
> <meta name="X-TIKA:Parsed-By"
> content="org.apache.tika.parser.DefaultParser"/>
> <meta name="X-TIKA:Parsed-By"
> content="org.apache.tika.parser.microsoft.OfficeParser"/>
> <meta name="extended-properties:TotalTime" content="3600000000"/>
> <meta name="Content-Length" content="22528"/>
> <meta name="meta:page-count" content="1"/>
> <meta name="Content-Type" content="application/msword"/>
> <title/>
> </head>
> <body><p><b>Conservation Security Program (CSP)
> </b></p>
> <p><b>Irrigation Enhancement Index Tool</b></p>
> <p>This tool is designed to help landowners conduct a self assessment of
> their eligibility for payment for enhanced irrigation systems in the
> Conservation Security Program. It may also serve as a means of
> documenting irrigation system components that can be utilized during
> individual interviews.
> </p>
> <p>This procedure is to be utilized on irrigated lands eligible for CSP
> and will result in assigning an Irrigation Enhancement Index value to
> the irrigation system being evaluated.
> </p>
> <p>This procedure starts with a base value that is assigned to the
> specific type of irrigation system in use. Systems that commonly have
> higher irrigation efficiencies and/or are easier to manage are assigned
> higher values. Modifiers are applied based on the level of management
> and the efficiency of the on-farm water delivery system. A bonus is
> given if runoff from the irrigated field is captured for re-use.</p>
> <p>The final calculation will require a value of the Soil Condition
> Index (SCI) multiplier. The exact value of this multiplier will be
> provided to you when NRCS staff computes your final SCI during your
> interview. The multiplier will be a value from 0.9 to 1.0 depending on
> your SCI.</p>
> <p>This self assessment is simple and should take less than 5 minutes to
> complete. A basic hand calculator is recommended. In addition, basic
> knowledge of the irrigation system and management practices in use is
> necessary. Definitions of the various terms are included in this tool.</p>
> <p>When the self assessment is complete, the landowner will have
> calculated an Irrigation Enhancement Index value for the irrigation
> system. The Irrigation Enhancement Index is not an efficiency number,
> but rather an indicator of how well the system may perform. If the
> Irrigation Enhancement Index value is 50 or more, the landowner may be
> eligible for CSP payments. If the Irrigation Enhancement Index value is
> less than 50, the applicant should consider utilizing other USDA
> programs to improve the irrigation system. If the Irrigation
> Enhancement Index is 60 or greater, the applicant may be eligible for
> increased payments.
> </p>
> </body></html>
>
Re: Next release?
Posted by Tilman Hausherr <TH...@t-online.de>.
I wanted to have a closer look at one of the files that claims to have
less output:
govdocs1/477/477727.ppt MBD0104A5C8.doc
so I ran tika-app 2.8.0 and 2.8.1 snapshot and got the same output. But
according to the excel table I should have these more in B
(TOP_10_MORE_IN_B column):
and: 3 | land: 3 | micro: 3 | reuse: 3 | sprinkler: 3 | water: 3 |
capture: 2 | collect: 2 | considered: 2 | leveled: 2
I can't find "sprinkler" in the text below nor in the WORD file.
<?xml version="1.0" encoding="UTF-8"?><html
xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta name="cp:revision" content="3"/>
<meta name="meta:word-count" content="322"/>
<meta name="extended-properties:Application" content="Microsoft Word 10.0"/>
<meta name="meta:last-author" content="Carolyn.Jones"/>
<meta name="dc:creator" content="NWCC"/>
<meta name="extended-properties:Company" content="USDA"/>
<meta name="xmpTPg:NPages" content="1"/>
<meta name="resourceName" content="MBD0104A5C8.doc"/>
<meta name="dcterms:created" content="2005-01-06T20:39:00Z"/>
<meta name="dcterms:modified" content="2005-03-29T22:02:00Z"/>
<meta name="meta:character-count" content="1836"/>
<meta name="extended-properties:Template" content="Normal.dot"/>
<meta name="X-TIKA:Parsed-By"
content="org.apache.tika.parser.DefaultParser"/>
<meta name="X-TIKA:Parsed-By"
content="org.apache.tika.parser.microsoft.OfficeParser"/>
<meta name="extended-properties:TotalTime" content="3600000000"/>
<meta name="Content-Length" content="22528"/>
<meta name="meta:page-count" content="1"/>
<meta name="Content-Type" content="application/msword"/>
<title/>
</head>
<body><p><b>Conservation Security Program (CSP)
</b></p>
<p><b>Irrigation Enhancement Index Tool</b></p>
<p>This tool is designed to help landowners conduct a self assessment of
their eligibility for payment for enhanced irrigation systems in the
Conservation Security Program. It may also serve as a means of
documenting irrigation system components that can be utilized during
individual interviews.
</p>
<p>This procedure is to be utilized on irrigated lands eligible for CSP
and will result in assigning an Irrigation Enhancement Index value to
the irrigation system being evaluated.
</p>
<p>This procedure starts with a base value that is assigned to the
specific type of irrigation system in use. Systems that commonly have
higher irrigation efficiencies and/or are easier to manage are assigned
higher values. Modifiers are applied based on the level of management
and the efficiency of the on-farm water delivery system. A bonus is
given if runoff from the irrigated field is captured for re-use.</p>
<p>The final calculation will require a value of the Soil Condition
Index (SCI) multiplier. The exact value of this multiplier will be
provided to you when NRCS staff computes your final SCI during your
interview. The multiplier will be a value from 0.9 to 1.0 depending on
your SCI.</p>
<p>This self assessment is simple and should take less than 5 minutes to
complete. A basic hand calculator is recommended. In addition, basic
knowledge of the irrigation system and management practices in use is
necessary. Definitions of the various terms are included in this tool.</p>
<p>When the self assessment is complete, the landowner will have
calculated an Irrigation Enhancement Index value for the irrigation
system. The Irrigation Enhancement Index is not an efficiency number,
but rather an indicator of how well the system may perform. If the
Irrigation Enhancement Index value is 50 or more, the landowner may be
eligible for CSP payments. If the Irrigation Enhancement Index value is
less than 50, the applicant should consider utilizing other USDA
programs to improve the irrigation system. If the Irrigation
Enhancement Index is 60 or greater, the applicant may be eligible for
increased payments.
</p>
</body></html>
Re: Next release?
Posted by Tilman Hausherr <TH...@t-online.de>.
On 25.07.2023 21:13, Tim Allison wrote:
> New reports are here:
> https://corpora.tika.apache.org/base/reports/tika-2.8.1-prerc1-b.tgz
>
> The alignment in tika-eval is still not working as planned, but the content
> looks ok...more work to do on tika-eval.
>
> We're getting many more exceptions with gz, and I noticed that we're also
> getting many fewer attachments in some OLE2 based Office files. Both items
> were happening in the last reports I ran...as I look back.
Agreed. I l always look at the TOP_10_MORE_IN_A column in the content
file, and the only occurrences where there's more look like something
got switched around in the same ppt file.
Tilman
Re: Next release?
Posted by Tim Allison <ta...@apache.org>.
New reports are here:
https://corpora.tika.apache.org/base/reports/tika-2.8.1-prerc1-b.tgz
The alignment in tika-eval is still not working as planned, but the content
looks ok...more work to do on tika-eval.
We're getting many more exceptions with gz, and I noticed that we're also
getting many fewer attachments in some OLE2 based Office files. Both items
were happening in the last reports I ran...as I look back.
On Fri, Jul 21, 2023 at 10:07 AM Tim Allison <ta...@apache.org> wrote:
> LOL...I knew about this 2 years ago:
> https://github.com/jai-imageio/jai-imageio-core/issues/57#issuecomment-870863133
>
> On Fri, Jul 21, 2023 at 6:17 AM Tim Allison <ta...@apache.org> wrote:
>
>> I'm still trying to figure this one out. I also noticed "sometimes a
>> problem and sometimes not."
>>
>> I did find that if the sun/java bmp parser is selected then there's no
>> NPE. If the jaiimageio BMP reader is selected, there's an NPE.
>>
>> The NPE comes from missing a "properties" file in the jaiimageio
>> library. I'm trying to figure out if that's a Tika packaging problem or
>> something else.
>>
>> On Thu, Jul 20, 2023 at 9:48 PM Tilman Hausherr <TH...@t-online.de>
>> wrote:
>>
>>> On 20.07.2023 17:39, Tim Allison wrote:
>>> > How about this:
>>> > https://corpora.tika.apache.org/base/reports/tika-2.8.1-pre-rc1-v3.tgz
>>>
>>>
>>> Thank you this looks much better.
>>>
>>> Now the the last file in new_exceptions_in_B_details.xlsx , r-cuda.bmp .
>>> When doing a drag and drop on tika-app, I SOMETIMES get the NPE. If I
>>> call tika app with the file from the command line, I don't.
>>>
>>> Btw why this for a bmp file?
>>>
>>> tiff:BitsPerSample: 5 65
>>> tiff:ImageLength: 384
>>> tiff:ImageWidth: 12
>>>
>>>
>>>
>>>
Re: Next release?
Posted by Tim Allison <ta...@apache.org>.
LOL...I knew about this 2 years ago:
https://github.com/jai-imageio/jai-imageio-core/issues/57#issuecomment-870863133
On Fri, Jul 21, 2023 at 6:17 AM Tim Allison <ta...@apache.org> wrote:
> I'm still trying to figure this one out. I also noticed "sometimes a
> problem and sometimes not."
>
> I did find that if the sun/java bmp parser is selected then there's no
> NPE. If the jaiimageio BMP reader is selected, there's an NPE.
>
> The NPE comes from missing a "properties" file in the jaiimageio library.
> I'm trying to figure out if that's a Tika packaging problem or something
> else.
>
> On Thu, Jul 20, 2023 at 9:48 PM Tilman Hausherr <TH...@t-online.de>
> wrote:
>
>> On 20.07.2023 17:39, Tim Allison wrote:
>> > How about this:
>> > https://corpora.tika.apache.org/base/reports/tika-2.8.1-pre-rc1-v3.tgz
>>
>>
>> Thank you this looks much better.
>>
>> Now the the last file in new_exceptions_in_B_details.xlsx , r-cuda.bmp .
>> When doing a drag and drop on tika-app, I SOMETIMES get the NPE. If I
>> call tika app with the file from the command line, I don't.
>>
>> Btw why this for a bmp file?
>>
>> tiff:BitsPerSample: 5 65
>> tiff:ImageLength: 384
>> tiff:ImageWidth: 12
>>
>>
>>
>>
Re: Next release?
Posted by Tim Allison <ta...@apache.org>.
I'm still trying to figure this one out. I also noticed "sometimes a
problem and sometimes not."
I did find that if the sun/java bmp parser is selected then there's no
NPE. If the jaiimageio BMP reader is selected, there's an NPE.
The NPE comes from missing a "properties" file in the jaiimageio library.
I'm trying to figure out if that's a Tika packaging problem or something
else.
On Thu, Jul 20, 2023 at 9:48 PM Tilman Hausherr <TH...@t-online.de>
wrote:
> On 20.07.2023 17:39, Tim Allison wrote:
> > How about this:
> > https://corpora.tika.apache.org/base/reports/tika-2.8.1-pre-rc1-v3.tgz
>
>
> Thank you this looks much better.
>
> Now the the last file in new_exceptions_in_B_details.xlsx , r-cuda.bmp .
> When doing a drag and drop on tika-app, I SOMETIMES get the NPE. If I
> call tika app with the file from the command line, I don't.
>
> Btw why this for a bmp file?
>
> tiff:BitsPerSample: 5 65
> tiff:ImageLength: 384
> tiff:ImageWidth: 12
>
>
>
>
Re: Next release?
Posted by Tilman Hausherr <TH...@t-online.de>.
On 20.07.2023 17:39, Tim Allison wrote:
> How about this:
> https://corpora.tika.apache.org/base/reports/tika-2.8.1-pre-rc1-v3.tgz
Thank you this looks much better.
Now the the last file in new_exceptions_in_B_details.xlsx , r-cuda.bmp .
When doing a drag and drop on tika-app, I SOMETIMES get the NPE. If I
call tika app with the file from the command line, I don't.
Btw why this for a bmp file?
tiff:BitsPerSample: 5 65
tiff:ImageLength: 384
tiff:ImageWidth: 12
Re: Next release?
Posted by Tim Allison <ta...@apache.org>.
How about this:
https://corpora.tika.apache.org/base/reports/tika-2.8.1-pre-rc1-v3.tgz
On Thu, Jul 20, 2023 at 11:00 AM Tim Allison <ta...@apache.org> wrote:
> How about these:
> https://corpora.tika.apache.org/base/reports/tika-2.8.1-pre-rc1-v2.tgz
>
> That adds the "file_name", which is either the container file name or, in
> the case of an embedded file, Tika's best guess as to what the container
> file named the embedded file. The "file_name" is problematic because it is
> "user defined data" and can be messy or malicious. We should include the
> embedded file id path, which is just numbers and slashes, but that'll take
> a bit more work.
>
> So, let me know if this helps any...
>
> On Thu, Jul 20, 2023 at 9:32 AM Tim Allison <ta...@apache.org> wrote:
>
>> Y, again, those are embedded files. I'll add this report to the ticket
>> as well to include embedded resource path. Thank you!
>>
>> On Wed, Jul 19, 2023 at 11:56 PM Tilman Hausherr <TH...@t-online.de>
>> wrote:
>>
>>> On 19.07.2023 19:19, Tim Allison wrote:
>>> > Results are here:
>>> > https://corpora.tika.apache.org/base/reports/tika-2.8.1-pre-rc1.tgz
>>>
>>>
>>>
>>> govdocs1/974/974098.ppt
>>>
>>> appears twice in the content_diffs_no_exceptions.xlsx file but with
>>> different content?!
>>>
>>>
>>> govdocs1/974/974098.ppt 304128 application/msword
>>> application/vnd.ms-excel 44 567 66 1421 eng
>>> 48 eng 39 -9 the:
>>> 6 | of: 5 | standard: 4 | 90: 3 | energy: 3 | 75: 2 | a: 2 | and: 2 |
>>> by: 2 | criteria: 2 330: 42 | 21008: 12 | 24371: 12 | 29977: 12 |
>>> energy: 9 | lcc: 8 | 1: 6 | 10: 6 | 11: 6 | 12: 6 BASIC_LATIN: 437
>>> BASIC_LATIN: 7140 the: 6 | of: 5 | standard: 4 | a: 2 | and: 2 |
>>> by: 2
>>> | criteria: 2 | technical: 2 | 1980: 1 | adopted: 1 330: 42 | 21008:
>>> 12
>>> | 24371: 12 | 29977: 12 | lcc: 8 | 1: 6 | 10: 6 | 11: 6 | 12: 6 | 13: 6
>>> the: 6 | of: 5 | standard: 4 | a: 2 | and: 2 | by: 2 | criteria: 2 |
>>> technical: 2 | 1980: 1 | adopted: 1 330: 42 | 21008: 12 | 24371: 12
>>> |
>>> 29977: 12 | lcc: 8 | 1: 6 | 10: 6 | 11: 6 | 12: 6 | 13: 6 0,013
>>> 0,012
>>>
>>> govdocs1/974/974098.ppt 304128 application/vnd.ms-excel
>>> application/vnd.ms-excel 567 148 1421 262 eng
>>> 39 eng 21 -18
>>> 330: 42 | 21008: 12 | 24371: 12 | 29977: 12 | energy: 9 | lcc: 8 | 1: 6
>>> | 10: 6 | 11: 6 | 12: 6 1: 10 | 30: 7 | 25: 5 | 218.06175: 4 |
>>> 436.1235: 4 | 654.18525: 4 | 872.247: 4 | energy: 4 | national: 4 | 50:
>>> 3 BASIC_LATIN: 7140 BASIC_LATIN: 1717 330: 42 | 21008:
>>> 12 | 24371: 12
>>> | 29977: 12 | lcc: 8 | 31: 6 | 32: 6 | 33: 6 | 34: 6 | 35: 6
>>> 218.06175:
>>> 4 | 436.1235: 4 | 654.18525: 4 | 872.247: 4 | national: 4 | 90.1: 3 |
>>> ashrae: 3 | 1265.469: 2 | 141.157: 2 | 149.503: 2 330: 42 | 21008:
>>> 12 |
>>> 24371: 12 | 29977: 12 | lcc: 8 | 31: 6 | 32: 6 | 33: 6 | 34: 6 | 35: 6
>>> 1: 4 | 218.06175: 4 | 436.1235: 4 | 654.18525: 4 | 872.247: 4 |
>>> national: 4 | 90.1: 3 | ashrae: 3 | 1265.469: 2 | 141.157: 2 0,092
>>> 0,096
>>>
>>>
>>>
Re: Next release?
Posted by Tim Allison <ta...@apache.org>.
How about these:
https://corpora.tika.apache.org/base/reports/tika-2.8.1-pre-rc1-v2.tgz
That adds the "file_name", which is either the container file name or, in
the case of an embedded file, Tika's best guess as to what the container
file named the embedded file. The "file_name" is problematic because it is
"user defined data" and can be messy or malicious. We should include the
embedded file id path, which is just numbers and slashes, but that'll take
a bit more work.
So, let me know if this helps any...
On Thu, Jul 20, 2023 at 9:32 AM Tim Allison <ta...@apache.org> wrote:
> Y, again, those are embedded files. I'll add this report to the ticket as
> well to include embedded resource path. Thank you!
>
> On Wed, Jul 19, 2023 at 11:56 PM Tilman Hausherr <TH...@t-online.de>
> wrote:
>
>> On 19.07.2023 19:19, Tim Allison wrote:
>> > Results are here:
>> > https://corpora.tika.apache.org/base/reports/tika-2.8.1-pre-rc1.tgz
>>
>>
>>
>> govdocs1/974/974098.ppt
>>
>> appears twice in the content_diffs_no_exceptions.xlsx file but with
>> different content?!
>>
>>
>> govdocs1/974/974098.ppt 304128 application/msword
>> application/vnd.ms-excel 44 567 66 1421 eng
>> 48 eng 39 -9 the:
>> 6 | of: 5 | standard: 4 | 90: 3 | energy: 3 | 75: 2 | a: 2 | and: 2 |
>> by: 2 | criteria: 2 330: 42 | 21008: 12 | 24371: 12 | 29977: 12 |
>> energy: 9 | lcc: 8 | 1: 6 | 10: 6 | 11: 6 | 12: 6 BASIC_LATIN: 437
>> BASIC_LATIN: 7140 the: 6 | of: 5 | standard: 4 | a: 2 | and: 2 |
>> by: 2
>> | criteria: 2 | technical: 2 | 1980: 1 | adopted: 1 330: 42 | 21008:
>> 12
>> | 24371: 12 | 29977: 12 | lcc: 8 | 1: 6 | 10: 6 | 11: 6 | 12: 6 | 13: 6
>> the: 6 | of: 5 | standard: 4 | a: 2 | and: 2 | by: 2 | criteria: 2 |
>> technical: 2 | 1980: 1 | adopted: 1 330: 42 | 21008: 12 | 24371: 12 |
>> 29977: 12 | lcc: 8 | 1: 6 | 10: 6 | 11: 6 | 12: 6 | 13: 6 0,013
>> 0,012
>>
>> govdocs1/974/974098.ppt 304128 application/vnd.ms-excel
>> application/vnd.ms-excel 567 148 1421 262 eng
>> 39 eng 21 -18
>> 330: 42 | 21008: 12 | 24371: 12 | 29977: 12 | energy: 9 | lcc: 8 | 1: 6
>> | 10: 6 | 11: 6 | 12: 6 1: 10 | 30: 7 | 25: 5 | 218.06175: 4 |
>> 436.1235: 4 | 654.18525: 4 | 872.247: 4 | energy: 4 | national: 4 | 50:
>> 3 BASIC_LATIN: 7140 BASIC_LATIN: 1717 330: 42 | 21008:
>> 12 | 24371: 12
>> | 29977: 12 | lcc: 8 | 31: 6 | 32: 6 | 33: 6 | 34: 6 | 35: 6
>> 218.06175:
>> 4 | 436.1235: 4 | 654.18525: 4 | 872.247: 4 | national: 4 | 90.1: 3 |
>> ashrae: 3 | 1265.469: 2 | 141.157: 2 | 149.503: 2 330: 42 | 21008:
>> 12 |
>> 24371: 12 | 29977: 12 | lcc: 8 | 31: 6 | 32: 6 | 33: 6 | 34: 6 | 35: 6
>> 1: 4 | 218.06175: 4 | 436.1235: 4 | 654.18525: 4 | 872.247: 4 |
>> national: 4 | 90.1: 3 | ashrae: 3 | 1265.469: 2 | 141.157: 2 0,092
>> 0,096
>>
>>
>>
Re: Next release?
Posted by Tim Allison <ta...@apache.org>.
Y, again, those are embedded files. I'll add this report to the ticket as
well to include embedded resource path. Thank you!
On Wed, Jul 19, 2023 at 11:56 PM Tilman Hausherr <TH...@t-online.de>
wrote:
> On 19.07.2023 19:19, Tim Allison wrote:
> > Results are here:
> > https://corpora.tika.apache.org/base/reports/tika-2.8.1-pre-rc1.tgz
>
>
>
> govdocs1/974/974098.ppt
>
> appears twice in the content_diffs_no_exceptions.xlsx file but with
> different content?!
>
>
> govdocs1/974/974098.ppt 304128 application/msword
> application/vnd.ms-excel 44 567 66 1421 eng
> 48 eng 39 -9 the:
> 6 | of: 5 | standard: 4 | 90: 3 | energy: 3 | 75: 2 | a: 2 | and: 2 |
> by: 2 | criteria: 2 330: 42 | 21008: 12 | 24371: 12 | 29977: 12 |
> energy: 9 | lcc: 8 | 1: 6 | 10: 6 | 11: 6 | 12: 6 BASIC_LATIN: 437
> BASIC_LATIN: 7140 the: 6 | of: 5 | standard: 4 | a: 2 | and: 2 | by:
> 2
> | criteria: 2 | technical: 2 | 1980: 1 | adopted: 1 330: 42 | 21008:
> 12
> | 24371: 12 | 29977: 12 | lcc: 8 | 1: 6 | 10: 6 | 11: 6 | 12: 6 | 13: 6
> the: 6 | of: 5 | standard: 4 | a: 2 | and: 2 | by: 2 | criteria: 2 |
> technical: 2 | 1980: 1 | adopted: 1 330: 42 | 21008: 12 | 24371: 12 |
> 29977: 12 | lcc: 8 | 1: 6 | 10: 6 | 11: 6 | 12: 6 | 13: 6 0,013
> 0,012
>
> govdocs1/974/974098.ppt 304128 application/vnd.ms-excel
> application/vnd.ms-excel 567 148 1421 262 eng
> 39 eng 21 -18
> 330: 42 | 21008: 12 | 24371: 12 | 29977: 12 | energy: 9 | lcc: 8 | 1: 6
> | 10: 6 | 11: 6 | 12: 6 1: 10 | 30: 7 | 25: 5 | 218.06175: 4 |
> 436.1235: 4 | 654.18525: 4 | 872.247: 4 | energy: 4 | national: 4 | 50:
> 3 BASIC_LATIN: 7140 BASIC_LATIN: 1717 330: 42 | 21008:
> 12 | 24371: 12
> | 29977: 12 | lcc: 8 | 31: 6 | 32: 6 | 33: 6 | 34: 6 | 35: 6 218.06175:
> 4 | 436.1235: 4 | 654.18525: 4 | 872.247: 4 | national: 4 | 90.1: 3 |
> ashrae: 3 | 1265.469: 2 | 141.157: 2 | 149.503: 2 330: 42 | 21008:
> 12 |
> 24371: 12 | 29977: 12 | lcc: 8 | 31: 6 | 32: 6 | 33: 6 | 34: 6 | 35: 6
> 1: 4 | 218.06175: 4 | 436.1235: 4 | 654.18525: 4 | 872.247: 4 |
> national: 4 | 90.1: 3 | ashrae: 3 | 1265.469: 2 | 141.157: 2 0,092
> 0,096
>
>
>
Re: Next release?
Posted by Tilman Hausherr <TH...@t-online.de>.
On 19.07.2023 19:19, Tim Allison wrote:
> Results are here:
> https://corpora.tika.apache.org/base/reports/tika-2.8.1-pre-rc1.tgz
govdocs1/974/974098.ppt
appears twice in the content_diffs_no_exceptions.xlsx file but with
different content?!
govdocs1/974/974098.ppt 304128 application/msword
application/vnd.ms-excel 44 567 66 1421 eng 48 eng 39 -9 the:
6 | of: 5 | standard: 4 | 90: 3 | energy: 3 | 75: 2 | a: 2 | and: 2 |
by: 2 | criteria: 2 330: 42 | 21008: 12 | 24371: 12 | 29977: 12 |
energy: 9 | lcc: 8 | 1: 6 | 10: 6 | 11: 6 | 12: 6 BASIC_LATIN: 437
BASIC_LATIN: 7140 the: 6 | of: 5 | standard: 4 | a: 2 | and: 2 | by: 2
| criteria: 2 | technical: 2 | 1980: 1 | adopted: 1 330: 42 | 21008: 12
| 24371: 12 | 29977: 12 | lcc: 8 | 1: 6 | 10: 6 | 11: 6 | 12: 6 | 13: 6
the: 6 | of: 5 | standard: 4 | a: 2 | and: 2 | by: 2 | criteria: 2 |
technical: 2 | 1980: 1 | adopted: 1 330: 42 | 21008: 12 | 24371: 12 |
29977: 12 | lcc: 8 | 1: 6 | 10: 6 | 11: 6 | 12: 6 | 13: 6 0,013 0,012
govdocs1/974/974098.ppt 304128 application/vnd.ms-excel
application/vnd.ms-excel 567 148 1421 262 eng 39 eng 21 -18
330: 42 | 21008: 12 | 24371: 12 | 29977: 12 | energy: 9 | lcc: 8 | 1: 6
| 10: 6 | 11: 6 | 12: 6 1: 10 | 30: 7 | 25: 5 | 218.06175: 4 |
436.1235: 4 | 654.18525: 4 | 872.247: 4 | energy: 4 | national: 4 | 50:
3 BASIC_LATIN: 7140 BASIC_LATIN: 1717 330: 42 | 21008: 12 | 24371: 12
| 29977: 12 | lcc: 8 | 31: 6 | 32: 6 | 33: 6 | 34: 6 | 35: 6 218.06175:
4 | 436.1235: 4 | 654.18525: 4 | 872.247: 4 | national: 4 | 90.1: 3 |
ashrae: 3 | 1265.469: 2 | 141.157: 2 | 149.503: 2 330: 42 | 21008: 12 |
24371: 12 | 29977: 12 | lcc: 8 | 31: 6 | 32: 6 | 33: 6 | 34: 6 | 35: 6
1: 4 | 218.06175: 4 | 436.1235: 4 | 654.18525: 4 | 872.247: 4 |
national: 4 | 90.1: 3 | ashrae: 3 | 1265.469: 2 | 141.157: 2 0,092 0,096
Re: Next release?
Posted by Tim Allison <ta...@apache.org>.
I confirmed that the drop in attachments is actually a good thing in the
handful of files that I randomly sampled. One file had 9 copies of the
same macro.
The mismatch in tika-eval's attachment matching is troubling, but I think
we're good to go. I'll kick off the release process tomorrow morning (ET)
unless there are objections.
Best,
Tim
On Sun, Aug 20, 2023 at 7:21 AM Tilman Hausherr <TH...@t-online.de>
wrote:
> The content diffs are impressive, the "more in A" column is almost fully
> empty. There are only 3 files that might be relevant:
>
> govdocs1/245/245359.doc _842859044.xls _842791279.doc
> govdocs1/491/491561.ppt UNKNOWN-0.xls UNKNOWN-1.doc
> govdocs1/752/752792.ppt UNKNOWN-0.xls UNKNOWN-1.doc
>
>
> but I notice that the 2nd and 3rd column have different names.
>
> Tilman
>
> On 18.08.2023 00:21, Tim Allison wrote:
> > Current reports are here:
> > https://corpora.tika.apache.org/base/reports/tika-2.8.1-rand1m-xyz.tgz
> >
> > I expect a bunch of ole2 files will have fewer attachments because we're
> no
> > longer duplicating/triplicating macros. I haven't had a chance to look,
> > but will look tomorrow.
> >
> > On Tue, Aug 15, 2023 at 11:29 AM Tim Allison<ta...@apache.org>
> wrote:
> >
> >> All,
> >>
> >> I'm back from vacation. I had really hoped to run this release before I
> >> left, but TIKA-4091 and TIKA-4048 left some surprises without quick
> fixes
> >> available.
> >>
> >> I'd like to fix small regressions left behind in TIKA-4091 (case
> >> insensitive object names in OLE2), the new TIKA-4116 (duplicate macros
> in
> >> some OLE2) and TIKA-4048 (the regression caused by setting extract all
> in
> >> compressor parsers).
> >>
> >> WIth those changes, I think we should increment the minor version ->
> 2.9.0.
> >>
> >> Any blockers left for the next release? Any objections to the version
> >> choice?
> >>
> >>
> >> Best,
> >>
> >> Tim
> >>
> >>
> >>
>
Re: Next release?
Posted by Tim Allison <ta...@apache.org>.
Y. Something is possibly more broken w the eval embedded file matcher. :(
Will confirm I ran the latest version of tika-eval.
I’ll also check these files.
I want to spot check new “missing attachment” files and then off we go?
On Sun, Aug 20, 2023 at 7:21 AM Tilman Hausherr <TH...@t-online.de>
wrote:
> The content diffs are impressive, the "more in A" column is almost fully
> empty. There are only 3 files that might be relevant:
>
> govdocs1/245/245359.doc _842859044.xls _842791279.doc
> govdocs1/491/491561.ppt UNKNOWN-0.xls UNKNOWN-1.doc
> govdocs1/752/752792.ppt UNKNOWN-0.xls UNKNOWN-1.doc
>
>
> but I notice that the 2nd and 3rd column have different names.
>
> Tilman
>
> On 18.08.2023 00:21, Tim Allison wrote:
> > Current reports are here:
> > https://corpora.tika.apache.org/base/reports/tika-2.8.1-rand1m-xyz.tgz
> >
> > I expect a bunch of ole2 files will have fewer attachments because we're
> no
> > longer duplicating/triplicating macros. I haven't had a chance to look,
> > but will look tomorrow.
> >
> > On Tue, Aug 15, 2023 at 11:29 AM Tim Allison<ta...@apache.org>
> wrote:
> >
> >> All,
> >>
> >> I'm back from vacation. I had really hoped to run this release before I
> >> left, but TIKA-4091 and TIKA-4048 left some surprises without quick
> fixes
> >> available.
> >>
> >> I'd like to fix small regressions left behind in TIKA-4091 (case
> >> insensitive object names in OLE2), the new TIKA-4116 (duplicate macros
> in
> >> some OLE2) and TIKA-4048 (the regression caused by setting extract all
> in
> >> compressor parsers).
> >>
> >> WIth those changes, I think we should increment the minor version ->
> 2.9.0.
> >>
> >> Any blockers left for the next release? Any objections to the version
> >> choice?
> >>
> >>
> >> Best,
> >>
> >> Tim
> >>
> >>
> >>
>
Re: Next release?
Posted by Tilman Hausherr <TH...@t-online.de>.
The content diffs are impressive, the "more in A" column is almost fully
empty. There are only 3 files that might be relevant:
govdocs1/245/245359.doc _842859044.xls _842791279.doc
govdocs1/491/491561.ppt UNKNOWN-0.xls UNKNOWN-1.doc
govdocs1/752/752792.ppt UNKNOWN-0.xls UNKNOWN-1.doc
but I notice that the 2nd and 3rd column have different names.
Tilman
On 18.08.2023 00:21, Tim Allison wrote:
> Current reports are here:
> https://corpora.tika.apache.org/base/reports/tika-2.8.1-rand1m-xyz.tgz
>
> I expect a bunch of ole2 files will have fewer attachments because we're no
> longer duplicating/triplicating macros. I haven't had a chance to look,
> but will look tomorrow.
>
> On Tue, Aug 15, 2023 at 11:29 AM Tim Allison<ta...@apache.org> wrote:
>
>> All,
>>
>> I'm back from vacation. I had really hoped to run this release before I
>> left, but TIKA-4091 and TIKA-4048 left some surprises without quick fixes
>> available.
>>
>> I'd like to fix small regressions left behind in TIKA-4091 (case
>> insensitive object names in OLE2), the new TIKA-4116 (duplicate macros in
>> some OLE2) and TIKA-4048 (the regression caused by setting extract all in
>> compressor parsers).
>>
>> WIth those changes, I think we should increment the minor version -> 2.9.0.
>>
>> Any blockers left for the next release? Any objections to the version
>> choice?
>>
>>
>> Best,
>>
>> Tim
>>
>>
>>
Re: Next release?
Posted by Tim Allison <ta...@apache.org>.
Current reports are here:
https://corpora.tika.apache.org/base/reports/tika-2.8.1-rand1m-xyz.tgz
I expect a bunch of ole2 files will have fewer attachments because we're no
longer duplicating/triplicating macros. I haven't had a chance to look,
but will look tomorrow.
On Tue, Aug 15, 2023 at 11:29 AM Tim Allison <ta...@apache.org> wrote:
> All,
>
> I'm back from vacation. I had really hoped to run this release before I
> left, but TIKA-4091 and TIKA-4048 left some surprises without quick fixes
> available.
>
> I'd like to fix small regressions left behind in TIKA-4091 (case
> insensitive object names in OLE2), the new TIKA-4116 (duplicate macros in
> some OLE2) and TIKA-4048 (the regression caused by setting extract all in
> compressor parsers).
>
> WIth those changes, I think we should increment the minor version -> 2.9.0.
>
> Any blockers left for the next release? Any objections to the version
> choice?
>
>
> Best,
>
> Tim
>
>
>
Re: Next release?
Posted by Tim Allison <ta...@apache.org>.
All,
I'm back from vacation. I had really hoped to run this release before I
left, but TIKA-4091 and TIKA-4048 left some surprises without quick fixes
available.
I'd like to fix small regressions left behind in TIKA-4091 (case
insensitive object names in OLE2), the new TIKA-4116 (duplicate macros in
some OLE2) and TIKA-4048 (the regression caused by setting extract all in
compressor parsers).
WIth those changes, I think we should increment the minor version -> 2.9.0.
Any blockers left for the next release? Any objections to the version
choice?
Best,
Tim
Re: Next release?
Posted by Tim Allison <ta...@apache.org>.
Y, those are embedded files within the zip (e.g. "/shuffle.bmp"). I should
add "embedded resource path" and or "resource name" as a field in that
report.
I'll open a ticket for that.
Those exceptions at least '43R5U3BXJUDJXDZ25OAE33ZU47362WLV' are actually
new...I need to figure out why we are getting those now but weren't with
2.8.0.
Thank you!
On Wed, Jul 19, 2023 at 11:47 PM Tilman Hausherr <TH...@t-online.de>
wrote:
> In the new_exceptions_in_B_details.xlsx file,
> commoncrawl3/43/43R5U3BXJUDJXDZ25OAE33ZU47362WLV is listed as "bmp", but
> it is a zip file. And I get no exception when trying to extract all
> attachments with the -z option
>
> Same for commoncrawl3/M4/M4J5KAPEC5F62UXFNCPRQATQWH3FSWPG
>
> Tilman
>
> On 19.07.2023 19:19, Tim Allison wrote:
> > Results are here:
> > https://corpora.tika.apache.org/base/reports/tika-2.8.1-pre-rc1.tgz
> >
> > This is on a new set of ~1.3 million files, including fewer truncated
> PDFs.
> >
> > I've only had a chance to look quickly. No showstoppers leapt out to me.
> > There are some expected differences, and a couple of surprises. I'm
> going
> > to dig a bit tomorrow and then start the release process unless anyone
> > finds anything concerning or has a blocker.
> >
> > Thank you, all!
> >
> > Best,
> >
> > Tim
> >
> > On Thu, Jul 13, 2023 at 7:00 PM Tim Allison <ta...@apache.org> wrote:
> >
> >> All,
> >> I think we’re at a good place for a minor version release? Should I
> >> start the regression tests tomorrow for potential release next week or
> week
> >> after?
> >> Any blockers or things we should try to get in?
> >>
> >> Thank you!
> >>
> >> Best,
> >>
> >> Tim
> >>
> >> On Thu, Jul 13, 2023 at 5:20 PM Nicolò Mendola (Jira) <ji...@apache.org>
> >> wrote:
> >>
> >>> [
> >>>
> https://issues.apache.org/jira/browse/TIKA-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17742957#comment-17742957
> >>> ]
> >>>
> >>> Nicolò Mendola edited comment on TIKA-4064 at 7/13/23 9:19 PM:
> >>> ---------------------------------------------------------------
> >>>
> >>> Just out of interest, is there an eta for release 2.8.1 to be
> published?
> >>>
> >>> Best regards
> >>>
> >>>
> >>> was (Author: JIRAUSER296595):
> >>> Just out of interest, is there an eta for release 2.8.1 to be
> published?
> >>>
> >>>
> >>>
> >>> Best regards
> >>>
> >>>> Update to 2.8.1
> >>>> ---------------
> >>>>
> >>>> Key: TIKA-4064
> >>>> URL: https://issues.apache.org/jira/browse/TIKA-4064
> >>>> Project: Tika
> >>>> Issue Type: Task
> >>>> Components: build
> >>>> Affects Versions: 2.8.0
> >>>> Reporter: Tilman Hausherr
> >>>> Priority: Minor
> >>>> Fix For: 2.8.1
> >>>>
> >>>>
> >>>> The latest maven versions plugin finds much more outdated stuff than
> >>> the previous one, so I'll do a few updates.
> >>>
> >>>
> >>>
> >>> --
> >>> This message was sent by Atlassian Jira
> >>> (v8.20.10#820010)
> >>>
>
>
Re: Next release?
Posted by Tilman Hausherr <TH...@t-online.de>.
In the new_exceptions_in_B_details.xlsx file,
commoncrawl3/43/43R5U3BXJUDJXDZ25OAE33ZU47362WLV is listed as "bmp", but
it is a zip file. And I get no exception when trying to extract all
attachments with the -z option
Same for commoncrawl3/M4/M4J5KAPEC5F62UXFNCPRQATQWH3FSWPG
Tilman
On 19.07.2023 19:19, Tim Allison wrote:
> Results are here:
> https://corpora.tika.apache.org/base/reports/tika-2.8.1-pre-rc1.tgz
>
> This is on a new set of ~1.3 million files, including fewer truncated PDFs.
>
> I've only had a chance to look quickly. No showstoppers leapt out to me.
> There are some expected differences, and a couple of surprises. I'm going
> to dig a bit tomorrow and then start the release process unless anyone
> finds anything concerning or has a blocker.
>
> Thank you, all!
>
> Best,
>
> Tim
>
> On Thu, Jul 13, 2023 at 7:00 PM Tim Allison <ta...@apache.org> wrote:
>
>> All,
>> I think we’re at a good place for a minor version release? Should I
>> start the regression tests tomorrow for potential release next week or week
>> after?
>> Any blockers or things we should try to get in?
>>
>> Thank you!
>>
>> Best,
>>
>> Tim
>>
>> On Thu, Jul 13, 2023 at 5:20 PM Nicolò Mendola (Jira) <ji...@apache.org>
>> wrote:
>>
>>> [
>>> https://issues.apache.org/jira/browse/TIKA-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17742957#comment-17742957
>>> ]
>>>
>>> Nicolò Mendola edited comment on TIKA-4064 at 7/13/23 9:19 PM:
>>> ---------------------------------------------------------------
>>>
>>> Just out of interest, is there an eta for release 2.8.1 to be published?
>>>
>>> Best regards
>>>
>>>
>>> was (Author: JIRAUSER296595):
>>> Just out of interest, is there an eta for release 2.8.1 to be published?
>>>
>>>
>>>
>>> Best regards
>>>
>>>> Update to 2.8.1
>>>> ---------------
>>>>
>>>> Key: TIKA-4064
>>>> URL: https://issues.apache.org/jira/browse/TIKA-4064
>>>> Project: Tika
>>>> Issue Type: Task
>>>> Components: build
>>>> Affects Versions: 2.8.0
>>>> Reporter: Tilman Hausherr
>>>> Priority: Minor
>>>> Fix For: 2.8.1
>>>>
>>>>
>>>> The latest maven versions plugin finds much more outdated stuff than
>>> the previous one, so I'll do a few updates.
>>>
>>>
>>>
>>> --
>>> This message was sent by Atlassian Jira
>>> (v8.20.10#820010)
>>>
Re: Next release?
Posted by Tim Allison <ta...@apache.org>.
Results are here:
https://corpora.tika.apache.org/base/reports/tika-2.8.1-pre-rc1.tgz
This is on a new set of ~1.3 million files, including fewer truncated PDFs.
I've only had a chance to look quickly. No showstoppers leapt out to me.
There are some expected differences, and a couple of surprises. I'm going
to dig a bit tomorrow and then start the release process unless anyone
finds anything concerning or has a blocker.
Thank you, all!
Best,
Tim
On Thu, Jul 13, 2023 at 7:00 PM Tim Allison <ta...@apache.org> wrote:
> All,
> I think we’re at a good place for a minor version release? Should I
> start the regression tests tomorrow for potential release next week or week
> after?
> Any blockers or things we should try to get in?
>
> Thank you!
>
> Best,
>
> Tim
>
> On Thu, Jul 13, 2023 at 5:20 PM Nicolò Mendola (Jira) <ji...@apache.org>
> wrote:
>
>>
>> [
>> https://issues.apache.org/jira/browse/TIKA-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17742957#comment-17742957
>> ]
>>
>> Nicolò Mendola edited comment on TIKA-4064 at 7/13/23 9:19 PM:
>> ---------------------------------------------------------------
>>
>> Just out of interest, is there an eta for release 2.8.1 to be published?
>>
>> Best regards
>>
>>
>> was (Author: JIRAUSER296595):
>> Just out of interest, is there an eta for release 2.8.1 to be published?
>>
>>
>>
>> Best regards
>>
>> > Update to 2.8.1
>> > ---------------
>> >
>> > Key: TIKA-4064
>> > URL: https://issues.apache.org/jira/browse/TIKA-4064
>> > Project: Tika
>> > Issue Type: Task
>> > Components: build
>> > Affects Versions: 2.8.0
>> > Reporter: Tilman Hausherr
>> > Priority: Minor
>> > Fix For: 2.8.1
>> >
>> >
>> > The latest maven versions plugin finds much more outdated stuff than
>> the previous one, so I'll do a few updates.
>>
>>
>>
>> --
>> This message was sent by Atlassian Jira
>> (v8.20.10#820010)
>>
>
Next release?
Posted by Tim Allison <ta...@apache.org>.
All,
I think we’re at a good place for a minor version release? Should I
start the regression tests tomorrow for potential release next week or week
after?
Any blockers or things we should try to get in?
Thank you!
Best,
Tim
On Thu, Jul 13, 2023 at 5:20 PM Nicolò Mendola (Jira) <ji...@apache.org>
wrote:
>
> [
> https://issues.apache.org/jira/browse/TIKA-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17742957#comment-17742957
> ]
>
> Nicolò Mendola edited comment on TIKA-4064 at 7/13/23 9:19 PM:
> ---------------------------------------------------------------
>
> Just out of interest, is there an eta for release 2.8.1 to be published?
>
> Best regards
>
>
> was (Author: JIRAUSER296595):
> Just out of interest, is there an eta for release 2.8.1 to be published?
>
>
>
> Best regards
>
> > Update to 2.8.1
> > ---------------
> >
> > Key: TIKA-4064
> > URL: https://issues.apache.org/jira/browse/TIKA-4064
> > Project: Tika
> > Issue Type: Task
> > Components: build
> > Affects Versions: 2.8.0
> > Reporter: Tilman Hausherr
> > Priority: Minor
> > Fix For: 2.8.1
> >
> >
> > The latest maven versions plugin finds much more outdated stuff than the
> previous one, so I'll do a few updates.
>
>
>
> --
> This message was sent by Atlassian Jira
> (v8.20.10#820010)
>