You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tika.apache.org by "Antoni Mylka (Created) (JIRA)" <ji...@apache.org> on 2011/12/13 21:33:30 UTC
[jira] [Created] (TIKA-814) Increase the amount of bytes read by
TextDetector
Increase the amount of bytes read by TextDetector
-------------------------------------------------
Key: TIKA-814
URL: https://issues.apache.org/jira/browse/TIKA-814
Project: Tika
Issue Type: Improvement
Affects Versions: 1.1
Reporter: Antoni Mylka
Attachments: tika-textdetector.patch
In TIKA-688 Jukka implemented a plain text detector. It is fired automatically inside MimeTypes. I find a number of files in my collections, which are binary but are still detected as plain text. They wouldn't be if the plain text detector were allowed to look at more than the initial 512 bytes. I think that the TextDetector should look at MimeTypes.getMinLength bytes. It is given a ByteArrayInputStream backed by an Array. It should read all bytes in that array.
The performance impact should be negligible (no I/O, no allocations, just pure array lookups), while my experiments show that there are cases when 512 bytes is not enough.
If anyone objects due to performance reasons, I'll create another patch, which will allow the users to decouple the TextDetector from MimeTypes and supply their own, with different settings.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (TIKA-814) Increase the amount of bytes read by
TextDetector
Posted by "Antoni Mylka (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TIKA-814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Antoni Mylka updated TIKA-814:
------------------------------
Attachment: tika-textdetector.patch
A patch, which makes the text detector work on the entire array supplied by MimeTypes
> Increase the amount of bytes read by TextDetector
> -------------------------------------------------
>
> Key: TIKA-814
> URL: https://issues.apache.org/jira/browse/TIKA-814
> Project: Tika
> Issue Type: Improvement
> Affects Versions: 1.1
> Reporter: Antoni Mylka
> Attachments: tika-textdetector.patch
>
>
> In TIKA-688 Jukka implemented a plain text detector. It is fired automatically inside MimeTypes. I find a number of files in my collections, which are binary but are still detected as plain text. They wouldn't be if the plain text detector were allowed to look at more than the initial 512 bytes. I think that the TextDetector should look at MimeTypes.getMinLength bytes. It is given a ByteArrayInputStream backed by an Array. It should read all bytes in that array.
> The performance impact should be negligible (no I/O, no allocations, just pure array lookups), while my experiments show that there are cases when 512 bytes is not enough.
> If anyone objects due to performance reasons, I'll create another patch, which will allow the users to decouple the TextDetector from MimeTypes and supply their own, with different settings.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (TIKA-814) Increase the amount of bytes read by
TextDetector
Posted by "Antoni Mylka (Closed) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TIKA-814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Antoni Mylka closed TIKA-814.
-----------------------------
Resolution: Fixed
Fix Version/s: 1.1
Committed in r1220698.
This is a change, which theoretically impacts all users of Tika invoking MimeTypes. I say it has negligible performance overhead and yields better results on 5 broken BMP files I have in my collections.
If you disagree: revert the change and reopen this issue. I'll create a second solution, with customizable plain text detection.
For now, I close this.
> Increase the amount of bytes read by TextDetector
> -------------------------------------------------
>
> Key: TIKA-814
> URL: https://issues.apache.org/jira/browse/TIKA-814
> Project: Tika
> Issue Type: Improvement
> Affects Versions: 1.1
> Reporter: Antoni Mylka
> Fix For: 1.1
>
> Attachments: tika-textdetector.patch
>
>
> In TIKA-688 Jukka implemented a plain text detector. It is fired automatically inside MimeTypes. I find a number of files in my collections, which are binary but are still detected as plain text. They wouldn't be if the plain text detector were allowed to look at more than the initial 512 bytes. I think that the TextDetector should look at MimeTypes.getMinLength bytes. It is given a ByteArrayInputStream backed by an Array. It should read all bytes in that array.
> The performance impact should be negligible (no I/O, no allocations, just pure array lookups), while my experiments show that there are cases when 512 bytes is not enough.
> If anyone objects due to performance reasons, I'll create another patch, which will allow the users to decouple the TextDetector from MimeTypes and supply their own, with different settings.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira