You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tika.apache.org by "Andrew Jackson (Created) (JIRA)" <ji...@apache.org> on 2012/01/17 12:08:39 UTC
[jira] [Created] (TIKA-847) Add regular expression support to the
MagicDetector
Add regular expression support to the MagicDetector
---------------------------------------------------
Key: TIKA-847
URL: https://issues.apache.org/jira/browse/TIKA-847
Project: Tika
Issue Type: New Feature
Components: mime
Affects Versions: 1.0
Reporter: Andrew Jackson
Following on from TIKA-86, we would like to add support for regular expressions to the MagicDetector. This would allow more signatures to be re-used from more sources (e.g. the file(1) command). As part of the SCAPE Project, we have added this functionality to our own Tika branch (e.g. https://github.com/openplanets/tika/commit/b8de9e77c8b432788e3f73a4dbccca8ea044b503) and are working to tidy this up to make a clean patch we can submit here.
BTW, are there any patch submission guidelines or coding standards we should check our work against first?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (TIKA-847) Add regular expression support to the
MagicDetector
Posted by "Peter May (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TIKA-847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter May updated TIKA-847:
---------------------------
Attachment: regex_support.patch
Patch updating MagicDetector and associated unit tests to incorporate regular expression support in the signature file (does not support EOF regular expressions).
This required a slight extension to the freedesktops mime-info to support a type="regex" attribute in the "match" element. Do you have an XML schema anywhere for mime-info, as this would also need updating?
I also noted (what I consider) a minor bug in the while loop at line 315 (https://github.com/apache/tika/blob/trunk/tika-core/src/main/java/org/apache/tika/detect/MagicDetector.java#L315) of MagicDetector, where the offset is not incremented by the number of read bytes. I have corrected that in this patch, but I can extract this out as a separate issue if preferred?
> Add regular expression support to the MagicDetector
> ---------------------------------------------------
>
> Key: TIKA-847
> URL: https://issues.apache.org/jira/browse/TIKA-847
> Project: Tika
> Issue Type: New Feature
> Components: mime
> Affects Versions: 1.0
> Reporter: Andrew Jackson
> Labels: detection, format
> Attachments: regex_support.patch
>
>
> Following on from TIKA-86, we would like to add support for regular expressions to the MagicDetector. This would allow more signatures to be re-used from more sources (e.g. the file(1) command). As part of the SCAPE Project, we have added this functionality to our own Tika branch (e.g. https://github.com/openplanets/tika/commit/b8de9e77c8b432788e3f73a4dbccca8ea044b503) and are working to tidy this up to make a clean patch we can submit here.
> BTW, are there any patch submission guidelines or coding standards we should check our work against first?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TIKA-847) Add regular expression support to the
MagicDetector
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TIKA-847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188402#comment-13188402 ]
Uwe Schindler commented on TIKA-847:
------------------------------------
You could use the original FSM library, Lucene's is a fork of: http://www.brics.dk/automaton/
> Add regular expression support to the MagicDetector
> ---------------------------------------------------
>
> Key: TIKA-847
> URL: https://issues.apache.org/jira/browse/TIKA-847
> Project: Tika
> Issue Type: New Feature
> Components: mime
> Affects Versions: 1.0
> Reporter: Andrew Jackson
> Labels: detection, format
>
> Following on from TIKA-86, we would like to add support for regular expressions to the MagicDetector. This would allow more signatures to be re-used from more sources (e.g. the file(1) command). As part of the SCAPE Project, we have added this functionality to our own Tika branch (e.g. https://github.com/openplanets/tika/commit/b8de9e77c8b432788e3f73a4dbccca8ea044b503) and are working to tidy this up to make a clean patch we can submit here.
> BTW, are there any patch submission guidelines or coding standards we should check our work against first?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (TIKA-847) Add regular expression
support to the MagicDetector
Posted by "Andrew Jackson (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TIKA-847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187587#comment-13187587 ]
Andrew Jackson edited comment on TIKA-847 at 1/17/12 11:09 AM:
---------------------------------------------------------------
Under TIKA-86, [~kkrugler] suggested we use the Lucene FSM RegEx engine (see https://issues.apache.org/jira/browse/LUCENE-1606 and http://search-lucene.com/jd/lucene/org/apache/lucene/search/RegexpQuery.html). However, I believe this code is not yet in a stable Lucene version (4.0-SNAPSHOT only) and so would rather we defer that optimisation to a later date (under a separate ticket).
was (Author: anjackson):
Under TIKA-86, @kkrugler suggested we use the Lucene FSM RegEx engine (see https://issues.apache.org/jira/browse/LUCENE-1606 and http://search-lucene.com/jd/lucene/org/apache/lucene/search/RegexpQuery.html). However, I believe this code is not yet in a stable Lucene version (4.0-SNAPSHOT only) and so would rather we defer that optimisation to a later date (under a separate ticket).
> Add regular expression support to the MagicDetector
> ---------------------------------------------------
>
> Key: TIKA-847
> URL: https://issues.apache.org/jira/browse/TIKA-847
> Project: Tika
> Issue Type: New Feature
> Components: mime
> Affects Versions: 1.0
> Reporter: Andrew Jackson
> Labels: detection, format
>
> Following on from TIKA-86, we would like to add support for regular expressions to the MagicDetector. This would allow more signatures to be re-used from more sources (e.g. the file(1) command). As part of the SCAPE Project, we have added this functionality to our own Tika branch (e.g. https://github.com/openplanets/tika/commit/b8de9e77c8b432788e3f73a4dbccca8ea044b503) and are working to tidy this up to make a clean patch we can submit here.
> BTW, are there any patch submission guidelines or coding standards we should check our work against first?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TIKA-847) Add regular expression support to the
MagicDetector
Posted by "Andrew Jackson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TIKA-847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202883#comment-13202883 ]
Andrew Jackson commented on TIKA-847:
-------------------------------------
As far as I can tell, Lucene's FSM engine is so very new it is only in the v4 snapshot, so I would prefer it if we stuck to the built-in RegEx functionality for now and revisited this optimisation under a separate ticket, at least until Lucene 4 is released.
Is there anything [~pete.s.may] or I can do to help this issue along?
> Add regular expression support to the MagicDetector
> ---------------------------------------------------
>
> Key: TIKA-847
> URL: https://issues.apache.org/jira/browse/TIKA-847
> Project: Tika
> Issue Type: New Feature
> Components: mime
> Affects Versions: 1.0
> Reporter: Andrew Jackson
> Labels: detection, format
> Attachments: regex_support.patch
>
>
> Following on from TIKA-86, we would like to add support for regular expressions to the MagicDetector. This would allow more signatures to be re-used from more sources (e.g. the file(1) command). As part of the SCAPE Project, we have added this functionality to our own Tika branch (e.g. https://github.com/openplanets/tika/commit/b8de9e77c8b432788e3f73a4dbccca8ea044b503) and are working to tidy this up to make a clean patch we can submit here.
> BTW, are there any patch submission guidelines or coding standards we should check our work against first?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TIKA-847) Add regular expression support to the
MagicDetector
Posted by "Andrew Jackson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TIKA-847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187587#comment-13187587 ]
Andrew Jackson commented on TIKA-847:
-------------------------------------
Under TIKA-86, @kkrugler suggested we use the Lucene FSM RegEx engine (see https://issues.apache.org/jira/browse/LUCENE-1606 and http://search-lucene.com/jd/lucene/org/apache/lucene/search/RegexpQuery.html). However, I believe this code is not yet in a stable Lucene version (4.0-SNAPSHOT only) and so would rather we defer that optimisation to a later date (under a separate ticket).
> Add regular expression support to the MagicDetector
> ---------------------------------------------------
>
> Key: TIKA-847
> URL: https://issues.apache.org/jira/browse/TIKA-847
> Project: Tika
> Issue Type: New Feature
> Components: mime
> Affects Versions: 1.0
> Reporter: Andrew Jackson
> Labels: detection, format
>
> Following on from TIKA-86, we would like to add support for regular expressions to the MagicDetector. This would allow more signatures to be re-used from more sources (e.g. the file(1) command). As part of the SCAPE Project, we have added this functionality to our own Tika branch (e.g. https://github.com/openplanets/tika/commit/b8de9e77c8b432788e3f73a4dbccca8ea044b503) and are working to tidy this up to make a clean patch we can submit here.
> BTW, are there any patch submission guidelines or coding standards we should check our work against first?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (TIKA-847) Add regular expression support to the
MagicDetector
Posted by "Jukka Zitting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TIKA-847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jukka Zitting resolved TIKA-847.
--------------------------------
Resolution: Fixed
Fix Version/s: 1.2
Assignee: Jukka Zitting
Sorry for the delay on this, the issue fell off our radar.
I committed the patch in revision 1355752. Thanks!
> Add regular expression support to the MagicDetector
> ---------------------------------------------------
>
> Key: TIKA-847
> URL: https://issues.apache.org/jira/browse/TIKA-847
> Project: Tika
> Issue Type: New Feature
> Components: mime
> Affects Versions: 1.0
> Reporter: Andrew Jackson
> Assignee: Jukka Zitting
> Labels: detection, format
> Fix For: 1.2
>
> Attachments: regex_support.patch
>
>
> Following on from TIKA-86, we would like to add support for regular expressions to the MagicDetector. This would allow more signatures to be re-used from more sources (e.g. the file(1) command). As part of the SCAPE Project, we have added this functionality to our own Tika branch (e.g. https://github.com/openplanets/tika/commit/b8de9e77c8b432788e3f73a4dbccca8ea044b503) and are working to tidy this up to make a clean patch we can submit here.
> BTW, are there any patch submission guidelines or coding standards we should check our work against first?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira