You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tika.apache.org by "Joshua Turner (JIRA)" <ji...@apache.org> on 2010/07/08 16:45:49 UTC
[jira] Created: (TIKA-461) RFC822 messages not parsed
RFC822 messages not parsed
--------------------------
Key: TIKA-461
URL: https://issues.apache.org/jira/browse/TIKA-461
Project: Tika
Issue Type: Bug
Components: parser
Affects Versions: 0.7
Reporter: Joshua Turner
Presented with an RFC822 message exported from Thunderbird, AutodetectParser produces an empty body, and a Metadata containing only one key-value pair: "Content-Type=message/rfc822". Directly calling MboxParser likewise gives an empty body, but with two metadata pairs: "Content-Encoding=us-ascii Content-Type=application/mbox".
A quick peek at the source of MboxParser shows that the implementation is pretty naive. If the wiring can be sorted out, something like Apache James' mime4j might be a better bet.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (TIKA-461) RFC822 messages not parsed
Posted by "Julien Nioche (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TIKA-461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Julien Nioche reassigned TIKA-461:
----------------------------------
Assignee: Julien Nioche
> RFC822 messages not parsed
> --------------------------
>
> Key: TIKA-461
> URL: https://issues.apache.org/jira/browse/TIKA-461
> Project: Tika
> Issue Type: Bug
> Components: parser
> Affects Versions: 0.7
> Reporter: Joshua Turner
> Assignee: Julien Nioche
>
> Presented with an RFC822 message exported from Thunderbird, AutodetectParser produces an empty body, and a Metadata containing only one key-value pair: "Content-Type=message/rfc822". Directly calling MboxParser likewise gives an empty body, but with two metadata pairs: "Content-Encoding=us-ascii Content-Type=application/mbox".
> A quick peek at the source of MboxParser shows that the implementation is pretty naive. If the wiring can be sorted out, something like Apache James' mime4j might be a better bet.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (TIKA-461) RFC822 messages not parsed
Posted by "Julien Nioche (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TIKA-461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12906468#action_12906468 ]
Julien Nioche commented on TIKA-461:
------------------------------------
I'll have a look at mime4j and try to use it in Tika
> RFC822 messages not parsed
> --------------------------
>
> Key: TIKA-461
> URL: https://issues.apache.org/jira/browse/TIKA-461
> Project: Tika
> Issue Type: Bug
> Components: parser
> Affects Versions: 0.7
> Reporter: Joshua Turner
> Assignee: Julien Nioche
>
> Presented with an RFC822 message exported from Thunderbird, AutodetectParser produces an empty body, and a Metadata containing only one key-value pair: "Content-Type=message/rfc822". Directly calling MboxParser likewise gives an empty body, but with two metadata pairs: "Content-Encoding=us-ascii Content-Type=application/mbox".
> A quick peek at the source of MboxParser shows that the implementation is pretty naive. If the wiring can be sorted out, something like Apache James' mime4j might be a better bet.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (TIKA-461) RFC822 messages not parsed
Posted by "Nick Burch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TIKA-461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12915285#action_12915285 ]
Nick Burch commented on TIKA-461:
---------------------------------
It'd probably be good to see some more tests with it. For now, just checking your basic message should be fine, but I'd suggest we also try to get an email with plain text, html, images and similar in to check the more complex bits.
In terms of the nested parser, I'm tempted to say we do something so that plain text comes out without any extra work needed. Anything else gets handled via a Parser fetched from the ParseContext if required, much as we're doing for container formats like zip, .docx etc. That way, you can throw a simple email at it and get the text, but the rest of the parts are available if you want them
Also, the james jars need to be listed in the tika bundle pom so they get properly included
> RFC822 messages not parsed
> --------------------------
>
> Key: TIKA-461
> URL: https://issues.apache.org/jira/browse/TIKA-461
> Project: Tika
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.7
> Reporter: Joshua Turner
> Assignee: Julien Nioche
> Attachments: TIKA-461.patch
>
>
> Presented with an RFC822 message exported from Thunderbird, AutodetectParser produces an empty body, and a Metadata containing only one key-value pair: "Content-Type=message/rfc822". Directly calling MboxParser likewise gives an empty body, but with two metadata pairs: "Content-Encoding=us-ascii Content-Type=application/mbox".
> A quick peek at the source of MboxParser shows that the implementation is pretty naive. If the wiring can be sorted out, something like Apache James' mime4j might be a better bet.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (TIKA-461) RFC822 messages not parsed
Posted by "Julien Nioche (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TIKA-461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930180#action_12930180 ]
Julien Nioche commented on TIKA-461:
------------------------------------
Nope. I was planning to refactor the parser first along the lines of what you described (org.apache.tika.extractor) and haven't found the time to do it so far.
> RFC822 messages not parsed
> --------------------------
>
> Key: TIKA-461
> URL: https://issues.apache.org/jira/browse/TIKA-461
> Project: Tika
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.7
> Reporter: Joshua Turner
> Assignee: Julien Nioche
> Attachments: TIKA-461.patch
>
>
> Presented with an RFC822 message exported from Thunderbird, AutodetectParser produces an empty body, and a Metadata containing only one key-value pair: "Content-Type=message/rfc822". Directly calling MboxParser likewise gives an empty body, but with two metadata pairs: "Content-Encoding=us-ascii Content-Type=application/mbox".
> A quick peek at the source of MboxParser shows that the implementation is pretty naive. If the wiring can be sorted out, something like Apache James' mime4j might be a better bet.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (TIKA-461) RFC822 messages not parsed
Posted by "Nick Burch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TIKA-461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930182#action_12930182 ]
Nick Burch commented on TIKA-461:
---------------------------------
No worries, we'll look forward to the new patch and the tests when you get some time :)
> RFC822 messages not parsed
> --------------------------
>
> Key: TIKA-461
> URL: https://issues.apache.org/jira/browse/TIKA-461
> Project: Tika
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.7
> Reporter: Joshua Turner
> Assignee: Julien Nioche
> Attachments: TIKA-461.patch
>
>
> Presented with an RFC822 message exported from Thunderbird, AutodetectParser produces an empty body, and a Metadata containing only one key-value pair: "Content-Type=message/rfc822". Directly calling MboxParser likewise gives an empty body, but with two metadata pairs: "Content-Encoding=us-ascii Content-Type=application/mbox".
> A quick peek at the source of MboxParser shows that the implementation is pretty naive. If the wiring can be sorted out, something like Apache James' mime4j might be a better bet.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (TIKA-461) RFC822 messages not parsed
Posted by "Julien Nioche (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TIKA-461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Julien Nioche updated TIKA-461:
-------------------------------
Issue Type: New Feature (was: Bug)
changed from bug to new feature
> RFC822 messages not parsed
> --------------------------
>
> Key: TIKA-461
> URL: https://issues.apache.org/jira/browse/TIKA-461
> Project: Tika
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.7
> Reporter: Joshua Turner
> Assignee: Julien Nioche
> Attachments: TIKA-461.patch
>
>
> Presented with an RFC822 message exported from Thunderbird, AutodetectParser produces an empty body, and a Metadata containing only one key-value pair: "Content-Type=message/rfc822". Directly calling MboxParser likewise gives an empty body, but with two metadata pairs: "Content-Encoding=us-ascii Content-Type=application/mbox".
> A quick peek at the source of MboxParser shows that the implementation is pretty naive. If the wiring can be sorted out, something like Apache James' mime4j might be a better bet.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (TIKA-461) RFC822 messages not parsed
Posted by "Julien Nioche (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TIKA-461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12915708#action_12915708 ]
Julien Nioche commented on TIKA-461:
------------------------------------
Nick,
Thanks for taking the time to review my patch.
bq. It'd probably be good to see some more tests with it. For now, just checking your basic message should be fine, but I'd suggest we also try to get an email with plain text, html, images and similar in to check the more complex bits.
Agreed
bq. In terms of the nested parser, I'm tempted to say we do something so that plain text comes out without any extra work needed. Anything else gets handled via a Parser fetched from the ParseContext if required, much as we're doing for container formats like zip, .docx etc. That way, you can throw a simple email at it and get the text, but the rest of the parts are available if you want them
I hadn't noticed that you've added org.apache.tika.extractor, seems an elegant way of doing. Will have a closer look and see how I can leverage it in RFC822Parser
bq. Also, the james jars need to be listed in the tika bundle pom so they get properly included
Ok, did not know about that. Thanks
> RFC822 messages not parsed
> --------------------------
>
> Key: TIKA-461
> URL: https://issues.apache.org/jira/browse/TIKA-461
> Project: Tika
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.7
> Reporter: Joshua Turner
> Assignee: Julien Nioche
> Attachments: TIKA-461.patch
>
>
> Presented with an RFC822 message exported from Thunderbird, AutodetectParser produces an empty body, and a Metadata containing only one key-value pair: "Content-Type=message/rfc822". Directly calling MboxParser likewise gives an empty body, but with two metadata pairs: "Content-Encoding=us-ascii Content-Type=application/mbox".
> A quick peek at the source of MboxParser shows that the implementation is pretty naive. If the wiring can be sorted out, something like Apache James' mime4j might be a better bet.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (TIKA-461) RFC822 messages not parsed
Posted by "Nick Burch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TIKA-461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930175#action_12930175 ]
Nick Burch commented on TIKA-461:
---------------------------------
Julien - Did you have any luck knocking up some tests to go with the patch?
> RFC822 messages not parsed
> --------------------------
>
> Key: TIKA-461
> URL: https://issues.apache.org/jira/browse/TIKA-461
> Project: Tika
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.7
> Reporter: Joshua Turner
> Assignee: Julien Nioche
> Attachments: TIKA-461.patch
>
>
> Presented with an RFC822 message exported from Thunderbird, AutodetectParser produces an empty body, and a Metadata containing only one key-value pair: "Content-Type=message/rfc822". Directly calling MboxParser likewise gives an empty body, but with two metadata pairs: "Content-Encoding=us-ascii Content-Type=application/mbox".
> A quick peek at the source of MboxParser shows that the implementation is pretty naive. If the wiring can be sorted out, something like Apache James' mime4j might be a better bet.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (TIKA-461) RFC822 messages not parsed
Posted by "Julien Nioche (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TIKA-461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Julien Nioche updated TIKA-461:
-------------------------------
Attachment: TIKA-461.patch
This patch contains an initial version of the RFC822Parser which uses apache-mime4j
The metadata are currently created only for the main message and not for the parts. Note that the parts are put inside <p> elements and parsed with the right parser for their declared mime-type e.g. txt, html etc... The multiparts seem to be properly handled as well.
There is definitely space for improvements, as usual comments are more than welcome
> RFC822 messages not parsed
> --------------------------
>
> Key: TIKA-461
> URL: https://issues.apache.org/jira/browse/TIKA-461
> Project: Tika
> Issue Type: Bug
> Components: parser
> Affects Versions: 0.7
> Reporter: Joshua Turner
> Assignee: Julien Nioche
> Attachments: TIKA-461.patch
>
>
> Presented with an RFC822 message exported from Thunderbird, AutodetectParser produces an empty body, and a Metadata containing only one key-value pair: "Content-Type=message/rfc822". Directly calling MboxParser likewise gives an empty body, but with two metadata pairs: "Content-Encoding=us-ascii Content-Type=application/mbox".
> A quick peek at the source of MboxParser shows that the implementation is pretty naive. If the wiring can be sorted out, something like Apache James' mime4j might be a better bet.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (TIKA-461) RFC822 messages not parsed
Posted by "Julien Nioche (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TIKA-461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12915269#action_12915269 ]
Julien Nioche commented on TIKA-461:
------------------------------------
Hi guys,
Could anyone have a look at the patch and review it please? I will assume no one has seen any major issues with it and commit at the end of this week otherwise
Thanks
Julien
> RFC822 messages not parsed
> --------------------------
>
> Key: TIKA-461
> URL: https://issues.apache.org/jira/browse/TIKA-461
> Project: Tika
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.7
> Reporter: Joshua Turner
> Assignee: Julien Nioche
> Attachments: TIKA-461.patch
>
>
> Presented with an RFC822 message exported from Thunderbird, AutodetectParser produces an empty body, and a Metadata containing only one key-value pair: "Content-Type=message/rfc822". Directly calling MboxParser likewise gives an empty body, but with two metadata pairs: "Content-Encoding=us-ascii Content-Type=application/mbox".
> A quick peek at the source of MboxParser shows that the implementation is pretty naive. If the wiring can be sorted out, something like Apache James' mime4j might be a better bet.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.