You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tika.apache.org by "Andrew Jackson (Created) (JIRA)" <ji...@apache.org> on 2012/01/23 14:30:39 UTC
[jira] [Created] (TIKA-849) Identify and parse the Apple iBooks
format
Identify and parse the Apple iBooks format
------------------------------------------
Key: TIKA-849
URL: https://issues.apache.org/jira/browse/TIKA-849
Project: Tika
Issue Type: New Feature
Components: mime, parser
Affects Versions: 1.1
Reporter: Andrew Jackson
With the release of iBooks Author 1.0, Apple have created a new eBook format very similar to ePub. Tika could be extended to identify and parse this new format, re-using the existing ePub code wherever possible.
I have created an initial patch, which I will attach to this issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TIKA-849) Identify and parse the Apple iBooks
format
Posted by "Nick Burch (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TIKA-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192079#comment-13192079 ]
Nick Burch commented on TIKA-849:
---------------------------------
We might be able to use the same handler, but it'd need one of our XML experts to weigh in and say for sure!
I've fixed a bug in r1235215 where start/end document was being called for each html/xhtml file, which solves the use of --metadata from TikaCLI but doesn't affect the text extraction
For the mimetype subtype, I thought it made sense as both use the same structure and a very similar format. If you think that's wrong, we could maybe invent a common supertype for the two, to show their relation
> Identify and parse the Apple iBooks format
> ------------------------------------------
>
> Key: TIKA-849
> URL: https://issues.apache.org/jira/browse/TIKA-849
> Project: Tika
> Issue Type: New Feature
> Components: mime, parser
> Affects Versions: 1.1
> Reporter: Andrew Jackson
> Attachments: ibooks-support.patch
>
>
> With the release of iBooks Author 1.0, Apple have created a new eBook format very similar to ePub. Tika could be extended to identify and parse this new format, re-using the existing ePub code wherever possible.
> I have created an initial patch, which I will attach to this issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (TIKA-849) Identify and parse the
Apple iBooks format
Posted by "Andrew Jackson (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TIKA-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192078#comment-13192078 ]
Andrew Jackson edited comment on TIKA-849 at 1/24/12 11:54 AM:
---------------------------------------------------------------
By the way, I notice you added a
{code:xml}
<sub-class-of type="application/epub+zip" />
{code}
to the iBooks identification information. I'm not sure this is correct, as I was under the impression that this was a strict conformance hierarchy. This would imply that every x-ibooks+zip also conforms to the epub+zip format, but this is not the case, because the new MIME type declaration means it violates the ePub standard.
was (Author: anjackson):
By the way, I notice you added a
<code>
<sub-class-of type="application/epub+zip" />
</code>
to the iBooks identification information. I'm not sure this is correct, as I was under the impression that this was a strict conformance hierarchy. This would imply that every x-ibooks+zip also conforms to the epub+zip format, but this is not the case, because the new MIME type declaration means it violated the ePub standard.
> Identify and parse the Apple iBooks format
> ------------------------------------------
>
> Key: TIKA-849
> URL: https://issues.apache.org/jira/browse/TIKA-849
> Project: Tika
> Issue Type: New Feature
> Components: mime, parser
> Affects Versions: 1.1
> Reporter: Andrew Jackson
> Attachments: ibooks-support.patch
>
>
> With the release of iBooks Author 1.0, Apple have created a new eBook format very similar to ePub. Tika could be extended to identify and parse this new format, re-using the existing ePub code wherever possible.
> I have created an initial patch, which I will attach to this issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (TIKA-849) Identify and parse the
Apple iBooks format
Posted by "Andrew Jackson (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TIKA-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191148#comment-13191148 ]
Andrew Jackson edited comment on TIKA-849 at 1/23/12 1:35 PM:
--------------------------------------------------------------
This patch identifies \*.ibooks files, and passes them through the ePub parser. Unfortunately, the content of the eBook is not extracted successfully. I'm not sure why this is, and feedback would be appreciated here. I think the content\*.xhtml files are actually HTML5 and it seems that the current XHTML-based parser is unable to parse them. A unit test and test file is included.
was (Author: anjackson):
This patch identifies *.ibooks files, and passes them through the ePub parser. Unfortunately, the content of the eBook is not extracted successfully. I'm not sure why this is, and feedback would be appreciated here. I think the content*.xhtml files are actually HTML5 and it seems that the current XHTML-based parser is unable to parse them. A unit test and test file is included.
> Identify and parse the Apple iBooks format
> ------------------------------------------
>
> Key: TIKA-849
> URL: https://issues.apache.org/jira/browse/TIKA-849
> Project: Tika
> Issue Type: New Feature
> Components: mime, parser
> Affects Versions: 1.1
> Reporter: Andrew Jackson
> Attachments: ibooks-support.patch
>
>
> With the release of iBooks Author 1.0, Apple have created a new eBook format very similar to ePub. Tika could be extended to identify and parse this new format, re-using the existing ePub code wherever possible.
> I have created an initial patch, which I will attach to this issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TIKA-849) Identify and parse the Apple iBooks
format
Posted by "Nick Burch (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TIKA-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191242#comment-13191242 ]
Nick Burch commented on TIKA-849:
---------------------------------
Sample file committed in r1234886, along with a unit test for Zip Container Aware detection of epub files (epub seems to use the same scheme as the Open Document Format does)
> Identify and parse the Apple iBooks format
> ------------------------------------------
>
> Key: TIKA-849
> URL: https://issues.apache.org/jira/browse/TIKA-849
> Project: Tika
> Issue Type: New Feature
> Components: mime, parser
> Affects Versions: 1.1
> Reporter: Andrew Jackson
> Attachments: ibooks-support.patch
>
>
> With the release of iBooks Author 1.0, Apple have created a new eBook format very similar to ePub. Tika could be extended to identify and parse this new format, re-using the existing ePub code wherever possible.
> I have created an initial patch, which I will attach to this issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TIKA-849) Identify and parse the Apple iBooks
format
Posted by "Andrew Jackson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TIKA-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192078#comment-13192078 ]
Andrew Jackson commented on TIKA-849:
-------------------------------------
By the way, I notice you added a
<code>
<sub-class-of type="application/epub+zip" />
</code>
to the iBooks identification information. I'm not sure this is correct, as I was under the impression that this was a strict conformance hierarchy. This would imply that every x-ibooks+zip also conforms to the epub+zip format, but this is not the case, because the new MIME type declaration means it violated the ePub standard.
> Identify and parse the Apple iBooks format
> ------------------------------------------
>
> Key: TIKA-849
> URL: https://issues.apache.org/jira/browse/TIKA-849
> Project: Tika
> Issue Type: New Feature
> Components: mime, parser
> Affects Versions: 1.1
> Reporter: Andrew Jackson
> Attachments: ibooks-support.patch
>
>
> With the release of iBooks Author 1.0, Apple have created a new eBook format very similar to ePub. Tika could be extended to identify and parse this new format, re-using the existing ePub code wherever possible.
> I have created an initial patch, which I will attach to this issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TIKA-849) Identify and parse the Apple iBooks
format
Posted by "Nick Burch (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TIKA-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191259#comment-13191259 ]
Nick Burch commented on TIKA-849:
---------------------------------
Test and parser change committed in r1234904, thanks
It looks like all the text is stored within things like
<object type="application/x-ibooks+shape" id="textShape-89">
It's all in regular html div tags inside those objects, though sometimes that object is nested inside another of type "application/x-ibooks+flowhead"
Images seem to be within svg tags
I guess we'll need to do something for the ibooks case to treat these objects as regular xhtml?
> Identify and parse the Apple iBooks format
> ------------------------------------------
>
> Key: TIKA-849
> URL: https://issues.apache.org/jira/browse/TIKA-849
> Project: Tika
> Issue Type: New Feature
> Components: mime, parser
> Affects Versions: 1.1
> Reporter: Andrew Jackson
> Attachments: ibooks-support.patch
>
>
> With the release of iBooks Author 1.0, Apple have created a new eBook format very similar to ePub. Tika could be extended to identify and parse this new format, re-using the existing ePub code wherever possible.
> I have created an initial patch, which I will attach to this issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TIKA-849) Identify and parse the Apple iBooks
format
Posted by "Andrew Jackson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TIKA-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13217999#comment-13217999 ]
Andrew Jackson commented on TIKA-849:
-------------------------------------
Having thought about it, I think your proposed sub-type relationship is a more sensible approach, and my interpretation of the relationship was overly strict.
> Identify and parse the Apple iBooks format
> ------------------------------------------
>
> Key: TIKA-849
> URL: https://issues.apache.org/jira/browse/TIKA-849
> Project: Tika
> Issue Type: New Feature
> Components: mime, parser
> Affects Versions: 1.1
> Reporter: Andrew Jackson
> Attachments: ibooks-support.patch
>
>
> With the release of iBooks Author 1.0, Apple have created a new eBook format very similar to ePub. Tika could be extended to identify and parse this new format, re-using the existing ePub code wherever possible.
> I have created an initial patch, which I will attach to this issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (TIKA-849) Identify and parse the Apple iBooks
format
Posted by "Andrew Jackson (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TIKA-849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Jackson updated TIKA-849:
--------------------------------
Attachment: ibooks-support.patch
This patch identifies *.ibooks files, and passes them through the ePub parser. Unfortunately, the content of the eBook is not extracted successfully. I'm not sure why this is, and feedback would be appreciated here. I think the content*.xhtml files are actually HTML5 and it seems that the current XHTML-based parser is unable to parse them. A unit test and test file is included.
> Identify and parse the Apple iBooks format
> ------------------------------------------
>
> Key: TIKA-849
> URL: https://issues.apache.org/jira/browse/TIKA-849
> Project: Tika
> Issue Type: New Feature
> Components: mime, parser
> Affects Versions: 1.1
> Reporter: Andrew Jackson
> Attachments: ibooks-support.patch
>
>
> With the release of iBooks Author 1.0, Apple have created a new eBook format very similar to ePub. Tika could be extended to identify and parse this new format, re-using the existing ePub code wherever possible.
> I have created an initial patch, which I will attach to this issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TIKA-849) Identify and parse the Apple iBooks
format
Posted by "Andrew Jackson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TIKA-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192074#comment-13192074 ]
Andrew Jackson commented on TIKA-849:
-------------------------------------
I'm not that familiar with the content handling infrastructure of Tika at present, so I'm having trouble working out how to do this. I can see the ePubParser delegating the content handling to the ePubContentParser which uses an XHTMLContentHandler, but I don't really understand that code well enough to see why it discards the <object> elements and lets the others through. Or perhaps that filtering is done at a later stage?
In any case, I suppose we would need a new iBooksParser and a new iBooksContentParser, based on the ePub code, but using a different ContentHandler?
> Identify and parse the Apple iBooks format
> ------------------------------------------
>
> Key: TIKA-849
> URL: https://issues.apache.org/jira/browse/TIKA-849
> Project: Tika
> Issue Type: New Feature
> Components: mime, parser
> Affects Versions: 1.1
> Reporter: Andrew Jackson
> Attachments: ibooks-support.patch
>
>
> With the release of iBooks Author 1.0, Apple have created a new eBook format very similar to ePub. Tika could be extended to identify and parse this new format, re-using the existing ePub code wherever possible.
> I have created an initial patch, which I will attach to this issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira