[jira] [Created] (COR-31) Identification of Document Format Tool Progressions: Access, Creation, Testing, Assessment, Validation, Forensics

["Dennis E. Hamilton (JIRA)" <ji...@apache.org>]

Dennis E. Hamilton created COR-31:
-------------------------------------

             Summary: Identification of Document Format Tool Progressions: Access, Creation, Testing, Assessment, Validation, Forensics
                 Key: COR-31
                 URL: https://issues.apache.org/jira/browse/COR-31
             Project: Corinthia
          Issue Type: Task
            Reporter: Dennis E. Hamilton


There are many needs, and opportunities, for command-line and library-level tools that support the development of processors for different document formats.  

Many small tools can be developed as part of the application and verification of what will be larger solutions with regard to particular formats.  

This task is for identification of which such tools will be defined as work-product and deliverables for Corinthia, even in an initial provisional list.  Having an identified structure points for defined deliverables should aid in having different aspects of Corinthia available for development and testing by many hands and eyes.

SKETCH

There are different levels of tools, and the layers provide fixtures for exercising lower layers of code and also composing them into layers above.

To be concrete, here is a sketch of the levels of tooling that can be byproducts and aids in the confirmation of correct handling of a document format.

There are two "raw" formats that are handled in building document files of interest to us: text files and Zip packages (or other carriers of composite structures, such as MIME multi-part, tar files, Microsoft DocFiles, etc.).  

There are flat file formats atop text-file formats.  Examples are Microsoft RTF, XML, and HTML.  These are accompanied by character-set encoding variations that must be dealt with.  There are also cases of linking that arise in these formats.

RTF is a document format.  XML carries document formats such as the single-file ODF format, the single-file XML formats defined for Microsoft Office, etc.  There are already HTML-format usages that provide for fidelity preservation in round trip between HTML and Microsoft Office formats.  There may be something similar that has lived in OpenOffice.org.  These are very handy formats for creation of simple test documents that exercise the respective document models.  They also provide experience with the document formats and efforts to abstract the document that is represented in those formats.

Zip usage as carriers raises its own needs for well-defined tools, both for use in the inspection of document files but also the validation and forensic analysis of the Zip usage for ODF, OOXML, and other formats, such as ePub.  Now we're dealing with composite document files with multiple parts using flat formats, such as HTML and XML, and other formats, including binary formats not mentioned as part of this progressive layering.  There are now more elaborate structures to abstract from the parts of the Zip package and the cross-references among them.

These are all tooling opportunities and they support the testing and confirmation of the development of the document-processing functions that Corinthia makes available.

The richness of this can be illustrated by the need for forensic and validation tools and how they may become interdependent.

Consider the simple verification of a Zip file.  There are two levels of verification that matter.  

First there is of the fundamental invariant structure that a Zip archive must possess.  In practical use, it is desirable to rapidly abstract the presence of a correct Zip and its components.  It is desirable to be able to produce or update one efficiently.   One wants a fail-safe and resilient response when an unacceptable Zip is encountered.

At the same time, one wants a way to assess and inspect a Zip that is well-formed or is considered defective.  A separate tool would be handier for that, but needed to support document processing by providing inspection and reporting of how the Zip is unacceptable.  That's more involved and not something one wants to endure just to get going working with a document.  At the same time, there is a good case for some reused common code as well, and these kinds of tools aid in the confirmation of that code too.

Suppose a Zip is concluded to be damaged.  Another level is goes beyond detection of damage to determination of how much of the Zip can be recovered and what to do with the areas of damage.  This is about rescuing documents.  Yet another opportunity.  Yet another elaborate use that can involve some shared underlying code.

We're now at the second level and that intersects with the use of a Zip as a particular kind of document container.  A zip may be well-formed, but there are additional limitations and functions that go into recognizing the Zip usage as a carrier of a particular document format.  It can even be a generic carrier format, such as the Open Packaging Conventions (OPC) used for carrying OOXML, XPS, and other artifacts, and the OpenDocument 1.2 Package used for carrying ODF.

There need to be analysis and inspection tools at this second level of generic Zip usage.  This also has a cross-over value in the forensic problem of recovering what is recoverable in a damaged Zip archive.  When it is known what additional structure is expected to be present, this can inform the identification of breakage and determination of loss.

It's not all one-sided.  What appears to be a well-formed Zip package for a given document format can still expose damage in the recording or compression (oh yes, compression and decompression) of any of its parts.

This sketch is still at the plumbing level.  The abstraction of document features is yet to happen.  That's raising up another level.

This is all just to point out how many opportunities for tools and supporting libraries there are. The tools are important for bootstrapping up the levels of Corinthia and for being able to check our own work, to devise tests and demonstrations, and to provide forensic support in the face of problems that may arise in the software or simply in circumstances that arise for users.

The idea behind this task and its subtasks is to see what could be identified as point deliverables, even if fundamentally for our own work process, so that they become definable and something to work on, to be available in higher levels of operation, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)