You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openoffice.apache.org by Andre Fischer <aw...@gmail.com> on 2014/06/03 09:41:32 UTC
News about the new OOXML framework.
I would like to give a short status update about the new OOXML framework.
- Created the new module main/ooxml
There are not yet any makefiles that build the contents of the ooxml/
module nor link it into the build process of OpenOffice. However, you
can use e.g. Eclipse to import the Java projects that are described below.
- Moved the old Office Open XML wiki pages out of the way and create two
new ones:
= "OfficeOpenXML" contains an introduction into the OOXML file
format, a status overview of the implementation progress and links to
more detailed information.
= "OOXML" and "ooxml" (uppercase/lowercase) redirect to
"OfficeOpenXML" so that there is no excuse to not find this page.
= "OOXML Framework" contains more detailed information about the new
framework.
- Created a new Java project at ooxml/main/source/framework/SchemaParser
that parses the XML schema files that come with the ECMA-376
specification files.
Its purpose is to read the schema files and create a skeleton OOXML
parser from it. This skeleton can then be filled in with code for
importing certain elements of OOXML documents.
- Created a new Java project at
ooxml/main/source/framework/JavaOOXMLParser. Its purpose is testing and
debugging of and experimenting with the schema parser. It is not
intended to become a runtime component of OpenOffice.
The SchemaParser is able to parse all files of the ECMA-376
specification both in the old (1st edition of 2006) and new (4th edition
of 2012) versions. It looks like we need both since the new one is the
current standard (equivalent to the ISO standard) while the old on is
actually used.
Not all details of the schema files are handled yet.
The JavaOOXMLParser, based on parser tables created by the SchemaParser,
is already able to parse the large DOCX file of the 1st edition
specification. When pretty printed it is about 90 MB large. It takes
the parser about 90 s to read it. Note that the parser is not optimized
in any way (if it where then it would be optimized for readability, not
for speed) and that it writes about 650 MB of log files in the process.
If anyone would like to play with the parsers, I will gladly provide
more details.
Best regards,
Andre
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: News about the new OOXML framework.
Posted by Dave Fisher <wa...@apache.org>.
Great, makes sense. I've been way too busy at $job and am barely reading my email.
Regards,
Dave
Sent from my iPhone
> On Jun 3, 2014, at 7:41 AM, Jürgen Schmidt <jo...@gmail.com> wrote:
>
>> On 03/06/14 13:20, Dave Fisher wrote:
>> Hi,
>>
>> Are you aware of Apache POI?
>>
>> Reads and writes most Ooxml in Java.
>
> yes, we know POI and Andrea pointed already on thread where this was
> discussed.
>
> When you look in the details Java is used for the SchemaParser and later
> on to generate a C++ parser. No plans to use Java during runtime in the
> office to parse OOXML. But Java is perfect and productive for the
> development of the tooling etc.
>
> Juergen
>
>>
>> Regards,
>> Dave
>>
>> Apache POI and OpenOffice PMCs
>>
>> Sent from my iPhone
>>
>>> On Jun 3, 2014, at 3:41 AM, Andre Fischer <aw...@gmail.com> wrote:
>>>
>>> I would like to give a short status update about the new OOXML framework.
>>>
>>> - Created the new module main/ooxml
>>> There are not yet any makefiles that build the contents of the ooxml/ module nor link it into the build process of OpenOffice. However, you can use e.g. Eclipse to import the Java projects that are described below.
>>>
>>> - Moved the old Office Open XML wiki pages out of the way and create two new ones:
>>> = "OfficeOpenXML" contains an introduction into the OOXML file format, a status overview of the implementation progress and links to more detailed information.
>>> = "OOXML" and "ooxml" (uppercase/lowercase) redirect to "OfficeOpenXML" so that there is no excuse to not find this page.
>>> = "OOXML Framework" contains more detailed information about the new framework.
>>>
>>> - Created a new Java project at ooxml/main/source/framework/SchemaParser that parses the XML schema files that come with the ECMA-376 specification files.
>>> Its purpose is to read the schema files and create a skeleton OOXML parser from it. This skeleton can then be filled in with code for importing certain elements of OOXML documents.
>>>
>>> - Created a new Java project at ooxml/main/source/framework/JavaOOXMLParser. Its purpose is testing and debugging of and experimenting with the schema parser. It is not intended to become a runtime component of OpenOffice.
>>>
>>>
>>> The SchemaParser is able to parse all files of the ECMA-376 specification both in the old (1st edition of 2006) and new (4th edition of 2012) versions. It looks like we need both since the new one is the current standard (equivalent to the ISO standard) while the old on is actually used.
>>> Not all details of the schema files are handled yet.
>>>
>>> The JavaOOXMLParser, based on parser tables created by the SchemaParser, is already able to parse the large DOCX file of the 1st edition specification. When pretty printed it is about 90 MB large. It takes the parser about 90 s to read it. Note that the parser is not optimized in any way (if it where then it would be optimized for readability, not for speed) and that it writes about 650 MB of log files in the process.
>>>
>>> If anyone would like to play with the parsers, I will gladly provide more details.
>>>
>>> Best regards,
>>> Andre
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>> For additional commands, e-mail: dev-help@openoffice.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: News about the new OOXML framework.
Posted by Jürgen Schmidt <jo...@gmail.com>.
On 03/06/14 13:20, Dave Fisher wrote:
> Hi,
>
> Are you aware of Apache POI?
>
> Reads and writes most Ooxml in Java.
yes, we know POI and Andrea pointed already on thread where this was
discussed.
When you look in the details Java is used for the SchemaParser and later
on to generate a C++ parser. No plans to use Java during runtime in the
office to parse OOXML. But Java is perfect and productive for the
development of the tooling etc.
Juergen
>
> Regards,
> Dave
>
> Apache POI and OpenOffice PMCs
>
> Sent from my iPhone
>
>> On Jun 3, 2014, at 3:41 AM, Andre Fischer <aw...@gmail.com> wrote:
>>
>> I would like to give a short status update about the new OOXML framework.
>>
>> - Created the new module main/ooxml
>> There are not yet any makefiles that build the contents of the ooxml/ module nor link it into the build process of OpenOffice. However, you can use e.g. Eclipse to import the Java projects that are described below.
>>
>> - Moved the old Office Open XML wiki pages out of the way and create two new ones:
>> = "OfficeOpenXML" contains an introduction into the OOXML file format, a status overview of the implementation progress and links to more detailed information.
>> = "OOXML" and "ooxml" (uppercase/lowercase) redirect to "OfficeOpenXML" so that there is no excuse to not find this page.
>> = "OOXML Framework" contains more detailed information about the new framework.
>>
>> - Created a new Java project at ooxml/main/source/framework/SchemaParser that parses the XML schema files that come with the ECMA-376 specification files.
>> Its purpose is to read the schema files and create a skeleton OOXML parser from it. This skeleton can then be filled in with code for importing certain elements of OOXML documents.
>>
>> - Created a new Java project at ooxml/main/source/framework/JavaOOXMLParser. Its purpose is testing and debugging of and experimenting with the schema parser. It is not intended to become a runtime component of OpenOffice.
>>
>>
>> The SchemaParser is able to parse all files of the ECMA-376 specification both in the old (1st edition of 2006) and new (4th edition of 2012) versions. It looks like we need both since the new one is the current standard (equivalent to the ISO standard) while the old on is actually used.
>> Not all details of the schema files are handled yet.
>>
>> The JavaOOXMLParser, based on parser tables created by the SchemaParser, is already able to parse the large DOCX file of the 1st edition specification. When pretty printed it is about 90 MB large. It takes the parser about 90 s to read it. Note that the parser is not optimized in any way (if it where then it would be optimized for readability, not for speed) and that it writes about 650 MB of log files in the process.
>>
>> If anyone would like to play with the parsers, I will gladly provide more details.
>>
>> Best regards,
>> Andre
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: News about the new OOXML framework.
Posted by Andrea Pescetti <pe...@apache.org>.
Dave Fisher wrote:
> Are you aware of Apache POI?
> Reads and writes most Ooxml in Java.
Already discussed here: http://markmail.org/message/jhdsrqxfdczvoyy4
And thanks, Andre, for the nice progress and detailed information!
Regards,
Andrea.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: News about the new OOXML framework.
Posted by Dave Fisher <wa...@apache.org>.
Hi,
Are you aware of Apache POI?
Reads and writes most Ooxml in Java.
Regards,
Dave
Apache POI and OpenOffice PMCs
Sent from my iPhone
> On Jun 3, 2014, at 3:41 AM, Andre Fischer <aw...@gmail.com> wrote:
>
> I would like to give a short status update about the new OOXML framework.
>
> - Created the new module main/ooxml
> There are not yet any makefiles that build the contents of the ooxml/ module nor link it into the build process of OpenOffice. However, you can use e.g. Eclipse to import the Java projects that are described below.
>
> - Moved the old Office Open XML wiki pages out of the way and create two new ones:
> = "OfficeOpenXML" contains an introduction into the OOXML file format, a status overview of the implementation progress and links to more detailed information.
> = "OOXML" and "ooxml" (uppercase/lowercase) redirect to "OfficeOpenXML" so that there is no excuse to not find this page.
> = "OOXML Framework" contains more detailed information about the new framework.
>
> - Created a new Java project at ooxml/main/source/framework/SchemaParser that parses the XML schema files that come with the ECMA-376 specification files.
> Its purpose is to read the schema files and create a skeleton OOXML parser from it. This skeleton can then be filled in with code for importing certain elements of OOXML documents.
>
> - Created a new Java project at ooxml/main/source/framework/JavaOOXMLParser. Its purpose is testing and debugging of and experimenting with the schema parser. It is not intended to become a runtime component of OpenOffice.
>
>
> The SchemaParser is able to parse all files of the ECMA-376 specification both in the old (1st edition of 2006) and new (4th edition of 2012) versions. It looks like we need both since the new one is the current standard (equivalent to the ISO standard) while the old on is actually used.
> Not all details of the schema files are handled yet.
>
> The JavaOOXMLParser, based on parser tables created by the SchemaParser, is already able to parse the large DOCX file of the 1st edition specification. When pretty printed it is about 90 MB large. It takes the parser about 90 s to read it. Note that the parser is not optimized in any way (if it where then it would be optimized for readability, not for speed) and that it writes about 650 MB of log files in the process.
>
> If anyone would like to play with the parsers, I will gladly provide more details.
>
> Best regards,
> Andre
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: News about the new OOXML framework.
Posted by Jürgen Schmidt <jo...@gmail.com>.
On 03/06/14 09:41, Andre Fischer wrote:
> I would like to give a short status update about the new OOXML framework.
>
> - Created the new module main/ooxml
> There are not yet any makefiles that build the contents of the ooxml/
> module nor link it into the build process of OpenOffice. However, you
> can use e.g. Eclipse to import the Java projects that are described below.
>
> - Moved the old Office Open XML wiki pages out of the way and create two
> new ones:
> = "OfficeOpenXML" contains an introduction into the OOXML file format,
> a status overview of the implementation progress and links to more
> detailed information.
> = "OOXML" and "ooxml" (uppercase/lowercase) redirect to
> "OfficeOpenXML" so that there is no excuse to not find this page.
> = "OOXML Framework" contains more detailed information about the new
> framework.
for convenience, the wiki pages can be found under
https://wiki.openoffice.org/wiki/OOXML
https://wiki.openoffice.org/wiki/OOXML_Framework
>
> - Created a new Java project at ooxml/main/source/framework/SchemaParser
> that parses the XML schema files that come with the ECMA-376
> specification files.
> Its purpose is to read the schema files and create a skeleton OOXML
> parser from it. This skeleton can then be filled in with code for
> importing certain elements of OOXML documents.
>
> - Created a new Java project at
> ooxml/main/source/framework/JavaOOXMLParser. Its purpose is testing and
> debugging of and experimenting with the schema parser. It is not
> intended to become a runtime component of OpenOffice.
>
>
> The SchemaParser is able to parse all files of the ECMA-376
> specification both in the old (1st edition of 2006) and new (4th edition
> of 2012) versions. It looks like we need both since the new one is the
> current standard (equivalent to the ISO standard) while the old on is
> actually used.
> Not all details of the schema files are handled yet.
>
> The JavaOOXMLParser, based on parser tables created by the SchemaParser,
> is already able to parse the large DOCX file of the 1st edition
> specification. When pretty printed it is about 90 MB large. It takes
> the parser about 90 s to read it. Note that the parser is not optimized
> in any way (if it where then it would be optimized for readability, not
> for speed) and that it writes about 650 MB of log files in the process.
>
> If anyone would like to play with the parsers, I will gladly provide
> more details.
Thanks for the update, I think it's good to know that you are already
able to read both versions as described above.
I believe most users of OOXML don't care about the spec and even don't
know that the files produced today are by default OOXML transitional.
OOXML is far more complex than reading one spec ... but nobody cares
about the complexity and simply want pr expect a 1:1 support ;-)
Juergen
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: News about the new OOXML framework.
Posted by Kay Schenk <ka...@gmail.com>.
On Tue, Jun 10, 2014 at 2:58 AM, Andre Fischer <aw...@gmail.com> wrote:
> Another update of my progress.
>
> I can now create a validating parser, i.e. one that checks that a document
> conforms to the specs while it parses its content.
> At the moment the validation is restricted to complex types (as opposed to
> simple types and attributes) but I think that is the hardest part.
>
> One NFA (non-deterministic finite automaton) is created for each complex
> type and one for the top level elements. The NFAs are then converted into
> equivalent DFAs (deterministic FAs) and finally minimized (via the Hopcroft
> algorithm). The minimization step became necessary when I added support
> for the 'all' schema element which states that its children each occur once
> in arbitrary order. Recognizing this with an FA leads to enumerate all
> permutations of the children. With n children there are n! permutations.
> Luckily the 'all' element is used only once and then only for 7 children
> (7! = 5040).
>
> Here are some numbers:
> The 1st and 4th edition of the ECMA-376 specification (1st edition is what
> is used by MS Office, 4th edition is equivalent to the ISO standard) have
> 40 schema files.
> These contain 1917 complex types and 781 simple types.
> Used are 1851 complex types and 727 simple types (have to check if there
> are really unused complex types or if my optimization is broken).
>
> The non-validating parser has 1853 states and 6987 transitions.
>
> The validating parser has 129530 states and 43512 transitions after
> creating the NFAs.
> After conversion to DFAs there remain 20999 states and 73772 transitions.
> After minimization there are 6097 states and 34286 transitions.
>
> Please note that the time for parsing OOXML documents does not depend on
> the number of states or transitions. It only depends on the length of the
> input. The number of states and transitions only make the parser bigger.
>
> Progress and commits are tracked in issue 125035.
Thanks for the update -- very impressive!
>
>
> Best regards,
> Andre
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
>
--
-------------------------------------------------------------------------------------------------
MzK
"In the midst of winter, I found there was, within me,
an invincible summer."
-- Albert Camus
Re: News about the new OOXML framework.
Posted by Oliver-Rainer Wittmann <or...@googlemail.com>.
Hi,
in the last days I made up my mind about a further part of the OOXML
framework - namely some tooling to handle the different XML streams
(called 'part' in the OOXML specification) and the corresponding meta
data about its content types and its relations to each other (called
'relationships' in the OOXML specification).
Based on Andre's framework and in some pair-programming sessions with
Andre a prototype in Java had been worked out to verify my ideas. This
prototype was also the first usage case to create corresponding imports
of the meta data using Andre's prototyped OOXMLParser.
The ideas could be implemented quite well and worked. Further analysis
and concept validation will follow.
Best regards, Oliver.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: News about the new OOXML framework.
Posted by Jürgen Schmidt <jo...@gmail.com>.
On 10/06/14 23:54, Marcus (OOo) wrote:
> Am 06/10/2014 11:58 AM, schrieb Andre Fischer:
>> Another update of my progress.
>>
>> I can now create a validating parser, i.e. one that checks that a
>> document conforms to the specs while it parses its content.
>> At the moment the validation is restricted to complex types (as opposed
>> to simple types and attributes) but I think that is the hardest part.
>>
>> One NFA (non-deterministic finite automaton) is created for each complex
>> type and one for the top level elements. The NFAs are then converted
>> into equivalent DFAs (deterministic FAs) and finally minimized (via the
>> Hopcroft algorithm). The minimization step became necessary when I added
>> support for the 'all' schema element which states that its children each
>> occur once in arbitrary order. Recognizing this with an FA leads to
>> enumerate all permutations of the children. With n children there are n!
>> permutations. Luckily the 'all' element is used only once and then only
>> for 7 children (7! = 5040).
>>
>> Here are some numbers:
>> The 1st and 4th edition of the ECMA-376 specification (1st edition is
>> what is used by MS Office, 4th edition is equivalent to the ISO
>> standard) have 40 schema files.
>> These contain 1917 complex types and 781 simple types.
>> Used are 1851 complex types and 727 simple types (have to check if there
>> are really unused complex types or if my optimization is broken).
>>
>> The non-validating parser has 1853 states and 6987 transitions.
>>
>> The validating parser has 129530 states and 43512 transitions after
>> creating the NFAs.
>> After conversion to DFAs there remain 20999 states and 73772 transitions.
>> After minimization there are 6097 states and 34286 transitions.
>>
>> Please note that the time for parsing OOXML documents does not depend on
>> the number of states or transitions. It only depends on the length of
>> the input. The number of states and transitions only make the parser
>> bigger.
>>
>> Progress and commits are tracked in issue 125035.
>
> that's existing. I cannot really follow the technical stuff and what
> this all means but it sounds like a great progress in the OOXML area.
> And for me this is exiting.
well it simply means that we spent already a lot of time in analyzing
the stuff we have, what and how we can improve it. And it shows that we
follow a very professional approach ;-)
We mean it serious with an improved and better OOXML support including
export. It is what users need and expect because the interoperability
with MS Office is still very important and OOXML becomes more and more
popular and broader used. If we like it or not.
This covers also a lot of ground work, refactoring and feature
development over time to make it possible. The parsing framework is only
one but a very important part of it.
And we will propose, discuss all the things we have in mind in detail
here on the list that everybody who is interested can join our efforts.
Juergen
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: News about the new OOXML framework.
Posted by "Marcus (OOo)" <ma...@wtnet.de>.
Am 06/10/2014 11:58 AM, schrieb Andre Fischer:
> Another update of my progress.
>
> I can now create a validating parser, i.e. one that checks that a
> document conforms to the specs while it parses its content.
> At the moment the validation is restricted to complex types (as opposed
> to simple types and attributes) but I think that is the hardest part.
>
> One NFA (non-deterministic finite automaton) is created for each complex
> type and one for the top level elements. The NFAs are then converted
> into equivalent DFAs (deterministic FAs) and finally minimized (via the
> Hopcroft algorithm). The minimization step became necessary when I added
> support for the 'all' schema element which states that its children each
> occur once in arbitrary order. Recognizing this with an FA leads to
> enumerate all permutations of the children. With n children there are n!
> permutations. Luckily the 'all' element is used only once and then only
> for 7 children (7! = 5040).
>
> Here are some numbers:
> The 1st and 4th edition of the ECMA-376 specification (1st edition is
> what is used by MS Office, 4th edition is equivalent to the ISO
> standard) have 40 schema files.
> These contain 1917 complex types and 781 simple types.
> Used are 1851 complex types and 727 simple types (have to check if there
> are really unused complex types or if my optimization is broken).
>
> The non-validating parser has 1853 states and 6987 transitions.
>
> The validating parser has 129530 states and 43512 transitions after
> creating the NFAs.
> After conversion to DFAs there remain 20999 states and 73772 transitions.
> After minimization there are 6097 states and 34286 transitions.
>
> Please note that the time for parsing OOXML documents does not depend on
> the number of states or transitions. It only depends on the length of
> the input. The number of states and transitions only make the parser
> bigger.
>
> Progress and commits are tracked in issue 125035.
that's existing. I cannot really follow the technical stuff and what
this all means but it sounds like a great progress in the OOXML area.
And for me this is exiting.
Marcus
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: News about the new OOXML framework.
Posted by Andre Fischer <aw...@gmail.com>.
Another update of my progress.
I can now create a validating parser, i.e. one that checks that a
document conforms to the specs while it parses its content.
At the moment the validation is restricted to complex types (as opposed
to simple types and attributes) but I think that is the hardest part.
One NFA (non-deterministic finite automaton) is created for each complex
type and one for the top level elements. The NFAs are then converted
into equivalent DFAs (deterministic FAs) and finally minimized (via the
Hopcroft algorithm). The minimization step became necessary when I
added support for the 'all' schema element which states that its
children each occur once in arbitrary order. Recognizing this with an FA
leads to enumerate all permutations of the children. With n children
there are n! permutations. Luckily the 'all' element is used only once
and then only for 7 children (7! = 5040).
Here are some numbers:
The 1st and 4th edition of the ECMA-376 specification (1st edition is
what is used by MS Office, 4th edition is equivalent to the ISO
standard) have 40 schema files.
These contain 1917 complex types and 781 simple types.
Used are 1851 complex types and 727 simple types (have to check if there
are really unused complex types or if my optimization is broken).
The non-validating parser has 1853 states and 6987 transitions.
The validating parser has 129530 states and 43512 transitions after
creating the NFAs.
After conversion to DFAs there remain 20999 states and 73772 transitions.
After minimization there are 6097 states and 34286 transitions.
Please note that the time for parsing OOXML documents does not depend on
the number of states or transitions. It only depends on the length of
the input. The number of states and transitions only make the parser
bigger.
Progress and commits are tracked in issue 125035.
Best regards,
Andre
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org