You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@daffodil.apache.org by sl...@apache.org on 2018/02/06 17:05:50 UTC
[incubator-daffodil-site] branch master updated: Miscellaneous
website updates
This is an automated email from the ASF dual-hosted git repository.
slawrence pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-daffodil-site.git
The following commit(s) were added to refs/heads/master by this push:
new 2e855eb Miscellaneous website updates
2e855eb is described below
commit 2e855eb10f9ef67e16a60f9ac71892a87126be2f
Author: Steve Lawrence <sl...@tresys.com>
AuthorDate: Tue Feb 6 11:08:03 2018 -0500
Miscellaneous website updates
- Add /infoset and /tdml pages from old wiki
- Add Josh Adams' githubID
- Modify GitHub link to point to both the daffodil and daffodil-site repos
- Update the community page to provide a bit more information
- Updates to 2.1.0 changelog page to include missing bugs and clean up
grammar
---
site/_data/contributors.yml | 2 +-
site/_includes/themes/apache/_navigation.html | 4 +-
site/_releases/2.1.0.md | 48 ++-
site/community.md | 44 ++-
site/infoset.md | 513 ++++++++++++++++++++++++++
site/tdml.md | 295 +++++++++++++++
6 files changed, 868 insertions(+), 38 deletions(-)
diff --git a/site/_data/contributors.yml b/site/_data/contributors.yml
index 352134a..e2c3bb7 100644
--- a/site/_data/contributors.yml
+++ b/site/_data/contributors.yml
@@ -16,7 +16,7 @@
- first: Josh
last: Adams
apacheId: jadams
- githubId:
+ githubId: jadams-tresys
role: PPMC
org: Tresys Technology
diff --git a/site/_includes/themes/apache/_navigation.html b/site/_includes/themes/apache/_navigation.html
index 9232c9a..4c4d573 100644
--- a/site/_includes/themes/apache/_navigation.html
+++ b/site/_includes/themes/apache/_navigation.html
@@ -37,8 +37,8 @@
<li id="development">
<a href="#" data-toggle="dropdown" class="dropdown-toggle">Development<b class="caret"></b></a>
<ul class="dropdown-menu dropdown-left">
- <li><a class="external" href="https://cwiki.apache.org/confluence/display/DAFFODIL/Code+Contributor+Workflow">Contributor Workflow</a></li>
- <li><a class="external" href="https://github.com/apache/incubator-daffodil">GitHub</a></li>
+ <li><a class="external" href="https://cwiki.apache.org/confluence/display/DAFFODIL/">Wiki</a></li>
+ <li><a class="external" href="https://github.com/search?q=repo%3Aapache%2Fincubator-daffodil+repo%3Aapache%2Fincubator-daffodil-site&type=Repositories">GitHub</a></li>
<li><a class="external" href="https://issues.apache.org/jira/projects/DAFFODIL/">JIRA</a></li>
</ul>
</li>
diff --git a/site/_releases/2.1.0.md b/site/_releases/2.1.0.md
index 93fa211..0219985 100644
--- a/site/_releases/2.1.0.md
+++ b/site/_releases/2.1.0.md
@@ -50,50 +50,64 @@ Also implemented are ``dfdl:binaryPackedSignCodes`` and
#### Apache Incubation Support
-Multiple changes were made to support the move to Apache incubation. This
+Multiple changes were made to support the move to the Apache incubator. This
includes changes to the source directory structure, relicensing files to Apache
v2, changing the project package to org.apache.daffodil, updating the build
infrastructure to SBT 1.x, and adding TravisCI continuous integration support.
+* {% jira 1612 %} sbt - branches without tags cause sbt compile to fail
+* {% jira 1841 %} daffodil root jar is published but shouldn't be, and its contents are wrong
* {% jira 1856 %} Changes necessary for apache incubation
* {% jira 1857 %} Update sbt to 1.x, simplifing build config and replacing deprecated syntax
-* {% jira 1870 %} invalid and nonfunctional URLs in source files
* {% jira 1858 %} Add support for TravisCI
+* {% jira 1859 %} Switch to apache.org namespace and Apache v2 license
+* {% jira 1870 %} invalid and nonfunctional URLs in source files
* {% jira 1873 %} sbt warnings/errors
-* {% jira 1612 %} sbt - branches without tags cause sbt compile to fail
#### Ambiguous Path Expressions
Previously, ambiguous path expressions (i.e. path expressions that could
-potentially reference multiple non-array elements of the same name) were not
-allowed and caused a schema definition error at schema. As of 2.1.0, such
-ambiguities are detected at runtime but are now only considered schema
-definition warnings. However, if a path expression ever evaluates to more than
-on non-array element at runtime, it will then be considered an error and cause a
-runtime schema definition error. This can allow for schemas that have ambiguous
-path expression, but in practice only ever evaluate to a single element.
+potentially reference multiple non-array elements of the same name) were
+forbidden and caused a schema definition error at schema compilation time. As
+of 2.1.0, such ambiguities are detected at compile time but are now only
+considered schema definition warnings. However, if a path expression ever
+evaluates to more than one non-array element at runtime, it will then be
+considered an error and cause a runtime schema definition error. This change
+allows for schemas that appear to have ambiguous path expressions, but in
+only ever evaluate to a single element in practice.
* {% jira 1773 %} Choice ambiguous element name results in failed expression
-* {% jira 1869 %} Nato link16 doesn't work on 2.0.0 Daffodil
-* {% jira 1877 %} Slow schema compilation due to unambiguous checks
* {% jira 1854 %} Remove concept of slots from InfosetImpl.scala
* {% jira 1860 %} Minimize the number of HashMap allocations/insertions in the Infoset
+* {% jira 1869 %} Nato link16 doesn't work on 2.0.0 Daffodil
+* {% jira 1877 %} Slow schema compilation due to unambiguous checks
#### TDML Runner Enhancements
-Changes were made to the TDML runner to support embedded defaultConfig option
-and to verify expected schema definition warnings.
+Changes were made to the TDML runner to support an embedded defaultConfig
+option and to verify expected schema definition warnings.
-* {% jira 1868 %} TDML Runner defaultConfig cannot be embedded. Not checked properly.
* {% jira 1583 %} TDML Runner - expected warnings feature
* {% jira 1846 %} TDML Runner - Infoset.contents is stripping off all attributes
+* {% jira 1868 %} TDML Runner defaultConfig cannot be embedded. Not checked properly.
#### BitOrder/ByteOrder/Encoding Refactoring
+The way bitOrder, byteOrder, and encoding consistency was checked required
+fairly complicated logic at schema compilation time. Not only was this logic
+difficult to validate, which led to bugs, it also contributed to slow schema
+compilation times. This complex logic was removed and replaced with an
+efficient mechanism for runtime validation. This reduced code complexity, fixerd
+some bugs, and will facilitate future performance optimizations during schema
+compilation.
+
* {% jira 1001 %} Invalid bitOrder and byteOrder combination should produce SDE
* {% jira 1597 %} Too many ways that encoding, byteOrder, etc. are being setup
+* {% jira 1732 %} BitOrderChangeParser not inserted correctly for bitmap schema
+* {% jira 1767 %} byteOrderChangeParser optimization not working
* {% jira 1843 %} Unparser bitOrder change and OVC (outputValueCalc) interaction
* {% jira 1876 %} Improve performance regression due to FormatInfo changes
+* {% jira 1884 %} Regression in bitOrder changing
#### XML Output
@@ -107,12 +121,12 @@ users reported of receiving only partial data.
#### Miscellaneous Bug Fixes
-* {% jira 1867 %} Memory leak in Register
-* {% jira 1862 %} Better logging/reset mechanism for MarkPool
* {% jira 1851 %} Unparsing delimited data with escape characters could lead to truncation
* {% jira 1855 %} Property Resolution - resolve onto ref objects
+* {% jira 1862 %} Better logging/reset mechanism for MarkPool
* {% jira 1864 %} spurious SDE about all-text schema being non-scannable
* {% jira 1866 %} Eclipse classpaths need fixing
+* {% jira 1867 %} Memory leak in Register
* {% jira 1872 %} Eclipse classpaths for daffodil-io module need fixing
#### Deprecation/Compatability
diff --git a/site/community.md b/site/community.md
index e798578..ea07fb0 100644
--- a/site/community.md
+++ b/site/community.md
@@ -35,35 +35,43 @@ You can:
* Report bugs and submit patches.
* Contribute code, javadocs, documentation.
-### Mailing list
+### Mailing Lists
Get help using Daffodil or contribute to the project on our mailing lists:
-* [users@daffodil.incubator.apache.org](mailto:users@daffodil.incubator.apache.org) is for usage questions, help, and announcements -
- [subscribe](mailto:users-subscribe@daffodil.incubator.apache.org?subject=send this email to subscribe),
- [unsubscribe](mailto:users-unsubscribe@daffodil.incubator.apache.org?subject=send this email to unsubscribe),
- [archives](https://lists.apache.org/list.html?users@daffodil.apache.org)
-* [dev@daffodil.incubator.apache.org](mailto:dev@daffodil.incubator.apache.org) is for people who want to contribute code to Daffodil -
- [subscribe](mailto:dev-subscribe@daffodil.incubator.apache.org?subject=send this email to subscribe),
- [unsubscribe](mailto:dev-unsubcribe@daffodil.incubator.apache.org?subject=send this email to unsubscribe),
- [archives](https://lists.apache.org/list.html?dev@daffodil.apache.org)
-* [commits@daffodil.incubator.apache.org](mailto:comits@daffodil.incubator.apache.org) is for commit messages and patches to Daffodil -
- [subscribe](mailto:commits-subscribe@daffodil.incubator.apache.org?subject=send this email to subscribe),
- [unsubscribe](mailto:commits-unsubcribe@daffodil.incubator.apache.org?subject=send this email to unsubscribe),
- [archives](https://lists.apache.org/list.html?commits@daffodil.apache.org)
+[users@daffodil.apache.org](mailto:users@daffodil.apache.org) -- [subscribe](mailto:users-subscribe@daffodil.apache.org?subject=send this email to subscribe), [unsubscribe](mailto:users-unsubscribe@daffodil.apache.org?subject=send this email to unsubscribe), [archives](https://lists.apache.org/list.html?users@daffodil.apache.org)
+
+ : For usage/help questions (e.g. DFDL schema creation, CLI/API usage) and announcements
+
+[dev@daffodil.apache.org](mailto:dev@daffodil.apache.org) -- [subscribe](mailto:dev-subscribe@daffodil.apache.org?subject=send this email to subscribe), [unsubscribe](mailto:dev-unsubcribe@daffodil.apache.org?subject=send this email to unsubscribe), [archives](https://lists.apache.org/list.html?dev@daffodil.apache.org)
+
+ : For those who want to contribute code to Daffodil, join in on design discussions, track issues and Daffodil development, etc.
+
+[commits@daffodil.apache.org](mailto:comits@daffodil.apache.org) -- [subscribe](mailto:commits-subscribe@daffodil.apache.org?subject=send this email to subscribe), [unsubscribe](mailto:commits-unsubcribe@daffodil.apache.org?subject=send this email to unsubscribe), [archives](https://lists.apache.org/list.html?commits@daffodil.apache.org)
+
+ : For tracking new commits to the Daffodil codebase
### Issue tracker
#### Bug Reports
-Found bug? Enter an issue in the [Daffodil Issue Tracker](https://issues.apache.org/jira/browse/DAFFODIL) on JIRA.
+Found bug? Enter an issue in the [Daffodil Issue Tracker](https://issues.apache.org/jira/browse/DAFFODIL)
+on JIRA. Be sure to provide as much detail as possible in the issue description
+on how to reproduce the bug. It is also helpful and highly encouraged to create
+and submit a [Test Data Markup Language (TMDL)](/tdml) file that illustrates
+the issue, enabling developers to quickly reproduce the issue.
Before submitting an issue, please:
* Verify that the bug does in fact exist.
-* Search the issue tracker to verify there is no existing issue reporting the bug you've found.
-* Consider tracking down the bug yourself in the Wink's source and submitting a patch along with your bug report. This is a great time saver for the Wink developers and helps ensure the bug will be fixed quickly.
+* Search the issue tracker to verify there are no existing issues reporting the
+ same bug you have found. If you do find an existing bug that sounds similar or
+ closely related, feel free to add your information as a comment to provide
+ extra details to help the developers fix the issue.
+* Consider tracking down and fixing the bug and submitting a pull request along
+ with creating a bug report. Visit the [Daffodil Developers wiki](https://cwiki.apache.org/confluence/display/DAFFODIL/)
+ for information to get started contributing.
#### Feature Requests
@@ -71,8 +79,8 @@ Enhancement requests for new features are also welcome. The more concrete and ra
### Source Code
-The project sources are accessible via [GitHub](https://github.com/apache/incubator-daffodil) and [ASF](https://gitbox.apache.org/repos/asf?p=incubator-daffodil.git)
+The project sources are accessible via [GitHub](https://github.com/apache/incubator-daffodil) and [ASF](https://gitbox.apache.org/repos/asf?p=incubator-daffodil.git).
### Website Source Code
-The project website sources are accessible via [GitHub](https://github.com/apache/incubator-daffodil-site) and [ASF](https://gitbox.apache.org/repos/asf?p=incubator-daffodil-site.git)
+The project website sources are accessible via [GitHub](https://github.com/apache/incubator-daffodil-site) and [ASF](https://gitbox.apache.org/repos/asf?p=incubator-daffodil-site.git).
diff --git a/site/infoset.md b/site/infoset.md
new file mode 100644
index 0000000..7cd365b
--- /dev/null
+++ b/site/infoset.md
@@ -0,0 +1,513 @@
+---
+layout: page
+title: Daffodil and the DFDL Infoset
+description: Daffodil and the DFDL Infoset
+group: nav-right
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+
+## Daffodil and the DFDL Infoset
+
+Daffodil is an implementation of DFDL which supports multiple methods to
+represent the DFDL Infoset, including various XML representations and JSON.
+However, the DFDL Infoset is somewhat different from the representations that
+Daffodil creates since Daffodil approximates the DFDL Infoset using a subset of
+features of XML/JSON. The below tables describe how Daffodil maps the DFDL
+Infoset to the supported representations.
+
+<ul class="nav nav-tabs">
+ <li class="active"><a data-toggle="tab" href="#jdom">JDOM</a></li>
+ <li><a data-toggle="tab" href="#w3cdom">W3C DOM</a></li>
+ <li><a data-toggle="tab" href="#scalanode">Scala Node</a></li>
+ <li><a data-toggle="tab" href="#xmltext">XML Text</a></li>
+ <li><a data-toggle="tab" href="#json">JSON</a></li>
+</ul>
+
+<div class="tab-content">
+ <div id="jdom" class="tab-pane fade in active">
+ <table class="table">
+ <tr>
+ <td class="col-md-3">Document Information Item</td>
+ <td class="col-md-9">org.jdom.Document</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">root</td>
+ <td class="col-md-9">org.jdom.Element getRootElement()</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">dfdlVersion</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">schema (reserved for future use)</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">unicodeByteOrderMark</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">Element Information Item</td>
+ <td class="col-md-9">org.jdom.Element</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">namespace</td>
+ <td class="col-md-9">org.jdom.Namespace getNamespace()</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">name</td>
+ <td class="col-md-9">String getName()</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">document</td>
+ <td class="col-md-9">org.jdom.Document getDocument()</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">datatype</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">dataValue</td>
+ <td class="col-md-9">
+ For simple types other than xs:string, the canonical
+ XML representation of the value as returned by String getText(). See <a
+ href="#xml-illegal-characters">XML Illegal Characters</a> for xs:string
+ types containing XML illegal characters.
+ </td>
+ </tr>
+ <tr>
+ <td class="col-md-3">nilled</td>
+ <td class="col-md-9">The "nilled" attribute in the "xsi" namespace.</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">children</td>
+ <td class="col-md-9">java.util.List<Element> getChildren()</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">parent</td>
+ <td class="col-md-9">org.jdom.Parent getParent()</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">schema</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">valid</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">unionMemberSchema</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">"No Value"</td>
+ <td class="col-md-9">An org.jdom.Element with no children (not even Text nodes) is the representation of an element with "no value".</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">Augmented Infoset</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ </table>
+ </div>
+ <div id="w3cdom" class="tab-pane fade">
+ <table class="table">
+ <tr>
+ <td class="col-md-3">Document Information Item</td>
+ <td class="col-md-9">org.w3c.dom.Document</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">root</td>
+ <td class="col-md-9">org.w3c.dom.Node getFirstChild()</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">dfdlVersion</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">schema (reserved for future use)</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">unicodeByteOrderMark</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">Element Information Item</td>
+ <td class="col-md-9">org.w3c.dom.Element</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">namespace</td>
+ <td class="col-md-9">String getNamespaceURI()</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">name</td>
+ <td class="col-md-9">String getNodeName() if getNamespaceURI() == null, String getLocalName() otherwise</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">document</td>
+ <td class="col-md-9">org.jdom.Document getOwnerDocument()</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">datatype</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">dataValue</td>
+ <td class="col-md-9">
+ For simple types other than xs:string, the canonical
+ XML representation of the value as returned by String getWholeText(). See <a
+ href="#xml-illegal-characters">XML Illegal Characters</a> for xs:string
+ types containing XML illegal characters.
+ </td>
+ </tr>
+ <tr>
+ <td class="col-md-3">nilled</td>
+ <td class="col-md-9">The "nilled" attribute in the "xsi" namespace.</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">children</td>
+ <td class="col-md-9">org.w3c.dom.NodeList getChildNodes()</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">parent</td>
+ <td class="col-md-9">org.w3c.dom.Node getParentNode()</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">schema</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">valid</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">unionMemberSchema</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">"No Value"</td>
+ <td class="col-md-9">An org.w3c.dom.Element with no children (not even Text nodes) is the representation of an element with "no value".</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">Augmented Infoset</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ </table>
+ </div>
+ <div id="scalanode" class="tab-pane fade">
+ <table class="table">
+ <tr>
+ <td class="col-md-3">Document Information Item</td>
+ <td class="col-md-9">The document is represented by the root element. There is no separate document item.</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">root</td>
+ <td class="col-md-9"><i>not supported</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">dfdlVersion</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">schema (reserved for future use)</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">unicodeByteOrderMark</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">Element Information Item</td>
+ <td class="col-md-9">scala.xml.Elem</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">namespace</td>
+ <td class="col-md-9">def namespace: String</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">name</td>
+ <td class="col-md-9">def name: String</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">document</td>
+ <td class="col-md-9"><i>not supported</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">datatype</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">dataValue</td>
+ <td class="col-md-9">
+ For simple types other than xs:string, the canonical
+ XML representation of the value as returned by def text: String. See <a
+ href="#xml-illegal-characters">XML Illegal Characters</a> for xs:string
+ types containing XML illegal characters.
+ </td>
+ </tr>
+ <tr>
+ <td class="col-md-3">nilled</td>
+ <td class="col-md-9">The "nilled" attribute in the "xsi" namespace.</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">children</td>
+ <td class="col-md-9">def child: Node*</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">parent</td>
+ <td class="col-md-9"><i>not supported</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">schema</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">valid</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">unionMemberSchema</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">"No Value"</td>
+ <td class="col-md-9">A scala.xml.Elem with no children.</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">Augmented Infoset</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ </table>
+ </div>
+ <div id="xmltext" class="tab-pane fade">
+ <table class="table">
+ <tr>
+ <td class="col-md-3">Document Information Item</td>
+ <td class="col-md-9">The full text is the document.</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">root</td>
+ <td class="col-md-9">The first XML tag in the document.</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">dfdlVersion</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">schema (reserved for future use)</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">unicodeByteOrderMark</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">Element Information Item</td>
+ <td class="col-md-9">An XML tag</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">namespace</td>
+ <td class="col-md-9">Defined using standard XML namespacing (e.g. xmlns="..." and element prefixes)</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">name</td>
+ <td class="col-md-9">XML tag name</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">document</td>
+ <td class="col-md-9">The full text is the document</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">datatype</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">dataValue</td>
+ <td class="col-md-9">
+ For simple types other than xs:string, the canonical
+ XML representation of the value inside the opening/closing XML tags. See <a
+ href="#xml-illegal-characters">XML Illegal Characters</a> for xs:string
+ types containing XML illegal characters.
+ </td>
+ </tr>
+ <tr>
+ <td class="col-md-3">nilled</td>
+ <td class="col-md-9">The "nilled" attribute in the "xsi" namespace.</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">children</td>
+ <td class="col-md-9">Child XML tags</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">parent</td>
+ <td class="col-md-9">Parent XML tags</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">schema</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">valid</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">unionMemberSchema</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">"No Value"</td>
+ <td class="col-md-9">An XML tag with no content in between the opening and closing tags</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">Augmented Infoset</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ </table>
+ </div>
+ <div id="json" class="tab-pane fade">
+ <table class="table">
+ <tr>
+ <td class="col-md-3">Document Information Item</td>
+ <td class="col-md-9">The full text is the document, containing a JSON single object.</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">root</td>
+ <td class="col-md-9">The first (and only) JSON string in the doucment object.</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">dfdlVersion</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">schema (reserved for future use)</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">unicodeByteOrderMark</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">Element Information Item</td>
+ <td class="col-md-9">The first JSON string in an object.</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">namespace</td>
+ <td class="col-md-9"><i>not supported</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">name</td>
+ <td class="col-md-9">The first JSON string in an object.</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">document</td>
+ <td class="col-md-9">The full text is the document</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">datatype</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">dataValue</td>
+ <td class="col-md-9">
+ For simple types other than xs:string, the canonical XML representation
+ of the value inside double quotes. For xs:string types, a JSON escaped
+ string in double quotes.
+ </td>
+ </tr>
+ <tr>
+ <td class="col-md-3">nilled</td>
+ <td class="col-md-9">The value of the element is <i>null</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">children</td>
+ <td class="col-md-9">Child JSON objects</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">parent</td>
+ <td class="col-md-9">Parent JSON tags</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">schema</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">valid</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">unionMemberSchema</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ <tr>
+ <td class="col-md-3">"No Value"</td>
+ <td class="col-md-9">The value of the element is empty double quotes.</td>
+ </tr>
+ <tr>
+ <td class="col-md-3">Augmented Infoset</td>
+ <td class="col-md-9"><i>not yet implemented</i></td>
+ </tr>
+ </table>
+ </div>
+</div>
+
+### XML Illegal Characters
+
+Since DFDL strings can contain characters that are not allowed in XML at all,
+for the XML based representations, these characters are mapped into the Unicode
+Private Use Areas (PUA). This is similar to the scheme used by Microsoft
+Visio (See: <a href="http://msdn.microsoft.com/en-us/library/office/aa218415%28v=office.10%29.aspx">http://msdn.microsoft.com/en-us/library/office/aa218415%28v=office.10%29.aspx</a>),
+but extended to handle all the XML 1.0 illegal characters including those
+with 16-bit codepoint values. This mapping is used bi-directionally, that is,
+illegal characters are replaced by their legal counterparts when parsing, and
+the reverse transformation is performed when unparsing, thereby allowing the
+creation of data containing the XML illegal characters from legal XML
+documents that contain only the mapped PUA corresponding characters.
+
+These are the legal XML characters (for XML v1.0):
+
+```
+#x0 | #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF] | #xD (treated specially)
+```
+
+Illegal characters from ``#x00`` to ``#x1F`` are mapped to the PUA
+by adding ``#xE000`` to their character code.
+
+Illegal characters from ``#xD800`` to ``#xDFFF`` are mapped to the PUA by adding
+``#x1000`` to their character code. So ``#xD800`` maps to ``#xE800``, and
+``#xDFFF`` maps to ``#xEFFF``.
+
+Illegal characters ``#xFFFE`` and ``#xFFFF`` are mapped to the PUA by
+subtracting ``#x0F00`` from their character code, so to characters ``#xF0FE``
+and ``#xF0FF``.
+
+Character ``#xD`` (Carriage Return or CR) is mapped to ``#xA`` (Line Feed, or
+LF). The CR character is allowed in the textual representation of XML
+documents, but is always converted to LF in the XML Infoset. That is, it is
+read by XML processors, but CRLF is converted to just LF, and CR alone is
+converted to LF. Daffodil is in a sense a different 'reader' of data into the
+XML infoset, so to be consistent with XML we map CR and CRLF to LF.
+
+It is a processing error when parsing if any DFDL infoset string contains
+characters in the parts of the PUA used by this mapping for illegal XML
+codepoints.
+
+It is a processing error if any DFDL infoset string character is created with a
+character code greater than ``#x10FFFF``.
diff --git a/site/tdml.md b/site/tdml.md
new file mode 100644
index 0000000..f4e118b
--- /dev/null
+++ b/site/tdml.md
@@ -0,0 +1,295 @@
+---
+layout: page
+title: Test Data Markup Language (TDML)
+description: Test Data Markup Language
+group: nav-right
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+
+## Test Data Markup Language (TDML)
+
+Test Data Markup Language (TDML) it is a way of specifying a DFDL schema, input
+test data, and expected result or expected error/diagnostic messages, all
+self-contained in an XML file. IBM created TDML to capture tests for their own
+DFDL implementation. Daffodil latched onto this and has since extended it a
+bit, though there is an effort to reconcile TDML dialects so that all
+implementations can run the same tests.
+
+A TDML file is often useful just to ask a question about how something in DFDL
+works, for example, to get a clarification. It allows for a level of precision
+that is often lacking, but also often required when discussing complex data
+format issues. As such, providing a TDML file along with a bug report is the
+absolutely best way to demonstrate a problem.
+
+By convention, a TDML file uses the file extension ``.tdml``, or ``.tdml.xml``
+when used with the TDML "tutorial" feature.
+
+The schema for a TDML file is available on [GitHub](https://github.com/apache/incubator-daffodil/blob/master/daffodil-lib/src/main/resources/org/apache/daffodil/xsd/tdml.xsd).
+
+Below is an annotated TDML file for a very simple example:
+
+```xml
+<?xml version="1.0" encoding="ASCII"?>
+
+<tdml:testSuite
+ suiteName="Bug Report TDML Template"
+ description="Illustration of TDML for bug reporting."
+ xmlns:tdml="http://www.ibm.com/xmlns/dfdl/testData"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xmlns:xml="http://www.w3.org/XML/1998/namespace"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:ex="http://example.com"
+ xmlns:gpf="http://www.ibm.com/dfdl/GeneralPurposeFormat"
+ xmlns:daf="urn:ogf:dfdl:2013:imp:daffodil.apache.org:2018:ext"
+ xmlns="http://www.w3.org/1999/xhtml"
+ xsi:schemaLocation="http://www.ibm.com/xmlns/dfdl/testData tdml.xsd"
+ defaultRoundTrip="false">
+
+ <!--
+ This example TDML file is for a self-contained bug report.
+
+ It shows the definition of an inline schema and parse test and unparse test that use that schema.
+ -->
+
+ <!--
+ A DFDL schema is defined inside the tdml:defineSchema element. The contents
+ are similar to a normal DFDL schema, allowing for imports, defining a
+ global format via dfdl:defineFormat and dfdl:format, and defining schema
+ xs:elements/groups/types/etc.
+ -->
+
+ <tdml:defineSchema name="bug01Schema" elementFormDefault="unqualified">
+
+ <xs:import
+ namespace="http://www.ibm.com/dfdl/GeneralPurposeFormat"
+ schemaLocation="IBMdefined/GeneralPurposeFormat.xsd" />
+
+ <dfdl:defineFormat name="myFormat">
+ <dfdl:format ref="gpf:GeneralPurposeFormat"
+ lengthKind="implicit"
+ representation="text"
+ encoding="ASCII"
+ initiator=""
+ terminator=""
+ separator="" />
+ </dfdl:defineFormat>
+
+ <dfdl:format ref="ex:myFormat" />
+
+ <xs:element name="myTestRoot" type="xs:dateTime"
+ dfdl:calendarPattern="MM.dd.yyyy 'at' HH:mm:ssZZZZZ"
+ dfdl:calendarPatternKind="explicit"
+ dfdl:lengthKind="delimited"
+ dfdl:terminator="%NL;" />
+
+ </tdml:defineSchema>
+
+ <!--
+ Define a parse test case, using the above schema and root element. Input
+ data is defined along with the expected infoset.
+ -->
+
+ <tdml:parserTestCase name="dateTimeTest" root="myTestRoot" model="bug01Schema"
+ description="A hypothetical bug illustration about parsing a date time.">
+
+ <tdml:document>
+ <tdml:documentPart type="text"
+ replaceDFDLEntities="true"><![CDATA[04.02.2013 at 14:00:56 GMT-05:00%LF;]]></tdml:documentPart>
+ </tdml:document>
+
+ <tdml:infoset>
+ <tdml:dfdlInfoset>
+ <ex:myTestRoot>2013-04-02T14:00:56.000000-05:00</ex:myTestRoot>
+ </tdml:dfdlInfoset>
+ </tdml:infoset>
+
+ </tdml:parserTestCase>
+
+ <!--
+ Define an unparse test case, using the above schema and root element. An
+ input infoset is defined along with the expected unparsed data.
+ -->
+
+ <tdml:unparserTestCase name="unparseDateTimeTest" root="myTestRoot" model="bug01Schema"
+ description="Another bug illustration, this time unparsing direction.">
+
+ <tdml:infoset>
+ <tdml:dfdlInfoset>
+ <ex:myTestRoot>2013-04-02T14:00:56.000000-05:00</ex:myTestRoot>
+ </tdml:dfdlInfoset>
+ </tdml:infoset>
+
+ <tdml:document>
+ <tdml:documentPart type="text"
+ replaceDFDLEntities="true"><![CDATA[04.02.2013 at 14:00:56-05:00%CR;%LF;]]></tdml:documentPart>
+ </tdml:document>
+
+ </tdml:unparserTestCase>
+
+</tdml:testSuite>
+```
+
+Suppose you save the above out as a file ``myDateTimeBug.tdml``. You can then run
+it using the ``test`` subcommand of the [Daffodil Command Line Interface](/cli):
+
+```
+$ daffodil test myDateTimeBug.tdml
+```
+
+### Specifying Test Data
+
+Test data can be specified in text, hexadecimal, individual bits, or in an
+external file by setting the ``type`` attribute in the ``tdml:documentPart``
+element. Multiple ``tdml:documentPart`` elements are combined to create the
+test data. The different documentPart types are illustrated below.
+
+```xml
+<tdml:document>
+ <!--
+ A document part with type="text" is text. It is often a good idea to use
+ CDATA to avoid whitespace changes made by some autoindenting IDE's.
+
+ So in the example below, the line ending after '250;' and after '967;' are
+ intentional parts of the data so as to illustrate that the whitespace is
+ preserved.
+
+ If you care exactly which kind of line ending is used, then you can use
+ DFDL character entities to insert a %CR; %LF; or both. In this example,
+ because the whitespace is expressed as whitespace, it depends on the
+ platform where you edit this file whether the line ending is a LF (Unix
+ convention), or a CRLF (MS Windows convention). By convention, it is
+ usually recommended to use Unix style line-endings in TDML files and use
+ character entities if explicit line endings are being tested.
+
+ If you want to use DFDL character entities, you must turn on the
+ replaceDFDLEntities="true" feature of the documentPart element.
+ -->
+
+ <tdml:documentPart type="text"><![CDATA[quantity:250;
+hardnessRating:967;
+]]></tdml:documentPart>
+
+ <!--
+ In 'text' both XML character entities, and DFDL's own character entities
+ are interpreted.
+
+ So here is a NUL terminated string that contains a date with some Japanese
+ Kanji characters. The Japanese characters are expressed using XML numeric
+ character entities. The NUL termination is expressed using a DFDL character
+ entity.
+
+ In this example one has no choice but to use a DFDL character entity. The
+ NUL character (which has character code zero), is not allowed in XML
+ documents, not even using an XML character entity. So you have to write
+ '%NUL;' or '%#x00;' to express it using DFDL character entities.
+ -->
+
+ <tdml:documentPart type="text"
+ replaceDFDLEntities="true"><![CDATA[1987年10月日 BCE%NUL;]]></tdml:documentPart>
+
+ <!--
+ Type 'byte' means use hexadecimal to specify the data. Freeform whitespace
+ is allowed and any character that is not a-zA-Z0-9 is ignored. So you can
+ use "." or "-" to separate groups of hex digits if you like.
+ -->
+
+ <tdml:documentPart type="byte">
+ 9Abf e4c3
+ A5-E9-FF-00
+ </tdml:documentPart>
+
+ <!--
+ Type 'bits' allows you to specify individual 0 and 1. Any character other
+ than 0 or 1 is ignored.
+
+ The number of bits does not have to be a multiple of 8. That is, whole
+ bytes are not required.
+ -->
+
+ <tdml:documentPart type="bits">
+ 1.110 0.011 1 First 5 bit fields.
+ </tdml:documentPart>
+
+ <!--
+ Type 'file' means the content is a file name where to get the data
+ -->
+
+ <tdml:documentPart type="file">/some/directory/testData.in.dat</tdml:documentPart>
+
+</tdml:document>
+```
+Note that in order for a test to be considered successful, it must consume all
+the data defined in the ``tdml:document`` element. Otherwise the test will fail
+with a message about "left over data".
+
+### Specifying the Infoset
+
+The infoset can be provided either as an inline XML infoset or as a path to an
+external file by setting the ``type`` attribute on the ``tdml:dfdlInfoset``
+element. If not provided, the type defaults to inline XML. For example:
+
+```xml
+<tdml:infoset>
+ <tdml:dfdlInfoset type="infoset">
+ <ex:myTestRoot>2013-04-02T14:00:56.000000-05:00</ex:myTestRoot>
+ </tdml:dfdlInfoset>
+</tdml:infoset>
+
+<tdml:infoset>
+ <tdml:dfdlInfoset type="file">/some/directory/testData.in.xml</tdml:dfdlInfoset>
+</tdml:infoset>
+```
+
+Note that the ``tdml:dfdlInfoset`` may need to contain characters that are not
+legal in XML documents. Daffodil remaps these characters into legal XML
+characters in the Unicode Private Use Areas (PUA). See [XML Illegal
+Characters](/infoset#xml-illegal-characters) for details.
+
+### Negative Tests: Expecting Errors/Warnings
+
+A poor or missing diagnostic message is a bug just as much as a broken feature.
+TDML allows for creating negative tests to expect errors and warnings. To do
+expect errors, replace the ``tdml:infoset`` element with a ``tdml:errors``
+element:
+
+```xml
+<tdml:errors>
+ <tdml:error>Schema Definition Error</tdml:error>
+ <tdml:error>testElementName</tdml:error>
+</tdml:errors>
+```
+
+Each ``tdml:error`` child element contains a sub-string which must be found
+somewhere in the set of diagnostic messages that come out of the test. The
+comparison is case-insensitive.
+
+The ``tdml:warnings`` and ``tdml:warning`` elements behave just like the error
+counterparts to define warnings that should be created during the test. Note
+that warnings are considered non-fatal and so can appear alongside
+``tdml:errors`` and ``tdml:infoset`` elements.
+
+```xml
+<tdml:warnings>
+ <tdml:warning>Schema Definition Warning</tdml:warning>
+ <tdml:warning>'http://www.ogf.org/dfdl/dfdl-1.0/' should be 'http://www.ogf.org/dfdl/'</tdml:warning>
+</tdml:warnings>
+```
--
To stop receiving notification emails like this one, please contact
slawrence@apache.org.