You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@daffodil.apache.org by gi...@apache.org on 2025/11/04 17:20:35 UTC
(daffodil-site) branch asf-site updated: Publishing from 17f2e9c58ab3fb775a9adb8c8252309147ba4c8e
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/daffodil-site.git
The following commit(s) were added to refs/heads/asf-site by this push:
new bed108a Publishing from 17f2e9c58ab3fb775a9adb8c8252309147ba4c8e
bed108a is described below
commit bed108a9fec46ac4143783b25982369b6e23b35f
Author: Apache Daffodil Site Autobuild <de...@daffodil.apache.org>
AuthorDate: Tue Nov 4 17:20:31 2025 +0000
Publishing from 17f2e9c58ab3fb775a9adb8c8252309147ba4c8e
---
content/binary-large-objects/index.html | 281 ++++++++++++++++++++++
content/dfdl-extensions/index.html | 407 ++++++++++++++++++++++++++++----
content/layers/index.html | 68 ++++--
content/unsupported/index.html | 2 +-
4 files changed, 690 insertions(+), 68 deletions(-)
diff --git a/content/binary-large-objects/index.html b/content/binary-large-objects/index.html
new file mode 100644
index 0000000..4783e41
--- /dev/null
+++ b/content/binary-large-objects/index.html
@@ -0,0 +1,281 @@
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <title>Apache Daffodil | Binary Large Objects Feature</title>
+ <meta name="description" content="Binary Large Objects Feature">
+ <meta name="author" content="">
+
+ <!-- Enable responsive viewport -->
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <!-- HTML5 shim, for IE6-8 support of HTML elements -->
+ <!--[if lt IE 9]>
+ <script src="http://html5shim.googlecode.com/svn/trunk/html5.js"></script>
+ <![endif]-->
+
+ <link href="/assets/themes/apache/img/apache-daffodil-icon.png" rel="icon" type="image/png">
+
+ <link href="/assets/themes/apache/bootstrap/css/bootstrap.css" rel="stylesheet">
+ <link href="/assets/themes/apache/css/style.css?body=1" rel="stylesheet" type="text/css">
+ <link href="/assets/themes/apache/css/syntax.css" rel="stylesheet" type="text/css" media="screen" />
+
+ </head>
+
+ <body>
+
+ <div class="navbar navbar-inverse" role="navigation">
+ <div class="container">
+ <div class="navbar-header"><a class="navbar-brand" href="/"><img src="/assets/themes/apache/img/apache-daffodil-logo.png" alt="Apache Daffodil"/></a></div>
+ <nav role="navigation">
+ <ul class="nav navbar-nav navbar-right">
+ <li><a href="/releases">Releases</a></li>
+ <li id="extensions">
+ <a href="#" data-toggle="dropdown" class="dropdown-toggle">Extensions<b class="caret"></b></a>
+ <ul class="dropdown-menu dropdown-left">
+ <li><a href="/vscode">VS Code</a></li>
+ <li><a href="/sbt">SBT</a></li>
+ </ul>
+ </li>
+ <li id="documentation">
+ <a href="#" data-toggle="dropdown" class="dropdown-toggle">Docs<b class="caret"></b></a>
+ <ul class="dropdown-menu dropdown-left">
+ <li><a href="/getting-started/">Getting Started</a></li>
+ <li><a href="/examples/">Examples</a></li>
+ <li><a href="/docs/latest/javadoc/">API</a></li>
+ <li><a href="/docs/dfdl/">DFDL Specification</a></li>
+ <li><a href="/unsupported/">Unsupported Features</a></li>
+ <li><a href="/faq/">Frequently Asked Questions</a></li>
+ <li><a href="/dfdl-extensions/">Daffodil DFDL Language Extensions</a></li>
+ </ul>
+ </li>
+ <li id="community">
+ <a href="#" data-toggle="dropdown" class="dropdown-toggle">Community<b class="caret"></b></a>
+ <ul class="dropdown-menu dropdown-left">
+ <li><a href="/community">Get Involved</a></li>
+ <li><a href="/people">People</a></li>
+ </ul>
+ </li>
+ <li id="development">
+ <a href="#" data-toggle="dropdown" class="dropdown-toggle">Development<b class="caret"></b></a>
+ <ul class="dropdown-menu dropdown-left">
+ <li><a class="external" href="https://cwiki.apache.org/confluence/display/DAFFODIL/">Wiki</a></li>
+ <li><a class="external" href="https://github.com/apache/?q=daffodil">GitHub</a></li>
+ <li><a class="external" href="https://issues.apache.org/jira/projects/DAFFODIL/">JIRA</a></li>
+ </ul>
+ </li>
+ <li id="apache">
+ <a href="#" data-toggle="dropdown" class="dropdown-toggle">Apache<b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a class="external" href="https://www.apache.org/">Foundation</a></li>
+ <li><a class="external" href="https://www.apache.org/licenses/">License</a></li>
+ <li><a class="external" href="https://www.apache.org/events/current-event">Events</a></li>
+ <li><a class="external" href="https://www.apache.org/security">Security</a></li>
+ <li><a class="external" href="https://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a class="external" href="https://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a class="external" href="https://privacy.apache.org/policies/privacy-policy-public.html">Privacy Policy</a></li>
+ </ul>
+ </li>
+ </ul>
+ </nav>
+ </div>
+ </div>
+
+
+
+
+<div class="title">
+ <div class="container"></div>
+</div>
+<div class="container">
+ <h1>Binary Large Objects Feature</h1>
+</div>
+
+
+
+
+ <div class="container">
+ <div class="row">
+ <div class="col-md-12">
+ <!--
+
+-->
+<h3 class="no_toc" id="table-of-contents">Table of Contents</h3>
+<!-- The {: .no_toc } excludes the above heading from the ToC -->
+
+<ol id="markdown-toc">
+ <li><a href="#introduction" id="markdown-toc-introduction">Introduction</a></li>
+ <li><a href="#type-xsanyuri-and-property-dfdlxobjectkind" id="markdown-toc-type-xsanyuri-and-property-dfdlxobjectkind">Type <code class="language-plaintext highlighter-rouge">xs:anyURI</code> and Property <code class="language-plaintext highlighter-rouge">dfdlx:objectKind</code></a></li>
+ <li><a href="#daffodil-blob-api" id="markdown-toc-daffodil-blob-api">Daffodil BLOB API</a></li>
+ <li><a href="#dfdl-expressions" id="markdown-toc-dfdl-expressions">DFDL Expressions</a></li>
+ <li><a href="#testing-dfdl-schemas-using-blobs-via-the-tdml-runner" id="markdown-toc-testing-dfdl-schemas-using-blobs-via-the-tdml-runner">Testing DFDL Schemas using BLOBs via the TDML Runner</a></li>
+ <li><a href="#command-line-interface" id="markdown-toc-command-line-interface">Command Line Interface</a></li>
+</ol>
+<!-- note the above line {:toc} cannot have whitespace at the start -->
+
+<!--
+This page is linked from https://s.apache.org/daffodil-blob-feature.
+ If this page content moves, please update that link from https://s.apache.org.
+-->
+
+<h2 id="introduction">Introduction</h2>
+
+<p>Daffodil has implemented a DFDL extension that allows data much larger than memory to be manipulated.</p>
+
+<p>A variety of data formats, such as for image and video files, consist of fields of what is effectively metadata, surrounding large blocks of data containing compressed image or video data.</p>
+
+<p>An important use case for DFDL is to expose this metadata for easy use, and to provide access to
+the large data via a streaming mechanism akin to opening a file, thereby avoiding
+large <code class="language-plaintext highlighter-rouge">xs:hexBinary</code> strings in the infoset.</p>
+
+<p>In RDBMS systems, BLOB (Binary Large Object) is the type used when the data row returned from an SQL query will not contain the actual value data, but rather a handle that can be used to open/read/write/close the BLOB.</p>
+
+<p>Daffodil has an analogous BLOB capability.
+This enables processing of images or video of arbitrary size without the need to ever hold all the data in memory.</p>
+
+<p>This also bypasses the limitation on object size.</p>
+
+<h2 id="type-xsanyuri-and-property-dfdlxobjectkind">Type <code class="language-plaintext highlighter-rouge">xs:anyURI</code> and Property <code class="language-plaintext highlighter-rouge">dfdlx:objectKind</code></h2>
+
+<p>DFDL is extended to allow simple types to have the <code class="language-plaintext highlighter-rouge">xs:anyURI</code> type.
+Elements with this type will be treated as BLOB objects.</p>
+
+<p>The <code class="language-plaintext highlighter-rouge">dfdlx:objectKind</code> property is added to define what type of object it is.
+The valid value for this property is only <code class="language-plaintext highlighter-rouge">"bytes"</code> specifying binary large objects.
+All other values reserved for future extensions of this feature.</p>
+
+<p>An example of this usage in a DFDL schema may look something like this:</p>
+
+<pre><code class="language-xsd"><xs:schema
+ xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/">
+
+ <xs:element name="data" type="xs:anyURI"
+ dfdlx:objectKind="bytes"
+ dfdl:lengthUnits="bytes"
+ dfdl:length="1024" />
+
+</xs:schema>
+</code></pre>
+
+<p>The resulting infoset (as XML) will look something like this:</p>
+
+<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><data></span>file:///path/to/blob/data<span class="nt"></data></span>
+</code></pre></div></div>
+
+<p>With the 1024 bytes of data being written to a file at location <code class="language-plaintext highlighter-rouge">/path/to/blob/data</code>.</p>
+
+<p>The BLOB URI must always use the <em>file scheme</em> and must be absolute.</p>
+
+<h2 id="daffodil-blob-api">Daffodil BLOB API</h2>
+
+<p>API calls are used to specify where Daffodil should write the BLOB files.</p>
+
+<p>Two functions are used on the Daffodil <code class="language-plaintext highlighter-rouge">InfosetOutputter</code>.</p>
+
+<p>The first API function allows a way to set the properties used when
+creating BLOB files, including the output directory, and prefix/suffix
+for the BLOB file.</p>
+
+<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cm">/**
+ * Set the attributes for how to create blob files.
+ *
+ * @param dir the Path the the directory to create files. If the directory
+ * does not exist, Daffodil will attempt to create it before
+ * writing a blob.
+ * @param prefix the prefix string to be used in generating a blob file name
+ * @param suffix the suffix string to be used in generating a blob file name
+ */</span>
+<span class="k">final</span> <span class="k">def</span> <span class="nf">setBlobAttributes</span><span class="o">(</span><span class="n">directory</span><span class="k">:</span> <span class="kt">Path</span><span class="o">,</span> <span class="n">prefix</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">suffix</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
+</code></pre></div></div>
+
+<p>The second API function allows a way for the API user to get a list of
+all BLOB files that were created during <code class="language-plaintext highlighter-rouge">parse()</code>.</p>
+
+<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cm">/**
+ * Get the list of blob paths that were output in the infoset.
+ *
+ * This is the same as what would be found by iterating over the infoset.
+ */</span>
+<span class="k">final</span> <span class="k">def</span> <span class="nf">getBlobFiles</span><span class="o">()</span><span class="k">:</span> <span class="kt">Seq</span><span class="o">[</span><span class="kt">Path</span><span class="o">]</span>
+</code></pre></div></div>
+
+<p>Note that no changes to the <code class="language-plaintext highlighter-rouge">unparse()</code> API are required, since the BLOB URI provides
+all the necessary information to retrieve files containing BLOB data.</p>
+
+<p>BLOB files are not automatically deleted.
+It is the responsibility of the API user to determine when files are no
+longer needed and remove them.</p>
+
+<h2 id="dfdl-expressions">DFDL Expressions</h2>
+
+<p>Any expression access to the <em>data</em> of a BLOB element will result in a
+Schema Definition Error during schema compilation.</p>
+
+<p>The <em>length</em> of a BLOB element is available since it is very common in
+data formats to include both a BLOB payload and the length of that
+payload. On unparse, we can calculate the length of the BLOB data so
+that the value can be output in a length field in the data. This is
+done using the regular <code class="language-plaintext highlighter-rouge">dfdl:contentLength()</code> and <code class="language-plaintext highlighter-rouge">dfdl:valueLength()</code>
+functions.</p>
+
+<h2 id="testing-dfdl-schemas-using-blobs-via-the-tdml-runner">Testing DFDL Schemas using BLOBs via the TDML Runner</h2>
+
+<p>The TDML language is extended to support the <code class="language-plaintext highlighter-rouge">xsi:type="xs:anyURI"</code> annotation on XML data elements.</p>
+
+<p>For example:</p>
+
+<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><tdml:dfdlInfoset></span>
+ <span class="nt"><data</span> <span class="na">xsi:type=</span><span class="s">"xs:anyURI"</span><span class="nt">></span>path/to/blob/data<span class="nt"></data></span>
+<span class="nt"></tdml:dfdlInfoset></span>
+</code></pre></div></div>
+
+<p>The path provided as the URI value can be, and usually will be, a relative path within the
+<code class="language-plaintext highlighter-rouge">src/test/resources</code> directory of the DFDL schema project.
+During Infoset comparisons the TDML Runner will compare the contents of this file
+with the BLOB file in the corresponding element (having type <code class="language-plaintext highlighter-rouge">xs:anyURI</code>) of the infoset.</p>
+
+<p>BLOB files created when running the tests are deleted when the test completes.</p>
+
+<h2 id="command-line-interface">Command Line Interface</h2>
+
+<p>The CLI supports ad-hoc testing of the use of BLOBs.
+BLOBs are written to the directory given by the JVM <em>System Property</em> <code class="language-plaintext highlighter-rouge">user.dir</code> into
+a subdirectory of it named <code class="language-plaintext highlighter-rouge">daffodil-blobs</code>.
+If it does not exist, Daffodil will attempt to create the <code class="language-plaintext highlighter-rouge">daffodil-blobs</code> directory.
+The CLI does not delete any BLOB files.</p>
+
+ </div>
+</div>
+
+
+ <footer>
+ <footer class="site-footer">
+ <div class="wrapper">
+ <div class="footer-col-wrapper" style="font-size: .85em;">
+ <hr>
+ <div>
+ <div style="text-align: center;">
+ Copyright © 2025 <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version
+ 2.0</a>.
+ <br>
+ Apache, Apache Daffodil, Daffodil, and the Apache Daffodil logo
+ are trademarks of The Apache Software Foundation.
+ </div>
+ </div>
+ </div>
+ </div>
+</footer>
+
+ </footer>
+ </div>
+
+ <script src="/assets/themes/apache/jquery/jquery-2.1.1.min.js"></script>
+
+ <script src="/assets/themes/apache/bootstrap/js/bootstrap.min.js"></script>
+
+
+ </body>
+</html>
+
diff --git a/content/dfdl-extensions/index.html b/content/dfdl-extensions/index.html
index 5b2bcb6..17b8125 100644
--- a/content/dfdl-extensions/index.html
+++ b/content/dfdl-extensions/index.html
@@ -2,7 +2,7 @@
<html lang="en">
<head>
<meta charset="utf-8">
- <title>Apache Daffodil | DFDL Extensions</title>
+ <title>Apache Daffodil | Daffodil Extensions to the DFDL Language</title>
<meta name="author" content="">
@@ -88,7 +88,7 @@
<div class="container"></div>
</div>
<div class="container">
- <h1>DFDL Extensions</h1>
+ <h1>Daffodil Extensions to the DFDL Language</h1>
</div>
@@ -101,35 +101,96 @@
-->
+<h3 class="no_toc" id="table-of-contents">Table of Contents</h3>
+<!-- The {: .no_toc } excludes the above heading from the ToC -->
+
+<ol id="markdown-toc">
+ <li><a href="#introduction" id="markdown-toc-introduction">Introduction</a></li>
+ <li><a href="#binary-large-objects-blob-feature" id="markdown-toc-binary-large-objects-blob-feature">Binary Large Objects (BLOB) Feature</a></li>
+ <li><a href="#expression-functions" id="markdown-toc-expression-functions">Expression Functions</a> <ol>
+ <li><a href="#dfdlxtracevalue-label" id="markdown-toc-dfdlxtracevalue-label"><code class="language-plaintext highlighter-rouge">dfdlx:trace(value, label)</code></a></li>
+ <li><a href="#dfdlxlookaheadoffset-bitsize" id="markdown-toc-dfdlxlookaheadoffset-bitsize"><code class="language-plaintext highlighter-rouge">dfdlx:lookAhead(offset, bitSize)</code></a> <ol>
+ <li><a href="#examples-of-dfdlxlookahead" id="markdown-toc-examples-of-dfdlxlookahead">Examples of <code class="language-plaintext highlighter-rouge">dfdlx:lookAhead</code></a></li>
+ </ol>
+ </li>
+ <li><a href="#bitwise-functions-bitand-bitor-bitxor-bitnot-leftshift-rightshift" id="markdown-toc-bitwise-functions-bitand-bitor-bitxor-bitnot-leftshift-rightshift">Bitwise Functions: <code class="language-plaintext highlighter-rouge">bitAnd</code>, <code class="language-plaintext highlighter-rouge">bitOr</code>, <code class="language-plaintext highlighter-rouge">bitXor</code>, <code class="language-plaintext highlighter-rouge">bitNot</code>, <code class="language-plaintext highlig [...]
+ <li><a href="#dfdlxbitandarg1-arg2" id="markdown-toc-dfdlxbitandarg1-arg2"><code class="language-plaintext highlighter-rouge">dfdlx:bitAnd(arg1, arg2)</code></a></li>
+ <li><a href="#dfdlxbitorarg1-arg2" id="markdown-toc-dfdlxbitorarg1-arg2"><code class="language-plaintext highlighter-rouge">dfdlx:bitOr(arg1, arg2)</code></a></li>
+ <li><a href="#dfdlxbitxorarg1-arg2" id="markdown-toc-dfdlxbitxorarg1-arg2"><code class="language-plaintext highlighter-rouge">dfdlx:bitXor(arg1, arg2)</code></a></li>
+ <li><a href="#dfdlxbitnotarg" id="markdown-toc-dfdlxbitnotarg"><code class="language-plaintext highlighter-rouge">dfdlx:bitNot(arg)</code></a></li>
+ <li><a href="#dfdlxleftshiftvalue-shiftcount" id="markdown-toc-dfdlxleftshiftvalue-shiftcount"><code class="language-plaintext highlighter-rouge">dfdlx:leftShift(value, shiftCount)</code></a></li>
+ <li><a href="#dfdlxrightshiftvalue-shiftcount" id="markdown-toc-dfdlxrightshiftvalue-shiftcount"><code class="language-plaintext highlighter-rouge">dfdlx:rightShift(value, shiftCount)</code></a></li>
+ </ol>
+ </li>
+ <li><a href="#dfdlxdoublefromrawlonglongarg-and-dfdlxdoubletorawlongdoublearg" id="markdown-toc-dfdlxdoublefromrawlonglongarg-and-dfdlxdoubletorawlongdoublearg"><code class="language-plaintext highlighter-rouge">dfdlx:doubleFromRawLong(longArg)</code> and <code class="language-plaintext highlighter-rouge">dfdlx:doubleToRawLong(doubleArg)</code></a></li>
+ </ol>
+ </li>
+ <li><a href="#properties" id="markdown-toc-properties">Properties</a> <ol>
+ <li><a href="#dfdlxalignmentkind" id="markdown-toc-dfdlxalignmentkind"><code class="language-plaintext highlighter-rouge">dfdlx:alignmentKind</code></a></li>
+ <li><a href="#dfdlxparseunparsepolicy" id="markdown-toc-dfdlxparseunparsepolicy"><code class="language-plaintext highlighter-rouge">dfdlx:parseUnparsePolicy</code></a></li>
+ <li><a href="#dfdlxlayer" id="markdown-toc-dfdlxlayer"><code class="language-plaintext highlighter-rouge">dfdlx:layer</code></a></li>
+ <li><a href="#dfdlxdirection" id="markdown-toc-dfdlxdirection"><code class="language-plaintext highlighter-rouge">dfdlx:direction</code></a></li>
+ <li><a href="#enumerations-dfdlxreptype-dfdlxrepvalues" id="markdown-toc-enumerations-dfdlxreptype-dfdlxrepvalues">Enumerations: <code class="language-plaintext highlighter-rouge">dfdlx:repType</code>, <code class="language-plaintext highlighter-rouge">dfdlx:repValues</code></a> <ol>
+ <li><a href="#dfdlxreptype" id="markdown-toc-dfdlxreptype"><code class="language-plaintext highlighter-rouge">dfdlx:repType</code></a></li>
+ <li><a href="#dfdlxrepvalues" id="markdown-toc-dfdlxrepvalues"><code class="language-plaintext highlighter-rouge">dfdlx:repValues</code></a></li>
+ <li><a href="#examples-of-enumerations-in-daffodil-dfdl" id="markdown-toc-examples-of-enumerations-in-daffodil-dfdl">Examples of Enumerations in Daffodil DFDL</a></li>
+ </ol>
+ </li>
+ </ol>
+ </li>
+ <li><a href="#extended-behaviors-for-dfdl-types" id="markdown-toc-extended-behaviors-for-dfdl-types">Extended Behaviors for DFDL Types</a> <ol>
+ <li><a href="#type-xshexbinary" id="markdown-toc-type-xshexbinary">Type <code class="language-plaintext highlighter-rouge">xs:hexBinary</code></a></li>
+ </ol>
+ </li>
+</ol>
+<!-- note the above line {:toc} cannot have whitespace at the start -->
+
+<h2 id="introduction">Introduction</h2>
+
<p>Daffodil provides extensions to the DFDL specification.
-These properties are in the namespace defined by the URI
+These functions and properties are in the namespace defined by the URI
<code class="language-plaintext highlighter-rouge">http://www.ogf.org/dfdl/dfdl-1.0/extensions</code> which is normally bound to the <code class="language-plaintext highlighter-rouge">dfdlx</code> prefix
like so:</p>
-<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><xs:schema</span> <span class="na">xmlns:xs=</span><span class="s">"http://www.w3.org/2001/XMLSchema"</span>
- <span class="na">xmlns:dfdl=</span><span class="s">"http://www.ogf.org/dfdl/dfdl-1.0/"</span>
- <span class="na">xmlns:dfdlx=</span><span class="s">"http://www.ogf.org/dfdl/dfdl-1.0/extensions"</span>
-<span class="nt">></span>
+<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><schema</span> <span class="na">xmlns=</span><span class="s">"http://www.w3.org/2001/XMLSchema"</span>
+ <span class="na">xmlns:dfdl=</span><span class="s">"http://www.ogf.org/dfdl/dfdl-1.0/"</span>
+ <span class="na">xmlns:dfdlx=</span><span class="s">"http://www.ogf.org/dfdl/dfdl-1.0/extensions"</span><span class="nt">></span>
</code></pre></div></div>
-<p>The following symbols defined in this namespace are described below.</p>
+<p>The DFDL language extensions described below have Long Term Support (LTS) in Daffodil
+going forward, and are proposed for inclusion in a future revision of the DFDL
+standard.
+DFDL schema authors can depend on the features and behaviors defined here without fear
+that these extensions will be withdrawn in the future.</p>
-<h2 id="expression-functions">Expression Functions</h2>
+<h2 id="binary-large-objects-blob-feature">Binary Large Objects (BLOB) Feature</h2>
-<h3 id="daferror"><code class="language-plaintext highlighter-rouge">daf:error()</code></h3>
+<p>Daffodil supports processing data that contains large opaque binary objects,
+also known as <em>BLOBs</em>.
+These enable processing of data types such as images, audio, or video where the
+data content is surrounded by important metadata.
+The DFDL Schema can expose the metadata fields for processing and carry
+along the opaque BLOB data in files.</p>
-<p>A function that can be used in DFDL expressions. This functions does not return a value or accept any arguments. When called, it causes a Parse Error or Unparse Error.</p>
+<p>There is
+<a href="/binary-large-objects">separate documentation for the Binary Large Object (BLOB) feature</a>.</p>
-<p><em>This function is deprecated as of Daffodil 2.0.0. Use the <code class="language-plaintext highlighter-rouge">fn:error(...)</code> function instead.</em></p>
+<h2 id="expression-functions">Expression Functions</h2>
-<h3 id="dfdlxtracevalue-label"><code class="language-plaintext highlighter-rouge">dfdlx:trace($value, $label)</code></h3>
+<h3 id="dfdlxtracevalue-label"><code class="language-plaintext highlighter-rouge">dfdlx:trace(value, label)</code></h3>
-<p>A function that can be used in DFDL expressions, similar to the <code class="language-plaintext highlighter-rouge">fn:trace()</code> function. This logs the string <code class="language-plaintext highlighter-rouge">$label</code> followed by <code class="language-plaintext highlighter-rouge">$value</code> converted to a string and returns <code class="language-plaintext highlighter-rouge">$value</code>. The second argument must be of type <code class="language-plaintext highlighter-rou [...]
+<p>A function that can be used to debug DFDL expressions, similar to
+the <a href="https://www.w3.org/TR/xpath-functions-31/#func-trace">XPath <code class="language-plaintext highlighter-rouge">fn:trace(value, label)</code></a>
+function.
+This creates a message from the string argument <code class="language-plaintext highlighter-rouge">label</code> followed by <code class="language-plaintext highlighter-rouge">value</code> converted to a
+string and logs the message.
+The function returns the <code class="language-plaintext highlighter-rouge">value</code>.
+The second <code class="language-plaintext highlighter-rouge">label</code> argument must be of type <code class="language-plaintext highlighter-rouge">xs:string</code>.</p>
<h3 id="dfdlxlookaheadoffset-bitsize"><code class="language-plaintext highlighter-rouge">dfdlx:lookAhead(offset, bitSize)</code></h3>
<p>Read <code class="language-plaintext highlighter-rouge">bitSize</code> bits, where the first bit is located at an <code class="language-plaintext highlighter-rouge">offset</code> (in bits)
-from the current location. The result is a <code class="language-plaintext highlighter-rouge">xs:nonNegativeInteger</code>. Restrictions:</p>
+ from the current location. The result is a <code class="language-plaintext highlighter-rouge">xs:nonNegativeInteger</code>. Restrictions:</p>
<ul>
<li>offset >=0</li>
@@ -148,39 +209,127 @@ the data being read will not be used.</li>
<h4 id="examples-of-dfdlxlookahead">Examples of <code class="language-plaintext highlighter-rouge">dfdlx:lookAhead</code></h4>
<p>The following two elements both populate element <code class="language-plaintext highlighter-rouge">a</code> with the value of the next 3 bits as an
-unsignedInt. They are not completely equivalent because the first will consume 3 bits of the
+unsignedInt.
+They are not completely equivalent because the first will consume 3 bits of the
input stream where the second will not advance the input stream.</p>
+
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><xs:element</span> <span class="na">name=</span><span class="s">"a"</span> <span class="na">type=</span><span class="s">"xs:unsignedInt"</span> <span class="na">dfdl:length=</span><span class="s">"3"</span> <span class="na">dfdl:lengthUnits=</span><span class="s">"bits"</span> <span class="nt">/></span>
<span class="nt"><xs:element</span> <span class="na">name=</span><span class="s">"a"</span> <span class="na">type=</span><span class="s">"xs:unsignedInt"</span> <span class="na">dfdl:inputValueCalc=</span><span class="s">"{ dfdlx:lookAhead(0,3) }"</span> <span class="nt">/></span>
</code></pre></div></div>
<p>The following example demonstrates using lookAhead to branch based on a field in the future.
In this case the choice of elements <code class="language-plaintext highlighter-rouge">a</code> vs. <code class="language-plaintext highlighter-rouge">b</code> depends on the value of the <code class="language-plaintext highlighter-rouge">tag</code> field which is
-found after fields <code class="language-plaintext highlighter-rouge">a</code> and <code class="language-plaintext highlighter-rouge">b</code>:
-```</p>
-<xs:choice dfdl:choiceDispatchKey="{ dfdlx:lookAhead(16,8) }">
-<xs:element name="a" type="xs:int" dfdl:length="16" dfdl:choiceBranchKey="1" />
-<xs:element name="b" type="xs:int" dfdl:length="16" dfdl:choiceBranchKey="2" />
-</xs:choice>
-<xs:element name="tag" type="xs:int" dfdl:length="8" />
-
-<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> ``` # Bitwise Functions
+found after fields <code class="language-plaintext highlighter-rouge">a</code> and <code class="language-plaintext highlighter-rouge">b</code>:</p>
+<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code><xs:choice dfdl:choiceDispatchKey="{ dfdlx:lookAhead(16,8) }">
+ <xs:element name="a" type="xs:int" dfdl:length="16" dfdl:choiceBranchKey="1"/>
+ <xs:element name="b" type="xs:int" dfdl:length="16" dfdl:choiceBranchKey="2"/>
+</xs:choice>
+<xs:element name="tag" type="xs:int" dfdl:length="8" />
</code></pre></div></div>
-<p>TBD, but the complete list (all <code class="language-plaintext highlighter-rouge">dfdlx</code>) is <code class="language-plaintext highlighter-rouge">BitAnd</code>, <code class="language-plaintext highlighter-rouge">BitNot</code>, <code class="language-plaintext highlighter-rouge">BitOr</code>, <code class="language-plaintext highlighter-rouge">BitXor</code>, <code class="language-plaintext highlighter-rouge">LeftShift</code>,
-<code class="language-plaintext highlighter-rouge">RightShift</code></p>
+<h3 id="bitwise-functions-bitand-bitor-bitxor-bitnot-leftshift-rightshift">Bitwise Functions: <code class="language-plaintext highlighter-rouge">bitAnd</code>, <code class="language-plaintext highlighter-rouge">bitOr</code>, <code class="language-plaintext highlighter-rouge">bitXor</code>, <code class="language-plaintext highlighter-rouge">bitNot</code>, <code class="language-plaintext highlighter-rouge">leftShift</code>, <code class="language-plaintext highlighter-rouge">rightShift</code></h3>
+
+<p>These functions are defined on types <code class="language-plaintext highlighter-rouge">long</code>, <code class="language-plaintext highlighter-rouge">int</code>, <code class="language-plaintext highlighter-rouge">short</code>, <code class="language-plaintext highlighter-rouge">byte</code>, <code class="language-plaintext highlighter-rouge">unsignedLong</code>,
+<code class="language-plaintext highlighter-rouge">unsignedInt</code>, <code class="language-plaintext highlighter-rouge">unsignedShort</code>, and <code class="language-plaintext highlighter-rouge">unsignedByte</code></p>
+
+<h4 id="dfdlxbitandarg1-arg2"><code class="language-plaintext highlighter-rouge">dfdlx:bitAnd(arg1, arg2)</code></h4>
+
+<p>This computes the bitwise AND of two integers.</p>
+
+<ul>
+ <li>Both arguments must be signed, or both must be unsigned.</li>
+ <li>If the two arguments are not the same type the smaller one is converted into the type of the
+larger one.</li>
+ <li>If the smaller argument is signed, this conversion does sign-extension.</li>
+ <li>The result type is the that of the largest argument.</li>
+</ul>
+
+<h4 id="dfdlxbitorarg1-arg2"><code class="language-plaintext highlighter-rouge">dfdlx:bitOr(arg1, arg2)</code></h4>
+
+<p>This computes the bitwise OR of two integers.</p>
+
+<ul>
+ <li>Both arguments must be signed, or both must be unsigned.</li>
+ <li>If the two arguments are not the same type the smaller one is converted into the type of the
+larger one.</li>
+ <li>If the smaller argument is signed, this conversion does sign-extension.</li>
+ <li>The result type is the that of the largest argument.</li>
+</ul>
+
+<h4 id="dfdlxbitxorarg1-arg2"><code class="language-plaintext highlighter-rouge">dfdlx:bitXor(arg1, arg2)</code></h4>
+
+<p>This computes the bitwise Exclusive OR of two integers.</p>
+
+<ul>
+ <li>Both arguments must be signed, or both must be unsigned.</li>
+ <li>If the two arguments are not the same type the smaller one is converted into the type of the
+larger one.</li>
+ <li>If the smaller argument is signed, this conversion does sign-extension.</li>
+ <li>The result type is the that of the largest argument.</li>
+</ul>
+
+<h4 id="dfdlxbitnotarg"><code class="language-plaintext highlighter-rouge">dfdlx:bitNot(arg)</code></h4>
+
+<p>This computes the bitwise NOT of an integer. Every bit is inverted. The result type is the same
+as the argument type.</p>
+
+<h4 id="dfdlxleftshiftvalue-shiftcount"><code class="language-plaintext highlighter-rouge">dfdlx:leftShift(value, shiftCount)</code></h4>
-<h3 id="dfdlxdoublefromrawlong-and-dfdlxdoubletorawlong"><code class="language-plaintext highlighter-rouge">dfdlx:doubleFromRawLong</code> and <code class="language-plaintext highlighter-rouge">dfdlx:doubleToRawLong</code></h3>
+<p>This is the <em>logical</em> shift left, meaning that bits are shifted from less-significant positions
+to more-significant positions.</p>
-<p>Converting binary floating point numbers to/from base 10 text can result in lost information.
-The base 10 representation, converted back to binary representation, may not be bit-for-bit
-identical. These functions can be used to carry 8-byte double precision IEEE floating point
-numbers as type <code class="language-plaintext highlighter-rouge">xs:long</code> so that no information is lost. The DFDL schema can still obtain
-and operate on the floating point value by converting these <code class="language-plaintext highlighter-rouge">xs:long</code> values into type
-<code class="language-plaintext highlighter-rouge">xs:double</code>, and back if necessary for unparsing a new value.</p>
+<ul>
+ <li>The left-most bits shifted out are discarded.</li>
+ <li>Zeros are shifted in for the right-most bits.</li>
+ <li>The result type is the same as the <code class="language-plaintext highlighter-rouge">value</code> argument type.</li>
+ <li>It is a processing error if the <code class="language-plaintext highlighter-rouge">shiftCount</code> argument is < 0.</li>
+ <li>It is a processing error if the <code class="language-plaintext highlighter-rouge">shiftCount</code> argument is greater than the number of
+bits in the type of the value argument.</li>
+</ul>
+
+<h4 id="dfdlxrightshiftvalue-shiftcount"><code class="language-plaintext highlighter-rouge">dfdlx:rightShift(value, shiftCount)</code></h4>
+
+<p>This is the <em>arithmetic</em> shift right, meaning bits move from most-significant to
+less-significant positions.
+If <em>logical</em> (zero-filling) shift right is needed, you must use unsigned types.</p>
+
+<ul>
+ <li>The <code class="language-plaintext highlighter-rouge">value</code> argument is shifted by the <code class="language-plaintext highlighter-rouge">shiftCount</code>.</li>
+ <li>The right-most bits shifted out are discarded.</li>
+ <li>If the <code class="language-plaintext highlighter-rouge">value</code> is signed, then the sign bit is shifted in for the left-most bits.</li>
+ <li>If the <code class="language-plaintext highlighter-rouge">value</code> is unsigned, then zeros are shifted in for the left-most bits.</li>
+ <li>The result type is the same as the <code class="language-plaintext highlighter-rouge">value</code> argument type.</li>
+ <li>It is a processing error if the <code class="language-plaintext highlighter-rouge">shiftCount</code> argument is < 0.</li>
+ <li>It is a processing error if the <code class="language-plaintext highlighter-rouge">shiftCount</code> argument is greater than the number of
+bits in the type of the value argument.</li>
+</ul>
+
+<h3 id="dfdlxdoublefromrawlonglongarg-and-dfdlxdoubletorawlongdoublearg"><code class="language-plaintext highlighter-rouge">dfdlx:doubleFromRawLong(longArg)</code> and <code class="language-plaintext highlighter-rouge">dfdlx:doubleToRawLong(doubleArg)</code></h3>
+
+<p>IEEE binary float and double values that are not NaN will parse to base 10 text and unparse back
+to the same exact IEEE binary bits.
+However, the same cannot be said for NaN (not a number) values, of which there are many bit
+patterns.
+To preserve float and double NaN values bit for bit you can use these functions to compute
+<code class="language-plaintext highlighter-rouge">xs:long</code> values that enable the DFDL Infoset to preserve the bits of a float or double value
+even if it is a NaN.</p>
<h2 id="properties">Properties</h2>
+<h3 id="dfdlxalignmentkind"><code class="language-plaintext highlighter-rouge">dfdlx:alignmentKind</code></h3>
+
+<p>Valid values for this property are <code class="language-plaintext highlighter-rouge">manual</code> or <code class="language-plaintext highlighter-rouge">automatic</code> with <code class="language-plaintext highlighter-rouge">automatic</code> being the default
+behavior.
+When specified, the <code class="language-plaintext highlighter-rouge">manual</code> value turns off all automatic alignment based on the
+<code class="language-plaintext highlighter-rouge">dfdl:alignment</code> and <code class="language-plaintext highlighter-rouge">dfdl:alignmentUnits</code> properties.
+The schema author must use <code class="language-plaintext highlighter-rouge">dfdl:leadingSkip</code>, <code class="language-plaintext highlighter-rouge">dfdl:trailingSkip</code>, or just ensure all the
+elements/terms are aligned based on their length.</p>
+
+<p>This property is sometimes needed to facilitate creation of schemas where interactions occur
+between computed lengths (that is, stored length fields) and
+alignment regions that are automatically being inserted.
+It can be easier to do all alignment manually than to debug these interactions.</p>
+
<h3 id="dfdlxparseunparsepolicy"><code class="language-plaintext highlighter-rouge">dfdlx:parseUnparsePolicy</code></h3>
<p>A property applied to simple and complex elements, which specifies whether the element supports only parsing, only unparsing, or both parsing and unparse. Valid values for this property are <code class="language-plaintext highlighter-rouge">parse</code>, <code class="language-plaintext highlighter-rouge">unparse</code>, or <code class="language-plaintext highlighter-rouge">both</code>. This allows one to leave off properties that are required for only parse or only unparse, such as <c [...]
@@ -191,24 +340,200 @@ and operate on the floating point value by converting these <code class="languag
<h3 id="dfdlxlayer"><code class="language-plaintext highlighter-rouge">dfdlx:layer</code></h3>
-<p><a href="/layers">Layers</a> provide algorithmic capabilities for decoding/encoding data or computing
-checksums. Some are built-in to Daffodil. New layers can be created in Java/Scala and
-plugged-in to Daffodil dynamically.</p>
+<p><em>Layers</em> provide algorithmic capabilities for decoding/encoding data or computing
+ checksums. Some are built-in to Daffodil. New layers can be created in Java/Scala and
+ plugged-in to Daffodil dynamically.
+There is <a href="/layers">separate Layer documentation</a>.</p>
<h3 id="dfdlxdirection"><code class="language-plaintext highlighter-rouge">dfdlx:direction</code></h3>
-<p>TBD</p>
+<p>This property can appear only on DFDL <code class="language-plaintext highlighter-rouge">defineVariable</code> statement annotations.
+This property has possible values <code class="language-plaintext highlighter-rouge">both</code> (the default), <code class="language-plaintext highlighter-rouge">parseOnly</code>, or <code class="language-plaintext highlighter-rouge">unparseOnly</code>.
+It declares
+whether the variable is to be available for only parsing, only unparsing, or both.
+Since this is a newly introduced extension property and existing schemas won't contain a definition
+for it, it has a default value of <code class="language-plaintext highlighter-rouge">both</code>.</p>
+
+<p>This property can conflict with the <code class="language-plaintext highlighter-rouge">dfdlx:parseUnparsePolicy</code> property which takes the same
+values (<code class="language-plaintext highlighter-rouge">both</code>, <code class="language-plaintext highlighter-rouge">parseOnly</code>, and <code class="language-plaintext highlighter-rouge">unparseOnly</code>).
+If <code class="language-plaintext highlighter-rouge">dfdlx:parseUnparsePolicy='parseOnly'</code> then it is a Schema Definition Error if
+variables in the DFDL schema have <code class="language-plaintext highlighter-rouge">dfdlx:direction='unparseOnly'</code>.
+Similarly if <code class="language-plaintext highlighter-rouge">dfdlx:parseUnparsePolicy='unparseOnly'</code> then it is a Schema Definition Error if
+variables in the DFDL schema have <code class="language-plaintext highlighter-rouge">dfdlx:direction='parseOnly'</code>.</p>
+
+<p>It is a Schema Definition Error if a variable defined with direction <code class="language-plaintext highlighter-rouge">parseOnly</code> is accessed
+from an expression used by the unparser.
+Symmetrically, it is a Schema Definition Error if a variable defined with direction
+<code class="language-plaintext highlighter-rouge">unparseOnly</code> is accessed from an expression used by the parser.
+This error is detected at DFDL schema compilation time, not runtime.</p>
+
+<p>These properties take expressions for their values and are generally evaluated at both parse and
+unparse time.
+Hence, unless the whole schema is constrained by <code class="language-plaintext highlighter-rouge">dfdlx:parseUnparsePolicy</code>, any expressions for
+these properties<sup id="fnref:moreProps"><a href="#fn:moreProps" class="footnote" rel="footnote" role="doc-noteref">1</a></sup> cannot<br />
+cannot reference DFDL variables with <code class="language-plaintext highlighter-rouge">dfdlx:direction</code> of <code class="language-plaintext highlighter-rouge">parseOnly</code> or <code class="language-plaintext highlighter-rouge">unparseOnly</code>.</p>
+
+<ul>
+ <li><code class="language-plaintext highlighter-rouge">byteOrder</code></li>
+ <li><code class="language-plaintext highlighter-rouge">encoding</code></li>
+ <li><code class="language-plaintext highlighter-rouge">initiator</code></li>
+ <li><code class="language-plaintext highlighter-rouge">terminator</code></li>
+ <li><code class="language-plaintext highlighter-rouge">separator</code></li>
+ <li><code class="language-plaintext highlighter-rouge">escapeCharacter</code></li>
+ <li><code class="language-plaintext highlighter-rouge">escapeEscapeCharacter</code></li>
+ <li><code class="language-plaintext highlighter-rouge">length</code></li>
+ <li><code class="language-plaintext highlighter-rouge">occursCount</code></li>
+ <li><code class="language-plaintext highlighter-rouge">textStandardDecimalSeparator</code></li>
+ <li><code class="language-plaintext highlighter-rouge">textStandardGroupingSeparator</code></li>
+ <li><code class="language-plaintext highlighter-rouge">textStandardExponentRep</code></li>
+ <li><code class="language-plaintext highlighter-rouge">binaryFloatRep</code></li>
+ <li><code class="language-plaintext highlighter-rouge">textBooleanTrueRep</code></li>
+ <li><code class="language-plaintext highlighter-rouge">textbooleanFalseRep</code></li>
+ <li><code class="language-plaintext highlighter-rouge">calendarLanguage</code></li>
+ <li><code class="language-plaintext highlighter-rouge">dfdl:setVariable</code>, a <code class="language-plaintext highlighter-rouge">dfdl:newVariableInstance</code> default value expression, or a
+<code class="language-plaintext highlighter-rouge">dfdl:defineVariable</code> default value expression when
+that variable being set/defaulted is itself referenced from a another expression and the variable
+being set/defaulted has <code class="language-plaintext highlighter-rouge">dfdlx:direction</code> of <code class="language-plaintext highlighter-rouge">both</code> (the default)</li>
+</ul>
-<h3 id="dfdlxreptype-dfdlxrepvalues-and-dfdlxrepvalueranges"><code class="language-plaintext highlighter-rouge">dfdlx:repType</code>, <code class="language-plaintext highlighter-rouge">dfdlx:repValues</code>, and <code class="language-plaintext highlighter-rouge">dfdlx:repValueRanges</code></h3>
+<!-- footnotes must be all one big long line -->
-<p>TBD</p>
+<p>Parser-specific expressions include</p>
+
+<ul>
+ <li><code class="language-plaintext highlighter-rouge">dfdl:inputValueCalc</code></li>
+ <li><code class="language-plaintext highlighter-rouge">dfdl:length</code> (when dfdl:lengthKind='explicit')</li>
+ <li><code class="language-plaintext highlighter-rouge">dfdl:occursCount</code> (when `dfdl:occursCountKind='expression')</li>
+ <li><code class="language-plaintext highlighter-rouge">dfdl:choiceDispatchKey</code></li>
+ <li>the <code class="language-plaintext highlighter-rouge">message</code> and <code class="language-plaintext highlighter-rouge">test</code> attributes of the <code class="language-plaintext highlighter-rouge">dfdl:assert</code> and <code class="language-plaintext highlighter-rouge">dfdl:discriminator</code> statement annotations</li>
+ <li><code class="language-plaintext highlighter-rouge">dfdl:setVariable</code>, a <code class="language-plaintext highlighter-rouge">dfdl:newVariableInstance</code> default value expression, or a
+<code class="language-plaintext highlighter-rouge">dfdl:defineVariable</code> default value expression when
+that variable being set/defaulted is itself referenced from a another expression being
+accessed at parser creation time, and the variable being set/defaulted has <code class="language-plaintext highlighter-rouge">dfdlx:direction</code>
+of <code class="language-plaintext highlighter-rouge">parseOnly</code></li>
+</ul>
+
+<p>Unparser-specific expressions include:</p>
+
+<ul>
+ <li><code class="language-plaintext highlighter-rouge">dfdl:outputValueCalc</code></li>
+ <li><code class="language-plaintext highlighter-rouge">dfdl:length</code> (when `dfdl:lengthKind='explicit')</li>
+ <li><code class="language-plaintext highlighter-rouge">dfdl:outputNewLine</code></li>
+ <li><code class="language-plaintext highlighter-rouge">dfdl:setVariable</code>, a <code class="language-plaintext highlighter-rouge">dfdl:newVariableInstance</code> default value expression, or a
+<code class="language-plaintext highlighter-rouge">dfdl:defineVariable</code> default value expression when
+that variable being set/defaulted is itself referenced from a another expression being
+accessed at unparser creation time, and the variable being set/defaulted has <code class="language-plaintext highlighter-rouge">dfdlx:direction</code>
+of <code class="language-plaintext highlighter-rouge">unparseOnly</code></li>
+</ul>
-<h2 id="extended-behaviors">Extended Behaviors</h2>
+<h3 id="enumerations-dfdlxreptype-dfdlxrepvalues">Enumerations: <code class="language-plaintext highlighter-rouge">dfdlx:repType</code>, <code class="language-plaintext highlighter-rouge">dfdlx:repValues</code></h3>
+
+<p>These properties work together to allow DFDL schemas to define <em>enumerations</em>;
+that is, symbolic representations for integer constants.
+When parsing, Daffodil will convert these integers into the corresponding string values.
+When unparsing, Daffodil will convert strings into the corresponding integers.</p>
+
+<p>An element of type (or derived from) <code class="language-plaintext highlighter-rouge">xs:string</code> can be defined using XSD <code class="language-plaintext highlighter-rouge">enumeration</code> facets
+which constrain the valid values of this string.
+These enumeration values are effectively symbolic constants.
+The <code class="language-plaintext highlighter-rouge">dfdlx:repType</code> and <code class="language-plaintext highlighter-rouge">dfdlx:repValues</code> properties are then used to define the correspondence of
+the symbolic strings to the corresponding integer values.</p>
+
+<h4 id="dfdlxreptype"><code class="language-plaintext highlighter-rouge">dfdlx:repType</code></h4>
+
+<p>The value of this property is an XSD QName of a simple type definition that must be derived
+from <code class="language-plaintext highlighter-rouge">xs:int</code>, or <code class="language-plaintext highlighter-rouge">xs:unsignedInt</code>.
+A simple type definition for a string can be annotated with <code class="language-plaintext highlighter-rouge">dfdlx:repType</code>
+in order to declare that the representation of the string is not as text characters but is a
+numeric integer value.
+The type referenced from <code class="language-plaintext highlighter-rouge">dfdlx:repType</code> is usually a fixed length binary integer, but can be any
+DFDL type derived from <code class="language-plaintext highlighter-rouge">xs:int</code> or <code class="language-plaintext highlighter-rouge">xs:unsignedInt</code>, with any DFDL representation properties.</p>
+
+<p>The mapping between the representation integer and the symbolic constants is specified using the
+<code class="language-plaintext highlighter-rouge">dfdlx:repValues</code> property.</p>
+
+<h4 id="dfdlxrepvalues"><code class="language-plaintext highlighter-rouge">dfdlx:repValues</code></h4>
+
+<p>The value of this property is one or more integer values within
+the numeric range defined for the type referenced by <code class="language-plaintext highlighter-rouge">dfdlx:repType</code>. When more than one value
+is specified, they are in a whitespace separated list.</p>
+
+<p>This property is placed on the <code class="language-plaintext highlighter-rouge">xs:enumeration</code> facets of a symbolic string constant having a
+<code class="language-plaintext highlighter-rouge">dfdlx:repType</code>.
+At parse time, if the value of the <code class="language-plaintext highlighter-rouge">dfdlx:repType</code> integer is found within the <code class="language-plaintext highlighter-rouge">dfdlx:repValues</code>
+list, then the infoset value for the symbolic string gets the corresponing enumeration facet value.
+It is a parse error if the <code class="language-plaintext highlighter-rouge">dfdlx:repType</code> integer is not found in any of the <code class="language-plaintext highlighter-rouge">dfdlx:repValues</code>
+lists of the <code class="language-plaintext highlighter-rouge">xs:enumeration</code> facets.
+At unparse time, the symbolic constant is mapped to the first integer in the <code class="language-plaintext highlighter-rouge">dfdlx:repValues</code> list.
+It is an unparse error if the symbolic string value is not found among the <code class="language-plaintext highlighter-rouge">xs:enumeration</code>
+facet values of the symbolic string type.</p>
+
+<h4 id="examples-of-enumerations-in-daffodil-dfdl">Examples of Enumerations in Daffodil DFDL</h4>
+
+<p>A simple example of a basic enum is:</p>
+
+<pre><code class="language-xsd"> <simpleType name="rep3Bit" dfdl:lengthUnits="bits" dfdl:length="3" dfdl:lengthKind="explicit">
+ <restriction base="xs:unsignedInt"/>
+ </simpleType>
+
+ <simpleType name="precedenceEnum" dfdlx:repType="pre:rep3Bit">
+ <restriction base="xs:string">
+ <enumeration value="Reserved_0" dfdlx:repValues="0"/>
+ <enumeration value="Reserved_1" dfdlx:repValues="1"/>
+ <enumeration value="Emergency" dfdlx:repValues="2"/>
+ <enumeration value="Reserved_3" dfdlx:repValues="3"/>
+ <enumeration value="Flash" dfdlx:repValues="4"/>
+ <enumeration value="Immediate" dfdlx:repValues="5"/>
+ <enumeration value="Priority" dfdlx:repValues="6"/>
+ <enumeration value="Routine" dfdlx:repValues="7"/>
+ </restriction>
+ </simpleType>
+</code></pre>
+
+<p>Above we see the <code class="language-plaintext highlighter-rouge">dfdlx:repType</code> is <code class="language-plaintext highlighter-rouge">rep3Bit</code> which is a 3 bit <code class="language-plaintext highlighter-rouge">xs:unsignedInt</code>. This can
+represent the values 0 to 7 which one can see are the <code class="language-plaintext highlighter-rouge">dfdlx:repValues</code> of the <code class="language-plaintext highlighter-rouge">xs:enumeration</code>
+facets for this enumeration string type which is named <code class="language-plaintext highlighter-rouge">precedenceEnum</code>.</p>
+
+<p>In the above you can also see that the symbolic strings are in one-to-one correspondence with
+every possible value of the 3-bit representation integer.
+This one-to-one correspondence assures that data that is first parsed and then unparsed will
+recreate the exact numeric bits used.</p>
+
+<p>However, in data security applications the following may be preferred:</p>
+<pre><code class="language-xsd"> <simpleType name="precedenceEnum" dfdlx:repType="pre:rep3Bit">
+ <restriction base="xs:string">
+ <enumeration value="Reserved" dfdlx:repValues="0 1 3"/>
+ <enumeration value="Emergency" dfdlx:repValues="2"/>
+ <enumeration value="Flash" dfdlx:repValues="4"/>
+ <enumeration value="Immediate" dfdlx:repValues="5"/>
+ <enumeration value="Priority" dfdlx:repValues="6"/>
+ <enumeration value="Routine" dfdlx:repValues="7"/>
+ </restriction>
+ </simpleType>
+</code></pre>
+
+<p>In the above we see that three numeric values, 0, 1, and 3 are the <code class="language-plaintext highlighter-rouge">dfdlx:repValues</code> mapped to
+the symbolic string <code class="language-plaintext highlighter-rouge">Reserved</code>.
+This technique has the advantage of blocking covert signals being transmitted by use of the
+different reserved values since when unparsed, the constant string <code class="language-plaintext highlighter-rouge">Reserved</code> will always be
+<em>canonicalized</em> to integer 0.
+Putting data into canonical form when unparsing generally improves data security.</p>
+
+<h2 id="extended-behaviors-for-dfdl-types">Extended Behaviors for DFDL Types</h2>
<h3 id="type-xshexbinary">Type <code class="language-plaintext highlighter-rouge">xs:hexBinary</code></h3>
<p>Daffodil allows <code class="language-plaintext highlighter-rouge">dfdlx:lengthUnits='bits'</code> for this simple type.</p>
+<hr />
+<div class="footnotes" role="doc-endnotes">
+ <ol>
+ <li id="fn:moreProps">
+ <p>New properties added as part of errata corrections to the DFDL v1.0 standard which take expressions for their values will need to be added to this list or those for parser-specific or unparser-specific properties. <a href="#fnref:moreProps" class="reversefootnote" role="doc-backlink">↩</a></p>
+ </li>
+ </ol>
+</div>
+
</div>
</div>
diff --git a/content/layers/index.html b/content/layers/index.html
index 87511d6..41258fb 100644
--- a/content/layers/index.html
+++ b/content/layers/index.html
@@ -100,16 +100,18 @@
<!--
-->
-
<h3 class="no_toc" id="table-of-contents">Table of Contents</h3>
+<!-- The {: .no_toc } excludes the above heading from the ToC -->
<ol id="markdown-toc">
- <li><a href="#introduction" id="markdown-toc-introduction">Introduction</a></li>
- <li><a href="#built-in-layers" id="markdown-toc-built-in-layers">Built-in Layers</a></li>
- <li><a href="#custom-plug-in-layers" id="markdown-toc-custom-plug-in-layers">Custom Plug-In Layers</a></li>
- <li><a href="#layer-kinds-transforming-layers-and-checksum-layers" id="markdown-toc-layer-kinds-transforming-layers-and-checksum-layers">Layer Kinds: Transforming Layers and Checksum Layers</a> <ol>
- <li><a href="#transforming-layers" id="markdown-toc-transforming-layers">Transforming Layers</a></li>
- <li><a href="#checksum-layers" id="markdown-toc-checksum-layers">Checksum Layers</a></li>
+ <li><a href="#introduction" id="markdown-toc-introduction">Introduction</a> <ol>
+ <li><a href="#built-in-layers" id="markdown-toc-built-in-layers">Built-in Layers</a></li>
+ <li><a href="#custom-plug-in-layers" id="markdown-toc-custom-plug-in-layers">Custom Plug-In Layers</a></li>
+ <li><a href="#layer-kinds-transforming-layers-and-checksum-layers" id="markdown-toc-layer-kinds-transforming-layers-and-checksum-layers">Layer Kinds: Transforming Layers and Checksum Layers</a> <ol>
+ <li><a href="#transforming-layers" id="markdown-toc-transforming-layers">Transforming Layers</a></li>
+ <li><a href="#checksum-layers" id="markdown-toc-checksum-layers">Checksum Layers</a></li>
+ </ol>
+ </li>
</ol>
</li>
<li><a href="#using-layers" id="markdown-toc-using-layers">Using Layers</a> <ol>
@@ -121,11 +123,8 @@
</li>
</ol>
</li>
- <li><a href="#examples" id="markdown-toc-examples">Examples</a> <ol>
- <li><a href="#line-folding" id="markdown-toc-line-folding">Line Folding</a></li>
- <li><a href="#base64-gzip-and-boundarymark-layers-used-together" id="markdown-toc-base64-gzip-and-boundarymark-layers-used-together">Base64, GZip, and BoundaryMark Layers used Together</a></li>
- </ol>
- </li>
+ <li><a href="#example-line-folding" id="markdown-toc-example-line-folding">Example: Line Folding</a></li>
+ <li><a href="#example-base64-gzip-and-boundarymark-layers-used-together" id="markdown-toc-example-base64-gzip-and-boundarymark-layers-used-together">Example: Base64, GZip, and BoundaryMark Layers used Together</a></li>
<li><a href="#using-custom-plug-in-layers" id="markdown-toc-using-custom-plug-in-layers">Using Custom Plug-In Layers</a></li>
<li><a href="#daffodil-built-in-layer-documentation" id="markdown-toc-daffodil-built-in-layer-documentation">Daffodil Built-In Layer Documentation</a> <ol>
<li><a href="#base64-mime-layer" id="markdown-toc-base64-mime-layer">Base64 MIME Layer</a></li>
@@ -142,6 +141,7 @@
</ol>
</li>
</ol>
+<!-- note the above line {:toc} cannot have whitespace at the start -->
<h2 id="introduction">Introduction</h2>
@@ -170,7 +170,7 @@ There is no limit to this depth.</p>
<p>In the section on <a href="#UsingLayers">Using Layers</a> below we will look at an example that uses
multiple layers together.</p>
-<h2 id="built-in-layers">Built-in Layers</h2>
+<h3 id="built-in-layers">Built-in Layers</h3>
<p>Daffodil includes several built-in layers:</p>
<ul>
@@ -194,20 +194,20 @@ These are:</p>
<a href="#daffodil-built-in-layer-documentation">documented separately below</a> with examples of their
usage.</p>
-<h2 id="custom-plug-in-layers">Custom Plug-In Layers</h2>
+<h3 id="custom-plug-in-layers">Custom Plug-In Layers</h3>
<p>Additional layers can be written in Java or Scala and deployed as <em>plug-ins</em> for Daffodil.
These are generally packaged as DFDL <em>layer schemas</em>, a kind of <em>component schema</em>,
that provide the layer packaged for import by other DFDL <em>assembly</em> schemas that use the
layer in the data format they describe.</p>
-<h2 id="layer-kinds-transforming-layers-and-checksum-layers">Layer Kinds: Transforming Layers and Checksum Layers</h2>
+<h3 id="layer-kinds-transforming-layers-and-checksum-layers">Layer Kinds: Transforming Layers and Checksum Layers</h3>
<p>There are two different kinds of layers, though they share many characteristics. They are
<em>transforming</em> layers, and <em>checksum</em> layers. Both run small algorithms over part (or all) of
the data stream. The difference is the purpose of the algorithm and its output.</p>
-<h3 id="transforming-layers">Transforming Layers</h3>
+<h4 id="transforming-layers">Transforming Layers</h4>
<p>These layers decode data (when parsing), and encode data (when unparsing).
The simplest example of a transforming layer is the <code class="language-plaintext highlighter-rouge">base64_MIME</code> layer which
@@ -222,7 +222,7 @@ They can also assign computed result values to DFDL variables, though this is un
<a href="/docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html"><code class="language-plaintext highlighter-rouge">Layer</code></a> class
which is introduced in a later section.</p>
-<h3 id="checksum-layers">Checksum Layers</h3>
+<h4 id="checksum-layers">Checksum Layers</h4>
<p>Checksum layers are a simplified kind of layer which do not decode or encode data, they simply
pass through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic
@@ -245,6 +245,8 @@ DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header chec
html"><code class="language-plaintext highlighter-rouge">ChecksumLayer</code></a>
class, which is introduced in a later section.</p>
+<hr />
+
<h2 id="using-layers">Using Layers</h2>
<p>To use a layer you must know</p>
@@ -320,9 +322,7 @@ use the DFDL <code class="language-plaintext highlighter-rouge">layerLength</cod
<p>Layers may specify restrictions on the minimum and maximum allowed values of these lengths,
and passing an out-of-range value for the variable is a processing error.</p>
-<h2 id="examples">Examples</h2>
-
-<h3 id="line-folding">Line Folding</h3>
+<h2 id="example-line-folding">Example: Line Folding</h2>
<p>Consider the line folding layer, specifically the <code class="language-plaintext highlighter-rouge">lineFolded_IMF</code> layer,
which is built-in to Daffodil.</p>
@@ -338,7 +338,7 @@ a limited line length.</p>
</code></pre></div></div>
<p>This data has been <em>line folded</em> at roughly 72 characters by inserting a CRLF
before an existing space in the data.
-Each line ends with a CRLF (\r\n) and the second through fourth lines begin
+Each line ends with a CRLF (<code class="language-plaintext highlighter-rouge">\r\n</code>) and the second through fourth lines begin
with a space as a way of indicating that they are extension lines.
This data is supposed to be reassembled to form a long single-line string by removing
all CRLF pairs.</p>
@@ -379,7 +379,7 @@ Other examples will show how the layer length can be limited to a sub-region of
<p>More detailed documentation for the <a href="#line-folded-layers">Line Folded Layers</a> is below.</p>
-<h3 id="base64-gzip-and-boundarymark-layers-used-together">Base64, GZip, and BoundaryMark Layers used Together</h3>
+<h2 id="example-base64-gzip-and-boundarymark-layers-used-together">Example: Base64, GZip, and BoundaryMark Layers used Together</h2>
<p>In this example, the data consists of a preliminary string, a section of CSV-like data, and a
final string element.
@@ -539,6 +539,8 @@ This group definition is the last thing in the schema:</p>
</code></pre></div></div>
<p>The above schema works both to parse, but also to unparse this data.</p>
+<hr />
+
<h2 id="using-custom-plug-in-layers">Using Custom Plug-In Layers</h2>
<p>A custom plug-in layer is used in the same manner as the built-in Daffodil layers with just a few
@@ -571,6 +573,9 @@ base class.</p>
<p>Further details on how to define custom plug-in layers is in the Javadoc for the
<a href="/docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/package-summary.html">Layer API</a></p>
+<hr />
+<hr />
+
<h2 id="daffodil-built-in-layer-documentation">Daffodil Built-In Layer Documentation</h2>
<p>Each of the layers built-in to the Daffodil implementation are documented in a section below
@@ -586,6 +591,8 @@ which gives the name, namespace, variables, and some usage notes.</p>
<li><a href="#line-folded-layers">lineFolded_iCalendar</a></li>
</ul>
+<hr />
+
<h3 id="base64-mime-layer">Base64 MIME Layer</h3>
<ul>
@@ -603,11 +610,13 @@ which gives the name, namespace, variables, and some usage notes.</p>
<p>This is specified by <a href="https://www.ietf.org/rfc/rfc2045.txt">RFC 2045</a>.
The encoded output must be represented in lines of no more than 76 characters
-each and uses a carriage return '\r' followed immediately by a linefeed '\n' as the line separator.
+each and uses a carriage return <code class="language-plaintext highlighter-rouge">\r</code> followed immediately by a linefeed <code class="language-plaintext highlighter-rouge">\n</code> as the line separator.
No line separator is added to the end of the encoded output.
All line separators or other characters not found in the base64 alphabet table are ignored in
decoding operation.</p>
+<hr />
+
<h3 id="boundarymark-layer">BoundaryMark Layer</h3>
<ul>
@@ -657,6 +666,8 @@ of any child element enclosed within the layer, or even the lengths of other lay
within the scope of this boundary mark layer are not considered and do not disrupt the search
for the boundary mark string.</p>
+<hr />
+
<h3 id="byte-swapping-layers">Byte-Swapping Layers</h3>
<ul>
@@ -698,6 +709,8 @@ order 2 1 4 3 6 5 8 7 10 9.</p>
<p>If <code class="language-plaintext highlighter-rouge">requireLengthInWholeWords</code> is bound to "yes", then if the length is not a multiple of the
word size a processing error occurs.</p>
+<hr />
+
<h3 id="fixedlength-layer">FixedLength Layer</h3>
<ul>
@@ -721,6 +734,8 @@ The entire fixed length region of the data will be pulled into a byte buffer in
</li>
</ul>
+<hr />
+
<h3 id="gzip-layer">GZIP Layer</h3>
<ul>
@@ -745,6 +760,8 @@ depending on the Java version used.
To avoid inconsistent behavior of test failures that expect a certain byte value this layer
always writes a consistent header (header byte 9 of 255) regardless of the Java version.</p>
+<hr />
+
<h3 id="line-folded-layers">Line Folded Layers</h3>
<ul>
@@ -762,18 +779,17 @@ always writes a consistent header (header byte 9 of 255) regardless of the Java
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <xs:import namespace="urn:org.apache.daffodil.layers.lineFolded"
schemaLocation="/org/apache/daffodil/layers/xsd/lineFoldedLayer.dfdl.xsd"/>
</code></pre></div> </div>
+ <h4 id="general-usage">General Usage</h4>
</li>
</ul>
-<h4 id="general-usage">General Usage</h4>
-
<p>There is a limitation on the compatibility of line folding of data
with adjacent parts of the format which also use line-endings.
For example, line folding can interact badly with surrounding elements of <code class="language-plaintext highlighter-rouge">dfdl:lengthKind
'pattern'</code> if the pattern is, for example <code class="language-plaintext highlighter-rouge">".*?\\r\\n(?!(?:\\t|\\ ))"</code> which is anything up to
and including a CRLF not followed by a space or tab.
The problem is that line folding
-converts isolated \n or \r into \r\n, and if this just happens to be followed by a
+converts isolated <code class="language-plaintext highlighter-rouge">\n</code> or <code class="language-plaintext highlighter-rouge">\r</code> into <code class="language-plaintext highlighter-rouge">\r\n</code>, and if this just happens to be followed by a
non space/tab character this will have inserted an end-of-data in the middle of the
data.</p>
diff --git a/content/unsupported/index.html b/content/unsupported/index.html
index 677a54e..b681e7e 100644
--- a/content/unsupported/index.html
+++ b/content/unsupported/index.html
@@ -133,7 +133,7 @@ but may return in a future version of the DFDL spec.)</li>
<ul>
<li>fixed [<a href="https://issues.apache.org/jira/browse/DAFFODIL-117">DAFFODIL-117</a>]</li>
- <li>default [<a href="https://issues.apache.org/jira/browse/DAFFODIL-115">DAFFODIL-115</a>] [<a href="https://issues.apache.org/jira/browse/DAFFODIL-1277">DAFFODIL-1277</a>]</li>
+ <li>default [<a href="https://issues.apache.org/jira/browse/DAFFODIL-115">DAFFODIL-115</a>]</li>
</ul>
<h2 id="properties-and-property-enumerations">Properties and Property Enumerations</h2>