You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@daffodil.apache.org by "mbeckerle (via GitHub)" <gi...@apache.org> on 2024/05/21 23:19:25 UTC
[PR] WIP: Checkpoint - start on layers documentation in layers.md [daffodil-site]
mbeckerle opened a new pull request, #132:
URL: https://github.com/apache/daffodil-site/pull/132
DAFFODIL-2569
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Documentation on layers feature in layers.md and updated javadoc for layers [daffodil-site]
Posted by "mbeckerle (via GitHub)" <gi...@apache.org>.
mbeckerle merged PR #132:
URL: https://github.com/apache/daffodil-site/pull/132
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] WIP: Checkpoint - start on layers documentation in layers.md [daffodil-site]
Posted by "mbeckerle (via GitHub)" <gi...@apache.org>.
mbeckerle commented on code in PR #132:
URL: https://github.com/apache/daffodil-site/pull/132#discussion_r1612207243
##########
site/layers.md:
##########
@@ -0,0 +1,382 @@
+---
+layout: page
+title: Layers - Pluggable Extensions to Enable Algorithmic Transformations in DFDL
+description: Pluggable Extensions to Enable Algorithmic Transformations in DFDL
+group: nav-right
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- base64_MIME
+- fourbyteswap
+- twobyteswap
+- gzip
+- lineFolded_IMF
+- lineFolded_iCalendar
+
+Daffodil also includes two utility layers that are used in combination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- boundaryMark
+- fixedLength
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+## Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html) class
+which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass-through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic Redundancy Check (CRC) over the data stream.
+The value of the checksum (or hash or CRC) is assigned to a DFDL variable as the result of the
+layer. This makes the value available for use by the DFDL schema that uses the checksum layer.
+When parsing, the value of this DFDL variable can then be compared to a checksum field in the
+data, and either an invalid data element or an parse-error can be created if the checksum in the
+data stream does not match the computed value.
+When unparsing, the value of this DFDL variable can be written to an element using the
+`dfdl:outputValueCalc` property.
+
+An example of a layer plug-in is in the <a href="https://github.com/DFDLSchemas/ethernetIP">EthernetIP</a>
+DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header checksum algorithm.
+
+Custom checksum layers are created by deriving an implementation class from the Daffodil API's
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.
+html)
+class, which is introduced in a later section.
+
+## Layer Composition
+
+Layers can be piled on top of each other. Once a layer is in place, another layer can be used in
+conjunction with it.
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses both the
+built-in _gzip_ layer, and the _base64_MIME_ layer together.
+
+# Using Layers
+
+To use a layer you must know
+- the layer's namespace URI
+- the layer's name
+- the names of any layer parameter variables
+- the names of any layer result variables
+
+## Example: Line Folding
+
+As a first example, let's look at the line folding layer, specifically the `lineFolded_IMF`layer,
+which is built-in to Daffodil.
+
+Line folding is a way of encoding textual data formats so that no line of text is longer than
+a limited line length.
+
+Consider this data :
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem
+ ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+This data has been _line folded_ at roughly 72 characters by inserting a CRLF
+before an existing space in the data.
+Each line ends with a CRLF (\r\n) and the second through fourth lines begin
+with a space as a way of indicating that they are extension lines.
+This data is supposed to be reassembled to form a long single-line string by removing
+all CRLF pairs.
+
+The result should be this single longer string which does not contain any line endings:
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+To achieve this we would use the `lineFolded_IMF` layer.
+This layer has a specific namespace which our DFDL schema will define a prefix `lf` for like this:
+```xml
+xmlns:lf="urn:org.apache.daffodil.layers.lineFolded"
+```
+Our DFDL schema will import the layer schema for the line-folded layer with this import:
+```xml
+<import namespace="urn:org.apache.daffodil.layers.lineFolded"
+ schemaLocation="/org/apache/daffodil/layers/xsd/lineFoldedLayer.dfdl.xsd"/>
+```
+Then the layer is incorporated into our DFDL schema like this:
+```xml
+ <sequence dfdlx:layer="lf:lineFolded_IMF">
+ ... elements to be parsed from unfolded layer data go here ...
+ </sequence>
+```
+You can see that use of a layer is described using the `dfdx:layer` property, and the specific layer
+is identified by a QName using the previously defined namespace prefix.
+The scope of the layer is the duration of the sequence it appears on.
+The `dfdlx:layer` property can only be used on an XSD `sequence`.
+
+The `lineFolded_IMF` layer does not define any DFDL variables in its namespace as it has no
+parameters and produces no results.
+In fact, as of Daffodil 3.8.0, the `import` statement above is optional for the
+line-folded layers as the DFDL schema `lineFoldedLayer.dfdl.xsd` does not contain any definitions.
+In the future however, parameters may be added, so for uniformity all layers define a DFDL
+schema to be imported as part of using the layer.
+
+More detailed documentation for the [Line Folded Layers](#LineFoldedLayers) is below.
+
+## Example: Base64, GZip, and BoundaryMark
+
+[EXAMPLE TBD] - should explain elements of specified length, or use of boundaryMark or
+fixed length layers.
+
+# Defining Custom Plug-In Layers
+
+Plug-in custom layers are dynamically loaded using the Java Service Provider Interface (SPI).
+Hence, they must be compiled into Jar files with specific META-INF metadata, and must appear on
+the Java CLASSPATH so that they can be found and loaded.
+
+The layer API is defined via Java clases and interfaces to enable writing of custom layers in
+either Java or Scala.
+(One of the built-in layers (Gzip) is written in Java now, by way of proving
+that one can write a Layer in Java.)
+
+A layer implementation class must obey various naming
+conventions that allow Java/Scala reflection to associate the names of Java/Scala code methods and
+method arguments with DFDL variables in that layer's namespace, that share those names.
+
+Transformer layer classes are derived from the
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html)
+base class.
+
+A name and target namespace, and hence a QName is required to identify a layer for use from
+a DFDL schema.
+This namespace is OWNED by the layer.
+All DFDL variables
+defined in that namespace are either used to pass parameters to the
+layer code, or receive results (such as a checksum) back from the
+layer code.
+This is enforced.
+A layer that has no DFDL variables does
+not have to define a DFDL schema that defines the layer's target
+namespace, but any layer that uses DFDL variables *must* define a
+schema with the layer's namespace as its target namespace, with the
+variables declared in it (using `dfdl:defineVariable`).
+
+There is also an abstract base class for defining checksum layers
+called,
+
+ org.apache.daffodil.runtime1.layers.api.ChecksumLayer
+
+### Layer Limiting
+
+A layer can process an entire input file/stream.
+Such a layer is said to be _unlimited_.
+More commonly, a layer has a limited region within the data stream that it is supposed to process.
+The restriction of the layer so that it only processes the expected part of the data is called
+_layer limiting_.
+
+There are two ways to do layer limiting, that differ in an
+important way.
+
+#### Layer Limiting using Elements of Specified Length
+
+If the `sequence` for a layer is the model group of the
+complex type of an element, and that element has `dfdl:lengthKind` of `'explicit'`, `'implicit'`,
+or `'pattern'`, then the length of the element limits the length of the layer within it.
+
+In this case the length of the layer is limited when parsing, but it is NOT limited when
+unparsing.
+
+#### Layer Limiting using the `fixedLength` Utility Layer
+
+If you use the layer `fixedLength`, that dictates the parse and
+unparse length to be the value of the `layerLength` DFDL variable.
+
+Similarly, if you use a `checksum` layer built with the Daffodil API `ChecksumLayer` base class,
+the length is controlled for both parsing and unparsing as it works
+similarly to the way the `fixedLength` layer works.
+
+#### Layer Limiting using the `boundaryMark` Utility Layer
+
+The `boundaryMark` layer uses a variable named `boundaryMark` which provides a parameter to the
+layer which is a delimiter of the layer data when parsing, and
+which is inserted after the (otherwise unbounded) layer data when unparsing.
+
+### Layer Variables:
Review Comment:
This section to the javadoc. Down to the section on Built-in layers.
##########
site/layers.md:
##########
@@ -0,0 +1,382 @@
+---
+layout: page
+title: Layers - Pluggable Extensions to Enable Algorithmic Transformations in DFDL
+description: Pluggable Extensions to Enable Algorithmic Transformations in DFDL
+group: nav-right
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- base64_MIME
+- fourbyteswap
+- twobyteswap
+- gzip
+- lineFolded_IMF
+- lineFolded_iCalendar
+
+Daffodil also includes two utility layers that are used in combination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- boundaryMark
+- fixedLength
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+## Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html) class
+which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass-through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic Redundancy Check (CRC) over the data stream.
+The value of the checksum (or hash or CRC) is assigned to a DFDL variable as the result of the
+layer. This makes the value available for use by the DFDL schema that uses the checksum layer.
+When parsing, the value of this DFDL variable can then be compared to a checksum field in the
+data, and either an invalid data element or an parse-error can be created if the checksum in the
+data stream does not match the computed value.
+When unparsing, the value of this DFDL variable can be written to an element using the
+`dfdl:outputValueCalc` property.
+
+An example of a layer plug-in is in the <a href="https://github.com/DFDLSchemas/ethernetIP">EthernetIP</a>
+DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header checksum algorithm.
+
+Custom checksum layers are created by deriving an implementation class from the Daffodil API's
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.
+html)
+class, which is introduced in a later section.
+
+## Layer Composition
+
+Layers can be piled on top of each other. Once a layer is in place, another layer can be used in
+conjunction with it.
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses both the
+built-in _gzip_ layer, and the _base64_MIME_ layer together.
+
+# Using Layers
+
+To use a layer you must know
+- the layer's namespace URI
+- the layer's name
+- the names of any layer parameter variables
+- the names of any layer result variables
+
+## Example: Line Folding
+
+As a first example, let's look at the line folding layer, specifically the `lineFolded_IMF`layer,
+which is built-in to Daffodil.
+
+Line folding is a way of encoding textual data formats so that no line of text is longer than
+a limited line length.
+
+Consider this data :
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem
+ ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+This data has been _line folded_ at roughly 72 characters by inserting a CRLF
+before an existing space in the data.
+Each line ends with a CRLF (\r\n) and the second through fourth lines begin
+with a space as a way of indicating that they are extension lines.
+This data is supposed to be reassembled to form a long single-line string by removing
+all CRLF pairs.
+
+The result should be this single longer string which does not contain any line endings:
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+To achieve this we would use the `lineFolded_IMF` layer.
+This layer has a specific namespace which our DFDL schema will define a prefix `lf` for like this:
+```xml
+xmlns:lf="urn:org.apache.daffodil.layers.lineFolded"
+```
+Our DFDL schema will import the layer schema for the line-folded layer with this import:
+```xml
+<import namespace="urn:org.apache.daffodil.layers.lineFolded"
+ schemaLocation="/org/apache/daffodil/layers/xsd/lineFoldedLayer.dfdl.xsd"/>
+```
+Then the layer is incorporated into our DFDL schema like this:
+```xml
+ <sequence dfdlx:layer="lf:lineFolded_IMF">
+ ... elements to be parsed from unfolded layer data go here ...
+ </sequence>
+```
+You can see that use of a layer is described using the `dfdx:layer` property, and the specific layer
+is identified by a QName using the previously defined namespace prefix.
+The scope of the layer is the duration of the sequence it appears on.
+The `dfdlx:layer` property can only be used on an XSD `sequence`.
+
+The `lineFolded_IMF` layer does not define any DFDL variables in its namespace as it has no
+parameters and produces no results.
+In fact, as of Daffodil 3.8.0, the `import` statement above is optional for the
+line-folded layers as the DFDL schema `lineFoldedLayer.dfdl.xsd` does not contain any definitions.
+In the future however, parameters may be added, so for uniformity all layers define a DFDL
+schema to be imported as part of using the layer.
+
+More detailed documentation for the [Line Folded Layers](#LineFoldedLayers) is below.
+
+## Example: Base64, GZip, and BoundaryMark
+
+[EXAMPLE TBD] - should explain elements of specified length, or use of boundaryMark or
+fixed length layers.
+
+# Defining Custom Plug-In Layers
+
+Plug-in custom layers are dynamically loaded using the Java Service Provider Interface (SPI).
+Hence, they must be compiled into Jar files with specific META-INF metadata, and must appear on
+the Java CLASSPATH so that they can be found and loaded.
+
+The layer API is defined via Java clases and interfaces to enable writing of custom layers in
+either Java or Scala.
+(One of the built-in layers (Gzip) is written in Java now, by way of proving
+that one can write a Layer in Java.)
+
+A layer implementation class must obey various naming
Review Comment:
Move most details about writing custom layers to the javadoc. This section should be about USING custom layers, and how it differs from using Daffodil built-in layers.
For guidance on coding custom layers, just link to the javadoc. All of that should be in the javadoc for the layers.api package, and for the Layer and ChecksumLayer classes.
##########
site/layers.md:
##########
@@ -0,0 +1,382 @@
+---
+layout: page
+title: Layers - Pluggable Extensions to Enable Algorithmic Transformations in DFDL
+description: Pluggable Extensions to Enable Algorithmic Transformations in DFDL
+group: nav-right
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- base64_MIME
+- fourbyteswap
+- twobyteswap
+- gzip
+- lineFolded_IMF
+- lineFolded_iCalendar
+
+Daffodil also includes two utility layers that are used in combination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- boundaryMark
+- fixedLength
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+## Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html) class
+which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass-through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic Redundancy Check (CRC) over the data stream.
+The value of the checksum (or hash or CRC) is assigned to a DFDL variable as the result of the
+layer. This makes the value available for use by the DFDL schema that uses the checksum layer.
+When parsing, the value of this DFDL variable can then be compared to a checksum field in the
+data, and either an invalid data element or an parse-error can be created if the checksum in the
+data stream does not match the computed value.
+When unparsing, the value of this DFDL variable can be written to an element using the
+`dfdl:outputValueCalc` property.
+
+An example of a layer plug-in is in the <a href="https://github.com/DFDLSchemas/ethernetIP">EthernetIP</a>
+DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header checksum algorithm.
+
+Custom checksum layers are created by deriving an implementation class from the Daffodil API's
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.
+html)
+class, which is introduced in a later section.
+
+## Layer Composition
+
+Layers can be piled on top of each other. Once a layer is in place, another layer can be used in
+conjunction with it.
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses both the
+built-in _gzip_ layer, and the _base64_MIME_ layer together.
+
+# Using Layers
+
+To use a layer you must know
+- the layer's namespace URI
+- the layer's name
+- the names of any layer parameter variables
+- the names of any layer result variables
+
+## Example: Line Folding
+
+As a first example, let's look at the line folding layer, specifically the `lineFolded_IMF`layer,
+which is built-in to Daffodil.
+
+Line folding is a way of encoding textual data formats so that no line of text is longer than
+a limited line length.
+
+Consider this data :
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem
+ ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+This data has been _line folded_ at roughly 72 characters by inserting a CRLF
+before an existing space in the data.
+Each line ends with a CRLF (\r\n) and the second through fourth lines begin
+with a space as a way of indicating that they are extension lines.
+This data is supposed to be reassembled to form a long single-line string by removing
+all CRLF pairs.
+
+The result should be this single longer string which does not contain any line endings:
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+To achieve this we would use the `lineFolded_IMF` layer.
+This layer has a specific namespace which our DFDL schema will define a prefix `lf` for like this:
+```xml
+xmlns:lf="urn:org.apache.daffodil.layers.lineFolded"
+```
+Our DFDL schema will import the layer schema for the line-folded layer with this import:
+```xml
+<import namespace="urn:org.apache.daffodil.layers.lineFolded"
+ schemaLocation="/org/apache/daffodil/layers/xsd/lineFoldedLayer.dfdl.xsd"/>
+```
+Then the layer is incorporated into our DFDL schema like this:
+```xml
+ <sequence dfdlx:layer="lf:lineFolded_IMF">
+ ... elements to be parsed from unfolded layer data go here ...
+ </sequence>
+```
+You can see that use of a layer is described using the `dfdx:layer` property, and the specific layer
+is identified by a QName using the previously defined namespace prefix.
+The scope of the layer is the duration of the sequence it appears on.
+The `dfdlx:layer` property can only be used on an XSD `sequence`.
+
+The `lineFolded_IMF` layer does not define any DFDL variables in its namespace as it has no
+parameters and produces no results.
+In fact, as of Daffodil 3.8.0, the `import` statement above is optional for the
+line-folded layers as the DFDL schema `lineFoldedLayer.dfdl.xsd` does not contain any definitions.
+In the future however, parameters may be added, so for uniformity all layers define a DFDL
+schema to be imported as part of using the layer.
+
+More detailed documentation for the [Line Folded Layers](#LineFoldedLayers) is below.
+
+## Example: Base64, GZip, and BoundaryMark
+
+[EXAMPLE TBD] - should explain elements of specified length, or use of boundaryMark or
+fixed length layers.
+
+# Defining Custom Plug-In Layers
+
+Plug-in custom layers are dynamically loaded using the Java Service Provider Interface (SPI).
+Hence, they must be compiled into Jar files with specific META-INF metadata, and must appear on
+the Java CLASSPATH so that they can be found and loaded.
+
+The layer API is defined via Java clases and interfaces to enable writing of custom layers in
+either Java or Scala.
+(One of the built-in layers (Gzip) is written in Java now, by way of proving
+that one can write a Layer in Java.)
+
+A layer implementation class must obey various naming
+conventions that allow Java/Scala reflection to associate the names of Java/Scala code methods and
+method arguments with DFDL variables in that layer's namespace, that share those names.
+
+Transformer layer classes are derived from the
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html)
+base class.
+
+A name and target namespace, and hence a QName is required to identify a layer for use from
+a DFDL schema.
+This namespace is OWNED by the layer.
+All DFDL variables
+defined in that namespace are either used to pass parameters to the
+layer code, or receive results (such as a checksum) back from the
+layer code.
+This is enforced.
+A layer that has no DFDL variables does
+not have to define a DFDL schema that defines the layer's target
+namespace, but any layer that uses DFDL variables *must* define a
+schema with the layer's namespace as its target namespace, with the
+variables declared in it (using `dfdl:defineVariable`).
+
+There is also an abstract base class for defining checksum layers
+called,
+
+ org.apache.daffodil.runtime1.layers.api.ChecksumLayer
+
+### Layer Limiting
+
+A layer can process an entire input file/stream.
+Such a layer is said to be _unlimited_.
+More commonly, a layer has a limited region within the data stream that it is supposed to process.
+The restriction of the layer so that it only processes the expected part of the data is called
+_layer limiting_.
+
+There are two ways to do layer limiting, that differ in an
+important way.
+
+#### Layer Limiting using Elements of Specified Length
+
+If the `sequence` for a layer is the model group of the
+complex type of an element, and that element has `dfdl:lengthKind` of `'explicit'`, `'implicit'`,
+or `'pattern'`, then the length of the element limits the length of the layer within it.
+
+In this case the length of the layer is limited when parsing, but it is NOT limited when
+unparsing.
+
+#### Layer Limiting using the `fixedLength` Utility Layer
+
+If you use the layer `fixedLength`, that dictates the parse and
+unparse length to be the value of the `layerLength` DFDL variable.
+
+Similarly, if you use a `checksum` layer built with the Daffodil API `ChecksumLayer` base class,
+the length is controlled for both parsing and unparsing as it works
+similarly to the way the `fixedLength` layer works.
+
+#### Layer Limiting using the `boundaryMark` Utility Layer
+
+The `boundaryMark` layer uses a variable named `boundaryMark` which provides a parameter to the
+layer which is a delimiter of the layer data when parsing, and
+which is inserted after the (otherwise unbounded) layer data when unparsing.
+
+### Layer Variables:
+
+Each variable is either a parameter passed to a special setter named
+`setLayerVariableParameters`, or is a return variable which is populated
+from a special getter.
+The special setter args and getters taken
+together must use up all the DFDL variables defined in the layer's
+namespace. (See the javadoc for the Layer base class.)
+
+A special getter example for a checksum layer:
+
+ int getLayerVariableResult_checksum() { .... }
+
+This corresponds to, and returns the value that populates, the DFDL
+variable named `checksum` in the layer's namespace.
+That DFDL variable must have a DFDL type corresponding to the `int` type (`xs:unsignedShort` or
+`xs:int`).
+
+Layers could have multiple result variables, but we have no actual
+examples of more than one value being returned.
+(Returning multiple variables is tested.
+We just have no real use cases for it at this time.)
+
+### Layer Exception Handling:
+
+The method
+
+ setProcessingErrorException(...)
+
+allows the layer to specify that if the layer throws specific
+exceptions or runtime exceptions that they are converted into processing
+errors.
+This eliminates most need for layer code to contain try-catches.
+
+#### About Testing:
+
+There are tests for every way that
+a user can goof up the definition of a layer class, and there are
+tests for `processingError`, `runtimeSchemaDefinitionError`, and throwing
+an `Exception` from every place a user-defined Layer could cause these.
+Parse errors cause backtracking in all sensible cases.
+Nothing aborts.
+
+## Compatibility with Daffodil 3.7.0 and prior
+
+This new Layer API is entirely incompatible with schemas or layer code
+from Daffodil 3.7.0 and all prior versions of Daffodil.
+
+The layer
+feature was just an experimental feature in Daffodil, so we reserved
+the right to change it, and for Daffodil 3.8.0 it has changed radically.
+
+However, it is our hope that the Layer API introduced in
+Daffodil 3.8.0 will prove to be stable and supportable long term.
+
+
+
+# Daffodil Built-In Layer Documentation
+
+## Line Folded Layers
+
+- Namespace URI: urn:org.apache.daffodil.layers.lineFolded
+- No DFDL Variables are read or set
+- There are two layer names for two variations:
+- - [lineFolded_IMF](#layer-name-linefolded_imf) - conforms to IETF RFC 2822 Internet Message Format (IMF)
+- - [lineFolded_iCalendar](#layer-name-linefolded_icalendar) - conforms to IETF RFC 5545 iCalendar
+
+### General Usage
+
+These layers would normally be used in a context of either an enclosing element
+which bounds its length (`dfdl:lengthKind 'explicit'`, `'prefixed'`, or `'pattern'`)
+or an enclosing utility layer that bounds its length (ex: `boundaryMark`).
+
+There is a limitation on the compatibility of line folding of data
+with adjacent parts of the format which also use line-endings.
+For example, line folding can interact badly with surrounding elements of `dfdl:lengthKind
+'pattern'` if the pattern is, for example `".*?\\r\\n(?!(?:\\t|\\ ))"` which is anything up to
+and including a CRLF not followed by a space or tab.
+The problem is that line folding
+converts isolated \n or \r into \r\n, and if this just happens to be followed by a
+non space/tab character this will have inserted an end-of-data in the middle of the
+data.
+
+### Layer Name: lineFolded_IMF
+
+For IMF, unfolding simply removes CRLFs if they are followed by a space or tab.
+
+When unparsing, the folding is more complex, as CRLFs can only be inserted before
+a space/tab that appears in the data. If the data has no spaces, then no
+folding is possible.
+If there are spaces/tabs, the one closest to (and before) position 78 is used unless it is
+followed by punctuation, in which case a prior space/tab (if it exists) is used.
+(This preference for spaces not followed by punctuation is optional, it is
+not required, but is preferred in the IMF RFC.)
+
+Note: folding is done by some systems in a manner that does not respect
+character boundaries - i.e., in utf-8, a multi-byte character sequence may be
+broken in the middle by insertion of a CRLF. Hence, unfolding initially treats
+the text as iso-8859-1, i.e., just bytes, and removes CRLFs, then subsequently
+re-interprets the bytes as the expected charset such as utf-8.
+
+IMF is supposed to be US-ASCII, but implementations have gone to 8-bit characters
+being preserved, so the above problem really can occur.
+
+IMF has a maximum line length of 998 characters per line excluding the CRLF.
+
+- _WARNING This check for 998 characters is not implemented (As of Daffodil version 3.8.0)._
+
+The layer should fail (cause a parse error) if a line longer than this is encountered
+or constructed after unfolding. When unparsing, if a line longer than 998 cannot be
+folded due to no spaces/tabs being present in it, then it is an unparse error.
+
+Note that i/vCalendar, vCard, and MIME payloads held by IMF do not run into
+the IMF line length issues, in that they have their own line length limits that
+are smaller than those of IMF, and which do not require accommodation by having
+pre-existing spaces/tabs in the data. So such data will always be short
+enough lines.
+
+### Layer Name: lineFolded_iCalendar
+
+For iCalendar (including vCard and vCalendar), the maximum is 75 bytes plus the CRLF, for
+a total of 77. Folding is inserted by inserting CRLF + a space or tab. The
+CRLF and the following space or tab are removed to unfold. If data happened to
+contain a CRLF followed by a space or tab initially, then that will be lost when
+the data is parsed.
+
+For MIME, the maximum line length is 76.
+
+
+-----------
Review Comment:
Add sections for other built-in layers.
##########
site/layers.md:
##########
@@ -0,0 +1,382 @@
+---
+layout: page
+title: Layers - Pluggable Extensions to Enable Algorithmic Transformations in DFDL
+description: Pluggable Extensions to Enable Algorithmic Transformations in DFDL
+group: nav-right
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- base64_MIME
+- fourbyteswap
+- twobyteswap
+- gzip
+- lineFolded_IMF
+- lineFolded_iCalendar
+
+Daffodil also includes two utility layers that are used in combination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- boundaryMark
+- fixedLength
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+## Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html) class
+which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass-through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic Redundancy Check (CRC) over the data stream.
+The value of the checksum (or hash or CRC) is assigned to a DFDL variable as the result of the
+layer. This makes the value available for use by the DFDL schema that uses the checksum layer.
+When parsing, the value of this DFDL variable can then be compared to a checksum field in the
+data, and either an invalid data element or an parse-error can be created if the checksum in the
+data stream does not match the computed value.
+When unparsing, the value of this DFDL variable can be written to an element using the
+`dfdl:outputValueCalc` property.
+
+An example of a layer plug-in is in the <a href="https://github.com/DFDLSchemas/ethernetIP">EthernetIP</a>
+DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header checksum algorithm.
+
+Custom checksum layers are created by deriving an implementation class from the Daffodil API's
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.
+html)
+class, which is introduced in a later section.
+
+## Layer Composition
+
+Layers can be piled on top of each other. Once a layer is in place, another layer can be used in
+conjunction with it.
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses both the
+built-in _gzip_ layer, and the _base64_MIME_ layer together.
+
+# Using Layers
+
+To use a layer you must know
+- the layer's namespace URI
+- the layer's name
+- the names of any layer parameter variables
+- the names of any layer result variables
+
+## Example: Line Folding
+
+As a first example, let's look at the line folding layer, specifically the `lineFolded_IMF`layer,
+which is built-in to Daffodil.
+
+Line folding is a way of encoding textual data formats so that no line of text is longer than
+a limited line length.
+
+Consider this data :
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem
+ ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+This data has been _line folded_ at roughly 72 characters by inserting a CRLF
+before an existing space in the data.
+Each line ends with a CRLF (\r\n) and the second through fourth lines begin
+with a space as a way of indicating that they are extension lines.
+This data is supposed to be reassembled to form a long single-line string by removing
+all CRLF pairs.
+
+The result should be this single longer string which does not contain any line endings:
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+To achieve this we would use the `lineFolded_IMF` layer.
+This layer has a specific namespace which our DFDL schema will define a prefix `lf` for like this:
+```xml
+xmlns:lf="urn:org.apache.daffodil.layers.lineFolded"
+```
+Our DFDL schema will import the layer schema for the line-folded layer with this import:
+```xml
+<import namespace="urn:org.apache.daffodil.layers.lineFolded"
+ schemaLocation="/org/apache/daffodil/layers/xsd/lineFoldedLayer.dfdl.xsd"/>
+```
+Then the layer is incorporated into our DFDL schema like this:
+```xml
+ <sequence dfdlx:layer="lf:lineFolded_IMF">
+ ... elements to be parsed from unfolded layer data go here ...
+ </sequence>
+```
+You can see that use of a layer is described using the `dfdx:layer` property, and the specific layer
+is identified by a QName using the previously defined namespace prefix.
+The scope of the layer is the duration of the sequence it appears on.
+The `dfdlx:layer` property can only be used on an XSD `sequence`.
+
+The `lineFolded_IMF` layer does not define any DFDL variables in its namespace as it has no
+parameters and produces no results.
+In fact, as of Daffodil 3.8.0, the `import` statement above is optional for the
+line-folded layers as the DFDL schema `lineFoldedLayer.dfdl.xsd` does not contain any definitions.
+In the future however, parameters may be added, so for uniformity all layers define a DFDL
+schema to be imported as part of using the layer.
+
+More detailed documentation for the [Line Folded Layers](#LineFoldedLayers) is below.
+
+## Example: Base64, GZip, and BoundaryMark
+
+[EXAMPLE TBD] - should explain elements of specified length, or use of boundaryMark or
+fixed length layers.
+
+# Defining Custom Plug-In Layers
+
+Plug-in custom layers are dynamically loaded using the Java Service Provider Interface (SPI).
+Hence, they must be compiled into Jar files with specific META-INF metadata, and must appear on
+the Java CLASSPATH so that they can be found and loaded.
+
+The layer API is defined via Java clases and interfaces to enable writing of custom layers in
+either Java or Scala.
+(One of the built-in layers (Gzip) is written in Java now, by way of proving
+that one can write a Layer in Java.)
+
+A layer implementation class must obey various naming
+conventions that allow Java/Scala reflection to associate the names of Java/Scala code methods and
+method arguments with DFDL variables in that layer's namespace, that share those names.
+
+Transformer layer classes are derived from the
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html)
+base class.
+
+A name and target namespace, and hence a QName is required to identify a layer for use from
Review Comment:
Thiis paragraph to javadoc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] WIP: Checkpoint - start on layers documentation in layers.md [daffodil-site]
Posted by "mbeckerle (via GitHub)" <gi...@apache.org>.
mbeckerle commented on code in PR #132:
URL: https://github.com/apache/daffodil-site/pull/132#discussion_r1612133663
##########
site/layersTOC.md:
##########
@@ -0,0 +1,64 @@
+---
+description: Pluggable Extensions to Enable Algorithmic Transformations
+ in DFDL
+group: 'nav-right'
+layout: page
+title: 'Layers - Algorithmic
+ Extensions for DFDL'
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+### Table of Contents
+
+<!--
+{% comment %}
+NOTE: Generate using `pandoc -s --toc layers.md -o layersTOC.md`
+
+Followed by fixing up the layerTOC.md page so it works.
+{% endcomment %}
+-->
+
+- [Introduction](#introduction)
+ - [Built-in Layers](#built-in-layers)
+ - [Custom Plug-In Layers](#custom-plug-in-layers)
+ - [Transforming Layers and Checksum
+ Layers](#transforming-layers-and-checksum-layers)
+ - [Transforming Layers](#transforming-layers)
+ - [Checksum Layers](#checksum-layers)
+ - [Layer Composition](#layer-composition)
+- [Using Layers](#using-layers)
+ - [Example: Line Folding](#example-line-folding)
+ - [Example: Base64, GZip, and
+ BoundaryMark](#example-base64-gzip-and-boundarymark)
+- [Defining Custom Plug-In Layers](#defining-custom-plug-in-layers)
+ - [Layer Variables:](#layer-variables)
+ - [Layer Exception Handling:](#layer-exception-handling)
+ - [Compatibility with Daffodil 3.7.0 and
+ prior](#compatibility-with-daffodil-3.7.0-and-prior)
+- [Daffodil Built-In Layer
+ Documentation](#daffodil-built-in-layer-documentation)
+ - [Line Folded Layers](#line-folded-layers)
+ - [General Usage](#general-usage)
+ - [Layer Name: lineFolded\_IMF](#layer-name-linefolded_imf)
+ - [Layer Name:
+ lineFolded\_iCalendar](#layer-name-linefolded_icalendar)
+
+
+{% include_relative layers.md %}
Review Comment:
Works. Thanks. Much simpler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] WIP: Checkpoint - start on layers documentation in layers.md [daffodil-site]
Posted by "tuxji (via GitHub)" <gi...@apache.org>.
tuxji commented on code in PR #132:
URL: https://github.com/apache/daffodil-site/pull/132#discussion_r1609874552
##########
site/layers.md:
##########
@@ -0,0 +1,380 @@
+---
+layout: page
+title: Layers - Pluggable Extensions to Enable Algorithmic Transformations in DFDL
+description: Pluggable Extensions to Enable Algorithmic Transformations in DFDL
+group: nav-right
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- base64_MIME
+- fourbyteswap
+- twobyteswap
+- gzip
+- lineFolded_IMF
+- lineFolded_iCalendar
+
+Daffodil also includes two utility layers that are used incombination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- boundaryMark
+- fixedLength
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+## Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+`Layer` class which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass-through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic Redundancy Check (CRC) over the data stream.
Review Comment:
Long line
##########
site/layers.md:
##########
@@ -0,0 +1,380 @@
+---
+layout: page
+title: Layers - Pluggable Extensions to Enable Algorithmic Transformations in DFDL
+description: Pluggable Extensions to Enable Algorithmic Transformations in DFDL
+group: nav-right
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- base64_MIME
+- fourbyteswap
+- twobyteswap
+- gzip
+- lineFolded_IMF
+- lineFolded_iCalendar
+
+Daffodil also includes two utility layers that are used incombination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- boundaryMark
+- fixedLength
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+## Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+`Layer` class which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass-through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic Redundancy Check (CRC) over the data stream.
+The value of the checksum (or hash or CRC) is assigned to a DFDL variable as the result of the
+layer. This makes the value available for use by the DFDL schema that uses the checksum layer.
+When parsing, the value of this DFDL variable can then be compared to a checksum field in the
+data, and either an invalid data element or an parse-error can be created if the checksum in the
+data stream does not match the computed value.
+When unparsing, the value of this DFDL variable can be written to an element using the
+`dfdl:outputValueCalc` property.
+
+An example of a layer plug-in is in the <a href="https://github.com/DFDLSchemas/ethernetIP">EthernetIP</a>
+DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header checksum algorithm.
+
+Custom checksum layers are created by deriving an implementation class from the Daffodil API's
+`ChecksumLayer` class, which is introduced in a later section.
+
+## Layer Composition
+
+Layers can be piled on top of each other. Once a layer is in place, another layer can be used in
Review Comment:
Consider another verb like "layered on top of each other"?
##########
site/layers.md:
##########
@@ -0,0 +1,380 @@
+---
+layout: page
+title: Layers - Pluggable Extensions to Enable Algorithmic Transformations in DFDL
+description: Pluggable Extensions to Enable Algorithmic Transformations in DFDL
+group: nav-right
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- base64_MIME
+- fourbyteswap
+- twobyteswap
+- gzip
+- lineFolded_IMF
+- lineFolded_iCalendar
+
+Daffodil also includes two utility layers that are used incombination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- boundaryMark
+- fixedLength
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+## Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+`Layer` class which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass-through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic Redundancy Check (CRC) over the data stream.
+The value of the checksum (or hash or CRC) is assigned to a DFDL variable as the result of the
+layer. This makes the value available for use by the DFDL schema that uses the checksum layer.
+When parsing, the value of this DFDL variable can then be compared to a checksum field in the
+data, and either an invalid data element or an parse-error can be created if the checksum in the
+data stream does not match the computed value.
+When unparsing, the value of this DFDL variable can be written to an element using the
+`dfdl:outputValueCalc` property.
+
+An example of a layer plug-in is in the <a href="https://github.com/DFDLSchemas/ethernetIP">EthernetIP</a>
+DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header checksum algorithm.
+
+Custom checksum layers are created by deriving an implementation class from the Daffodil API's
+`ChecksumLayer` class, which is introduced in a later section.
+
+## Layer Composition
+
+Layers can be piled on top of each other. Once a layer is in place, another layer can be used in
+conjunction with it.
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses both the
+built-in _gzip_ layer, and the _base64_MIME_ layer together.
+
+# Using Layers
+
+To use a layer you must know
+- the layer's namespace URI
+- the layer's name
+- the names of any layer parameter variables
+- the names of any layer result variables
+
+## Example: Line Folding
+
+As a first example, let's look at the line folding layer, specifically the `lineFolded_IMF`layer,
Review Comment:
Insert space between `IMF` and layer
##########
site/layers.md:
##########
@@ -0,0 +1,380 @@
+---
+layout: page
+title: Layers - Pluggable Extensions to Enable Algorithmic Transformations in DFDL
+description: Pluggable Extensions to Enable Algorithmic Transformations in DFDL
+group: nav-right
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- base64_MIME
+- fourbyteswap
+- twobyteswap
+- gzip
+- lineFolded_IMF
+- lineFolded_iCalendar
+
+Daffodil also includes two utility layers that are used incombination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- boundaryMark
+- fixedLength
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+## Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+`Layer` class which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass-through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic Redundancy Check (CRC) over the data stream.
+The value of the checksum (or hash or CRC) is assigned to a DFDL variable as the result of the
+layer. This makes the value available for use by the DFDL schema that uses the checksum layer.
+When parsing, the value of this DFDL variable can then be compared to a checksum field in the
+data, and either an invalid data element or an parse-error can be created if the checksum in the
+data stream does not match the computed value.
+When unparsing, the value of this DFDL variable can be written to an element using the
+`dfdl:outputValueCalc` property.
+
+An example of a layer plug-in is in the <a href="https://github.com/DFDLSchemas/ethernetIP">EthernetIP</a>
+DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header checksum algorithm.
+
+Custom checksum layers are created by deriving an implementation class from the Daffodil API's
+`ChecksumLayer` class, which is introduced in a later section.
+
+## Layer Composition
+
+Layers can be piled on top of each other. Once a layer is in place, another layer can be used in
+conjunction with it.
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses both the
+built-in _gzip_ layer, and the _base64_MIME_ layer together.
+
+# Using Layers
+
+To use a layer you must know
+- the layer's namespace URI
+- the layer's name
+- the names of any layer parameter variables
+- the names of any layer result variables
+
+## Example: Line Folding
+
+As a first example, let's look at the line folding layer, specifically the `lineFolded_IMF`layer,
+which is built-in to Daffodil.
+
+Line folding is a way of encoding textual data formats so that no line of text is longer than
+a limited line length.
+
+Consider this data :
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem
+ ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+This data has been _line folded_ at roughly 72 characters by inserting a CRLF
+before an existing space in the data.
+Each line ends with a CRLF (\r\n) and the second through fourth lines begin
+with a space as a way of indicating that they are extension lines.
+This data is supposed to be reassembled to form a long single-line string by removing
+all CRLF pairs.
+
+The result should be this single longer string which does not contain any line endings:
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+To achieve this we would use the `lineFolded_IMF` layer.
+This layer has a specific namespace which our DFDL schema will define a prefix `lf` for like this:
+```xml
+xmlns:lf="urn:org.apache.daffodil.layers.lineFolded"
+```
+Our DFDL schema will import the layer schema for the line-folded layer with this import:
+```xml
+<import namespace="urn:org.apache.daffodil.layers.lineFolded"
+ schemaLocation="/org/apache/daffodil/layers/xsd/lineFoldedLayer.dfdl.xsd"/>
+```
+Then the layer is incorporated into our DFDL schema like this:
+```xml
+ <sequence dfdlx:layer="lf:lineFolded_IMF">
+ ... elements to be parsed from unfolded layer data go here ...
+ </sequence>
+```
+You can see that use of a layer is described using the `dfdx:layer` property, and the specific layer
+is identified by a QName using the previously defined namespace prefix.
+The scope of the layer is the duration of the sequence it appears on.
+The `dfdlx:layer` property can only be used on an XSD `sequence`.
+
+The `lineFolded_IMF` layer does not define any DFDL variables in its namespace as it has no
+parameters and produces no results.
+In fact, as of Daffodil 3.8.0, the `import` statement above is optional for the
+line-folded layers as the DFDL schema `lineFoldedLayer.dfdl.xsd` does not contain any definitions.
+In the future however, parameters may be added, so for uniformity all layers define a DFDL
+schema to be imported as part of using the layer.
+
+More detailed documentation for the [Line Folded Layers](#LineFoldedLayers) is below.
+
+## Example: Base64, GZip, and BoundaryMark
+
+[EXAMPLE TBD] - should explain elements of specified length, or use of boundaryMark or
+fixed length layers.
+
+# Defining Custom Plug-In Layers
+
+[NOTE: Try not to be redundant with the javadoc for Layer and ChecksumLayer classes]
+[Link to them where relevant.]
+
+Plug-in custom layers are dynamically loaded using the Java Service Provider Interface (SPI).
+Hence, they must be compiled into Jar files with specific META-INF metadata, and must appear on
+the Java CLASSPATH so that they can be found and loaded.
+
+The layer API is defined via Java clases and interfaces to enable writing of custom layers in
+either Java or Scala.
+(One of the built-in layers (Gzip) is written in Java now, by way of proving
+that one can write a Layer in Java.)
+
+Transformer layer classes are derived from
+
+ org.apache.daffodil.runtime1.layers.api.Layer
+
+A layer a name and target namespace, and hence a QName is required to
+identify it.
+This namespace is OWNED by the layer.
+All DFDL variables
+defined in that namespace are either used to pass parameters to the
+layer code, or receive results (such as a checksum) back from the
+layer code.
+This is enforced.
+A layer that has no DFDL variables does
+not have to define a DFDL schema that defines the layer's target
+namespace, but any layer that uses DFDL variables *must* define a
+schema with the layer's namespace as its target namespace, with the
+variables declared in it (using `dfdl:defineVariable`).
+
+There is also an abstract base class for defining checksum layers
+called,
+
+ org.apache.daffodil.runtime1.layers.api.ChecksumLayer
+
+### Layer Limiting
+
+A layer can process an entire input file/stream.
+Such a layer is said to be _unlimited_.
+More commonly, a layer has a limited region within the data stream that it is supposed to process.
+The restriction of the layer so that it only processes the expected part of the data is called
+_layer limiting_.
+
+There are two ways to do layer limiting, that differ in an
+important way.
Review Comment:
I seem to see three ways to do layer limiting below, not only two?
##########
site/layers.md:
##########
@@ -0,0 +1,380 @@
+---
+layout: page
+title: Layers - Pluggable Extensions to Enable Algorithmic Transformations in DFDL
+description: Pluggable Extensions to Enable Algorithmic Transformations in DFDL
+group: nav-right
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- base64_MIME
+- fourbyteswap
+- twobyteswap
+- gzip
+- lineFolded_IMF
+- lineFolded_iCalendar
+
+Daffodil also includes two utility layers that are used incombination with other layers to
Review Comment:
*in combination
##########
site/layers.md:
##########
@@ -0,0 +1,380 @@
+---
+layout: page
+title: Layers - Pluggable Extensions to Enable Algorithmic Transformations in DFDL
+description: Pluggable Extensions to Enable Algorithmic Transformations in DFDL
+group: nav-right
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- base64_MIME
+- fourbyteswap
+- twobyteswap
+- gzip
+- lineFolded_IMF
+- lineFolded_iCalendar
+
+Daffodil also includes two utility layers that are used incombination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- boundaryMark
+- fixedLength
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+## Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+`Layer` class which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass-through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic Redundancy Check (CRC) over the data stream.
+The value of the checksum (or hash or CRC) is assigned to a DFDL variable as the result of the
+layer. This makes the value available for use by the DFDL schema that uses the checksum layer.
+When parsing, the value of this DFDL variable can then be compared to a checksum field in the
+data, and either an invalid data element or an parse-error can be created if the checksum in the
+data stream does not match the computed value.
+When unparsing, the value of this DFDL variable can be written to an element using the
+`dfdl:outputValueCalc` property.
+
+An example of a layer plug-in is in the <a href="https://github.com/DFDLSchemas/ethernetIP">EthernetIP</a>
+DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header checksum algorithm.
+
+Custom checksum layers are created by deriving an implementation class from the Daffodil API's
+`ChecksumLayer` class, which is introduced in a later section.
Review Comment:
Markdown [supports](https://stackoverflow.com/questions/2822089/how-to-link-to-part-of-the-same-document-in-markdown) intra-document links. Also, if the kramdown automatic TOC generator Steve suggested doesn't work, this StackOverflow [page](https://stackoverflow.com/questions/11948245/markdown-to-create-pages-and-table-of-contents) mentions other ways (IDE plugins, running `pandoc -s --toc input.md -o input_toc.md`, etc.).
##########
site/layers.md:
##########
@@ -0,0 +1,380 @@
+---
+layout: page
+title: Layers - Pluggable Extensions to Enable Algorithmic Transformations in DFDL
+description: Pluggable Extensions to Enable Algorithmic Transformations in DFDL
+group: nav-right
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- base64_MIME
+- fourbyteswap
+- twobyteswap
+- gzip
+- lineFolded_IMF
+- lineFolded_iCalendar
+
+Daffodil also includes two utility layers that are used incombination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- boundaryMark
+- fixedLength
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+## Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+`Layer` class which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass-through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic Redundancy Check (CRC) over the data stream.
+The value of the checksum (or hash or CRC) is assigned to a DFDL variable as the result of the
+layer. This makes the value available for use by the DFDL schema that uses the checksum layer.
+When parsing, the value of this DFDL variable can then be compared to a checksum field in the
+data, and either an invalid data element or an parse-error can be created if the checksum in the
+data stream does not match the computed value.
+When unparsing, the value of this DFDL variable can be written to an element using the
+`dfdl:outputValueCalc` property.
+
+An example of a layer plug-in is in the <a href="https://github.com/DFDLSchemas/ethernetIP">EthernetIP</a>
+DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header checksum algorithm.
+
+Custom checksum layers are created by deriving an implementation class from the Daffodil API's
+`ChecksumLayer` class, which is introduced in a later section.
+
+## Layer Composition
+
+Layers can be piled on top of each other. Once a layer is in place, another layer can be used in
+conjunction with it.
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses both the
+built-in _gzip_ layer, and the _base64_MIME_ layer together.
+
+# Using Layers
+
+To use a layer you must know
+- the layer's namespace URI
+- the layer's name
+- the names of any layer parameter variables
+- the names of any layer result variables
+
+## Example: Line Folding
+
+As a first example, let's look at the line folding layer, specifically the `lineFolded_IMF`layer,
+which is built-in to Daffodil.
+
+Line folding is a way of encoding textual data formats so that no line of text is longer than
+a limited line length.
+
+Consider this data :
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem
+ ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+This data has been _line folded_ at roughly 72 characters by inserting a CRLF
+before an existing space in the data.
+Each line ends with a CRLF (\r\n) and the second through fourth lines begin
+with a space as a way of indicating that they are extension lines.
+This data is supposed to be reassembled to form a long single-line string by removing
+all CRLF pairs.
+
+The result should be this single longer string which does not contain any line endings:
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+To achieve this we would use the `lineFolded_IMF` layer.
+This layer has a specific namespace which our DFDL schema will define a prefix `lf` for like this:
+```xml
+xmlns:lf="urn:org.apache.daffodil.layers.lineFolded"
+```
+Our DFDL schema will import the layer schema for the line-folded layer with this import:
+```xml
+<import namespace="urn:org.apache.daffodil.layers.lineFolded"
+ schemaLocation="/org/apache/daffodil/layers/xsd/lineFoldedLayer.dfdl.xsd"/>
+```
+Then the layer is incorporated into our DFDL schema like this:
+```xml
+ <sequence dfdlx:layer="lf:lineFolded_IMF">
+ ... elements to be parsed from unfolded layer data go here ...
+ </sequence>
+```
+You can see that use of a layer is described using the `dfdx:layer` property, and the specific layer
+is identified by a QName using the previously defined namespace prefix.
+The scope of the layer is the duration of the sequence it appears on.
+The `dfdlx:layer` property can only be used on an XSD `sequence`.
+
+The `lineFolded_IMF` layer does not define any DFDL variables in its namespace as it has no
+parameters and produces no results.
+In fact, as of Daffodil 3.8.0, the `import` statement above is optional for the
+line-folded layers as the DFDL schema `lineFoldedLayer.dfdl.xsd` does not contain any definitions.
+In the future however, parameters may be added, so for uniformity all layers define a DFDL
+schema to be imported as part of using the layer.
+
+More detailed documentation for the [Line Folded Layers](#LineFoldedLayers) is below.
+
+## Example: Base64, GZip, and BoundaryMark
+
+[EXAMPLE TBD] - should explain elements of specified length, or use of boundaryMark or
+fixed length layers.
+
+# Defining Custom Plug-In Layers
+
+[NOTE: Try not to be redundant with the javadoc for Layer and ChecksumLayer classes]
+[Link to them where relevant.]
+
+Plug-in custom layers are dynamically loaded using the Java Service Provider Interface (SPI).
+Hence, they must be compiled into Jar files with specific META-INF metadata, and must appear on
+the Java CLASSPATH so that they can be found and loaded.
+
+The layer API is defined via Java clases and interfaces to enable writing of custom layers in
+either Java or Scala.
+(One of the built-in layers (Gzip) is written in Java now, by way of proving
+that one can write a Layer in Java.)
+
+Transformer layer classes are derived from
+
+ org.apache.daffodil.runtime1.layers.api.Layer
+
+A layer a name and target namespace, and hence a QName is required to
+identify it.
+This namespace is OWNED by the layer.
+All DFDL variables
+defined in that namespace are either used to pass parameters to the
+layer code, or receive results (such as a checksum) back from the
+layer code.
+This is enforced.
+A layer that has no DFDL variables does
+not have to define a DFDL schema that defines the layer's target
+namespace, but any layer that uses DFDL variables *must* define a
+schema with the layer's namespace as its target namespace, with the
+variables declared in it (using `dfdl:defineVariable`).
+
+There is also an abstract base class for defining checksum layers
+called,
+
+ org.apache.daffodil.runtime1.layers.api.ChecksumLayer
+
+### Layer Limiting
+
+A layer can process an entire input file/stream.
+Such a layer is said to be _unlimited_.
+More commonly, a layer has a limited region within the data stream that it is supposed to process.
+The restriction of the layer so that it only processes the expected part of the data is called
+_layer limiting_.
+
+There are two ways to do layer limiting, that differ in an
+important way.
+
+#### Layer Limiting using Elements of Specified Length
+
+If the `sequence` for a layer is the model group of the
+complex type of an element, and that element has `dfdl:lengthKind` of `'explicit'`, `'implicit'`,
+or `'pattern'`, then the length of the element limits the length of the layer within it.
+
+In this case the length of the layer is limited when parsing, but it is NOT limited when
+unparsing.
+
+#### Layer Limiting using the `fixedLength` Utility Layer
+
+If you use the layer `fixedLength`, that dictates the parse and
+unparse length to be the value of the `layerLength` DFDL variable.
+
+Similarly, if you use a `checksum` layer built with the `ChecksumLayer` base class,
+the length is controlled for both parsing and unparsing as it works
+similarly to the way the `fixedLength` layer works.
+
+#### Layer Limiting using the `boundaryMark` Utility Layer
+
+The `boundaryMark` layer uses a variable named `boundaryMark` which provides a parameter to the
+layer which is a delimiter of the layer data when parsing, and
+which is inserted after the (otherwise unbounded) layer data when unparsing.
+
+### Layer Variables:
+
+Each variable is either a parameter passed to a special setter named
+`setLayerVariableParameters`, or is a return variable which is populated
+from a special getter.
+The special setter args and getters taken
+together must use up all the DFDL variables defined in the layer's
+namespace. (See the javadoc for the Layer base class.)
+
+A special getter example for a checksum layer:
+
+ int getLayerVariableResult_checksum() { .... }
+
+This corresponds to, and returns the value that populates, the DFDL
+variable named `checksum` in the layer's namespace.
+That DFDL variable must have a DFDL type corresponding to the `int` type (`xs:unsignedShort` or
+`xs:int`).
+
+Layers could have multiple result variables, but we have no actual
+examples of more than one value being returned.
+(Returning multiple variables is tested.
+We just have no real use cases for it at this time.)
+
+### Layer Exception Handling:
+
+The method
+
+ setProcessingErrorException(...)
+
+allows the layer to specify that if the layer throws specific
+exceptions or runtime exceptions that they are converted into processing
+errors.
+This eliminates most need for layers to contain try-catches.
Review Comment:
Please clarify and show an example of what goes into this method. What would one put inside the (...) parentheses and inside the method's body (assuming the layer defines this method instead of importing it from somewhere and calling it)?
##########
site/layers.md:
##########
@@ -0,0 +1,380 @@
+---
+layout: page
+title: Layers - Pluggable Extensions to Enable Algorithmic Transformations in DFDL
+description: Pluggable Extensions to Enable Algorithmic Transformations in DFDL
+group: nav-right
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- base64_MIME
+- fourbyteswap
+- twobyteswap
+- gzip
+- lineFolded_IMF
+- lineFolded_iCalendar
+
+Daffodil also includes two utility layers that are used incombination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- boundaryMark
+- fixedLength
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+## Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+`Layer` class which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass-through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic Redundancy Check (CRC) over the data stream.
+The value of the checksum (or hash or CRC) is assigned to a DFDL variable as the result of the
+layer. This makes the value available for use by the DFDL schema that uses the checksum layer.
+When parsing, the value of this DFDL variable can then be compared to a checksum field in the
+data, and either an invalid data element or an parse-error can be created if the checksum in the
+data stream does not match the computed value.
+When unparsing, the value of this DFDL variable can be written to an element using the
+`dfdl:outputValueCalc` property.
+
+An example of a layer plug-in is in the <a href="https://github.com/DFDLSchemas/ethernetIP">EthernetIP</a>
+DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header checksum algorithm.
+
+Custom checksum layers are created by deriving an implementation class from the Daffodil API's
+`ChecksumLayer` class, which is introduced in a later section.
+
+## Layer Composition
+
+Layers can be piled on top of each other. Once a layer is in place, another layer can be used in
+conjunction with it.
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses both the
+built-in _gzip_ layer, and the _base64_MIME_ layer together.
+
+# Using Layers
+
+To use a layer you must know
+- the layer's namespace URI
+- the layer's name
+- the names of any layer parameter variables
+- the names of any layer result variables
+
+## Example: Line Folding
+
+As a first example, let's look at the line folding layer, specifically the `lineFolded_IMF`layer,
+which is built-in to Daffodil.
+
+Line folding is a way of encoding textual data formats so that no line of text is longer than
+a limited line length.
+
+Consider this data :
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem
+ ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+This data has been _line folded_ at roughly 72 characters by inserting a CRLF
+before an existing space in the data.
+Each line ends with a CRLF (\r\n) and the second through fourth lines begin
+with a space as a way of indicating that they are extension lines.
+This data is supposed to be reassembled to form a long single-line string by removing
+all CRLF pairs.
+
+The result should be this single longer string which does not contain any line endings:
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+To achieve this we would use the `lineFolded_IMF` layer.
+This layer has a specific namespace which our DFDL schema will define a prefix `lf` for like this:
+```xml
+xmlns:lf="urn:org.apache.daffodil.layers.lineFolded"
+```
+Our DFDL schema will import the layer schema for the line-folded layer with this import:
+```xml
+<import namespace="urn:org.apache.daffodil.layers.lineFolded"
+ schemaLocation="/org/apache/daffodil/layers/xsd/lineFoldedLayer.dfdl.xsd"/>
+```
+Then the layer is incorporated into our DFDL schema like this:
+```xml
+ <sequence dfdlx:layer="lf:lineFolded_IMF">
+ ... elements to be parsed from unfolded layer data go here ...
+ </sequence>
+```
+You can see that use of a layer is described using the `dfdx:layer` property, and the specific layer
+is identified by a QName using the previously defined namespace prefix.
+The scope of the layer is the duration of the sequence it appears on.
+The `dfdlx:layer` property can only be used on an XSD `sequence`.
+
+The `lineFolded_IMF` layer does not define any DFDL variables in its namespace as it has no
+parameters and produces no results.
+In fact, as of Daffodil 3.8.0, the `import` statement above is optional for the
+line-folded layers as the DFDL schema `lineFoldedLayer.dfdl.xsd` does not contain any definitions.
+In the future however, parameters may be added, so for uniformity all layers define a DFDL
+schema to be imported as part of using the layer.
+
+More detailed documentation for the [Line Folded Layers](#LineFoldedLayers) is below.
+
+## Example: Base64, GZip, and BoundaryMark
+
+[EXAMPLE TBD] - should explain elements of specified length, or use of boundaryMark or
+fixed length layers.
+
+# Defining Custom Plug-In Layers
+
+[NOTE: Try not to be redundant with the javadoc for Layer and ChecksumLayer classes]
+[Link to them where relevant.]
+
+Plug-in custom layers are dynamically loaded using the Java Service Provider Interface (SPI).
+Hence, they must be compiled into Jar files with specific META-INF metadata, and must appear on
+the Java CLASSPATH so that they can be found and loaded.
+
+The layer API is defined via Java clases and interfaces to enable writing of custom layers in
+either Java or Scala.
+(One of the built-in layers (Gzip) is written in Java now, by way of proving
+that one can write a Layer in Java.)
+
+Transformer layer classes are derived from
+
+ org.apache.daffodil.runtime1.layers.api.Layer
+
+A layer a name and target namespace, and hence a QName is required to
+identify it.
+This namespace is OWNED by the layer.
+All DFDL variables
+defined in that namespace are either used to pass parameters to the
+layer code, or receive results (such as a checksum) back from the
+layer code.
+This is enforced.
+A layer that has no DFDL variables does
+not have to define a DFDL schema that defines the layer's target
+namespace, but any layer that uses DFDL variables *must* define a
+schema with the layer's namespace as its target namespace, with the
+variables declared in it (using `dfdl:defineVariable`).
+
+There is also an abstract base class for defining checksum layers
+called,
+
+ org.apache.daffodil.runtime1.layers.api.ChecksumLayer
+
+### Layer Limiting
+
+A layer can process an entire input file/stream.
+Such a layer is said to be _unlimited_.
+More commonly, a layer has a limited region within the data stream that it is supposed to process.
+The restriction of the layer so that it only processes the expected part of the data is called
+_layer limiting_.
+
+There are two ways to do layer limiting, that differ in an
+important way.
+
+#### Layer Limiting using Elements of Specified Length
+
+If the `sequence` for a layer is the model group of the
+complex type of an element, and that element has `dfdl:lengthKind` of `'explicit'`, `'implicit'`,
+or `'pattern'`, then the length of the element limits the length of the layer within it.
+
+In this case the length of the layer is limited when parsing, but it is NOT limited when
+unparsing.
+
+#### Layer Limiting using the `fixedLength` Utility Layer
+
+If you use the layer `fixedLength`, that dictates the parse and
+unparse length to be the value of the `layerLength` DFDL variable.
+
+Similarly, if you use a `checksum` layer built with the `ChecksumLayer` base class,
+the length is controlled for both parsing and unparsing as it works
+similarly to the way the `fixedLength` layer works.
+
+#### Layer Limiting using the `boundaryMark` Utility Layer
+
+The `boundaryMark` layer uses a variable named `boundaryMark` which provides a parameter to the
+layer which is a delimiter of the layer data when parsing, and
+which is inserted after the (otherwise unbounded) layer data when unparsing.
+
+### Layer Variables:
Review Comment:
Headers typically don't have punctuation, and this line is the first time that I've noticed punctuation (all previous headers have no punctuation).
##########
site/layers.md:
##########
@@ -0,0 +1,380 @@
+---
+layout: page
+title: Layers - Pluggable Extensions to Enable Algorithmic Transformations in DFDL
+description: Pluggable Extensions to Enable Algorithmic Transformations in DFDL
+group: nav-right
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- base64_MIME
+- fourbyteswap
+- twobyteswap
+- gzip
+- lineFolded_IMF
+- lineFolded_iCalendar
+
+Daffodil also includes two utility layers that are used incombination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- boundaryMark
+- fixedLength
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+## Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+`Layer` class which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass-through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic Redundancy Check (CRC) over the data stream.
+The value of the checksum (or hash or CRC) is assigned to a DFDL variable as the result of the
+layer. This makes the value available for use by the DFDL schema that uses the checksum layer.
+When parsing, the value of this DFDL variable can then be compared to a checksum field in the
+data, and either an invalid data element or an parse-error can be created if the checksum in the
+data stream does not match the computed value.
+When unparsing, the value of this DFDL variable can be written to an element using the
+`dfdl:outputValueCalc` property.
+
+An example of a layer plug-in is in the <a href="https://github.com/DFDLSchemas/ethernetIP">EthernetIP</a>
+DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header checksum algorithm.
+
+Custom checksum layers are created by deriving an implementation class from the Daffodil API's
+`ChecksumLayer` class, which is introduced in a later section.
+
+## Layer Composition
+
+Layers can be piled on top of each other. Once a layer is in place, another layer can be used in
+conjunction with it.
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses both the
+built-in _gzip_ layer, and the _base64_MIME_ layer together.
+
+# Using Layers
+
+To use a layer you must know
+- the layer's namespace URI
+- the layer's name
+- the names of any layer parameter variables
+- the names of any layer result variables
+
+## Example: Line Folding
+
+As a first example, let's look at the line folding layer, specifically the `lineFolded_IMF`layer,
+which is built-in to Daffodil.
+
+Line folding is a way of encoding textual data formats so that no line of text is longer than
+a limited line length.
+
+Consider this data :
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem
+ ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+This data has been _line folded_ at roughly 72 characters by inserting a CRLF
+before an existing space in the data.
+Each line ends with a CRLF (\r\n) and the second through fourth lines begin
+with a space as a way of indicating that they are extension lines.
+This data is supposed to be reassembled to form a long single-line string by removing
+all CRLF pairs.
+
+The result should be this single longer string which does not contain any line endings:
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+To achieve this we would use the `lineFolded_IMF` layer.
+This layer has a specific namespace which our DFDL schema will define a prefix `lf` for like this:
+```xml
+xmlns:lf="urn:org.apache.daffodil.layers.lineFolded"
+```
+Our DFDL schema will import the layer schema for the line-folded layer with this import:
+```xml
+<import namespace="urn:org.apache.daffodil.layers.lineFolded"
+ schemaLocation="/org/apache/daffodil/layers/xsd/lineFoldedLayer.dfdl.xsd"/>
+```
+Then the layer is incorporated into our DFDL schema like this:
+```xml
+ <sequence dfdlx:layer="lf:lineFolded_IMF">
+ ... elements to be parsed from unfolded layer data go here ...
+ </sequence>
+```
+You can see that use of a layer is described using the `dfdx:layer` property, and the specific layer
+is identified by a QName using the previously defined namespace prefix.
+The scope of the layer is the duration of the sequence it appears on.
+The `dfdlx:layer` property can only be used on an XSD `sequence`.
+
+The `lineFolded_IMF` layer does not define any DFDL variables in its namespace as it has no
+parameters and produces no results.
+In fact, as of Daffodil 3.8.0, the `import` statement above is optional for the
+line-folded layers as the DFDL schema `lineFoldedLayer.dfdl.xsd` does not contain any definitions.
+In the future however, parameters may be added, so for uniformity all layers define a DFDL
+schema to be imported as part of using the layer.
+
+More detailed documentation for the [Line Folded Layers](#LineFoldedLayers) is below.
+
+## Example: Base64, GZip, and BoundaryMark
+
+[EXAMPLE TBD] - should explain elements of specified length, or use of boundaryMark or
+fixed length layers.
+
+# Defining Custom Plug-In Layers
+
+[NOTE: Try not to be redundant with the javadoc for Layer and ChecksumLayer classes]
+[Link to them where relevant.]
+
+Plug-in custom layers are dynamically loaded using the Java Service Provider Interface (SPI).
+Hence, they must be compiled into Jar files with specific META-INF metadata, and must appear on
+the Java CLASSPATH so that they can be found and loaded.
+
+The layer API is defined via Java clases and interfaces to enable writing of custom layers in
+either Java or Scala.
+(One of the built-in layers (Gzip) is written in Java now, by way of proving
+that one can write a Layer in Java.)
+
+Transformer layer classes are derived from
+
+ org.apache.daffodil.runtime1.layers.api.Layer
+
+A layer a name and target namespace, and hence a QName is required to
+identify it.
+This namespace is OWNED by the layer.
+All DFDL variables
+defined in that namespace are either used to pass parameters to the
+layer code, or receive results (such as a checksum) back from the
+layer code.
+This is enforced.
+A layer that has no DFDL variables does
+not have to define a DFDL schema that defines the layer's target
+namespace, but any layer that uses DFDL variables *must* define a
+schema with the layer's namespace as its target namespace, with the
+variables declared in it (using `dfdl:defineVariable`).
+
+There is also an abstract base class for defining checksum layers
+called,
+
+ org.apache.daffodil.runtime1.layers.api.ChecksumLayer
+
+### Layer Limiting
+
+A layer can process an entire input file/stream.
+Such a layer is said to be _unlimited_.
+More commonly, a layer has a limited region within the data stream that it is supposed to process.
+The restriction of the layer so that it only processes the expected part of the data is called
+_layer limiting_.
+
+There are two ways to do layer limiting, that differ in an
+important way.
+
+#### Layer Limiting using Elements of Specified Length
+
+If the `sequence` for a layer is the model group of the
+complex type of an element, and that element has `dfdl:lengthKind` of `'explicit'`, `'implicit'`,
+or `'pattern'`, then the length of the element limits the length of the layer within it.
+
+In this case the length of the layer is limited when parsing, but it is NOT limited when
+unparsing.
+
+#### Layer Limiting using the `fixedLength` Utility Layer
+
+If you use the layer `fixedLength`, that dictates the parse and
+unparse length to be the value of the `layerLength` DFDL variable.
+
+Similarly, if you use a `checksum` layer built with the `ChecksumLayer` base class,
+the length is controlled for both parsing and unparsing as it works
+similarly to the way the `fixedLength` layer works.
+
+#### Layer Limiting using the `boundaryMark` Utility Layer
+
+The `boundaryMark` layer uses a variable named `boundaryMark` which provides a parameter to the
+layer which is a delimiter of the layer data when parsing, and
+which is inserted after the (otherwise unbounded) layer data when unparsing.
+
+### Layer Variables:
+
+Each variable is either a parameter passed to a special setter named
+`setLayerVariableParameters`, or is a return variable which is populated
+from a special getter.
+The special setter args and getters taken
+together must use up all the DFDL variables defined in the layer's
+namespace. (See the javadoc for the Layer base class.)
+
+A special getter example for a checksum layer:
+
+ int getLayerVariableResult_checksum() { .... }
+
+This corresponds to, and returns the value that populates, the DFDL
+variable named `checksum` in the layer's namespace.
+That DFDL variable must have a DFDL type corresponding to the `int` type (`xs:unsignedShort` or
+`xs:int`).
+
+Layers could have multiple result variables, but we have no actual
+examples of more than one value being returned.
+(Returning multiple variables is tested.
+We just have no real use cases for it at this time.)
+
+### Layer Exception Handling:
+
+The method
+
+ setProcessingErrorException(...)
+
+allows the layer to specify that if the layer throws specific
+exceptions or runtime exceptions that they are converted into processing
+errors.
+This eliminates most need for layers to contain try-catches.
+
+#### About Testing:
+
+There are tests for every way that
+a user can goof up the definition of a layer class, and there are
+tests for `processingError`, `runtimeSchemaDefinitionError`, and throwing
+an `Exception` from every place a user-defined Layer could cause these.
+Parse errors cause backtracking in all sensible cases.
+Nothing aborts.
+
+## Compatibility with Daffodil 3.7.0 and prior
+
+This new Layer API is 100% incompatible with schemas or layer code
+from Daffodil 3.7.0 and all prior versions of Daffodil.
+The layer
+feature was just an experimental feature in Daffodil, so we reserved
+the right to change it.
+
+However, it is our hope that the new Layer API introduced in
+Daffodil 3.8.0 will prove to be stable and supportable indefinitely.
+
+
+
+# Daffodil Built-In Layer Documentation
+
+## Line Folded Layers
+
+- Namespace URI: urn:org.apache.daffodil.layers.lineFolded
+- No DFDL Variables are read or set
+- There are two layer names for two variations:
+- - [lineFolded_IMF](#layer-name-linefolded_imf) - conforms to IETF RFC 2822 Internet Message Format (IMF)
+- - [lineFolded_iCalendar](#layer-name-linefolded_icalendar) - conforms to IETF RFC 5545 iCalendar
+
+### General Usage
+
+[NOTE: THis next paragraph could be used to explain length-limiting in general, not just for
+these layers]
+
+These layers would normally be used in a context of either an enclosing element
+which bounds its length (`dfdl:lengthKind 'explicit'`, `'prefixed'`, or `'pattern'`)
+or an enclosing utility layer that bounds its length (ex: `boundaryMark`).
+
+There is a limitation on the compatibility of line folding of data
+with adjacent parts of the format which also use line-endings.
+For example, line folding can interact badly with surrounding elements of `dfdl:lengthKind
+'pattern'` if the pattern is, for example `".*?\\r\\n(?!(?:\\t|\\ ))"` which is anything up to
+and including a CRLF not followed by a space or tab.
+The problem is that line folding
+converts isolated \n or \r into \r\n, and if this just happens to be followed by a
+non space/tab character this will have inserted an end-of-data in the middle of the
+data.
+
+### Layer Name: lineFolded_IMF
+
+For IMF, unfolding simply removes CRLFs if they are followed by a space or tab.
+
+When unparsing, the folding is more complex, as CRLFs can only be inserted before
+a space/tab that appears in the data. If the data has no spaces, then no
+folding is possible.
+If there are spaces/tabs, the one closest to (and before) position 78 is used unless it is
+followed by punctuation, in which case a prior space/tab (if it exists) is used.
+(This preference for spaces not followed by punctuation is optional, it is
+not required, but is preferred in the IMF RFC.)
+
+Note: folding is done by some systems in a manner that does not respect
+character boundaries - i.e., in utf-8, a multi-byte character sequence may be
+broken in the middle by insertion of a CRLF. Hence, unfolding initially treats
+the text as iso-8859-1, i.e., just bytes, and removes CRLFs, then subsequently
+re-interprets the bytes as the expected charset such as utf-8.
+
+IMF is supposed to be US-ASCII, but implementations have gone to 8-bit characters
+being preserved, so the above problem really can occur.
+
+IMF has a maximum line length of 998 characters per line excluding the CRLF.
+
+- _WARNING This check for 998 characters is not implemented (As of Daffodil version 3.8.0)._
+
+The layer should fail (cause a parse error) if a line longer than this is encountered
Review Comment:
*longer than 998 characters
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Documentation on layers feature in layers.md and updated javadoc for layers [daffodil-site]
Posted by "mbeckerle (via GitHub)" <gi...@apache.org>.
mbeckerle commented on code in PR #132:
URL: https://github.com/apache/daffodil-site/pull/132#discussion_r1619461543
##########
site/_includes/themes/apache/_navigation.html:
##########
@@ -21,6 +21,7 @@
<li><a href="/docs/dfdl/">DFDL Specification</a></li>
<li><a href="/unsupported/">Unsupported Features</a></li>
<li><a href="/faq/">Frequently Asked Questions</a></li>
+ <li><a href="/layers/">Layers (a DFDL Extension)</a></li>
Review Comment:
I updated the extensions page somewhat, and linked that from the navagation headings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Documentation on layers feature in layers.md and updated javadoc for layers [daffodil-site]
Posted by "mbeckerle (via GitHub)" <gi...@apache.org>.
mbeckerle commented on PR #132:
URL: https://github.com/apache/daffodil-site/pull/132#issuecomment-2135835321
I believe this is all set now. I will remove the update to the "latest" link for the javadoc/scaladoc so that this can be committed earlier than when 3.8.0 is actually released, since it is quite hard to review this sort of thing without seeing the formatted site.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Documentation on layers feature in layers.md and updated javadoc for layers [daffodil-site]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on code in PR #132:
URL: https://github.com/apache/daffodil-site/pull/132#discussion_r1619088194
##########
site/_includes/themes/apache/_navigation.html:
##########
@@ -21,6 +21,7 @@
<li><a href="/docs/dfdl/">DFDL Specification</a></li>
<li><a href="/unsupported/">Unsupported Features</a></li>
<li><a href="/faq/">Frequently Asked Questions</a></li>
+ <li><a href="/layers/">Layers (a DFDL Extension)</a></li>
Review Comment:
Note, we do have the https://daffodil.apache.org/dfdl-extensions/ page. This is not up to date at all (the `daf` prefix it mentions has been deprecated for a while), so I'm not sure it's worth updating now.
But what we should probably plan to do (not for this PR) is to create a separate page for each extension (e.g. `daffodil.apache.org/dfdl-extensions/layers` and each page goes into user-oriented details about that specific DFDL extension. Then the `dfdl-extensions` page can just become a list of short descriptions and links to each of those pages. And we would probably replace this link with a link to the the dfdl-extensions page.
Using jekyll we can pretty easily auto generate the dfdl-extensions page so adding a new extension page just means adding a new markdown file in a special directory, similar to what we do for releases.
##########
site/docs/latest:
##########
@@ -1 +1 @@
-3.7.0
\ No newline at end of file
+3.8.0
Review Comment:
We should revert this change and update this symlink when 3.8.0 is officially released.
##########
site/layers.md:
##########
@@ -0,0 +1,655 @@
+---
+description: Pluggable Extensions to Enable Algorithmic Transformations
+ in DFDL
+group: 'nav-right'
+layout: page
+title: 'Layers - Algorithmic
+ Extensions for DFDL'
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+### Table of Contents
+{:.no_toc}
+
+1. use ordered table of contents
+{:toc}
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+Layers are _byte oriented_.
+The smallest non-zero layer size is a single byte, and the length of
+all layers are in bytes.
+Throughout this discussion of layers, when length is discussed, the units of measure are always
+bytes.
+
+Layers can be composed together, that is, with one on top of another, and their composition is
+orthogonal (as in non-interacting).
+This is actually very common. For example, binary data being included into a textual data format
+can be both gzip compressed, and then base64 encoded, which requires 2 layers to describe.
+There is no limit to this depth.
+
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses
+multiple layers together.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- [base64_MIME](#base64-mime-layer)
+- [fourbyteswap](#byte-swapping-layers)
+- [twobyteswap](#byte-swapping-layers)
+- [gzip](#gzip-layer)
+- [lineFolded_IMF](#line-folded-layers)
+- [lineFolded_iCalendar](#line-folded-layers)
+
+Daffodil also includes two utility layers that are used in combination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- [boundaryMark](#boundaryMark-layer)
+- [fixedLength](#fixedLength-layer)
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+### Layer Kinds: Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html) class
+which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic
+Redundancy Check (CRC) over the data stream.
+
+The value of the checksum (or hash or CRC) is assigned to a DFDL variable as the result of the
+layer. This makes the value available for use by the DFDL schema that uses the checksum layer.
+When parsing, the value of this DFDL variable can then be compared to a checksum field in the
+data, and either an invalid data element or an parse-error can be created if the checksum in the
+data stream does not match the computed value.
+When unparsing, the value of this DFDL variable can be written to an element using the
+`dfdl:outputValueCalc` property.
+
+An example of a checksum layer plug-in is in the
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header checksum algorithm.
+
+Custom checksum layers are created by deriving an implementation class from the Daffodil API's
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.
+html)
+class, which is introduced in a later section.
+
+# Using Layers
+
+To use a layer you must know
+- the layer's namespace URI
+- the layer's name
+- the names of any layer parameter variables
+- the names of any layer result variables
+
+Each layer normally has a _layer schema file_ (.dfdl.xsd) which contains the DFDL variable's
+`dfdl:defineVariable` declarations.
+
+A `sequence` group encloses the scope, within the DFDL schema, of the layer.
+Often layers are used to describe checksums or decding/encoding on small regions within a
Review Comment:
s/deciding/decoding/
##########
site/layers.md:
##########
@@ -0,0 +1,655 @@
+---
+description: Pluggable Extensions to Enable Algorithmic Transformations
+ in DFDL
+group: 'nav-right'
+layout: page
+title: 'Layers - Algorithmic
+ Extensions for DFDL'
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+### Table of Contents
+{:.no_toc}
+
+1. use ordered table of contents
+{:toc}
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+Layers are _byte oriented_.
+The smallest non-zero layer size is a single byte, and the length of
+all layers are in bytes.
+Throughout this discussion of layers, when length is discussed, the units of measure are always
+bytes.
+
+Layers can be composed together, that is, with one on top of another, and their composition is
+orthogonal (as in non-interacting).
+This is actually very common. For example, binary data being included into a textual data format
+can be both gzip compressed, and then base64 encoded, which requires 2 layers to describe.
+There is no limit to this depth.
+
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses
+multiple layers together.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- [base64_MIME](#base64-mime-layer)
+- [fourbyteswap](#byte-swapping-layers)
+- [twobyteswap](#byte-swapping-layers)
+- [gzip](#gzip-layer)
+- [lineFolded_IMF](#line-folded-layers)
+- [lineFolded_iCalendar](#line-folded-layers)
+
+Daffodil also includes two utility layers that are used in combination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- [boundaryMark](#boundaryMark-layer)
+- [fixedLength](#fixedLength-layer)
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+### Layer Kinds: Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html) class
+which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic
+Redundancy Check (CRC) over the data stream.
+
+The value of the checksum (or hash or CRC) is assigned to a DFDL variable as the result of the
+layer. This makes the value available for use by the DFDL schema that uses the checksum layer.
+When parsing, the value of this DFDL variable can then be compared to a checksum field in the
+data, and either an invalid data element or an parse-error can be created if the checksum in the
+data stream does not match the computed value.
+When unparsing, the value of this DFDL variable can be written to an element using the
+`dfdl:outputValueCalc` property.
+
+An example of a checksum layer plug-in is in the
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header checksum algorithm.
+
+Custom checksum layers are created by deriving an implementation class from the Daffodil API's
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.
+html)
+class, which is introduced in a later section.
+
+# Using Layers
+
+To use a layer you must know
+- the layer's namespace URI
+- the layer's name
+- the names of any layer parameter variables
+- the names of any layer result variables
+
+Each layer normally has a _layer schema file_ (.dfdl.xsd) which contains the DFDL variable's
+`dfdl:defineVariable` declarations.
+
+A `sequence` group encloses the scope, within the DFDL schema, of the layer.
+Often layers are used to describe checksums or decding/encoding on small regions within a
+larger data format.
+
+The DFDL extension property `dfdlx:layer` declares that the data underlying a `sequence` is to
+use a layer. The value of the property is a QName identifying the namespace (via prefix) and
+name of the layer.
+
+A layer's namespace may define DFDL variables that are used to pass parameters to the layer, or
+to receive results back from the layer (usually for checksum layers).
+
+Lastly, there is the concept of _layer length limiting_.
+
+### Layer Length Limiting
+
+A layer can process an entire input file/stream.
+Such a layer is said to be _unlimited_.
+
+A layer can also have a specific data length built into it.
+The `IPv4Checksum` layer (in the aforementioned
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema) has a fixed length of 20 bytes.
+Such a layer is said to be _self limiting_.
+
+More commonly, a layer has a limited region within the data stream that it is supposed to describe.
+The restriction of the layer so that it only describes the expected part of the data is called
+_layer limiting_.
+
+There are these ways to do layer limiting, that differ in important ways:
+- using enclosing elements of specified length
+- using a fixedLength layer
+- binding a DFDL variable specific to a particular layer
+- using a boundaryMark layer
+
+#### Layer Limiting using Enclosing Elements of Specified Length
+
+If the `sequence` for a layer is the model group of the
+complex type of an element, and that element has specified length (meaning it has
+`dfdl:lengthKind` property of `explicit`, `implicit`, `prefixed` or `pattern`), then the length
+of the element limits the length of the layer within it.
+
+In this case the length of the layer is limited when parsing, but it is NOT limited when
+unparsing.
+
+#### Layer Limiting using the `fixedLength` Utility Layer
+
+The layer `fixedLength` is a utility layer intended for use in conjunction with other layers.
+It dictates the parse _and unparse_ length to be the value of the `layerLength` DFDL variable.
+Setting or binding this DFDL variable fixes the length of the `fixedLength` layer, and then
+any layer on top of that layer will be limited by the underlying fixed length.
+
+#### Layer Limiting by Binding a DFDL Variable
+
+Some layers will define a DFDL variable that must be set to a non-negative
+integer value to specify the layer length that the layer will use.
+All checksum layers built with the Daffodil API `ChecksumLayer` base class,
+use the DFDL `layerLength` variable to specify the length (in bytes) of the layer.
+
+Layers may specify restrictions on the minimum and maximum allowed values of these lengths,
+and passing an out-of-range value for the variable is a processing error.
+
+#### Layer Limiting using the `boundaryMark` Utility Layer
+
+The `boundaryMark` layer uses a variable named `boundaryMark` which provides a string parameter
+to the layer.
+This string is a delimiter of the layer data when parsing, and when unparsing the string is
+inserted after the (otherwise unbounded length) layer data.
+Unlike DFDL's normal delimiter properties( `dfdl:separator`, `dfdl:terminator`, and
+`dfdl:initiator`), the scanning for a boundary mark string does not use any escape scheme.
Review Comment:
Does the layer consume the delimiter? I believe it does, but it might be worth mentioned to make it more clear. Maybe that is define in teh boundmark layer section.
##########
site/layers.md:
##########
@@ -0,0 +1,655 @@
+---
+description: Pluggable Extensions to Enable Algorithmic Transformations
+ in DFDL
+group: 'nav-right'
+layout: page
+title: 'Layers - Algorithmic
+ Extensions for DFDL'
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+### Table of Contents
+{:.no_toc}
+
+1. use ordered table of contents
+{:toc}
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+Layers are _byte oriented_.
+The smallest non-zero layer size is a single byte, and the length of
+all layers are in bytes.
+Throughout this discussion of layers, when length is discussed, the units of measure are always
+bytes.
+
+Layers can be composed together, that is, with one on top of another, and their composition is
+orthogonal (as in non-interacting).
+This is actually very common. For example, binary data being included into a textual data format
+can be both gzip compressed, and then base64 encoded, which requires 2 layers to describe.
+There is no limit to this depth.
+
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses
+multiple layers together.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- [base64_MIME](#base64-mime-layer)
+- [fourbyteswap](#byte-swapping-layers)
+- [twobyteswap](#byte-swapping-layers)
+- [gzip](#gzip-layer)
+- [lineFolded_IMF](#line-folded-layers)
+- [lineFolded_iCalendar](#line-folded-layers)
+
+Daffodil also includes two utility layers that are used in combination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- [boundaryMark](#boundaryMark-layer)
+- [fixedLength](#fixedLength-layer)
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+### Layer Kinds: Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html) class
+which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic
+Redundancy Check (CRC) over the data stream.
+
+The value of the checksum (or hash or CRC) is assigned to a DFDL variable as the result of the
+layer. This makes the value available for use by the DFDL schema that uses the checksum layer.
+When parsing, the value of this DFDL variable can then be compared to a checksum field in the
+data, and either an invalid data element or an parse-error can be created if the checksum in the
+data stream does not match the computed value.
+When unparsing, the value of this DFDL variable can be written to an element using the
+`dfdl:outputValueCalc` property.
+
+An example of a checksum layer plug-in is in the
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header checksum algorithm.
+
+Custom checksum layers are created by deriving an implementation class from the Daffodil API's
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.
+html)
+class, which is introduced in a later section.
+
+# Using Layers
+
+To use a layer you must know
+- the layer's namespace URI
+- the layer's name
+- the names of any layer parameter variables
+- the names of any layer result variables
+
+Each layer normally has a _layer schema file_ (.dfdl.xsd) which contains the DFDL variable's
+`dfdl:defineVariable` declarations.
+
+A `sequence` group encloses the scope, within the DFDL schema, of the layer.
+Often layers are used to describe checksums or decding/encoding on small regions within a
+larger data format.
+
+The DFDL extension property `dfdlx:layer` declares that the data underlying a `sequence` is to
+use a layer. The value of the property is a QName identifying the namespace (via prefix) and
+name of the layer.
+
+A layer's namespace may define DFDL variables that are used to pass parameters to the layer, or
+to receive results back from the layer (usually for checksum layers).
+
+Lastly, there is the concept of _layer length limiting_.
+
+### Layer Length Limiting
+
+A layer can process an entire input file/stream.
+Such a layer is said to be _unlimited_.
+
+A layer can also have a specific data length built into it.
+The `IPv4Checksum` layer (in the aforementioned
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema) has a fixed length of 20 bytes.
+Such a layer is said to be _self limiting_.
+
+More commonly, a layer has a limited region within the data stream that it is supposed to describe.
+The restriction of the layer so that it only describes the expected part of the data is called
+_layer limiting_.
+
+There are these ways to do layer limiting, that differ in important ways:
+- using enclosing elements of specified length
+- using a fixedLength layer
+- binding a DFDL variable specific to a particular layer
+- using a boundaryMark layer
+
+#### Layer Limiting using Enclosing Elements of Specified Length
+
+If the `sequence` for a layer is the model group of the
+complex type of an element, and that element has specified length (meaning it has
+`dfdl:lengthKind` property of `explicit`, `implicit`, `prefixed` or `pattern`), then the length
+of the element limits the length of the layer within it.
+
+In this case the length of the layer is limited when parsing, but it is NOT limited when
+unparsing.
+
+#### Layer Limiting using the `fixedLength` Utility Layer
+
+The layer `fixedLength` is a utility layer intended for use in conjunction with other layers.
+It dictates the parse _and unparse_ length to be the value of the `layerLength` DFDL variable.
+Setting or binding this DFDL variable fixes the length of the `fixedLength` layer, and then
+any layer on top of that layer will be limited by the underlying fixed length.
+
+#### Layer Limiting by Binding a DFDL Variable
+
+Some layers will define a DFDL variable that must be set to a non-negative
+integer value to specify the layer length that the layer will use.
+All checksum layers built with the Daffodil API `ChecksumLayer` base class,
+use the DFDL `layerLength` variable to specify the length (in bytes) of the layer.
+
+Layers may specify restrictions on the minimum and maximum allowed values of these lengths,
+and passing an out-of-range value for the variable is a processing error.
+
+#### Layer Limiting using the `boundaryMark` Utility Layer
+
+The `boundaryMark` layer uses a variable named `boundaryMark` which provides a string parameter
+to the layer.
+This string is a delimiter of the layer data when parsing, and when unparsing the string is
+inserted after the (otherwise unbounded length) layer data.
+Unlike DFDL's normal delimiter properties( `dfdl:separator`, `dfdl:terminator`, and
+`dfdl:initiator`), the scanning for a boundary mark string does not use any escape scheme.
+
+
+## Example: Line Folding
+
+Consider the line folding layer, specifically the `lineFolded_IMF` layer,
+which is built-in to Daffodil.
+
+Line folding is a way of encoding textual data formats so that no line of text is longer than
+a limited line length.
+
+Consider this data :
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem
+ ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+This data has been _line folded_ at roughly 72 characters by inserting a CRLF
+before an existing space in the data.
+Each line ends with a CRLF (\r\n) and the second through fourth lines begin
+with a space as a way of indicating that they are extension lines.
+This data is supposed to be reassembled to form a long single-line string by removing
+all CRLF pairs.
+
+The result should be this single longer string which does not contain any line endings:
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+To achieve this we would use the `lineFolded_IMF` layer.
+This layer has a specific namespace which our DFDL schema will define a prefix `lf` for like this:
+```xml
+xmlns:lf="urn:org.apache.daffodil.layers.lineFolded"
+```
+Our DFDL schema will import the layer schema for the line-folded layer with this import:
+```xml
+<import namespace="urn:org.apache.daffodil.layers.lineFolded"
+ schemaLocation="/org/apache/daffodil/layers/xsd/lineFoldedLayer.dfdl.xsd"/>
+```
+Then the layer is incorporated into our DFDL schema like this:
+```xml
+<sequence dfdlx:layer="lf:lineFolded_IMF">
+ ... elements to be parsed from the unfolded layer data go here ...
+</sequence>
+```
+You can see that use of a layer is described using the `dfdx:layer` property, and the specific layer
+is identified by a QName using the previously defined namespace prefix.
+The scope of the layer is the duration of the sequence it appears on.
+The `dfdlx:layer` property can _only_ be used on an XSD `sequence`.
+
+The `lineFolded_IMF` layer does not define any DFDL variables in its namespace as it has no
+parameters and produces no results.
+As of Daffodil 3.8.0, the `import` statement above is optional for the
+line-folded layers as the DFDL schema file `lineFoldedLayer.dfdl.xsd` does not contain any
+definitions.
+In the future however, parameters may be added, so for uniformity all layers define a DFDL
+schema to be imported as part of using the layer.
+
+If the line-folding applies to the entire data stream (not uncommon for formats that use line
+folding), then what has been described is all one needs to describe the layer aspects of a data
+format.
+Other examples will show how the layer length can be limited to a sub-region of the data.
+
+More detailed documentation for the [Line Folded Layers](#LineFoldedLayers) is below.
Review Comment:
Broken link
##########
site/layers.md:
##########
@@ -0,0 +1,655 @@
+---
+description: Pluggable Extensions to Enable Algorithmic Transformations
+ in DFDL
+group: 'nav-right'
+layout: page
+title: 'Layers - Algorithmic
+ Extensions for DFDL'
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+### Table of Contents
+{:.no_toc}
+
+1. use ordered table of contents
+{:toc}
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+Layers are _byte oriented_.
+The smallest non-zero layer size is a single byte, and the length of
+all layers are in bytes.
+Throughout this discussion of layers, when length is discussed, the units of measure are always
+bytes.
+
+Layers can be composed together, that is, with one on top of another, and their composition is
+orthogonal (as in non-interacting).
+This is actually very common. For example, binary data being included into a textual data format
+can be both gzip compressed, and then base64 encoded, which requires 2 layers to describe.
+There is no limit to this depth.
+
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses
+multiple layers together.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- [base64_MIME](#base64-mime-layer)
+- [fourbyteswap](#byte-swapping-layers)
+- [twobyteswap](#byte-swapping-layers)
+- [gzip](#gzip-layer)
+- [lineFolded_IMF](#line-folded-layers)
+- [lineFolded_iCalendar](#line-folded-layers)
+
+Daffodil also includes two utility layers that are used in combination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- [boundaryMark](#boundaryMark-layer)
+- [fixedLength](#fixedLength-layer)
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+### Layer Kinds: Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html) class
+which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic
+Redundancy Check (CRC) over the data stream.
+
+The value of the checksum (or hash or CRC) is assigned to a DFDL variable as the result of the
+layer. This makes the value available for use by the DFDL schema that uses the checksum layer.
+When parsing, the value of this DFDL variable can then be compared to a checksum field in the
+data, and either an invalid data element or an parse-error can be created if the checksum in the
+data stream does not match the computed value.
+When unparsing, the value of this DFDL variable can be written to an element using the
+`dfdl:outputValueCalc` property.
+
+An example of a checksum layer plug-in is in the
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header checksum algorithm.
+
+Custom checksum layers are created by deriving an implementation class from the Daffodil API's
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.
+html)
+class, which is introduced in a later section.
+
+# Using Layers
+
+To use a layer you must know
+- the layer's namespace URI
+- the layer's name
+- the names of any layer parameter variables
+- the names of any layer result variables
+
+Each layer normally has a _layer schema file_ (.dfdl.xsd) which contains the DFDL variable's
+`dfdl:defineVariable` declarations.
+
+A `sequence` group encloses the scope, within the DFDL schema, of the layer.
+Often layers are used to describe checksums or decding/encoding on small regions within a
+larger data format.
+
+The DFDL extension property `dfdlx:layer` declares that the data underlying a `sequence` is to
+use a layer. The value of the property is a QName identifying the namespace (via prefix) and
+name of the layer.
+
+A layer's namespace may define DFDL variables that are used to pass parameters to the layer, or
+to receive results back from the layer (usually for checksum layers).
+
+Lastly, there is the concept of _layer length limiting_.
+
+### Layer Length Limiting
+
+A layer can process an entire input file/stream.
+Such a layer is said to be _unlimited_.
+
+A layer can also have a specific data length built into it.
+The `IPv4Checksum` layer (in the aforementioned
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema) has a fixed length of 20 bytes.
+Such a layer is said to be _self limiting_.
+
+More commonly, a layer has a limited region within the data stream that it is supposed to describe.
+The restriction of the layer so that it only describes the expected part of the data is called
+_layer limiting_.
+
+There are these ways to do layer limiting, that differ in important ways:
+- using enclosing elements of specified length
+- using a fixedLength layer
+- binding a DFDL variable specific to a particular layer
+- using a boundaryMark layer
+
+#### Layer Limiting using Enclosing Elements of Specified Length
+
+If the `sequence` for a layer is the model group of the
+complex type of an element, and that element has specified length (meaning it has
+`dfdl:lengthKind` property of `explicit`, `implicit`, `prefixed` or `pattern`), then the length
+of the element limits the length of the layer within it.
+
+In this case the length of the layer is limited when parsing, but it is NOT limited when
+unparsing.
+
+#### Layer Limiting using the `fixedLength` Utility Layer
+
+The layer `fixedLength` is a utility layer intended for use in conjunction with other layers.
+It dictates the parse _and unparse_ length to be the value of the `layerLength` DFDL variable.
+Setting or binding this DFDL variable fixes the length of the `fixedLength` layer, and then
+any layer on top of that layer will be limited by the underlying fixed length.
+
+#### Layer Limiting by Binding a DFDL Variable
+
+Some layers will define a DFDL variable that must be set to a non-negative
+integer value to specify the layer length that the layer will use.
+All checksum layers built with the Daffodil API `ChecksumLayer` base class,
+use the DFDL `layerLength` variable to specify the length (in bytes) of the layer.
+
+Layers may specify restrictions on the minimum and maximum allowed values of these lengths,
+and passing an out-of-range value for the variable is a processing error.
+
+#### Layer Limiting using the `boundaryMark` Utility Layer
+
+The `boundaryMark` layer uses a variable named `boundaryMark` which provides a string parameter
+to the layer.
+This string is a delimiter of the layer data when parsing, and when unparsing the string is
+inserted after the (otherwise unbounded length) layer data.
+Unlike DFDL's normal delimiter properties( `dfdl:separator`, `dfdl:terminator`, and
+`dfdl:initiator`), the scanning for a boundary mark string does not use any escape scheme.
+
+
+## Example: Line Folding
+
+Consider the line folding layer, specifically the `lineFolded_IMF` layer,
+which is built-in to Daffodil.
+
+Line folding is a way of encoding textual data formats so that no line of text is longer than
+a limited line length.
+
+Consider this data :
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem
+ ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+This data has been _line folded_ at roughly 72 characters by inserting a CRLF
+before an existing space in the data.
+Each line ends with a CRLF (\r\n) and the second through fourth lines begin
+with a space as a way of indicating that they are extension lines.
+This data is supposed to be reassembled to form a long single-line string by removing
+all CRLF pairs.
+
+The result should be this single longer string which does not contain any line endings:
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+To achieve this we would use the `lineFolded_IMF` layer.
+This layer has a specific namespace which our DFDL schema will define a prefix `lf` for like this:
+```xml
+xmlns:lf="urn:org.apache.daffodil.layers.lineFolded"
+```
+Our DFDL schema will import the layer schema for the line-folded layer with this import:
+```xml
+<import namespace="urn:org.apache.daffodil.layers.lineFolded"
+ schemaLocation="/org/apache/daffodil/layers/xsd/lineFoldedLayer.dfdl.xsd"/>
+```
+Then the layer is incorporated into our DFDL schema like this:
+```xml
+<sequence dfdlx:layer="lf:lineFolded_IMF">
+ ... elements to be parsed from the unfolded layer data go here ...
+</sequence>
+```
+You can see that use of a layer is described using the `dfdx:layer` property, and the specific layer
+is identified by a QName using the previously defined namespace prefix.
+The scope of the layer is the duration of the sequence it appears on.
+The `dfdlx:layer` property can _only_ be used on an XSD `sequence`.
+
+The `lineFolded_IMF` layer does not define any DFDL variables in its namespace as it has no
+parameters and produces no results.
+As of Daffodil 3.8.0, the `import` statement above is optional for the
+line-folded layers as the DFDL schema file `lineFoldedLayer.dfdl.xsd` does not contain any
+definitions.
+In the future however, parameters may be added, so for uniformity all layers define a DFDL
+schema to be imported as part of using the layer.
+
+If the line-folding applies to the entire data stream (not uncommon for formats that use line
+folding), then what has been described is all one needs to describe the layer aspects of a data
+format.
+Other examples will show how the layer length can be limited to a sub-region of the data.
+
+More detailed documentation for the [Line Folded Layers](#LineFoldedLayers) is below.
+
+## Example: Base64, GZip, and BoundaryMark Layers used Together
+
+In this example, the data consists of a preliminary string, a section of CSV-like data, and a
+final string element.
+The CSV-like data section in the middle is both gzipped, and then base64 encoded, with the
+boundary value mark of "`=_END_=`" indicating where the base64 part ends.
+
+The point of the preliminary string and the final string is to show that this sort of layering
+composes properly with surrounding data elements in the schema. Those strings could be anything
+at all, they're just strings to make this example simpler.
+
+The physical data string/stream looks like this:
+```text
+example of data before the base64 gzipped part
+H4sIAAAAAAAA/y3JQQqAIBCF4b1nmYG0iNxG+84wpqFhI4xuun0Gbd4P78tUG5xJ+t7J+xxg21dV
+79QiSHFBGjgh9oVB22XBYUQzqatErv36CjGwPJ/OOGg0Y1cOFUgegoPaLwa1VS9htd+UbgAAAA==
+=_END_=
+example of data after the base64 gzipped part
+```
+That base64 encoded part, if decoded and decompressed looks like this CSV-like data:
+```
+last,first,middle,DOB
+smith,robert,brandon,1988-03-24
+johnson,john,henry,1986-01-23
+jones,arya,cat,1986-02-19
+```
+
+The Infoset, presented as XML, that we will create for this entire data stream
+using Daffodil parsing looks like:
+```xml
+<ex:data>
+ <before>example of data before the base64 gzipped part</before>
+ <header><title>last</title><title>first</title><title>middle</title><title>DOB</title></header>
+ <record><item>smith</item><item>robert</item><item>brandon</item><item>1988-03-24</item></record>
+ <record><item>johnson</item><item>john</item><item>henry</item><item>1986-01-23</item></record>
+ <record><item>jones</item><item>arya</item><item>cat</item><item>1986-02-19</item></record>
+ <after>example of data after the base64 gzipped part</after>
+</ex:data>
+```
+
+With that characterization of the problem, let's look at the DFDL schema with the Daffodil
+layering properties to do the boundaryMark, base64 decode, and gzip decompression.
+(Note that each of these layers is described further in its own section below on this page.)
+
+First there is the start of the schema which introduces namespace prefix definitions for DFDL,
+for the DFDL extensions (dfdlx) prefix which is used for the `dfdlx:layer` property.
+
+This also sets up 3 namespace prefixes for the namespaces of the boundaryMark (bm), gzip (gz)
+and base64_MIME (b64) layers which will be imported just after this header.
+
+```xml
+<schema
+ xmlns="http://www.w3.org/2001/XMLSchema"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:fn="http://www.w3.org/2005/xpath-functions"
+ xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions"
+ xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:bm="urn:org.apache.daffodil.layers.boundaryMark"
+ xmlns:b64="urn:org.apache.daffodil.layers.base64_MIME"
+ xmlns:gz="urn:org.apache.daffodil.layers.gzip"
+ xmlns:ex="http://example.com"
+ targetNamespace="http://example.com">
+```
+Next we import the Daffodil built in general format which provides most DFDL properties with
+sensible starting values, and import the namespaces of our boundaryMark, gzip, and base64_MIME
+layers, and lastly, set the default format properties for use by this schema file:
+```xml
+<include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+
+<import namespace="urn:org.apache.daffodil.layers.boundaryMark"
+ schemaLocation="/org/apache/daffodil/layers/xsd/boundaryMarkLayer.dfdl.xsd"/>
+<import namespace="urn:org.apache.daffodil.layers.gzip"
+ schemaLocation="/org/apache/daffodil/layers/xsd/gzipLayer.dfdl.xsd"/>
+<import namespace="urn:org.apache.daffodil.layers.base64_MIME"
+ schemaLocation="/org/apache/daffodil/layers/xsd/base64_MIMELayer.dfdl.xsd"/>
+
+<annotation><appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:format ref="ex:GeneralFormat" representation="binary"/>
+</appinfo></annotation>
+```
+The root element of the schema is `data`, and this element has type `dataType`.
+(A separate named complex type definition for the root element is a
+recommended style for DFDL schemas as it allows someone else to reuse the schema without
+getting involved in element namespace prefix complexities.)
+The `dataType` just shows us the `before` and `after` elements, and references a group
+named `layeredDataGroup` for the layered part in between.
+```xml
+<element name="data" type="ex:dataType"/>
+
+<complexType name="dataType">
+ <sequence dfdl:separator="%NL;">
+ <element name="before" type="string" dfdl:lengthKind="delimited"/>
+ <group ref="ex:layeredDataGroup"/>
+ <element name="after" type="string" dfdl:lengthKind="delimited"/>
+ </sequence>
+</complexType>
+```
+Now we see the definition of the layering.
+All 3 layers are declared in this named group.
+The lines here are numbered so that the text that follows can refer to them:
+```xml
+1 <group name="layeredDataGroup">
+2 <sequence dfdlx:layer="bm:boundaryMark">
+3 <annotation>
+4 <appinfo source="http://www.ogf.org/dfdl/">
+5 <dfdl:newVariableInstance ref="bm:layerEncoding" defaultValue="iso-8859-1"/>
+6 <dfdl:newVariableInstance ref="bm:boundaryMark" defaultValue="=_END_="/>
+7 </appinfo>
+8 </annotation>
+9 <sequence dfdlx:layer="b64:base64_MIME">
+10 <sequence dfdlx:layer="gz:gzip">
+11 <group ref="ex:dataFieldsGroup"/>
+12 </sequence>
+13 </sequence>
+14 </sequence>
+15 </group>
+```
+In the above, at line 2 we see the sequence declares the boundaryMark layer.
+This layer requires 2 parameters to be passed by binding DFDL variables, which occurs on
+lines 5 and 6.
+Line 5 binds the `layerEncoding` variable and note the prefix since DFDL variable names can be
+in a specific namespace.
+Line 6 binds the `boundaryMark` variable to the string "`=_END_=`".
+Note that the variable and the layer both are named `boundaryMark`, but this is just coincidence.
+Layers do not have to have parameters that match their name, nor even any parameters at all.
+The boundary mark is the string which marks the end of the data based on scanning for that ending
+mark.
+
+The data region now being isolated, the `boundaryMark` layer contains just one sequence, and line
+9 which itself declares that the data is base64 encoded by specifying the base64_MIME layer.
+This layer has no parameters, so there are no `dfdl:newVariableInstance` statements to bind any
+variables.
+The region of data that is base64 encoded is then described by the sequence at line 10 which
+declares that the data is compressed by the `gzip` algorithm.
+
+Finally, at line 11 we reference a group which describes what the data format of the underlying
+CSV-like data.
+That group's definition is regular DFDL without any of the complexity of layers.
+This is useful for testing, as data can be parsed using this group (in the complexType of an
+element) to ensure it works properly without involving the complexity of layers.
+This group definition is the last thing in the schema:
+```xml
+<group name="dataFieldsGroup">
+ <sequence dfdl:separator="%NL;" dfdl:separatorPosition="postfix">
+ <element name="header" minOccurs="0" dfdl:occursCountKind="implicit">
+ <complexType>
+ <sequence dfdl:separator=",">
+ <element name="title" type="xs:string" maxOccurs="unbounded" dfdl:lengthKind="delimited"/>
+ </sequence>
+ </complexType>
+ </element>
+ <element name="record" maxOccurs="unbounded">
+ <complexType>
+ <sequence dfdl:separator=",">
+ <element name="item" type="xs:string" maxOccurs="unbounded"
+ dfdl:lengthKind="delimited"
+ dfdl:occursCount="{ fn:count(../../header/title) }"
+ dfdl:occursCountKind="expression"/>
+ </sequence>
+ </complexType>
+ </element>
+ </sequence>
+</group>
+
+</schema>
+```
+The above schema works both to parse, but also to unparse this data.
+
+----
+
+# Using Custom Plug-In Layers
+
+A custom plug-in layer is used in the same manner as the built-in Daffodil layers with just a few
+additional details:
+1. The custom layer will have a DFDL schema that declares any DFDL variables it uses, and
+ defines any data formats needed to facilitate use of the layer.
+ This schema will need to be included/imported.
+ Normally this schema would be a small single-file schema and it can be loaded from the
+ classpath.
+2. The custom layer's compiled Java/Scala code is dynamically loaded using the
+ Java Service Provider Interface (SPI).
+ Hence, they must be compiled into Jar files with specific META-INF metadata enclosed, and
+ these jars must appear on the Java CLASSPATH so that they can be found and loaded.
+
+The layer API is defined via Java clases and interfaces to enable writing of custom layers in
+either Java or Scala.
+(One of the built-in layers (Gzip) is written in Java now, by way of proving
+that one can write a Layer in Java.)
+
+Transformer layer classes are derived from the
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html)
+base class.
+
+Checksum layer classes are derived from the
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.html)
+base class.
+
+Further details on how to define custom plug-in layers is in the Javadoc for the
+[Layer API](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/package-summary.html)
+
+----
+
+# Daffodil Built-In Layer Documentation
Review Comment:
Should be `##`
##########
site/layers.md:
##########
@@ -0,0 +1,655 @@
+---
+description: Pluggable Extensions to Enable Algorithmic Transformations
+ in DFDL
+group: 'nav-right'
+layout: page
+title: 'Layers - Algorithmic
+ Extensions for DFDL'
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+### Table of Contents
+{:.no_toc}
+
+1. use ordered table of contents
+{:toc}
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+Layers are _byte oriented_.
+The smallest non-zero layer size is a single byte, and the length of
+all layers are in bytes.
+Throughout this discussion of layers, when length is discussed, the units of measure are always
+bytes.
+
+Layers can be composed together, that is, with one on top of another, and their composition is
+orthogonal (as in non-interacting).
+This is actually very common. For example, binary data being included into a textual data format
+can be both gzip compressed, and then base64 encoded, which requires 2 layers to describe.
+There is no limit to this depth.
+
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses
+multiple layers together.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- [base64_MIME](#base64-mime-layer)
+- [fourbyteswap](#byte-swapping-layers)
+- [twobyteswap](#byte-swapping-layers)
+- [gzip](#gzip-layer)
+- [lineFolded_IMF](#line-folded-layers)
+- [lineFolded_iCalendar](#line-folded-layers)
+
+Daffodil also includes two utility layers that are used in combination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- [boundaryMark](#boundaryMark-layer)
+- [fixedLength](#fixedLength-layer)
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+### Layer Kinds: Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html) class
+which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic
+Redundancy Check (CRC) over the data stream.
+
+The value of the checksum (or hash or CRC) is assigned to a DFDL variable as the result of the
+layer. This makes the value available for use by the DFDL schema that uses the checksum layer.
+When parsing, the value of this DFDL variable can then be compared to a checksum field in the
+data, and either an invalid data element or an parse-error can be created if the checksum in the
+data stream does not match the computed value.
+When unparsing, the value of this DFDL variable can be written to an element using the
+`dfdl:outputValueCalc` property.
+
+An example of a checksum layer plug-in is in the
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header checksum algorithm.
+
+Custom checksum layers are created by deriving an implementation class from the Daffodil API's
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.
+html)
+class, which is introduced in a later section.
+
+# Using Layers
+
+To use a layer you must know
+- the layer's namespace URI
+- the layer's name
+- the names of any layer parameter variables
+- the names of any layer result variables
+
+Each layer normally has a _layer schema file_ (.dfdl.xsd) which contains the DFDL variable's
+`dfdl:defineVariable` declarations.
+
+A `sequence` group encloses the scope, within the DFDL schema, of the layer.
+Often layers are used to describe checksums or decding/encoding on small regions within a
+larger data format.
+
+The DFDL extension property `dfdlx:layer` declares that the data underlying a `sequence` is to
+use a layer. The value of the property is a QName identifying the namespace (via prefix) and
+name of the layer.
+
+A layer's namespace may define DFDL variables that are used to pass parameters to the layer, or
+to receive results back from the layer (usually for checksum layers).
+
+Lastly, there is the concept of _layer length limiting_.
+
+### Layer Length Limiting
+
+A layer can process an entire input file/stream.
+Such a layer is said to be _unlimited_.
+
+A layer can also have a specific data length built into it.
+The `IPv4Checksum` layer (in the aforementioned
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema) has a fixed length of 20 bytes.
+Such a layer is said to be _self limiting_.
+
+More commonly, a layer has a limited region within the data stream that it is supposed to describe.
+The restriction of the layer so that it only describes the expected part of the data is called
+_layer limiting_.
+
+There are these ways to do layer limiting, that differ in important ways:
+- using enclosing elements of specified length
+- using a fixedLength layer
+- binding a DFDL variable specific to a particular layer
+- using a boundaryMark layer
+
+#### Layer Limiting using Enclosing Elements of Specified Length
+
+If the `sequence` for a layer is the model group of the
+complex type of an element, and that element has specified length (meaning it has
+`dfdl:lengthKind` property of `explicit`, `implicit`, `prefixed` or `pattern`), then the length
+of the element limits the length of the layer within it.
+
+In this case the length of the layer is limited when parsing, but it is NOT limited when
+unparsing.
+
+#### Layer Limiting using the `fixedLength` Utility Layer
+
+The layer `fixedLength` is a utility layer intended for use in conjunction with other layers.
+It dictates the parse _and unparse_ length to be the value of the `layerLength` DFDL variable.
+Setting or binding this DFDL variable fixes the length of the `fixedLength` layer, and then
+any layer on top of that layer will be limited by the underlying fixed length.
+
+#### Layer Limiting by Binding a DFDL Variable
+
+Some layers will define a DFDL variable that must be set to a non-negative
+integer value to specify the layer length that the layer will use.
+All checksum layers built with the Daffodil API `ChecksumLayer` base class,
+use the DFDL `layerLength` variable to specify the length (in bytes) of the layer.
+
+Layers may specify restrictions on the minimum and maximum allowed values of these lengths,
+and passing an out-of-range value for the variable is a processing error.
+
+#### Layer Limiting using the `boundaryMark` Utility Layer
+
+The `boundaryMark` layer uses a variable named `boundaryMark` which provides a string parameter
+to the layer.
+This string is a delimiter of the layer data when parsing, and when unparsing the string is
+inserted after the (otherwise unbounded length) layer data.
+Unlike DFDL's normal delimiter properties( `dfdl:separator`, `dfdl:terminator`, and
+`dfdl:initiator`), the scanning for a boundary mark string does not use any escape scheme.
+
+
+## Example: Line Folding
+
+Consider the line folding layer, specifically the `lineFolded_IMF` layer,
+which is built-in to Daffodil.
+
+Line folding is a way of encoding textual data formats so that no line of text is longer than
+a limited line length.
+
+Consider this data :
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem
+ ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+This data has been _line folded_ at roughly 72 characters by inserting a CRLF
+before an existing space in the data.
+Each line ends with a CRLF (\r\n) and the second through fourth lines begin
+with a space as a way of indicating that they are extension lines.
+This data is supposed to be reassembled to form a long single-line string by removing
+all CRLF pairs.
+
+The result should be this single longer string which does not contain any line endings:
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+To achieve this we would use the `lineFolded_IMF` layer.
+This layer has a specific namespace which our DFDL schema will define a prefix `lf` for like this:
+```xml
+xmlns:lf="urn:org.apache.daffodil.layers.lineFolded"
+```
+Our DFDL schema will import the layer schema for the line-folded layer with this import:
+```xml
+<import namespace="urn:org.apache.daffodil.layers.lineFolded"
+ schemaLocation="/org/apache/daffodil/layers/xsd/lineFoldedLayer.dfdl.xsd"/>
+```
+Then the layer is incorporated into our DFDL schema like this:
+```xml
+<sequence dfdlx:layer="lf:lineFolded_IMF">
+ ... elements to be parsed from the unfolded layer data go here ...
+</sequence>
+```
+You can see that use of a layer is described using the `dfdx:layer` property, and the specific layer
+is identified by a QName using the previously defined namespace prefix.
+The scope of the layer is the duration of the sequence it appears on.
+The `dfdlx:layer` property can _only_ be used on an XSD `sequence`.
+
+The `lineFolded_IMF` layer does not define any DFDL variables in its namespace as it has no
+parameters and produces no results.
+As of Daffodil 3.8.0, the `import` statement above is optional for the
+line-folded layers as the DFDL schema file `lineFoldedLayer.dfdl.xsd` does not contain any
+definitions.
+In the future however, parameters may be added, so for uniformity all layers define a DFDL
+schema to be imported as part of using the layer.
+
+If the line-folding applies to the entire data stream (not uncommon for formats that use line
+folding), then what has been described is all one needs to describe the layer aspects of a data
+format.
+Other examples will show how the layer length can be limited to a sub-region of the data.
+
+More detailed documentation for the [Line Folded Layers](#LineFoldedLayers) is below.
+
+## Example: Base64, GZip, and BoundaryMark Layers used Together
+
+In this example, the data consists of a preliminary string, a section of CSV-like data, and a
+final string element.
+The CSV-like data section in the middle is both gzipped, and then base64 encoded, with the
+boundary value mark of "`=_END_=`" indicating where the base64 part ends.
+
+The point of the preliminary string and the final string is to show that this sort of layering
+composes properly with surrounding data elements in the schema. Those strings could be anything
+at all, they're just strings to make this example simpler.
+
+The physical data string/stream looks like this:
+```text
+example of data before the base64 gzipped part
+H4sIAAAAAAAA/y3JQQqAIBCF4b1nmYG0iNxG+84wpqFhI4xuun0Gbd4P78tUG5xJ+t7J+xxg21dV
+79QiSHFBGjgh9oVB22XBYUQzqatErv36CjGwPJ/OOGg0Y1cOFUgegoPaLwa1VS9htd+UbgAAAA==
+=_END_=
+example of data after the base64 gzipped part
+```
+That base64 encoded part, if decoded and decompressed looks like this CSV-like data:
+```
+last,first,middle,DOB
+smith,robert,brandon,1988-03-24
+johnson,john,henry,1986-01-23
+jones,arya,cat,1986-02-19
+```
+
+The Infoset, presented as XML, that we will create for this entire data stream
+using Daffodil parsing looks like:
+```xml
+<ex:data>
+ <before>example of data before the base64 gzipped part</before>
+ <header><title>last</title><title>first</title><title>middle</title><title>DOB</title></header>
+ <record><item>smith</item><item>robert</item><item>brandon</item><item>1988-03-24</item></record>
+ <record><item>johnson</item><item>john</item><item>henry</item><item>1986-01-23</item></record>
+ <record><item>jones</item><item>arya</item><item>cat</item><item>1986-02-19</item></record>
+ <after>example of data after the base64 gzipped part</after>
+</ex:data>
+```
+
+With that characterization of the problem, let's look at the DFDL schema with the Daffodil
+layering properties to do the boundaryMark, base64 decode, and gzip decompression.
+(Note that each of these layers is described further in its own section below on this page.)
+
+First there is the start of the schema which introduces namespace prefix definitions for DFDL,
+for the DFDL extensions (dfdlx) prefix which is used for the `dfdlx:layer` property.
+
+This also sets up 3 namespace prefixes for the namespaces of the boundaryMark (bm), gzip (gz)
+and base64_MIME (b64) layers which will be imported just after this header.
+
+```xml
+<schema
+ xmlns="http://www.w3.org/2001/XMLSchema"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:fn="http://www.w3.org/2005/xpath-functions"
+ xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions"
+ xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:bm="urn:org.apache.daffodil.layers.boundaryMark"
+ xmlns:b64="urn:org.apache.daffodil.layers.base64_MIME"
+ xmlns:gz="urn:org.apache.daffodil.layers.gzip"
+ xmlns:ex="http://example.com"
+ targetNamespace="http://example.com">
+```
+Next we import the Daffodil built in general format which provides most DFDL properties with
+sensible starting values, and import the namespaces of our boundaryMark, gzip, and base64_MIME
+layers, and lastly, set the default format properties for use by this schema file:
+```xml
+<include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+
+<import namespace="urn:org.apache.daffodil.layers.boundaryMark"
+ schemaLocation="/org/apache/daffodil/layers/xsd/boundaryMarkLayer.dfdl.xsd"/>
+<import namespace="urn:org.apache.daffodil.layers.gzip"
+ schemaLocation="/org/apache/daffodil/layers/xsd/gzipLayer.dfdl.xsd"/>
+<import namespace="urn:org.apache.daffodil.layers.base64_MIME"
+ schemaLocation="/org/apache/daffodil/layers/xsd/base64_MIMELayer.dfdl.xsd"/>
+
+<annotation><appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:format ref="ex:GeneralFormat" representation="binary"/>
+</appinfo></annotation>
+```
+The root element of the schema is `data`, and this element has type `dataType`.
+(A separate named complex type definition for the root element is a
+recommended style for DFDL schemas as it allows someone else to reuse the schema without
+getting involved in element namespace prefix complexities.)
+The `dataType` just shows us the `before` and `after` elements, and references a group
+named `layeredDataGroup` for the layered part in between.
+```xml
+<element name="data" type="ex:dataType"/>
+
+<complexType name="dataType">
+ <sequence dfdl:separator="%NL;">
+ <element name="before" type="string" dfdl:lengthKind="delimited"/>
+ <group ref="ex:layeredDataGroup"/>
+ <element name="after" type="string" dfdl:lengthKind="delimited"/>
+ </sequence>
+</complexType>
+```
+Now we see the definition of the layering.
+All 3 layers are declared in this named group.
+The lines here are numbered so that the text that follows can refer to them:
+```xml
+1 <group name="layeredDataGroup">
+2 <sequence dfdlx:layer="bm:boundaryMark">
+3 <annotation>
+4 <appinfo source="http://www.ogf.org/dfdl/">
+5 <dfdl:newVariableInstance ref="bm:layerEncoding" defaultValue="iso-8859-1"/>
+6 <dfdl:newVariableInstance ref="bm:boundaryMark" defaultValue="=_END_="/>
+7 </appinfo>
+8 </annotation>
+9 <sequence dfdlx:layer="b64:base64_MIME">
+10 <sequence dfdlx:layer="gz:gzip">
+11 <group ref="ex:dataFieldsGroup"/>
+12 </sequence>
+13 </sequence>
+14 </sequence>
+15 </group>
+```
+In the above, at line 2 we see the sequence declares the boundaryMark layer.
+This layer requires 2 parameters to be passed by binding DFDL variables, which occurs on
+lines 5 and 6.
+Line 5 binds the `layerEncoding` variable and note the prefix since DFDL variable names can be
+in a specific namespace.
+Line 6 binds the `boundaryMark` variable to the string "`=_END_=`".
+Note that the variable and the layer both are named `boundaryMark`, but this is just coincidence.
+Layers do not have to have parameters that match their name, nor even any parameters at all.
+The boundary mark is the string which marks the end of the data based on scanning for that ending
+mark.
+
+The data region now being isolated, the `boundaryMark` layer contains just one sequence, and line
+9 which itself declares that the data is base64 encoded by specifying the base64_MIME layer.
+This layer has no parameters, so there are no `dfdl:newVariableInstance` statements to bind any
+variables.
+The region of data that is base64 encoded is then described by the sequence at line 10 which
+declares that the data is compressed by the `gzip` algorithm.
+
+Finally, at line 11 we reference a group which describes what the data format of the underlying
+CSV-like data.
+That group's definition is regular DFDL without any of the complexity of layers.
+This is useful for testing, as data can be parsed using this group (in the complexType of an
+element) to ensure it works properly without involving the complexity of layers.
+This group definition is the last thing in the schema:
+```xml
+<group name="dataFieldsGroup">
+ <sequence dfdl:separator="%NL;" dfdl:separatorPosition="postfix">
+ <element name="header" minOccurs="0" dfdl:occursCountKind="implicit">
+ <complexType>
+ <sequence dfdl:separator=",">
+ <element name="title" type="xs:string" maxOccurs="unbounded" dfdl:lengthKind="delimited"/>
+ </sequence>
+ </complexType>
+ </element>
+ <element name="record" maxOccurs="unbounded">
+ <complexType>
+ <sequence dfdl:separator=",">
+ <element name="item" type="xs:string" maxOccurs="unbounded"
+ dfdl:lengthKind="delimited"
+ dfdl:occursCount="{ fn:count(../../header/title) }"
+ dfdl:occursCountKind="expression"/>
+ </sequence>
+ </complexType>
+ </element>
+ </sequence>
+</group>
+
+</schema>
+```
+The above schema works both to parse, but also to unparse this data.
+
+----
+
+# Using Custom Plug-In Layers
+
+A custom plug-in layer is used in the same manner as the built-in Daffodil layers with just a few
+additional details:
+1. The custom layer will have a DFDL schema that declares any DFDL variables it uses, and
+ defines any data formats needed to facilitate use of the layer.
+ This schema will need to be included/imported.
+ Normally this schema would be a small single-file schema and it can be loaded from the
+ classpath.
+2. The custom layer's compiled Java/Scala code is dynamically loaded using the
+ Java Service Provider Interface (SPI).
+ Hence, they must be compiled into Jar files with specific META-INF metadata enclosed, and
+ these jars must appear on the Java CLASSPATH so that they can be found and loaded.
+
+The layer API is defined via Java clases and interfaces to enable writing of custom layers in
+either Java or Scala.
+(One of the built-in layers (Gzip) is written in Java now, by way of proving
+that one can write a Layer in Java.)
+
+Transformer layer classes are derived from the
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html)
+base class.
+
+Checksum layer classes are derived from the
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.html)
+base class.
+
+Further details on how to define custom plug-in layers is in the Javadoc for the
+[Layer API](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/package-summary.html)
+
+----
+
+# Daffodil Built-In Layer Documentation
+
+Each of the layers built-in to the Daffodil implementation are documented in a section below
+which gives the name, namespace, variables, and some usage notes.
+
+----
+
+## Base64 MIME Layer
+
+- Name: base64_MIME
+- Namespace URI: urn:org.apache.daffodil.layers.base64_MIME
+- No DFDL variables are read or written.
+
+This uses the standard `java.util.Base64` classes.
+
+----
+
+## BoundaryMark Layer
+
+- Name: boundaryMark
+- Namespace URI: urn:org.apache.daffodil.layers.boundaryMark
+- Reads variables
Review Comment:
Suggest changing to an "Parameter Variable" and "Result Variables" bullets, and they are just "None" for layers that don't have them. Just to make this more consistent per layer description.
##########
site/layers.md:
##########
@@ -0,0 +1,655 @@
+---
+description: Pluggable Extensions to Enable Algorithmic Transformations
+ in DFDL
+group: 'nav-right'
+layout: page
+title: 'Layers - Algorithmic
+ Extensions for DFDL'
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+### Table of Contents
+{:.no_toc}
+
+1. use ordered table of contents
+{:toc}
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+Layers are _byte oriented_.
+The smallest non-zero layer size is a single byte, and the length of
+all layers are in bytes.
+Throughout this discussion of layers, when length is discussed, the units of measure are always
+bytes.
+
+Layers can be composed together, that is, with one on top of another, and their composition is
+orthogonal (as in non-interacting).
+This is actually very common. For example, binary data being included into a textual data format
+can be both gzip compressed, and then base64 encoded, which requires 2 layers to describe.
+There is no limit to this depth.
+
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses
+multiple layers together.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- [base64_MIME](#base64-mime-layer)
+- [fourbyteswap](#byte-swapping-layers)
+- [twobyteswap](#byte-swapping-layers)
+- [gzip](#gzip-layer)
+- [lineFolded_IMF](#line-folded-layers)
+- [lineFolded_iCalendar](#line-folded-layers)
+
+Daffodil also includes two utility layers that are used in combination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- [boundaryMark](#boundaryMark-layer)
+- [fixedLength](#fixedLength-layer)
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+### Layer Kinds: Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html) class
+which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic
+Redundancy Check (CRC) over the data stream.
+
+The value of the checksum (or hash or CRC) is assigned to a DFDL variable as the result of the
+layer. This makes the value available for use by the DFDL schema that uses the checksum layer.
+When parsing, the value of this DFDL variable can then be compared to a checksum field in the
+data, and either an invalid data element or an parse-error can be created if the checksum in the
+data stream does not match the computed value.
+When unparsing, the value of this DFDL variable can be written to an element using the
+`dfdl:outputValueCalc` property.
+
+An example of a checksum layer plug-in is in the
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header checksum algorithm.
+
+Custom checksum layers are created by deriving an implementation class from the Daffodil API's
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.
+html)
+class, which is introduced in a later section.
+
+# Using Layers
+
+To use a layer you must know
+- the layer's namespace URI
+- the layer's name
+- the names of any layer parameter variables
+- the names of any layer result variables
+
+Each layer normally has a _layer schema file_ (.dfdl.xsd) which contains the DFDL variable's
+`dfdl:defineVariable` declarations.
+
+A `sequence` group encloses the scope, within the DFDL schema, of the layer.
+Often layers are used to describe checksums or decding/encoding on small regions within a
+larger data format.
+
+The DFDL extension property `dfdlx:layer` declares that the data underlying a `sequence` is to
+use a layer. The value of the property is a QName identifying the namespace (via prefix) and
+name of the layer.
+
+A layer's namespace may define DFDL variables that are used to pass parameters to the layer, or
+to receive results back from the layer (usually for checksum layers).
+
+Lastly, there is the concept of _layer length limiting_.
+
+### Layer Length Limiting
+
+A layer can process an entire input file/stream.
+Such a layer is said to be _unlimited_.
+
+A layer can also have a specific data length built into it.
+The `IPv4Checksum` layer (in the aforementioned
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema) has a fixed length of 20 bytes.
+Such a layer is said to be _self limiting_.
+
+More commonly, a layer has a limited region within the data stream that it is supposed to describe.
+The restriction of the layer so that it only describes the expected part of the data is called
+_layer limiting_.
+
+There are these ways to do layer limiting, that differ in important ways:
+- using enclosing elements of specified length
+- using a fixedLength layer
+- binding a DFDL variable specific to a particular layer
+- using a boundaryMark layer
+
+#### Layer Limiting using Enclosing Elements of Specified Length
+
+If the `sequence` for a layer is the model group of the
+complex type of an element, and that element has specified length (meaning it has
+`dfdl:lengthKind` property of `explicit`, `implicit`, `prefixed` or `pattern`), then the length
+of the element limits the length of the layer within it.
+
+In this case the length of the layer is limited when parsing, but it is NOT limited when
+unparsing.
+
+#### Layer Limiting using the `fixedLength` Utility Layer
+
+The layer `fixedLength` is a utility layer intended for use in conjunction with other layers.
+It dictates the parse _and unparse_ length to be the value of the `layerLength` DFDL variable.
+Setting or binding this DFDL variable fixes the length of the `fixedLength` layer, and then
+any layer on top of that layer will be limited by the underlying fixed length.
+
+#### Layer Limiting by Binding a DFDL Variable
+
+Some layers will define a DFDL variable that must be set to a non-negative
+integer value to specify the layer length that the layer will use.
+All checksum layers built with the Daffodil API `ChecksumLayer` base class,
+use the DFDL `layerLength` variable to specify the length (in bytes) of the layer.
+
+Layers may specify restrictions on the minimum and maximum allowed values of these lengths,
+and passing an out-of-range value for the variable is a processing error.
+
+#### Layer Limiting using the `boundaryMark` Utility Layer
Review Comment:
It feels like maybe the sections layer limiting by boundaryMark or fixedLength layers may duplicate information in the respective built-in layer documentation below. Maybe instead you combine this into a single section that is something like `Layer Limiting using Utility Layers" and just say something generic like
> some layers do not decode or encode data but only restrict the length for subsequent layers, for example by using a length layer variable or delimiter. See the boundaryMark and fixedLength layers for examples.
And then link to those to built-in layers.
##########
site/layers.md:
##########
@@ -0,0 +1,655 @@
+---
+description: Pluggable Extensions to Enable Algorithmic Transformations
+ in DFDL
+group: 'nav-right'
+layout: page
+title: 'Layers - Algorithmic
+ Extensions for DFDL'
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+### Table of Contents
+{:.no_toc}
+
+1. use ordered table of contents
+{:toc}
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+Layers are _byte oriented_.
+The smallest non-zero layer size is a single byte, and the length of
+all layers are in bytes.
+Throughout this discussion of layers, when length is discussed, the units of measure are always
+bytes.
+
+Layers can be composed together, that is, with one on top of another, and their composition is
+orthogonal (as in non-interacting).
+This is actually very common. For example, binary data being included into a textual data format
+can be both gzip compressed, and then base64 encoded, which requires 2 layers to describe.
+There is no limit to this depth.
+
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses
+multiple layers together.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- [base64_MIME](#base64-mime-layer)
+- [fourbyteswap](#byte-swapping-layers)
+- [twobyteswap](#byte-swapping-layers)
+- [gzip](#gzip-layer)
+- [lineFolded_IMF](#line-folded-layers)
+- [lineFolded_iCalendar](#line-folded-layers)
+
+Daffodil also includes two utility layers that are used in combination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- [boundaryMark](#boundaryMark-layer)
+- [fixedLength](#fixedLength-layer)
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+### Layer Kinds: Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html) class
Review Comment:
Suggest linking `/docs/latest/...` instead of `../docs/latest/...`. Using absolute paths means we can move this page (like suggested we might want to do in the future) without having to worry about breaking links.
Also note, if we revert the `latest` docs symlink change then this link will be broken until we do an actual release. I think that's probably fine for now since the next release should be relatively soon, and a lot of this documentation isn't even relevant until that release is out.
##########
site/layers.md:
##########
@@ -0,0 +1,655 @@
+---
+description: Pluggable Extensions to Enable Algorithmic Transformations
+ in DFDL
+group: 'nav-right'
+layout: page
+title: 'Layers - Algorithmic
+ Extensions for DFDL'
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+### Table of Contents
+{:.no_toc}
+
+1. use ordered table of contents
+{:toc}
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+Layers are _byte oriented_.
+The smallest non-zero layer size is a single byte, and the length of
+all layers are in bytes.
+Throughout this discussion of layers, when length is discussed, the units of measure are always
+bytes.
+
+Layers can be composed together, that is, with one on top of another, and their composition is
+orthogonal (as in non-interacting).
+This is actually very common. For example, binary data being included into a textual data format
+can be both gzip compressed, and then base64 encoded, which requires 2 layers to describe.
+There is no limit to this depth.
+
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses
+multiple layers together.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- [base64_MIME](#base64-mime-layer)
+- [fourbyteswap](#byte-swapping-layers)
+- [twobyteswap](#byte-swapping-layers)
+- [gzip](#gzip-layer)
+- [lineFolded_IMF](#line-folded-layers)
+- [lineFolded_iCalendar](#line-folded-layers)
+
+Daffodil also includes two utility layers that are used in combination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- [boundaryMark](#boundaryMark-layer)
+- [fixedLength](#fixedLength-layer)
Review Comment:
These links are broken, I think all lowercase will fix it.
##########
site/layers.md:
##########
@@ -0,0 +1,655 @@
+---
+description: Pluggable Extensions to Enable Algorithmic Transformations
+ in DFDL
+group: 'nav-right'
+layout: page
+title: 'Layers - Algorithmic
+ Extensions for DFDL'
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+### Table of Contents
+{:.no_toc}
+
+1. use ordered table of contents
+{:toc}
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+Layers are _byte oriented_.
+The smallest non-zero layer size is a single byte, and the length of
+all layers are in bytes.
+Throughout this discussion of layers, when length is discussed, the units of measure are always
+bytes.
+
+Layers can be composed together, that is, with one on top of another, and their composition is
+orthogonal (as in non-interacting).
+This is actually very common. For example, binary data being included into a textual data format
+can be both gzip compressed, and then base64 encoded, which requires 2 layers to describe.
+There is no limit to this depth.
+
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses
+multiple layers together.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- [base64_MIME](#base64-mime-layer)
+- [fourbyteswap](#byte-swapping-layers)
+- [twobyteswap](#byte-swapping-layers)
+- [gzip](#gzip-layer)
+- [lineFolded_IMF](#line-folded-layers)
+- [lineFolded_iCalendar](#line-folded-layers)
+
+Daffodil also includes two utility layers that are used in combination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- [boundaryMark](#boundaryMark-layer)
+- [fixedLength](#fixedLength-layer)
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+### Layer Kinds: Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html) class
+which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic
+Redundancy Check (CRC) over the data stream.
+
+The value of the checksum (or hash or CRC) is assigned to a DFDL variable as the result of the
+layer. This makes the value available for use by the DFDL schema that uses the checksum layer.
+When parsing, the value of this DFDL variable can then be compared to a checksum field in the
+data, and either an invalid data element or an parse-error can be created if the checksum in the
+data stream does not match the computed value.
+When unparsing, the value of this DFDL variable can be written to an element using the
+`dfdl:outputValueCalc` property.
+
+An example of a checksum layer plug-in is in the
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header checksum algorithm.
+
+Custom checksum layers are created by deriving an implementation class from the Daffodil API's
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.
+html)
+class, which is introduced in a later section.
+
+# Using Layers
+
+To use a layer you must know
+- the layer's namespace URI
+- the layer's name
+- the names of any layer parameter variables
+- the names of any layer result variables
+
+Each layer normally has a _layer schema file_ (.dfdl.xsd) which contains the DFDL variable's
+`dfdl:defineVariable` declarations.
+
+A `sequence` group encloses the scope, within the DFDL schema, of the layer.
+Often layers are used to describe checksums or decding/encoding on small regions within a
+larger data format.
+
+The DFDL extension property `dfdlx:layer` declares that the data underlying a `sequence` is to
+use a layer. The value of the property is a QName identifying the namespace (via prefix) and
+name of the layer.
+
+A layer's namespace may define DFDL variables that are used to pass parameters to the layer, or
+to receive results back from the layer (usually for checksum layers).
+
+Lastly, there is the concept of _layer length limiting_.
+
+### Layer Length Limiting
+
+A layer can process an entire input file/stream.
+Such a layer is said to be _unlimited_.
+
+A layer can also have a specific data length built into it.
+The `IPv4Checksum` layer (in the aforementioned
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema) has a fixed length of 20 bytes.
+Such a layer is said to be _self limiting_.
+
+More commonly, a layer has a limited region within the data stream that it is supposed to describe.
+The restriction of the layer so that it only describes the expected part of the data is called
+_layer limiting_.
+
+There are these ways to do layer limiting, that differ in important ways:
+- using enclosing elements of specified length
+- using a fixedLength layer
+- binding a DFDL variable specific to a particular layer
+- using a boundaryMark layer
+
+#### Layer Limiting using Enclosing Elements of Specified Length
+
+If the `sequence` for a layer is the model group of the
+complex type of an element, and that element has specified length (meaning it has
+`dfdl:lengthKind` property of `explicit`, `implicit`, `prefixed` or `pattern`), then the length
+of the element limits the length of the layer within it.
+
+In this case the length of the layer is limited when parsing, but it is NOT limited when
+unparsing.
+
+#### Layer Limiting using the `fixedLength` Utility Layer
+
+The layer `fixedLength` is a utility layer intended for use in conjunction with other layers.
+It dictates the parse _and unparse_ length to be the value of the `layerLength` DFDL variable.
+Setting or binding this DFDL variable fixes the length of the `fixedLength` layer, and then
+any layer on top of that layer will be limited by the underlying fixed length.
+
+#### Layer Limiting by Binding a DFDL Variable
+
+Some layers will define a DFDL variable that must be set to a non-negative
+integer value to specify the layer length that the layer will use.
+All checksum layers built with the Daffodil API `ChecksumLayer` base class,
+use the DFDL `layerLength` variable to specify the length (in bytes) of the layer.
+
+Layers may specify restrictions on the minimum and maximum allowed values of these lengths,
+and passing an out-of-range value for the variable is a processing error.
+
+#### Layer Limiting using the `boundaryMark` Utility Layer
+
+The `boundaryMark` layer uses a variable named `boundaryMark` which provides a string parameter
+to the layer.
+This string is a delimiter of the layer data when parsing, and when unparsing the string is
+inserted after the (otherwise unbounded length) layer data.
+Unlike DFDL's normal delimiter properties( `dfdl:separator`, `dfdl:terminator`, and
+`dfdl:initiator`), the scanning for a boundary mark string does not use any escape scheme.
+
+
+## Example: Line Folding
+
+Consider the line folding layer, specifically the `lineFolded_IMF` layer,
+which is built-in to Daffodil.
+
+Line folding is a way of encoding textual data formats so that no line of text is longer than
+a limited line length.
+
+Consider this data :
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem
+ ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+This data has been _line folded_ at roughly 72 characters by inserting a CRLF
+before an existing space in the data.
+Each line ends with a CRLF (\r\n) and the second through fourth lines begin
+with a space as a way of indicating that they are extension lines.
+This data is supposed to be reassembled to form a long single-line string by removing
+all CRLF pairs.
+
+The result should be this single longer string which does not contain any line endings:
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+To achieve this we would use the `lineFolded_IMF` layer.
+This layer has a specific namespace which our DFDL schema will define a prefix `lf` for like this:
+```xml
+xmlns:lf="urn:org.apache.daffodil.layers.lineFolded"
Review Comment:
Suggest change to
```
```xml
<schema xmlns:lf="urn:org.apache.daffodil.layers.lineFolded" ...>
```
```
so that it adds syntax highlighting.
##########
site/layers.md:
##########
@@ -0,0 +1,655 @@
+---
+description: Pluggable Extensions to Enable Algorithmic Transformations
+ in DFDL
+group: 'nav-right'
+layout: page
+title: 'Layers - Algorithmic
+ Extensions for DFDL'
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+### Table of Contents
+{:.no_toc}
+
+1. use ordered table of contents
+{:toc}
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+Layers are _byte oriented_.
+The smallest non-zero layer size is a single byte, and the length of
+all layers are in bytes.
+Throughout this discussion of layers, when length is discussed, the units of measure are always
+bytes.
+
+Layers can be composed together, that is, with one on top of another, and their composition is
+orthogonal (as in non-interacting).
+This is actually very common. For example, binary data being included into a textual data format
+can be both gzip compressed, and then base64 encoded, which requires 2 layers to describe.
+There is no limit to this depth.
+
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses
+multiple layers together.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- [base64_MIME](#base64-mime-layer)
+- [fourbyteswap](#byte-swapping-layers)
+- [twobyteswap](#byte-swapping-layers)
+- [gzip](#gzip-layer)
+- [lineFolded_IMF](#line-folded-layers)
+- [lineFolded_iCalendar](#line-folded-layers)
+
+Daffodil also includes two utility layers that are used in combination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- [boundaryMark](#boundaryMark-layer)
+- [fixedLength](#fixedLength-layer)
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+### Layer Kinds: Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html) class
+which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic
+Redundancy Check (CRC) over the data stream.
+
+The value of the checksum (or hash or CRC) is assigned to a DFDL variable as the result of the
+layer. This makes the value available for use by the DFDL schema that uses the checksum layer.
+When parsing, the value of this DFDL variable can then be compared to a checksum field in the
+data, and either an invalid data element or an parse-error can be created if the checksum in the
+data stream does not match the computed value.
+When unparsing, the value of this DFDL variable can be written to an element using the
+`dfdl:outputValueCalc` property.
+
+An example of a checksum layer plug-in is in the
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header checksum algorithm.
+
+Custom checksum layers are created by deriving an implementation class from the Daffodil API's
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.
+html)
+class, which is introduced in a later section.
+
+# Using Layers
+
+To use a layer you must know
+- the layer's namespace URI
+- the layer's name
+- the names of any layer parameter variables
+- the names of any layer result variables
+
+Each layer normally has a _layer schema file_ (.dfdl.xsd) which contains the DFDL variable's
+`dfdl:defineVariable` declarations.
+
+A `sequence` group encloses the scope, within the DFDL schema, of the layer.
+Often layers are used to describe checksums or decding/encoding on small regions within a
+larger data format.
+
+The DFDL extension property `dfdlx:layer` declares that the data underlying a `sequence` is to
+use a layer. The value of the property is a QName identifying the namespace (via prefix) and
+name of the layer.
+
+A layer's namespace may define DFDL variables that are used to pass parameters to the layer, or
+to receive results back from the layer (usually for checksum layers).
+
+Lastly, there is the concept of _layer length limiting_.
+
+### Layer Length Limiting
+
+A layer can process an entire input file/stream.
+Such a layer is said to be _unlimited_.
+
+A layer can also have a specific data length built into it.
+The `IPv4Checksum` layer (in the aforementioned
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema) has a fixed length of 20 bytes.
+Such a layer is said to be _self limiting_.
+
+More commonly, a layer has a limited region within the data stream that it is supposed to describe.
+The restriction of the layer so that it only describes the expected part of the data is called
+_layer limiting_.
+
+There are these ways to do layer limiting, that differ in important ways:
+- using enclosing elements of specified length
+- using a fixedLength layer
+- binding a DFDL variable specific to a particular layer
+- using a boundaryMark layer
+
+#### Layer Limiting using Enclosing Elements of Specified Length
+
+If the `sequence` for a layer is the model group of the
+complex type of an element, and that element has specified length (meaning it has
+`dfdl:lengthKind` property of `explicit`, `implicit`, `prefixed` or `pattern`), then the length
+of the element limits the length of the layer within it.
+
+In this case the length of the layer is limited when parsing, but it is NOT limited when
+unparsing.
+
+#### Layer Limiting using the `fixedLength` Utility Layer
+
+The layer `fixedLength` is a utility layer intended for use in conjunction with other layers.
+It dictates the parse _and unparse_ length to be the value of the `layerLength` DFDL variable.
+Setting or binding this DFDL variable fixes the length of the `fixedLength` layer, and then
+any layer on top of that layer will be limited by the underlying fixed length.
Review Comment:
Should we link to the fixedLength layer in this section?
##########
site/layers.md:
##########
@@ -0,0 +1,655 @@
+---
+description: Pluggable Extensions to Enable Algorithmic Transformations
+ in DFDL
+group: 'nav-right'
+layout: page
+title: 'Layers - Algorithmic
+ Extensions for DFDL'
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+### Table of Contents
+{:.no_toc}
+
+1. use ordered table of contents
+{:toc}
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+Layers are _byte oriented_.
+The smallest non-zero layer size is a single byte, and the length of
+all layers are in bytes.
+Throughout this discussion of layers, when length is discussed, the units of measure are always
+bytes.
+
+Layers can be composed together, that is, with one on top of another, and their composition is
+orthogonal (as in non-interacting).
+This is actually very common. For example, binary data being included into a textual data format
+can be both gzip compressed, and then base64 encoded, which requires 2 layers to describe.
+There is no limit to this depth.
+
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses
+multiple layers together.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- [base64_MIME](#base64-mime-layer)
+- [fourbyteswap](#byte-swapping-layers)
+- [twobyteswap](#byte-swapping-layers)
+- [gzip](#gzip-layer)
+- [lineFolded_IMF](#line-folded-layers)
+- [lineFolded_iCalendar](#line-folded-layers)
+
+Daffodil also includes two utility layers that are used in combination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- [boundaryMark](#boundaryMark-layer)
+- [fixedLength](#fixedLength-layer)
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+### Layer Kinds: Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html) class
+which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
Review Comment:
`kind of layer` singular
##########
site/layers.md:
##########
@@ -0,0 +1,655 @@
+---
+description: Pluggable Extensions to Enable Algorithmic Transformations
+ in DFDL
+group: 'nav-right'
+layout: page
+title: 'Layers - Algorithmic
+ Extensions for DFDL'
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+### Table of Contents
+{:.no_toc}
+
+1. use ordered table of contents
+{:toc}
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+Layers are _byte oriented_.
+The smallest non-zero layer size is a single byte, and the length of
+all layers are in bytes.
+Throughout this discussion of layers, when length is discussed, the units of measure are always
+bytes.
+
+Layers can be composed together, that is, with one on top of another, and their composition is
+orthogonal (as in non-interacting).
+This is actually very common. For example, binary data being included into a textual data format
+can be both gzip compressed, and then base64 encoded, which requires 2 layers to describe.
+There is no limit to this depth.
+
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses
+multiple layers together.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- [base64_MIME](#base64-mime-layer)
+- [fourbyteswap](#byte-swapping-layers)
+- [twobyteswap](#byte-swapping-layers)
+- [gzip](#gzip-layer)
+- [lineFolded_IMF](#line-folded-layers)
+- [lineFolded_iCalendar](#line-folded-layers)
+
+Daffodil also includes two utility layers that are used in combination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- [boundaryMark](#boundaryMark-layer)
+- [fixedLength](#fixedLength-layer)
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+### Layer Kinds: Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html) class
+which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic
+Redundancy Check (CRC) over the data stream.
+
+The value of the checksum (or hash or CRC) is assigned to a DFDL variable as the result of the
+layer. This makes the value available for use by the DFDL schema that uses the checksum layer.
+When parsing, the value of this DFDL variable can then be compared to a checksum field in the
+data, and either an invalid data element or an parse-error can be created if the checksum in the
+data stream does not match the computed value.
+When unparsing, the value of this DFDL variable can be written to an element using the
+`dfdl:outputValueCalc` property.
+
+An example of a checksum layer plug-in is in the
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header checksum algorithm.
+
+Custom checksum layers are created by deriving an implementation class from the Daffodil API's
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.
+html)
+class, which is introduced in a later section.
+
+# Using Layers
+
+To use a layer you must know
+- the layer's namespace URI
+- the layer's name
+- the names of any layer parameter variables
+- the names of any layer result variables
+
+Each layer normally has a _layer schema file_ (.dfdl.xsd) which contains the DFDL variable's
+`dfdl:defineVariable` declarations.
+
+A `sequence` group encloses the scope, within the DFDL schema, of the layer.
+Often layers are used to describe checksums or decding/encoding on small regions within a
+larger data format.
+
+The DFDL extension property `dfdlx:layer` declares that the data underlying a `sequence` is to
+use a layer. The value of the property is a QName identifying the namespace (via prefix) and
+name of the layer.
+
+A layer's namespace may define DFDL variables that are used to pass parameters to the layer, or
+to receive results back from the layer (usually for checksum layers).
+
+Lastly, there is the concept of _layer length limiting_.
+
+### Layer Length Limiting
+
+A layer can process an entire input file/stream.
+Such a layer is said to be _unlimited_.
+
+A layer can also have a specific data length built into it.
+The `IPv4Checksum` layer (in the aforementioned
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema) has a fixed length of 20 bytes.
+Such a layer is said to be _self limiting_.
+
+More commonly, a layer has a limited region within the data stream that it is supposed to describe.
+The restriction of the layer so that it only describes the expected part of the data is called
+_layer limiting_.
+
+There are these ways to do layer limiting, that differ in important ways:
+- using enclosing elements of specified length
+- using a fixedLength layer
+- binding a DFDL variable specific to a particular layer
+- using a boundaryMark layer
+
+#### Layer Limiting using Enclosing Elements of Specified Length
+
+If the `sequence` for a layer is the model group of the
+complex type of an element, and that element has specified length (meaning it has
+`dfdl:lengthKind` property of `explicit`, `implicit`, `prefixed` or `pattern`), then the length
Review Comment:
Does a complex type with implicit length limit anything? Should that be removed from this list?
##########
site/layers.md:
##########
@@ -0,0 +1,655 @@
+---
+description: Pluggable Extensions to Enable Algorithmic Transformations
+ in DFDL
+group: 'nav-right'
+layout: page
+title: 'Layers - Algorithmic
+ Extensions for DFDL'
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+### Table of Contents
+{:.no_toc}
+
+1. use ordered table of contents
+{:toc}
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+Layers are _byte oriented_.
+The smallest non-zero layer size is a single byte, and the length of
+all layers are in bytes.
+Throughout this discussion of layers, when length is discussed, the units of measure are always
+bytes.
+
+Layers can be composed together, that is, with one on top of another, and their composition is
+orthogonal (as in non-interacting).
+This is actually very common. For example, binary data being included into a textual data format
+can be both gzip compressed, and then base64 encoded, which requires 2 layers to describe.
+There is no limit to this depth.
+
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses
+multiple layers together.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- [base64_MIME](#base64-mime-layer)
+- [fourbyteswap](#byte-swapping-layers)
+- [twobyteswap](#byte-swapping-layers)
+- [gzip](#gzip-layer)
+- [lineFolded_IMF](#line-folded-layers)
+- [lineFolded_iCalendar](#line-folded-layers)
+
+Daffodil also includes two utility layers that are used in combination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- [boundaryMark](#boundaryMark-layer)
+- [fixedLength](#fixedLength-layer)
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+### Layer Kinds: Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html) class
+which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic
+Redundancy Check (CRC) over the data stream.
+
+The value of the checksum (or hash or CRC) is assigned to a DFDL variable as the result of the
+layer. This makes the value available for use by the DFDL schema that uses the checksum layer.
+When parsing, the value of this DFDL variable can then be compared to a checksum field in the
+data, and either an invalid data element or an parse-error can be created if the checksum in the
+data stream does not match the computed value.
+When unparsing, the value of this DFDL variable can be written to an element using the
+`dfdl:outputValueCalc` property.
+
+An example of a checksum layer plug-in is in the
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header checksum algorithm.
+
+Custom checksum layers are created by deriving an implementation class from the Daffodil API's
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.
+html)
+class, which is introduced in a later section.
+
+# Using Layers
+
+To use a layer you must know
+- the layer's namespace URI
+- the layer's name
+- the names of any layer parameter variables
+- the names of any layer result variables
+
+Each layer normally has a _layer schema file_ (.dfdl.xsd) which contains the DFDL variable's
+`dfdl:defineVariable` declarations.
+
+A `sequence` group encloses the scope, within the DFDL schema, of the layer.
+Often layers are used to describe checksums or decding/encoding on small regions within a
+larger data format.
+
+The DFDL extension property `dfdlx:layer` declares that the data underlying a `sequence` is to
+use a layer. The value of the property is a QName identifying the namespace (via prefix) and
+name of the layer.
+
+A layer's namespace may define DFDL variables that are used to pass parameters to the layer, or
+to receive results back from the layer (usually for checksum layers).
+
+Lastly, there is the concept of _layer length limiting_.
+
+### Layer Length Limiting
+
+A layer can process an entire input file/stream.
+Such a layer is said to be _unlimited_.
+
+A layer can also have a specific data length built into it.
+The `IPv4Checksum` layer (in the aforementioned
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema) has a fixed length of 20 bytes.
+Such a layer is said to be _self limiting_.
+
+More commonly, a layer has a limited region within the data stream that it is supposed to describe.
+The restriction of the layer so that it only describes the expected part of the data is called
+_layer limiting_.
+
+There are these ways to do layer limiting, that differ in important ways:
+- using enclosing elements of specified length
+- using a fixedLength layer
+- binding a DFDL variable specific to a particular layer
+- using a boundaryMark layer
+
+#### Layer Limiting using Enclosing Elements of Specified Length
+
+If the `sequence` for a layer is the model group of the
+complex type of an element, and that element has specified length (meaning it has
+`dfdl:lengthKind` property of `explicit`, `implicit`, `prefixed` or `pattern`), then the length
+of the element limits the length of the layer within it.
+
+In this case the length of the layer is limited when parsing, but it is NOT limited when
+unparsing.
+
+#### Layer Limiting using the `fixedLength` Utility Layer
+
+The layer `fixedLength` is a utility layer intended for use in conjunction with other layers.
+It dictates the parse _and unparse_ length to be the value of the `layerLength` DFDL variable.
+Setting or binding this DFDL variable fixes the length of the `fixedLength` layer, and then
+any layer on top of that layer will be limited by the underlying fixed length.
+
+#### Layer Limiting by Binding a DFDL Variable
+
+Some layers will define a DFDL variable that must be set to a non-negative
+integer value to specify the layer length that the layer will use.
+All checksum layers built with the Daffodil API `ChecksumLayer` base class,
+use the DFDL `layerLength` variable to specify the length (in bytes) of the layer.
+
+Layers may specify restrictions on the minimum and maximum allowed values of these lengths,
+and passing an out-of-range value for the variable is a processing error.
+
+#### Layer Limiting using the `boundaryMark` Utility Layer
+
+The `boundaryMark` layer uses a variable named `boundaryMark` which provides a string parameter
+to the layer.
+This string is a delimiter of the layer data when parsing, and when unparsing the string is
+inserted after the (otherwise unbounded length) layer data.
+Unlike DFDL's normal delimiter properties( `dfdl:separator`, `dfdl:terminator`, and
+`dfdl:initiator`), the scanning for a boundary mark string does not use any escape scheme.
+
Review Comment:
Should we link to the bondary mark layer section?
##########
site/layers.md:
##########
@@ -0,0 +1,655 @@
+---
+description: Pluggable Extensions to Enable Algorithmic Transformations
+ in DFDL
+group: 'nav-right'
+layout: page
+title: 'Layers - Algorithmic
+ Extensions for DFDL'
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+### Table of Contents
+{:.no_toc}
+
+1. use ordered table of contents
+{:toc}
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+Layers are _byte oriented_.
+The smallest non-zero layer size is a single byte, and the length of
+all layers are in bytes.
+Throughout this discussion of layers, when length is discussed, the units of measure are always
+bytes.
+
+Layers can be composed together, that is, with one on top of another, and their composition is
+orthogonal (as in non-interacting).
+This is actually very common. For example, binary data being included into a textual data format
+can be both gzip compressed, and then base64 encoded, which requires 2 layers to describe.
+There is no limit to this depth.
+
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses
+multiple layers together.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- [base64_MIME](#base64-mime-layer)
+- [fourbyteswap](#byte-swapping-layers)
+- [twobyteswap](#byte-swapping-layers)
+- [gzip](#gzip-layer)
+- [lineFolded_IMF](#line-folded-layers)
+- [lineFolded_iCalendar](#line-folded-layers)
+
+Daffodil also includes two utility layers that are used in combination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- [boundaryMark](#boundaryMark-layer)
+- [fixedLength](#fixedLength-layer)
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+### Layer Kinds: Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html) class
+which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic
+Redundancy Check (CRC) over the data stream.
+
+The value of the checksum (or hash or CRC) is assigned to a DFDL variable as the result of the
+layer. This makes the value available for use by the DFDL schema that uses the checksum layer.
+When parsing, the value of this DFDL variable can then be compared to a checksum field in the
+data, and either an invalid data element or an parse-error can be created if the checksum in the
+data stream does not match the computed value.
+When unparsing, the value of this DFDL variable can be written to an element using the
+`dfdl:outputValueCalc` property.
+
+An example of a checksum layer plug-in is in the
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header checksum algorithm.
+
+Custom checksum layers are created by deriving an implementation class from the Daffodil API's
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.
+html)
+class, which is introduced in a later section.
+
+# Using Layers
Review Comment:
I think the convention this website styling uses is all headings in markdown files want to be at least `##` headings. This is kindof annoying not being able to using single `#` sections, so we may want to fix that at some point, but that should be a separate PR.
But it means this wants to be a `##` section and the later sections want to add an additional `#`.
##########
site/layers.md:
##########
@@ -0,0 +1,655 @@
+---
+description: Pluggable Extensions to Enable Algorithmic Transformations
+ in DFDL
+group: 'nav-right'
+layout: page
+title: 'Layers - Algorithmic
+ Extensions for DFDL'
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+### Table of Contents
+{:.no_toc}
+
+1. use ordered table of contents
+{:toc}
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+Layers are _byte oriented_.
+The smallest non-zero layer size is a single byte, and the length of
+all layers are in bytes.
+Throughout this discussion of layers, when length is discussed, the units of measure are always
+bytes.
+
+Layers can be composed together, that is, with one on top of another, and their composition is
+orthogonal (as in non-interacting).
+This is actually very common. For example, binary data being included into a textual data format
+can be both gzip compressed, and then base64 encoded, which requires 2 layers to describe.
+There is no limit to this depth.
+
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses
+multiple layers together.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- [base64_MIME](#base64-mime-layer)
+- [fourbyteswap](#byte-swapping-layers)
+- [twobyteswap](#byte-swapping-layers)
+- [gzip](#gzip-layer)
+- [lineFolded_IMF](#line-folded-layers)
+- [lineFolded_iCalendar](#line-folded-layers)
+
+Daffodil also includes two utility layers that are used in combination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- [boundaryMark](#boundaryMark-layer)
+- [fixedLength](#fixedLength-layer)
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+### Layer Kinds: Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html) class
+which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic
+Redundancy Check (CRC) over the data stream.
+
+The value of the checksum (or hash or CRC) is assigned to a DFDL variable as the result of the
+layer. This makes the value available for use by the DFDL schema that uses the checksum layer.
+When parsing, the value of this DFDL variable can then be compared to a checksum field in the
+data, and either an invalid data element or an parse-error can be created if the checksum in the
+data stream does not match the computed value.
+When unparsing, the value of this DFDL variable can be written to an element using the
+`dfdl:outputValueCalc` property.
+
+An example of a checksum layer plug-in is in the
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header checksum algorithm.
+
+Custom checksum layers are created by deriving an implementation class from the Daffodil API's
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.
+html)
+class, which is introduced in a later section.
+
+# Using Layers
+
+To use a layer you must know
+- the layer's namespace URI
+- the layer's name
+- the names of any layer parameter variables
+- the names of any layer result variables
+
+Each layer normally has a _layer schema file_ (.dfdl.xsd) which contains the DFDL variable's
+`dfdl:defineVariable` declarations.
+
+A `sequence` group encloses the scope, within the DFDL schema, of the layer.
+Often layers are used to describe checksums or decding/encoding on small regions within a
+larger data format.
+
+The DFDL extension property `dfdlx:layer` declares that the data underlying a `sequence` is to
+use a layer. The value of the property is a QName identifying the namespace (via prefix) and
+name of the layer.
+
+A layer's namespace may define DFDL variables that are used to pass parameters to the layer, or
+to receive results back from the layer (usually for checksum layers).
+
+Lastly, there is the concept of _layer length limiting_.
+
+### Layer Length Limiting
+
+A layer can process an entire input file/stream.
+Such a layer is said to be _unlimited_.
+
+A layer can also have a specific data length built into it.
+The `IPv4Checksum` layer (in the aforementioned
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema) has a fixed length of 20 bytes.
+Such a layer is said to be _self limiting_.
+
+More commonly, a layer has a limited region within the data stream that it is supposed to describe.
+The restriction of the layer so that it only describes the expected part of the data is called
+_layer limiting_.
+
+There are these ways to do layer limiting, that differ in important ways:
+- using enclosing elements of specified length
+- using a fixedLength layer
+- binding a DFDL variable specific to a particular layer
+- using a boundaryMark layer
+
+#### Layer Limiting using Enclosing Elements of Specified Length
+
+If the `sequence` for a layer is the model group of the
+complex type of an element, and that element has specified length (meaning it has
+`dfdl:lengthKind` property of `explicit`, `implicit`, `prefixed` or `pattern`), then the length
+of the element limits the length of the layer within it.
+
+In this case the length of the layer is limited when parsing, but it is NOT limited when
+unparsing.
+
+#### Layer Limiting using the `fixedLength` Utility Layer
+
+The layer `fixedLength` is a utility layer intended for use in conjunction with other layers.
+It dictates the parse _and unparse_ length to be the value of the `layerLength` DFDL variable.
+Setting or binding this DFDL variable fixes the length of the `fixedLength` layer, and then
+any layer on top of that layer will be limited by the underlying fixed length.
+
+#### Layer Limiting by Binding a DFDL Variable
+
+Some layers will define a DFDL variable that must be set to a non-negative
+integer value to specify the layer length that the layer will use.
+All checksum layers built with the Daffodil API `ChecksumLayer` base class,
+use the DFDL `layerLength` variable to specify the length (in bytes) of the layer.
+
+Layers may specify restrictions on the minimum and maximum allowed values of these lengths,
+and passing an out-of-range value for the variable is a processing error.
+
+#### Layer Limiting using the `boundaryMark` Utility Layer
+
+The `boundaryMark` layer uses a variable named `boundaryMark` which provides a string parameter
+to the layer.
+This string is a delimiter of the layer data when parsing, and when unparsing the string is
+inserted after the (otherwise unbounded length) layer data.
+Unlike DFDL's normal delimiter properties( `dfdl:separator`, `dfdl:terminator`, and
+`dfdl:initiator`), the scanning for a boundary mark string does not use any escape scheme.
+
+
+## Example: Line Folding
+
+Consider the line folding layer, specifically the `lineFolded_IMF` layer,
+which is built-in to Daffodil.
+
+Line folding is a way of encoding textual data formats so that no line of text is longer than
+a limited line length.
+
+Consider this data :
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem
+ ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+This data has been _line folded_ at roughly 72 characters by inserting a CRLF
+before an existing space in the data.
+Each line ends with a CRLF (\r\n) and the second through fourth lines begin
+with a space as a way of indicating that they are extension lines.
+This data is supposed to be reassembled to form a long single-line string by removing
+all CRLF pairs.
+
+The result should be this single longer string which does not contain any line endings:
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+To achieve this we would use the `lineFolded_IMF` layer.
+This layer has a specific namespace which our DFDL schema will define a prefix `lf` for like this:
+```xml
+xmlns:lf="urn:org.apache.daffodil.layers.lineFolded"
+```
+Our DFDL schema will import the layer schema for the line-folded layer with this import:
+```xml
+<import namespace="urn:org.apache.daffodil.layers.lineFolded"
+ schemaLocation="/org/apache/daffodil/layers/xsd/lineFoldedLayer.dfdl.xsd"/>
+```
+Then the layer is incorporated into our DFDL schema like this:
+```xml
+<sequence dfdlx:layer="lf:lineFolded_IMF">
+ ... elements to be parsed from the unfolded layer data go here ...
+</sequence>
+```
+You can see that use of a layer is described using the `dfdx:layer` property, and the specific layer
+is identified by a QName using the previously defined namespace prefix.
+The scope of the layer is the duration of the sequence it appears on.
+The `dfdlx:layer` property can _only_ be used on an XSD `sequence`.
+
+The `lineFolded_IMF` layer does not define any DFDL variables in its namespace as it has no
+parameters and produces no results.
+As of Daffodil 3.8.0, the `import` statement above is optional for the
+line-folded layers as the DFDL schema file `lineFoldedLayer.dfdl.xsd` does not contain any
+definitions.
+In the future however, parameters may be added, so for uniformity all layers define a DFDL
+schema to be imported as part of using the layer.
+
+If the line-folding applies to the entire data stream (not uncommon for formats that use line
+folding), then what has been described is all one needs to describe the layer aspects of a data
+format.
+Other examples will show how the layer length can be limited to a sub-region of the data.
+
+More detailed documentation for the [Line Folded Layers](#LineFoldedLayers) is below.
+
+## Example: Base64, GZip, and BoundaryMark Layers used Together
+
+In this example, the data consists of a preliminary string, a section of CSV-like data, and a
+final string element.
+The CSV-like data section in the middle is both gzipped, and then base64 encoded, with the
+boundary value mark of "`=_END_=`" indicating where the base64 part ends.
+
+The point of the preliminary string and the final string is to show that this sort of layering
+composes properly with surrounding data elements in the schema. Those strings could be anything
+at all, they're just strings to make this example simpler.
+
+The physical data string/stream looks like this:
+```text
+example of data before the base64 gzipped part
+H4sIAAAAAAAA/y3JQQqAIBCF4b1nmYG0iNxG+84wpqFhI4xuun0Gbd4P78tUG5xJ+t7J+xxg21dV
+79QiSHFBGjgh9oVB22XBYUQzqatErv36CjGwPJ/OOGg0Y1cOFUgegoPaLwa1VS9htd+UbgAAAA==
+=_END_=
+example of data after the base64 gzipped part
+```
+That base64 encoded part, if decoded and decompressed looks like this CSV-like data:
+```
+last,first,middle,DOB
+smith,robert,brandon,1988-03-24
+johnson,john,henry,1986-01-23
+jones,arya,cat,1986-02-19
+```
+
+The Infoset, presented as XML, that we will create for this entire data stream
+using Daffodil parsing looks like:
+```xml
+<ex:data>
+ <before>example of data before the base64 gzipped part</before>
+ <header><title>last</title><title>first</title><title>middle</title><title>DOB</title></header>
+ <record><item>smith</item><item>robert</item><item>brandon</item><item>1988-03-24</item></record>
+ <record><item>johnson</item><item>john</item><item>henry</item><item>1986-01-23</item></record>
+ <record><item>jones</item><item>arya</item><item>cat</item><item>1986-02-19</item></record>
+ <after>example of data after the base64 gzipped part</after>
+</ex:data>
+```
+
+With that characterization of the problem, let's look at the DFDL schema with the Daffodil
+layering properties to do the boundaryMark, base64 decode, and gzip decompression.
+(Note that each of these layers is described further in its own section below on this page.)
+
+First there is the start of the schema which introduces namespace prefix definitions for DFDL,
+for the DFDL extensions (dfdlx) prefix which is used for the `dfdlx:layer` property.
+
+This also sets up 3 namespace prefixes for the namespaces of the boundaryMark (bm), gzip (gz)
+and base64_MIME (b64) layers which will be imported just after this header.
+
+```xml
+<schema
+ xmlns="http://www.w3.org/2001/XMLSchema"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:fn="http://www.w3.org/2005/xpath-functions"
Review Comment:
Might make this a little less verbose teo remove the xsi and fn namespaces that arent used.
##########
site/layers.md:
##########
@@ -0,0 +1,655 @@
+---
+description: Pluggable Extensions to Enable Algorithmic Transformations
+ in DFDL
+group: 'nav-right'
+layout: page
+title: 'Layers - Algorithmic
+ Extensions for DFDL'
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+### Table of Contents
+{:.no_toc}
+
+1. use ordered table of contents
+{:toc}
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+Layers are _byte oriented_.
+The smallest non-zero layer size is a single byte, and the length of
+all layers are in bytes.
+Throughout this discussion of layers, when length is discussed, the units of measure are always
+bytes.
+
+Layers can be composed together, that is, with one on top of another, and their composition is
+orthogonal (as in non-interacting).
+This is actually very common. For example, binary data being included into a textual data format
+can be both gzip compressed, and then base64 encoded, which requires 2 layers to describe.
+There is no limit to this depth.
+
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses
+multiple layers together.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- [base64_MIME](#base64-mime-layer)
+- [fourbyteswap](#byte-swapping-layers)
+- [twobyteswap](#byte-swapping-layers)
+- [gzip](#gzip-layer)
+- [lineFolded_IMF](#line-folded-layers)
+- [lineFolded_iCalendar](#line-folded-layers)
+
+Daffodil also includes two utility layers that are used in combination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- [boundaryMark](#boundaryMark-layer)
+- [fixedLength](#fixedLength-layer)
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+### Layer Kinds: Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html) class
+which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic
+Redundancy Check (CRC) over the data stream.
+
+The value of the checksum (or hash or CRC) is assigned to a DFDL variable as the result of the
+layer. This makes the value available for use by the DFDL schema that uses the checksum layer.
+When parsing, the value of this DFDL variable can then be compared to a checksum field in the
+data, and either an invalid data element or an parse-error can be created if the checksum in the
+data stream does not match the computed value.
+When unparsing, the value of this DFDL variable can be written to an element using the
+`dfdl:outputValueCalc` property.
+
+An example of a checksum layer plug-in is in the
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header checksum algorithm.
+
+Custom checksum layers are created by deriving an implementation class from the Daffodil API's
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.
+html)
+class, which is introduced in a later section.
+
+# Using Layers
+
+To use a layer you must know
+- the layer's namespace URI
+- the layer's name
+- the names of any layer parameter variables
+- the names of any layer result variables
+
+Each layer normally has a _layer schema file_ (.dfdl.xsd) which contains the DFDL variable's
+`dfdl:defineVariable` declarations.
+
+A `sequence` group encloses the scope, within the DFDL schema, of the layer.
+Often layers are used to describe checksums or decding/encoding on small regions within a
+larger data format.
+
+The DFDL extension property `dfdlx:layer` declares that the data underlying a `sequence` is to
+use a layer. The value of the property is a QName identifying the namespace (via prefix) and
+name of the layer.
+
+A layer's namespace may define DFDL variables that are used to pass parameters to the layer, or
+to receive results back from the layer (usually for checksum layers).
+
+Lastly, there is the concept of _layer length limiting_.
+
+### Layer Length Limiting
+
+A layer can process an entire input file/stream.
+Such a layer is said to be _unlimited_.
+
+A layer can also have a specific data length built into it.
+The `IPv4Checksum` layer (in the aforementioned
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema) has a fixed length of 20 bytes.
+Such a layer is said to be _self limiting_.
+
+More commonly, a layer has a limited region within the data stream that it is supposed to describe.
+The restriction of the layer so that it only describes the expected part of the data is called
+_layer limiting_.
+
+There are these ways to do layer limiting, that differ in important ways:
+- using enclosing elements of specified length
+- using a fixedLength layer
+- binding a DFDL variable specific to a particular layer
+- using a boundaryMark layer
+
+#### Layer Limiting using Enclosing Elements of Specified Length
+
+If the `sequence` for a layer is the model group of the
+complex type of an element, and that element has specified length (meaning it has
+`dfdl:lengthKind` property of `explicit`, `implicit`, `prefixed` or `pattern`), then the length
+of the element limits the length of the layer within it.
+
+In this case the length of the layer is limited when parsing, but it is NOT limited when
+unparsing.
+
+#### Layer Limiting using the `fixedLength` Utility Layer
+
+The layer `fixedLength` is a utility layer intended for use in conjunction with other layers.
+It dictates the parse _and unparse_ length to be the value of the `layerLength` DFDL variable.
+Setting or binding this DFDL variable fixes the length of the `fixedLength` layer, and then
+any layer on top of that layer will be limited by the underlying fixed length.
+
+#### Layer Limiting by Binding a DFDL Variable
+
+Some layers will define a DFDL variable that must be set to a non-negative
+integer value to specify the layer length that the layer will use.
+All checksum layers built with the Daffodil API `ChecksumLayer` base class,
+use the DFDL `layerLength` variable to specify the length (in bytes) of the layer.
+
+Layers may specify restrictions on the minimum and maximum allowed values of these lengths,
+and passing an out-of-range value for the variable is a processing error.
+
+#### Layer Limiting using the `boundaryMark` Utility Layer
+
+The `boundaryMark` layer uses a variable named `boundaryMark` which provides a string parameter
+to the layer.
+This string is a delimiter of the layer data when parsing, and when unparsing the string is
+inserted after the (otherwise unbounded length) layer data.
+Unlike DFDL's normal delimiter properties( `dfdl:separator`, `dfdl:terminator`, and
+`dfdl:initiator`), the scanning for a boundary mark string does not use any escape scheme.
+
+
+## Example: Line Folding
+
+Consider the line folding layer, specifically the `lineFolded_IMF` layer,
+which is built-in to Daffodil.
+
+Line folding is a way of encoding textual data formats so that no line of text is longer than
+a limited line length.
+
+Consider this data :
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem
+ ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+This data has been _line folded_ at roughly 72 characters by inserting a CRLF
+before an existing space in the data.
+Each line ends with a CRLF (\r\n) and the second through fourth lines begin
+with a space as a way of indicating that they are extension lines.
+This data is supposed to be reassembled to form a long single-line string by removing
+all CRLF pairs.
+
+The result should be this single longer string which does not contain any line endings:
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+To achieve this we would use the `lineFolded_IMF` layer.
+This layer has a specific namespace which our DFDL schema will define a prefix `lf` for like this:
+```xml
+xmlns:lf="urn:org.apache.daffodil.layers.lineFolded"
+```
+Our DFDL schema will import the layer schema for the line-folded layer with this import:
+```xml
+<import namespace="urn:org.apache.daffodil.layers.lineFolded"
+ schemaLocation="/org/apache/daffodil/layers/xsd/lineFoldedLayer.dfdl.xsd"/>
+```
+Then the layer is incorporated into our DFDL schema like this:
+```xml
+<sequence dfdlx:layer="lf:lineFolded_IMF">
+ ... elements to be parsed from the unfolded layer data go here ...
+</sequence>
+```
+You can see that use of a layer is described using the `dfdx:layer` property, and the specific layer
+is identified by a QName using the previously defined namespace prefix.
+The scope of the layer is the duration of the sequence it appears on.
+The `dfdlx:layer` property can _only_ be used on an XSD `sequence`.
+
+The `lineFolded_IMF` layer does not define any DFDL variables in its namespace as it has no
+parameters and produces no results.
+As of Daffodil 3.8.0, the `import` statement above is optional for the
+line-folded layers as the DFDL schema file `lineFoldedLayer.dfdl.xsd` does not contain any
+definitions.
+In the future however, parameters may be added, so for uniformity all layers define a DFDL
+schema to be imported as part of using the layer.
+
+If the line-folding applies to the entire data stream (not uncommon for formats that use line
+folding), then what has been described is all one needs to describe the layer aspects of a data
+format.
+Other examples will show how the layer length can be limited to a sub-region of the data.
+
+More detailed documentation for the [Line Folded Layers](#LineFoldedLayers) is below.
+
+## Example: Base64, GZip, and BoundaryMark Layers used Together
+
+In this example, the data consists of a preliminary string, a section of CSV-like data, and a
+final string element.
+The CSV-like data section in the middle is both gzipped, and then base64 encoded, with the
+boundary value mark of "`=_END_=`" indicating where the base64 part ends.
+
+The point of the preliminary string and the final string is to show that this sort of layering
+composes properly with surrounding data elements in the schema. Those strings could be anything
+at all, they're just strings to make this example simpler.
+
+The physical data string/stream looks like this:
+```text
+example of data before the base64 gzipped part
+H4sIAAAAAAAA/y3JQQqAIBCF4b1nmYG0iNxG+84wpqFhI4xuun0Gbd4P78tUG5xJ+t7J+xxg21dV
+79QiSHFBGjgh9oVB22XBYUQzqatErv36CjGwPJ/OOGg0Y1cOFUgegoPaLwa1VS9htd+UbgAAAA==
+=_END_=
+example of data after the base64 gzipped part
+```
+That base64 encoded part, if decoded and decompressed looks like this CSV-like data:
+```
+last,first,middle,DOB
+smith,robert,brandon,1988-03-24
+johnson,john,henry,1986-01-23
+jones,arya,cat,1986-02-19
+```
+
+The Infoset, presented as XML, that we will create for this entire data stream
+using Daffodil parsing looks like:
+```xml
+<ex:data>
+ <before>example of data before the base64 gzipped part</before>
+ <header><title>last</title><title>first</title><title>middle</title><title>DOB</title></header>
+ <record><item>smith</item><item>robert</item><item>brandon</item><item>1988-03-24</item></record>
+ <record><item>johnson</item><item>john</item><item>henry</item><item>1986-01-23</item></record>
+ <record><item>jones</item><item>arya</item><item>cat</item><item>1986-02-19</item></record>
+ <after>example of data after the base64 gzipped part</after>
+</ex:data>
+```
+
+With that characterization of the problem, let's look at the DFDL schema with the Daffodil
+layering properties to do the boundaryMark, base64 decode, and gzip decompression.
+(Note that each of these layers is described further in its own section below on this page.)
+
+First there is the start of the schema which introduces namespace prefix definitions for DFDL,
+for the DFDL extensions (dfdlx) prefix which is used for the `dfdlx:layer` property.
+
+This also sets up 3 namespace prefixes for the namespaces of the boundaryMark (bm), gzip (gz)
+and base64_MIME (b64) layers which will be imported just after this header.
+
+```xml
+<schema
+ xmlns="http://www.w3.org/2001/XMLSchema"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:fn="http://www.w3.org/2005/xpath-functions"
+ xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions"
+ xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:bm="urn:org.apache.daffodil.layers.boundaryMark"
+ xmlns:b64="urn:org.apache.daffodil.layers.base64_MIME"
+ xmlns:gz="urn:org.apache.daffodil.layers.gzip"
+ xmlns:ex="http://example.com"
+ targetNamespace="http://example.com">
+```
+Next we import the Daffodil built in general format which provides most DFDL properties with
+sensible starting values, and import the namespaces of our boundaryMark, gzip, and base64_MIME
+layers, and lastly, set the default format properties for use by this schema file:
+```xml
+<include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+
+<import namespace="urn:org.apache.daffodil.layers.boundaryMark"
+ schemaLocation="/org/apache/daffodil/layers/xsd/boundaryMarkLayer.dfdl.xsd"/>
+<import namespace="urn:org.apache.daffodil.layers.gzip"
+ schemaLocation="/org/apache/daffodil/layers/xsd/gzipLayer.dfdl.xsd"/>
+<import namespace="urn:org.apache.daffodil.layers.base64_MIME"
+ schemaLocation="/org/apache/daffodil/layers/xsd/base64_MIMELayer.dfdl.xsd"/>
+
+<annotation><appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:format ref="ex:GeneralFormat" representation="binary"/>
+</appinfo></annotation>
+```
+The root element of the schema is `data`, and this element has type `dataType`.
+(A separate named complex type definition for the root element is a
+recommended style for DFDL schemas as it allows someone else to reuse the schema without
+getting involved in element namespace prefix complexities.)
+The `dataType` just shows us the `before` and `after` elements, and references a group
+named `layeredDataGroup` for the layered part in between.
+```xml
+<element name="data" type="ex:dataType"/>
+
+<complexType name="dataType">
+ <sequence dfdl:separator="%NL;">
+ <element name="before" type="string" dfdl:lengthKind="delimited"/>
+ <group ref="ex:layeredDataGroup"/>
+ <element name="after" type="string" dfdl:lengthKind="delimited"/>
+ </sequence>
+</complexType>
+```
+Now we see the definition of the layering.
+All 3 layers are declared in this named group.
+The lines here are numbered so that the text that follows can refer to them:
+```xml
+1 <group name="layeredDataGroup">
+2 <sequence dfdlx:layer="bm:boundaryMark">
+3 <annotation>
+4 <appinfo source="http://www.ogf.org/dfdl/">
+5 <dfdl:newVariableInstance ref="bm:layerEncoding" defaultValue="iso-8859-1"/>
+6 <dfdl:newVariableInstance ref="bm:boundaryMark" defaultValue="=_END_="/>
+7 </appinfo>
+8 </annotation>
+9 <sequence dfdlx:layer="b64:base64_MIME">
+10 <sequence dfdlx:layer="gz:gzip">
+11 <group ref="ex:dataFieldsGroup"/>
+12 </sequence>
+13 </sequence>
+14 </sequence>
+15 </group>
+```
+In the above, at line 2 we see the sequence declares the boundaryMark layer.
+This layer requires 2 parameters to be passed by binding DFDL variables, which occurs on
+lines 5 and 6.
+Line 5 binds the `layerEncoding` variable and note the prefix since DFDL variable names can be
+in a specific namespace.
+Line 6 binds the `boundaryMark` variable to the string "`=_END_=`".
+Note that the variable and the layer both are named `boundaryMark`, but this is just coincidence.
+Layers do not have to have parameters that match their name, nor even any parameters at all.
+The boundary mark is the string which marks the end of the data based on scanning for that ending
+mark.
+
+The data region now being isolated, the `boundaryMark` layer contains just one sequence, and line
+9 which itself declares that the data is base64 encoded by specifying the base64_MIME layer.
+This layer has no parameters, so there are no `dfdl:newVariableInstance` statements to bind any
+variables.
+The region of data that is base64 encoded is then described by the sequence at line 10 which
+declares that the data is compressed by the `gzip` algorithm.
+
+Finally, at line 11 we reference a group which describes what the data format of the underlying
+CSV-like data.
+That group's definition is regular DFDL without any of the complexity of layers.
+This is useful for testing, as data can be parsed using this group (in the complexType of an
+element) to ensure it works properly without involving the complexity of layers.
+This group definition is the last thing in the schema:
+```xml
+<group name="dataFieldsGroup">
+ <sequence dfdl:separator="%NL;" dfdl:separatorPosition="postfix">
+ <element name="header" minOccurs="0" dfdl:occursCountKind="implicit">
+ <complexType>
+ <sequence dfdl:separator=",">
+ <element name="title" type="xs:string" maxOccurs="unbounded" dfdl:lengthKind="delimited"/>
+ </sequence>
+ </complexType>
+ </element>
+ <element name="record" maxOccurs="unbounded">
+ <complexType>
+ <sequence dfdl:separator=",">
+ <element name="item" type="xs:string" maxOccurs="unbounded"
+ dfdl:lengthKind="delimited"
+ dfdl:occursCount="{ fn:count(../../header/title) }"
+ dfdl:occursCountKind="expression"/>
+ </sequence>
+ </complexType>
+ </element>
+ </sequence>
+</group>
+
+</schema>
+```
+The above schema works both to parse, but also to unparse this data.
+
+----
+
+# Using Custom Plug-In Layers
+
+A custom plug-in layer is used in the same manner as the built-in Daffodil layers with just a few
+additional details:
+1. The custom layer will have a DFDL schema that declares any DFDL variables it uses, and
+ defines any data formats needed to facilitate use of the layer.
+ This schema will need to be included/imported.
+ Normally this schema would be a small single-file schema and it can be loaded from the
+ classpath.
+2. The custom layer's compiled Java/Scala code is dynamically loaded using the
+ Java Service Provider Interface (SPI).
+ Hence, they must be compiled into Jar files with specific META-INF metadata enclosed, and
+ these jars must appear on the Java CLASSPATH so that they can be found and loaded.
+
+The layer API is defined via Java clases and interfaces to enable writing of custom layers in
+either Java or Scala.
+(One of the built-in layers (Gzip) is written in Java now, by way of proving
+that one can write a Layer in Java.)
+
+Transformer layer classes are derived from the
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html)
+base class.
+
+Checksum layer classes are derived from the
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.html)
+base class.
+
+Further details on how to define custom plug-in layers is in the Javadoc for the
+[Layer API](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/package-summary.html)
+
+----
Review Comment:
Personally I find the hr's inserted because of the `----` look a little weird. Thoughts on removing?
##########
site/layers.md:
##########
@@ -0,0 +1,655 @@
+---
+description: Pluggable Extensions to Enable Algorithmic Transformations
+ in DFDL
+group: 'nav-right'
+layout: page
+title: 'Layers - Algorithmic
+ Extensions for DFDL'
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+### Table of Contents
+{:.no_toc}
+
+1. use ordered table of contents
+{:toc}
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+Layers are _byte oriented_.
+The smallest non-zero layer size is a single byte, and the length of
+all layers are in bytes.
+Throughout this discussion of layers, when length is discussed, the units of measure are always
+bytes.
+
+Layers can be composed together, that is, with one on top of another, and their composition is
+orthogonal (as in non-interacting).
+This is actually very common. For example, binary data being included into a textual data format
+can be both gzip compressed, and then base64 encoded, which requires 2 layers to describe.
+There is no limit to this depth.
+
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses
+multiple layers together.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- [base64_MIME](#base64-mime-layer)
+- [fourbyteswap](#byte-swapping-layers)
+- [twobyteswap](#byte-swapping-layers)
+- [gzip](#gzip-layer)
+- [lineFolded_IMF](#line-folded-layers)
+- [lineFolded_iCalendar](#line-folded-layers)
+
+Daffodil also includes two utility layers that are used in combination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- [boundaryMark](#boundaryMark-layer)
+- [fixedLength](#fixedLength-layer)
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+### Layer Kinds: Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html) class
+which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic
+Redundancy Check (CRC) over the data stream.
+
+The value of the checksum (or hash or CRC) is assigned to a DFDL variable as the result of the
+layer. This makes the value available for use by the DFDL schema that uses the checksum layer.
+When parsing, the value of this DFDL variable can then be compared to a checksum field in the
+data, and either an invalid data element or an parse-error can be created if the checksum in the
+data stream does not match the computed value.
+When unparsing, the value of this DFDL variable can be written to an element using the
+`dfdl:outputValueCalc` property.
+
+An example of a checksum layer plug-in is in the
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header checksum algorithm.
+
+Custom checksum layers are created by deriving an implementation class from the Daffodil API's
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.
Review Comment:
Use absolute path here.
##########
site/layers.md:
##########
@@ -0,0 +1,655 @@
+---
+description: Pluggable Extensions to Enable Algorithmic Transformations
+ in DFDL
+group: 'nav-right'
+layout: page
+title: 'Layers - Algorithmic
+ Extensions for DFDL'
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+### Table of Contents
+{:.no_toc}
+
+1. use ordered table of contents
+{:toc}
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+Layers are _byte oriented_.
+The smallest non-zero layer size is a single byte, and the length of
+all layers are in bytes.
+Throughout this discussion of layers, when length is discussed, the units of measure are always
+bytes.
+
+Layers can be composed together, that is, with one on top of another, and their composition is
+orthogonal (as in non-interacting).
+This is actually very common. For example, binary data being included into a textual data format
+can be both gzip compressed, and then base64 encoded, which requires 2 layers to describe.
+There is no limit to this depth.
+
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses
+multiple layers together.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- [base64_MIME](#base64-mime-layer)
+- [fourbyteswap](#byte-swapping-layers)
+- [twobyteswap](#byte-swapping-layers)
+- [gzip](#gzip-layer)
+- [lineFolded_IMF](#line-folded-layers)
+- [lineFolded_iCalendar](#line-folded-layers)
+
+Daffodil also includes two utility layers that are used in combination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- [boundaryMark](#boundaryMark-layer)
+- [fixedLength](#fixedLength-layer)
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+### Layer Kinds: Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html) class
+which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic
+Redundancy Check (CRC) over the data stream.
+
+The value of the checksum (or hash or CRC) is assigned to a DFDL variable as the result of the
+layer. This makes the value available for use by the DFDL schema that uses the checksum layer.
+When parsing, the value of this DFDL variable can then be compared to a checksum field in the
+data, and either an invalid data element or an parse-error can be created if the checksum in the
+data stream does not match the computed value.
+When unparsing, the value of this DFDL variable can be written to an element using the
+`dfdl:outputValueCalc` property.
+
+An example of a checksum layer plug-in is in the
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header checksum algorithm.
+
+Custom checksum layers are created by deriving an implementation class from the Daffodil API's
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.
+html)
+class, which is introduced in a later section.
+
+# Using Layers
+
+To use a layer you must know
+- the layer's namespace URI
+- the layer's name
+- the names of any layer parameter variables
+- the names of any layer result variables
+
+Each layer normally has a _layer schema file_ (.dfdl.xsd) which contains the DFDL variable's
+`dfdl:defineVariable` declarations.
+
+A `sequence` group encloses the scope, within the DFDL schema, of the layer.
+Often layers are used to describe checksums or decding/encoding on small regions within a
+larger data format.
+
+The DFDL extension property `dfdlx:layer` declares that the data underlying a `sequence` is to
+use a layer. The value of the property is a QName identifying the namespace (via prefix) and
+name of the layer.
+
+A layer's namespace may define DFDL variables that are used to pass parameters to the layer, or
+to receive results back from the layer (usually for checksum layers).
+
+Lastly, there is the concept of _layer length limiting_.
+
+### Layer Length Limiting
+
+A layer can process an entire input file/stream.
+Such a layer is said to be _unlimited_.
+
+A layer can also have a specific data length built into it.
+The `IPv4Checksum` layer (in the aforementioned
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema) has a fixed length of 20 bytes.
+Such a layer is said to be _self limiting_.
+
+More commonly, a layer has a limited region within the data stream that it is supposed to describe.
+The restriction of the layer so that it only describes the expected part of the data is called
+_layer limiting_.
+
+There are these ways to do layer limiting, that differ in important ways:
+- using enclosing elements of specified length
+- using a fixedLength layer
+- binding a DFDL variable specific to a particular layer
+- using a boundaryMark layer
+
+#### Layer Limiting using Enclosing Elements of Specified Length
+
+If the `sequence` for a layer is the model group of the
+complex type of an element, and that element has specified length (meaning it has
+`dfdl:lengthKind` property of `explicit`, `implicit`, `prefixed` or `pattern`), then the length
+of the element limits the length of the layer within it.
+
+In this case the length of the layer is limited when parsing, but it is NOT limited when
+unparsing.
+
+#### Layer Limiting using the `fixedLength` Utility Layer
+
+The layer `fixedLength` is a utility layer intended for use in conjunction with other layers.
+It dictates the parse _and unparse_ length to be the value of the `layerLength` DFDL variable.
+Setting or binding this DFDL variable fixes the length of the `fixedLength` layer, and then
+any layer on top of that layer will be limited by the underlying fixed length.
+
+#### Layer Limiting by Binding a DFDL Variable
+
+Some layers will define a DFDL variable that must be set to a non-negative
+integer value to specify the layer length that the layer will use.
+All checksum layers built with the Daffodil API `ChecksumLayer` base class,
+use the DFDL `layerLength` variable to specify the length (in bytes) of the layer.
+
+Layers may specify restrictions on the minimum and maximum allowed values of these lengths,
+and passing an out-of-range value for the variable is a processing error.
+
+#### Layer Limiting using the `boundaryMark` Utility Layer
+
+The `boundaryMark` layer uses a variable named `boundaryMark` which provides a string parameter
+to the layer.
+This string is a delimiter of the layer data when parsing, and when unparsing the string is
+inserted after the (otherwise unbounded length) layer data.
+Unlike DFDL's normal delimiter properties( `dfdl:separator`, `dfdl:terminator`, and
+`dfdl:initiator`), the scanning for a boundary mark string does not use any escape scheme.
+
+
+## Example: Line Folding
+
+Consider the line folding layer, specifically the `lineFolded_IMF` layer,
+which is built-in to Daffodil.
+
+Line folding is a way of encoding textual data formats so that no line of text is longer than
+a limited line length.
+
+Consider this data :
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem
+ ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+This data has been _line folded_ at roughly 72 characters by inserting a CRLF
+before an existing space in the data.
+Each line ends with a CRLF (\r\n) and the second through fourth lines begin
+with a space as a way of indicating that they are extension lines.
+This data is supposed to be reassembled to form a long single-line string by removing
+all CRLF pairs.
+
+The result should be this single longer string which does not contain any line endings:
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+To achieve this we would use the `lineFolded_IMF` layer.
+This layer has a specific namespace which our DFDL schema will define a prefix `lf` for like this:
+```xml
+xmlns:lf="urn:org.apache.daffodil.layers.lineFolded"
+```
+Our DFDL schema will import the layer schema for the line-folded layer with this import:
+```xml
+<import namespace="urn:org.apache.daffodil.layers.lineFolded"
+ schemaLocation="/org/apache/daffodil/layers/xsd/lineFoldedLayer.dfdl.xsd"/>
+```
+Then the layer is incorporated into our DFDL schema like this:
+```xml
+<sequence dfdlx:layer="lf:lineFolded_IMF">
+ ... elements to be parsed from the unfolded layer data go here ...
+</sequence>
+```
+You can see that use of a layer is described using the `dfdx:layer` property, and the specific layer
+is identified by a QName using the previously defined namespace prefix.
+The scope of the layer is the duration of the sequence it appears on.
+The `dfdlx:layer` property can _only_ be used on an XSD `sequence`.
+
+The `lineFolded_IMF` layer does not define any DFDL variables in its namespace as it has no
+parameters and produces no results.
+As of Daffodil 3.8.0, the `import` statement above is optional for the
+line-folded layers as the DFDL schema file `lineFoldedLayer.dfdl.xsd` does not contain any
+definitions.
+In the future however, parameters may be added, so for uniformity all layers define a DFDL
+schema to be imported as part of using the layer.
+
+If the line-folding applies to the entire data stream (not uncommon for formats that use line
+folding), then what has been described is all one needs to describe the layer aspects of a data
+format.
+Other examples will show how the layer length can be limited to a sub-region of the data.
+
+More detailed documentation for the [Line Folded Layers](#LineFoldedLayers) is below.
+
+## Example: Base64, GZip, and BoundaryMark Layers used Together
+
+In this example, the data consists of a preliminary string, a section of CSV-like data, and a
+final string element.
+The CSV-like data section in the middle is both gzipped, and then base64 encoded, with the
+boundary value mark of "`=_END_=`" indicating where the base64 part ends.
+
+The point of the preliminary string and the final string is to show that this sort of layering
+composes properly with surrounding data elements in the schema. Those strings could be anything
+at all, they're just strings to make this example simpler.
+
+The physical data string/stream looks like this:
+```text
+example of data before the base64 gzipped part
+H4sIAAAAAAAA/y3JQQqAIBCF4b1nmYG0iNxG+84wpqFhI4xuun0Gbd4P78tUG5xJ+t7J+xxg21dV
+79QiSHFBGjgh9oVB22XBYUQzqatErv36CjGwPJ/OOGg0Y1cOFUgegoPaLwa1VS9htd+UbgAAAA==
+=_END_=
+example of data after the base64 gzipped part
+```
+That base64 encoded part, if decoded and decompressed looks like this CSV-like data:
+```
+last,first,middle,DOB
+smith,robert,brandon,1988-03-24
+johnson,john,henry,1986-01-23
+jones,arya,cat,1986-02-19
+```
+
+The Infoset, presented as XML, that we will create for this entire data stream
+using Daffodil parsing looks like:
+```xml
+<ex:data>
+ <before>example of data before the base64 gzipped part</before>
+ <header><title>last</title><title>first</title><title>middle</title><title>DOB</title></header>
+ <record><item>smith</item><item>robert</item><item>brandon</item><item>1988-03-24</item></record>
+ <record><item>johnson</item><item>john</item><item>henry</item><item>1986-01-23</item></record>
+ <record><item>jones</item><item>arya</item><item>cat</item><item>1986-02-19</item></record>
+ <after>example of data after the base64 gzipped part</after>
+</ex:data>
+```
+
+With that characterization of the problem, let's look at the DFDL schema with the Daffodil
+layering properties to do the boundaryMark, base64 decode, and gzip decompression.
+(Note that each of these layers is described further in its own section below on this page.)
+
+First there is the start of the schema which introduces namespace prefix definitions for DFDL,
+for the DFDL extensions (dfdlx) prefix which is used for the `dfdlx:layer` property.
+
+This also sets up 3 namespace prefixes for the namespaces of the boundaryMark (bm), gzip (gz)
+and base64_MIME (b64) layers which will be imported just after this header.
+
+```xml
+<schema
+ xmlns="http://www.w3.org/2001/XMLSchema"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:fn="http://www.w3.org/2005/xpath-functions"
+ xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions"
+ xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:bm="urn:org.apache.daffodil.layers.boundaryMark"
+ xmlns:b64="urn:org.apache.daffodil.layers.base64_MIME"
+ xmlns:gz="urn:org.apache.daffodil.layers.gzip"
+ xmlns:ex="http://example.com"
+ targetNamespace="http://example.com">
+```
+Next we import the Daffodil built in general format which provides most DFDL properties with
+sensible starting values, and import the namespaces of our boundaryMark, gzip, and base64_MIME
+layers, and lastly, set the default format properties for use by this schema file:
+```xml
+<include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+
+<import namespace="urn:org.apache.daffodil.layers.boundaryMark"
+ schemaLocation="/org/apache/daffodil/layers/xsd/boundaryMarkLayer.dfdl.xsd"/>
+<import namespace="urn:org.apache.daffodil.layers.gzip"
+ schemaLocation="/org/apache/daffodil/layers/xsd/gzipLayer.dfdl.xsd"/>
+<import namespace="urn:org.apache.daffodil.layers.base64_MIME"
+ schemaLocation="/org/apache/daffodil/layers/xsd/base64_MIMELayer.dfdl.xsd"/>
+
+<annotation><appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:format ref="ex:GeneralFormat" representation="binary"/>
+</appinfo></annotation>
+```
+The root element of the schema is `data`, and this element has type `dataType`.
+(A separate named complex type definition for the root element is a
+recommended style for DFDL schemas as it allows someone else to reuse the schema without
+getting involved in element namespace prefix complexities.)
+The `dataType` just shows us the `before` and `after` elements, and references a group
+named `layeredDataGroup` for the layered part in between.
+```xml
+<element name="data" type="ex:dataType"/>
+
+<complexType name="dataType">
+ <sequence dfdl:separator="%NL;">
+ <element name="before" type="string" dfdl:lengthKind="delimited"/>
+ <group ref="ex:layeredDataGroup"/>
+ <element name="after" type="string" dfdl:lengthKind="delimited"/>
+ </sequence>
+</complexType>
+```
+Now we see the definition of the layering.
+All 3 layers are declared in this named group.
+The lines here are numbered so that the text that follows can refer to them:
+```xml
+1 <group name="layeredDataGroup">
+2 <sequence dfdlx:layer="bm:boundaryMark">
+3 <annotation>
+4 <appinfo source="http://www.ogf.org/dfdl/">
+5 <dfdl:newVariableInstance ref="bm:layerEncoding" defaultValue="iso-8859-1"/>
+6 <dfdl:newVariableInstance ref="bm:boundaryMark" defaultValue="=_END_="/>
+7 </appinfo>
+8 </annotation>
+9 <sequence dfdlx:layer="b64:base64_MIME">
+10 <sequence dfdlx:layer="gz:gzip">
+11 <group ref="ex:dataFieldsGroup"/>
+12 </sequence>
+13 </sequence>
+14 </sequence>
+15 </group>
+```
+In the above, at line 2 we see the sequence declares the boundaryMark layer.
+This layer requires 2 parameters to be passed by binding DFDL variables, which occurs on
+lines 5 and 6.
+Line 5 binds the `layerEncoding` variable and note the prefix since DFDL variable names can be
+in a specific namespace.
+Line 6 binds the `boundaryMark` variable to the string "`=_END_=`".
+Note that the variable and the layer both are named `boundaryMark`, but this is just coincidence.
+Layers do not have to have parameters that match their name, nor even any parameters at all.
+The boundary mark is the string which marks the end of the data based on scanning for that ending
+mark.
+
+The data region now being isolated, the `boundaryMark` layer contains just one sequence, and line
+9 which itself declares that the data is base64 encoded by specifying the base64_MIME layer.
+This layer has no parameters, so there are no `dfdl:newVariableInstance` statements to bind any
+variables.
+The region of data that is base64 encoded is then described by the sequence at line 10 which
+declares that the data is compressed by the `gzip` algorithm.
+
+Finally, at line 11 we reference a group which describes what the data format of the underlying
+CSV-like data.
+That group's definition is regular DFDL without any of the complexity of layers.
+This is useful for testing, as data can be parsed using this group (in the complexType of an
+element) to ensure it works properly without involving the complexity of layers.
+This group definition is the last thing in the schema:
+```xml
+<group name="dataFieldsGroup">
+ <sequence dfdl:separator="%NL;" dfdl:separatorPosition="postfix">
+ <element name="header" minOccurs="0" dfdl:occursCountKind="implicit">
+ <complexType>
+ <sequence dfdl:separator=",">
+ <element name="title" type="xs:string" maxOccurs="unbounded" dfdl:lengthKind="delimited"/>
+ </sequence>
+ </complexType>
+ </element>
+ <element name="record" maxOccurs="unbounded">
+ <complexType>
+ <sequence dfdl:separator=",">
+ <element name="item" type="xs:string" maxOccurs="unbounded"
+ dfdl:lengthKind="delimited"
+ dfdl:occursCount="{ fn:count(../../header/title) }"
+ dfdl:occursCountKind="expression"/>
+ </sequence>
+ </complexType>
+ </element>
+ </sequence>
+</group>
+
+</schema>
+```
+The above schema works both to parse, but also to unparse this data.
+
+----
+
+# Using Custom Plug-In Layers
Review Comment:
Should be `##`
##########
site/layers.md:
##########
@@ -0,0 +1,655 @@
+---
+description: Pluggable Extensions to Enable Algorithmic Transformations
+ in DFDL
+group: 'nav-right'
+layout: page
+title: 'Layers - Algorithmic
+ Extensions for DFDL'
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+### Table of Contents
+{:.no_toc}
+
+1. use ordered table of contents
+{:toc}
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+Layers are _byte oriented_.
+The smallest non-zero layer size is a single byte, and the length of
+all layers are in bytes.
+Throughout this discussion of layers, when length is discussed, the units of measure are always
+bytes.
+
+Layers can be composed together, that is, with one on top of another, and their composition is
+orthogonal (as in non-interacting).
+This is actually very common. For example, binary data being included into a textual data format
+can be both gzip compressed, and then base64 encoded, which requires 2 layers to describe.
+There is no limit to this depth.
+
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses
+multiple layers together.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- [base64_MIME](#base64-mime-layer)
+- [fourbyteswap](#byte-swapping-layers)
+- [twobyteswap](#byte-swapping-layers)
+- [gzip](#gzip-layer)
+- [lineFolded_IMF](#line-folded-layers)
+- [lineFolded_iCalendar](#line-folded-layers)
+
+Daffodil also includes two utility layers that are used in combination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- [boundaryMark](#boundaryMark-layer)
+- [fixedLength](#fixedLength-layer)
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+### Layer Kinds: Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html) class
+which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic
+Redundancy Check (CRC) over the data stream.
+
+The value of the checksum (or hash or CRC) is assigned to a DFDL variable as the result of the
+layer. This makes the value available for use by the DFDL schema that uses the checksum layer.
+When parsing, the value of this DFDL variable can then be compared to a checksum field in the
+data, and either an invalid data element or an parse-error can be created if the checksum in the
+data stream does not match the computed value.
+When unparsing, the value of this DFDL variable can be written to an element using the
+`dfdl:outputValueCalc` property.
+
+An example of a checksum layer plug-in is in the
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header checksum algorithm.
+
+Custom checksum layers are created by deriving an implementation class from the Daffodil API's
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.
+html)
+class, which is introduced in a later section.
+
+# Using Layers
+
+To use a layer you must know
+- the layer's namespace URI
+- the layer's name
+- the names of any layer parameter variables
+- the names of any layer result variables
+
+Each layer normally has a _layer schema file_ (.dfdl.xsd) which contains the DFDL variable's
+`dfdl:defineVariable` declarations.
+
+A `sequence` group encloses the scope, within the DFDL schema, of the layer.
+Often layers are used to describe checksums or decding/encoding on small regions within a
+larger data format.
+
+The DFDL extension property `dfdlx:layer` declares that the data underlying a `sequence` is to
+use a layer. The value of the property is a QName identifying the namespace (via prefix) and
+name of the layer.
+
+A layer's namespace may define DFDL variables that are used to pass parameters to the layer, or
+to receive results back from the layer (usually for checksum layers).
+
+Lastly, there is the concept of _layer length limiting_.
+
+### Layer Length Limiting
+
+A layer can process an entire input file/stream.
+Such a layer is said to be _unlimited_.
+
+A layer can also have a specific data length built into it.
+The `IPv4Checksum` layer (in the aforementioned
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema) has a fixed length of 20 bytes.
+Such a layer is said to be _self limiting_.
+
+More commonly, a layer has a limited region within the data stream that it is supposed to describe.
+The restriction of the layer so that it only describes the expected part of the data is called
+_layer limiting_.
+
+There are these ways to do layer limiting, that differ in important ways:
+- using enclosing elements of specified length
+- using a fixedLength layer
+- binding a DFDL variable specific to a particular layer
+- using a boundaryMark layer
+
+#### Layer Limiting using Enclosing Elements of Specified Length
+
+If the `sequence` for a layer is the model group of the
+complex type of an element, and that element has specified length (meaning it has
+`dfdl:lengthKind` property of `explicit`, `implicit`, `prefixed` or `pattern`), then the length
+of the element limits the length of the layer within it.
+
+In this case the length of the layer is limited when parsing, but it is NOT limited when
+unparsing.
+
+#### Layer Limiting using the `fixedLength` Utility Layer
+
+The layer `fixedLength` is a utility layer intended for use in conjunction with other layers.
+It dictates the parse _and unparse_ length to be the value of the `layerLength` DFDL variable.
+Setting or binding this DFDL variable fixes the length of the `fixedLength` layer, and then
+any layer on top of that layer will be limited by the underlying fixed length.
+
+#### Layer Limiting by Binding a DFDL Variable
+
+Some layers will define a DFDL variable that must be set to a non-negative
+integer value to specify the layer length that the layer will use.
+All checksum layers built with the Daffodil API `ChecksumLayer` base class,
+use the DFDL `layerLength` variable to specify the length (in bytes) of the layer.
+
+Layers may specify restrictions on the minimum and maximum allowed values of these lengths,
+and passing an out-of-range value for the variable is a processing error.
+
+#### Layer Limiting using the `boundaryMark` Utility Layer
+
+The `boundaryMark` layer uses a variable named `boundaryMark` which provides a string parameter
+to the layer.
+This string is a delimiter of the layer data when parsing, and when unparsing the string is
+inserted after the (otherwise unbounded length) layer data.
+Unlike DFDL's normal delimiter properties( `dfdl:separator`, `dfdl:terminator`, and
+`dfdl:initiator`), the scanning for a boundary mark string does not use any escape scheme.
+
+
+## Example: Line Folding
+
+Consider the line folding layer, specifically the `lineFolded_IMF` layer,
+which is built-in to Daffodil.
+
+Line folding is a way of encoding textual data formats so that no line of text is longer than
+a limited line length.
+
+Consider this data :
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem
+ ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+This data has been _line folded_ at roughly 72 characters by inserting a CRLF
+before an existing space in the data.
+Each line ends with a CRLF (\r\n) and the second through fourth lines begin
+with a space as a way of indicating that they are extension lines.
+This data is supposed to be reassembled to form a long single-line string by removing
+all CRLF pairs.
+
+The result should be this single longer string which does not contain any line endings:
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+To achieve this we would use the `lineFolded_IMF` layer.
+This layer has a specific namespace which our DFDL schema will define a prefix `lf` for like this:
+```xml
+xmlns:lf="urn:org.apache.daffodil.layers.lineFolded"
+```
+Our DFDL schema will import the layer schema for the line-folded layer with this import:
+```xml
+<import namespace="urn:org.apache.daffodil.layers.lineFolded"
+ schemaLocation="/org/apache/daffodil/layers/xsd/lineFoldedLayer.dfdl.xsd"/>
+```
+Then the layer is incorporated into our DFDL schema like this:
+```xml
+<sequence dfdlx:layer="lf:lineFolded_IMF">
+ ... elements to be parsed from the unfolded layer data go here ...
+</sequence>
+```
+You can see that use of a layer is described using the `dfdx:layer` property, and the specific layer
+is identified by a QName using the previously defined namespace prefix.
+The scope of the layer is the duration of the sequence it appears on.
+The `dfdlx:layer` property can _only_ be used on an XSD `sequence`.
+
+The `lineFolded_IMF` layer does not define any DFDL variables in its namespace as it has no
+parameters and produces no results.
+As of Daffodil 3.8.0, the `import` statement above is optional for the
+line-folded layers as the DFDL schema file `lineFoldedLayer.dfdl.xsd` does not contain any
+definitions.
+In the future however, parameters may be added, so for uniformity all layers define a DFDL
+schema to be imported as part of using the layer.
+
+If the line-folding applies to the entire data stream (not uncommon for formats that use line
+folding), then what has been described is all one needs to describe the layer aspects of a data
+format.
+Other examples will show how the layer length can be limited to a sub-region of the data.
+
+More detailed documentation for the [Line Folded Layers](#LineFoldedLayers) is below.
+
+## Example: Base64, GZip, and BoundaryMark Layers used Together
+
+In this example, the data consists of a preliminary string, a section of CSV-like data, and a
+final string element.
+The CSV-like data section in the middle is both gzipped, and then base64 encoded, with the
+boundary value mark of "`=_END_=`" indicating where the base64 part ends.
+
+The point of the preliminary string and the final string is to show that this sort of layering
+composes properly with surrounding data elements in the schema. Those strings could be anything
+at all, they're just strings to make this example simpler.
+
+The physical data string/stream looks like this:
+```text
+example of data before the base64 gzipped part
+H4sIAAAAAAAA/y3JQQqAIBCF4b1nmYG0iNxG+84wpqFhI4xuun0Gbd4P78tUG5xJ+t7J+xxg21dV
+79QiSHFBGjgh9oVB22XBYUQzqatErv36CjGwPJ/OOGg0Y1cOFUgegoPaLwa1VS9htd+UbgAAAA==
+=_END_=
+example of data after the base64 gzipped part
+```
+That base64 encoded part, if decoded and decompressed looks like this CSV-like data:
+```
+last,first,middle,DOB
+smith,robert,brandon,1988-03-24
+johnson,john,henry,1986-01-23
+jones,arya,cat,1986-02-19
+```
+
+The Infoset, presented as XML, that we will create for this entire data stream
+using Daffodil parsing looks like:
+```xml
+<ex:data>
+ <before>example of data before the base64 gzipped part</before>
+ <header><title>last</title><title>first</title><title>middle</title><title>DOB</title></header>
+ <record><item>smith</item><item>robert</item><item>brandon</item><item>1988-03-24</item></record>
+ <record><item>johnson</item><item>john</item><item>henry</item><item>1986-01-23</item></record>
+ <record><item>jones</item><item>arya</item><item>cat</item><item>1986-02-19</item></record>
+ <after>example of data after the base64 gzipped part</after>
+</ex:data>
+```
+
+With that characterization of the problem, let's look at the DFDL schema with the Daffodil
+layering properties to do the boundaryMark, base64 decode, and gzip decompression.
+(Note that each of these layers is described further in its own section below on this page.)
+
+First there is the start of the schema which introduces namespace prefix definitions for DFDL,
+for the DFDL extensions (dfdlx) prefix which is used for the `dfdlx:layer` property.
+
+This also sets up 3 namespace prefixes for the namespaces of the boundaryMark (bm), gzip (gz)
+and base64_MIME (b64) layers which will be imported just after this header.
+
+```xml
+<schema
+ xmlns="http://www.w3.org/2001/XMLSchema"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:fn="http://www.w3.org/2005/xpath-functions"
+ xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions"
+ xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:bm="urn:org.apache.daffodil.layers.boundaryMark"
+ xmlns:b64="urn:org.apache.daffodil.layers.base64_MIME"
+ xmlns:gz="urn:org.apache.daffodil.layers.gzip"
+ xmlns:ex="http://example.com"
+ targetNamespace="http://example.com">
+```
+Next we import the Daffodil built in general format which provides most DFDL properties with
+sensible starting values, and import the namespaces of our boundaryMark, gzip, and base64_MIME
+layers, and lastly, set the default format properties for use by this schema file:
+```xml
+<include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+
+<import namespace="urn:org.apache.daffodil.layers.boundaryMark"
+ schemaLocation="/org/apache/daffodil/layers/xsd/boundaryMarkLayer.dfdl.xsd"/>
+<import namespace="urn:org.apache.daffodil.layers.gzip"
+ schemaLocation="/org/apache/daffodil/layers/xsd/gzipLayer.dfdl.xsd"/>
+<import namespace="urn:org.apache.daffodil.layers.base64_MIME"
+ schemaLocation="/org/apache/daffodil/layers/xsd/base64_MIMELayer.dfdl.xsd"/>
+
+<annotation><appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:format ref="ex:GeneralFormat" representation="binary"/>
+</appinfo></annotation>
+```
+The root element of the schema is `data`, and this element has type `dataType`.
+(A separate named complex type definition for the root element is a
+recommended style for DFDL schemas as it allows someone else to reuse the schema without
+getting involved in element namespace prefix complexities.)
+The `dataType` just shows us the `before` and `after` elements, and references a group
+named `layeredDataGroup` for the layered part in between.
+```xml
+<element name="data" type="ex:dataType"/>
+
+<complexType name="dataType">
+ <sequence dfdl:separator="%NL;">
+ <element name="before" type="string" dfdl:lengthKind="delimited"/>
+ <group ref="ex:layeredDataGroup"/>
+ <element name="after" type="string" dfdl:lengthKind="delimited"/>
+ </sequence>
+</complexType>
+```
+Now we see the definition of the layering.
+All 3 layers are declared in this named group.
+The lines here are numbered so that the text that follows can refer to them:
+```xml
+1 <group name="layeredDataGroup">
+2 <sequence dfdlx:layer="bm:boundaryMark">
+3 <annotation>
+4 <appinfo source="http://www.ogf.org/dfdl/">
+5 <dfdl:newVariableInstance ref="bm:layerEncoding" defaultValue="iso-8859-1"/>
+6 <dfdl:newVariableInstance ref="bm:boundaryMark" defaultValue="=_END_="/>
+7 </appinfo>
+8 </annotation>
+9 <sequence dfdlx:layer="b64:base64_MIME">
+10 <sequence dfdlx:layer="gz:gzip">
+11 <group ref="ex:dataFieldsGroup"/>
+12 </sequence>
+13 </sequence>
+14 </sequence>
+15 </group>
+```
+In the above, at line 2 we see the sequence declares the boundaryMark layer.
+This layer requires 2 parameters to be passed by binding DFDL variables, which occurs on
+lines 5 and 6.
+Line 5 binds the `layerEncoding` variable and note the prefix since DFDL variable names can be
+in a specific namespace.
+Line 6 binds the `boundaryMark` variable to the string "`=_END_=`".
+Note that the variable and the layer both are named `boundaryMark`, but this is just coincidence.
+Layers do not have to have parameters that match their name, nor even any parameters at all.
+The boundary mark is the string which marks the end of the data based on scanning for that ending
+mark.
+
+The data region now being isolated, the `boundaryMark` layer contains just one sequence, and line
+9 which itself declares that the data is base64 encoded by specifying the base64_MIME layer.
+This layer has no parameters, so there are no `dfdl:newVariableInstance` statements to bind any
+variables.
+The region of data that is base64 encoded is then described by the sequence at line 10 which
+declares that the data is compressed by the `gzip` algorithm.
+
+Finally, at line 11 we reference a group which describes what the data format of the underlying
+CSV-like data.
+That group's definition is regular DFDL without any of the complexity of layers.
+This is useful for testing, as data can be parsed using this group (in the complexType of an
+element) to ensure it works properly without involving the complexity of layers.
+This group definition is the last thing in the schema:
+```xml
+<group name="dataFieldsGroup">
+ <sequence dfdl:separator="%NL;" dfdl:separatorPosition="postfix">
+ <element name="header" minOccurs="0" dfdl:occursCountKind="implicit">
+ <complexType>
+ <sequence dfdl:separator=",">
+ <element name="title" type="xs:string" maxOccurs="unbounded" dfdl:lengthKind="delimited"/>
+ </sequence>
+ </complexType>
+ </element>
+ <element name="record" maxOccurs="unbounded">
+ <complexType>
+ <sequence dfdl:separator=",">
+ <element name="item" type="xs:string" maxOccurs="unbounded"
+ dfdl:lengthKind="delimited"
+ dfdl:occursCount="{ fn:count(../../header/title) }"
+ dfdl:occursCountKind="expression"/>
+ </sequence>
+ </complexType>
+ </element>
+ </sequence>
+</group>
+
+</schema>
+```
+The above schema works both to parse, but also to unparse this data.
+
+----
+
+# Using Custom Plug-In Layers
+
+A custom plug-in layer is used in the same manner as the built-in Daffodil layers with just a few
+additional details:
+1. The custom layer will have a DFDL schema that declares any DFDL variables it uses, and
+ defines any data formats needed to facilitate use of the layer.
+ This schema will need to be included/imported.
+ Normally this schema would be a small single-file schema and it can be loaded from the
+ classpath.
+2. The custom layer's compiled Java/Scala code is dynamically loaded using the
+ Java Service Provider Interface (SPI).
+ Hence, they must be compiled into Jar files with specific META-INF metadata enclosed, and
+ these jars must appear on the Java CLASSPATH so that they can be found and loaded.
+
+The layer API is defined via Java clases and interfaces to enable writing of custom layers in
+either Java or Scala.
+(One of the built-in layers (Gzip) is written in Java now, by way of proving
+that one can write a Layer in Java.)
+
+Transformer layer classes are derived from the
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html)
+base class.
+
+Checksum layer classes are derived from the
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.html)
+base class.
+
+Further details on how to define custom plug-in layers is in the Javadoc for the
+[Layer API](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/package-summary.html)
+
+----
+
+# Daffodil Built-In Layer Documentation
+
+Each of the layers built-in to the Daffodil implementation are documented in a section below
+which gives the name, namespace, variables, and some usage notes.
+
+----
+
+## Base64 MIME Layer
Review Comment:
These should all be `###`
##########
site/layers.md:
##########
@@ -0,0 +1,655 @@
+---
+description: Pluggable Extensions to Enable Algorithmic Transformations
+ in DFDL
+group: 'nav-right'
+layout: page
+title: 'Layers - Algorithmic
+ Extensions for DFDL'
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+### Table of Contents
+{:.no_toc}
+
+1. use ordered table of contents
+{:toc}
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+Layers are _byte oriented_.
+The smallest non-zero layer size is a single byte, and the length of
+all layers are in bytes.
+Throughout this discussion of layers, when length is discussed, the units of measure are always
+bytes.
+
+Layers can be composed together, that is, with one on top of another, and their composition is
+orthogonal (as in non-interacting).
+This is actually very common. For example, binary data being included into a textual data format
+can be both gzip compressed, and then base64 encoded, which requires 2 layers to describe.
+There is no limit to this depth.
+
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses
+multiple layers together.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- [base64_MIME](#base64-mime-layer)
+- [fourbyteswap](#byte-swapping-layers)
+- [twobyteswap](#byte-swapping-layers)
+- [gzip](#gzip-layer)
+- [lineFolded_IMF](#line-folded-layers)
+- [lineFolded_iCalendar](#line-folded-layers)
+
+Daffodil also includes two utility layers that are used in combination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- [boundaryMark](#boundaryMark-layer)
+- [fixedLength](#fixedLength-layer)
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+### Layer Kinds: Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html) class
+which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic
+Redundancy Check (CRC) over the data stream.
+
+The value of the checksum (or hash or CRC) is assigned to a DFDL variable as the result of the
+layer. This makes the value available for use by the DFDL schema that uses the checksum layer.
+When parsing, the value of this DFDL variable can then be compared to a checksum field in the
+data, and either an invalid data element or an parse-error can be created if the checksum in the
+data stream does not match the computed value.
+When unparsing, the value of this DFDL variable can be written to an element using the
+`dfdl:outputValueCalc` property.
+
+An example of a checksum layer plug-in is in the
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header checksum algorithm.
+
+Custom checksum layers are created by deriving an implementation class from the Daffodil API's
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.
+html)
+class, which is introduced in a later section.
+
+# Using Layers
+
+To use a layer you must know
+- the layer's namespace URI
+- the layer's name
+- the names of any layer parameter variables
+- the names of any layer result variables
+
+Each layer normally has a _layer schema file_ (.dfdl.xsd) which contains the DFDL variable's
+`dfdl:defineVariable` declarations.
+
+A `sequence` group encloses the scope, within the DFDL schema, of the layer.
+Often layers are used to describe checksums or decding/encoding on small regions within a
+larger data format.
+
+The DFDL extension property `dfdlx:layer` declares that the data underlying a `sequence` is to
+use a layer. The value of the property is a QName identifying the namespace (via prefix) and
+name of the layer.
+
+A layer's namespace may define DFDL variables that are used to pass parameters to the layer, or
+to receive results back from the layer (usually for checksum layers).
+
+Lastly, there is the concept of _layer length limiting_.
+
+### Layer Length Limiting
+
+A layer can process an entire input file/stream.
+Such a layer is said to be _unlimited_.
+
+A layer can also have a specific data length built into it.
+The `IPv4Checksum` layer (in the aforementioned
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema) has a fixed length of 20 bytes.
+Such a layer is said to be _self limiting_.
+
+More commonly, a layer has a limited region within the data stream that it is supposed to describe.
+The restriction of the layer so that it only describes the expected part of the data is called
+_layer limiting_.
+
+There are these ways to do layer limiting, that differ in important ways:
+- using enclosing elements of specified length
+- using a fixedLength layer
+- binding a DFDL variable specific to a particular layer
+- using a boundaryMark layer
+
+#### Layer Limiting using Enclosing Elements of Specified Length
+
+If the `sequence` for a layer is the model group of the
+complex type of an element, and that element has specified length (meaning it has
+`dfdl:lengthKind` property of `explicit`, `implicit`, `prefixed` or `pattern`), then the length
+of the element limits the length of the layer within it.
+
+In this case the length of the layer is limited when parsing, but it is NOT limited when
+unparsing.
+
+#### Layer Limiting using the `fixedLength` Utility Layer
+
+The layer `fixedLength` is a utility layer intended for use in conjunction with other layers.
+It dictates the parse _and unparse_ length to be the value of the `layerLength` DFDL variable.
+Setting or binding this DFDL variable fixes the length of the `fixedLength` layer, and then
+any layer on top of that layer will be limited by the underlying fixed length.
+
+#### Layer Limiting by Binding a DFDL Variable
+
+Some layers will define a DFDL variable that must be set to a non-negative
+integer value to specify the layer length that the layer will use.
+All checksum layers built with the Daffodil API `ChecksumLayer` base class,
+use the DFDL `layerLength` variable to specify the length (in bytes) of the layer.
+
+Layers may specify restrictions on the minimum and maximum allowed values of these lengths,
+and passing an out-of-range value for the variable is a processing error.
+
+#### Layer Limiting using the `boundaryMark` Utility Layer
+
+The `boundaryMark` layer uses a variable named `boundaryMark` which provides a string parameter
+to the layer.
+This string is a delimiter of the layer data when parsing, and when unparsing the string is
+inserted after the (otherwise unbounded length) layer data.
+Unlike DFDL's normal delimiter properties( `dfdl:separator`, `dfdl:terminator`, and
+`dfdl:initiator`), the scanning for a boundary mark string does not use any escape scheme.
+
+
+## Example: Line Folding
+
+Consider the line folding layer, specifically the `lineFolded_IMF` layer,
+which is built-in to Daffodil.
+
+Line folding is a way of encoding textual data formats so that no line of text is longer than
+a limited line length.
+
+Consider this data :
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem
+ ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+This data has been _line folded_ at roughly 72 characters by inserting a CRLF
+before an existing space in the data.
+Each line ends with a CRLF (\r\n) and the second through fourth lines begin
+with a space as a way of indicating that they are extension lines.
+This data is supposed to be reassembled to form a long single-line string by removing
+all CRLF pairs.
+
+The result should be this single longer string which does not contain any line endings:
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+To achieve this we would use the `lineFolded_IMF` layer.
+This layer has a specific namespace which our DFDL schema will define a prefix `lf` for like this:
+```xml
+xmlns:lf="urn:org.apache.daffodil.layers.lineFolded"
+```
+Our DFDL schema will import the layer schema for the line-folded layer with this import:
+```xml
+<import namespace="urn:org.apache.daffodil.layers.lineFolded"
+ schemaLocation="/org/apache/daffodil/layers/xsd/lineFoldedLayer.dfdl.xsd"/>
+```
+Then the layer is incorporated into our DFDL schema like this:
+```xml
+<sequence dfdlx:layer="lf:lineFolded_IMF">
+ ... elements to be parsed from the unfolded layer data go here ...
+</sequence>
+```
+You can see that use of a layer is described using the `dfdx:layer` property, and the specific layer
+is identified by a QName using the previously defined namespace prefix.
+The scope of the layer is the duration of the sequence it appears on.
+The `dfdlx:layer` property can _only_ be used on an XSD `sequence`.
+
+The `lineFolded_IMF` layer does not define any DFDL variables in its namespace as it has no
+parameters and produces no results.
+As of Daffodil 3.8.0, the `import` statement above is optional for the
+line-folded layers as the DFDL schema file `lineFoldedLayer.dfdl.xsd` does not contain any
+definitions.
+In the future however, parameters may be added, so for uniformity all layers define a DFDL
+schema to be imported as part of using the layer.
+
+If the line-folding applies to the entire data stream (not uncommon for formats that use line
+folding), then what has been described is all one needs to describe the layer aspects of a data
+format.
+Other examples will show how the layer length can be limited to a sub-region of the data.
+
+More detailed documentation for the [Line Folded Layers](#LineFoldedLayers) is below.
+
+## Example: Base64, GZip, and BoundaryMark Layers used Together
+
+In this example, the data consists of a preliminary string, a section of CSV-like data, and a
+final string element.
+The CSV-like data section in the middle is both gzipped, and then base64 encoded, with the
+boundary value mark of "`=_END_=`" indicating where the base64 part ends.
+
+The point of the preliminary string and the final string is to show that this sort of layering
+composes properly with surrounding data elements in the schema. Those strings could be anything
+at all, they're just strings to make this example simpler.
+
+The physical data string/stream looks like this:
+```text
+example of data before the base64 gzipped part
+H4sIAAAAAAAA/y3JQQqAIBCF4b1nmYG0iNxG+84wpqFhI4xuun0Gbd4P78tUG5xJ+t7J+xxg21dV
+79QiSHFBGjgh9oVB22XBYUQzqatErv36CjGwPJ/OOGg0Y1cOFUgegoPaLwa1VS9htd+UbgAAAA==
+=_END_=
+example of data after the base64 gzipped part
+```
+That base64 encoded part, if decoded and decompressed looks like this CSV-like data:
+```
+last,first,middle,DOB
+smith,robert,brandon,1988-03-24
+johnson,john,henry,1986-01-23
+jones,arya,cat,1986-02-19
+```
+
+The Infoset, presented as XML, that we will create for this entire data stream
+using Daffodil parsing looks like:
+```xml
+<ex:data>
+ <before>example of data before the base64 gzipped part</before>
+ <header><title>last</title><title>first</title><title>middle</title><title>DOB</title></header>
+ <record><item>smith</item><item>robert</item><item>brandon</item><item>1988-03-24</item></record>
+ <record><item>johnson</item><item>john</item><item>henry</item><item>1986-01-23</item></record>
+ <record><item>jones</item><item>arya</item><item>cat</item><item>1986-02-19</item></record>
+ <after>example of data after the base64 gzipped part</after>
+</ex:data>
+```
+
+With that characterization of the problem, let's look at the DFDL schema with the Daffodil
+layering properties to do the boundaryMark, base64 decode, and gzip decompression.
+(Note that each of these layers is described further in its own section below on this page.)
+
+First there is the start of the schema which introduces namespace prefix definitions for DFDL,
+for the DFDL extensions (dfdlx) prefix which is used for the `dfdlx:layer` property.
+
+This also sets up 3 namespace prefixes for the namespaces of the boundaryMark (bm), gzip (gz)
+and base64_MIME (b64) layers which will be imported just after this header.
+
+```xml
+<schema
+ xmlns="http://www.w3.org/2001/XMLSchema"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:fn="http://www.w3.org/2005/xpath-functions"
+ xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions"
+ xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:bm="urn:org.apache.daffodil.layers.boundaryMark"
+ xmlns:b64="urn:org.apache.daffodil.layers.base64_MIME"
+ xmlns:gz="urn:org.apache.daffodil.layers.gzip"
+ xmlns:ex="http://example.com"
+ targetNamespace="http://example.com">
+```
+Next we import the Daffodil built in general format which provides most DFDL properties with
+sensible starting values, and import the namespaces of our boundaryMark, gzip, and base64_MIME
+layers, and lastly, set the default format properties for use by this schema file:
+```xml
+<include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+
+<import namespace="urn:org.apache.daffodil.layers.boundaryMark"
+ schemaLocation="/org/apache/daffodil/layers/xsd/boundaryMarkLayer.dfdl.xsd"/>
+<import namespace="urn:org.apache.daffodil.layers.gzip"
+ schemaLocation="/org/apache/daffodil/layers/xsd/gzipLayer.dfdl.xsd"/>
+<import namespace="urn:org.apache.daffodil.layers.base64_MIME"
+ schemaLocation="/org/apache/daffodil/layers/xsd/base64_MIMELayer.dfdl.xsd"/>
+
+<annotation><appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:format ref="ex:GeneralFormat" representation="binary"/>
+</appinfo></annotation>
+```
+The root element of the schema is `data`, and this element has type `dataType`.
+(A separate named complex type definition for the root element is a
+recommended style for DFDL schemas as it allows someone else to reuse the schema without
+getting involved in element namespace prefix complexities.)
+The `dataType` just shows us the `before` and `after` elements, and references a group
+named `layeredDataGroup` for the layered part in between.
+```xml
+<element name="data" type="ex:dataType"/>
+
+<complexType name="dataType">
+ <sequence dfdl:separator="%NL;">
+ <element name="before" type="string" dfdl:lengthKind="delimited"/>
+ <group ref="ex:layeredDataGroup"/>
+ <element name="after" type="string" dfdl:lengthKind="delimited"/>
+ </sequence>
+</complexType>
+```
+Now we see the definition of the layering.
+All 3 layers are declared in this named group.
+The lines here are numbered so that the text that follows can refer to them:
+```xml
+1 <group name="layeredDataGroup">
+2 <sequence dfdlx:layer="bm:boundaryMark">
+3 <annotation>
+4 <appinfo source="http://www.ogf.org/dfdl/">
+5 <dfdl:newVariableInstance ref="bm:layerEncoding" defaultValue="iso-8859-1"/>
+6 <dfdl:newVariableInstance ref="bm:boundaryMark" defaultValue="=_END_="/>
+7 </appinfo>
+8 </annotation>
+9 <sequence dfdlx:layer="b64:base64_MIME">
+10 <sequence dfdlx:layer="gz:gzip">
+11 <group ref="ex:dataFieldsGroup"/>
+12 </sequence>
+13 </sequence>
+14 </sequence>
+15 </group>
+```
+In the above, at line 2 we see the sequence declares the boundaryMark layer.
+This layer requires 2 parameters to be passed by binding DFDL variables, which occurs on
+lines 5 and 6.
+Line 5 binds the `layerEncoding` variable and note the prefix since DFDL variable names can be
+in a specific namespace.
+Line 6 binds the `boundaryMark` variable to the string "`=_END_=`".
+Note that the variable and the layer both are named `boundaryMark`, but this is just coincidence.
+Layers do not have to have parameters that match their name, nor even any parameters at all.
+The boundary mark is the string which marks the end of the data based on scanning for that ending
+mark.
+
+The data region now being isolated, the `boundaryMark` layer contains just one sequence, and line
+9 which itself declares that the data is base64 encoded by specifying the base64_MIME layer.
+This layer has no parameters, so there are no `dfdl:newVariableInstance` statements to bind any
+variables.
+The region of data that is base64 encoded is then described by the sequence at line 10 which
+declares that the data is compressed by the `gzip` algorithm.
+
+Finally, at line 11 we reference a group which describes what the data format of the underlying
+CSV-like data.
+That group's definition is regular DFDL without any of the complexity of layers.
+This is useful for testing, as data can be parsed using this group (in the complexType of an
+element) to ensure it works properly without involving the complexity of layers.
+This group definition is the last thing in the schema:
+```xml
+<group name="dataFieldsGroup">
+ <sequence dfdl:separator="%NL;" dfdl:separatorPosition="postfix">
+ <element name="header" minOccurs="0" dfdl:occursCountKind="implicit">
+ <complexType>
+ <sequence dfdl:separator=",">
+ <element name="title" type="xs:string" maxOccurs="unbounded" dfdl:lengthKind="delimited"/>
+ </sequence>
+ </complexType>
+ </element>
+ <element name="record" maxOccurs="unbounded">
+ <complexType>
+ <sequence dfdl:separator=",">
+ <element name="item" type="xs:string" maxOccurs="unbounded"
+ dfdl:lengthKind="delimited"
+ dfdl:occursCount="{ fn:count(../../header/title) }"
+ dfdl:occursCountKind="expression"/>
+ </sequence>
+ </complexType>
+ </element>
+ </sequence>
+</group>
+
+</schema>
+```
+The above schema works both to parse, but also to unparse this data.
+
+----
+
+# Using Custom Plug-In Layers
+
+A custom plug-in layer is used in the same manner as the built-in Daffodil layers with just a few
+additional details:
+1. The custom layer will have a DFDL schema that declares any DFDL variables it uses, and
+ defines any data formats needed to facilitate use of the layer.
+ This schema will need to be included/imported.
+ Normally this schema would be a small single-file schema and it can be loaded from the
+ classpath.
+2. The custom layer's compiled Java/Scala code is dynamically loaded using the
+ Java Service Provider Interface (SPI).
+ Hence, they must be compiled into Jar files with specific META-INF metadata enclosed, and
+ these jars must appear on the Java CLASSPATH so that they can be found and loaded.
+
+The layer API is defined via Java clases and interfaces to enable writing of custom layers in
+either Java or Scala.
+(One of the built-in layers (Gzip) is written in Java now, by way of proving
+that one can write a Layer in Java.)
+
+Transformer layer classes are derived from the
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html)
+base class.
+
+Checksum layer classes are derived from the
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.html)
+base class.
+
+Further details on how to define custom plug-in layers is in the Javadoc for the
+[Layer API](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/package-summary.html)
Review Comment:
Makes these links absolute.
##########
site/layers.md:
##########
@@ -0,0 +1,655 @@
+---
+description: Pluggable Extensions to Enable Algorithmic Transformations
+ in DFDL
+group: 'nav-right'
+layout: page
+title: 'Layers - Algorithmic
+ Extensions for DFDL'
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+### Table of Contents
+{:.no_toc}
+
+1. use ordered table of contents
+{:toc}
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+Layers are _byte oriented_.
+The smallest non-zero layer size is a single byte, and the length of
+all layers are in bytes.
+Throughout this discussion of layers, when length is discussed, the units of measure are always
+bytes.
+
+Layers can be composed together, that is, with one on top of another, and their composition is
+orthogonal (as in non-interacting).
+This is actually very common. For example, binary data being included into a textual data format
+can be both gzip compressed, and then base64 encoded, which requires 2 layers to describe.
+There is no limit to this depth.
+
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses
+multiple layers together.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- [base64_MIME](#base64-mime-layer)
+- [fourbyteswap](#byte-swapping-layers)
+- [twobyteswap](#byte-swapping-layers)
+- [gzip](#gzip-layer)
+- [lineFolded_IMF](#line-folded-layers)
+- [lineFolded_iCalendar](#line-folded-layers)
+
+Daffodil also includes two utility layers that are used in combination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- [boundaryMark](#boundaryMark-layer)
+- [fixedLength](#fixedLength-layer)
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+### Layer Kinds: Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html) class
+which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic
+Redundancy Check (CRC) over the data stream.
+
+The value of the checksum (or hash or CRC) is assigned to a DFDL variable as the result of the
+layer. This makes the value available for use by the DFDL schema that uses the checksum layer.
+When parsing, the value of this DFDL variable can then be compared to a checksum field in the
+data, and either an invalid data element or an parse-error can be created if the checksum in the
+data stream does not match the computed value.
+When unparsing, the value of this DFDL variable can be written to an element using the
+`dfdl:outputValueCalc` property.
+
+An example of a checksum layer plug-in is in the
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header checksum algorithm.
+
+Custom checksum layers are created by deriving an implementation class from the Daffodil API's
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.
+html)
+class, which is introduced in a later section.
+
+# Using Layers
+
+To use a layer you must know
+- the layer's namespace URI
+- the layer's name
+- the names of any layer parameter variables
+- the names of any layer result variables
+
+Each layer normally has a _layer schema file_ (.dfdl.xsd) which contains the DFDL variable's
+`dfdl:defineVariable` declarations.
+
+A `sequence` group encloses the scope, within the DFDL schema, of the layer.
+Often layers are used to describe checksums or decding/encoding on small regions within a
+larger data format.
+
+The DFDL extension property `dfdlx:layer` declares that the data underlying a `sequence` is to
+use a layer. The value of the property is a QName identifying the namespace (via prefix) and
+name of the layer.
+
+A layer's namespace may define DFDL variables that are used to pass parameters to the layer, or
+to receive results back from the layer (usually for checksum layers).
+
+Lastly, there is the concept of _layer length limiting_.
+
+### Layer Length Limiting
+
+A layer can process an entire input file/stream.
+Such a layer is said to be _unlimited_.
+
+A layer can also have a specific data length built into it.
+The `IPv4Checksum` layer (in the aforementioned
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema) has a fixed length of 20 bytes.
+Such a layer is said to be _self limiting_.
+
+More commonly, a layer has a limited region within the data stream that it is supposed to describe.
+The restriction of the layer so that it only describes the expected part of the data is called
+_layer limiting_.
+
+There are these ways to do layer limiting, that differ in important ways:
+- using enclosing elements of specified length
+- using a fixedLength layer
+- binding a DFDL variable specific to a particular layer
+- using a boundaryMark layer
+
+#### Layer Limiting using Enclosing Elements of Specified Length
+
+If the `sequence` for a layer is the model group of the
+complex type of an element, and that element has specified length (meaning it has
+`dfdl:lengthKind` property of `explicit`, `implicit`, `prefixed` or `pattern`), then the length
+of the element limits the length of the layer within it.
+
+In this case the length of the layer is limited when parsing, but it is NOT limited when
+unparsing.
+
+#### Layer Limiting using the `fixedLength` Utility Layer
+
+The layer `fixedLength` is a utility layer intended for use in conjunction with other layers.
+It dictates the parse _and unparse_ length to be the value of the `layerLength` DFDL variable.
+Setting or binding this DFDL variable fixes the length of the `fixedLength` layer, and then
+any layer on top of that layer will be limited by the underlying fixed length.
+
+#### Layer Limiting by Binding a DFDL Variable
+
+Some layers will define a DFDL variable that must be set to a non-negative
+integer value to specify the layer length that the layer will use.
+All checksum layers built with the Daffodil API `ChecksumLayer` base class,
+use the DFDL `layerLength` variable to specify the length (in bytes) of the layer.
+
+Layers may specify restrictions on the minimum and maximum allowed values of these lengths,
+and passing an out-of-range value for the variable is a processing error.
+
+#### Layer Limiting using the `boundaryMark` Utility Layer
+
+The `boundaryMark` layer uses a variable named `boundaryMark` which provides a string parameter
+to the layer.
+This string is a delimiter of the layer data when parsing, and when unparsing the string is
+inserted after the (otherwise unbounded length) layer data.
+Unlike DFDL's normal delimiter properties( `dfdl:separator`, `dfdl:terminator`, and
+`dfdl:initiator`), the scanning for a boundary mark string does not use any escape scheme.
+
+
+## Example: Line Folding
+
+Consider the line folding layer, specifically the `lineFolded_IMF` layer,
+which is built-in to Daffodil.
+
+Line folding is a way of encoding textual data formats so that no line of text is longer than
+a limited line length.
+
+Consider this data :
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem
+ ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+This data has been _line folded_ at roughly 72 characters by inserting a CRLF
+before an existing space in the data.
+Each line ends with a CRLF (\r\n) and the second through fourth lines begin
+with a space as a way of indicating that they are extension lines.
+This data is supposed to be reassembled to form a long single-line string by removing
+all CRLF pairs.
+
+The result should be this single longer string which does not contain any line endings:
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+To achieve this we would use the `lineFolded_IMF` layer.
+This layer has a specific namespace which our DFDL schema will define a prefix `lf` for like this:
+```xml
+xmlns:lf="urn:org.apache.daffodil.layers.lineFolded"
+```
+Our DFDL schema will import the layer schema for the line-folded layer with this import:
+```xml
+<import namespace="urn:org.apache.daffodil.layers.lineFolded"
+ schemaLocation="/org/apache/daffodil/layers/xsd/lineFoldedLayer.dfdl.xsd"/>
+```
+Then the layer is incorporated into our DFDL schema like this:
+```xml
+<sequence dfdlx:layer="lf:lineFolded_IMF">
+ ... elements to be parsed from the unfolded layer data go here ...
+</sequence>
+```
+You can see that use of a layer is described using the `dfdx:layer` property, and the specific layer
Review Comment:
s/dfdx/dfdlx
##########
site/tutorials/DFDLTutorialStylesheet.xsl:
##########
@@ -207,20 +215,20 @@
</nav>
</div>
</div>
- <div class="title">
- <div class="container">
- <h2><xsl:value-of select="tdml:testSuite/@suiteName"/></h2>
- </div>
- </div>
+ <div class="title">
<div class="container">
- <div class="row">
- <div class="col-md-12">
- <section>
- <xsl:apply-templates select="*"/>
- </section>
- </div>
- </div>
- <footer class="pt-footer">
+ <h2>
+ <xsl:value-of select="tdml:testSuite/@suiteName"/>
+ </h2>
+ </div>
+ </div>
+ <div class="container">
+ <div class="row">
+ <section>
+ <xsl:apply-templates select="*"/>
+ </section>
+ </div>
+ <footer class="pt-footer">
<footer class="site-footer">
Review Comment:
The release candidate process copies the `tutorials` files from the daffodil repo, so these changes will be overwritten when we make a release. This change probably wants to be reverted and applied to the daffodil repo instead. When we do the 3.8.0 release they'll be copied over.
##########
site/layers.md:
##########
@@ -0,0 +1,655 @@
+---
+description: Pluggable Extensions to Enable Algorithmic Transformations
+ in DFDL
+group: 'nav-right'
+layout: page
+title: 'Layers - Algorithmic
+ Extensions for DFDL'
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+### Table of Contents
+{:.no_toc}
+
+1. use ordered table of contents
+{:toc}
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+Layers are _byte oriented_.
+The smallest non-zero layer size is a single byte, and the length of
+all layers are in bytes.
+Throughout this discussion of layers, when length is discussed, the units of measure are always
+bytes.
+
+Layers can be composed together, that is, with one on top of another, and their composition is
+orthogonal (as in non-interacting).
+This is actually very common. For example, binary data being included into a textual data format
+can be both gzip compressed, and then base64 encoded, which requires 2 layers to describe.
+There is no limit to this depth.
+
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses
+multiple layers together.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- [base64_MIME](#base64-mime-layer)
+- [fourbyteswap](#byte-swapping-layers)
+- [twobyteswap](#byte-swapping-layers)
+- [gzip](#gzip-layer)
+- [lineFolded_IMF](#line-folded-layers)
+- [lineFolded_iCalendar](#line-folded-layers)
+
+Daffodil also includes two utility layers that are used in combination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- [boundaryMark](#boundaryMark-layer)
+- [fixedLength](#fixedLength-layer)
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+### Layer Kinds: Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html) class
+which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic
+Redundancy Check (CRC) over the data stream.
+
+The value of the checksum (or hash or CRC) is assigned to a DFDL variable as the result of the
+layer. This makes the value available for use by the DFDL schema that uses the checksum layer.
+When parsing, the value of this DFDL variable can then be compared to a checksum field in the
+data, and either an invalid data element or an parse-error can be created if the checksum in the
+data stream does not match the computed value.
+When unparsing, the value of this DFDL variable can be written to an element using the
+`dfdl:outputValueCalc` property.
+
+An example of a checksum layer plug-in is in the
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header checksum algorithm.
+
+Custom checksum layers are created by deriving an implementation class from the Daffodil API's
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.
+html)
+class, which is introduced in a later section.
+
+# Using Layers
+
+To use a layer you must know
+- the layer's namespace URI
+- the layer's name
+- the names of any layer parameter variables
+- the names of any layer result variables
+
+Each layer normally has a _layer schema file_ (.dfdl.xsd) which contains the DFDL variable's
+`dfdl:defineVariable` declarations.
+
+A `sequence` group encloses the scope, within the DFDL schema, of the layer.
+Often layers are used to describe checksums or decding/encoding on small regions within a
+larger data format.
+
+The DFDL extension property `dfdlx:layer` declares that the data underlying a `sequence` is to
+use a layer. The value of the property is a QName identifying the namespace (via prefix) and
+name of the layer.
+
+A layer's namespace may define DFDL variables that are used to pass parameters to the layer, or
+to receive results back from the layer (usually for checksum layers).
+
+Lastly, there is the concept of _layer length limiting_.
+
+### Layer Length Limiting
+
+A layer can process an entire input file/stream.
+Such a layer is said to be _unlimited_.
+
+A layer can also have a specific data length built into it.
+The `IPv4Checksum` layer (in the aforementioned
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema) has a fixed length of 20 bytes.
+Such a layer is said to be _self limiting_.
+
+More commonly, a layer has a limited region within the data stream that it is supposed to describe.
+The restriction of the layer so that it only describes the expected part of the data is called
+_layer limiting_.
+
+There are these ways to do layer limiting, that differ in important ways:
+- using enclosing elements of specified length
+- using a fixedLength layer
+- binding a DFDL variable specific to a particular layer
+- using a boundaryMark layer
+
+#### Layer Limiting using Enclosing Elements of Specified Length
+
+If the `sequence` for a layer is the model group of the
+complex type of an element, and that element has specified length (meaning it has
+`dfdl:lengthKind` property of `explicit`, `implicit`, `prefixed` or `pattern`), then the length
+of the element limits the length of the layer within it.
+
+In this case the length of the layer is limited when parsing, but it is NOT limited when
+unparsing.
+
+#### Layer Limiting using the `fixedLength` Utility Layer
+
+The layer `fixedLength` is a utility layer intended for use in conjunction with other layers.
+It dictates the parse _and unparse_ length to be the value of the `layerLength` DFDL variable.
+Setting or binding this DFDL variable fixes the length of the `fixedLength` layer, and then
+any layer on top of that layer will be limited by the underlying fixed length.
+
+#### Layer Limiting by Binding a DFDL Variable
+
+Some layers will define a DFDL variable that must be set to a non-negative
+integer value to specify the layer length that the layer will use.
+All checksum layers built with the Daffodil API `ChecksumLayer` base class,
+use the DFDL `layerLength` variable to specify the length (in bytes) of the layer.
+
+Layers may specify restrictions on the minimum and maximum allowed values of these lengths,
+and passing an out-of-range value for the variable is a processing error.
+
+#### Layer Limiting using the `boundaryMark` Utility Layer
+
+The `boundaryMark` layer uses a variable named `boundaryMark` which provides a string parameter
+to the layer.
+This string is a delimiter of the layer data when parsing, and when unparsing the string is
+inserted after the (otherwise unbounded length) layer data.
+Unlike DFDL's normal delimiter properties( `dfdl:separator`, `dfdl:terminator`, and
+`dfdl:initiator`), the scanning for a boundary mark string does not use any escape scheme.
+
+
+## Example: Line Folding
+
+Consider the line folding layer, specifically the `lineFolded_IMF` layer,
+which is built-in to Daffodil.
+
+Line folding is a way of encoding textual data formats so that no line of text is longer than
+a limited line length.
+
+Consider this data :
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem
+ ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+This data has been _line folded_ at roughly 72 characters by inserting a CRLF
+before an existing space in the data.
+Each line ends with a CRLF (\r\n) and the second through fourth lines begin
+with a space as a way of indicating that they are extension lines.
+This data is supposed to be reassembled to form a long single-line string by removing
+all CRLF pairs.
+
+The result should be this single longer string which does not contain any line endings:
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+To achieve this we would use the `lineFolded_IMF` layer.
+This layer has a specific namespace which our DFDL schema will define a prefix `lf` for like this:
+```xml
+xmlns:lf="urn:org.apache.daffodil.layers.lineFolded"
+```
+Our DFDL schema will import the layer schema for the line-folded layer with this import:
+```xml
+<import namespace="urn:org.apache.daffodil.layers.lineFolded"
+ schemaLocation="/org/apache/daffodil/layers/xsd/lineFoldedLayer.dfdl.xsd"/>
+```
+Then the layer is incorporated into our DFDL schema like this:
+```xml
+<sequence dfdlx:layer="lf:lineFolded_IMF">
+ ... elements to be parsed from the unfolded layer data go here ...
+</sequence>
+```
+You can see that use of a layer is described using the `dfdx:layer` property, and the specific layer
+is identified by a QName using the previously defined namespace prefix.
+The scope of the layer is the duration of the sequence it appears on.
+The `dfdlx:layer` property can _only_ be used on an XSD `sequence`.
+
+The `lineFolded_IMF` layer does not define any DFDL variables in its namespace as it has no
+parameters and produces no results.
+As of Daffodil 3.8.0, the `import` statement above is optional for the
+line-folded layers as the DFDL schema file `lineFoldedLayer.dfdl.xsd` does not contain any
+definitions.
+In the future however, parameters may be added, so for uniformity all layers define a DFDL
+schema to be imported as part of using the layer.
+
+If the line-folding applies to the entire data stream (not uncommon for formats that use line
+folding), then what has been described is all one needs to describe the layer aspects of a data
+format.
+Other examples will show how the layer length can be limited to a sub-region of the data.
+
+More detailed documentation for the [Line Folded Layers](#LineFoldedLayers) is below.
+
+## Example: Base64, GZip, and BoundaryMark Layers used Together
+
+In this example, the data consists of a preliminary string, a section of CSV-like data, and a
+final string element.
+The CSV-like data section in the middle is both gzipped, and then base64 encoded, with the
+boundary value mark of "`=_END_=`" indicating where the base64 part ends.
+
+The point of the preliminary string and the final string is to show that this sort of layering
+composes properly with surrounding data elements in the schema. Those strings could be anything
+at all, they're just strings to make this example simpler.
+
+The physical data string/stream looks like this:
+```text
+example of data before the base64 gzipped part
+H4sIAAAAAAAA/y3JQQqAIBCF4b1nmYG0iNxG+84wpqFhI4xuun0Gbd4P78tUG5xJ+t7J+xxg21dV
+79QiSHFBGjgh9oVB22XBYUQzqatErv36CjGwPJ/OOGg0Y1cOFUgegoPaLwa1VS9htd+UbgAAAA==
+=_END_=
+example of data after the base64 gzipped part
+```
+That base64 encoded part, if decoded and decompressed looks like this CSV-like data:
+```
+last,first,middle,DOB
+smith,robert,brandon,1988-03-24
+johnson,john,henry,1986-01-23
+jones,arya,cat,1986-02-19
+```
+
+The Infoset, presented as XML, that we will create for this entire data stream
+using Daffodil parsing looks like:
+```xml
+<ex:data>
+ <before>example of data before the base64 gzipped part</before>
+ <header><title>last</title><title>first</title><title>middle</title><title>DOB</title></header>
+ <record><item>smith</item><item>robert</item><item>brandon</item><item>1988-03-24</item></record>
+ <record><item>johnson</item><item>john</item><item>henry</item><item>1986-01-23</item></record>
+ <record><item>jones</item><item>arya</item><item>cat</item><item>1986-02-19</item></record>
+ <after>example of data after the base64 gzipped part</after>
+</ex:data>
+```
+
+With that characterization of the problem, let's look at the DFDL schema with the Daffodil
+layering properties to do the boundaryMark, base64 decode, and gzip decompression.
+(Note that each of these layers is described further in its own section below on this page.)
+
+First there is the start of the schema which introduces namespace prefix definitions for DFDL,
+for the DFDL extensions (dfdlx) prefix which is used for the `dfdlx:layer` property.
+
+This also sets up 3 namespace prefixes for the namespaces of the boundaryMark (bm), gzip (gz)
+and base64_MIME (b64) layers which will be imported just after this header.
+
+```xml
+<schema
+ xmlns="http://www.w3.org/2001/XMLSchema"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:fn="http://www.w3.org/2005/xpath-functions"
+ xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions"
+ xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:bm="urn:org.apache.daffodil.layers.boundaryMark"
+ xmlns:b64="urn:org.apache.daffodil.layers.base64_MIME"
+ xmlns:gz="urn:org.apache.daffodil.layers.gzip"
+ xmlns:ex="http://example.com"
+ targetNamespace="http://example.com">
+```
+Next we import the Daffodil built in general format which provides most DFDL properties with
+sensible starting values, and import the namespaces of our boundaryMark, gzip, and base64_MIME
+layers, and lastly, set the default format properties for use by this schema file:
+```xml
+<include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+
+<import namespace="urn:org.apache.daffodil.layers.boundaryMark"
+ schemaLocation="/org/apache/daffodil/layers/xsd/boundaryMarkLayer.dfdl.xsd"/>
+<import namespace="urn:org.apache.daffodil.layers.gzip"
+ schemaLocation="/org/apache/daffodil/layers/xsd/gzipLayer.dfdl.xsd"/>
+<import namespace="urn:org.apache.daffodil.layers.base64_MIME"
+ schemaLocation="/org/apache/daffodil/layers/xsd/base64_MIMELayer.dfdl.xsd"/>
+
+<annotation><appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:format ref="ex:GeneralFormat" representation="binary"/>
+</appinfo></annotation>
+```
+The root element of the schema is `data`, and this element has type `dataType`.
+(A separate named complex type definition for the root element is a
+recommended style for DFDL schemas as it allows someone else to reuse the schema without
+getting involved in element namespace prefix complexities.)
+The `dataType` just shows us the `before` and `after` elements, and references a group
+named `layeredDataGroup` for the layered part in between.
+```xml
+<element name="data" type="ex:dataType"/>
+
+<complexType name="dataType">
+ <sequence dfdl:separator="%NL;">
+ <element name="before" type="string" dfdl:lengthKind="delimited"/>
+ <group ref="ex:layeredDataGroup"/>
+ <element name="after" type="string" dfdl:lengthKind="delimited"/>
+ </sequence>
+</complexType>
+```
+Now we see the definition of the layering.
+All 3 layers are declared in this named group.
+The lines here are numbered so that the text that follows can refer to them:
+```xml
+1 <group name="layeredDataGroup">
+2 <sequence dfdlx:layer="bm:boundaryMark">
+3 <annotation>
+4 <appinfo source="http://www.ogf.org/dfdl/">
+5 <dfdl:newVariableInstance ref="bm:layerEncoding" defaultValue="iso-8859-1"/>
+6 <dfdl:newVariableInstance ref="bm:boundaryMark" defaultValue="=_END_="/>
+7 </appinfo>
+8 </annotation>
+9 <sequence dfdlx:layer="b64:base64_MIME">
+10 <sequence dfdlx:layer="gz:gzip">
+11 <group ref="ex:dataFieldsGroup"/>
+12 </sequence>
+13 </sequence>
+14 </sequence>
+15 </group>
+```
+In the above, at line 2 we see the sequence declares the boundaryMark layer.
+This layer requires 2 parameters to be passed by binding DFDL variables, which occurs on
+lines 5 and 6.
+Line 5 binds the `layerEncoding` variable and note the prefix since DFDL variable names can be
+in a specific namespace.
+Line 6 binds the `boundaryMark` variable to the string "`=_END_=`".
+Note that the variable and the layer both are named `boundaryMark`, but this is just coincidence.
+Layers do not have to have parameters that match their name, nor even any parameters at all.
+The boundary mark is the string which marks the end of the data based on scanning for that ending
+mark.
+
+The data region now being isolated, the `boundaryMark` layer contains just one sequence, and line
+9 which itself declares that the data is base64 encoded by specifying the base64_MIME layer.
+This layer has no parameters, so there are no `dfdl:newVariableInstance` statements to bind any
+variables.
+The region of data that is base64 encoded is then described by the sequence at line 10 which
+declares that the data is compressed by the `gzip` algorithm.
+
+Finally, at line 11 we reference a group which describes what the data format of the underlying
+CSV-like data.
+That group's definition is regular DFDL without any of the complexity of layers.
+This is useful for testing, as data can be parsed using this group (in the complexType of an
+element) to ensure it works properly without involving the complexity of layers.
+This group definition is the last thing in the schema:
+```xml
+<group name="dataFieldsGroup">
+ <sequence dfdl:separator="%NL;" dfdl:separatorPosition="postfix">
+ <element name="header" minOccurs="0" dfdl:occursCountKind="implicit">
+ <complexType>
+ <sequence dfdl:separator=",">
+ <element name="title" type="xs:string" maxOccurs="unbounded" dfdl:lengthKind="delimited"/>
+ </sequence>
+ </complexType>
+ </element>
+ <element name="record" maxOccurs="unbounded">
+ <complexType>
+ <sequence dfdl:separator=",">
+ <element name="item" type="xs:string" maxOccurs="unbounded"
+ dfdl:lengthKind="delimited"
+ dfdl:occursCount="{ fn:count(../../header/title) }"
+ dfdl:occursCountKind="expression"/>
+ </sequence>
+ </complexType>
+ </element>
+ </sequence>
+</group>
+
+</schema>
+```
+The above schema works both to parse, but also to unparse this data.
+
+----
+
+# Using Custom Plug-In Layers
+
+A custom plug-in layer is used in the same manner as the built-in Daffodil layers with just a few
+additional details:
+1. The custom layer will have a DFDL schema that declares any DFDL variables it uses, and
+ defines any data formats needed to facilitate use of the layer.
+ This schema will need to be included/imported.
+ Normally this schema would be a small single-file schema and it can be loaded from the
+ classpath.
+2. The custom layer's compiled Java/Scala code is dynamically loaded using the
+ Java Service Provider Interface (SPI).
+ Hence, they must be compiled into Jar files with specific META-INF metadata enclosed, and
+ these jars must appear on the Java CLASSPATH so that they can be found and loaded.
+
+The layer API is defined via Java clases and interfaces to enable writing of custom layers in
+either Java or Scala.
+(One of the built-in layers (Gzip) is written in Java now, by way of proving
+that one can write a Layer in Java.)
+
+Transformer layer classes are derived from the
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html)
+base class.
+
+Checksum layer classes are derived from the
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.html)
+base class.
+
+Further details on how to define custom plug-in layers is in the Javadoc for the
+[Layer API](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/package-summary.html)
+
+----
+
+# Daffodil Built-In Layer Documentation
+
+Each of the layers built-in to the Daffodil implementation are documented in a section below
+which gives the name, namespace, variables, and some usage notes.
+
+----
+
+## Base64 MIME Layer
+
+- Name: base64_MIME
+- Namespace URI: urn:org.apache.daffodil.layers.base64_MIME
+- No DFDL variables are read or written.
Review Comment:
Is it worth adding a bullet about how each layer is limited, in addition to name, namespace, and variables?
For example, I can't remember if base64 can do some kind of self limiting if it see's the = pad characters, or if it just always decodes everything it can and errors if there's is more data past pad chars.
##########
site/layers.md:
##########
@@ -0,0 +1,655 @@
+---
+description: Pluggable Extensions to Enable Algorithmic Transformations
+ in DFDL
+group: 'nav-right'
+layout: page
+title: 'Layers - Algorithmic
+ Extensions for DFDL'
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+### Table of Contents
+{:.no_toc}
+
+1. use ordered table of contents
+{:toc}
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+Layers are _byte oriented_.
+The smallest non-zero layer size is a single byte, and the length of
+all layers are in bytes.
+Throughout this discussion of layers, when length is discussed, the units of measure are always
+bytes.
+
+Layers can be composed together, that is, with one on top of another, and their composition is
+orthogonal (as in non-interacting).
+This is actually very common. For example, binary data being included into a textual data format
+can be both gzip compressed, and then base64 encoded, which requires 2 layers to describe.
+There is no limit to this depth.
+
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses
+multiple layers together.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- [base64_MIME](#base64-mime-layer)
+- [fourbyteswap](#byte-swapping-layers)
+- [twobyteswap](#byte-swapping-layers)
+- [gzip](#gzip-layer)
+- [lineFolded_IMF](#line-folded-layers)
+- [lineFolded_iCalendar](#line-folded-layers)
+
+Daffodil also includes two utility layers that are used in combination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- [boundaryMark](#boundaryMark-layer)
+- [fixedLength](#fixedLength-layer)
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+### Layer Kinds: Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html) class
+which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic
+Redundancy Check (CRC) over the data stream.
+
+The value of the checksum (or hash or CRC) is assigned to a DFDL variable as the result of the
+layer. This makes the value available for use by the DFDL schema that uses the checksum layer.
+When parsing, the value of this DFDL variable can then be compared to a checksum field in the
+data, and either an invalid data element or an parse-error can be created if the checksum in the
+data stream does not match the computed value.
+When unparsing, the value of this DFDL variable can be written to an element using the
+`dfdl:outputValueCalc` property.
+
+An example of a checksum layer plug-in is in the
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header checksum algorithm.
+
+Custom checksum layers are created by deriving an implementation class from the Daffodil API's
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.
+html)
+class, which is introduced in a later section.
+
+# Using Layers
+
+To use a layer you must know
+- the layer's namespace URI
+- the layer's name
+- the names of any layer parameter variables
+- the names of any layer result variables
+
+Each layer normally has a _layer schema file_ (.dfdl.xsd) which contains the DFDL variable's
+`dfdl:defineVariable` declarations.
+
+A `sequence` group encloses the scope, within the DFDL schema, of the layer.
+Often layers are used to describe checksums or decding/encoding on small regions within a
+larger data format.
+
+The DFDL extension property `dfdlx:layer` declares that the data underlying a `sequence` is to
+use a layer. The value of the property is a QName identifying the namespace (via prefix) and
+name of the layer.
+
+A layer's namespace may define DFDL variables that are used to pass parameters to the layer, or
+to receive results back from the layer (usually for checksum layers).
+
+Lastly, there is the concept of _layer length limiting_.
+
+### Layer Length Limiting
+
+A layer can process an entire input file/stream.
+Such a layer is said to be _unlimited_.
+
+A layer can also have a specific data length built into it.
+The `IPv4Checksum` layer (in the aforementioned
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema) has a fixed length of 20 bytes.
+Such a layer is said to be _self limiting_.
+
+More commonly, a layer has a limited region within the data stream that it is supposed to describe.
+The restriction of the layer so that it only describes the expected part of the data is called
+_layer limiting_.
+
+There are these ways to do layer limiting, that differ in important ways:
+- using enclosing elements of specified length
+- using a fixedLength layer
+- binding a DFDL variable specific to a particular layer
+- using a boundaryMark layer
+
+#### Layer Limiting using Enclosing Elements of Specified Length
+
+If the `sequence` for a layer is the model group of the
+complex type of an element, and that element has specified length (meaning it has
+`dfdl:lengthKind` property of `explicit`, `implicit`, `prefixed` or `pattern`), then the length
+of the element limits the length of the layer within it.
+
+In this case the length of the layer is limited when parsing, but it is NOT limited when
+unparsing.
+
+#### Layer Limiting using the `fixedLength` Utility Layer
+
+The layer `fixedLength` is a utility layer intended for use in conjunction with other layers.
+It dictates the parse _and unparse_ length to be the value of the `layerLength` DFDL variable.
+Setting or binding this DFDL variable fixes the length of the `fixedLength` layer, and then
+any layer on top of that layer will be limited by the underlying fixed length.
+
+#### Layer Limiting by Binding a DFDL Variable
+
+Some layers will define a DFDL variable that must be set to a non-negative
+integer value to specify the layer length that the layer will use.
+All checksum layers built with the Daffodil API `ChecksumLayer` base class,
+use the DFDL `layerLength` variable to specify the length (in bytes) of the layer.
+
+Layers may specify restrictions on the minimum and maximum allowed values of these lengths,
+and passing an out-of-range value for the variable is a processing error.
+
+#### Layer Limiting using the `boundaryMark` Utility Layer
+
+The `boundaryMark` layer uses a variable named `boundaryMark` which provides a string parameter
+to the layer.
+This string is a delimiter of the layer data when parsing, and when unparsing the string is
+inserted after the (otherwise unbounded length) layer data.
+Unlike DFDL's normal delimiter properties( `dfdl:separator`, `dfdl:terminator`, and
+`dfdl:initiator`), the scanning for a boundary mark string does not use any escape scheme.
+
+
+## Example: Line Folding
+
+Consider the line folding layer, specifically the `lineFolded_IMF` layer,
+which is built-in to Daffodil.
+
+Line folding is a way of encoding textual data formats so that no line of text is longer than
+a limited line length.
+
+Consider this data :
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem
+ ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+This data has been _line folded_ at roughly 72 characters by inserting a CRLF
+before an existing space in the data.
+Each line ends with a CRLF (\r\n) and the second through fourth lines begin
+with a space as a way of indicating that they are extension lines.
+This data is supposed to be reassembled to form a long single-line string by removing
+all CRLF pairs.
+
+The result should be this single longer string which does not contain any line endings:
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+To achieve this we would use the `lineFolded_IMF` layer.
+This layer has a specific namespace which our DFDL schema will define a prefix `lf` for like this:
+```xml
+xmlns:lf="urn:org.apache.daffodil.layers.lineFolded"
+```
+Our DFDL schema will import the layer schema for the line-folded layer with this import:
+```xml
+<import namespace="urn:org.apache.daffodil.layers.lineFolded"
+ schemaLocation="/org/apache/daffodil/layers/xsd/lineFoldedLayer.dfdl.xsd"/>
+```
+Then the layer is incorporated into our DFDL schema like this:
+```xml
+<sequence dfdlx:layer="lf:lineFolded_IMF">
+ ... elements to be parsed from the unfolded layer data go here ...
+</sequence>
+```
+You can see that use of a layer is described using the `dfdx:layer` property, and the specific layer
+is identified by a QName using the previously defined namespace prefix.
+The scope of the layer is the duration of the sequence it appears on.
+The `dfdlx:layer` property can _only_ be used on an XSD `sequence`.
+
+The `lineFolded_IMF` layer does not define any DFDL variables in its namespace as it has no
+parameters and produces no results.
+As of Daffodil 3.8.0, the `import` statement above is optional for the
+line-folded layers as the DFDL schema file `lineFoldedLayer.dfdl.xsd` does not contain any
+definitions.
+In the future however, parameters may be added, so for uniformity all layers define a DFDL
+schema to be imported as part of using the layer.
+
+If the line-folding applies to the entire data stream (not uncommon for formats that use line
+folding), then what has been described is all one needs to describe the layer aspects of a data
+format.
+Other examples will show how the layer length can be limited to a sub-region of the data.
+
+More detailed documentation for the [Line Folded Layers](#LineFoldedLayers) is below.
+
+## Example: Base64, GZip, and BoundaryMark Layers used Together
+
+In this example, the data consists of a preliminary string, a section of CSV-like data, and a
+final string element.
+The CSV-like data section in the middle is both gzipped, and then base64 encoded, with the
+boundary value mark of "`=_END_=`" indicating where the base64 part ends.
+
+The point of the preliminary string and the final string is to show that this sort of layering
+composes properly with surrounding data elements in the schema. Those strings could be anything
+at all, they're just strings to make this example simpler.
+
+The physical data string/stream looks like this:
+```text
+example of data before the base64 gzipped part
+H4sIAAAAAAAA/y3JQQqAIBCF4b1nmYG0iNxG+84wpqFhI4xuun0Gbd4P78tUG5xJ+t7J+xxg21dV
+79QiSHFBGjgh9oVB22XBYUQzqatErv36CjGwPJ/OOGg0Y1cOFUgegoPaLwa1VS9htd+UbgAAAA==
+=_END_=
+example of data after the base64 gzipped part
+```
+That base64 encoded part, if decoded and decompressed looks like this CSV-like data:
+```
+last,first,middle,DOB
+smith,robert,brandon,1988-03-24
+johnson,john,henry,1986-01-23
+jones,arya,cat,1986-02-19
+```
+
+The Infoset, presented as XML, that we will create for this entire data stream
+using Daffodil parsing looks like:
+```xml
+<ex:data>
+ <before>example of data before the base64 gzipped part</before>
+ <header><title>last</title><title>first</title><title>middle</title><title>DOB</title></header>
+ <record><item>smith</item><item>robert</item><item>brandon</item><item>1988-03-24</item></record>
+ <record><item>johnson</item><item>john</item><item>henry</item><item>1986-01-23</item></record>
+ <record><item>jones</item><item>arya</item><item>cat</item><item>1986-02-19</item></record>
+ <after>example of data after the base64 gzipped part</after>
+</ex:data>
+```
+
+With that characterization of the problem, let's look at the DFDL schema with the Daffodil
+layering properties to do the boundaryMark, base64 decode, and gzip decompression.
+(Note that each of these layers is described further in its own section below on this page.)
+
+First there is the start of the schema which introduces namespace prefix definitions for DFDL,
+for the DFDL extensions (dfdlx) prefix which is used for the `dfdlx:layer` property.
+
+This also sets up 3 namespace prefixes for the namespaces of the boundaryMark (bm), gzip (gz)
+and base64_MIME (b64) layers which will be imported just after this header.
+
+```xml
+<schema
+ xmlns="http://www.w3.org/2001/XMLSchema"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:fn="http://www.w3.org/2005/xpath-functions"
+ xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions"
+ xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:bm="urn:org.apache.daffodil.layers.boundaryMark"
+ xmlns:b64="urn:org.apache.daffodil.layers.base64_MIME"
+ xmlns:gz="urn:org.apache.daffodil.layers.gzip"
+ xmlns:ex="http://example.com"
+ targetNamespace="http://example.com">
+```
+Next we import the Daffodil built in general format which provides most DFDL properties with
+sensible starting values, and import the namespaces of our boundaryMark, gzip, and base64_MIME
+layers, and lastly, set the default format properties for use by this schema file:
+```xml
+<include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+
+<import namespace="urn:org.apache.daffodil.layers.boundaryMark"
+ schemaLocation="/org/apache/daffodil/layers/xsd/boundaryMarkLayer.dfdl.xsd"/>
+<import namespace="urn:org.apache.daffodil.layers.gzip"
+ schemaLocation="/org/apache/daffodil/layers/xsd/gzipLayer.dfdl.xsd"/>
+<import namespace="urn:org.apache.daffodil.layers.base64_MIME"
+ schemaLocation="/org/apache/daffodil/layers/xsd/base64_MIMELayer.dfdl.xsd"/>
+
+<annotation><appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:format ref="ex:GeneralFormat" representation="binary"/>
+</appinfo></annotation>
+```
+The root element of the schema is `data`, and this element has type `dataType`.
+(A separate named complex type definition for the root element is a
+recommended style for DFDL schemas as it allows someone else to reuse the schema without
+getting involved in element namespace prefix complexities.)
+The `dataType` just shows us the `before` and `after` elements, and references a group
+named `layeredDataGroup` for the layered part in between.
+```xml
+<element name="data" type="ex:dataType"/>
+
+<complexType name="dataType">
+ <sequence dfdl:separator="%NL;">
+ <element name="before" type="string" dfdl:lengthKind="delimited"/>
+ <group ref="ex:layeredDataGroup"/>
+ <element name="after" type="string" dfdl:lengthKind="delimited"/>
+ </sequence>
+</complexType>
+```
+Now we see the definition of the layering.
+All 3 layers are declared in this named group.
+The lines here are numbered so that the text that follows can refer to them:
+```xml
+1 <group name="layeredDataGroup">
+2 <sequence dfdlx:layer="bm:boundaryMark">
+3 <annotation>
+4 <appinfo source="http://www.ogf.org/dfdl/">
+5 <dfdl:newVariableInstance ref="bm:layerEncoding" defaultValue="iso-8859-1"/>
+6 <dfdl:newVariableInstance ref="bm:boundaryMark" defaultValue="=_END_="/>
+7 </appinfo>
+8 </annotation>
+9 <sequence dfdlx:layer="b64:base64_MIME">
+10 <sequence dfdlx:layer="gz:gzip">
+11 <group ref="ex:dataFieldsGroup"/>
+12 </sequence>
+13 </sequence>
+14 </sequence>
+15 </group>
+```
+In the above, at line 2 we see the sequence declares the boundaryMark layer.
+This layer requires 2 parameters to be passed by binding DFDL variables, which occurs on
+lines 5 and 6.
+Line 5 binds the `layerEncoding` variable and note the prefix since DFDL variable names can be
+in a specific namespace.
+Line 6 binds the `boundaryMark` variable to the string "`=_END_=`".
+Note that the variable and the layer both are named `boundaryMark`, but this is just coincidence.
+Layers do not have to have parameters that match their name, nor even any parameters at all.
+The boundary mark is the string which marks the end of the data based on scanning for that ending
+mark.
+
+The data region now being isolated, the `boundaryMark` layer contains just one sequence, and line
+9 which itself declares that the data is base64 encoded by specifying the base64_MIME layer.
+This layer has no parameters, so there are no `dfdl:newVariableInstance` statements to bind any
+variables.
+The region of data that is base64 encoded is then described by the sequence at line 10 which
+declares that the data is compressed by the `gzip` algorithm.
+
+Finally, at line 11 we reference a group which describes what the data format of the underlying
+CSV-like data.
+That group's definition is regular DFDL without any of the complexity of layers.
+This is useful for testing, as data can be parsed using this group (in the complexType of an
+element) to ensure it works properly without involving the complexity of layers.
+This group definition is the last thing in the schema:
+```xml
+<group name="dataFieldsGroup">
+ <sequence dfdl:separator="%NL;" dfdl:separatorPosition="postfix">
+ <element name="header" minOccurs="0" dfdl:occursCountKind="implicit">
+ <complexType>
+ <sequence dfdl:separator=",">
+ <element name="title" type="xs:string" maxOccurs="unbounded" dfdl:lengthKind="delimited"/>
+ </sequence>
+ </complexType>
+ </element>
+ <element name="record" maxOccurs="unbounded">
+ <complexType>
+ <sequence dfdl:separator=",">
+ <element name="item" type="xs:string" maxOccurs="unbounded"
+ dfdl:lengthKind="delimited"
+ dfdl:occursCount="{ fn:count(../../header/title) }"
+ dfdl:occursCountKind="expression"/>
+ </sequence>
+ </complexType>
+ </element>
+ </sequence>
+</group>
+
+</schema>
+```
+The above schema works both to parse, but also to unparse this data.
+
+----
+
+# Using Custom Plug-In Layers
+
+A custom plug-in layer is used in the same manner as the built-in Daffodil layers with just a few
+additional details:
+1. The custom layer will have a DFDL schema that declares any DFDL variables it uses, and
+ defines any data formats needed to facilitate use of the layer.
+ This schema will need to be included/imported.
+ Normally this schema would be a small single-file schema and it can be loaded from the
+ classpath.
+2. The custom layer's compiled Java/Scala code is dynamically loaded using the
+ Java Service Provider Interface (SPI).
+ Hence, they must be compiled into Jar files with specific META-INF metadata enclosed, and
+ these jars must appear on the Java CLASSPATH so that they can be found and loaded.
+
+The layer API is defined via Java clases and interfaces to enable writing of custom layers in
+either Java or Scala.
+(One of the built-in layers (Gzip) is written in Java now, by way of proving
+that one can write a Layer in Java.)
+
+Transformer layer classes are derived from the
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html)
+base class.
+
+Checksum layer classes are derived from the
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.html)
+base class.
+
+Further details on how to define custom plug-in layers is in the Javadoc for the
+[Layer API](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/package-summary.html)
+
+----
+
+# Daffodil Built-In Layer Documentation
+
+Each of the layers built-in to the Daffodil implementation are documented in a section below
+which gives the name, namespace, variables, and some usage notes.
+
+----
+
+## Base64 MIME Layer
+
+- Name: base64_MIME
+- Namespace URI: urn:org.apache.daffodil.layers.base64_MIME
+- No DFDL variables are read or written.
+
+This uses the standard `java.util.Base64` classes.
+
+----
+
+## BoundaryMark Layer
+
+- Name: boundaryMark
+- Namespace URI: urn:org.apache.daffodil.layers.boundaryMark
+- Reads variables
+ - `boundaryMark` a string which delimits the end of the layer data.
+ - No escape schemes are applied.
+ - DFDL character entities and character class entities are not allowed.
+ - `layerEncoding` a string which names a character set encoding which is used in the scanning
+ for the boundary mark.
+
+Isolates text data by way of a boundary mark string.
+This is like a terminating delimiter of a DFDL element or sequence/ choice group, except that
+there is no escaping mechanism, so the data cannot in any way contain the boundary mark string,
+and elements within the layer can have any `dfdl:lengthKind`, but this does not affect the search
+for the boundary mark.
+
+For example, when using an element with `dfdl:lengthKind="delimited"` and a `dfdl:terminator="END"`,
+if that element contains a child element with `dfdl:lengthKind="explicit"`, then the search for
+the "END" terminator is suspended for the length of the child element, and that search resumes
+after the child element's length has been parsed.
+
+In contrast to this, if a boundary mark layer is used with the `boundaryMark` variable bound
+to "END", then the data stream is decoded as characters in the charset encoding given by the
+`layerEncoding` variable, and the layer continues until the "END" is found. The `dfdl:lengthKind`
+of any child element enclosed within the layer, or even the lengths of other layers found
+within the scope of this boundary mark layer are not considered and do not disrupt the search
+for the boundary mark string.
+
+----
+
+## Byte-Swapping Layers
+
+- Layer Names:
+ - `twobyteswap`
+ - `fourbyteswap`
+- Namespace URI: urn:org.apache.daffodil.layers.byteSwap
+- No DFDL Variables are read or set
+- There are two layer names for two variations:
+
+Layer that re-orders bytes according to the word size which is 2 for `twobyteswap` and 4 for
+`fourbyteswap` respectively.
+This layer implements streaming behavior.
+It does not require buffering up the data.
+So it can be used on very large data objects.
+Bytes within the wrapped input stream are re-ordered _word size_ bytes at a time.
+
+For example, if
+the wrapped input stream contains 10 bytes and word size is 4, then the bytes from the wrapped
+input stream are returned in the order 4 3 2 1 8 7 6 5 10 9.
+
+If wordsize were 2 then the bytes from the wrapped input stream are returned in the
+order 2 1 4 3 6 5 8 7 10 9.
+
+----
+
+## FixedLength Layer
+
+- Name: fixedLength
+- Namespace URI: urn:org.apache.daffodil.layers.fixedLength
+- Reads variable `fixedLength` an `unsignedInt` which gives the length, in bytes, of the layer
+ data.
+ - This must be in the range from 0 to 32767 inclusive. It is a processing error otherwise.
+ - This length is enforced on both parsing and unparsing the layer.
+
+Suitable only for small sections of data, not large data streams or large files.
+The entire fixed length region of the data will be pulled into a byte buffer in memory.
+One DFDL Variable is a parameter
+
+----
+
+## GZIP Layer
+
+- Name: gzip
+- Namespace URI: urn:org.apache.daffodil.layers.gzip
+- No DFDL Variables are read or set
+
+This layer uses the `java.util.zip.GZIPInputStream` and `java.util.zip.GZIPOutputStream`
+libraries to decode and encode.
+
+Prior to Java 16, the `java.util.zip.GZIPOutputStream` wrote a value of zero for the OS field
+in the header (byte index 9).
+In Java 16, this was changed to a value of 255 to better abide by the GZIP specification.
+Unfortunately, this means unparsed data using a GZIP layer might have a single byte difference,
+depending on the Java version used.
+To avoid inconsistent behavior of test failures that expect a certain byte value this layer
+always writes a consistent header (header byte 9 of 255) regardless of the Java version.
+
+----
+
+## Line Folded Layers
+
+- Layer Names:
+ - [lineFolded_IMF](#layer-name-linefolded_imf) - conforms to IETF RFC 2822 Internet Message
+ Format (IMF)
+ - [lineFolded_iCalendar](#layer-name-linefolded_icalendar) - conforms to IETF RFC 5545 iCalendar
+- Namespace URI: urn:org.apache.daffodil.layers.lineFolded
+- No DFDL Variables are read or set
+
+#### General Usage
+
+These layers would normally be used in a context of either an enclosing element
+which bounds its length (`dfdl:lengthKind 'explicit'`, `'prefixed'`, or `'pattern'`)
+or an enclosing utility layer that bounds its length (ex: `boundaryMark`).
Review Comment:
I feel like line folded layers often are on an entire file and aren't necessarily limited by anything?
##########
site/layers.md:
##########
@@ -0,0 +1,655 @@
+---
+description: Pluggable Extensions to Enable Algorithmic Transformations
+ in DFDL
+group: 'nav-right'
+layout: page
+title: 'Layers - Algorithmic
+ Extensions for DFDL'
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+### Table of Contents
+{:.no_toc}
+
+1. use ordered table of contents
+{:toc}
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+Layers are _byte oriented_.
+The smallest non-zero layer size is a single byte, and the length of
+all layers are in bytes.
+Throughout this discussion of layers, when length is discussed, the units of measure are always
+bytes.
+
+Layers can be composed together, that is, with one on top of another, and their composition is
+orthogonal (as in non-interacting).
+This is actually very common. For example, binary data being included into a textual data format
+can be both gzip compressed, and then base64 encoded, which requires 2 layers to describe.
+There is no limit to this depth.
+
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses
+multiple layers together.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- [base64_MIME](#base64-mime-layer)
+- [fourbyteswap](#byte-swapping-layers)
+- [twobyteswap](#byte-swapping-layers)
+- [gzip](#gzip-layer)
+- [lineFolded_IMF](#line-folded-layers)
+- [lineFolded_iCalendar](#line-folded-layers)
+
+Daffodil also includes two utility layers that are used in combination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- [boundaryMark](#boundaryMark-layer)
+- [fixedLength](#fixedLength-layer)
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+### Layer Kinds: Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html) class
+which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic
+Redundancy Check (CRC) over the data stream.
+
+The value of the checksum (or hash or CRC) is assigned to a DFDL variable as the result of the
+layer. This makes the value available for use by the DFDL schema that uses the checksum layer.
+When parsing, the value of this DFDL variable can then be compared to a checksum field in the
+data, and either an invalid data element or an parse-error can be created if the checksum in the
+data stream does not match the computed value.
+When unparsing, the value of this DFDL variable can be written to an element using the
+`dfdl:outputValueCalc` property.
+
+An example of a checksum layer plug-in is in the
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header checksum algorithm.
+
+Custom checksum layers are created by deriving an implementation class from the Daffodil API's
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.
+html)
+class, which is introduced in a later section.
+
+# Using Layers
+
+To use a layer you must know
+- the layer's namespace URI
+- the layer's name
+- the names of any layer parameter variables
+- the names of any layer result variables
+
+Each layer normally has a _layer schema file_ (.dfdl.xsd) which contains the DFDL variable's
+`dfdl:defineVariable` declarations.
+
+A `sequence` group encloses the scope, within the DFDL schema, of the layer.
+Often layers are used to describe checksums or decding/encoding on small regions within a
+larger data format.
+
+The DFDL extension property `dfdlx:layer` declares that the data underlying a `sequence` is to
+use a layer. The value of the property is a QName identifying the namespace (via prefix) and
+name of the layer.
+
+A layer's namespace may define DFDL variables that are used to pass parameters to the layer, or
+to receive results back from the layer (usually for checksum layers).
+
+Lastly, there is the concept of _layer length limiting_.
+
+### Layer Length Limiting
+
+A layer can process an entire input file/stream.
+Such a layer is said to be _unlimited_.
+
+A layer can also have a specific data length built into it.
+The `IPv4Checksum` layer (in the aforementioned
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema) has a fixed length of 20 bytes.
+Such a layer is said to be _self limiting_.
+
+More commonly, a layer has a limited region within the data stream that it is supposed to describe.
+The restriction of the layer so that it only describes the expected part of the data is called
+_layer limiting_.
+
+There are these ways to do layer limiting, that differ in important ways:
+- using enclosing elements of specified length
+- using a fixedLength layer
+- binding a DFDL variable specific to a particular layer
+- using a boundaryMark layer
+
+#### Layer Limiting using Enclosing Elements of Specified Length
+
+If the `sequence` for a layer is the model group of the
+complex type of an element, and that element has specified length (meaning it has
+`dfdl:lengthKind` property of `explicit`, `implicit`, `prefixed` or `pattern`), then the length
+of the element limits the length of the layer within it.
+
+In this case the length of the layer is limited when parsing, but it is NOT limited when
+unparsing.
+
+#### Layer Limiting using the `fixedLength` Utility Layer
+
+The layer `fixedLength` is a utility layer intended for use in conjunction with other layers.
+It dictates the parse _and unparse_ length to be the value of the `layerLength` DFDL variable.
+Setting or binding this DFDL variable fixes the length of the `fixedLength` layer, and then
+any layer on top of that layer will be limited by the underlying fixed length.
+
+#### Layer Limiting by Binding a DFDL Variable
+
+Some layers will define a DFDL variable that must be set to a non-negative
+integer value to specify the layer length that the layer will use.
+All checksum layers built with the Daffodil API `ChecksumLayer` base class,
+use the DFDL `layerLength` variable to specify the length (in bytes) of the layer.
+
+Layers may specify restrictions on the minimum and maximum allowed values of these lengths,
+and passing an out-of-range value for the variable is a processing error.
+
+#### Layer Limiting using the `boundaryMark` Utility Layer
+
+The `boundaryMark` layer uses a variable named `boundaryMark` which provides a string parameter
+to the layer.
+This string is a delimiter of the layer data when parsing, and when unparsing the string is
+inserted after the (otherwise unbounded length) layer data.
+Unlike DFDL's normal delimiter properties( `dfdl:separator`, `dfdl:terminator`, and
+`dfdl:initiator`), the scanning for a boundary mark string does not use any escape scheme.
+
+
+## Example: Line Folding
+
+Consider the line folding layer, specifically the `lineFolded_IMF` layer,
+which is built-in to Daffodil.
+
+Line folding is a way of encoding textual data formats so that no line of text is longer than
+a limited line length.
+
+Consider this data :
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem
+ ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+This data has been _line folded_ at roughly 72 characters by inserting a CRLF
+before an existing space in the data.
+Each line ends with a CRLF (\r\n) and the second through fourth lines begin
+with a space as a way of indicating that they are extension lines.
+This data is supposed to be reassembled to form a long single-line string by removing
+all CRLF pairs.
+
+The result should be this single longer string which does not contain any line endings:
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+To achieve this we would use the `lineFolded_IMF` layer.
+This layer has a specific namespace which our DFDL schema will define a prefix `lf` for like this:
+```xml
+xmlns:lf="urn:org.apache.daffodil.layers.lineFolded"
+```
+Our DFDL schema will import the layer schema for the line-folded layer with this import:
+```xml
+<import namespace="urn:org.apache.daffodil.layers.lineFolded"
+ schemaLocation="/org/apache/daffodil/layers/xsd/lineFoldedLayer.dfdl.xsd"/>
+```
+Then the layer is incorporated into our DFDL schema like this:
+```xml
+<sequence dfdlx:layer="lf:lineFolded_IMF">
+ ... elements to be parsed from the unfolded layer data go here ...
+</sequence>
+```
+You can see that use of a layer is described using the `dfdx:layer` property, and the specific layer
+is identified by a QName using the previously defined namespace prefix.
+The scope of the layer is the duration of the sequence it appears on.
+The `dfdlx:layer` property can _only_ be used on an XSD `sequence`.
+
+The `lineFolded_IMF` layer does not define any DFDL variables in its namespace as it has no
+parameters and produces no results.
+As of Daffodil 3.8.0, the `import` statement above is optional for the
+line-folded layers as the DFDL schema file `lineFoldedLayer.dfdl.xsd` does not contain any
+definitions.
+In the future however, parameters may be added, so for uniformity all layers define a DFDL
+schema to be imported as part of using the layer.
+
+If the line-folding applies to the entire data stream (not uncommon for formats that use line
+folding), then what has been described is all one needs to describe the layer aspects of a data
+format.
+Other examples will show how the layer length can be limited to a sub-region of the data.
+
+More detailed documentation for the [Line Folded Layers](#LineFoldedLayers) is below.
+
+## Example: Base64, GZip, and BoundaryMark Layers used Together
+
+In this example, the data consists of a preliminary string, a section of CSV-like data, and a
+final string element.
+The CSV-like data section in the middle is both gzipped, and then base64 encoded, with the
+boundary value mark of "`=_END_=`" indicating where the base64 part ends.
+
+The point of the preliminary string and the final string is to show that this sort of layering
+composes properly with surrounding data elements in the schema. Those strings could be anything
+at all, they're just strings to make this example simpler.
+
+The physical data string/stream looks like this:
+```text
+example of data before the base64 gzipped part
+H4sIAAAAAAAA/y3JQQqAIBCF4b1nmYG0iNxG+84wpqFhI4xuun0Gbd4P78tUG5xJ+t7J+xxg21dV
+79QiSHFBGjgh9oVB22XBYUQzqatErv36CjGwPJ/OOGg0Y1cOFUgegoPaLwa1VS9htd+UbgAAAA==
+=_END_=
+example of data after the base64 gzipped part
+```
+That base64 encoded part, if decoded and decompressed looks like this CSV-like data:
+```
+last,first,middle,DOB
+smith,robert,brandon,1988-03-24
+johnson,john,henry,1986-01-23
+jones,arya,cat,1986-02-19
+```
+
+The Infoset, presented as XML, that we will create for this entire data stream
+using Daffodil parsing looks like:
+```xml
+<ex:data>
+ <before>example of data before the base64 gzipped part</before>
+ <header><title>last</title><title>first</title><title>middle</title><title>DOB</title></header>
+ <record><item>smith</item><item>robert</item><item>brandon</item><item>1988-03-24</item></record>
+ <record><item>johnson</item><item>john</item><item>henry</item><item>1986-01-23</item></record>
+ <record><item>jones</item><item>arya</item><item>cat</item><item>1986-02-19</item></record>
+ <after>example of data after the base64 gzipped part</after>
+</ex:data>
+```
+
+With that characterization of the problem, let's look at the DFDL schema with the Daffodil
+layering properties to do the boundaryMark, base64 decode, and gzip decompression.
+(Note that each of these layers is described further in its own section below on this page.)
+
+First there is the start of the schema which introduces namespace prefix definitions for DFDL,
+for the DFDL extensions (dfdlx) prefix which is used for the `dfdlx:layer` property.
+
+This also sets up 3 namespace prefixes for the namespaces of the boundaryMark (bm), gzip (gz)
+and base64_MIME (b64) layers which will be imported just after this header.
+
+```xml
+<schema
+ xmlns="http://www.w3.org/2001/XMLSchema"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:fn="http://www.w3.org/2005/xpath-functions"
+ xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions"
+ xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:bm="urn:org.apache.daffodil.layers.boundaryMark"
+ xmlns:b64="urn:org.apache.daffodil.layers.base64_MIME"
+ xmlns:gz="urn:org.apache.daffodil.layers.gzip"
+ xmlns:ex="http://example.com"
+ targetNamespace="http://example.com">
+```
+Next we import the Daffodil built in general format which provides most DFDL properties with
+sensible starting values, and import the namespaces of our boundaryMark, gzip, and base64_MIME
+layers, and lastly, set the default format properties for use by this schema file:
+```xml
+<include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+
+<import namespace="urn:org.apache.daffodil.layers.boundaryMark"
+ schemaLocation="/org/apache/daffodil/layers/xsd/boundaryMarkLayer.dfdl.xsd"/>
+<import namespace="urn:org.apache.daffodil.layers.gzip"
+ schemaLocation="/org/apache/daffodil/layers/xsd/gzipLayer.dfdl.xsd"/>
+<import namespace="urn:org.apache.daffodil.layers.base64_MIME"
+ schemaLocation="/org/apache/daffodil/layers/xsd/base64_MIMELayer.dfdl.xsd"/>
+
+<annotation><appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:format ref="ex:GeneralFormat" representation="binary"/>
+</appinfo></annotation>
+```
+The root element of the schema is `data`, and this element has type `dataType`.
+(A separate named complex type definition for the root element is a
+recommended style for DFDL schemas as it allows someone else to reuse the schema without
+getting involved in element namespace prefix complexities.)
+The `dataType` just shows us the `before` and `after` elements, and references a group
+named `layeredDataGroup` for the layered part in between.
+```xml
+<element name="data" type="ex:dataType"/>
+
+<complexType name="dataType">
+ <sequence dfdl:separator="%NL;">
+ <element name="before" type="string" dfdl:lengthKind="delimited"/>
+ <group ref="ex:layeredDataGroup"/>
+ <element name="after" type="string" dfdl:lengthKind="delimited"/>
+ </sequence>
+</complexType>
+```
+Now we see the definition of the layering.
+All 3 layers are declared in this named group.
+The lines here are numbered so that the text that follows can refer to them:
+```xml
+1 <group name="layeredDataGroup">
+2 <sequence dfdlx:layer="bm:boundaryMark">
+3 <annotation>
+4 <appinfo source="http://www.ogf.org/dfdl/">
+5 <dfdl:newVariableInstance ref="bm:layerEncoding" defaultValue="iso-8859-1"/>
+6 <dfdl:newVariableInstance ref="bm:boundaryMark" defaultValue="=_END_="/>
+7 </appinfo>
+8 </annotation>
+9 <sequence dfdlx:layer="b64:base64_MIME">
+10 <sequence dfdlx:layer="gz:gzip">
+11 <group ref="ex:dataFieldsGroup"/>
+12 </sequence>
+13 </sequence>
+14 </sequence>
+15 </group>
+```
+In the above, at line 2 we see the sequence declares the boundaryMark layer.
+This layer requires 2 parameters to be passed by binding DFDL variables, which occurs on
+lines 5 and 6.
+Line 5 binds the `layerEncoding` variable and note the prefix since DFDL variable names can be
+in a specific namespace.
+Line 6 binds the `boundaryMark` variable to the string "`=_END_=`".
+Note that the variable and the layer both are named `boundaryMark`, but this is just coincidence.
+Layers do not have to have parameters that match their name, nor even any parameters at all.
+The boundary mark is the string which marks the end of the data based on scanning for that ending
+mark.
+
+The data region now being isolated, the `boundaryMark` layer contains just one sequence, and line
+9 which itself declares that the data is base64 encoded by specifying the base64_MIME layer.
+This layer has no parameters, so there are no `dfdl:newVariableInstance` statements to bind any
+variables.
+The region of data that is base64 encoded is then described by the sequence at line 10 which
+declares that the data is compressed by the `gzip` algorithm.
+
+Finally, at line 11 we reference a group which describes what the data format of the underlying
+CSV-like data.
+That group's definition is regular DFDL without any of the complexity of layers.
+This is useful for testing, as data can be parsed using this group (in the complexType of an
+element) to ensure it works properly without involving the complexity of layers.
+This group definition is the last thing in the schema:
+```xml
+<group name="dataFieldsGroup">
+ <sequence dfdl:separator="%NL;" dfdl:separatorPosition="postfix">
+ <element name="header" minOccurs="0" dfdl:occursCountKind="implicit">
+ <complexType>
+ <sequence dfdl:separator=",">
+ <element name="title" type="xs:string" maxOccurs="unbounded" dfdl:lengthKind="delimited"/>
+ </sequence>
+ </complexType>
+ </element>
+ <element name="record" maxOccurs="unbounded">
+ <complexType>
+ <sequence dfdl:separator=",">
+ <element name="item" type="xs:string" maxOccurs="unbounded"
+ dfdl:lengthKind="delimited"
+ dfdl:occursCount="{ fn:count(../../header/title) }"
+ dfdl:occursCountKind="expression"/>
+ </sequence>
+ </complexType>
+ </element>
+ </sequence>
+</group>
+
+</schema>
+```
+The above schema works both to parse, but also to unparse this data.
+
+----
+
+# Using Custom Plug-In Layers
+
+A custom plug-in layer is used in the same manner as the built-in Daffodil layers with just a few
+additional details:
+1. The custom layer will have a DFDL schema that declares any DFDL variables it uses, and
+ defines any data formats needed to facilitate use of the layer.
+ This schema will need to be included/imported.
+ Normally this schema would be a small single-file schema and it can be loaded from the
+ classpath.
+2. The custom layer's compiled Java/Scala code is dynamically loaded using the
+ Java Service Provider Interface (SPI).
+ Hence, they must be compiled into Jar files with specific META-INF metadata enclosed, and
+ these jars must appear on the Java CLASSPATH so that they can be found and loaded.
+
+The layer API is defined via Java clases and interfaces to enable writing of custom layers in
+either Java or Scala.
+(One of the built-in layers (Gzip) is written in Java now, by way of proving
+that one can write a Layer in Java.)
+
+Transformer layer classes are derived from the
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html)
+base class.
+
+Checksum layer classes are derived from the
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.html)
+base class.
+
+Further details on how to define custom plug-in layers is in the Javadoc for the
+[Layer API](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/package-summary.html)
+
+----
+
+# Daffodil Built-In Layer Documentation
+
+Each of the layers built-in to the Daffodil implementation are documented in a section below
+which gives the name, namespace, variables, and some usage notes.
+
+----
+
+## Base64 MIME Layer
+
+- Name: base64_MIME
+- Namespace URI: urn:org.apache.daffodil.layers.base64_MIME
+- No DFDL variables are read or written.
+
+This uses the standard `java.util.Base64` classes.
+
+----
+
+## BoundaryMark Layer
+
+- Name: boundaryMark
+- Namespace URI: urn:org.apache.daffodil.layers.boundaryMark
+- Reads variables
+ - `boundaryMark` a string which delimits the end of the layer data.
+ - No escape schemes are applied.
+ - DFDL character entities and character class entities are not allowed.
+ - `layerEncoding` a string which names a character set encoding which is used in the scanning
+ for the boundary mark.
+
+Isolates text data by way of a boundary mark string.
+This is like a terminating delimiter of a DFDL element or sequence/ choice group, except that
+there is no escaping mechanism, so the data cannot in any way contain the boundary mark string,
+and elements within the layer can have any `dfdl:lengthKind`, but this does not affect the search
+for the boundary mark.
+
+For example, when using an element with `dfdl:lengthKind="delimited"` and a `dfdl:terminator="END"`,
+if that element contains a child element with `dfdl:lengthKind="explicit"`, then the search for
+the "END" terminator is suspended for the length of the child element, and that search resumes
+after the child element's length has been parsed.
+
+In contrast to this, if a boundary mark layer is used with the `boundaryMark` variable bound
+to "END", then the data stream is decoded as characters in the charset encoding given by the
+`layerEncoding` variable, and the layer continues until the "END" is found. The `dfdl:lengthKind`
+of any child element enclosed within the layer, or even the lengths of other layers found
+within the scope of this boundary mark layer are not considered and do not disrupt the search
+for the boundary mark string.
+
+----
+
+## Byte-Swapping Layers
+
+- Layer Names:
+ - `twobyteswap`
+ - `fourbyteswap`
+- Namespace URI: urn:org.apache.daffodil.layers.byteSwap
+- No DFDL Variables are read or set
+- There are two layer names for two variations:
+
+Layer that re-orders bytes according to the word size which is 2 for `twobyteswap` and 4 for
+`fourbyteswap` respectively.
+This layer implements streaming behavior.
+It does not require buffering up the data.
+So it can be used on very large data objects.
+Bytes within the wrapped input stream are re-ordered _word size_ bytes at a time.
+
+For example, if
+the wrapped input stream contains 10 bytes and word size is 4, then the bytes from the wrapped
+input stream are returned in the order 4 3 2 1 8 7 6 5 10 9.
+
+If wordsize were 2 then the bytes from the wrapped input stream are returned in the
+order 2 1 4 3 6 5 8 7 10 9.
+
+----
+
+## FixedLength Layer
+
+- Name: fixedLength
+- Namespace URI: urn:org.apache.daffodil.layers.fixedLength
+- Reads variable `fixedLength` an `unsignedInt` which gives the length, in bytes, of the layer
+ data.
+ - This must be in the range from 0 to 32767 inclusive. It is a processing error otherwise.
+ - This length is enforced on both parsing and unparsing the layer.
+
+Suitable only for small sections of data, not large data streams or large files.
+The entire fixed length region of the data will be pulled into a byte buffer in memory.
+One DFDL Variable is a parameter
Review Comment:
I don't understand "One DFDL Variable is a paramter", should that be removed?
##########
site/layers.md:
##########
@@ -0,0 +1,655 @@
+---
+description: Pluggable Extensions to Enable Algorithmic Transformations
+ in DFDL
+group: 'nav-right'
+layout: page
+title: 'Layers - Algorithmic
+ Extensions for DFDL'
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+### Table of Contents
+{:.no_toc}
+
+1. use ordered table of contents
+{:toc}
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+Layers are _byte oriented_.
+The smallest non-zero layer size is a single byte, and the length of
+all layers are in bytes.
+Throughout this discussion of layers, when length is discussed, the units of measure are always
+bytes.
+
+Layers can be composed together, that is, with one on top of another, and their composition is
+orthogonal (as in non-interacting).
+This is actually very common. For example, binary data being included into a textual data format
+can be both gzip compressed, and then base64 encoded, which requires 2 layers to describe.
+There is no limit to this depth.
+
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses
+multiple layers together.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- [base64_MIME](#base64-mime-layer)
+- [fourbyteswap](#byte-swapping-layers)
+- [twobyteswap](#byte-swapping-layers)
+- [gzip](#gzip-layer)
+- [lineFolded_IMF](#line-folded-layers)
+- [lineFolded_iCalendar](#line-folded-layers)
+
+Daffodil also includes two utility layers that are used in combination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- [boundaryMark](#boundaryMark-layer)
+- [fixedLength](#fixedLength-layer)
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+### Layer Kinds: Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html) class
+which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic
+Redundancy Check (CRC) over the data stream.
+
+The value of the checksum (or hash or CRC) is assigned to a DFDL variable as the result of the
+layer. This makes the value available for use by the DFDL schema that uses the checksum layer.
+When parsing, the value of this DFDL variable can then be compared to a checksum field in the
+data, and either an invalid data element or an parse-error can be created if the checksum in the
+data stream does not match the computed value.
+When unparsing, the value of this DFDL variable can be written to an element using the
+`dfdl:outputValueCalc` property.
+
+An example of a checksum layer plug-in is in the
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header checksum algorithm.
+
+Custom checksum layers are created by deriving an implementation class from the Daffodil API's
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.
+html)
+class, which is introduced in a later section.
+
+# Using Layers
+
+To use a layer you must know
+- the layer's namespace URI
+- the layer's name
+- the names of any layer parameter variables
+- the names of any layer result variables
+
+Each layer normally has a _layer schema file_ (.dfdl.xsd) which contains the DFDL variable's
+`dfdl:defineVariable` declarations.
+
+A `sequence` group encloses the scope, within the DFDL schema, of the layer.
+Often layers are used to describe checksums or decding/encoding on small regions within a
+larger data format.
+
+The DFDL extension property `dfdlx:layer` declares that the data underlying a `sequence` is to
+use a layer. The value of the property is a QName identifying the namespace (via prefix) and
+name of the layer.
+
+A layer's namespace may define DFDL variables that are used to pass parameters to the layer, or
+to receive results back from the layer (usually for checksum layers).
+
+Lastly, there is the concept of _layer length limiting_.
+
+### Layer Length Limiting
+
+A layer can process an entire input file/stream.
+Such a layer is said to be _unlimited_.
+
+A layer can also have a specific data length built into it.
+The `IPv4Checksum` layer (in the aforementioned
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema) has a fixed length of 20 bytes.
+Such a layer is said to be _self limiting_.
+
+More commonly, a layer has a limited region within the data stream that it is supposed to describe.
+The restriction of the layer so that it only describes the expected part of the data is called
+_layer limiting_.
+
+There are these ways to do layer limiting, that differ in important ways:
+- using enclosing elements of specified length
+- using a fixedLength layer
+- binding a DFDL variable specific to a particular layer
+- using a boundaryMark layer
+
+#### Layer Limiting using Enclosing Elements of Specified Length
+
+If the `sequence` for a layer is the model group of the
+complex type of an element, and that element has specified length (meaning it has
+`dfdl:lengthKind` property of `explicit`, `implicit`, `prefixed` or `pattern`), then the length
+of the element limits the length of the layer within it.
+
+In this case the length of the layer is limited when parsing, but it is NOT limited when
+unparsing.
+
+#### Layer Limiting using the `fixedLength` Utility Layer
+
+The layer `fixedLength` is a utility layer intended for use in conjunction with other layers.
+It dictates the parse _and unparse_ length to be the value of the `layerLength` DFDL variable.
+Setting or binding this DFDL variable fixes the length of the `fixedLength` layer, and then
+any layer on top of that layer will be limited by the underlying fixed length.
+
+#### Layer Limiting by Binding a DFDL Variable
+
+Some layers will define a DFDL variable that must be set to a non-negative
+integer value to specify the layer length that the layer will use.
+All checksum layers built with the Daffodil API `ChecksumLayer` base class,
+use the DFDL `layerLength` variable to specify the length (in bytes) of the layer.
+
+Layers may specify restrictions on the minimum and maximum allowed values of these lengths,
+and passing an out-of-range value for the variable is a processing error.
+
+#### Layer Limiting using the `boundaryMark` Utility Layer
+
+The `boundaryMark` layer uses a variable named `boundaryMark` which provides a string parameter
+to the layer.
+This string is a delimiter of the layer data when parsing, and when unparsing the string is
+inserted after the (otherwise unbounded length) layer data.
+Unlike DFDL's normal delimiter properties( `dfdl:separator`, `dfdl:terminator`, and
+`dfdl:initiator`), the scanning for a boundary mark string does not use any escape scheme.
+
+
+## Example: Line Folding
+
+Consider the line folding layer, specifically the `lineFolded_IMF` layer,
+which is built-in to Daffodil.
+
+Line folding is a way of encoding textual data formats so that no line of text is longer than
+a limited line length.
+
+Consider this data :
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem
+ ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+This data has been _line folded_ at roughly 72 characters by inserting a CRLF
+before an existing space in the data.
+Each line ends with a CRLF (\r\n) and the second through fourth lines begin
+with a space as a way of indicating that they are extension lines.
+This data is supposed to be reassembled to form a long single-line string by removing
+all CRLF pairs.
+
+The result should be this single longer string which does not contain any line endings:
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+To achieve this we would use the `lineFolded_IMF` layer.
+This layer has a specific namespace which our DFDL schema will define a prefix `lf` for like this:
+```xml
+xmlns:lf="urn:org.apache.daffodil.layers.lineFolded"
+```
+Our DFDL schema will import the layer schema for the line-folded layer with this import:
+```xml
+<import namespace="urn:org.apache.daffodil.layers.lineFolded"
+ schemaLocation="/org/apache/daffodil/layers/xsd/lineFoldedLayer.dfdl.xsd"/>
+```
+Then the layer is incorporated into our DFDL schema like this:
+```xml
+<sequence dfdlx:layer="lf:lineFolded_IMF">
+ ... elements to be parsed from the unfolded layer data go here ...
+</sequence>
+```
+You can see that use of a layer is described using the `dfdx:layer` property, and the specific layer
+is identified by a QName using the previously defined namespace prefix.
+The scope of the layer is the duration of the sequence it appears on.
+The `dfdlx:layer` property can _only_ be used on an XSD `sequence`.
+
+The `lineFolded_IMF` layer does not define any DFDL variables in its namespace as it has no
+parameters and produces no results.
+As of Daffodil 3.8.0, the `import` statement above is optional for the
+line-folded layers as the DFDL schema file `lineFoldedLayer.dfdl.xsd` does not contain any
+definitions.
+In the future however, parameters may be added, so for uniformity all layers define a DFDL
+schema to be imported as part of using the layer.
+
+If the line-folding applies to the entire data stream (not uncommon for formats that use line
+folding), then what has been described is all one needs to describe the layer aspects of a data
+format.
+Other examples will show how the layer length can be limited to a sub-region of the data.
+
+More detailed documentation for the [Line Folded Layers](#LineFoldedLayers) is below.
+
+## Example: Base64, GZip, and BoundaryMark Layers used Together
+
+In this example, the data consists of a preliminary string, a section of CSV-like data, and a
+final string element.
+The CSV-like data section in the middle is both gzipped, and then base64 encoded, with the
+boundary value mark of "`=_END_=`" indicating where the base64 part ends.
+
+The point of the preliminary string and the final string is to show that this sort of layering
+composes properly with surrounding data elements in the schema. Those strings could be anything
+at all, they're just strings to make this example simpler.
+
+The physical data string/stream looks like this:
+```text
+example of data before the base64 gzipped part
+H4sIAAAAAAAA/y3JQQqAIBCF4b1nmYG0iNxG+84wpqFhI4xuun0Gbd4P78tUG5xJ+t7J+xxg21dV
+79QiSHFBGjgh9oVB22XBYUQzqatErv36CjGwPJ/OOGg0Y1cOFUgegoPaLwa1VS9htd+UbgAAAA==
+=_END_=
+example of data after the base64 gzipped part
+```
+That base64 encoded part, if decoded and decompressed looks like this CSV-like data:
+```
+last,first,middle,DOB
+smith,robert,brandon,1988-03-24
+johnson,john,henry,1986-01-23
+jones,arya,cat,1986-02-19
+```
+
+The Infoset, presented as XML, that we will create for this entire data stream
+using Daffodil parsing looks like:
+```xml
+<ex:data>
+ <before>example of data before the base64 gzipped part</before>
+ <header><title>last</title><title>first</title><title>middle</title><title>DOB</title></header>
+ <record><item>smith</item><item>robert</item><item>brandon</item><item>1988-03-24</item></record>
+ <record><item>johnson</item><item>john</item><item>henry</item><item>1986-01-23</item></record>
+ <record><item>jones</item><item>arya</item><item>cat</item><item>1986-02-19</item></record>
+ <after>example of data after the base64 gzipped part</after>
+</ex:data>
+```
+
+With that characterization of the problem, let's look at the DFDL schema with the Daffodil
+layering properties to do the boundaryMark, base64 decode, and gzip decompression.
+(Note that each of these layers is described further in its own section below on this page.)
+
+First there is the start of the schema which introduces namespace prefix definitions for DFDL,
+for the DFDL extensions (dfdlx) prefix which is used for the `dfdlx:layer` property.
+
+This also sets up 3 namespace prefixes for the namespaces of the boundaryMark (bm), gzip (gz)
+and base64_MIME (b64) layers which will be imported just after this header.
+
+```xml
+<schema
+ xmlns="http://www.w3.org/2001/XMLSchema"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:fn="http://www.w3.org/2005/xpath-functions"
+ xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions"
+ xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:bm="urn:org.apache.daffodil.layers.boundaryMark"
+ xmlns:b64="urn:org.apache.daffodil.layers.base64_MIME"
+ xmlns:gz="urn:org.apache.daffodil.layers.gzip"
+ xmlns:ex="http://example.com"
+ targetNamespace="http://example.com">
+```
+Next we import the Daffodil built in general format which provides most DFDL properties with
+sensible starting values, and import the namespaces of our boundaryMark, gzip, and base64_MIME
+layers, and lastly, set the default format properties for use by this schema file:
+```xml
+<include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+
+<import namespace="urn:org.apache.daffodil.layers.boundaryMark"
+ schemaLocation="/org/apache/daffodil/layers/xsd/boundaryMarkLayer.dfdl.xsd"/>
+<import namespace="urn:org.apache.daffodil.layers.gzip"
+ schemaLocation="/org/apache/daffodil/layers/xsd/gzipLayer.dfdl.xsd"/>
+<import namespace="urn:org.apache.daffodil.layers.base64_MIME"
+ schemaLocation="/org/apache/daffodil/layers/xsd/base64_MIMELayer.dfdl.xsd"/>
+
+<annotation><appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:format ref="ex:GeneralFormat" representation="binary"/>
+</appinfo></annotation>
+```
+The root element of the schema is `data`, and this element has type `dataType`.
+(A separate named complex type definition for the root element is a
+recommended style for DFDL schemas as it allows someone else to reuse the schema without
+getting involved in element namespace prefix complexities.)
+The `dataType` just shows us the `before` and `after` elements, and references a group
+named `layeredDataGroup` for the layered part in between.
+```xml
+<element name="data" type="ex:dataType"/>
+
+<complexType name="dataType">
+ <sequence dfdl:separator="%NL;">
+ <element name="before" type="string" dfdl:lengthKind="delimited"/>
+ <group ref="ex:layeredDataGroup"/>
+ <element name="after" type="string" dfdl:lengthKind="delimited"/>
+ </sequence>
+</complexType>
+```
+Now we see the definition of the layering.
+All 3 layers are declared in this named group.
+The lines here are numbered so that the text that follows can refer to them:
+```xml
+1 <group name="layeredDataGroup">
+2 <sequence dfdlx:layer="bm:boundaryMark">
+3 <annotation>
+4 <appinfo source="http://www.ogf.org/dfdl/">
+5 <dfdl:newVariableInstance ref="bm:layerEncoding" defaultValue="iso-8859-1"/>
+6 <dfdl:newVariableInstance ref="bm:boundaryMark" defaultValue="=_END_="/>
+7 </appinfo>
+8 </annotation>
+9 <sequence dfdlx:layer="b64:base64_MIME">
+10 <sequence dfdlx:layer="gz:gzip">
+11 <group ref="ex:dataFieldsGroup"/>
+12 </sequence>
+13 </sequence>
+14 </sequence>
+15 </group>
+```
+In the above, at line 2 we see the sequence declares the boundaryMark layer.
+This layer requires 2 parameters to be passed by binding DFDL variables, which occurs on
+lines 5 and 6.
+Line 5 binds the `layerEncoding` variable and note the prefix since DFDL variable names can be
+in a specific namespace.
+Line 6 binds the `boundaryMark` variable to the string "`=_END_=`".
+Note that the variable and the layer both are named `boundaryMark`, but this is just coincidence.
+Layers do not have to have parameters that match their name, nor even any parameters at all.
+The boundary mark is the string which marks the end of the data based on scanning for that ending
+mark.
+
+The data region now being isolated, the `boundaryMark` layer contains just one sequence, and line
+9 which itself declares that the data is base64 encoded by specifying the base64_MIME layer.
+This layer has no parameters, so there are no `dfdl:newVariableInstance` statements to bind any
+variables.
+The region of data that is base64 encoded is then described by the sequence at line 10 which
+declares that the data is compressed by the `gzip` algorithm.
+
+Finally, at line 11 we reference a group which describes what the data format of the underlying
+CSV-like data.
+That group's definition is regular DFDL without any of the complexity of layers.
+This is useful for testing, as data can be parsed using this group (in the complexType of an
+element) to ensure it works properly without involving the complexity of layers.
+This group definition is the last thing in the schema:
+```xml
+<group name="dataFieldsGroup">
+ <sequence dfdl:separator="%NL;" dfdl:separatorPosition="postfix">
+ <element name="header" minOccurs="0" dfdl:occursCountKind="implicit">
+ <complexType>
+ <sequence dfdl:separator=",">
+ <element name="title" type="xs:string" maxOccurs="unbounded" dfdl:lengthKind="delimited"/>
+ </sequence>
+ </complexType>
+ </element>
+ <element name="record" maxOccurs="unbounded">
+ <complexType>
+ <sequence dfdl:separator=",">
+ <element name="item" type="xs:string" maxOccurs="unbounded"
+ dfdl:lengthKind="delimited"
+ dfdl:occursCount="{ fn:count(../../header/title) }"
+ dfdl:occursCountKind="expression"/>
+ </sequence>
+ </complexType>
+ </element>
+ </sequence>
+</group>
+
+</schema>
+```
+The above schema works both to parse, but also to unparse this data.
+
+----
+
+# Using Custom Plug-In Layers
+
+A custom plug-in layer is used in the same manner as the built-in Daffodil layers with just a few
+additional details:
+1. The custom layer will have a DFDL schema that declares any DFDL variables it uses, and
+ defines any data formats needed to facilitate use of the layer.
+ This schema will need to be included/imported.
+ Normally this schema would be a small single-file schema and it can be loaded from the
+ classpath.
+2. The custom layer's compiled Java/Scala code is dynamically loaded using the
+ Java Service Provider Interface (SPI).
+ Hence, they must be compiled into Jar files with specific META-INF metadata enclosed, and
+ these jars must appear on the Java CLASSPATH so that they can be found and loaded.
+
+The layer API is defined via Java clases and interfaces to enable writing of custom layers in
+either Java or Scala.
+(One of the built-in layers (Gzip) is written in Java now, by way of proving
+that one can write a Layer in Java.)
+
+Transformer layer classes are derived from the
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html)
+base class.
+
+Checksum layer classes are derived from the
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.html)
+base class.
+
+Further details on how to define custom plug-in layers is in the Javadoc for the
+[Layer API](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/package-summary.html)
+
+----
+
+# Daffodil Built-In Layer Documentation
+
+Each of the layers built-in to the Daffodil implementation are documented in a section below
+which gives the name, namespace, variables, and some usage notes.
+
+----
+
+## Base64 MIME Layer
+
+- Name: base64_MIME
+- Namespace URI: urn:org.apache.daffodil.layers.base64_MIME
+- No DFDL variables are read or written.
+
+This uses the standard `java.util.Base64` classes.
+
+----
+
+## BoundaryMark Layer
+
+- Name: boundaryMark
+- Namespace URI: urn:org.apache.daffodil.layers.boundaryMark
+- Reads variables
+ - `boundaryMark` a string which delimits the end of the layer data.
+ - No escape schemes are applied.
+ - DFDL character entities and character class entities are not allowed.
+ - `layerEncoding` a string which names a character set encoding which is used in the scanning
+ for the boundary mark.
+
+Isolates text data by way of a boundary mark string.
+This is like a terminating delimiter of a DFDL element or sequence/ choice group, except that
+there is no escaping mechanism, so the data cannot in any way contain the boundary mark string,
+and elements within the layer can have any `dfdl:lengthKind`, but this does not affect the search
+for the boundary mark.
+
+For example, when using an element with `dfdl:lengthKind="delimited"` and a `dfdl:terminator="END"`,
+if that element contains a child element with `dfdl:lengthKind="explicit"`, then the search for
+the "END" terminator is suspended for the length of the child element, and that search resumes
+after the child element's length has been parsed.
+
+In contrast to this, if a boundary mark layer is used with the `boundaryMark` variable bound
+to "END", then the data stream is decoded as characters in the charset encoding given by the
+`layerEncoding` variable, and the layer continues until the "END" is found. The `dfdl:lengthKind`
+of any child element enclosed within the layer, or even the lengths of other layers found
+within the scope of this boundary mark layer are not considered and do not disrupt the search
+for the boundary mark string.
+
+----
+
+## Byte-Swapping Layers
+
+- Layer Names:
+ - `twobyteswap`
+ - `fourbyteswap`
+- Namespace URI: urn:org.apache.daffodil.layers.byteSwap
+- No DFDL Variables are read or set
+- There are two layer names for two variations:
+
+Layer that re-orders bytes according to the word size which is 2 for `twobyteswap` and 4 for
+`fourbyteswap` respectively.
+This layer implements streaming behavior.
+It does not require buffering up the data.
+So it can be used on very large data objects.
+Bytes within the wrapped input stream are re-ordered _word size_ bytes at a time.
+
+For example, if
+the wrapped input stream contains 10 bytes and word size is 4, then the bytes from the wrapped
+input stream are returned in the order 4 3 2 1 8 7 6 5 10 9.
+
+If wordsize were 2 then the bytes from the wrapped input stream are returned in the
+order 2 1 4 3 6 5 8 7 10 9.
+
+----
+
+## FixedLength Layer
+
+- Name: fixedLength
+- Namespace URI: urn:org.apache.daffodil.layers.fixedLength
+- Reads variable `fixedLength` an `unsignedInt` which gives the length, in bytes, of the layer
Review Comment:
Just for consistency, suggest making fixedLength and its description a bullet under a Reads variables bullet similar to boundary mark.
##########
site/layers.md:
##########
@@ -0,0 +1,655 @@
+---
+description: Pluggable Extensions to Enable Algorithmic Transformations
+ in DFDL
+group: 'nav-right'
+layout: page
+title: 'Layers - Algorithmic
+ Extensions for DFDL'
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+### Table of Contents
+{:.no_toc}
+
+1. use ordered table of contents
+{:toc}
+
+## Introduction
+
+This page describes a DFDL language extension known as _Layers_.
+
+A _layer_ is an algorithmic transformation of the data stream that cannot be expressed using
+regular DFDL properties.
+When parsing it is like a pre-processing of the data stream which
+happens before parsing.
+When unparsing it is like a post-processing of the data stream which
+happens after unparsing.
+The layer can underlie a part of the data stream, or all of it.
+
+Layers are _byte oriented_.
+The smallest non-zero layer size is a single byte, and the length of
+all layers are in bytes.
+Throughout this discussion of layers, when length is discussed, the units of measure are always
+bytes.
+
+Layers can be composed together, that is, with one on top of another, and their composition is
+orthogonal (as in non-interacting).
+This is actually very common. For example, binary data being included into a textual data format
+can be both gzip compressed, and then base64 encoded, which requires 2 layers to describe.
+There is no limit to this depth.
+
+In the section on [Using Layers](#UsingLayers) below we will look at an example that uses
+multiple layers together.
+
+### Built-in Layers
+
+Daffodil includes several built-in layers:
+- [base64_MIME](#base64-mime-layer)
+- [fourbyteswap](#byte-swapping-layers)
+- [twobyteswap](#byte-swapping-layers)
+- [gzip](#gzip-layer)
+- [lineFolded_IMF](#line-folded-layers)
+- [lineFolded_iCalendar](#line-folded-layers)
+
+Daffodil also includes two utility layers that are used in combination with other layers to
+isolate the subset of the data stream the layer algorithm will operate upon.
+These are:
+- [boundaryMark](#boundaryMark-layer)
+- [fixedLength](#fixedLength-layer)
+
+Each of the built-in layers will be documented separately below with examples of their usage.
+
+### Custom Plug-In Layers
+
+Additional layers can be written in Java or Scala and deployed as _plug-ins_ for Daffodil.
+These are generally packaged as DFDL _layer schemas_, a kind of _component schema_,
+that provide the layer packaged for import by other DFDL _assembly_ schemas that use the
+layer in the data format they describe.
+
+### Layer Kinds: Transforming Layers and Checksum Layers
+
+There are two different kinds of layers, though they share many characteristics. They are
+_transforming_ layers, and _checksum_ layers. Both run small algorithms over part (or all) of
+the data stream. The difference is the purpose of the algorithm and its output.
+
+#### Transforming Layers
+
+These layers decode data (when parsing), and encode data (when unparsing).
+The simplest example of a transforming layer is the `base64_MIME` layer which
+decodes the well known base64 encoding which is commonly used to encode binary
+data inside textual data formats.
+
+Besides decoding and encoding the data stream for the parser/unparser, transforming layers can be
+parameterized using DFDL variables.
+They can also assign computed result values to DFDL variables, though this is uncommon.
+
+Custom transforming layers are created by deriving an implementation from the Daffodil API's
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html) class
+which is introduced in a later section.
+
+#### Checksum Layers
+
+Checksum layers are a simplified kind of layers which do not decode or encode data, they simply
+pass through the data unmodified, but while doing so they compute a checksum, hash, or Cyclic
+Redundancy Check (CRC) over the data stream.
+
+The value of the checksum (or hash or CRC) is assigned to a DFDL variable as the result of the
+layer. This makes the value available for use by the DFDL schema that uses the checksum layer.
+When parsing, the value of this DFDL variable can then be compared to a checksum field in the
+data, and either an invalid data element or an parse-error can be created if the checksum in the
+data stream does not match the computed value.
+When unparsing, the value of this DFDL variable can be written to an element using the
+`dfdl:outputValueCalc` property.
+
+An example of a checksum layer plug-in is in the
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema, which uses a Daffodil layer to describe the IPv4 packet header checksum algorithm.
+
+Custom checksum layers are created by deriving an implementation class from the Daffodil API's
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.
+html)
+class, which is introduced in a later section.
+
+# Using Layers
+
+To use a layer you must know
+- the layer's namespace URI
+- the layer's name
+- the names of any layer parameter variables
+- the names of any layer result variables
+
+Each layer normally has a _layer schema file_ (.dfdl.xsd) which contains the DFDL variable's
+`dfdl:defineVariable` declarations.
+
+A `sequence` group encloses the scope, within the DFDL schema, of the layer.
+Often layers are used to describe checksums or decding/encoding on small regions within a
+larger data format.
+
+The DFDL extension property `dfdlx:layer` declares that the data underlying a `sequence` is to
+use a layer. The value of the property is a QName identifying the namespace (via prefix) and
+name of the layer.
+
+A layer's namespace may define DFDL variables that are used to pass parameters to the layer, or
+to receive results back from the layer (usually for checksum layers).
+
+Lastly, there is the concept of _layer length limiting_.
+
+### Layer Length Limiting
+
+A layer can process an entire input file/stream.
+Such a layer is said to be _unlimited_.
+
+A layer can also have a specific data length built into it.
+The `IPv4Checksum` layer (in the aforementioned
+[EthernetIP](https://github.com/DFDLSchemas/ethernetIP)
+DFDL schema) has a fixed length of 20 bytes.
+Such a layer is said to be _self limiting_.
+
+More commonly, a layer has a limited region within the data stream that it is supposed to describe.
+The restriction of the layer so that it only describes the expected part of the data is called
+_layer limiting_.
+
+There are these ways to do layer limiting, that differ in important ways:
+- using enclosing elements of specified length
+- using a fixedLength layer
+- binding a DFDL variable specific to a particular layer
+- using a boundaryMark layer
+
+#### Layer Limiting using Enclosing Elements of Specified Length
+
+If the `sequence` for a layer is the model group of the
+complex type of an element, and that element has specified length (meaning it has
+`dfdl:lengthKind` property of `explicit`, `implicit`, `prefixed` or `pattern`), then the length
+of the element limits the length of the layer within it.
+
+In this case the length of the layer is limited when parsing, but it is NOT limited when
+unparsing.
+
+#### Layer Limiting using the `fixedLength` Utility Layer
+
+The layer `fixedLength` is a utility layer intended for use in conjunction with other layers.
+It dictates the parse _and unparse_ length to be the value of the `layerLength` DFDL variable.
+Setting or binding this DFDL variable fixes the length of the `fixedLength` layer, and then
+any layer on top of that layer will be limited by the underlying fixed length.
+
+#### Layer Limiting by Binding a DFDL Variable
+
+Some layers will define a DFDL variable that must be set to a non-negative
+integer value to specify the layer length that the layer will use.
+All checksum layers built with the Daffodil API `ChecksumLayer` base class,
+use the DFDL `layerLength` variable to specify the length (in bytes) of the layer.
+
+Layers may specify restrictions on the minimum and maximum allowed values of these lengths,
+and passing an out-of-range value for the variable is a processing error.
+
+#### Layer Limiting using the `boundaryMark` Utility Layer
+
+The `boundaryMark` layer uses a variable named `boundaryMark` which provides a string parameter
+to the layer.
+This string is a delimiter of the layer data when parsing, and when unparsing the string is
+inserted after the (otherwise unbounded length) layer data.
+Unlike DFDL's normal delimiter properties( `dfdl:separator`, `dfdl:terminator`, and
+`dfdl:initiator`), the scanning for a boundary mark string does not use any escape scheme.
+
+
+## Example: Line Folding
+
+Consider the line folding layer, specifically the `lineFolded_IMF` layer,
+which is built-in to Daffodil.
+
+Line folding is a way of encoding textual data formats so that no line of text is longer than
+a limited line length.
+
+Consider this data :
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem
+ ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+This data has been _line folded_ at roughly 72 characters by inserting a CRLF
+before an existing space in the data.
+Each line ends with a CRLF (\r\n) and the second through fourth lines begin
+with a space as a way of indicating that they are extension lines.
+This data is supposed to be reassembled to form a long single-line string by removing
+all CRLF pairs.
+
+The result should be this single longer string which does not contain any line endings:
+```
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
+```
+To achieve this we would use the `lineFolded_IMF` layer.
+This layer has a specific namespace which our DFDL schema will define a prefix `lf` for like this:
+```xml
+xmlns:lf="urn:org.apache.daffodil.layers.lineFolded"
+```
+Our DFDL schema will import the layer schema for the line-folded layer with this import:
+```xml
+<import namespace="urn:org.apache.daffodil.layers.lineFolded"
+ schemaLocation="/org/apache/daffodil/layers/xsd/lineFoldedLayer.dfdl.xsd"/>
+```
+Then the layer is incorporated into our DFDL schema like this:
+```xml
+<sequence dfdlx:layer="lf:lineFolded_IMF">
+ ... elements to be parsed from the unfolded layer data go here ...
+</sequence>
+```
+You can see that use of a layer is described using the `dfdx:layer` property, and the specific layer
+is identified by a QName using the previously defined namespace prefix.
+The scope of the layer is the duration of the sequence it appears on.
+The `dfdlx:layer` property can _only_ be used on an XSD `sequence`.
+
+The `lineFolded_IMF` layer does not define any DFDL variables in its namespace as it has no
+parameters and produces no results.
+As of Daffodil 3.8.0, the `import` statement above is optional for the
+line-folded layers as the DFDL schema file `lineFoldedLayer.dfdl.xsd` does not contain any
+definitions.
+In the future however, parameters may be added, so for uniformity all layers define a DFDL
+schema to be imported as part of using the layer.
+
+If the line-folding applies to the entire data stream (not uncommon for formats that use line
+folding), then what has been described is all one needs to describe the layer aspects of a data
+format.
+Other examples will show how the layer length can be limited to a sub-region of the data.
+
+More detailed documentation for the [Line Folded Layers](#LineFoldedLayers) is below.
+
+## Example: Base64, GZip, and BoundaryMark Layers used Together
+
+In this example, the data consists of a preliminary string, a section of CSV-like data, and a
+final string element.
+The CSV-like data section in the middle is both gzipped, and then base64 encoded, with the
+boundary value mark of "`=_END_=`" indicating where the base64 part ends.
+
+The point of the preliminary string and the final string is to show that this sort of layering
+composes properly with surrounding data elements in the schema. Those strings could be anything
+at all, they're just strings to make this example simpler.
+
+The physical data string/stream looks like this:
+```text
+example of data before the base64 gzipped part
+H4sIAAAAAAAA/y3JQQqAIBCF4b1nmYG0iNxG+84wpqFhI4xuun0Gbd4P78tUG5xJ+t7J+xxg21dV
+79QiSHFBGjgh9oVB22XBYUQzqatErv36CjGwPJ/OOGg0Y1cOFUgegoPaLwa1VS9htd+UbgAAAA==
+=_END_=
+example of data after the base64 gzipped part
+```
+That base64 encoded part, if decoded and decompressed looks like this CSV-like data:
+```
+last,first,middle,DOB
+smith,robert,brandon,1988-03-24
+johnson,john,henry,1986-01-23
+jones,arya,cat,1986-02-19
+```
+
+The Infoset, presented as XML, that we will create for this entire data stream
+using Daffodil parsing looks like:
+```xml
+<ex:data>
+ <before>example of data before the base64 gzipped part</before>
+ <header><title>last</title><title>first</title><title>middle</title><title>DOB</title></header>
+ <record><item>smith</item><item>robert</item><item>brandon</item><item>1988-03-24</item></record>
+ <record><item>johnson</item><item>john</item><item>henry</item><item>1986-01-23</item></record>
+ <record><item>jones</item><item>arya</item><item>cat</item><item>1986-02-19</item></record>
+ <after>example of data after the base64 gzipped part</after>
+</ex:data>
+```
+
+With that characterization of the problem, let's look at the DFDL schema with the Daffodil
+layering properties to do the boundaryMark, base64 decode, and gzip decompression.
+(Note that each of these layers is described further in its own section below on this page.)
+
+First there is the start of the schema which introduces namespace prefix definitions for DFDL,
+for the DFDL extensions (dfdlx) prefix which is used for the `dfdlx:layer` property.
+
+This also sets up 3 namespace prefixes for the namespaces of the boundaryMark (bm), gzip (gz)
+and base64_MIME (b64) layers which will be imported just after this header.
+
+```xml
+<schema
+ xmlns="http://www.w3.org/2001/XMLSchema"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:fn="http://www.w3.org/2005/xpath-functions"
+ xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions"
+ xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:bm="urn:org.apache.daffodil.layers.boundaryMark"
+ xmlns:b64="urn:org.apache.daffodil.layers.base64_MIME"
+ xmlns:gz="urn:org.apache.daffodil.layers.gzip"
+ xmlns:ex="http://example.com"
+ targetNamespace="http://example.com">
+```
+Next we import the Daffodil built in general format which provides most DFDL properties with
+sensible starting values, and import the namespaces of our boundaryMark, gzip, and base64_MIME
+layers, and lastly, set the default format properties for use by this schema file:
+```xml
+<include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+
+<import namespace="urn:org.apache.daffodil.layers.boundaryMark"
+ schemaLocation="/org/apache/daffodil/layers/xsd/boundaryMarkLayer.dfdl.xsd"/>
+<import namespace="urn:org.apache.daffodil.layers.gzip"
+ schemaLocation="/org/apache/daffodil/layers/xsd/gzipLayer.dfdl.xsd"/>
+<import namespace="urn:org.apache.daffodil.layers.base64_MIME"
+ schemaLocation="/org/apache/daffodil/layers/xsd/base64_MIMELayer.dfdl.xsd"/>
+
+<annotation><appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:format ref="ex:GeneralFormat" representation="binary"/>
+</appinfo></annotation>
+```
+The root element of the schema is `data`, and this element has type `dataType`.
+(A separate named complex type definition for the root element is a
+recommended style for DFDL schemas as it allows someone else to reuse the schema without
+getting involved in element namespace prefix complexities.)
+The `dataType` just shows us the `before` and `after` elements, and references a group
+named `layeredDataGroup` for the layered part in between.
+```xml
+<element name="data" type="ex:dataType"/>
+
+<complexType name="dataType">
+ <sequence dfdl:separator="%NL;">
+ <element name="before" type="string" dfdl:lengthKind="delimited"/>
+ <group ref="ex:layeredDataGroup"/>
+ <element name="after" type="string" dfdl:lengthKind="delimited"/>
+ </sequence>
+</complexType>
+```
+Now we see the definition of the layering.
+All 3 layers are declared in this named group.
+The lines here are numbered so that the text that follows can refer to them:
+```xml
+1 <group name="layeredDataGroup">
+2 <sequence dfdlx:layer="bm:boundaryMark">
+3 <annotation>
+4 <appinfo source="http://www.ogf.org/dfdl/">
+5 <dfdl:newVariableInstance ref="bm:layerEncoding" defaultValue="iso-8859-1"/>
+6 <dfdl:newVariableInstance ref="bm:boundaryMark" defaultValue="=_END_="/>
+7 </appinfo>
+8 </annotation>
+9 <sequence dfdlx:layer="b64:base64_MIME">
+10 <sequence dfdlx:layer="gz:gzip">
+11 <group ref="ex:dataFieldsGroup"/>
+12 </sequence>
+13 </sequence>
+14 </sequence>
+15 </group>
+```
+In the above, at line 2 we see the sequence declares the boundaryMark layer.
+This layer requires 2 parameters to be passed by binding DFDL variables, which occurs on
+lines 5 and 6.
+Line 5 binds the `layerEncoding` variable and note the prefix since DFDL variable names can be
+in a specific namespace.
+Line 6 binds the `boundaryMark` variable to the string "`=_END_=`".
+Note that the variable and the layer both are named `boundaryMark`, but this is just coincidence.
+Layers do not have to have parameters that match their name, nor even any parameters at all.
+The boundary mark is the string which marks the end of the data based on scanning for that ending
+mark.
+
+The data region now being isolated, the `boundaryMark` layer contains just one sequence, and line
+9 which itself declares that the data is base64 encoded by specifying the base64_MIME layer.
+This layer has no parameters, so there are no `dfdl:newVariableInstance` statements to bind any
+variables.
+The region of data that is base64 encoded is then described by the sequence at line 10 which
+declares that the data is compressed by the `gzip` algorithm.
+
+Finally, at line 11 we reference a group which describes what the data format of the underlying
+CSV-like data.
+That group's definition is regular DFDL without any of the complexity of layers.
+This is useful for testing, as data can be parsed using this group (in the complexType of an
+element) to ensure it works properly without involving the complexity of layers.
+This group definition is the last thing in the schema:
+```xml
+<group name="dataFieldsGroup">
+ <sequence dfdl:separator="%NL;" dfdl:separatorPosition="postfix">
+ <element name="header" minOccurs="0" dfdl:occursCountKind="implicit">
+ <complexType>
+ <sequence dfdl:separator=",">
+ <element name="title" type="xs:string" maxOccurs="unbounded" dfdl:lengthKind="delimited"/>
+ </sequence>
+ </complexType>
+ </element>
+ <element name="record" maxOccurs="unbounded">
+ <complexType>
+ <sequence dfdl:separator=",">
+ <element name="item" type="xs:string" maxOccurs="unbounded"
+ dfdl:lengthKind="delimited"
+ dfdl:occursCount="{ fn:count(../../header/title) }"
+ dfdl:occursCountKind="expression"/>
+ </sequence>
+ </complexType>
+ </element>
+ </sequence>
+</group>
+
+</schema>
+```
+The above schema works both to parse, but also to unparse this data.
+
+----
+
+# Using Custom Plug-In Layers
+
+A custom plug-in layer is used in the same manner as the built-in Daffodil layers with just a few
+additional details:
+1. The custom layer will have a DFDL schema that declares any DFDL variables it uses, and
+ defines any data formats needed to facilitate use of the layer.
+ This schema will need to be included/imported.
+ Normally this schema would be a small single-file schema and it can be loaded from the
+ classpath.
+2. The custom layer's compiled Java/Scala code is dynamically loaded using the
+ Java Service Provider Interface (SPI).
+ Hence, they must be compiled into Jar files with specific META-INF metadata enclosed, and
+ these jars must appear on the Java CLASSPATH so that they can be found and loaded.
+
+The layer API is defined via Java clases and interfaces to enable writing of custom layers in
+either Java or Scala.
+(One of the built-in layers (Gzip) is written in Java now, by way of proving
+that one can write a Layer in Java.)
+
+Transformer layer classes are derived from the
+[`Layer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html)
+base class.
+
+Checksum layer classes are derived from the
+[`ChecksumLayer`](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/ChecksumLayer.html)
+base class.
+
+Further details on how to define custom plug-in layers is in the Javadoc for the
+[Layer API](../docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/package-summary.html)
+
+----
+
+# Daffodil Built-In Layer Documentation
+
+Each of the layers built-in to the Daffodil implementation are documented in a section below
+which gives the name, namespace, variables, and some usage notes.
+
+----
+
+## Base64 MIME Layer
+
+- Name: base64_MIME
+- Namespace URI: urn:org.apache.daffodil.layers.base64_MIME
+- No DFDL variables are read or written.
+
+This uses the standard `java.util.Base64` classes.
+
+----
+
+## BoundaryMark Layer
+
+- Name: boundaryMark
+- Namespace URI: urn:org.apache.daffodil.layers.boundaryMark
+- Reads variables
+ - `boundaryMark` a string which delimits the end of the layer data.
+ - No escape schemes are applied.
+ - DFDL character entities and character class entities are not allowed.
+ - `layerEncoding` a string which names a character set encoding which is used in the scanning
+ for the boundary mark.
+
+Isolates text data by way of a boundary mark string.
+This is like a terminating delimiter of a DFDL element or sequence/ choice group, except that
+there is no escaping mechanism, so the data cannot in any way contain the boundary mark string,
+and elements within the layer can have any `dfdl:lengthKind`, but this does not affect the search
+for the boundary mark.
+
+For example, when using an element with `dfdl:lengthKind="delimited"` and a `dfdl:terminator="END"`,
+if that element contains a child element with `dfdl:lengthKind="explicit"`, then the search for
+the "END" terminator is suspended for the length of the child element, and that search resumes
+after the child element's length has been parsed.
+
+In contrast to this, if a boundary mark layer is used with the `boundaryMark` variable bound
+to "END", then the data stream is decoded as characters in the charset encoding given by the
+`layerEncoding` variable, and the layer continues until the "END" is found. The `dfdl:lengthKind`
+of any child element enclosed within the layer, or even the lengths of other layers found
+within the scope of this boundary mark layer are not considered and do not disrupt the search
+for the boundary mark string.
+
+----
+
+## Byte-Swapping Layers
+
+- Layer Names:
+ - `twobyteswap`
+ - `fourbyteswap`
+- Namespace URI: urn:org.apache.daffodil.layers.byteSwap
+- No DFDL Variables are read or set
+- There are two layer names for two variations:
Review Comment:
Is this bullet missing something after the colon? Maybe it can just be removed? The below notes seems to cover what this is talking about.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] WIP: Checkpoint - start on layers documentation in layers.md [daffodil-site]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on code in PR #132:
URL: https://github.com/apache/daffodil-site/pull/132#discussion_r1609853753
##########
site/layersTOC.md:
##########
@@ -0,0 +1,64 @@
+---
+description: Pluggable Extensions to Enable Algorithmic Transformations
+ in DFDL
+group: 'nav-right'
+layout: page
+title: 'Layers - Algorithmic
+ Extensions for DFDL'
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+### Table of Contents
+
+<!--
+{% comment %}
+NOTE: Generate using `pandoc -s --toc layers.md -o layersTOC.md`
+
+Followed by fixing up the layerTOC.md page so it works.
+{% endcomment %}
+-->
+
+- [Introduction](#introduction)
+ - [Built-in Layers](#built-in-layers)
+ - [Custom Plug-In Layers](#custom-plug-in-layers)
+ - [Transforming Layers and Checksum
+ Layers](#transforming-layers-and-checksum-layers)
+ - [Transforming Layers](#transforming-layers)
+ - [Checksum Layers](#checksum-layers)
+ - [Layer Composition](#layer-composition)
+- [Using Layers](#using-layers)
+ - [Example: Line Folding](#example-line-folding)
+ - [Example: Base64, GZip, and
+ BoundaryMark](#example-base64-gzip-and-boundarymark)
+- [Defining Custom Plug-In Layers](#defining-custom-plug-in-layers)
+ - [Layer Variables:](#layer-variables)
+ - [Layer Exception Handling:](#layer-exception-handling)
+ - [Compatibility with Daffodil 3.7.0 and
+ prior](#compatibility-with-daffodil-3.7.0-and-prior)
+- [Daffodil Built-In Layer
+ Documentation](#daffodil-built-in-layer-documentation)
+ - [Line Folded Layers](#line-folded-layers)
+ - [General Usage](#general-usage)
+ - [Layer Name: lineFolded\_IMF](#layer-name-linefolded_imf)
+ - [Layer Name:
+ lineFolded\_iCalendar](#layer-name-linefolded_icalendar)
+
+
+{% include_relative layers.md %}
Review Comment:
It looks like kramdown (the markdown parser used by jekyll) supports automatic generation of table of contents: https://kramdown.gettalong.org/converter/html.html#toc
If that works decently we should use that so these links don't get out of sync as updates are made.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org