You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@daffodil.apache.org by GitBox <gi...@apache.org> on 2022/11/15 22:24:46 UTC
[GitHub] [daffodil] tuxji commented on a diff in pull request #876: Add Daffodil Developer Guide
tuxji commented on code in PR #876:
URL: https://github.com/apache/daffodil/pull/876#discussion_r1023016404
##########
DEVELOP.md:
##########
@@ -0,0 +1,485 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Daffodil Developer Guide
+
+This guide is for a software engineer who wants to develop Daffodil
+code. This guide will help you become familiar with DFDL schemas,
+DFDL processors, and Daffodil development including its code style,
+code workflow, development environment, directory organization,
+documentation, and tests.
+
+## DFDL
+
+The [Data Format Description
+Language](https://en.wikipedia.org/wiki/Data_Format_Description_Language)
+(DFDL) is a language used to describe almost all data formats both
+logically and physically. DFDL is not a data format or a procedural
+language; rather, it is a data modeling language based on a subset of
+XML Schema annotated with DFDL properties describing the
+representation and layout of each element of the schema inside a
+native text or binary data format. A DFDL schema allows data to be
+converted between its native data format (physical representation,
+also called a text or binary file) and a DFDL information set (logical
+representation, also called an infoset) such as an XML document, EXI
+document, JSON document, SAX callbacks, or several document object
+model APIs in memory (JDOM, Scala Node, W3C DOM). When you have a
+DFDL schema for a native data format, you can pick whichever infoset
+type is easiest for you or your application to use and tell a DFDL
+processor to read ("parse") a text or binary file from its native data
+format to that infoset type. You or your application can do whatever
+you need to do with the infoset and then you can use the same DFDL
+schema and DFDL processor to write ("unparse") the infoset back to its
+native text or binary file format again, completing a round trip from
+native data to infoset to native data again.
+
+Using DFDL avoids inventing a completely new data modeling language,
+avoids writing any parsing and serialization code (with all the bugs
+that normally arise from implementing such code procedurally), and
+makes it much easier to convert any native data format to an infoset,
+operate on the infoset, and convert an infoset back to its native data
+format again.
+
+To learn more about DFDL, you can watch two short
+[videos](https://community.ibm.com/community/user/integration/viewdocument/get-started-with-the-data-format-de)
+put together by Steve Hanson, co-chair of the DFDL working group, read
+a
+[slideshow](https://www.slideshare.net/mbeckerle/dfdl-and-apache-daffodil-overview-from-owl-cyber-defense)
+written by Mike Beckerle, co-chair of the DFDL working group, or go
+through some [tutorials](http://www.xfront.com/DFDL/) written by Roger
+Costello, chair of the DFTVL working group (Data Format Transformation
+and Validation Language, a future not-yet-defined language to specify
+policies for cross-domain system devices).
+
+## Apache Daffodil, IBM DFDL, and ESA DFDL4S
+
+The standards organization in which DFDL started, the Open Grid Forum,
+required 2 implementations in order to move forward with the
+standardization process. This means that there are two leading DFDL
+processors, a commercial implementation called IBM DFDL bundled into
+IBM's [Integration
+Bus](https://www.ibm.com/docs/en/integration-bus/10.0?topic=model-data-format-description-language-dfdl)
+products and an open source implementation called [Apache
+Daffodil](https://daffodil.apache.org/) hosted by the Apache Software
+Foundation. The European Space Agency also has created a proprietary
+implementation called [ESA
+DFDL4S](https://eop-cfi.esa.int/index.php/applications/dfdl4s), which
+can be used only with their satellite communication formats and is
+provided only in the form of binary libraries, not source code.
+
+Among these three DFDL processors, Apache Daffodil is considered the
+most modern and thorough implementation of the [Data Format
+Description Language v1.0
+Specification](https://daffodil.apache.org/docs/dfdl/). Even so,
+Apache Daffodil lists some [unsupported
+features](https://daffodil.apache.org/unsupported/) of the DFDL
+specification. IBM DFDL lists some more [unsupported
+features](https://www.ibm.com/docs/en/integration-bus/10.0?topic=dfdl-unsupported-features)
+and also lists some [implementation-specific
+limits](https://www.ibm.com/docs/en/integration-bus/10.0?topic=dfdl-implementation-specific-limits).
+These limitations will not prevent you from writing DFDL schemas for
+almost all data formats, but they will reveal which parts of the DFDL
+specification are rarely used.
+
+## Daffodil Development
+
+The Apache Software Foundation hosts the Apache Daffodil project on
+the following ASF infrastructure:
+
+- Daffodil's [issue
+ tracker](https://issues.apache.org/jira/projects/DAFFODIL/) is
+ hosted on JIRA
+- Daffodil's
+ [users](https://lists.apache.org/list.html?users@daffodil.apache.org),
+ [dev](https://lists.apache.org/list.html?dev@daffodil.apache.org),
+ and
+ [commits](https://lists.apache.org/list.html?commits@daffodil.apache.org)
+ mailing lists are hosted on Apache Pony Mail
Review Comment:
Yes, I've changed to "mailing list archives" and added in parentheses that subscription is required to send to these lists. The archive pages have "subscribe" buttons so we don't really need redundant subscribe links.
##########
DEVELOP.md:
##########
@@ -0,0 +1,485 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Daffodil Developer Guide
+
+This guide is for a software engineer who wants to develop Daffodil
+code. This guide will help you become familiar with DFDL schemas,
+DFDL processors, and Daffodil development including its code style,
+code workflow, development environment, directory organization,
+documentation, and tests.
+
+## DFDL
+
+The [Data Format Description
+Language](https://en.wikipedia.org/wiki/Data_Format_Description_Language)
+(DFDL) is a language used to describe almost all data formats both
Review Comment:
I converted the first few links, skipped some one-off links, encountered some links that needed better labels, found out how to use labels (a second pair of [] after the first []), liked the idea, and ended up converting almost all the links. I found 4 benefits which actually should reduce future maintenance burden, not increase it:
1. Text sections can become easier to edit.
2. Link text and labels can be made different by using two pairs of [].
3. Links can be reused in several places without duplicating the urls.
4. Categories of urls can be grouped together so future en masse updates can be easier.
##########
DEVELOP.md:
##########
@@ -0,0 +1,485 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Daffodil Developer Guide
+
+This guide is for a software engineer who wants to develop Daffodil
+code. This guide will help you become familiar with DFDL schemas,
+DFDL processors, and Daffodil development including its code style,
+code workflow, development environment, directory organization,
+documentation, and tests.
+
+## DFDL
+
+The [Data Format Description
+Language](https://en.wikipedia.org/wiki/Data_Format_Description_Language)
+(DFDL) is a language used to describe almost all data formats both
+logically and physically. DFDL is not a data format or a procedural
+language; rather, it is a data modeling language based on a subset of
+XML Schema annotated with DFDL properties describing the
+representation and layout of each element of the schema inside a
+native text or binary data format. A DFDL schema allows data to be
+converted between its native data format (physical representation,
+also called a text or binary file) and a DFDL information set (logical
+representation, also called an infoset) such as an XML document, EXI
+document, JSON document, SAX callbacks, or several document object
+model APIs in memory (JDOM, Scala Node, W3C DOM). When you have a
+DFDL schema for a native data format, you can pick whichever infoset
+type is easiest for you or your application to use and tell a DFDL
+processor to read ("parse") a text or binary file from its native data
+format to that infoset type. You or your application can do whatever
+you need to do with the infoset and then you can use the same DFDL
+schema and DFDL processor to write ("unparse") the infoset back to its
+native text or binary file format again, completing a round trip from
+native data to infoset to native data again.
+
+Using DFDL avoids inventing a completely new data modeling language,
+avoids writing any parsing and serialization code (with all the bugs
+that normally arise from implementing such code procedurally), and
+makes it much easier to convert any native data format to an infoset,
+operate on the infoset, and convert an infoset back to its native data
+format again.
+
+To learn more about DFDL, you can watch two short
+[videos](https://community.ibm.com/community/user/integration/viewdocument/get-started-with-the-data-format-de)
+put together by Steve Hanson, co-chair of the DFDL working group, read
+a
+[slideshow](https://www.slideshare.net/mbeckerle/dfdl-and-apache-daffodil-overview-from-owl-cyber-defense)
+written by Mike Beckerle, co-chair of the DFDL working group, or go
+through some [tutorials](http://www.xfront.com/DFDL/) written by Roger
+Costello, chair of the DFTVL working group (Data Format Transformation
+and Validation Language, a future not-yet-defined language to specify
+policies for cross-domain system devices).
+
+## Apache Daffodil, IBM DFDL, and ESA DFDL4S
+
+The standards organization in which DFDL started, the Open Grid Forum,
+required 2 implementations in order to move forward with the
+standardization process. This means that there are two leading DFDL
+processors, a commercial implementation called IBM DFDL bundled into
+IBM's [Integration
+Bus](https://www.ibm.com/docs/en/integration-bus/10.0?topic=model-data-format-description-language-dfdl)
+products and an open source implementation called [Apache
+Daffodil](https://daffodil.apache.org/) hosted by the Apache Software
+Foundation. The European Space Agency also has created a proprietary
+implementation called [ESA
+DFDL4S](https://eop-cfi.esa.int/index.php/applications/dfdl4s), which
+can be used only with their satellite communication formats and is
+provided only in the form of binary libraries, not source code.
+
+Among these three DFDL processors, Apache Daffodil is considered the
+most modern and thorough implementation of the [Data Format
+Description Language v1.0
+Specification](https://daffodil.apache.org/docs/dfdl/). Even so,
+Apache Daffodil lists some [unsupported
+features](https://daffodil.apache.org/unsupported/) of the DFDL
+specification. IBM DFDL lists some more [unsupported
+features](https://www.ibm.com/docs/en/integration-bus/10.0?topic=dfdl-unsupported-features)
+and also lists some [implementation-specific
+limits](https://www.ibm.com/docs/en/integration-bus/10.0?topic=dfdl-implementation-specific-limits).
+These limitations will not prevent you from writing DFDL schemas for
+almost all data formats, but they will reveal which parts of the DFDL
+specification are rarely used.
+
+## Daffodil Development
+
+The Apache Software Foundation hosts the Apache Daffodil project on
+the following ASF infrastructure:
+
+- Daffodil's [issue
+ tracker](https://issues.apache.org/jira/projects/DAFFODIL/) is
+ hosted on JIRA
+- Daffodil's
+ [users](https://lists.apache.org/list.html?users@daffodil.apache.org),
+ [dev](https://lists.apache.org/list.html?dev@daffodil.apache.org),
+ and
+ [commits](https://lists.apache.org/list.html?commits@daffodil.apache.org)
+ mailing lists are hosted on Apache Pony Mail
+- Daffodil's [source code](https://github.com/apache/daffodil) is
+ hosted on GitHub
+- Daffodil's
+ [wiki](https://cwiki.apache.org/confluence/display/DAFFODIL/) is
+ hosted on Confluence
+- Daffodil's [website](https://daffodil.apache.org/) is hosted on
+ Apache with static content generated by Jekyll using jekyll-asciidoc
+ and asciidoctor-diagram plugins
+
+A good Daffodil developer has a GitHub account and two copies of
+Daffodil's source code repository, a reading copy directly cloned from
+the Apache repository and a working copy cloned from the developer's
+own fork of the Apache repository. A good Daffodil developer also
+gets an account on the Apache Software Foundation's JIRA and
+Confluence servers and subscribes to all three Daffodil mailing lists.
+
+You can clone your reading copy directly from the Apache repository,
+but you should never make any changes in it. The only commands you
+should ever run in your reading copy are `git pull`, `git log
+ORIG_HEAD..HEAD`, and `git diff ORIG_HEAD..HEAD` in order to see what
+changes have been made by other developers since your last pull from
+the Apache repository. Your reading copy should remain an exact copy
+of the Apache source code repository at all times which you can use
+for reading or occasionally running `diff` commands between your
+reading copy and working copy. To be safe, rename your reading copy
+from `daffodil` to `daffodil-asf` and never edit files or run `sbt` in
+it so there will be nothing to push to the Apache repository even if
+you accidentally run `git push` from the wrong source tree.
+
+You can set up your working copy by forking the Apache repository to
+your own GitHub account and then cloning your working copy from your
+fork. Cloning your working copy from your fork instead of the Apache
+repository ensures that any changes you commit in your working copy
+will be pushed safely to only your own fork, not the Apache
+repository.
Review Comment:
I appreciate the clarification about the write permission. Now I send readers to the "Code Contributor Workflow" first and then explain how to maintain a reading copy only as advice for Daffodil committers who do have write access to Daffodil's source code repositories, so they don't accidentally push anything from their reading copies.
BUILD, CONTRIBUTE, DEVELOP, README... I like it, although not in this PR (CONTRIBUTE and DEVELOP overlap a lot semantically, though).
##########
DEVELOP.md:
##########
@@ -0,0 +1,485 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Daffodil Developer Guide
+
+This guide is for a software engineer who wants to develop Daffodil
+code. This guide will help you become familiar with DFDL schemas,
+DFDL processors, and Daffodil development including its code style,
+code workflow, development environment, directory organization,
+documentation, and tests.
+
+## DFDL
+
+The [Data Format Description
+Language](https://en.wikipedia.org/wiki/Data_Format_Description_Language)
+(DFDL) is a language used to describe almost all data formats both
+logically and physically. DFDL is not a data format or a procedural
+language; rather, it is a data modeling language based on a subset of
+XML Schema annotated with DFDL properties describing the
+representation and layout of each element of the schema inside a
+native text or binary data format. A DFDL schema allows data to be
+converted between its native data format (physical representation,
+also called a text or binary file) and a DFDL information set (logical
+representation, also called an infoset) such as an XML document, EXI
+document, JSON document, SAX callbacks, or several document object
+model APIs in memory (JDOM, Scala Node, W3C DOM). When you have a
+DFDL schema for a native data format, you can pick whichever infoset
+type is easiest for you or your application to use and tell a DFDL
+processor to read ("parse") a text or binary file from its native data
+format to that infoset type. You or your application can do whatever
+you need to do with the infoset and then you can use the same DFDL
+schema and DFDL processor to write ("unparse") the infoset back to its
+native text or binary file format again, completing a round trip from
+native data to infoset to native data again.
+
+Using DFDL avoids inventing a completely new data modeling language,
+avoids writing any parsing and serialization code (with all the bugs
+that normally arise from implementing such code procedurally), and
+makes it much easier to convert any native data format to an infoset,
+operate on the infoset, and convert an infoset back to its native data
+format again.
+
+To learn more about DFDL, you can watch two short
+[videos](https://community.ibm.com/community/user/integration/viewdocument/get-started-with-the-data-format-de)
+put together by Steve Hanson, co-chair of the DFDL working group, read
+a
+[slideshow](https://www.slideshare.net/mbeckerle/dfdl-and-apache-daffodil-overview-from-owl-cyber-defense)
+written by Mike Beckerle, co-chair of the DFDL working group, or go
+through some [tutorials](http://www.xfront.com/DFDL/) written by Roger
+Costello, chair of the DFTVL working group (Data Format Transformation
+and Validation Language, a future not-yet-defined language to specify
+policies for cross-domain system devices).
+
+## Apache Daffodil, IBM DFDL, and ESA DFDL4S
+
+The standards organization in which DFDL started, the Open Grid Forum,
+required 2 implementations in order to move forward with the
+standardization process. This means that there are two leading DFDL
+processors, a commercial implementation called IBM DFDL bundled into
+IBM's [Integration
+Bus](https://www.ibm.com/docs/en/integration-bus/10.0?topic=model-data-format-description-language-dfdl)
+products and an open source implementation called [Apache
+Daffodil](https://daffodil.apache.org/) hosted by the Apache Software
+Foundation. The European Space Agency also has created a proprietary
+implementation called [ESA
+DFDL4S](https://eop-cfi.esa.int/index.php/applications/dfdl4s), which
+can be used only with their satellite communication formats and is
+provided only in the form of binary libraries, not source code.
+
+Among these three DFDL processors, Apache Daffodil is considered the
+most modern and thorough implementation of the [Data Format
+Description Language v1.0
+Specification](https://daffodil.apache.org/docs/dfdl/). Even so,
+Apache Daffodil lists some [unsupported
+features](https://daffodil.apache.org/unsupported/) of the DFDL
+specification. IBM DFDL lists some more [unsupported
+features](https://www.ibm.com/docs/en/integration-bus/10.0?topic=dfdl-unsupported-features)
+and also lists some [implementation-specific
+limits](https://www.ibm.com/docs/en/integration-bus/10.0?topic=dfdl-implementation-specific-limits).
+These limitations will not prevent you from writing DFDL schemas for
+almost all data formats, but they will reveal which parts of the DFDL
+specification are rarely used.
+
+## Daffodil Development
+
+The Apache Software Foundation hosts the Apache Daffodil project on
+the following ASF infrastructure:
+
+- Daffodil's [issue
+ tracker](https://issues.apache.org/jira/projects/DAFFODIL/) is
+ hosted on JIRA
+- Daffodil's
+ [users](https://lists.apache.org/list.html?users@daffodil.apache.org),
+ [dev](https://lists.apache.org/list.html?dev@daffodil.apache.org),
+ and
+ [commits](https://lists.apache.org/list.html?commits@daffodil.apache.org)
+ mailing lists are hosted on Apache Pony Mail
+- Daffodil's [source code](https://github.com/apache/daffodil) is
+ hosted on GitHub
+- Daffodil's
+ [wiki](https://cwiki.apache.org/confluence/display/DAFFODIL/) is
+ hosted on Confluence
+- Daffodil's [website](https://daffodil.apache.org/) is hosted on
+ Apache with static content generated by Jekyll using jekyll-asciidoc
+ and asciidoctor-diagram plugins
+
+A good Daffodil developer has a GitHub account and two copies of
+Daffodil's source code repository, a reading copy directly cloned from
+the Apache repository and a working copy cloned from the developer's
+own fork of the Apache repository. A good Daffodil developer also
+gets an account on the Apache Software Foundation's JIRA and
+Confluence servers and subscribes to all three Daffodil mailing lists.
+
+You can clone your reading copy directly from the Apache repository,
+but you should never make any changes in it. The only commands you
+should ever run in your reading copy are `git pull`, `git log
+ORIG_HEAD..HEAD`, and `git diff ORIG_HEAD..HEAD` in order to see what
+changes have been made by other developers since your last pull from
+the Apache repository. Your reading copy should remain an exact copy
+of the Apache source code repository at all times which you can use
+for reading or occasionally running `diff` commands between your
+reading copy and working copy. To be safe, rename your reading copy
+from `daffodil` to `daffodil-asf` and never edit files or run `sbt` in
+it so there will be nothing to push to the Apache repository even if
+you accidentally run `git push` from the wrong source tree.
+
+You can set up your working copy by forking the Apache repository to
+your own GitHub account and then cloning your working copy from your
+fork. Cloning your working copy from your fork instead of the Apache
+repository ensures that any changes you commit in your working copy
+will be pushed safely to only your own fork, not the Apache
+repository.
+
+### Code Style
+
+Daffodil mandates standard Scala formatting and its code generally
+keeps close to that formatting consistently. No one has run an
+automatic Scala formatter on the codebase yet (there has not been much
+need since the code is already formatted pretty well) but
+[DAFFODIL-2133](https://issues.apache.org/jira/browse/DAFFODIL-2133)
+is waiting for someone willing to do that work.
+
+Daffodil mandates that at least 80% of new code and modified code
+should be covered by unit tests or TDML tests. Daffodil naming
+conventions and more are covered in the Confluence pages [Code Style
+Guidelines](https://cwiki.apache.org/confluence/display/DAFFODIL/Code+Style+Guidelines)
+and [Coding for
+Performance](https://cwiki.apache.org/confluence/display/DAFFODIL/Coding+for+Performance).
+
+### Code Workflow
+
+It is very important that you fork the Apache repository to your own
+GitHub account before you start making changes to Apache Daffodil
+source code. You should always make changes only to your own fork of
+the upstream Apache repository, that is, clone your working copy from
+your fork and push commits back to your own fork only. The same
+workflow applies to Apache Daffodil's other two source code
+repositories, that is, always fork these repositories and clone from
+your forks before you make any changes to the [Daffodil VS Code
+Extension](https://github.com/apache/daffodil-vscode) or [Daffodil
+Website](https://github.com/apache/daffodil-site). When you are ready
+to contribute code from any of your forks to any of these upstream
+Apache repositories, you must create a pull request from your fork,
+allow the Apache Daffodil developers to review your contribution, make
+any suggested changes, and wait for their approval before you or
+someone else can merge your code into one of Apache's source code
+repositories.
+
+The Daffodil developers want each pull request to add only one commit
+to the Apache repository to keep its git history as readable and
+bisectable as possible. Your pull request should fix a JIRA issue in
+its entirety but it should not fix 2 or more unrelated JIRA issues in
+the same commit either. Your pull request should start with only one
+commit in it before you ask the developers to review it and your pull
+request should end with only one commit in it after you receive enough
+reviewer approvals (two +1's for Daffodil and Daffodil VS Code
+Extension, one +1 for Daffodil Website). If you have made multiple
+commits to your pull request due to reviewer suggestions (which is the
+usual case), you must not merge these multiple commits into the Apache
+repository. You must rebase and squash your pull request back to one
+commit as shown in the [Code Contributor
+Workflow](https://cwiki.apache.org/confluence/display/DAFFODIL/Code+Contributor+Workflow),
+which you should read and follow carefully to ensure a successful code
+contribution to Apache Daffodil.
+
+Before you change Daffodil code, you will need to know how to use JIRA
+as well as GitHub. You will assign a JIRA issue to yourself before
+you start working on it to avoid redundant effort in case another
+developer decides to work on the same issue. Then you will mark the
+issue as fixed after your pull request is merged. Here are the steps
+you need to perform in JIRA to mark the issue resolved after your pull
+request is merged (please don't forget these steps):
+
+1. Click "Resolve Issue"
+2. Click the "Fix Version/s" dropdown and mark the issue as fixed in
+ the currently unreleased version
+3. Paste "Fixed in commit xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
+ into the comment field (replacing the xxx's with the correct hash)
+4. Press the "Resolve" button
+
+That's it. You can expect someone else to test the change and close
Review Comment:
Yes, I revised the instructions to be clearer.
##########
DEVELOP.md:
##########
@@ -0,0 +1,485 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Daffodil Developer Guide
+
+This guide is for a software engineer who wants to develop Daffodil
+code. This guide will help you become familiar with DFDL schemas,
+DFDL processors, and Daffodil development including its code style,
+code workflow, development environment, directory organization,
+documentation, and tests.
+
+## DFDL
+
+The [Data Format Description
+Language](https://en.wikipedia.org/wiki/Data_Format_Description_Language)
+(DFDL) is a language used to describe almost all data formats both
+logically and physically. DFDL is not a data format or a procedural
+language; rather, it is a data modeling language based on a subset of
+XML Schema annotated with DFDL properties describing the
+representation and layout of each element of the schema inside a
+native text or binary data format. A DFDL schema allows data to be
+converted between its native data format (physical representation,
+also called a text or binary file) and a DFDL information set (logical
+representation, also called an infoset) such as an XML document, EXI
+document, JSON document, SAX callbacks, or several document object
+model APIs in memory (JDOM, Scala Node, W3C DOM). When you have a
+DFDL schema for a native data format, you can pick whichever infoset
+type is easiest for you or your application to use and tell a DFDL
+processor to read ("parse") a text or binary file from its native data
+format to that infoset type. You or your application can do whatever
+you need to do with the infoset and then you can use the same DFDL
+schema and DFDL processor to write ("unparse") the infoset back to its
+native text or binary file format again, completing a round trip from
+native data to infoset to native data again.
+
+Using DFDL avoids inventing a completely new data modeling language,
+avoids writing any parsing and serialization code (with all the bugs
+that normally arise from implementing such code procedurally), and
+makes it much easier to convert any native data format to an infoset,
+operate on the infoset, and convert an infoset back to its native data
+format again.
+
+To learn more about DFDL, you can watch two short
+[videos](https://community.ibm.com/community/user/integration/viewdocument/get-started-with-the-data-format-de)
+put together by Steve Hanson, co-chair of the DFDL working group, read
+a
+[slideshow](https://www.slideshare.net/mbeckerle/dfdl-and-apache-daffodil-overview-from-owl-cyber-defense)
+written by Mike Beckerle, co-chair of the DFDL working group, or go
+through some [tutorials](http://www.xfront.com/DFDL/) written by Roger
+Costello, chair of the DFTVL working group (Data Format Transformation
+and Validation Language, a future not-yet-defined language to specify
+policies for cross-domain system devices).
+
+## Apache Daffodil, IBM DFDL, and ESA DFDL4S
+
+The standards organization in which DFDL started, the Open Grid Forum,
+required 2 implementations in order to move forward with the
+standardization process. This means that there are two leading DFDL
+processors, a commercial implementation called IBM DFDL bundled into
+IBM's [Integration
+Bus](https://www.ibm.com/docs/en/integration-bus/10.0?topic=model-data-format-description-language-dfdl)
+products and an open source implementation called [Apache
+Daffodil](https://daffodil.apache.org/) hosted by the Apache Software
+Foundation. The European Space Agency also has created a proprietary
+implementation called [ESA
+DFDL4S](https://eop-cfi.esa.int/index.php/applications/dfdl4s), which
+can be used only with their satellite communication formats and is
+provided only in the form of binary libraries, not source code.
+
+Among these three DFDL processors, Apache Daffodil is considered the
+most modern and thorough implementation of the [Data Format
+Description Language v1.0
+Specification](https://daffodil.apache.org/docs/dfdl/). Even so,
+Apache Daffodil lists some [unsupported
+features](https://daffodil.apache.org/unsupported/) of the DFDL
+specification. IBM DFDL lists some more [unsupported
+features](https://www.ibm.com/docs/en/integration-bus/10.0?topic=dfdl-unsupported-features)
+and also lists some [implementation-specific
+limits](https://www.ibm.com/docs/en/integration-bus/10.0?topic=dfdl-implementation-specific-limits).
+These limitations will not prevent you from writing DFDL schemas for
+almost all data formats, but they will reveal which parts of the DFDL
+specification are rarely used.
+
+## Daffodil Development
+
+The Apache Software Foundation hosts the Apache Daffodil project on
+the following ASF infrastructure:
+
+- Daffodil's [issue
+ tracker](https://issues.apache.org/jira/projects/DAFFODIL/) is
+ hosted on JIRA
+- Daffodil's
+ [users](https://lists.apache.org/list.html?users@daffodil.apache.org),
+ [dev](https://lists.apache.org/list.html?dev@daffodil.apache.org),
+ and
+ [commits](https://lists.apache.org/list.html?commits@daffodil.apache.org)
+ mailing lists are hosted on Apache Pony Mail
+- Daffodil's [source code](https://github.com/apache/daffodil) is
+ hosted on GitHub
+- Daffodil's
+ [wiki](https://cwiki.apache.org/confluence/display/DAFFODIL/) is
+ hosted on Confluence
+- Daffodil's [website](https://daffodil.apache.org/) is hosted on
+ Apache with static content generated by Jekyll using jekyll-asciidoc
+ and asciidoctor-diagram plugins
+
+A good Daffodil developer has a GitHub account and two copies of
+Daffodil's source code repository, a reading copy directly cloned from
+the Apache repository and a working copy cloned from the developer's
+own fork of the Apache repository. A good Daffodil developer also
+gets an account on the Apache Software Foundation's JIRA and
+Confluence servers and subscribes to all three Daffodil mailing lists.
+
+You can clone your reading copy directly from the Apache repository,
+but you should never make any changes in it. The only commands you
+should ever run in your reading copy are `git pull`, `git log
+ORIG_HEAD..HEAD`, and `git diff ORIG_HEAD..HEAD` in order to see what
+changes have been made by other developers since your last pull from
+the Apache repository. Your reading copy should remain an exact copy
+of the Apache source code repository at all times which you can use
+for reading or occasionally running `diff` commands between your
+reading copy and working copy. To be safe, rename your reading copy
+from `daffodil` to `daffodil-asf` and never edit files or run `sbt` in
+it so there will be nothing to push to the Apache repository even if
+you accidentally run `git push` from the wrong source tree.
+
+You can set up your working copy by forking the Apache repository to
+your own GitHub account and then cloning your working copy from your
+fork. Cloning your working copy from your fork instead of the Apache
+repository ensures that any changes you commit in your working copy
+will be pushed safely to only your own fork, not the Apache
+repository.
+
+### Code Style
+
+Daffodil mandates standard Scala formatting and its code generally
+keeps close to that formatting consistently. No one has run an
+automatic Scala formatter on the codebase yet (there has not been much
+need since the code is already formatted pretty well) but
+[DAFFODIL-2133](https://issues.apache.org/jira/browse/DAFFODIL-2133)
+is waiting for someone willing to do that work.
+
+Daffodil mandates that at least 80% of new code and modified code
+should be covered by unit tests or TDML tests. Daffodil naming
+conventions and more are covered in the Confluence pages [Code Style
+Guidelines](https://cwiki.apache.org/confluence/display/DAFFODIL/Code+Style+Guidelines)
+and [Coding for
+Performance](https://cwiki.apache.org/confluence/display/DAFFODIL/Coding+for+Performance).
+
+### Code Workflow
+
+It is very important that you fork the Apache repository to your own
+GitHub account before you start making changes to Apache Daffodil
+source code. You should always make changes only to your own fork of
+the upstream Apache repository, that is, clone your working copy from
+your fork and push commits back to your own fork only. The same
+workflow applies to Apache Daffodil's other two source code
+repositories, that is, always fork these repositories and clone from
+your forks before you make any changes to the [Daffodil VS Code
+Extension](https://github.com/apache/daffodil-vscode) or [Daffodil
+Website](https://github.com/apache/daffodil-site). When you are ready
+to contribute code from any of your forks to any of these upstream
+Apache repositories, you must create a pull request from your fork,
+allow the Apache Daffodil developers to review your contribution, make
+any suggested changes, and wait for their approval before you or
+someone else can merge your code into one of Apache's source code
+repositories.
+
+The Daffodil developers want each pull request to add only one commit
+to the Apache repository to keep its git history as readable and
+bisectable as possible. Your pull request should fix a JIRA issue in
+its entirety but it should not fix 2 or more unrelated JIRA issues in
+the same commit either. Your pull request should start with only one
+commit in it before you ask the developers to review it and your pull
+request should end with only one commit in it after you receive enough
+reviewer approvals (two +1's for Daffodil and Daffodil VS Code
+Extension, one +1 for Daffodil Website). If you have made multiple
+commits to your pull request due to reviewer suggestions (which is the
+usual case), you must not merge these multiple commits into the Apache
+repository. You must rebase and squash your pull request back to one
+commit as shown in the [Code Contributor
+Workflow](https://cwiki.apache.org/confluence/display/DAFFODIL/Code+Contributor+Workflow),
+which you should read and follow carefully to ensure a successful code
+contribution to Apache Daffodil.
+
+Before you change Daffodil code, you will need to know how to use JIRA
+as well as GitHub. You will assign a JIRA issue to yourself before
+you start working on it to avoid redundant effort in case another
+developer decides to work on the same issue. Then you will mark the
+issue as fixed after your pull request is merged. Here are the steps
+you need to perform in JIRA to mark the issue resolved after your pull
+request is merged (please don't forget these steps):
+
+1. Click "Resolve Issue"
+2. Click the "Fix Version/s" dropdown and mark the issue as fixed in
+ the currently unreleased version
+3. Paste "Fixed in commit xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
+ into the comment field (replacing the xxx's with the correct hash)
+4. Press the "Resolve" button
+
+That's it. You can expect someone else to test the change and close
+the JIRA issue so you don't need to, and should not, close the issue
+yourself.
+
+### Development Environment
+
+You will find build instructions in [BUILD.md](BUILD.md) and
+[README.md](README.md). You will find more instructions how to set up
+your development environment in the Confluence pages [Java
+Setup](https://cwiki.apache.org/confluence/display/DAFFODIL/Java+Setup+and+Notes)
+and [IDEA
+Setup](https://cwiki.apache.org/confluence/display/DAFFODIL/IntelliJ+IDEA+Setup).
+
+### Directory Organization
+
+Daffodil is composed of one top-level project and 20
+subdirectory-level modules (some of these modules contain only
+generators, macros, tests and tutorials). Daffodil's source code
+repository is organized as follows:
+
+ daffodil/
+ ├── .asf.yaml - Enables/disables parts of GitHub's UI
+ ├── .gitattributes - Excludes KEY from source releases
+ ├── .gitignore - Ignores auto-generated files from source control
+ ├── .sbtopts - Tells sbt to use 4GB of memory
+ ├── BUILD.md - Lists Daffodil's build requirements
+ ├── KEYS - Lists public keys of Daffodil developers
+ ├── LICENSE - Licenses Daffodil under the Apache License
+ ├── NOTICE - Lists notices affecting LICENSE
+ ├── README.md - Explains how to get started
+ ├── build.sbt - Tells sbt how to build Daffodil
+ ├── containers/
+ │ └── release-candidate - Defines Docker image used to build release
+ ├── daffodil-cli/
+ │ ├── README.md - Explains how to start Daffodil
+ │ ├── bin.LICENSE - Contains licenses of Daffodil and subcomponents
+ │ ├── bin.NOTICE - Contains notices of Daffodil and subcomponents
+ │ ├── build.sbt - Tells sbt how to build CLI and installers
+ │ └── src/ - Contains CLI source code, tests, scripts, etc.
+ ├── daffodil-core/
+ │ └── src/ - Contains Daffodil's core
+ ├── daffodil-io/
+ │ └── src/ - Contains Daffodil's input/output/charsets
+ ├── daffodil-japi/
+ │ ├── build.sbt - Tells sbt how to generate javadocs
+ │ └── src/ - Contains Daffodil's Java API
+ ├── daffodil-lib/
+ │ └── src/ - Contains Daffodil's utilities
+ ├── daffodil-macro-lib/
+ │ └── src/ - Defines Daffodil's Scala 2.x macros
+ ├── daffodil-propgen/
+ │ └── src/ - Contains generators to generate more source code
+ ├── daffodil-runtime1/
+ │ └── src/ - Contains Daffodil's Scala parser
+ ├── daffodil-runtime1-layers/
+ │ └── src/ - Contains Daffodil's Scala layer transformers
+ ├── daffodil-runtime1-unparser/
+ │ └── src/ - Contains Daffodil's Scala unparser
+ ├── daffodil-runtime2/
+ │ └── src/ - Contains Daffodil's C code generator
+ ├── daffodil-sapi/
+ │ └── src/ - Contains Daffodil's Scala API
+ ├── daffodil-schematron/
+ │ ├── README.md - Explains how to validate with Schematron rules
+ │ └── src/ - Contains Daffodil's Schematron validator
+ ├── daffodil-tdml-lib/
+ │ └── src/ - Contains Daffodil's TDML runner
+ ├── daffodil-tdml-processor/
+ │ └── src/ - Contains Daffodil's Scala & C TDML implementations
+ ├── daffodil-test/
+ │ └── src/ - Contains Daffodil's TDML tests
+ ├── daffodil-test-ibm1/
+ │ └── src/ - Contains more TDML tests from IBM & Tresys
+ ├── daffodil-udf/
+ │ ├── README.md - Explains how to implement user defined functions
+ │ └── src/ - Contains Daffodil's UDF interfaces
+ ├── project/
+ │ ├── Dependencies.scala - Defines third party dependencies
+ │ ├── Rat.scala - Lists files to be ignored by license audit tool
+ │ ├── build.properties - Defines which sbt version to use
+ │ └── plugins.sbt - Adds third party plugins to sbt
+ ├── test-stdLayout/
+ │ └── src/ - Contains TDML tests laid out in different ways
+ └── tutorials/
+ └── src/ - Contains Daffodil's TDML test tutorials
+
+When compiled into a command line interface, Daffodil is composed of a
+script, a log4j2 configuration file, 14 Daffodil jars, and 37
Review Comment:
All right, I've removed the numbers.
##########
DEVELOP.md:
##########
@@ -0,0 +1,485 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Daffodil Developer Guide
+
+This guide is for a software engineer who wants to develop Daffodil
+code. This guide will help you become familiar with DFDL schemas,
+DFDL processors, and Daffodil development including its code style,
+code workflow, development environment, directory organization,
+documentation, and tests.
+
+## DFDL
+
+The [Data Format Description
+Language](https://en.wikipedia.org/wiki/Data_Format_Description_Language)
+(DFDL) is a language used to describe almost all data formats both
+logically and physically. DFDL is not a data format or a procedural
+language; rather, it is a data modeling language based on a subset of
+XML Schema annotated with DFDL properties describing the
+representation and layout of each element of the schema inside a
+native text or binary data format. A DFDL schema allows data to be
+converted between its native data format (physical representation,
+also called a text or binary file) and a DFDL information set (logical
+representation, also called an infoset) such as an XML document, EXI
+document, JSON document, SAX callbacks, or several document object
+model APIs in memory (JDOM, Scala Node, W3C DOM). When you have a
+DFDL schema for a native data format, you can pick whichever infoset
+type is easiest for you or your application to use and tell a DFDL
+processor to read ("parse") a text or binary file from its native data
+format to that infoset type. You or your application can do whatever
+you need to do with the infoset and then you can use the same DFDL
+schema and DFDL processor to write ("unparse") the infoset back to its
+native text or binary file format again, completing a round trip from
+native data to infoset to native data again.
+
+Using DFDL avoids inventing a completely new data modeling language,
+avoids writing any parsing and serialization code (with all the bugs
+that normally arise from implementing such code procedurally), and
+makes it much easier to convert any native data format to an infoset,
+operate on the infoset, and convert an infoset back to its native data
+format again.
+
+To learn more about DFDL, you can watch two short
+[videos](https://community.ibm.com/community/user/integration/viewdocument/get-started-with-the-data-format-de)
+put together by Steve Hanson, co-chair of the DFDL working group, read
+a
+[slideshow](https://www.slideshare.net/mbeckerle/dfdl-and-apache-daffodil-overview-from-owl-cyber-defense)
+written by Mike Beckerle, co-chair of the DFDL working group, or go
+through some [tutorials](http://www.xfront.com/DFDL/) written by Roger
+Costello, chair of the DFTVL working group (Data Format Transformation
+and Validation Language, a future not-yet-defined language to specify
+policies for cross-domain system devices).
+
+## Apache Daffodil, IBM DFDL, and ESA DFDL4S
+
+The standards organization in which DFDL started, the Open Grid Forum,
+required 2 implementations in order to move forward with the
+standardization process. This means that there are two leading DFDL
+processors, a commercial implementation called IBM DFDL bundled into
+IBM's [Integration
+Bus](https://www.ibm.com/docs/en/integration-bus/10.0?topic=model-data-format-description-language-dfdl)
+products and an open source implementation called [Apache
+Daffodil](https://daffodil.apache.org/) hosted by the Apache Software
+Foundation. The European Space Agency also has created a proprietary
Review Comment:
Yes, I've added (ASF) here.
##########
DEVELOP.md:
##########
@@ -0,0 +1,485 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Daffodil Developer Guide
+
+This guide is for a software engineer who wants to develop Daffodil
+code. This guide will help you become familiar with DFDL schemas,
+DFDL processors, and Daffodil development including its code style,
+code workflow, development environment, directory organization,
+documentation, and tests.
+
+## DFDL
+
+The [Data Format Description
+Language](https://en.wikipedia.org/wiki/Data_Format_Description_Language)
+(DFDL) is a language used to describe almost all data formats both
+logically and physically. DFDL is not a data format or a procedural
+language; rather, it is a data modeling language based on a subset of
+XML Schema annotated with DFDL properties describing the
+representation and layout of each element of the schema inside a
+native text or binary data format. A DFDL schema allows data to be
+converted between its native data format (physical representation,
+also called a text or binary file) and a DFDL information set (logical
+representation, also called an infoset) such as an XML document, EXI
+document, JSON document, SAX callbacks, or several document object
+model APIs in memory (JDOM, Scala Node, W3C DOM). When you have a
+DFDL schema for a native data format, you can pick whichever infoset
+type is easiest for you or your application to use and tell a DFDL
+processor to read ("parse") a text or binary file from its native data
+format to that infoset type. You or your application can do whatever
+you need to do with the infoset and then you can use the same DFDL
+schema and DFDL processor to write ("unparse") the infoset back to its
+native text or binary file format again, completing a round trip from
+native data to infoset to native data again.
+
+Using DFDL avoids inventing a completely new data modeling language,
+avoids writing any parsing and serialization code (with all the bugs
+that normally arise from implementing such code procedurally), and
+makes it much easier to convert any native data format to an infoset,
+operate on the infoset, and convert an infoset back to its native data
+format again.
+
+To learn more about DFDL, you can watch two short
+[videos](https://community.ibm.com/community/user/integration/viewdocument/get-started-with-the-data-format-de)
+put together by Steve Hanson, co-chair of the DFDL working group, read
+a
+[slideshow](https://www.slideshare.net/mbeckerle/dfdl-and-apache-daffodil-overview-from-owl-cyber-defense)
+written by Mike Beckerle, co-chair of the DFDL working group, or go
+through some [tutorials](http://www.xfront.com/DFDL/) written by Roger
+Costello, chair of the DFTVL working group (Data Format Transformation
+and Validation Language, a future not-yet-defined language to specify
+policies for cross-domain system devices).
+
+## Apache Daffodil, IBM DFDL, and ESA DFDL4S
+
+The standards organization in which DFDL started, the Open Grid Forum,
+required 2 implementations in order to move forward with the
+standardization process. This means that there are two leading DFDL
+processors, a commercial implementation called IBM DFDL bundled into
+IBM's [Integration
+Bus](https://www.ibm.com/docs/en/integration-bus/10.0?topic=model-data-format-description-language-dfdl)
+products and an open source implementation called [Apache
+Daffodil](https://daffodil.apache.org/) hosted by the Apache Software
+Foundation. The European Space Agency also has created a proprietary
+implementation called [ESA
+DFDL4S](https://eop-cfi.esa.int/index.php/applications/dfdl4s), which
+can be used only with their satellite communication formats and is
+provided only in the form of binary libraries, not source code.
+
+Among these three DFDL processors, Apache Daffodil is considered the
+most modern and thorough implementation of the [Data Format
+Description Language v1.0
+Specification](https://daffodil.apache.org/docs/dfdl/). Even so,
+Apache Daffodil lists some [unsupported
+features](https://daffodil.apache.org/unsupported/) of the DFDL
+specification. IBM DFDL lists some more [unsupported
+features](https://www.ibm.com/docs/en/integration-bus/10.0?topic=dfdl-unsupported-features)
+and also lists some [implementation-specific
+limits](https://www.ibm.com/docs/en/integration-bus/10.0?topic=dfdl-implementation-specific-limits).
+These limitations will not prevent you from writing DFDL schemas for
+almost all data formats, but they will reveal which parts of the DFDL
+specification are rarely used.
+
+## Daffodil Development
+
+The Apache Software Foundation hosts the Apache Daffodil project on
+the following ASF infrastructure:
+
+- Daffodil's [issue
+ tracker](https://issues.apache.org/jira/projects/DAFFODIL/) is
+ hosted on JIRA
+- Daffodil's
+ [users](https://lists.apache.org/list.html?users@daffodil.apache.org),
+ [dev](https://lists.apache.org/list.html?dev@daffodil.apache.org),
+ and
+ [commits](https://lists.apache.org/list.html?commits@daffodil.apache.org)
+ mailing lists are hosted on Apache Pony Mail
+- Daffodil's [source code](https://github.com/apache/daffodil) is
+ hosted on GitHub
+- Daffodil's
+ [wiki](https://cwiki.apache.org/confluence/display/DAFFODIL/) is
+ hosted on Confluence
+- Daffodil's [website](https://daffodil.apache.org/) is hosted on
+ Apache with static content generated by Jekyll using jekyll-asciidoc
+ and asciidoctor-diagram plugins
+
+A good Daffodil developer has a GitHub account and two copies of
+Daffodil's source code repository, a reading copy directly cloned from
+the Apache repository and a working copy cloned from the developer's
+own fork of the Apache repository. A good Daffodil developer also
+gets an account on the Apache Software Foundation's JIRA and
+Confluence servers and subscribes to all three Daffodil mailing lists.
+
+You can clone your reading copy directly from the Apache repository,
+but you should never make any changes in it. The only commands you
+should ever run in your reading copy are `git pull`, `git log
+ORIG_HEAD..HEAD`, and `git diff ORIG_HEAD..HEAD` in order to see what
+changes have been made by other developers since your last pull from
+the Apache repository. Your reading copy should remain an exact copy
+of the Apache source code repository at all times which you can use
+for reading or occasionally running `diff` commands between your
+reading copy and working copy. To be safe, rename your reading copy
+from `daffodil` to `daffodil-asf` and never edit files or run `sbt` in
+it so there will be nothing to push to the Apache repository even if
+you accidentally run `git push` from the wrong source tree.
+
+You can set up your working copy by forking the Apache repository to
+your own GitHub account and then cloning your working copy from your
+fork. Cloning your working copy from your fork instead of the Apache
+repository ensures that any changes you commit in your working copy
+will be pushed safely to only your own fork, not the Apache
+repository.
+
+### Code Style
+
+Daffodil mandates standard Scala formatting and its code generally
+keeps close to that formatting consistently. No one has run an
+automatic Scala formatter on the codebase yet (there has not been much
+need since the code is already formatted pretty well) but
+[DAFFODIL-2133](https://issues.apache.org/jira/browse/DAFFODIL-2133)
+is waiting for someone willing to do that work.
+
+Daffodil mandates that at least 80% of new code and modified code
+should be covered by unit tests or TDML tests. Daffodil naming
+conventions and more are covered in the Confluence pages [Code Style
+Guidelines](https://cwiki.apache.org/confluence/display/DAFFODIL/Code+Style+Guidelines)
+and [Coding for
+Performance](https://cwiki.apache.org/confluence/display/DAFFODIL/Coding+for+Performance).
+
+### Code Workflow
+
+It is very important that you fork the Apache repository to your own
+GitHub account before you start making changes to Apache Daffodil
+source code. You should always make changes only to your own fork of
+the upstream Apache repository, that is, clone your working copy from
+your fork and push commits back to your own fork only. The same
+workflow applies to Apache Daffodil's other two source code
+repositories, that is, always fork these repositories and clone from
+your forks before you make any changes to the [Daffodil VS Code
+Extension](https://github.com/apache/daffodil-vscode) or [Daffodil
+Website](https://github.com/apache/daffodil-site). When you are ready
+to contribute code from any of your forks to any of these upstream
+Apache repositories, you must create a pull request from your fork,
+allow the Apache Daffodil developers to review your contribution, make
+any suggested changes, and wait for their approval before you or
+someone else can merge your code into one of Apache's source code
+repositories.
+
+The Daffodil developers want each pull request to add only one commit
+to the Apache repository to keep its git history as readable and
+bisectable as possible. Your pull request should fix a JIRA issue in
+its entirety but it should not fix 2 or more unrelated JIRA issues in
+the same commit either. Your pull request should start with only one
+commit in it before you ask the developers to review it and your pull
+request should end with only one commit in it after you receive enough
+reviewer approvals (two +1's for Daffodil and Daffodil VS Code
+Extension, one +1 for Daffodil Website). If you have made multiple
+commits to your pull request due to reviewer suggestions (which is the
+usual case), you must not merge these multiple commits into the Apache
+repository. You must rebase and squash your pull request back to one
+commit as shown in the [Code Contributor
+Workflow](https://cwiki.apache.org/confluence/display/DAFFODIL/Code+Contributor+Workflow),
+which you should read and follow carefully to ensure a successful code
+contribution to Apache Daffodil.
+
+Before you change Daffodil code, you will need to know how to use JIRA
+as well as GitHub. You will assign a JIRA issue to yourself before
+you start working on it to avoid redundant effort in case another
+developer decides to work on the same issue. Then you will mark the
+issue as fixed after your pull request is merged. Here are the steps
+you need to perform in JIRA to mark the issue resolved after your pull
+request is merged (please don't forget these steps):
+
+1. Click "Resolve Issue"
+2. Click the "Fix Version/s" dropdown and mark the issue as fixed in
+ the currently unreleased version
+3. Paste "Fixed in commit xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
+ into the comment field (replacing the xxx's with the correct hash)
+4. Press the "Resolve" button
+
+That's it. You can expect someone else to test the change and close
+the JIRA issue so you don't need to, and should not, close the issue
+yourself.
+
+### Development Environment
+
+You will find build instructions in [BUILD.md](BUILD.md) and
+[README.md](README.md). You will find more instructions how to set up
+your development environment in the Confluence pages [Java
+Setup](https://cwiki.apache.org/confluence/display/DAFFODIL/Java+Setup+and+Notes)
+and [IDEA
+Setup](https://cwiki.apache.org/confluence/display/DAFFODIL/IntelliJ+IDEA+Setup).
+
+### Directory Organization
+
+Daffodil is composed of one top-level project and 20
+subdirectory-level modules (some of these modules contain only
+generators, macros, tests and tutorials). Daffodil's source code
+repository is organized as follows:
+
+ daffodil/
+ ├── .asf.yaml - Enables/disables parts of GitHub's UI
+ ├── .gitattributes - Excludes KEY from source releases
+ ├── .gitignore - Ignores auto-generated files from source control
+ ├── .sbtopts - Tells sbt to use 4GB of memory
+ ├── BUILD.md - Lists Daffodil's build requirements
+ ├── KEYS - Lists public keys of Daffodil developers
+ ├── LICENSE - Licenses Daffodil under the Apache License
+ ├── NOTICE - Lists notices affecting LICENSE
+ ├── README.md - Explains how to get started
+ ├── build.sbt - Tells sbt how to build Daffodil
+ ├── containers/
+ │ └── release-candidate - Defines Docker image used to build release
+ ├── daffodil-cli/
+ │ ├── README.md - Explains how to start Daffodil
+ │ ├── bin.LICENSE - Contains licenses of Daffodil and subcomponents
+ │ ├── bin.NOTICE - Contains notices of Daffodil and subcomponents
+ │ ├── build.sbt - Tells sbt how to build CLI and installers
+ │ └── src/ - Contains CLI source code, tests, scripts, etc.
+ ├── daffodil-core/
+ │ └── src/ - Contains Daffodil's core
+ ├── daffodil-io/
+ │ └── src/ - Contains Daffodil's input/output/charsets
+ ├── daffodil-japi/
+ │ ├── build.sbt - Tells sbt how to generate javadocs
+ │ └── src/ - Contains Daffodil's Java API
+ ├── daffodil-lib/
+ │ └── src/ - Contains Daffodil's utilities
+ ├── daffodil-macro-lib/
+ │ └── src/ - Defines Daffodil's Scala 2.x macros
+ ├── daffodil-propgen/
+ │ └── src/ - Contains generators to generate more source code
+ ├── daffodil-runtime1/
+ │ └── src/ - Contains Daffodil's Scala parser
+ ├── daffodil-runtime1-layers/
+ │ └── src/ - Contains Daffodil's Scala layer transformers
+ ├── daffodil-runtime1-unparser/
+ │ └── src/ - Contains Daffodil's Scala unparser
+ ├── daffodil-runtime2/
+ │ └── src/ - Contains Daffodil's C code generator
+ ├── daffodil-sapi/
+ │ └── src/ - Contains Daffodil's Scala API
+ ├── daffodil-schematron/
+ │ ├── README.md - Explains how to validate with Schematron rules
+ │ └── src/ - Contains Daffodil's Schematron validator
+ ├── daffodil-tdml-lib/
+ │ └── src/ - Contains Daffodil's TDML runner
+ ├── daffodil-tdml-processor/
+ │ └── src/ - Contains Daffodil's Scala & C TDML implementations
+ ├── daffodil-test/
+ │ └── src/ - Contains Daffodil's TDML tests
+ ├── daffodil-test-ibm1/
+ │ └── src/ - Contains more TDML tests from IBM & Tresys
+ ├── daffodil-udf/
+ │ ├── README.md - Explains how to implement user defined functions
+ │ └── src/ - Contains Daffodil's UDF interfaces
+ ├── project/
+ │ ├── Dependencies.scala - Defines third party dependencies
+ │ ├── Rat.scala - Lists files to be ignored by license audit tool
+ │ ├── build.properties - Defines which sbt version to use
+ │ └── plugins.sbt - Adds third party plugins to sbt
+ ├── test-stdLayout/
+ │ └── src/ - Contains TDML tests laid out in different ways
+ └── tutorials/
+ └── src/ - Contains Daffodil's TDML test tutorials
+
+When compiled into a command line interface, Daffodil is composed of a
+script, a log4j2 configuration file, 14 Daffodil jars, and 37
+third-party dependency jars. These numbers are accurate for Apache
+Daffodil release 3.4.0, but may be different for previous or
+later releases. Daffodil's command line interface is released in
+binary form as follows:
+
+ apache-daffodil-3.4.0-bin/
+ ├── LICENSE - Contains licenses of Daffodil and subcomponents
+ ├── NOTICE - Contains notices of Daffodil and subcomponents
+ ├── README.md - Explains how to start Daffodil
+ ├── bin/
+ │ ├── daffodil - Starts Daffodil on Linux
+ │ └── daffodil.bat - Starts Daffodil on Windows
+ ├── conf/
+ │ └── log4j2.xml - Configures log4j logger output
+ └── lib/ - Contains Daffodil jars and third party jars
+
+When an application uses Daffodil's Java or Scala API (not counting
+the jars used only by Daffodil's command line interface), it will need
+to put 8 Daffodil jars and 18 third-party dependency jars on its
+classpath. These numbers are accurate for Apache Daffodil release
+3.4.0, but may be slightly different for previous or later releases.
+The best way for a Java or Scala application to use Daffodil is to get
+Daffodil's Java API from Maven or get Daffodil's Scala API from SBT:
+
+```xml
+<dependency>
+ <groupId>org.apache.daffodil</groupId>
+ <artifactId>daffodil-japi_2.12</artifactId>
+ <version>3.4.0</version>
+</dependency>
+```
+
+```sbt
+libraryDependencies += "org.apache.daffodil" %% "daffodil-sapi" % "3.4.0"
+```
+
+When you use Daffodil's C code generator to generate C code from a
+DFDL schema, the newly created C code directory is organized as
+follows:
+
+ c/
+ ├── Makefile - Contains build and test targets
+ ├── libcli/
+ │ ├── cli_errors.c - Implements CLI error messages
+ │ ├── daffodil_getopt.c - Implements CLI options
+ │ ├── daffodil_main.c - Starts the CLI
+ │ ├── stack.c - Pushes XML nodes on a stack
+ │ ├── xml_reader.c - Reads an infoset from a XML file
+ │ └── xml_writer.c - Writes an infoset to a XML file
+ ├── libruntime/
+ │ ├── errors.c - Implements error messages
+ │ ├── generated_code.c - Implements functions to create/use infoset
+ │ ├── infoset.c - Implements functions to walk infoset
+ │ ├── parsers.c - Implements functions to read binary data
+ │ └── unparsers.c - Implements functions to write binary data
+ └── tests/
+ ├── bits.c - Tests integers with different lengths
+ └── extras.c - Avoids an undefined reference when linking
+
+You can use the Makefile to build a Daffodil executable (c/daffodil)
+and run c/daffodil with similar parse and unparse options as
+Daffodil's Scala command line interface. If you want to embed the
+generated C code into your own C application, you can call only
+libruntime to read and write binary data into C structs in memory or
+call both libruntime and libcli (replacing daffodil_main.c with your
+application's main.c) to read and write an XML infoset as well as
+binary data into C structs.
+
+### Documentation
+
+Unfortunately, it is not possible to search or edit all of Daffodil's
+documentation in one single place. Daffodil documentation is split
+between the
+[Confluence](https://cwiki.apache.org/confluence/display/DAFFODIL/)
+wiki, the [Daffodil](https://github.com/apache/daffodil) repository,
+and the [Daffodil Website](https://github.com/apache/daffodil-site)
+repository. Confluence has a page tree on the left side so you can
+find all the Confluence pages by expanding the page tree. The first
+link below is the only link you need to find or search all the pages
+in the wiki. However, there are a lot of pages in the wiki so here
+are some of the most important pages worth reading as well (sorted in
+alphabetical order).
+
+- [Apache Daffodil Wiki](https://cwiki.apache.org/confluence/display/DAFFODIL/)
+- [Code Contributor Workflow](https://cwiki.apache.org/confluence/display/DAFFODIL/Code+Contributor+Workflow)
+- [Code Style Guidelines](https://cwiki.apache.org/confluence/display/DAFFODIL/Code+Style+Guidelines)
+- [Coding for Performance](https://cwiki.apache.org/confluence/display/DAFFODIL/Coding+for+Performance)
+- [DFDL Schema Object Model](https://cwiki.apache.org/confluence/display/DAFFODIL/DFDL+Schema+Object+Model+%28DSOM%29+with+UML)
+- [Daffodil Maturity Model Assessment](https://cwiki.apache.org/confluence/display/DAFFODIL/Apache+Daffodil+Maturity+Model+Assessment)
+- [Daffodil Runtime 2](https://cwiki.apache.org/confluence/display/DAFFODIL/WIP%3A+Daffodil+Runtime+2)
+- [IntelliJ IDEA Setup](https://cwiki.apache.org/confluence/display/DAFFODIL/IntelliJ+IDEA+Setup)
+- [Object-Oriented Lazy Attribute Grammars](https://cwiki.apache.org/confluence/display/DAFFODIL/OOLAG+-+Object-Oriented+Lazy+Attribute+Grammars)
+- [Roadmap for Upcoming Releases](https://cwiki.apache.org/confluence/display/DAFFODIL/Roadmap+for+Upcoming+Releases)
+- [Scala References and Patterns](https://cwiki.apache.org/confluence/display/DAFFODIL/Scala+References+and+Patterns)
+
+Some of the Daffodil website pages don't seem to have any links
+pointing to them from other website pages. So that you can read all
+of the website pages in your browser more easily, here are direct
+links to every website page (sorted in alphabetical order).
+
+- [A TDML Tutorial](https://daffodil.apache.org/tutorials/tdmlTutorial.tdml.xml)
+- [About AsciiDoc for Documentation](https://daffodil.apache.org/dev/aboutAsciiDoc/)
+- [About TDML Tutorials](https://daffodil.apache.org/assets/tutorials/aboutTDMLTutorials/)
+- [Bug Report TDML Template](https://daffodil.apache.org/tutorials/bugReportTemplate.tdml.xml)
+- [Command Line Interface](https://daffodil.apache.org/cli/)
+- [Community](https://daffodil.apache.org/community/)
+- [Configuration](https://daffodil.apache.org/configuration/)
+- [DFDL Extensions](https://daffodil.apache.org/dfdl-extensions/)
+- [Daffodil Java API](https://daffodil.apache.org/docs/latest/javadoc/)
+- [Daffodil Scala API](https://daffodil.apache.org/docs/latest/scaladoc/)
+- [Daffodil and the DFDL Infoset](https://daffodil.apache.org/infoset/)
+- [Data Format Description Language v1.0 Specification](https://daffodil.apache.org/docs/dfdl/)
+- [Eclipse Configuration](https://daffodil.apache.org/eclipse-configuration/)
+- [Examples](https://daffodil.apache.org/examples/)
+- [Frequently Asked Questions](https://daffodil.apache.org/faq/)
+- [Getting Started](https://daffodil.apache.org/getting-started/)
+- [Hidden Groups](https://daffodil.apache.org/dev/design-notes/hidden-groups/)
+- [Home](https://daffodil.apache.org/)
+- [Infoset Inputters and Runtime 1 Streaming Unparser & NextElementResolver](https://daffodil.apache.org/dev/design-notes/infoset-inputter-streaming-unparser/)
+- [Interactive Debugger](https://daffodil.apache.org/debugger/)
+- [Namespace Binding Minimization](https://daffodil.apache.org/dev/design-notes/namespace-binding-minimization/)
+- [People](https://daffodil.apache.org/people/)
+- [Releases](https://daffodil.apache.org/releases/)
+- [Runtime2 ToDos](https://daffodil.apache.org/dev/design-notes/runtime2-todos/)
+- [Standard DFDL Schema Project Layout](https://daffodil.apache.org/dfdl-layout/)
+- [Term Sharing in the Schema Compiler](https://daffodil.apache.org/dev/design-notes/term-sharing-in-schema-compiler/)
+- [Test Data Markup Language](https://daffodil.apache.org/tdml/)
+- [Understanding dfdl:bitOrder](https://daffodil.apache.org/tutorials/bitorder.tutorial.tdml.xml)
+- [Unsupported Features and Errata](https://daffodil.apache.org/unsupported/)
+- [VS Code Extension](https://daffodil.apache.org/vscode/)
+- [design notes folder](https://daffodil.apache.org/dev/)
+
+### Tests
+
+The majority of Daffodil tests are written inside TDML files with
+corresponding Scala unit tests which allow your SBT or IDE to run
+these tests within the TDML files. Daffodil also has some pure Scala
+unit tests for testing smaller parts of Daffodil at a lower level.
+Both kinds of tests are important, although new Daffodil tests tend to
+be written in TDML and focus on testing issues with Daffodil's
+processing of DFDL schemas and parsing data or unparsing data.
+
+TDML tests are helpful for Daffodil developers because they allow a
+developer to write a TDML file with a schema, a test case, and test
+data which isolates a complex data format issue so the developer can
+work on that issue more easily. The developer is free to decide
+whether to define the schema and test data inline in the TDML file or
+leave the schema and test data in separate files which the TDML file
+reads.
+
Review Comment:
Yes, I've added commands showing how to run the tests.
##########
DEVELOP.md:
##########
@@ -0,0 +1,485 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Daffodil Developer Guide
+
+This guide is for a software engineer who wants to develop Daffodil
+code. This guide will help you become familiar with DFDL schemas,
+DFDL processors, and Daffodil development including its code style,
+code workflow, development environment, directory organization,
+documentation, and tests.
+
+## DFDL
+
+The [Data Format Description
+Language](https://en.wikipedia.org/wiki/Data_Format_Description_Language)
+(DFDL) is a language used to describe almost all data formats both
+logically and physically. DFDL is not a data format or a procedural
+language; rather, it is a data modeling language based on a subset of
+XML Schema annotated with DFDL properties describing the
+representation and layout of each element of the schema inside a
+native text or binary data format. A DFDL schema allows data to be
+converted between its native data format (physical representation,
+also called a text or binary file) and a DFDL information set (logical
+representation, also called an infoset) such as an XML document, EXI
+document, JSON document, SAX callbacks, or several document object
+model APIs in memory (JDOM, Scala Node, W3C DOM). When you have a
+DFDL schema for a native data format, you can pick whichever infoset
+type is easiest for you or your application to use and tell a DFDL
+processor to read ("parse") a text or binary file from its native data
+format to that infoset type. You or your application can do whatever
+you need to do with the infoset and then you can use the same DFDL
+schema and DFDL processor to write ("unparse") the infoset back to its
+native text or binary file format again, completing a round trip from
+native data to infoset to native data again.
+
+Using DFDL avoids inventing a completely new data modeling language,
+avoids writing any parsing and serialization code (with all the bugs
+that normally arise from implementing such code procedurally), and
+makes it much easier to convert any native data format to an infoset,
+operate on the infoset, and convert an infoset back to its native data
+format again.
+
+To learn more about DFDL, you can watch two short
+[videos](https://community.ibm.com/community/user/integration/viewdocument/get-started-with-the-data-format-de)
+put together by Steve Hanson, co-chair of the DFDL working group, read
+a
+[slideshow](https://www.slideshare.net/mbeckerle/dfdl-and-apache-daffodil-overview-from-owl-cyber-defense)
+written by Mike Beckerle, co-chair of the DFDL working group, or go
+through some [tutorials](http://www.xfront.com/DFDL/) written by Roger
+Costello, chair of the DFTVL working group (Data Format Transformation
+and Validation Language, a future not-yet-defined language to specify
+policies for cross-domain system devices).
+
+## Apache Daffodil, IBM DFDL, and ESA DFDL4S
+
+The standards organization in which DFDL started, the Open Grid Forum,
+required 2 implementations in order to move forward with the
+standardization process. This means that there are two leading DFDL
+processors, a commercial implementation called IBM DFDL bundled into
+IBM's [Integration
+Bus](https://www.ibm.com/docs/en/integration-bus/10.0?topic=model-data-format-description-language-dfdl)
+products and an open source implementation called [Apache
+Daffodil](https://daffodil.apache.org/) hosted by the Apache Software
+Foundation. The European Space Agency also has created a proprietary
+implementation called [ESA
+DFDL4S](https://eop-cfi.esa.int/index.php/applications/dfdl4s), which
+can be used only with their satellite communication formats and is
+provided only in the form of binary libraries, not source code.
+
+Among these three DFDL processors, Apache Daffodil is considered the
+most modern and thorough implementation of the [Data Format
+Description Language v1.0
+Specification](https://daffodil.apache.org/docs/dfdl/). Even so,
+Apache Daffodil lists some [unsupported
+features](https://daffodil.apache.org/unsupported/) of the DFDL
+specification. IBM DFDL lists some more [unsupported
+features](https://www.ibm.com/docs/en/integration-bus/10.0?topic=dfdl-unsupported-features)
+and also lists some [implementation-specific
+limits](https://www.ibm.com/docs/en/integration-bus/10.0?topic=dfdl-implementation-specific-limits).
+These limitations will not prevent you from writing DFDL schemas for
+almost all data formats, but they will reveal which parts of the DFDL
+specification are rarely used.
+
+## Daffodil Development
+
+The Apache Software Foundation hosts the Apache Daffodil project on
+the following ASF infrastructure:
+
+- Daffodil's [issue
+ tracker](https://issues.apache.org/jira/projects/DAFFODIL/) is
+ hosted on JIRA
+- Daffodil's
+ [users](https://lists.apache.org/list.html?users@daffodil.apache.org),
+ [dev](https://lists.apache.org/list.html?dev@daffodil.apache.org),
+ and
+ [commits](https://lists.apache.org/list.html?commits@daffodil.apache.org)
+ mailing lists are hosted on Apache Pony Mail
+- Daffodil's [source code](https://github.com/apache/daffodil) is
+ hosted on GitHub
+- Daffodil's
+ [wiki](https://cwiki.apache.org/confluence/display/DAFFODIL/) is
+ hosted on Confluence
+- Daffodil's [website](https://daffodil.apache.org/) is hosted on
+ Apache with static content generated by Jekyll using jekyll-asciidoc
+ and asciidoctor-diagram plugins
+
+A good Daffodil developer has a GitHub account and two copies of
+Daffodil's source code repository, a reading copy directly cloned from
+the Apache repository and a working copy cloned from the developer's
+own fork of the Apache repository. A good Daffodil developer also
+gets an account on the Apache Software Foundation's JIRA and
+Confluence servers and subscribes to all three Daffodil mailing lists.
+
+You can clone your reading copy directly from the Apache repository,
+but you should never make any changes in it. The only commands you
+should ever run in your reading copy are `git pull`, `git log
+ORIG_HEAD..HEAD`, and `git diff ORIG_HEAD..HEAD` in order to see what
+changes have been made by other developers since your last pull from
+the Apache repository. Your reading copy should remain an exact copy
+of the Apache source code repository at all times which you can use
+for reading or occasionally running `diff` commands between your
+reading copy and working copy. To be safe, rename your reading copy
+from `daffodil` to `daffodil-asf` and never edit files or run `sbt` in
+it so there will be nothing to push to the Apache repository even if
+you accidentally run `git push` from the wrong source tree.
+
+You can set up your working copy by forking the Apache repository to
+your own GitHub account and then cloning your working copy from your
+fork. Cloning your working copy from your fork instead of the Apache
+repository ensures that any changes you commit in your working copy
+will be pushed safely to only your own fork, not the Apache
+repository.
+
+### Code Style
+
+Daffodil mandates standard Scala formatting and its code generally
+keeps close to that formatting consistently. No one has run an
+automatic Scala formatter on the codebase yet (there has not been much
+need since the code is already formatted pretty well) but
+[DAFFODIL-2133](https://issues.apache.org/jira/browse/DAFFODIL-2133)
+is waiting for someone willing to do that work.
+
+Daffodil mandates that at least 80% of new code and modified code
Review Comment:
Ah, interesting to know. I'll leave the text as it is, though.
##########
DEVELOP.md:
##########
@@ -0,0 +1,485 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Daffodil Developer Guide
+
+This guide is for a software engineer who wants to develop Daffodil
+code. This guide will help you become familiar with DFDL schemas,
+DFDL processors, and Daffodil development including its code style,
+code workflow, development environment, directory organization,
+documentation, and tests.
+
+## DFDL
+
+The [Data Format Description
+Language](https://en.wikipedia.org/wiki/Data_Format_Description_Language)
+(DFDL) is a language used to describe almost all data formats both
+logically and physically. DFDL is not a data format or a procedural
+language; rather, it is a data modeling language based on a subset of
+XML Schema annotated with DFDL properties describing the
+representation and layout of each element of the schema inside a
+native text or binary data format. A DFDL schema allows data to be
+converted between its native data format (physical representation,
+also called a text or binary file) and a DFDL information set (logical
+representation, also called an infoset) such as an XML document, EXI
+document, JSON document, SAX callbacks, or several document object
+model APIs in memory (JDOM, Scala Node, W3C DOM). When you have a
+DFDL schema for a native data format, you can pick whichever infoset
+type is easiest for you or your application to use and tell a DFDL
+processor to read ("parse") a text or binary file from its native data
+format to that infoset type. You or your application can do whatever
+you need to do with the infoset and then you can use the same DFDL
+schema and DFDL processor to write ("unparse") the infoset back to its
+native text or binary file format again, completing a round trip from
+native data to infoset to native data again.
+
+Using DFDL avoids inventing a completely new data modeling language,
+avoids writing any parsing and serialization code (with all the bugs
+that normally arise from implementing such code procedurally), and
+makes it much easier to convert any native data format to an infoset,
+operate on the infoset, and convert an infoset back to its native data
+format again.
+
+To learn more about DFDL, you can watch two short
+[videos](https://community.ibm.com/community/user/integration/viewdocument/get-started-with-the-data-format-de)
+put together by Steve Hanson, co-chair of the DFDL working group, read
+a
+[slideshow](https://www.slideshare.net/mbeckerle/dfdl-and-apache-daffodil-overview-from-owl-cyber-defense)
+written by Mike Beckerle, co-chair of the DFDL working group, or go
+through some [tutorials](http://www.xfront.com/DFDL/) written by Roger
+Costello, chair of the DFTVL working group (Data Format Transformation
+and Validation Language, a future not-yet-defined language to specify
+policies for cross-domain system devices).
+
+## Apache Daffodil, IBM DFDL, and ESA DFDL4S
+
+The standards organization in which DFDL started, the Open Grid Forum,
+required 2 implementations in order to move forward with the
+standardization process. This means that there are two leading DFDL
+processors, a commercial implementation called IBM DFDL bundled into
+IBM's [Integration
+Bus](https://www.ibm.com/docs/en/integration-bus/10.0?topic=model-data-format-description-language-dfdl)
+products and an open source implementation called [Apache
+Daffodil](https://daffodil.apache.org/) hosted by the Apache Software
+Foundation. The European Space Agency also has created a proprietary
+implementation called [ESA
+DFDL4S](https://eop-cfi.esa.int/index.php/applications/dfdl4s), which
+can be used only with their satellite communication formats and is
+provided only in the form of binary libraries, not source code.
+
+Among these three DFDL processors, Apache Daffodil is considered the
+most modern and thorough implementation of the [Data Format
+Description Language v1.0
+Specification](https://daffodil.apache.org/docs/dfdl/). Even so,
+Apache Daffodil lists some [unsupported
+features](https://daffodil.apache.org/unsupported/) of the DFDL
+specification. IBM DFDL lists some more [unsupported
+features](https://www.ibm.com/docs/en/integration-bus/10.0?topic=dfdl-unsupported-features)
+and also lists some [implementation-specific
+limits](https://www.ibm.com/docs/en/integration-bus/10.0?topic=dfdl-implementation-specific-limits).
+These limitations will not prevent you from writing DFDL schemas for
+almost all data formats, but they will reveal which parts of the DFDL
+specification are rarely used.
+
+## Daffodil Development
+
+The Apache Software Foundation hosts the Apache Daffodil project on
+the following ASF infrastructure:
+
+- Daffodil's [issue
+ tracker](https://issues.apache.org/jira/projects/DAFFODIL/) is
+ hosted on JIRA
+- Daffodil's
+ [users](https://lists.apache.org/list.html?users@daffodil.apache.org),
+ [dev](https://lists.apache.org/list.html?dev@daffodil.apache.org),
+ and
+ [commits](https://lists.apache.org/list.html?commits@daffodil.apache.org)
+ mailing lists are hosted on Apache Pony Mail
+- Daffodil's [source code](https://github.com/apache/daffodil) is
+ hosted on GitHub
+- Daffodil's
+ [wiki](https://cwiki.apache.org/confluence/display/DAFFODIL/) is
+ hosted on Confluence
+- Daffodil's [website](https://daffodil.apache.org/) is hosted on
+ Apache with static content generated by Jekyll using jekyll-asciidoc
+ and asciidoctor-diagram plugins
+
+A good Daffodil developer has a GitHub account and two copies of
+Daffodil's source code repository, a reading copy directly cloned from
+the Apache repository and a working copy cloned from the developer's
+own fork of the Apache repository. A good Daffodil developer also
+gets an account on the Apache Software Foundation's JIRA and
+Confluence servers and subscribes to all three Daffodil mailing lists.
+
+You can clone your reading copy directly from the Apache repository,
+but you should never make any changes in it. The only commands you
+should ever run in your reading copy are `git pull`, `git log
+ORIG_HEAD..HEAD`, and `git diff ORIG_HEAD..HEAD` in order to see what
+changes have been made by other developers since your last pull from
+the Apache repository. Your reading copy should remain an exact copy
+of the Apache source code repository at all times which you can use
+for reading or occasionally running `diff` commands between your
+reading copy and working copy. To be safe, rename your reading copy
+from `daffodil` to `daffodil-asf` and never edit files or run `sbt` in
+it so there will be nothing to push to the Apache repository even if
+you accidentally run `git push` from the wrong source tree.
+
+You can set up your working copy by forking the Apache repository to
+your own GitHub account and then cloning your working copy from your
+fork. Cloning your working copy from your fork instead of the Apache
+repository ensures that any changes you commit in your working copy
+will be pushed safely to only your own fork, not the Apache
+repository.
+
+### Code Style
+
+Daffodil mandates standard Scala formatting and its code generally
+keeps close to that formatting consistently. No one has run an
+automatic Scala formatter on the codebase yet (there has not been much
+need since the code is already formatted pretty well) but
+[DAFFODIL-2133](https://issues.apache.org/jira/browse/DAFFODIL-2133)
+is waiting for someone willing to do that work.
+
+Daffodil mandates that at least 80% of new code and modified code
+should be covered by unit tests or TDML tests. Daffodil naming
+conventions and more are covered in the Confluence pages [Code Style
+Guidelines](https://cwiki.apache.org/confluence/display/DAFFODIL/Code+Style+Guidelines)
+and [Coding for
+Performance](https://cwiki.apache.org/confluence/display/DAFFODIL/Coding+for+Performance).
+
+### Code Workflow
+
+It is very important that you fork the Apache repository to your own
+GitHub account before you start making changes to Apache Daffodil
+source code. You should always make changes only to your own fork of
+the upstream Apache repository, that is, clone your working copy from
+your fork and push commits back to your own fork only. The same
+workflow applies to Apache Daffodil's other two source code
+repositories, that is, always fork these repositories and clone from
+your forks before you make any changes to the [Daffodil VS Code
+Extension](https://github.com/apache/daffodil-vscode) or [Daffodil
+Website](https://github.com/apache/daffodil-site). When you are ready
+to contribute code from any of your forks to any of these upstream
+Apache repositories, you must create a pull request from your fork,
+allow the Apache Daffodil developers to review your contribution, make
+any suggested changes, and wait for their approval before you or
+someone else can merge your code into one of Apache's source code
+repositories.
+
+The Daffodil developers want each pull request to add only one commit
+to the Apache repository to keep its git history as readable and
+bisectable as possible. Your pull request should fix a JIRA issue in
+its entirety but it should not fix 2 or more unrelated JIRA issues in
+the same commit either. Your pull request should start with only one
+commit in it before you ask the developers to review it and your pull
+request should end with only one commit in it after you receive enough
+reviewer approvals (two +1's for Daffodil and Daffodil VS Code
+Extension, one +1 for Daffodil Website). If you have made multiple
+commits to your pull request due to reviewer suggestions (which is the
+usual case), you must not merge these multiple commits into the Apache
+repository. You must rebase and squash your pull request back to one
+commit as shown in the [Code Contributor
+Workflow](https://cwiki.apache.org/confluence/display/DAFFODIL/Code+Contributor+Workflow),
+which you should read and follow carefully to ensure a successful code
+contribution to Apache Daffodil.
+
+Before you change Daffodil code, you will need to know how to use JIRA
+as well as GitHub. You will assign a JIRA issue to yourself before
+you start working on it to avoid redundant effort in case another
+developer decides to work on the same issue. Then you will mark the
+issue as fixed after your pull request is merged. Here are the steps
+you need to perform in JIRA to mark the issue resolved after your pull
+request is merged (please don't forget these steps):
+
+1. Click "Resolve Issue"
+2. Click the "Fix Version/s" dropdown and mark the issue as fixed in
+ the currently unreleased version
+3. Paste "Fixed in commit xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
+ into the comment field (replacing the xxx's with the correct hash)
+4. Press the "Resolve" button
+
+That's it. You can expect someone else to test the change and close
+the JIRA issue so you don't need to, and should not, close the issue
+yourself.
+
+### Development Environment
+
+You will find build instructions in [BUILD.md](BUILD.md) and
+[README.md](README.md). You will find more instructions how to set up
+your development environment in the Confluence pages [Java
+Setup](https://cwiki.apache.org/confluence/display/DAFFODIL/Java+Setup+and+Notes)
+and [IDEA
+Setup](https://cwiki.apache.org/confluence/display/DAFFODIL/IntelliJ+IDEA+Setup).
+
+### Directory Organization
+
+Daffodil is composed of one top-level project and 20
Review Comment:
Yes, I've made the replacement.
##########
DEVELOP.md:
##########
@@ -0,0 +1,485 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Daffodil Developer Guide
+
+This guide is for a software engineer who wants to develop Daffodil
+code. This guide will help you become familiar with DFDL schemas,
+DFDL processors, and Daffodil development including its code style,
+code workflow, development environment, directory organization,
+documentation, and tests.
+
+## DFDL
+
+The [Data Format Description
+Language](https://en.wikipedia.org/wiki/Data_Format_Description_Language)
+(DFDL) is a language used to describe almost all data formats both
+logically and physically. DFDL is not a data format or a procedural
+language; rather, it is a data modeling language based on a subset of
+XML Schema annotated with DFDL properties describing the
+representation and layout of each element of the schema inside a
+native text or binary data format. A DFDL schema allows data to be
+converted between its native data format (physical representation,
+also called a text or binary file) and a DFDL information set (logical
+representation, also called an infoset) such as an XML document, EXI
+document, JSON document, SAX callbacks, or several document object
+model APIs in memory (JDOM, Scala Node, W3C DOM). When you have a
+DFDL schema for a native data format, you can pick whichever infoset
+type is easiest for you or your application to use and tell a DFDL
+processor to read ("parse") a text or binary file from its native data
+format to that infoset type. You or your application can do whatever
+you need to do with the infoset and then you can use the same DFDL
+schema and DFDL processor to write ("unparse") the infoset back to its
+native text or binary file format again, completing a round trip from
+native data to infoset to native data again.
+
+Using DFDL avoids inventing a completely new data modeling language,
+avoids writing any parsing and serialization code (with all the bugs
+that normally arise from implementing such code procedurally), and
+makes it much easier to convert any native data format to an infoset,
+operate on the infoset, and convert an infoset back to its native data
+format again.
+
+To learn more about DFDL, you can watch two short
+[videos](https://community.ibm.com/community/user/integration/viewdocument/get-started-with-the-data-format-de)
+put together by Steve Hanson, co-chair of the DFDL working group, read
+a
+[slideshow](https://www.slideshare.net/mbeckerle/dfdl-and-apache-daffodil-overview-from-owl-cyber-defense)
+written by Mike Beckerle, co-chair of the DFDL working group, or go
+through some [tutorials](http://www.xfront.com/DFDL/) written by Roger
+Costello, chair of the DFTVL working group (Data Format Transformation
+and Validation Language, a future not-yet-defined language to specify
+policies for cross-domain system devices).
+
+## Apache Daffodil, IBM DFDL, and ESA DFDL4S
+
+The standards organization in which DFDL started, the Open Grid Forum,
+required 2 implementations in order to move forward with the
+standardization process. This means that there are two leading DFDL
+processors, a commercial implementation called IBM DFDL bundled into
+IBM's [Integration
+Bus](https://www.ibm.com/docs/en/integration-bus/10.0?topic=model-data-format-description-language-dfdl)
+products and an open source implementation called [Apache
+Daffodil](https://daffodil.apache.org/) hosted by the Apache Software
+Foundation. The European Space Agency also has created a proprietary
+implementation called [ESA
+DFDL4S](https://eop-cfi.esa.int/index.php/applications/dfdl4s), which
+can be used only with their satellite communication formats and is
+provided only in the form of binary libraries, not source code.
+
+Among these three DFDL processors, Apache Daffodil is considered the
+most modern and thorough implementation of the [Data Format
+Description Language v1.0
+Specification](https://daffodil.apache.org/docs/dfdl/). Even so,
+Apache Daffodil lists some [unsupported
+features](https://daffodil.apache.org/unsupported/) of the DFDL
+specification. IBM DFDL lists some more [unsupported
+features](https://www.ibm.com/docs/en/integration-bus/10.0?topic=dfdl-unsupported-features)
+and also lists some [implementation-specific
+limits](https://www.ibm.com/docs/en/integration-bus/10.0?topic=dfdl-implementation-specific-limits).
+These limitations will not prevent you from writing DFDL schemas for
+almost all data formats, but they will reveal which parts of the DFDL
+specification are rarely used.
+
+## Daffodil Development
+
+The Apache Software Foundation hosts the Apache Daffodil project on
+the following ASF infrastructure:
+
+- Daffodil's [issue
+ tracker](https://issues.apache.org/jira/projects/DAFFODIL/) is
+ hosted on JIRA
+- Daffodil's
+ [users](https://lists.apache.org/list.html?users@daffodil.apache.org),
+ [dev](https://lists.apache.org/list.html?dev@daffodil.apache.org),
+ and
+ [commits](https://lists.apache.org/list.html?commits@daffodil.apache.org)
+ mailing lists are hosted on Apache Pony Mail
+- Daffodil's [source code](https://github.com/apache/daffodil) is
+ hosted on GitHub
+- Daffodil's
+ [wiki](https://cwiki.apache.org/confluence/display/DAFFODIL/) is
+ hosted on Confluence
+- Daffodil's [website](https://daffodil.apache.org/) is hosted on
+ Apache with static content generated by Jekyll using jekyll-asciidoc
+ and asciidoctor-diagram plugins
+
+A good Daffodil developer has a GitHub account and two copies of
+Daffodil's source code repository, a reading copy directly cloned from
+the Apache repository and a working copy cloned from the developer's
+own fork of the Apache repository. A good Daffodil developer also
+gets an account on the Apache Software Foundation's JIRA and
+Confluence servers and subscribes to all three Daffodil mailing lists.
+
+You can clone your reading copy directly from the Apache repository,
+but you should never make any changes in it. The only commands you
+should ever run in your reading copy are `git pull`, `git log
+ORIG_HEAD..HEAD`, and `git diff ORIG_HEAD..HEAD` in order to see what
+changes have been made by other developers since your last pull from
+the Apache repository. Your reading copy should remain an exact copy
+of the Apache source code repository at all times which you can use
+for reading or occasionally running `diff` commands between your
+reading copy and working copy. To be safe, rename your reading copy
+from `daffodil` to `daffodil-asf` and never edit files or run `sbt` in
+it so there will be nothing to push to the Apache repository even if
+you accidentally run `git push` from the wrong source tree.
+
+You can set up your working copy by forking the Apache repository to
+your own GitHub account and then cloning your working copy from your
+fork. Cloning your working copy from your fork instead of the Apache
+repository ensures that any changes you commit in your working copy
+will be pushed safely to only your own fork, not the Apache
+repository.
+
+### Code Style
+
+Daffodil mandates standard Scala formatting and its code generally
+keeps close to that formatting consistently. No one has run an
+automatic Scala formatter on the codebase yet (there has not been much
+need since the code is already formatted pretty well) but
+[DAFFODIL-2133](https://issues.apache.org/jira/browse/DAFFODIL-2133)
+is waiting for someone willing to do that work.
+
+Daffodil mandates that at least 80% of new code and modified code
+should be covered by unit tests or TDML tests. Daffodil naming
+conventions and more are covered in the Confluence pages [Code Style
+Guidelines](https://cwiki.apache.org/confluence/display/DAFFODIL/Code+Style+Guidelines)
+and [Coding for
+Performance](https://cwiki.apache.org/confluence/display/DAFFODIL/Coding+for+Performance).
+
+### Code Workflow
+
+It is very important that you fork the Apache repository to your own
+GitHub account before you start making changes to Apache Daffodil
+source code. You should always make changes only to your own fork of
+the upstream Apache repository, that is, clone your working copy from
+your fork and push commits back to your own fork only. The same
+workflow applies to Apache Daffodil's other two source code
+repositories, that is, always fork these repositories and clone from
+your forks before you make any changes to the [Daffodil VS Code
+Extension](https://github.com/apache/daffodil-vscode) or [Daffodil
+Website](https://github.com/apache/daffodil-site). When you are ready
+to contribute code from any of your forks to any of these upstream
+Apache repositories, you must create a pull request from your fork,
+allow the Apache Daffodil developers to review your contribution, make
+any suggested changes, and wait for their approval before you or
+someone else can merge your code into one of Apache's source code
+repositories.
+
+The Daffodil developers want each pull request to add only one commit
+to the Apache repository to keep its git history as readable and
+bisectable as possible. Your pull request should fix a JIRA issue in
+its entirety but it should not fix 2 or more unrelated JIRA issues in
+the same commit either. Your pull request should start with only one
+commit in it before you ask the developers to review it and your pull
+request should end with only one commit in it after you receive enough
+reviewer approvals (two +1's for Daffodil and Daffodil VS Code
+Extension, one +1 for Daffodil Website). If you have made multiple
+commits to your pull request due to reviewer suggestions (which is the
+usual case), you must not merge these multiple commits into the Apache
+repository. You must rebase and squash your pull request back to one
+commit as shown in the [Code Contributor
+Workflow](https://cwiki.apache.org/confluence/display/DAFFODIL/Code+Contributor+Workflow),
+which you should read and follow carefully to ensure a successful code
+contribution to Apache Daffodil.
Review Comment:
I revised this section and trimmed some text in response to the suggestions about not having write permission and citing the Contributor workflow in an earlier section since I didn't want both sections to be too long. Now this section sends the reader to the Contributor workflow first and only adds some more things the reader should know too (keep the pull request to 1 issue/1 commit, and how to resolve the JIRA issue). The section should be easier to read now.
##########
DEVELOP.md:
##########
@@ -0,0 +1,485 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Daffodil Developer Guide
+
+This guide is for a software engineer who wants to develop Daffodil
+code. This guide will help you become familiar with DFDL schemas,
+DFDL processors, and Daffodil development including its code style,
+code workflow, development environment, directory organization,
+documentation, and tests.
+
+## DFDL
+
+The [Data Format Description
+Language](https://en.wikipedia.org/wiki/Data_Format_Description_Language)
+(DFDL) is a language used to describe almost all data formats both
+logically and physically. DFDL is not a data format or a procedural
+language; rather, it is a data modeling language based on a subset of
+XML Schema annotated with DFDL properties describing the
+representation and layout of each element of the schema inside a
+native text or binary data format. A DFDL schema allows data to be
+converted between its native data format (physical representation,
+also called a text or binary file) and a DFDL information set (logical
+representation, also called an infoset) such as an XML document, EXI
+document, JSON document, SAX callbacks, or several document object
+model APIs in memory (JDOM, Scala Node, W3C DOM). When you have a
+DFDL schema for a native data format, you can pick whichever infoset
+type is easiest for you or your application to use and tell a DFDL
+processor to read ("parse") a text or binary file from its native data
+format to that infoset type. You or your application can do whatever
+you need to do with the infoset and then you can use the same DFDL
+schema and DFDL processor to write ("unparse") the infoset back to its
+native text or binary file format again, completing a round trip from
+native data to infoset to native data again.
+
+Using DFDL avoids inventing a completely new data modeling language,
+avoids writing any parsing and serialization code (with all the bugs
+that normally arise from implementing such code procedurally), and
+makes it much easier to convert any native data format to an infoset,
+operate on the infoset, and convert an infoset back to its native data
+format again.
+
+To learn more about DFDL, you can watch two short
+[videos](https://community.ibm.com/community/user/integration/viewdocument/get-started-with-the-data-format-de)
+put together by Steve Hanson, co-chair of the DFDL working group, read
+a
+[slideshow](https://www.slideshare.net/mbeckerle/dfdl-and-apache-daffodil-overview-from-owl-cyber-defense)
+written by Mike Beckerle, co-chair of the DFDL working group, or go
+through some [tutorials](http://www.xfront.com/DFDL/) written by Roger
+Costello, chair of the DFTVL working group (Data Format Transformation
+and Validation Language, a future not-yet-defined language to specify
+policies for cross-domain system devices).
+
+## Apache Daffodil, IBM DFDL, and ESA DFDL4S
+
+The standards organization in which DFDL started, the Open Grid Forum,
+required 2 implementations in order to move forward with the
+standardization process. This means that there are two leading DFDL
+processors, a commercial implementation called IBM DFDL bundled into
+IBM's [Integration
+Bus](https://www.ibm.com/docs/en/integration-bus/10.0?topic=model-data-format-description-language-dfdl)
+products and an open source implementation called [Apache
+Daffodil](https://daffodil.apache.org/) hosted by the Apache Software
+Foundation. The European Space Agency also has created a proprietary
+implementation called [ESA
+DFDL4S](https://eop-cfi.esa.int/index.php/applications/dfdl4s), which
+can be used only with their satellite communication formats and is
+provided only in the form of binary libraries, not source code.
+
+Among these three DFDL processors, Apache Daffodil is considered the
+most modern and thorough implementation of the [Data Format
+Description Language v1.0
+Specification](https://daffodil.apache.org/docs/dfdl/). Even so,
+Apache Daffodil lists some [unsupported
+features](https://daffodil.apache.org/unsupported/) of the DFDL
+specification. IBM DFDL lists some more [unsupported
+features](https://www.ibm.com/docs/en/integration-bus/10.0?topic=dfdl-unsupported-features)
+and also lists some [implementation-specific
+limits](https://www.ibm.com/docs/en/integration-bus/10.0?topic=dfdl-implementation-specific-limits).
+These limitations will not prevent you from writing DFDL schemas for
+almost all data formats, but they will reveal which parts of the DFDL
+specification are rarely used.
+
+## Daffodil Development
+
+The Apache Software Foundation hosts the Apache Daffodil project on
+the following ASF infrastructure:
+
+- Daffodil's [issue
+ tracker](https://issues.apache.org/jira/projects/DAFFODIL/) is
+ hosted on JIRA
+- Daffodil's
+ [users](https://lists.apache.org/list.html?users@daffodil.apache.org),
+ [dev](https://lists.apache.org/list.html?dev@daffodil.apache.org),
+ and
+ [commits](https://lists.apache.org/list.html?commits@daffodil.apache.org)
+ mailing lists are hosted on Apache Pony Mail
+- Daffodil's [source code](https://github.com/apache/daffodil) is
+ hosted on GitHub
+- Daffodil's
+ [wiki](https://cwiki.apache.org/confluence/display/DAFFODIL/) is
+ hosted on Confluence
+- Daffodil's [website](https://daffodil.apache.org/) is hosted on
+ Apache with static content generated by Jekyll using jekyll-asciidoc
+ and asciidoctor-diagram plugins
+
+A good Daffodil developer has a GitHub account and two copies of
+Daffodil's source code repository, a reading copy directly cloned from
+the Apache repository and a working copy cloned from the developer's
+own fork of the Apache repository. A good Daffodil developer also
+gets an account on the Apache Software Foundation's JIRA and
+Confluence servers and subscribes to all three Daffodil mailing lists.
+
+You can clone your reading copy directly from the Apache repository,
+but you should never make any changes in it. The only commands you
+should ever run in your reading copy are `git pull`, `git log
+ORIG_HEAD..HEAD`, and `git diff ORIG_HEAD..HEAD` in order to see what
+changes have been made by other developers since your last pull from
+the Apache repository. Your reading copy should remain an exact copy
+of the Apache source code repository at all times which you can use
+for reading or occasionally running `diff` commands between your
+reading copy and working copy. To be safe, rename your reading copy
+from `daffodil` to `daffodil-asf` and never edit files or run `sbt` in
+it so there will be nothing to push to the Apache repository even if
+you accidentally run `git push` from the wrong source tree.
+
+You can set up your working copy by forking the Apache repository to
+your own GitHub account and then cloning your working copy from your
+fork. Cloning your working copy from your fork instead of the Apache
+repository ensures that any changes you commit in your working copy
+will be pushed safely to only your own fork, not the Apache
+repository.
+
+### Code Style
+
+Daffodil mandates standard Scala formatting and its code generally
+keeps close to that formatting consistently. No one has run an
+automatic Scala formatter on the codebase yet (there has not been much
+need since the code is already formatted pretty well) but
+[DAFFODIL-2133](https://issues.apache.org/jira/browse/DAFFODIL-2133)
+is waiting for someone willing to do that work.
+
+Daffodil mandates that at least 80% of new code and modified code
+should be covered by unit tests or TDML tests. Daffodil naming
+conventions and more are covered in the Confluence pages [Code Style
+Guidelines](https://cwiki.apache.org/confluence/display/DAFFODIL/Code+Style+Guidelines)
+and [Coding for
+Performance](https://cwiki.apache.org/confluence/display/DAFFODIL/Coding+for+Performance).
+
+### Code Workflow
+
+It is very important that you fork the Apache repository to your own
+GitHub account before you start making changes to Apache Daffodil
+source code. You should always make changes only to your own fork of
+the upstream Apache repository, that is, clone your working copy from
+your fork and push commits back to your own fork only. The same
+workflow applies to Apache Daffodil's other two source code
+repositories, that is, always fork these repositories and clone from
+your forks before you make any changes to the [Daffodil VS Code
+Extension](https://github.com/apache/daffodil-vscode) or [Daffodil
+Website](https://github.com/apache/daffodil-site). When you are ready
+to contribute code from any of your forks to any of these upstream
+Apache repositories, you must create a pull request from your fork,
+allow the Apache Daffodil developers to review your contribution, make
+any suggested changes, and wait for their approval before you or
+someone else can merge your code into one of Apache's source code
+repositories.
+
+The Daffodil developers want each pull request to add only one commit
+to the Apache repository to keep its git history as readable and
+bisectable as possible. Your pull request should fix a JIRA issue in
+its entirety but it should not fix 2 or more unrelated JIRA issues in
+the same commit either. Your pull request should start with only one
+commit in it before you ask the developers to review it and your pull
+request should end with only one commit in it after you receive enough
+reviewer approvals (two +1's for Daffodil and Daffodil VS Code
+Extension, one +1 for Daffodil Website). If you have made multiple
+commits to your pull request due to reviewer suggestions (which is the
+usual case), you must not merge these multiple commits into the Apache
+repository. You must rebase and squash your pull request back to one
+commit as shown in the [Code Contributor
+Workflow](https://cwiki.apache.org/confluence/display/DAFFODIL/Code+Contributor+Workflow),
+which you should read and follow carefully to ensure a successful code
+contribution to Apache Daffodil.
+
+Before you change Daffodil code, you will need to know how to use JIRA
+as well as GitHub. You will assign a JIRA issue to yourself before
+you start working on it to avoid redundant effort in case another
+developer decides to work on the same issue. Then you will mark the
+issue as fixed after your pull request is merged. Here are the steps
+you need to perform in JIRA to mark the issue resolved after your pull
+request is merged (please don't forget these steps):
Review Comment:
Ah, I told Peter to resolve the Add macOS build support issue himself and he had no problem doing it even though he's a non-committer. Let's keep the JIRA instructions since it makes sense for the code contributor to do the workflow steps from start to end and being able to resolve an issue yourself is a nice motivation to take up another issue.
##########
DEVELOP.md:
##########
@@ -0,0 +1,485 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Daffodil Developer Guide
+
+This guide is for a software engineer who wants to develop Daffodil
+code. This guide will help you become familiar with DFDL schemas,
+DFDL processors, and Daffodil development including its code style,
+code workflow, development environment, directory organization,
+documentation, and tests.
+
+## DFDL
+
+The [Data Format Description
+Language](https://en.wikipedia.org/wiki/Data_Format_Description_Language)
+(DFDL) is a language used to describe almost all data formats both
+logically and physically. DFDL is not a data format or a procedural
+language; rather, it is a data modeling language based on a subset of
+XML Schema annotated with DFDL properties describing the
+representation and layout of each element of the schema inside a
+native text or binary data format. A DFDL schema allows data to be
+converted between its native data format (physical representation,
+also called a text or binary file) and a DFDL information set (logical
+representation, also called an infoset) such as an XML document, EXI
+document, JSON document, SAX callbacks, or several document object
+model APIs in memory (JDOM, Scala Node, W3C DOM). When you have a
+DFDL schema for a native data format, you can pick whichever infoset
+type is easiest for you or your application to use and tell a DFDL
+processor to read ("parse") a text or binary file from its native data
+format to that infoset type. You or your application can do whatever
+you need to do with the infoset and then you can use the same DFDL
+schema and DFDL processor to write ("unparse") the infoset back to its
+native text or binary file format again, completing a round trip from
+native data to infoset to native data again.
+
+Using DFDL avoids inventing a completely new data modeling language,
+avoids writing any parsing and serialization code (with all the bugs
+that normally arise from implementing such code procedurally), and
+makes it much easier to convert any native data format to an infoset,
+operate on the infoset, and convert an infoset back to its native data
+format again.
+
+To learn more about DFDL, you can watch two short
+[videos](https://community.ibm.com/community/user/integration/viewdocument/get-started-with-the-data-format-de)
+put together by Steve Hanson, co-chair of the DFDL working group, read
+a
+[slideshow](https://www.slideshare.net/mbeckerle/dfdl-and-apache-daffodil-overview-from-owl-cyber-defense)
+written by Mike Beckerle, co-chair of the DFDL working group, or go
+through some [tutorials](http://www.xfront.com/DFDL/) written by Roger
+Costello, chair of the DFTVL working group (Data Format Transformation
+and Validation Language, a future not-yet-defined language to specify
+policies for cross-domain system devices).
+
+## Apache Daffodil, IBM DFDL, and ESA DFDL4S
+
+The standards organization in which DFDL started, the Open Grid Forum,
+required 2 implementations in order to move forward with the
+standardization process. This means that there are two leading DFDL
+processors, a commercial implementation called IBM DFDL bundled into
+IBM's [Integration
+Bus](https://www.ibm.com/docs/en/integration-bus/10.0?topic=model-data-format-description-language-dfdl)
+products and an open source implementation called [Apache
+Daffodil](https://daffodil.apache.org/) hosted by the Apache Software
+Foundation. The European Space Agency also has created a proprietary
+implementation called [ESA
+DFDL4S](https://eop-cfi.esa.int/index.php/applications/dfdl4s), which
+can be used only with their satellite communication formats and is
+provided only in the form of binary libraries, not source code.
+
+Among these three DFDL processors, Apache Daffodil is considered the
+most modern and thorough implementation of the [Data Format
+Description Language v1.0
+Specification](https://daffodil.apache.org/docs/dfdl/). Even so,
+Apache Daffodil lists some [unsupported
+features](https://daffodil.apache.org/unsupported/) of the DFDL
+specification. IBM DFDL lists some more [unsupported
+features](https://www.ibm.com/docs/en/integration-bus/10.0?topic=dfdl-unsupported-features)
+and also lists some [implementation-specific
+limits](https://www.ibm.com/docs/en/integration-bus/10.0?topic=dfdl-implementation-specific-limits).
+These limitations will not prevent you from writing DFDL schemas for
+almost all data formats, but they will reveal which parts of the DFDL
+specification are rarely used.
+
+## Daffodil Development
+
+The Apache Software Foundation hosts the Apache Daffodil project on
+the following ASF infrastructure:
+
+- Daffodil's [issue
+ tracker](https://issues.apache.org/jira/projects/DAFFODIL/) is
+ hosted on JIRA
+- Daffodil's
+ [users](https://lists.apache.org/list.html?users@daffodil.apache.org),
+ [dev](https://lists.apache.org/list.html?dev@daffodil.apache.org),
+ and
+ [commits](https://lists.apache.org/list.html?commits@daffodil.apache.org)
+ mailing lists are hosted on Apache Pony Mail
+- Daffodil's [source code](https://github.com/apache/daffodil) is
+ hosted on GitHub
+- Daffodil's
+ [wiki](https://cwiki.apache.org/confluence/display/DAFFODIL/) is
+ hosted on Confluence
+- Daffodil's [website](https://daffodil.apache.org/) is hosted on
+ Apache with static content generated by Jekyll using jekyll-asciidoc
+ and asciidoctor-diagram plugins
+
+A good Daffodil developer has a GitHub account and two copies of
+Daffodil's source code repository, a reading copy directly cloned from
+the Apache repository and a working copy cloned from the developer's
+own fork of the Apache repository. A good Daffodil developer also
+gets an account on the Apache Software Foundation's JIRA and
+Confluence servers and subscribes to all three Daffodil mailing lists.
+
+You can clone your reading copy directly from the Apache repository,
+but you should never make any changes in it. The only commands you
+should ever run in your reading copy are `git pull`, `git log
+ORIG_HEAD..HEAD`, and `git diff ORIG_HEAD..HEAD` in order to see what
+changes have been made by other developers since your last pull from
+the Apache repository. Your reading copy should remain an exact copy
+of the Apache source code repository at all times which you can use
+for reading or occasionally running `diff` commands between your
+reading copy and working copy. To be safe, rename your reading copy
+from `daffodil` to `daffodil-asf` and never edit files or run `sbt` in
+it so there will be nothing to push to the Apache repository even if
+you accidentally run `git push` from the wrong source tree.
+
+You can set up your working copy by forking the Apache repository to
+your own GitHub account and then cloning your working copy from your
+fork. Cloning your working copy from your fork instead of the Apache
+repository ensures that any changes you commit in your working copy
+will be pushed safely to only your own fork, not the Apache
+repository.
+
+### Code Style
+
+Daffodil mandates standard Scala formatting and its code generally
+keeps close to that formatting consistently. No one has run an
+automatic Scala formatter on the codebase yet (there has not been much
+need since the code is already formatted pretty well) but
+[DAFFODIL-2133](https://issues.apache.org/jira/browse/DAFFODIL-2133)
+is waiting for someone willing to do that work.
+
+Daffodil mandates that at least 80% of new code and modified code
+should be covered by unit tests or TDML tests. Daffodil naming
+conventions and more are covered in the Confluence pages [Code Style
+Guidelines](https://cwiki.apache.org/confluence/display/DAFFODIL/Code+Style+Guidelines)
+and [Coding for
+Performance](https://cwiki.apache.org/confluence/display/DAFFODIL/Coding+for+Performance).
+
+### Code Workflow
+
+It is very important that you fork the Apache repository to your own
+GitHub account before you start making changes to Apache Daffodil
+source code. You should always make changes only to your own fork of
+the upstream Apache repository, that is, clone your working copy from
+your fork and push commits back to your own fork only. The same
+workflow applies to Apache Daffodil's other two source code
+repositories, that is, always fork these repositories and clone from
+your forks before you make any changes to the [Daffodil VS Code
+Extension](https://github.com/apache/daffodil-vscode) or [Daffodil
+Website](https://github.com/apache/daffodil-site). When you are ready
+to contribute code from any of your forks to any of these upstream
+Apache repositories, you must create a pull request from your fork,
+allow the Apache Daffodil developers to review your contribution, make
+any suggested changes, and wait for their approval before you or
+someone else can merge your code into one of Apache's source code
+repositories.
+
+The Daffodil developers want each pull request to add only one commit
+to the Apache repository to keep its git history as readable and
+bisectable as possible. Your pull request should fix a JIRA issue in
+its entirety but it should not fix 2 or more unrelated JIRA issues in
+the same commit either. Your pull request should start with only one
+commit in it before you ask the developers to review it and your pull
+request should end with only one commit in it after you receive enough
+reviewer approvals (two +1's for Daffodil and Daffodil VS Code
+Extension, one +1 for Daffodil Website). If you have made multiple
+commits to your pull request due to reviewer suggestions (which is the
+usual case), you must not merge these multiple commits into the Apache
+repository. You must rebase and squash your pull request back to one
+commit as shown in the [Code Contributor
+Workflow](https://cwiki.apache.org/confluence/display/DAFFODIL/Code+Contributor+Workflow),
+which you should read and follow carefully to ensure a successful code
+contribution to Apache Daffodil.
+
+Before you change Daffodil code, you will need to know how to use JIRA
+as well as GitHub. You will assign a JIRA issue to yourself before
+you start working on it to avoid redundant effort in case another
+developer decides to work on the same issue. Then you will mark the
+issue as fixed after your pull request is merged. Here are the steps
+you need to perform in JIRA to mark the issue resolved after your pull
+request is merged (please don't forget these steps):
+
+1. Click "Resolve Issue"
+2. Click the "Fix Version/s" dropdown and mark the issue as fixed in
+ the currently unreleased version
+3. Paste "Fixed in commit xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
+ into the comment field (replacing the xxx's with the correct hash)
+4. Press the "Resolve" button
+
+That's it. You can expect someone else to test the change and close
+the JIRA issue so you don't need to, and should not, close the issue
+yourself.
+
+### Development Environment
+
+You will find build instructions in [BUILD.md](BUILD.md) and
+[README.md](README.md). You will find more instructions how to set up
+your development environment in the Confluence pages [Java
+Setup](https://cwiki.apache.org/confluence/display/DAFFODIL/Java+Setup+and+Notes)
+and [IDEA
+Setup](https://cwiki.apache.org/confluence/display/DAFFODIL/IntelliJ+IDEA+Setup).
+
+### Directory Organization
+
+Daffodil is composed of one top-level project and 20
+subdirectory-level modules (some of these modules contain only
+generators, macros, tests and tutorials). Daffodil's source code
+repository is organized as follows:
+
+ daffodil/
+ ├── .asf.yaml - Enables/disables parts of GitHub's UI
+ ├── .gitattributes - Excludes KEY from source releases
+ ├── .gitignore - Ignores auto-generated files from source control
+ ├── .sbtopts - Tells sbt to use 4GB of memory
+ ├── BUILD.md - Lists Daffodil's build requirements
+ ├── KEYS - Lists public keys of Daffodil developers
+ ├── LICENSE - Licenses Daffodil under the Apache License
+ ├── NOTICE - Lists notices affecting LICENSE
+ ├── README.md - Explains how to get started
+ ├── build.sbt - Tells sbt how to build Daffodil
+ ├── containers/
+ │ └── release-candidate - Defines Docker image used to build release
+ ├── daffodil-cli/
+ │ ├── README.md - Explains how to start Daffodil
+ │ ├── bin.LICENSE - Contains licenses of Daffodil and subcomponents
+ │ ├── bin.NOTICE - Contains notices of Daffodil and subcomponents
+ │ ├── build.sbt - Tells sbt how to build CLI and installers
+ │ └── src/ - Contains CLI source code, tests, scripts, etc.
+ ├── daffodil-core/
+ │ └── src/ - Contains Daffodil's core
+ ├── daffodil-io/
+ │ └── src/ - Contains Daffodil's input/output/charsets
+ ├── daffodil-japi/
+ │ ├── build.sbt - Tells sbt how to generate javadocs
+ │ └── src/ - Contains Daffodil's Java API
+ ├── daffodil-lib/
+ │ └── src/ - Contains Daffodil's utilities
+ ├── daffodil-macro-lib/
+ │ └── src/ - Defines Daffodil's Scala 2.x macros
+ ├── daffodil-propgen/
+ │ └── src/ - Contains generators to generate more source code
Review Comment:
Yes, I've removed the `src\` lines, moved the descriptions to the parent directory, and the directory tree does look more compact although I had to widen the whitespace between all the dirs and all the descriptions to make enough room. I also used code fencing instead of indentation to keep the lines from getting a lot wider in the editor.
##########
DEVELOP.md:
##########
@@ -0,0 +1,485 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Daffodil Developer Guide
+
+This guide is for a software engineer who wants to develop Daffodil
+code. This guide will help you become familiar with DFDL schemas,
+DFDL processors, and Daffodil development including its code style,
+code workflow, development environment, directory organization,
+documentation, and tests.
+
+## DFDL
+
+The [Data Format Description
+Language](https://en.wikipedia.org/wiki/Data_Format_Description_Language)
+(DFDL) is a language used to describe almost all data formats both
+logically and physically. DFDL is not a data format or a procedural
+language; rather, it is a data modeling language based on a subset of
+XML Schema annotated with DFDL properties describing the
+representation and layout of each element of the schema inside a
+native text or binary data format. A DFDL schema allows data to be
+converted between its native data format (physical representation,
+also called a text or binary file) and a DFDL information set (logical
+representation, also called an infoset) such as an XML document, EXI
+document, JSON document, SAX callbacks, or several document object
+model APIs in memory (JDOM, Scala Node, W3C DOM). When you have a
+DFDL schema for a native data format, you can pick whichever infoset
+type is easiest for you or your application to use and tell a DFDL
+processor to read ("parse") a text or binary file from its native data
+format to that infoset type. You or your application can do whatever
+you need to do with the infoset and then you can use the same DFDL
+schema and DFDL processor to write ("unparse") the infoset back to its
+native text or binary file format again, completing a round trip from
+native data to infoset to native data again.
+
+Using DFDL avoids inventing a completely new data modeling language,
+avoids writing any parsing and serialization code (with all the bugs
+that normally arise from implementing such code procedurally), and
+makes it much easier to convert any native data format to an infoset,
+operate on the infoset, and convert an infoset back to its native data
+format again.
+
+To learn more about DFDL, you can watch two short
+[videos](https://community.ibm.com/community/user/integration/viewdocument/get-started-with-the-data-format-de)
+put together by Steve Hanson, co-chair of the DFDL working group, read
+a
+[slideshow](https://www.slideshare.net/mbeckerle/dfdl-and-apache-daffodil-overview-from-owl-cyber-defense)
+written by Mike Beckerle, co-chair of the DFDL working group, or go
+through some [tutorials](http://www.xfront.com/DFDL/) written by Roger
+Costello, chair of the DFTVL working group (Data Format Transformation
+and Validation Language, a future not-yet-defined language to specify
+policies for cross-domain system devices).
+
+## Apache Daffodil, IBM DFDL, and ESA DFDL4S
+
+The standards organization in which DFDL started, the Open Grid Forum,
+required 2 implementations in order to move forward with the
+standardization process. This means that there are two leading DFDL
+processors, a commercial implementation called IBM DFDL bundled into
+IBM's [Integration
+Bus](https://www.ibm.com/docs/en/integration-bus/10.0?topic=model-data-format-description-language-dfdl)
+products and an open source implementation called [Apache
+Daffodil](https://daffodil.apache.org/) hosted by the Apache Software
+Foundation. The European Space Agency also has created a proprietary
+implementation called [ESA
+DFDL4S](https://eop-cfi.esa.int/index.php/applications/dfdl4s), which
+can be used only with their satellite communication formats and is
+provided only in the form of binary libraries, not source code.
+
+Among these three DFDL processors, Apache Daffodil is considered the
+most modern and thorough implementation of the [Data Format
+Description Language v1.0
+Specification](https://daffodil.apache.org/docs/dfdl/). Even so,
+Apache Daffodil lists some [unsupported
+features](https://daffodil.apache.org/unsupported/) of the DFDL
+specification. IBM DFDL lists some more [unsupported
+features](https://www.ibm.com/docs/en/integration-bus/10.0?topic=dfdl-unsupported-features)
+and also lists some [implementation-specific
+limits](https://www.ibm.com/docs/en/integration-bus/10.0?topic=dfdl-implementation-specific-limits).
+These limitations will not prevent you from writing DFDL schemas for
+almost all data formats, but they will reveal which parts of the DFDL
+specification are rarely used.
+
+## Daffodil Development
+
+The Apache Software Foundation hosts the Apache Daffodil project on
+the following ASF infrastructure:
+
+- Daffodil's [issue
+ tracker](https://issues.apache.org/jira/projects/DAFFODIL/) is
+ hosted on JIRA
+- Daffodil's
+ [users](https://lists.apache.org/list.html?users@daffodil.apache.org),
+ [dev](https://lists.apache.org/list.html?dev@daffodil.apache.org),
+ and
+ [commits](https://lists.apache.org/list.html?commits@daffodil.apache.org)
+ mailing lists are hosted on Apache Pony Mail
+- Daffodil's [source code](https://github.com/apache/daffodil) is
+ hosted on GitHub
+- Daffodil's
+ [wiki](https://cwiki.apache.org/confluence/display/DAFFODIL/) is
+ hosted on Confluence
+- Daffodil's [website](https://daffodil.apache.org/) is hosted on
+ Apache with static content generated by Jekyll using jekyll-asciidoc
+ and asciidoctor-diagram plugins
+
+A good Daffodil developer has a GitHub account and two copies of
+Daffodil's source code repository, a reading copy directly cloned from
+the Apache repository and a working copy cloned from the developer's
+own fork of the Apache repository. A good Daffodil developer also
+gets an account on the Apache Software Foundation's JIRA and
+Confluence servers and subscribes to all three Daffodil mailing lists.
+
+You can clone your reading copy directly from the Apache repository,
+but you should never make any changes in it. The only commands you
+should ever run in your reading copy are `git pull`, `git log
+ORIG_HEAD..HEAD`, and `git diff ORIG_HEAD..HEAD` in order to see what
+changes have been made by other developers since your last pull from
+the Apache repository. Your reading copy should remain an exact copy
+of the Apache source code repository at all times which you can use
+for reading or occasionally running `diff` commands between your
+reading copy and working copy. To be safe, rename your reading copy
+from `daffodil` to `daffodil-asf` and never edit files or run `sbt` in
+it so there will be nothing to push to the Apache repository even if
+you accidentally run `git push` from the wrong source tree.
+
+You can set up your working copy by forking the Apache repository to
+your own GitHub account and then cloning your working copy from your
+fork. Cloning your working copy from your fork instead of the Apache
+repository ensures that any changes you commit in your working copy
+will be pushed safely to only your own fork, not the Apache
+repository.
+
+### Code Style
+
+Daffodil mandates standard Scala formatting and its code generally
+keeps close to that formatting consistently. No one has run an
+automatic Scala formatter on the codebase yet (there has not been much
+need since the code is already formatted pretty well) but
+[DAFFODIL-2133](https://issues.apache.org/jira/browse/DAFFODIL-2133)
+is waiting for someone willing to do that work.
+
+Daffodil mandates that at least 80% of new code and modified code
+should be covered by unit tests or TDML tests. Daffodil naming
+conventions and more are covered in the Confluence pages [Code Style
+Guidelines](https://cwiki.apache.org/confluence/display/DAFFODIL/Code+Style+Guidelines)
+and [Coding for
+Performance](https://cwiki.apache.org/confluence/display/DAFFODIL/Coding+for+Performance).
+
+### Code Workflow
+
+It is very important that you fork the Apache repository to your own
+GitHub account before you start making changes to Apache Daffodil
+source code. You should always make changes only to your own fork of
+the upstream Apache repository, that is, clone your working copy from
+your fork and push commits back to your own fork only. The same
+workflow applies to Apache Daffodil's other two source code
+repositories, that is, always fork these repositories and clone from
+your forks before you make any changes to the [Daffodil VS Code
+Extension](https://github.com/apache/daffodil-vscode) or [Daffodil
+Website](https://github.com/apache/daffodil-site). When you are ready
+to contribute code from any of your forks to any of these upstream
+Apache repositories, you must create a pull request from your fork,
+allow the Apache Daffodil developers to review your contribution, make
+any suggested changes, and wait for their approval before you or
+someone else can merge your code into one of Apache's source code
+repositories.
+
+The Daffodil developers want each pull request to add only one commit
+to the Apache repository to keep its git history as readable and
+bisectable as possible. Your pull request should fix a JIRA issue in
+its entirety but it should not fix 2 or more unrelated JIRA issues in
+the same commit either. Your pull request should start with only one
+commit in it before you ask the developers to review it and your pull
+request should end with only one commit in it after you receive enough
+reviewer approvals (two +1's for Daffodil and Daffodil VS Code
+Extension, one +1 for Daffodil Website). If you have made multiple
+commits to your pull request due to reviewer suggestions (which is the
+usual case), you must not merge these multiple commits into the Apache
+repository. You must rebase and squash your pull request back to one
+commit as shown in the [Code Contributor
+Workflow](https://cwiki.apache.org/confluence/display/DAFFODIL/Code+Contributor+Workflow),
+which you should read and follow carefully to ensure a successful code
+contribution to Apache Daffodil.
+
+Before you change Daffodil code, you will need to know how to use JIRA
+as well as GitHub. You will assign a JIRA issue to yourself before
+you start working on it to avoid redundant effort in case another
+developer decides to work on the same issue. Then you will mark the
+issue as fixed after your pull request is merged. Here are the steps
+you need to perform in JIRA to mark the issue resolved after your pull
+request is merged (please don't forget these steps):
+
+1. Click "Resolve Issue"
+2. Click the "Fix Version/s" dropdown and mark the issue as fixed in
+ the currently unreleased version
+3. Paste "Fixed in commit xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
+ into the comment field (replacing the xxx's with the correct hash)
+4. Press the "Resolve" button
+
+That's it. You can expect someone else to test the change and close
+the JIRA issue so you don't need to, and should not, close the issue
+yourself.
+
+### Development Environment
+
+You will find build instructions in [BUILD.md](BUILD.md) and
+[README.md](README.md). You will find more instructions how to set up
+your development environment in the Confluence pages [Java
+Setup](https://cwiki.apache.org/confluence/display/DAFFODIL/Java+Setup+and+Notes)
+and [IDEA
+Setup](https://cwiki.apache.org/confluence/display/DAFFODIL/IntelliJ+IDEA+Setup).
+
+### Directory Organization
+
+Daffodil is composed of one top-level project and 20
+subdirectory-level modules (some of these modules contain only
+generators, macros, tests and tutorials). Daffodil's source code
+repository is organized as follows:
+
+ daffodil/
+ ├── .asf.yaml - Enables/disables parts of GitHub's UI
+ ├── .gitattributes - Excludes KEY from source releases
+ ├── .gitignore - Ignores auto-generated files from source control
+ ├── .sbtopts - Tells sbt to use 4GB of memory
+ ├── BUILD.md - Lists Daffodil's build requirements
+ ├── KEYS - Lists public keys of Daffodil developers
+ ├── LICENSE - Licenses Daffodil under the Apache License
+ ├── NOTICE - Lists notices affecting LICENSE
+ ├── README.md - Explains how to get started
+ ├── build.sbt - Tells sbt how to build Daffodil
+ ├── containers/
+ │ └── release-candidate - Defines Docker image used to build release
+ ├── daffodil-cli/
+ │ ├── README.md - Explains how to start Daffodil
+ │ ├── bin.LICENSE - Contains licenses of Daffodil and subcomponents
+ │ ├── bin.NOTICE - Contains notices of Daffodil and subcomponents
+ │ ├── build.sbt - Tells sbt how to build CLI and installers
+ │ └── src/ - Contains CLI source code, tests, scripts, etc.
+ ├── daffodil-core/
+ │ └── src/ - Contains Daffodil's core
+ ├── daffodil-io/
+ │ └── src/ - Contains Daffodil's input/output/charsets
+ ├── daffodil-japi/
+ │ ├── build.sbt - Tells sbt how to generate javadocs
+ │ └── src/ - Contains Daffodil's Java API
+ ├── daffodil-lib/
+ │ └── src/ - Contains Daffodil's utilities
+ ├── daffodil-macro-lib/
+ │ └── src/ - Defines Daffodil's Scala 2.x macros
+ ├── daffodil-propgen/
+ │ └── src/ - Contains generators to generate more source code
+ ├── daffodil-runtime1/
+ │ └── src/ - Contains Daffodil's Scala parser
+ ├── daffodil-runtime1-layers/
+ │ └── src/ - Contains Daffodil's Scala layer transformers
+ ├── daffodil-runtime1-unparser/
+ │ └── src/ - Contains Daffodil's Scala unparser
+ ├── daffodil-runtime2/
+ │ └── src/ - Contains Daffodil's C code generator
+ ├── daffodil-sapi/
+ │ └── src/ - Contains Daffodil's Scala API
+ ├── daffodil-schematron/
+ │ ├── README.md - Explains how to validate with Schematron rules
+ │ └── src/ - Contains Daffodil's Schematron validator
+ ├── daffodil-tdml-lib/
+ │ └── src/ - Contains Daffodil's TDML runner
+ ├── daffodil-tdml-processor/
+ │ └── src/ - Contains Daffodil's Scala & C TDML implementations
+ ├── daffodil-test/
+ │ └── src/ - Contains Daffodil's TDML tests
+ ├── daffodil-test-ibm1/
+ │ └── src/ - Contains more TDML tests from IBM & Tresys
+ ├── daffodil-udf/
+ │ ├── README.md - Explains how to implement user defined functions
+ │ └── src/ - Contains Daffodil's UDF interfaces
+ ├── project/
+ │ ├── Dependencies.scala - Defines third party dependencies
+ │ ├── Rat.scala - Lists files to be ignored by license audit tool
+ │ ├── build.properties - Defines which sbt version to use
+ │ └── plugins.sbt - Adds third party plugins to sbt
+ ├── test-stdLayout/
+ │ └── src/ - Contains TDML tests laid out in different ways
+ └── tutorials/
+ └── src/ - Contains Daffodil's TDML test tutorials
+
+When compiled into a command line interface, Daffodil is composed of a
+script, a log4j2 configuration file, 14 Daffodil jars, and 37
+third-party dependency jars. These numbers are accurate for Apache
+Daffodil release 3.4.0, but may be different for previous or
+later releases. Daffodil's command line interface is released in
+binary form as follows:
+
+ apache-daffodil-3.4.0-bin/
+ ├── LICENSE - Contains licenses of Daffodil and subcomponents
+ ├── NOTICE - Contains notices of Daffodil and subcomponents
+ ├── README.md - Explains how to start Daffodil
+ ├── bin/
+ │ ├── daffodil - Starts Daffodil on Linux
+ │ └── daffodil.bat - Starts Daffodil on Windows
+ ├── conf/
+ │ └── log4j2.xml - Configures log4j logger output
+ └── lib/ - Contains Daffodil jars and third party jars
+
+When an application uses Daffodil's Java or Scala API (not counting
+the jars used only by Daffodil's command line interface), it will need
+to put 8 Daffodil jars and 18 third-party dependency jars on its
+classpath. These numbers are accurate for Apache Daffodil release
Review Comment:
OK, I've removed the numbers.
##########
DEVELOP.md:
##########
@@ -0,0 +1,485 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Daffodil Developer Guide
+
+This guide is for a software engineer who wants to develop Daffodil
+code. This guide will help you become familiar with DFDL schemas,
+DFDL processors, and Daffodil development including its code style,
+code workflow, development environment, directory organization,
+documentation, and tests.
+
+## DFDL
+
+The [Data Format Description
+Language](https://en.wikipedia.org/wiki/Data_Format_Description_Language)
+(DFDL) is a language used to describe almost all data formats both
+logically and physically. DFDL is not a data format or a procedural
+language; rather, it is a data modeling language based on a subset of
+XML Schema annotated with DFDL properties describing the
+representation and layout of each element of the schema inside a
+native text or binary data format. A DFDL schema allows data to be
+converted between its native data format (physical representation,
+also called a text or binary file) and a DFDL information set (logical
+representation, also called an infoset) such as an XML document, EXI
+document, JSON document, SAX callbacks, or several document object
+model APIs in memory (JDOM, Scala Node, W3C DOM). When you have a
+DFDL schema for a native data format, you can pick whichever infoset
+type is easiest for you or your application to use and tell a DFDL
+processor to read ("parse") a text or binary file from its native data
+format to that infoset type. You or your application can do whatever
+you need to do with the infoset and then you can use the same DFDL
+schema and DFDL processor to write ("unparse") the infoset back to its
+native text or binary file format again, completing a round trip from
+native data to infoset to native data again.
+
+Using DFDL avoids inventing a completely new data modeling language,
+avoids writing any parsing and serialization code (with all the bugs
+that normally arise from implementing such code procedurally), and
+makes it much easier to convert any native data format to an infoset,
+operate on the infoset, and convert an infoset back to its native data
+format again.
+
+To learn more about DFDL, you can watch two short
+[videos](https://community.ibm.com/community/user/integration/viewdocument/get-started-with-the-data-format-de)
+put together by Steve Hanson, co-chair of the DFDL working group, read
+a
+[slideshow](https://www.slideshare.net/mbeckerle/dfdl-and-apache-daffodil-overview-from-owl-cyber-defense)
+written by Mike Beckerle, co-chair of the DFDL working group, or go
+through some [tutorials](http://www.xfront.com/DFDL/) written by Roger
+Costello, chair of the DFTVL working group (Data Format Transformation
+and Validation Language, a future not-yet-defined language to specify
+policies for cross-domain system devices).
Review Comment:
Thanks, I've made the correction.
##########
DEVELOP.md:
##########
@@ -0,0 +1,485 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Daffodil Developer Guide
+
+This guide is for a software engineer who wants to develop Daffodil
+code. This guide will help you become familiar with DFDL schemas,
+DFDL processors, and Daffodil development including its code style,
+code workflow, development environment, directory organization,
+documentation, and tests.
+
+## DFDL
+
+The [Data Format Description
+Language](https://en.wikipedia.org/wiki/Data_Format_Description_Language)
+(DFDL) is a language used to describe almost all data formats both
+logically and physically. DFDL is not a data format or a procedural
+language; rather, it is a data modeling language based on a subset of
+XML Schema annotated with DFDL properties describing the
+representation and layout of each element of the schema inside a
+native text or binary data format. A DFDL schema allows data to be
+converted between its native data format (physical representation,
+also called a text or binary file) and a DFDL information set (logical
+representation, also called an infoset) such as an XML document, EXI
+document, JSON document, SAX callbacks, or several document object
+model APIs in memory (JDOM, Scala Node, W3C DOM). When you have a
+DFDL schema for a native data format, you can pick whichever infoset
+type is easiest for you or your application to use and tell a DFDL
+processor to read ("parse") a text or binary file from its native data
+format to that infoset type. You or your application can do whatever
+you need to do with the infoset and then you can use the same DFDL
+schema and DFDL processor to write ("unparse") the infoset back to its
+native text or binary file format again, completing a round trip from
+native data to infoset to native data again.
+
+Using DFDL avoids inventing a completely new data modeling language,
+avoids writing any parsing and serialization code (with all the bugs
+that normally arise from implementing such code procedurally), and
+makes it much easier to convert any native data format to an infoset,
+operate on the infoset, and convert an infoset back to its native data
+format again.
+
+To learn more about DFDL, you can watch two short
+[videos](https://community.ibm.com/community/user/integration/viewdocument/get-started-with-the-data-format-de)
+put together by Steve Hanson, co-chair of the DFDL working group, read
+a
+[slideshow](https://www.slideshare.net/mbeckerle/dfdl-and-apache-daffodil-overview-from-owl-cyber-defense)
+written by Mike Beckerle, co-chair of the DFDL working group, or go
+through some [tutorials](http://www.xfront.com/DFDL/) written by Roger
+Costello, chair of the DFTVL working group (Data Format Transformation
+and Validation Language, a future not-yet-defined language to specify
+policies for cross-domain system devices).
+
+## Apache Daffodil, IBM DFDL, and ESA DFDL4S
+
+The standards organization in which DFDL started, the Open Grid Forum,
+required 2 implementations in order to move forward with the
+standardization process. This means that there are two leading DFDL
+processors, a commercial implementation called IBM DFDL bundled into
+IBM's [Integration
+Bus](https://www.ibm.com/docs/en/integration-bus/10.0?topic=model-data-format-description-language-dfdl)
+products and an open source implementation called [Apache
+Daffodil](https://daffodil.apache.org/) hosted by the Apache Software
+Foundation. The European Space Agency also has created a proprietary
+implementation called [ESA
+DFDL4S](https://eop-cfi.esa.int/index.php/applications/dfdl4s), which
+can be used only with their satellite communication formats and is
+provided only in the form of binary libraries, not source code.
+
+Among these three DFDL processors, Apache Daffodil is considered the
+most modern and thorough implementation of the [Data Format
+Description Language v1.0
+Specification](https://daffodil.apache.org/docs/dfdl/). Even so,
+Apache Daffodil lists some [unsupported
+features](https://daffodil.apache.org/unsupported/) of the DFDL
+specification. IBM DFDL lists some more [unsupported
+features](https://www.ibm.com/docs/en/integration-bus/10.0?topic=dfdl-unsupported-features)
+and also lists some [implementation-specific
+limits](https://www.ibm.com/docs/en/integration-bus/10.0?topic=dfdl-implementation-specific-limits).
+These limitations will not prevent you from writing DFDL schemas for
+almost all data formats, but they will reveal which parts of the DFDL
+specification are rarely used.
+
+## Daffodil Development
+
+The Apache Software Foundation hosts the Apache Daffodil project on
+the following ASF infrastructure:
+
+- Daffodil's [issue
+ tracker](https://issues.apache.org/jira/projects/DAFFODIL/) is
+ hosted on JIRA
+- Daffodil's
+ [users](https://lists.apache.org/list.html?users@daffodil.apache.org),
+ [dev](https://lists.apache.org/list.html?dev@daffodil.apache.org),
+ and
+ [commits](https://lists.apache.org/list.html?commits@daffodil.apache.org)
+ mailing lists are hosted on Apache Pony Mail
+- Daffodil's [source code](https://github.com/apache/daffodil) is
+ hosted on GitHub
+- Daffodil's
+ [wiki](https://cwiki.apache.org/confluence/display/DAFFODIL/) is
+ hosted on Confluence
+- Daffodil's [website](https://daffodil.apache.org/) is hosted on
+ Apache with static content generated by Jekyll using jekyll-asciidoc
+ and asciidoctor-diagram plugins
+
+A good Daffodil developer has a GitHub account and two copies of
+Daffodil's source code repository, a reading copy directly cloned from
+the Apache repository and a working copy cloned from the developer's
+own fork of the Apache repository. A good Daffodil developer also
+gets an account on the Apache Software Foundation's JIRA and
+Confluence servers and subscribes to all three Daffodil mailing lists.
+
+You can clone your reading copy directly from the Apache repository,
+but you should never make any changes in it. The only commands you
+should ever run in your reading copy are `git pull`, `git log
+ORIG_HEAD..HEAD`, and `git diff ORIG_HEAD..HEAD` in order to see what
+changes have been made by other developers since your last pull from
+the Apache repository. Your reading copy should remain an exact copy
+of the Apache source code repository at all times which you can use
+for reading or occasionally running `diff` commands between your
+reading copy and working copy. To be safe, rename your reading copy
+from `daffodil` to `daffodil-asf` and never edit files or run `sbt` in
+it so there will be nothing to push to the Apache repository even if
+you accidentally run `git push` from the wrong source tree.
+
+You can set up your working copy by forking the Apache repository to
+your own GitHub account and then cloning your working copy from your
+fork. Cloning your working copy from your fork instead of the Apache
+repository ensures that any changes you commit in your working copy
+will be pushed safely to only your own fork, not the Apache
+repository.
+
+### Code Style
+
+Daffodil mandates standard Scala formatting and its code generally
+keeps close to that formatting consistently. No one has run an
+automatic Scala formatter on the codebase yet (there has not been much
+need since the code is already formatted pretty well) but
+[DAFFODIL-2133](https://issues.apache.org/jira/browse/DAFFODIL-2133)
Review Comment:
Yes, I've added a reminder to the ticket.
##########
DEVELOP.md:
##########
@@ -0,0 +1,485 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Daffodil Developer Guide
+
+This guide is for a software engineer who wants to develop Daffodil
+code. This guide will help you become familiar with DFDL schemas,
+DFDL processors, and Daffodil development including its code style,
+code workflow, development environment, directory organization,
+documentation, and tests.
+
+## DFDL
+
+The [Data Format Description
+Language](https://en.wikipedia.org/wiki/Data_Format_Description_Language)
+(DFDL) is a language used to describe almost all data formats both
+logically and physically. DFDL is not a data format or a procedural
+language; rather, it is a data modeling language based on a subset of
+XML Schema annotated with DFDL properties describing the
+representation and layout of each element of the schema inside a
+native text or binary data format. A DFDL schema allows data to be
+converted between its native data format (physical representation,
+also called a text or binary file) and a DFDL information set (logical
+representation, also called an infoset) such as an XML document, EXI
+document, JSON document, SAX callbacks, or several document object
+model APIs in memory (JDOM, Scala Node, W3C DOM). When you have a
+DFDL schema for a native data format, you can pick whichever infoset
+type is easiest for you or your application to use and tell a DFDL
+processor to read ("parse") a text or binary file from its native data
+format to that infoset type. You or your application can do whatever
+you need to do with the infoset and then you can use the same DFDL
+schema and DFDL processor to write ("unparse") the infoset back to its
+native text or binary file format again, completing a round trip from
+native data to infoset to native data again.
+
+Using DFDL avoids inventing a completely new data modeling language,
+avoids writing any parsing and serialization code (with all the bugs
+that normally arise from implementing such code procedurally), and
+makes it much easier to convert any native data format to an infoset,
+operate on the infoset, and convert an infoset back to its native data
+format again.
+
+To learn more about DFDL, you can watch two short
+[videos](https://community.ibm.com/community/user/integration/viewdocument/get-started-with-the-data-format-de)
+put together by Steve Hanson, co-chair of the DFDL working group, read
+a
+[slideshow](https://www.slideshare.net/mbeckerle/dfdl-and-apache-daffodil-overview-from-owl-cyber-defense)
+written by Mike Beckerle, co-chair of the DFDL working group, or go
+through some [tutorials](http://www.xfront.com/DFDL/) written by Roger
+Costello, chair of the DFTVL working group (Data Format Transformation
+and Validation Language, a future not-yet-defined language to specify
+policies for cross-domain system devices).
+
+## Apache Daffodil, IBM DFDL, and ESA DFDL4S
+
+The standards organization in which DFDL started, the Open Grid Forum,
+required 2 implementations in order to move forward with the
+standardization process. This means that there are two leading DFDL
+processors, a commercial implementation called IBM DFDL bundled into
+IBM's [Integration
+Bus](https://www.ibm.com/docs/en/integration-bus/10.0?topic=model-data-format-description-language-dfdl)
+products and an open source implementation called [Apache
+Daffodil](https://daffodil.apache.org/) hosted by the Apache Software
+Foundation. The European Space Agency also has created a proprietary
+implementation called [ESA
+DFDL4S](https://eop-cfi.esa.int/index.php/applications/dfdl4s), which
+can be used only with their satellite communication formats and is
+provided only in the form of binary libraries, not source code.
+
+Among these three DFDL processors, Apache Daffodil is considered the
+most modern and thorough implementation of the [Data Format
+Description Language v1.0
+Specification](https://daffodil.apache.org/docs/dfdl/). Even so,
+Apache Daffodil lists some [unsupported
+features](https://daffodil.apache.org/unsupported/) of the DFDL
+specification. IBM DFDL lists some more [unsupported
+features](https://www.ibm.com/docs/en/integration-bus/10.0?topic=dfdl-unsupported-features)
+and also lists some [implementation-specific
+limits](https://www.ibm.com/docs/en/integration-bus/10.0?topic=dfdl-implementation-specific-limits).
+These limitations will not prevent you from writing DFDL schemas for
+almost all data formats, but they will reveal which parts of the DFDL
+specification are rarely used.
+
+## Daffodil Development
+
+The Apache Software Foundation hosts the Apache Daffodil project on
+the following ASF infrastructure:
+
+- Daffodil's [issue
+ tracker](https://issues.apache.org/jira/projects/DAFFODIL/) is
+ hosted on JIRA
+- Daffodil's
+ [users](https://lists.apache.org/list.html?users@daffodil.apache.org),
+ [dev](https://lists.apache.org/list.html?dev@daffodil.apache.org),
+ and
+ [commits](https://lists.apache.org/list.html?commits@daffodil.apache.org)
+ mailing lists are hosted on Apache Pony Mail
+- Daffodil's [source code](https://github.com/apache/daffodil) is
+ hosted on GitHub
+- Daffodil's
+ [wiki](https://cwiki.apache.org/confluence/display/DAFFODIL/) is
+ hosted on Confluence
+- Daffodil's [website](https://daffodil.apache.org/) is hosted on
+ Apache with static content generated by Jekyll using jekyll-asciidoc
+ and asciidoctor-diagram plugins
+
+A good Daffodil developer has a GitHub account and two copies of
+Daffodil's source code repository, a reading copy directly cloned from
+the Apache repository and a working copy cloned from the developer's
+own fork of the Apache repository. A good Daffodil developer also
+gets an account on the Apache Software Foundation's JIRA and
+Confluence servers and subscribes to all three Daffodil mailing lists.
+
+You can clone your reading copy directly from the Apache repository,
+but you should never make any changes in it. The only commands you
+should ever run in your reading copy are `git pull`, `git log
+ORIG_HEAD..HEAD`, and `git diff ORIG_HEAD..HEAD` in order to see what
+changes have been made by other developers since your last pull from
+the Apache repository. Your reading copy should remain an exact copy
+of the Apache source code repository at all times which you can use
+for reading or occasionally running `diff` commands between your
+reading copy and working copy. To be safe, rename your reading copy
+from `daffodil` to `daffodil-asf` and never edit files or run `sbt` in
+it so there will be nothing to push to the Apache repository even if
+you accidentally run `git push` from the wrong source tree.
+
+You can set up your working copy by forking the Apache repository to
+your own GitHub account and then cloning your working copy from your
+fork. Cloning your working copy from your fork instead of the Apache
+repository ensures that any changes you commit in your working copy
+will be pushed safely to only your own fork, not the Apache
+repository.
+
+### Code Style
+
+Daffodil mandates standard Scala formatting and its code generally
+keeps close to that formatting consistently. No one has run an
+automatic Scala formatter on the codebase yet (there has not been much
+need since the code is already formatted pretty well) but
+[DAFFODIL-2133](https://issues.apache.org/jira/browse/DAFFODIL-2133)
+is waiting for someone willing to do that work.
+
+Daffodil mandates that at least 80% of new code and modified code
+should be covered by unit tests or TDML tests. Daffodil naming
+conventions and more are covered in the Confluence pages [Code Style
+Guidelines](https://cwiki.apache.org/confluence/display/DAFFODIL/Code+Style+Guidelines)
+and [Coding for
+Performance](https://cwiki.apache.org/confluence/display/DAFFODIL/Coding+for+Performance).
+
+### Code Workflow
+
+It is very important that you fork the Apache repository to your own
+GitHub account before you start making changes to Apache Daffodil
+source code. You should always make changes only to your own fork of
+the upstream Apache repository, that is, clone your working copy from
+your fork and push commits back to your own fork only. The same
+workflow applies to Apache Daffodil's other two source code
+repositories, that is, always fork these repositories and clone from
+your forks before you make any changes to the [Daffodil VS Code
+Extension](https://github.com/apache/daffodil-vscode) or [Daffodil
+Website](https://github.com/apache/daffodil-site). When you are ready
+to contribute code from any of your forks to any of these upstream
+Apache repositories, you must create a pull request from your fork,
+allow the Apache Daffodil developers to review your contribution, make
+any suggested changes, and wait for their approval before you or
+someone else can merge your code into one of Apache's source code
+repositories.
+
+The Daffodil developers want each pull request to add only one commit
+to the Apache repository to keep its git history as readable and
+bisectable as possible. Your pull request should fix a JIRA issue in
+its entirety but it should not fix 2 or more unrelated JIRA issues in
+the same commit either. Your pull request should start with only one
+commit in it before you ask the developers to review it and your pull
+request should end with only one commit in it after you receive enough
+reviewer approvals (two +1's for Daffodil and Daffodil VS Code
+Extension, one +1 for Daffodil Website). If you have made multiple
+commits to your pull request due to reviewer suggestions (which is the
+usual case), you must not merge these multiple commits into the Apache
+repository. You must rebase and squash your pull request back to one
+commit as shown in the [Code Contributor
+Workflow](https://cwiki.apache.org/confluence/display/DAFFODIL/Code+Contributor+Workflow),
+which you should read and follow carefully to ensure a successful code
+contribution to Apache Daffodil.
+
+Before you change Daffodil code, you will need to know how to use JIRA
+as well as GitHub. You will assign a JIRA issue to yourself before
+you start working on it to avoid redundant effort in case another
+developer decides to work on the same issue. Then you will mark the
+issue as fixed after your pull request is merged. Here are the steps
+you need to perform in JIRA to mark the issue resolved after your pull
+request is merged (please don't forget these steps):
+
+1. Click "Resolve Issue"
+2. Click the "Fix Version/s" dropdown and mark the issue as fixed in
+ the currently unreleased version
+3. Paste "Fixed in commit xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
+ into the comment field (replacing the xxx's with the correct hash)
+4. Press the "Resolve" button
+
+That's it. You can expect someone else to test the change and close
+the JIRA issue so you don't need to, and should not, close the issue
+yourself.
+
+### Development Environment
+
+You will find build instructions in [BUILD.md](BUILD.md) and
+[README.md](README.md). You will find more instructions how to set up
+your development environment in the Confluence pages [Java
+Setup](https://cwiki.apache.org/confluence/display/DAFFODIL/Java+Setup+and+Notes)
+and [IDEA
+Setup](https://cwiki.apache.org/confluence/display/DAFFODIL/IntelliJ+IDEA+Setup).
+
+### Directory Organization
+
+Daffodil is composed of one top-level project and 20
+subdirectory-level modules (some of these modules contain only
+generators, macros, tests and tutorials). Daffodil's source code
+repository is organized as follows:
+
+ daffodil/
+ ├── .asf.yaml - Enables/disables parts of GitHub's UI
+ ├── .gitattributes - Excludes KEY from source releases
+ ├── .gitignore - Ignores auto-generated files from source control
+ ├── .sbtopts - Tells sbt to use 4GB of memory
+ ├── BUILD.md - Lists Daffodil's build requirements
+ ├── KEYS - Lists public keys of Daffodil developers
+ ├── LICENSE - Licenses Daffodil under the Apache License
+ ├── NOTICE - Lists notices affecting LICENSE
+ ├── README.md - Explains how to get started
+ ├── build.sbt - Tells sbt how to build Daffodil
+ ├── containers/
+ │ └── release-candidate - Defines Docker image used to build release
+ ├── daffodil-cli/
+ │ ├── README.md - Explains how to start Daffodil
+ │ ├── bin.LICENSE - Contains licenses of Daffodil and subcomponents
+ │ ├── bin.NOTICE - Contains notices of Daffodil and subcomponents
+ │ ├── build.sbt - Tells sbt how to build CLI and installers
+ │ └── src/ - Contains CLI source code, tests, scripts, etc.
+ ├── daffodil-core/
+ │ └── src/ - Contains Daffodil's core
+ ├── daffodil-io/
+ │ └── src/ - Contains Daffodil's input/output/charsets
+ ├── daffodil-japi/
+ │ ├── build.sbt - Tells sbt how to generate javadocs
+ │ └── src/ - Contains Daffodil's Java API
+ ├── daffodil-lib/
+ │ └── src/ - Contains Daffodil's utilities
+ ├── daffodil-macro-lib/
+ │ └── src/ - Defines Daffodil's Scala 2.x macros
+ ├── daffodil-propgen/
+ │ └── src/ - Contains generators to generate more source code
+ ├── daffodil-runtime1/
+ │ └── src/ - Contains Daffodil's Scala parser
+ ├── daffodil-runtime1-layers/
+ │ └── src/ - Contains Daffodil's Scala layer transformers
+ ├── daffodil-runtime1-unparser/
+ │ └── src/ - Contains Daffodil's Scala unparser
+ ├── daffodil-runtime2/
+ │ └── src/ - Contains Daffodil's C code generator
+ ├── daffodil-sapi/
+ │ └── src/ - Contains Daffodil's Scala API
+ ├── daffodil-schematron/
+ │ ├── README.md - Explains how to validate with Schematron rules
+ │ └── src/ - Contains Daffodil's Schematron validator
+ ├── daffodil-tdml-lib/
+ │ └── src/ - Contains Daffodil's TDML runner
+ ├── daffodil-tdml-processor/
+ │ └── src/ - Contains Daffodil's Scala & C TDML implementations
+ ├── daffodil-test/
+ │ └── src/ - Contains Daffodil's TDML tests
+ ├── daffodil-test-ibm1/
+ │ └── src/ - Contains more TDML tests from IBM & Tresys
+ ├── daffodil-udf/
+ │ ├── README.md - Explains how to implement user defined functions
+ │ └── src/ - Contains Daffodil's UDF interfaces
+ ├── project/
+ │ ├── Dependencies.scala - Defines third party dependencies
+ │ ├── Rat.scala - Lists files to be ignored by license audit tool
+ │ ├── build.properties - Defines which sbt version to use
+ │ └── plugins.sbt - Adds third party plugins to sbt
+ ├── test-stdLayout/
+ │ └── src/ - Contains TDML tests laid out in different ways
+ └── tutorials/
+ └── src/ - Contains Daffodil's TDML test tutorials
+
+When compiled into a command line interface, Daffodil is composed of a
+script, a log4j2 configuration file, 14 Daffodil jars, and 37
+third-party dependency jars. These numbers are accurate for Apache
+Daffodil release 3.4.0, but may be different for previous or
+later releases. Daffodil's command line interface is released in
+binary form as follows:
+
+ apache-daffodil-3.4.0-bin/
+ ├── LICENSE - Contains licenses of Daffodil and subcomponents
+ ├── NOTICE - Contains notices of Daffodil and subcomponents
+ ├── README.md - Explains how to start Daffodil
+ ├── bin/
+ │ ├── daffodil - Starts Daffodil on Linux
+ │ └── daffodil.bat - Starts Daffodil on Windows
+ ├── conf/
+ │ └── log4j2.xml - Configures log4j logger output
+ └── lib/ - Contains Daffodil jars and third party jars
+
+When an application uses Daffodil's Java or Scala API (not counting
+the jars used only by Daffodil's command line interface), it will need
+to put 8 Daffodil jars and 18 third-party dependency jars on its
+classpath. These numbers are accurate for Apache Daffodil release
+3.4.0, but may be slightly different for previous or later releases.
+The best way for a Java or Scala application to use Daffodil is to get
+Daffodil's Java API from Maven or get Daffodil's Scala API from SBT:
+
+```xml
+<dependency>
+ <groupId>org.apache.daffodil</groupId>
+ <artifactId>daffodil-japi_2.12</artifactId>
+ <version>3.4.0</version>
+</dependency>
+```
+
+```sbt
+libraryDependencies += "org.apache.daffodil" %% "daffodil-sapi" % "3.4.0"
+```
+
+When you use Daffodil's C code generator to generate C code from a
+DFDL schema, the newly created C code directory is organized as
+follows:
+
+ c/
+ ├── Makefile - Contains build and test targets
+ ├── libcli/
+ │ ├── cli_errors.c - Implements CLI error messages
+ │ ├── daffodil_getopt.c - Implements CLI options
+ │ ├── daffodil_main.c - Starts the CLI
+ │ ├── stack.c - Pushes XML nodes on a stack
+ │ ├── xml_reader.c - Reads an infoset from a XML file
+ │ └── xml_writer.c - Writes an infoset to a XML file
+ ├── libruntime/
+ │ ├── errors.c - Implements error messages
+ │ ├── generated_code.c - Implements functions to create/use infoset
+ │ ├── infoset.c - Implements functions to walk infoset
+ │ ├── parsers.c - Implements functions to read binary data
+ │ └── unparsers.c - Implements functions to write binary data
+ └── tests/
+ ├── bits.c - Tests integers with different lengths
+ └── extras.c - Avoids an undefined reference when linking
+
+You can use the Makefile to build a Daffodil executable (c/daffodil)
+and run c/daffodil with similar parse and unparse options as
+Daffodil's Scala command line interface. If you want to embed the
+generated C code into your own C application, you can call only
+libruntime to read and write binary data into C structs in memory or
+call both libruntime and libcli (replacing daffodil_main.c with your
+application's main.c) to read and write an XML infoset as well as
+binary data into C structs.
+
+### Documentation
+
+Unfortunately, it is not possible to search or edit all of Daffodil's
+documentation in one single place. Daffodil documentation is split
+between the
+[Confluence](https://cwiki.apache.org/confluence/display/DAFFODIL/)
+wiki, the [Daffodil](https://github.com/apache/daffodil) repository,
+and the [Daffodil Website](https://github.com/apache/daffodil-site)
+repository. Confluence has a page tree on the left side so you can
+find all the Confluence pages by expanding the page tree. The first
+link below is the only link you need to find or search all the pages
+in the wiki. However, there are a lot of pages in the wiki so here
+are some of the most important pages worth reading as well (sorted in
+alphabetical order).
+
+- [Apache Daffodil Wiki](https://cwiki.apache.org/confluence/display/DAFFODIL/)
+- [Code Contributor Workflow](https://cwiki.apache.org/confluence/display/DAFFODIL/Code+Contributor+Workflow)
+- [Code Style Guidelines](https://cwiki.apache.org/confluence/display/DAFFODIL/Code+Style+Guidelines)
+- [Coding for Performance](https://cwiki.apache.org/confluence/display/DAFFODIL/Coding+for+Performance)
+- [DFDL Schema Object Model](https://cwiki.apache.org/confluence/display/DAFFODIL/DFDL+Schema+Object+Model+%28DSOM%29+with+UML)
+- [Daffodil Maturity Model Assessment](https://cwiki.apache.org/confluence/display/DAFFODIL/Apache+Daffodil+Maturity+Model+Assessment)
+- [Daffodil Runtime 2](https://cwiki.apache.org/confluence/display/DAFFODIL/WIP%3A+Daffodil+Runtime+2)
+- [IntelliJ IDEA Setup](https://cwiki.apache.org/confluence/display/DAFFODIL/IntelliJ+IDEA+Setup)
+- [Object-Oriented Lazy Attribute Grammars](https://cwiki.apache.org/confluence/display/DAFFODIL/OOLAG+-+Object-Oriented+Lazy+Attribute+Grammars)
+- [Roadmap for Upcoming Releases](https://cwiki.apache.org/confluence/display/DAFFODIL/Roadmap+for+Upcoming+Releases)
+- [Scala References and Patterns](https://cwiki.apache.org/confluence/display/DAFFODIL/Scala+References+and+Patterns)
+
+Some of the Daffodil website pages don't seem to have any links
+pointing to them from other website pages. So that you can read all
+of the website pages in your browser more easily, here are direct
+links to every website page (sorted in alphabetical order).
+
+- [A TDML Tutorial](https://daffodil.apache.org/tutorials/tdmlTutorial.tdml.xml)
+- [About AsciiDoc for Documentation](https://daffodil.apache.org/dev/aboutAsciiDoc/)
+- [About TDML Tutorials](https://daffodil.apache.org/assets/tutorials/aboutTDMLTutorials/)
+- [Bug Report TDML Template](https://daffodil.apache.org/tutorials/bugReportTemplate.tdml.xml)
+- [Command Line Interface](https://daffodil.apache.org/cli/)
+- [Community](https://daffodil.apache.org/community/)
+- [Configuration](https://daffodil.apache.org/configuration/)
+- [DFDL Extensions](https://daffodil.apache.org/dfdl-extensions/)
+- [Daffodil Java API](https://daffodil.apache.org/docs/latest/javadoc/)
+- [Daffodil Scala API](https://daffodil.apache.org/docs/latest/scaladoc/)
+- [Daffodil and the DFDL Infoset](https://daffodil.apache.org/infoset/)
+- [Data Format Description Language v1.0 Specification](https://daffodil.apache.org/docs/dfdl/)
+- [Eclipse Configuration](https://daffodil.apache.org/eclipse-configuration/)
+- [Examples](https://daffodil.apache.org/examples/)
+- [Frequently Asked Questions](https://daffodil.apache.org/faq/)
+- [Getting Started](https://daffodil.apache.org/getting-started/)
+- [Hidden Groups](https://daffodil.apache.org/dev/design-notes/hidden-groups/)
+- [Home](https://daffodil.apache.org/)
+- [Infoset Inputters and Runtime 1 Streaming Unparser & NextElementResolver](https://daffodil.apache.org/dev/design-notes/infoset-inputter-streaming-unparser/)
+- [Interactive Debugger](https://daffodil.apache.org/debugger/)
+- [Namespace Binding Minimization](https://daffodil.apache.org/dev/design-notes/namespace-binding-minimization/)
+- [People](https://daffodil.apache.org/people/)
+- [Releases](https://daffodil.apache.org/releases/)
+- [Runtime2 ToDos](https://daffodil.apache.org/dev/design-notes/runtime2-todos/)
+- [Standard DFDL Schema Project Layout](https://daffodil.apache.org/dfdl-layout/)
+- [Term Sharing in the Schema Compiler](https://daffodil.apache.org/dev/design-notes/term-sharing-in-schema-compiler/)
+- [Test Data Markup Language](https://daffodil.apache.org/tdml/)
+- [Understanding dfdl:bitOrder](https://daffodil.apache.org/tutorials/bitorder.tutorial.tdml.xml)
+- [Unsupported Features and Errata](https://daffodil.apache.org/unsupported/)
+- [VS Code Extension](https://daffodil.apache.org/vscode/)
+- [design notes folder](https://daffodil.apache.org/dev/)
+
+### Tests
+
+The majority of Daffodil tests are written inside TDML files with
+corresponding Scala unit tests which allow your SBT or IDE to run
+these tests within the TDML files. Daffodil also has some pure Scala
+unit tests for testing smaller parts of Daffodil at a lower level.
+Both kinds of tests are important, although new Daffodil tests tend to
+be written in TDML and focus on testing issues with Daffodil's
+processing of DFDL schemas and parsing data or unparsing data.
+
+TDML tests are helpful for Daffodil developers because they allow a
+developer to write a TDML file with a schema, a test case, and test
+data which isolates a complex data format issue so the developer can
+work on that issue more easily. The developer is free to decide
+whether to define the schema and test data inline in the TDML file or
+leave the schema and test data in separate files which the TDML file
+reads.
+
+The developer also can choose which TDML implementation Daffodil
+should use to run a TDML test, either Daffodil's Scala backend or
+Daffodil's C code generator, simply by passing an option to Daffodil's
+test command on the command line or passing a TDMLImplementation
+instance to Daffodil's TDML Runner class in a Scala test class. The
+developer can use a TDML test to work on a part of DFDL which needs to
+be supported by the C code generator and keep running (or debugging)
+the TDML test until the generated C code works successfully:
+
+ daffodil test -I daffodilC nums.tdml
+
+Daffodil also has integration tests in the daffodil-cli module which
+test Daffodil's command line interface. These tests used to run
+Daffodil in a subprocess which took at least 20 minutes but now almost
+all of them run Daffodil in a thread which takes around 2 minutes. If
Review Comment:
OK, I've removed the sentence but added how to run the integration tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org