You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@daffodil.apache.org by "mbeckerle (via GitHub)" <gi...@apache.org> on 2023/03/01 18:07:27 UTC
[GitHub] [daffodil-site] mbeckerle opened a new pull request, #108: WIP: User doc on packaging and plugins. Drafts
mbeckerle opened a new pull request, #108:
URL: https://github.com/apache/daffodil-site/pull/108
(This PR replaces the PR https://github.com/apache/daffodil-site/pull/90 which was not on a named branch.)
At the moment no further review is necessary.
First draft of pages about packaging dfdl schemas, and about plugins.
All applications using Daffodil, at least via Runtime1 should, ideally, be taking advantage of packaging DFDL schemas in Jar files, using 'sbt publish' and managed-dependencies for inter-schema dependencies, etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [daffodil-site] mbeckerle commented on a diff in pull request #108: WIP: User doc on packaging and plugins. Drafts
Posted by "mbeckerle (via GitHub)" <gi...@apache.org>.
mbeckerle commented on code in PR #108:
URL: https://github.com/apache/daffodil-site/pull/108#discussion_r1122133654
##########
site/extensions.adoc:
##########
@@ -0,0 +1,60 @@
+:page-layout: page
+:url-asciidoctor: http://asciidoctor.org
+:keywords: plugins layering UDF charset
+// ///////////////////////////////////////////////////////////////////////////
Review Comment:
See comment https://github.com/apache/daffodil-site/pull/90#discussion_r889036765
##########
site/extensions.adoc:
##########
@@ -0,0 +1,60 @@
+:page-layout: page
+:url-asciidoctor: http://asciidoctor.org
+:keywords: plugins layering UDF charset
+// ///////////////////////////////////////////////////////////////////////////
+//
+// This file is written in AsciiDoc.
+//
+// If you can read this comment, your browser is not rendering asciidoc automatically.
+//
+// You need to install the asciidoc plugin to Chrome or Firefox
+// so that this page will be properly rendered for your viewing pleasure.
+//
+// You can get the plugins by searching the web for 'asciidoc plugin'
+//
+// You will want to change plugin settings to enable diagrams (they're off by default.)
+//
+// You need to view this page with Chrome or Firefox.
+//
+// ///////////////////////////////////////////////////////////////////////////
+//
+// When editing, please start each sentence on a new line.
+// See https://asciidoctor.org/docs/asciidoc-recommended-practices/#one-sentence-per-line[one sentence-per-line writing technique.]
+// This makes textual diffs of this file useful in a similar way to the way they work for code.
+//
+// //////////////////////////////////////////////////////////////////////////
+
+= DFDL Language Extensions in Daffodil
Review Comment:
See https://github.com/apache/daffodil-site/pull/90#discussion_r889749053
##########
site/extensions.adoc:
##########
@@ -0,0 +1,60 @@
+:page-layout: page
+:url-asciidoctor: http://asciidoctor.org
+:keywords: plugins layering UDF charset
+// ///////////////////////////////////////////////////////////////////////////
+//
+// This file is written in AsciiDoc.
+//
+// If you can read this comment, your browser is not rendering asciidoc automatically.
+//
+// You need to install the asciidoc plugin to Chrome or Firefox
+// so that this page will be properly rendered for your viewing pleasure.
+//
+// You can get the plugins by searching the web for 'asciidoc plugin'
+//
+// You will want to change plugin settings to enable diagrams (they're off by default.)
+//
+// You need to view this page with Chrome or Firefox.
+//
+// ///////////////////////////////////////////////////////////////////////////
+//
+// When editing, please start each sentence on a new line.
+// See https://asciidoctor.org/docs/asciidoc-recommended-practices/#one-sentence-per-line[one sentence-per-line writing technique.]
+// This makes textual diffs of this file useful in a similar way to the way they work for code.
+//
+// //////////////////////////////////////////////////////////////////////////
+
+= DFDL Language Extensions in Daffodil
+
+Daffodil contains numerous extensions to the DFDL v1.0 language.
+
+Many of these have been, or will be proposed for inclusion in a future version of the DFDL standard.
+
+This page provides a central starting point for the documentation of these extension features.
+
+== About Daffodil Plugins
+
+To provide some new advanced format capabilities such as checksums, compressed or encoded data regions, and user-defined-functions, DFDL schemas sometimes must use Daffodil-specific extensions and incorporate Daffodil plugins that provide the small algorithmic aspects needed by these formats.
+
+There are 2 kinds of plugins today supported by Daffodil 3.3.0
+
+- Layering Transformer (e.g., unzip/zip, verify/recompute checksums)
+- User Defined Function (UDF) (e.g., convert mean-sea-level elevation to height-above-ellipsoid)
+
+There is one additional kind of plugin that will be supported by Daffodil 3.4.0
Review Comment:
See https://github.com/apache/daffodil-site/pull/90#discussion_r1020806126
##########
site/extensions.adoc:
##########
@@ -0,0 +1,60 @@
+:page-layout: page
+:url-asciidoctor: http://asciidoctor.org
+:keywords: plugins layering UDF charset
+// ///////////////////////////////////////////////////////////////////////////
+//
+// This file is written in AsciiDoc.
+//
+// If you can read this comment, your browser is not rendering asciidoc automatically.
+//
+// You need to install the asciidoc plugin to Chrome or Firefox
+// so that this page will be properly rendered for your viewing pleasure.
+//
+// You can get the plugins by searching the web for 'asciidoc plugin'
+//
+// You will want to change plugin settings to enable diagrams (they're off by default.)
+//
+// You need to view this page with Chrome or Firefox.
+//
+// ///////////////////////////////////////////////////////////////////////////
+//
+// When editing, please start each sentence on a new line.
+// See https://asciidoctor.org/docs/asciidoc-recommended-practices/#one-sentence-per-line[one sentence-per-line writing technique.]
+// This makes textual diffs of this file useful in a similar way to the way they work for code.
+//
+// //////////////////////////////////////////////////////////////////////////
+
+= DFDL Language Extensions in Daffodil
+
+Daffodil contains numerous extensions to the DFDL v1.0 language.
+
+Many of these have been, or will be proposed for inclusion in a future version of the DFDL standard.
+
+This page provides a central starting point for the documentation of these extension features.
+
+== About Daffodil Plugins
+
+To provide some new advanced format capabilities such as checksums, compressed or encoded data regions, and user-defined-functions, DFDL schemas sometimes must use Daffodil-specific extensions and incorporate Daffodil plugins that provide the small algorithmic aspects needed by these formats.
+
+There are 2 kinds of plugins today supported by Daffodil 3.3.0
+
+- Layering Transformer (e.g., unzip/zip, verify/recompute checksums)
+- User Defined Function (UDF) (e.g., convert mean-sea-level elevation to height-above-ellipsoid)
+
+There is one additional kind of plugin that will be supported by Daffodil 3.4.0
+
+- Character Set Definitions (e.g., a specific 5-bit charset used only by a certain format)
+
+One needs to think of plugins as being part of the DFDL schema of a format, not part of Daffodil.
+
+Different DFDL schemas for different kinds of data will need their own such plugins.
+Hence the plugins, like the DFDL schema files themselves, are used in applications as part of a specific data-processing flow.
+
+Keeping in the spirit of DFDL in describing a format declaratively, plugins need to be very small pieces of code (ex: a character set definition should be 10 lines of code.)
+
+Plugins are compiled from Java/Scala code and would commonly be packaged into a jar file which may or may not also contain the DFDL schema files.
+The loading of the plugin is enabled using a standard Java technique for class loading where a special META-INF file identifies the jar as containing a particular type of plug-in.
+
+Configuring an application must put these jar files on the CLASSPATH so that the executing instance of Daffodil for a specific configured data processing flow finds them on the class path for the data format(s) that flow is processing.
+
+For greater assurance/trust, the plugin jars could be digitally signed by their creators, and applications could verify these signatures (using public keys) as a startup condition.
Review Comment:
See https://github.com/apache/daffodil-site/pull/90#discussion_r889048489
##########
site/packagingSchemas.adoc:
##########
@@ -0,0 +1,84 @@
+:page-layout: page
+:url-asciidoctor: http://asciidoctor.org
+:keywords: schema package jar
+// ///////////////////////////////////////////////////////////////////////////
+//
+// This file is written in AsciiDoc.
+//
+// If you can read this comment, your browser is not rendering asciidoc automatically.
+//
+// You need to install the asciidoc plugin to Chrome or Firefox
+// so that this page will be properly rendered for your viewing pleasure.
+//
+// You can get the plugins by searching the web for 'asciidoc plugin'
+//
+// You will want to change plugin settings to enable diagrams (they're off by default.)
+//
+// You need to view this page with Chrome or Firefox.
+//
+// ///////////////////////////////////////////////////////////////////////////
+//
+// When editing, please start each sentence on a new line.
+// See https://asciidoctor.org/docs/asciidoc-recommended-practices/#one-sentence-per-line[one sentence-per-line writing technique.]
+// This makes textual diffs of this file useful in a similar way to the way they work for code.
+//
+// //////////////////////////////////////////////////////////////////////////
+
+= Packaging DFDL Schemas for use in Daffodil Applications
+
+=== Advance Summary
+
+- The best way to use DFDL schemas is accessing them from Jar files
+- Include pre-compiled binary DFDL schema files also in the same Jar file.
Review Comment:
See https://github.com/apache/daffodil-site/pull/90#discussion_r889052055
##########
site/extensions.adoc:
##########
@@ -0,0 +1,60 @@
+:page-layout: page
+:url-asciidoctor: http://asciidoctor.org
+:keywords: plugins layering UDF charset
+// ///////////////////////////////////////////////////////////////////////////
+//
+// This file is written in AsciiDoc.
+//
+// If you can read this comment, your browser is not rendering asciidoc automatically.
+//
+// You need to install the asciidoc plugin to Chrome or Firefox
+// so that this page will be properly rendered for your viewing pleasure.
+//
+// You can get the plugins by searching the web for 'asciidoc plugin'
+//
+// You will want to change plugin settings to enable diagrams (they're off by default.)
+//
+// You need to view this page with Chrome or Firefox.
+//
+// ///////////////////////////////////////////////////////////////////////////
+//
+// When editing, please start each sentence on a new line.
+// See https://asciidoctor.org/docs/asciidoc-recommended-practices/#one-sentence-per-line[one sentence-per-line writing technique.]
+// This makes textual diffs of this file useful in a similar way to the way they work for code.
+//
+// //////////////////////////////////////////////////////////////////////////
+
+= DFDL Language Extensions in Daffodil
+
+Daffodil contains numerous extensions to the DFDL v1.0 language.
+
+Many of these have been, or will be proposed for inclusion in a future version of the DFDL standard.
+
+This page provides a central starting point for the documentation of these extension features.
+
+== About Daffodil Plugins
+
+To provide some new advanced format capabilities such as checksums, compressed or encoded data regions, and user-defined-functions, DFDL schemas sometimes must use Daffodil-specific extensions and incorporate Daffodil plugins that provide the small algorithmic aspects needed by these formats.
+
+There are 2 kinds of plugins today supported by Daffodil 3.3.0
+
+- Layering Transformer (e.g., unzip/zip, verify/recompute checksums)
+- User Defined Function (UDF) (e.g., convert mean-sea-level elevation to height-above-ellipsoid)
+
+There is one additional kind of plugin that will be supported by Daffodil 3.4.0
+
+- Character Set Definitions (e.g., a specific 5-bit charset used only by a certain format)
+
+One needs to think of plugins as being part of the DFDL schema of a format, not part of Daffodil.
+
+Different DFDL schemas for different kinds of data will need their own such plugins.
+Hence the plugins, like the DFDL schema files themselves, are used in applications as part of a specific data-processing flow.
+
+Keeping in the spirit of DFDL in describing a format declaratively, plugins need to be very small pieces of code (ex: a character set definition should be 10 lines of code.)
Review Comment:
See https://github.com/apache/daffodil-site/pull/90#discussion_r889749658
##########
site/extensions.adoc:
##########
@@ -0,0 +1,60 @@
+:page-layout: page
+:url-asciidoctor: http://asciidoctor.org
+:keywords: plugins layering UDF charset
+// ///////////////////////////////////////////////////////////////////////////
+//
+// This file is written in AsciiDoc.
+//
Review Comment:
See comment https://github.com/apache/daffodil-site/pull/90#discussion_r889035154
##########
site/extensions.adoc:
##########
@@ -0,0 +1,60 @@
+:page-layout: page
+:url-asciidoctor: http://asciidoctor.org
+:keywords: plugins layering UDF charset
+// ///////////////////////////////////////////////////////////////////////////
+//
+// This file is written in AsciiDoc.
+//
+// If you can read this comment, your browser is not rendering asciidoc automatically.
+//
+// You need to install the asciidoc plugin to Chrome or Firefox
+// so that this page will be properly rendered for your viewing pleasure.
+//
+// You can get the plugins by searching the web for 'asciidoc plugin'
+//
+// You will want to change plugin settings to enable diagrams (they're off by default.)
+//
+// You need to view this page with Chrome or Firefox.
+//
+// ///////////////////////////////////////////////////////////////////////////
+//
+// When editing, please start each sentence on a new line.
+// See https://asciidoctor.org/docs/asciidoc-recommended-practices/#one-sentence-per-line[one sentence-per-line writing technique.]
+// This makes textual diffs of this file useful in a similar way to the way they work for code.
+//
+// //////////////////////////////////////////////////////////////////////////
+
+= DFDL Language Extensions in Daffodil
+
+Daffodil contains numerous extensions to the DFDL v1.0 language.
Review Comment:
See https://github.com/apache/daffodil-site/pull/90#discussion_r889749305
##########
site/packagingSchemas.adoc:
##########
@@ -0,0 +1,84 @@
+:page-layout: page
+:url-asciidoctor: http://asciidoctor.org
+:keywords: schema package jar
+// ///////////////////////////////////////////////////////////////////////////
+//
+// This file is written in AsciiDoc.
+//
+// If you can read this comment, your browser is not rendering asciidoc automatically.
+//
+// You need to install the asciidoc plugin to Chrome or Firefox
+// so that this page will be properly rendered for your viewing pleasure.
+//
+// You can get the plugins by searching the web for 'asciidoc plugin'
+//
+// You will want to change plugin settings to enable diagrams (they're off by default.)
+//
+// You need to view this page with Chrome or Firefox.
+//
+// ///////////////////////////////////////////////////////////////////////////
+//
+// When editing, please start each sentence on a new line.
+// See https://asciidoctor.org/docs/asciidoc-recommended-practices/#one-sentence-per-line[one sentence-per-line writing technique.]
+// This makes textual diffs of this file useful in a similar way to the way they work for code.
+//
+// //////////////////////////////////////////////////////////////////////////
+
+= Packaging DFDL Schemas for use in Daffodil Applications
+
+=== Advance Summary
+
+- The best way to use DFDL schemas is accessing them from Jar files
+- Include pre-compiled binary DFDL schema files also in the same Jar file.
+- Include any Daffodil plugins (class files for compiled scala/java code) required by the DFDL schema also in the same Jar file (with the appropriate META-INF files) and optionally with the source code for the plugins.
+- Create _glue_ DFDL schemas that combine other DFDL schemas using managed dependencies (e.g., maven/sbt) on the Jar files of the dependency DFDL schemas.
+- Managed dependencies can be used to obtain specific versions of DFDL schemas for applications in the same way that applications obtain and depend upon Java libraries.
+- Digital signatures (signed jars) can enhance security by providing trust in the creator of the packaged DFDL schema jar.
+- Standard sbt tools facilitate all of this.
+
+
+=== Introduction to DFDL Schema Packaging
+
+DFDL schemas can be large collections of files.
+There are DFDL schemas with over 100 files spread over numerous directories.
+
+The organization of the files into these directory structures is not arbitrary.
+It can be needed to avoid file name clashes and serves the same role as the Java package-name directory structure does for Java programs.
+The directory hierarchy defines a Java package-like namespace structure for DFDL schemas.
Review Comment:
See https://github.com/apache/daffodil-site/pull/90#discussion_r889129286
##########
site/packagingSchemas.adoc:
##########
@@ -0,0 +1,84 @@
+:page-layout: page
+:url-asciidoctor: http://asciidoctor.org
+:keywords: schema package jar
+// ///////////////////////////////////////////////////////////////////////////
+//
+// This file is written in AsciiDoc.
+//
+// If you can read this comment, your browser is not rendering asciidoc automatically.
+//
+// You need to install the asciidoc plugin to Chrome or Firefox
+// so that this page will be properly rendered for your viewing pleasure.
+//
+// You can get the plugins by searching the web for 'asciidoc plugin'
+//
+// You will want to change plugin settings to enable diagrams (they're off by default.)
+//
+// You need to view this page with Chrome or Firefox.
+//
+// ///////////////////////////////////////////////////////////////////////////
+//
+// When editing, please start each sentence on a new line.
+// See https://asciidoctor.org/docs/asciidoc-recommended-practices/#one-sentence-per-line[one sentence-per-line writing technique.]
+// This makes textual diffs of this file useful in a similar way to the way they work for code.
+//
+// //////////////////////////////////////////////////////////////////////////
+
+= Packaging DFDL Schemas for use in Daffodil Applications
Review Comment:
See https://github.com/apache/daffodil-site/pull/90#discussion_r889130847
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [daffodil-site] tuxji commented on pull request #108: WIP: User doc on packaging and plugins. Drafts
Posted by "tuxji (via GitHub)" <gi...@apache.org>.
tuxji commented on PR #108:
URL: https://github.com/apache/daffodil-site/pull/108#issuecomment-1451137234
If I were to review this PR (I won't since you didn't ask for a review at this time), I would want the long lines in the file to be wrapped to at most 90 columns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org