You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@daffodil.apache.org by mb...@apache.org on 2018/05/03 18:17:07 UTC
[incubator-daffodil] branch master updated: All properties for
Base64/layering implemented.
This is an automated email from the ASF dual-hosted git repository.
mbeckerle pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-daffodil.git
The following commit(s) were added to refs/heads/master by this push:
new 1ea2290 All properties for Base64/layering implemented.
1ea2290 is described below
commit 1ea2290f28e9ede344cb7d8d06f38a7fe818dfc3
Author: Michael Beckerle <mb...@tresys.com>
AuthorDate: Tue Mar 6 08:34:33 2018 -0500
All properties for Base64/layering implemented.
Layers allowed are base64_MIME, gzip, lineFolded_IMF, lineFolded_iCalendar
base64_MIME supports layerLengthKind 'boundaryMark'
gzip supports layerLengthKind 'explcit'
lineFolded_XXX supports either 'boundaryMark' or 'implicit'
Characterized with tests, what is needed in order to implement
stream filters that impose terminators that are scanned for - using
Java's Scanner class.
Java's scanner class saves us the hassle of block filling/enlarging,
etc. that is otherwise necessary when you want to use the regex
libraries against a stream.
Also put catch for ClassNotFound around compiler.reload for Java 7 vs.
Java 8. Note: Base64 needs Java 8 classes. (there may be alternative
implementation that would work with Java 7.)
Identified a couple cleanups where java.io.HashMap can be replaced with
our NonAllocatingMap - which uses Maybe types instead of null/notNull.
Just code cleanups.
For future use in AIS layer: Added a charset so we can use the encoder
to unparse AIS data that must be armored with AIS_PAYLOAD_ARMORING
which is a 6 bit to 8 bit.
This is all part of layering, i.e., AIS format requires layering as does
base64, so merging this all onto same branch.
Combined tests that have base64 encoded regions that contain gzip data
having a computed length field also with the base64 region.... a test of
that works.
Verified that the nato-stanag-5516 schema still works with this build,
as well as VMF. Those are the most bit-order intensive schemas we have.
Changed terminology uniformly from terminator to boundaryMark for
layers.
Refactored BoundaryMarkLimitingStream to pull out a
RegexLimitingStream. Needed because line folded data is terminated by a
CRLF followed by either a tab or a space. Can't express this as a
boundary mark string.
DAFFODIL-1734, DAFFODIL-1805
---
.../org/apache/daffodil/compiler/Compiler.scala | 65 +--
.../daffodil/dsom/DFDLFormatAnnotation.scala | 2 -
.../RawCommonRuntimeValuedPropertiesMixin.scala | 13 +
.../daffodil/dsom/RuntimePropertyMixins.scala | 94 ++++
.../org/apache/daffodil/dsom/SequenceGroup.scala | 30 +-
.../daffodil/grammar/ModelGroupGrammarMixin.scala | 2 +-
.../daffodil/grammar/SequenceGrammarMixin.scala | 24 +-
.../grammar/primitives/LayeredSequence.scala | 43 ++
.../org/apache/daffodil/layers/TestLayers.scala | 361 ++++++++++++++++
.../scala/org/apache/daffodil/util/TestUtils.scala | 25 +-
.../daffodil/io/ByteBufferDataInputStream.scala | 17 +-
.../org/apache/daffodil/io/DataOutputStream.scala | 2 +-
.../daffodil/io/DataOutputStreamImplMixin.scala | 3 +-
.../io/DirectOrBufferedDataOutputStream.scala | 28 +-
.../daffodil/io/LimitingJavaIinputStreams.scala | 287 +++++++++++++
.../LimitingJavaOutputStreams.scala} | 36 +-
.../processors/charset/AIS_PAYLOAD_ARMORING.scala | 46 ++
.../daffodil/processors/charset/CharsetUtils.scala | 4 +
.../charset/DaffodilCharsetProvider.scala | 5 +-
.../charset/USASCII6BitPackedDecoder.scala | 19 +-
.../io/TestAISPayloadArmoringEncoder.scala | 98 +++++
.../org/apache/daffodil/layers/TestBase64.scala | 245 +++++++++++
.../apache/daffodil/layers/TestJavaIOStreams.scala | 184 ++++++++
.../layers/TestLimitingJavaIOStreams.scala | 204 +++++++++
.../org/apache/daffodil/cookers/Cookers.scala | 2 +
.../scala/org/apache/daffodil/xml/XMLUtils.scala | 18 +-
.../apache/daffodil/xsd/DFDL_part1_simpletypes.xsd | 131 ++++++
.../apache/daffodil/xsd/DFDL_part2_attributes.xsd | 46 ++
.../org/apache/daffodil/xsd/DFDL_part3_model.xsd | 2 +
.../unparsers/ElementKindUnparsers.scala | 2 +-
.../unparsers/LayeredSequenceUnparser.scala | 63 +++
.../org/apache/daffodil/infoset/InfosetImpl.scala | 2 +
.../daffodil/infoset/ScalaXMLInfosetInputter.scala | 21 +-
.../apache/daffodil/layers/Base64Transformer.scala | 98 +++++
.../apache/daffodil/layers/GZipTransformer.scala | 70 +++
.../apache/daffodil/layers/LayerTransformer.scala | 255 +++++++++++
.../daffodil/layers/LineFoldedTransformer.scala | 474 +++++++++++++++++++++
.../apache/daffodil/processors/EvEncoding.scala | 10 +-
.../apache/daffodil/processors/EvLayering.scala | 101 +++++
.../daffodil/processors/ProcessorStateBases.scala | 22 +-
.../org/apache/daffodil/processors/Runtime.scala | 11 +
.../processors/parsers/ElementKindParsers.scala | 3 +-
.../processors/parsers/LayeredSequenceParser.scala | 55 +++
.../TestLengthLimitedLineFoldingStreams.scala | 216 ++++++++++
.../daffodil/layers/TestLineFoldingStreams.scala | 231 ++++++++++
.../org/apache/daffodil/layers/layers.tdml | 257 +++++++++++
.../org/apache/daffodil/layers/TestLayers.scala | 42 +-
.../TestExternalVariablesNew.scala | 2 +-
48 files changed, 3854 insertions(+), 117 deletions(-)
diff --git a/daffodil-core/src/main/scala/org/apache/daffodil/compiler/Compiler.scala b/daffodil-core/src/main/scala/org/apache/daffodil/compiler/Compiler.scala
index bc018ba..81f1b91 100644
--- a/daffodil-core/src/main/scala/org/apache/daffodil/compiler/Compiler.scala
+++ b/daffodil-core/src/main/scala/org/apache/daffodil/compiler/Compiler.scala
@@ -17,44 +17,25 @@
package org.apache.daffodil.compiler
-import java.io.File
-import java.io.FileInputStream
-import java.io.ObjectInputStream
+import java.io.{ File, FileInputStream, ObjectInputStream, StreamCorruptedException }
import java.nio.channels.Channels
+import java.util.zip.{ GZIPInputStream, ZipException }
+
+import scala.collection.mutable.Queue
import scala.xml.Node
+
import org.apache.daffodil.ExecutionMode
-import org.apache.daffodil.api.DFDL
-import org.apache.daffodil.dsom.SchemaSet
-import org.apache.daffodil.oolag.OOLAG
+import org.apache.daffodil.api.{ DFDL, DaffodilSchemaSource, DaffodilTunables, URISchemaSource, UnitTestSchemaSource, ValidationMode }
+import org.apache.daffodil.dsom.{ ElementBase, SchemaComponent, SchemaComponentImpl, SchemaSet }
import org.apache.daffodil.exceptions.Assert
-import org.apache.daffodil.processors.DataProcessor
-import org.apache.daffodil.util.LogLevel
-import org.apache.daffodil.util.Logging
-import org.apache.daffodil.xml._
-import org.apache.daffodil.api.DFDL
-import org.apache.daffodil.api.DaffodilSchemaSource
-import org.apache.daffodil.api.UnitTestSchemaSource
-import org.apache.daffodil.externalvars.Binding
-import scala.collection.mutable.Queue
-import org.apache.daffodil.externalvars.ExternalVariablesLoader
-import org.apache.daffodil.processors.SchemaSetRuntimeData
-import org.apache.daffodil.util.CheckJavaVersion
-import org.apache.daffodil.util.InvalidJavaVersionException
-import org.apache.daffodil.api.ValidationMode
-import org.apache.daffodil.processors.VariableMap
-import java.util.zip.GZIPInputStream
-import java.util.zip.ZipException
-import java.io.StreamCorruptedException
-import org.apache.daffodil.dsom.ElementBase
-import org.apache.daffodil.api.URISchemaSource
-import org.apache.daffodil.processors.SerializableDataProcessor
-import org.apache.daffodil.processors.Processor
+import org.apache.daffodil.externalvars.{ Binding, ExternalVariablesLoader }
+import org.apache.daffodil.oolag.OOLAG
+import org.apache.daffodil.processors.{ DataProcessor, Processor, SchemaSetRuntimeData, SerializableDataProcessor, VariableMap }
import org.apache.daffodil.processors.parsers.NotParsableParser
import org.apache.daffodil.processors.unparsers.NotUnparsableUnparser
import org.apache.daffodil.schema.annotation.props.gen.ParseUnparsePolicy
-import org.apache.daffodil.dsom.SchemaComponent
-import org.apache.daffodil.dsom.SchemaComponentImpl
-import org.apache.daffodil.api.DaffodilTunables
+import org.apache.daffodil.util.{ CheckJavaVersion, InvalidJavaVersionException, LogLevel, Logging, Misc }
+import org.apache.daffodil.xml._
/**
* Some grammar rules need to be conditional based on whether we're trying
@@ -308,6 +289,28 @@ class Compiler(var validateDFDLSchemas: Boolean = true)
case ex: StreamCorruptedException => {
throw new InvalidParserException("The saved parser file is not a valid parser.", ex)
}
+ //
+ // If we are running on Java 7, and a class such as Base64 (only in Java 8)
+ // needs to be created as part of loading the schema, then we'll get a
+ // class not found exception. This catches that and issues a
+ // sensible diagnostic.
+ //
+ // Similarly, if a class *should* be on the classpath in order for this
+ // schema to reload, then we will get CNF, and we issue a diagnostic
+ // which also displays the classpath.
+ //
+ case cnf: ClassNotFoundException => {
+ val cpString =
+ if (Misc.classPath.length == 0) " empty."
+ else ":\n" + Misc.classPath.mkString("\n\t")
+ val msg = "%s\nThe class may not exist in this Java JVM version (%s)," +
+ "or it is missing from the classpath which is%s".format(
+ cnf.getMessage(),
+ scala.util.Properties.javaVersion,
+ cpString
+ )
+ throw new InvalidParserException(msg, cnf)
+ }
}
}
/**
diff --git a/daffodil-core/src/main/scala/org/apache/daffodil/dsom/DFDLFormatAnnotation.scala b/daffodil-core/src/main/scala/org/apache/daffodil/dsom/DFDLFormatAnnotation.scala
index fc105cf..679fcfb 100644
--- a/daffodil-core/src/main/scala/org/apache/daffodil/dsom/DFDLFormatAnnotation.scala
+++ b/daffodil-core/src/main/scala/org/apache/daffodil/dsom/DFDLFormatAnnotation.scala
@@ -31,8 +31,6 @@ import org.apache.daffodil.schema.annotation.props.LookupLocation
*/
abstract class DFDLFormatAnnotation(nodeArg: Node, annotatedSCArg: AnnotatedSchemaComponent)
extends DFDLAnnotation(nodeArg, annotatedSCArg)
- // with RawCommonRuntimeValuedPropertiesMixin
- // with RawEscapeSchemeRuntimeValuedPropertiesMixin
with LeafPropProvider {
requiredEvaluations(hasConflictingPropertyError)
diff --git a/daffodil-core/src/main/scala/org/apache/daffodil/dsom/RawCommonRuntimeValuedPropertiesMixin.scala b/daffodil-core/src/main/scala/org/apache/daffodil/dsom/RawCommonRuntimeValuedPropertiesMixin.scala
index 82f09b4..8caa35c 100644
--- a/daffodil-core/src/main/scala/org/apache/daffodil/dsom/RawCommonRuntimeValuedPropertiesMixin.scala
+++ b/daffodil-core/src/main/scala/org/apache/daffodil/dsom/RawCommonRuntimeValuedPropertiesMixin.scala
@@ -60,6 +60,19 @@ trait RawSequenceRuntimeValuedPropertiesMixin
protected final lazy val separatorRaw = requireProperty(optionSeparatorRaw)
}
+trait RawLayeringRuntimeValuedPropertiesMixin
+ extends PropertyMixin {
+ protected final lazy val optionLayerTransformRaw = findPropertyOption("layerTransform")
+ protected final lazy val layerTransformRaw = requireProperty(optionLayerTransformRaw)
+ protected final lazy val optionLayerEncodingRaw = findPropertyOption("layerEncoding")
+ protected final lazy val layerEncodingRaw = requireProperty(optionLayerEncodingRaw)
+ protected final lazy val optionLayerLengthRaw = findPropertyOption("layerLength")
+ protected final lazy val layerLengthRaw = requireProperty(optionLayerLengthRaw)
+ protected final lazy val optionLayerBoundaryMarkRaw = findPropertyOption("layerBoundaryMark")
+ protected final lazy val layerBoundaryMarkRaw = requireProperty(optionLayerBoundaryMarkRaw)
+
+}
+
trait RawEscapeSchemeRuntimeValuedPropertiesMixin
extends PropertyMixin {
diff --git a/daffodil-core/src/main/scala/org/apache/daffodil/dsom/RuntimePropertyMixins.scala b/daffodil-core/src/main/scala/org/apache/daffodil/dsom/RuntimePropertyMixins.scala
index c878db6..5fcc7a2 100644
--- a/daffodil-core/src/main/scala/org/apache/daffodil/dsom/RuntimePropertyMixins.scala
+++ b/daffodil-core/src/main/scala/org/apache/daffodil/dsom/RuntimePropertyMixins.scala
@@ -63,6 +63,11 @@ import org.apache.daffodil.schema.annotation.props.gen.NilKind
import org.apache.daffodil.schema.annotation.props.gen.TextTrimKind
import org.apache.daffodil.schema.annotation.props.gen.TextPadKind
import org.apache.daffodil.schema.annotation.props.gen.YesNo
+import org.apache.daffodil.processors.LayerTransformEv
+import org.apache.daffodil.processors.LayerEncodingEv
+import org.apache.daffodil.processors.LayerLengthInBytesEv
+import org.apache.daffodil.processors.LayerBoundaryMarkEv
+import org.apache.daffodil.processors.LayerCharsetEv
/*
* These are the DFDL properties which can have their values come
@@ -560,6 +565,95 @@ trait SequenceRuntimeValuedPropertiesMixin
}
+trait LayeringRuntimeValuedPropertiesMixin
+ extends RawLayeringRuntimeValuedPropertiesMixin { decl: SequenceTermBase =>
+
+ private lazy val layerTransformExpr = {
+ val qn = this.qNameForProperty("layerTransform")
+ ExpressionCompilers.String.compileProperty(qn, NodeInfo.NonEmptyString, layerTransformRaw, decl)
+ }
+
+ // final lazy val layerTransformEv = {
+ // if (maybeLayerTransformEv.isEmpty) layerTransformRaw // must be defined
+ // maybeLayerTransformEv.get
+ // }
+
+ final lazy val maybeLayerTransformEv = {
+ if (optionLayerTransformRaw.isDefined) {
+ val ev = new LayerTransformEv(layerTransformExpr, termRuntimeData)
+ ev.compile()
+ One(ev)
+ } else {
+ Nope
+ }
+ }
+
+ private lazy val layerEncodingExpr = {
+ val qn = this.qNameForProperty("layerEncoding")
+ ExpressionCompilers.String.compileProperty(qn, NodeInfo.NonEmptyString, layerEncodingRaw, decl)
+ }
+
+ private final lazy val layerEncodingEv = {
+ if (maybeLayerEncodingEv.isEmpty) layerEncodingRaw // must be defined
+ maybeLayerEncodingEv.get
+ }
+
+ private final lazy val maybeLayerEncodingEv = {
+ if (optionLayerEncodingRaw.isDefined) {
+ val ev = new LayerEncodingEv(layerEncodingExpr, termRuntimeData)
+ ev.compile()
+ One(ev)
+ } else {
+ Nope
+ }
+ }
+
+ final lazy val maybeLayerCharsetEv =
+ if (optionLayerEncodingRaw.isDefined) {
+ val ev = new LayerCharsetEv(layerEncodingEv, termRuntimeData)
+ ev.compile()
+ One(ev)
+ } else
+ Nope
+
+ private lazy val layerLengthExpr = {
+ val qn = this.qNameForProperty("layerLength")
+ ExpressionCompilers.JLong.compileProperty(qn, NodeInfo.Long, layerLengthRaw, decl)
+ }
+
+ final lazy val maybeLayerLengthInBytesEv = {
+ if (optionLayerLengthRaw.isDefined) {
+ layerLengthUnits
+ val ev = new LayerLengthInBytesEv(layerLengthExpr, termRuntimeData)
+ ev.compile()
+ One(ev)
+ } else {
+ Nope
+ }
+ }
+
+ private lazy val layerBoundaryMarkExpr = {
+ val qn = this.qNameForProperty("layerBoundaryMark")
+ ExpressionCompilers.String.compileProperty(qn, NodeInfo.String, layerBoundaryMarkRaw, decl)
+ }
+
+ // final lazy val layerBoundaryMarkEv = {
+ // if (maybeLayerBoundaryMarkEv.isEmpty) layerBoundaryMarkRaw // must be defined
+ // maybeLayerBoundaryMarkEv.get
+ // }
+
+ final lazy val maybeLayerBoundaryMarkEv = {
+ if (optionLayerBoundaryMarkRaw.isDefined) {
+ val ev = new LayerBoundaryMarkEv(layerBoundaryMarkExpr, termRuntimeData)
+ ev.compile()
+ One(ev)
+ } else {
+ Nope
+ }
+ }
+
+}
+
trait SimpleTypeRuntimeValuedPropertiesMixin
extends DFDLSimpleTypeMixin
with RawSimpleTypeRuntimeValuedPropertiesMixin { decl: ElementBase =>
diff --git a/daffodil-core/src/main/scala/org/apache/daffodil/dsom/SequenceGroup.scala b/daffodil-core/src/main/scala/org/apache/daffodil/dsom/SequenceGroup.scala
index 0b2947b..e5f6a59 100644
--- a/daffodil-core/src/main/scala/org/apache/daffodil/dsom/SequenceGroup.scala
+++ b/daffodil-core/src/main/scala/org/apache/daffodil/dsom/SequenceGroup.scala
@@ -35,6 +35,8 @@ import org.apache.daffodil.exceptions.Assert
import org.apache.daffodil.processors.SequenceRuntimeData
import org.apache.daffodil.schema.annotation.props.Found
import org.apache.daffodil.schema.annotation.props.PropertyLookupResult
+import org.apache.daffodil.processors.LayerTransformerEv
+import org.apache.daffodil.util.Maybe
abstract class SequenceTermBase(
final override val xml: Node,
@@ -44,7 +46,8 @@ abstract class SequenceTermBase(
with Sequence_AnnotationMixin
with SequenceRuntimeValuedPropertiesMixin
with SequenceGrammarMixin
- with SeparatorSuppressionPolicyMixin {
+ with SeparatorSuppressionPolicyMixin
+ with LayeringRuntimeValuedPropertiesMixin {
requiredEvaluations(checkIfValidUnorderedSequence)
requiredEvaluations(modelGroupRuntimeData.preSerialization)
@@ -245,6 +248,31 @@ abstract class SequenceTermBase(
maybeCheckBitOrderAndCharset)
}
+ private val layeredSequenceAllowedProps = Set("ref", "layerTransform", "layerEncoding", "layerLengthKind", "layerLength", "layerLengthUnits", "layerBoundaryMark")
+
+ final lazy val maybeLayerTransformerEv: Maybe[LayerTransformerEv] = {
+ if (maybeLayerTransformEv.isEmpty) Maybe.Nope
+ else { // need to check that only layering properties are specified
+ val localProps = this.formatAnnotation.justThisOneProperties
+ val localKeys = localProps.keySet
+ val disallowedKeys = localKeys.filterNot(k => layeredSequenceAllowedProps.contains(k))
+ if (disallowedKeys.size > 0)
+ SDE("Sequence has dfdl:layerTransform specified, so cannot have non-layering properties: %s", disallowedKeys.mkString(", "))
+
+ val lt = new LayerTransformerEv(maybeLayerTransformEv.get,
+ maybeLayerCharsetEv,
+ Maybe.toMaybe(optionLayerLengthKind),
+ maybeLayerLengthInBytesEv,
+ Maybe.toMaybe(optionLayerLengthUnits),
+ maybeLayerBoundaryMarkEv,
+ termRuntimeData)
+ lt.compile()
+ Maybe.One(lt)
+ }
+ }
+
+ final def isLayered = maybeLayerTransformerEv.isDefined
+
}
/**
diff --git a/daffodil-core/src/main/scala/org/apache/daffodil/grammar/ModelGroupGrammarMixin.scala b/daffodil-core/src/main/scala/org/apache/daffodil/grammar/ModelGroupGrammarMixin.scala
index 335a833..bcbbc86 100644
--- a/daffodil-core/src/main/scala/org/apache/daffodil/grammar/ModelGroupGrammarMixin.scala
+++ b/daffodil-core/src/main/scala/org/apache/daffodil/grammar/ModelGroupGrammarMixin.scala
@@ -52,7 +52,7 @@ trait ModelGroupGrammarMixin
val finalContent =
if (hasDelimiters ||
enclosingTerm.map(_.hasDelimiters).getOrElse(false) //
- // The above refernce to the delimiters of the enclosing term,
+ // The above reference to the delimiters of the enclosing term,
// has to do with the way our delim stack works.
// Even if this model group doesn't have delimiters,
// if the enclosing term did have delimiters, then we still need to
diff --git a/daffodil-core/src/main/scala/org/apache/daffodil/grammar/SequenceGrammarMixin.scala b/daffodil-core/src/main/scala/org/apache/daffodil/grammar/SequenceGrammarMixin.scala
index 41d57af..724b29c 100644
--- a/daffodil-core/src/main/scala/org/apache/daffodil/grammar/SequenceGrammarMixin.scala
+++ b/daffodil-core/src/main/scala/org/apache/daffodil/grammar/SequenceGrammarMixin.scala
@@ -19,25 +19,33 @@ package org.apache.daffodil.grammar
import org.apache.daffodil.schema.annotation.props.gen._
import org.apache.daffodil.grammar.primitives.SequenceCombinator
import org.apache.daffodil.dsom.SequenceTermBase
+import org.apache.daffodil.grammar.primitives.LayeredSequence
trait SequenceGrammarMixin extends GrammarMixin { self: SequenceTermBase =>
final override lazy val groupContent = prod("groupContent") {
- self.sequenceKind match {
- case SequenceKind.Ordered => orderedSequenceContent
- case SequenceKind.Unordered => subsetError("Unordered sequences are not supported.") // unorderedSequenceContent
+ if (isLayered) layeredSequenceContent
+ else {
+ self.sequenceKind match {
+ case SequenceKind.Ordered => orderedSequenceContent
+ case SequenceKind.Unordered => subsetError("Unordered sequences are not supported.") // unorderedSequenceContent
+ }
}
}
+ private lazy val layeredSequenceContent = {
+ schemaDefinitionUnless(groupMembers.length == 1, "Layered sequence can have only 1 child term. %s were found: %s", groupMembers.length,
+ groupMembers.mkString(", "))
+ val term = groupMembers(0)
+ schemaDefinitionWhen(term.isArray, "Layered sequence body cannot be an array.")
+ val termGram = term.termContentBody
+ LayeredSequence(this, termGram)
+ }
+
private lazy val orderedSequenceContent = prod("sequenceContent") {
SequenceCombinator(this, terms)
}
- // private lazy val unorderedSequenceContent = prod("unorderedSequenceContent") {
- // val uoseq = self.unorderedSeq.get
- // UnorderedSequenceCombinator(this, uoseq.terms)
- // }
-
protected lazy val terms = groupMembers.map { _.asTermInSequence }
/**
diff --git a/daffodil-core/src/main/scala/org/apache/daffodil/grammar/primitives/LayeredSequence.scala b/daffodil-core/src/main/scala/org/apache/daffodil/grammar/primitives/LayeredSequence.scala
new file mode 100644
index 0000000..0e355b1
--- /dev/null
+++ b/daffodil-core/src/main/scala/org/apache/daffodil/grammar/primitives/LayeredSequence.scala
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.grammar.primitives
+
+import org.apache.daffodil.grammar.Terminal
+import org.apache.daffodil.dsom._
+import org.apache.daffodil.processors.parsers.{ Parser => DaffodilParser }
+import org.apache.daffodil.processors.unparsers.{ Unparser => DaffodilUnparser }
+import org.apache.daffodil.grammar.Gram
+import org.apache.daffodil.util.Misc
+import org.apache.daffodil.processors.parsers.LayeredSequenceParser
+import org.apache.daffodil.processors.unparsers.LayeredSequenceUnparser
+
+case class LayeredSequence(sq: SequenceTermBase, bodyTerm: Gram)
+ extends Terminal(sq, true) {
+
+ override def toString() =
+ "<" + Misc.getNameFromClass(this) + ">" +
+ bodyTerm.toString() +
+ "</" + Misc.getNameFromClass(this) + ">"
+
+ override lazy val parser: DaffodilParser =
+ new LayeredSequenceParser(sq.termRuntimeData, sq.maybeLayerTransformerEv.get, bodyTerm.parser)
+
+ override lazy val unparser: DaffodilUnparser = {
+ new LayeredSequenceUnparser(sq.modelGroupRuntimeData, sq.maybeLayerTransformerEv.get, bodyTerm.unparser)
+ }
+}
diff --git a/daffodil-core/src/test/scala/org/apache/daffodil/layers/TestLayers.scala b/daffodil-core/src/test/scala/org/apache/daffodil/layers/TestLayers.scala
new file mode 100644
index 0000000..1dc7224
--- /dev/null
+++ b/daffodil-core/src/test/scala/org/apache/daffodil/layers/TestLayers.scala
@@ -0,0 +1,361 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.layers
+
+import org.apache.daffodil.util._
+import org.junit.Test
+import org.junit.Assert._
+import org.apache.daffodil.util.TestUtils
+import java.io.ByteArrayOutputStream
+import org.apache.commons.io.IOUtils
+import java.nio.charset.StandardCharsets
+import java.io.ByteArrayInputStream
+import java.io.InputStreamReader
+import scala.collection.JavaConversions._
+import org.apache.daffodil.xml.XMLUtils
+
+class TestLayers {
+
+ val example = XMLUtils.EXAMPLE_NAMESPACE
+
+ val B64Layer1Schema =
+ SchemaUtils.dfdlTestSchema(
+ <dfdl:format ref="tns:GeneralFormat"/>,
+ <xs:element name="e1" dfdl:lengthKind="implicit">
+ <xs:complexType>
+ <xs:sequence dfdl:layerTransform="base64_MIME" dfdl:layerLengthKind="boundaryMark" dfdl:layerBoundaryMark="!" dfdl:layerEncoding="iso-8859-1">
+ <xs:element name="s1" type="xs:string" dfdl:lengthKind="explicit" dfdl:length="3"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>, elementFormDefault = "unqualified")
+
+ @Test def testParseB64Layer1() {
+ val sch = B64Layer1Schema
+ val data = "cGxl!" // encoding of "ple" + "!"
+ val infoset = <ex:e1 xmlns:ex={ example }><s1>ple</s1></ex:e1>
+ val (_, actual) = TestUtils.testString(sch, data)
+ TestUtils.assertEqualsXMLElements(infoset, actual)
+
+ val areTracing = false
+ TestUtils.testUnparsing(sch, infoset, data, areTracing)
+ }
+
+ val B64Layer2Schema =
+ SchemaUtils.dfdlTestSchema(
+ <dfdl:format ref="tns:GeneralFormat" lengthKind='delimited'/>,
+ <xs:element name="e1" dfdl:lengthKind="implicit">
+ <xs:complexType>
+ <xs:sequence dfdl:layerTransform="base64_MIME" dfdl:layerLengthKind="boundaryMark" dfdl:layerBoundaryMark="!" dfdl:layerEncoding="iso-8859-1">
+ <xs:element name="s1" type="xs:string"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>, elementFormDefault = "unqualified")
+
+ @Test def testParseB64Layer2() {
+ val sch = B64Layer2Schema
+ val data = "cGxl!" // encoding of "ple" + "!"
+ val infoset = <ex:e1 xmlns:ex={ example }><s1>ple</s1></ex:e1>
+ val (_, actual) = TestUtils.testString(sch, data)
+ TestUtils.assertEqualsXMLElements(infoset, actual)
+
+ val areTracing = false
+ TestUtils.testUnparsing(sch, infoset, data, areTracing)
+ }
+
+ val B64Layer3Schema =
+ SchemaUtils.dfdlTestSchema(
+ <dfdl:format ref="tns:GeneralFormat" lengthKind='delimited'/>,
+ <xs:element name="e1" dfdl:lengthKind="implicit">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:sequence dfdl:layerTransform="base64_MIME" dfdl:layerLengthKind="boundaryMark" dfdl:layerBoundaryMark="!" dfdl:layerEncoding="iso-8859-1">
+ <xs:element name="s1" type="xs:string"/>
+ </xs:sequence>
+ <xs:element name="s2" type="xs:string"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>, elementFormDefault = "unqualified")
+
+ @Test def testParseB64Layer3() {
+ val sch = B64Layer3Schema
+ val data = "cGxl" + "!" + "moreDataAfter"
+ val infoset = <ex:e1 xmlns:ex={ example }><s1>ple</s1><s2>moreDataAfter</s2></ex:e1>
+ val (_, actual) = TestUtils.testString(sch, data)
+ TestUtils.assertEqualsXMLElements(infoset, actual)
+
+ val areTracing = false
+ TestUtils.testUnparsing(sch, infoset, data, areTracing)
+ }
+
+ def makeGZIPData(text: String) = {
+ val baos = new ByteArrayOutputStream()
+ val gzos = new java.util.zip.GZIPOutputStream(baos)
+ IOUtils.write(text, gzos, StandardCharsets.UTF_8)
+ gzos.close()
+ val data = baos.toByteArray()
+ data
+ }
+
+ val text = """This is just some made up text that is intended to be
+a few lines long. If this had been real text, it would not have been quite
+so boring to read. Use of famous quotes or song lyrics or anything like that
+introduces copyright notice issues, so it is easier to simply make up
+a few lines of pointless text like this.""".replace("\n", " ")
+
+ @Test def testGZIPRoundTrips() {
+ val bais = new ByteArrayInputStream(makeGZIPData(text))
+ val gzis = new java.util.zip.GZIPInputStream(bais)
+ val rdr = new InputStreamReader(gzis, StandardCharsets.UTF_8)
+ val lines = IOUtils.readLines(rdr)
+ val textBack = lines.head
+ assertEquals(text, textBack)
+ }
+
+ val GZIPLayer1Schema =
+ SchemaUtils.dfdlTestSchema(
+ <dfdl:format ref="tns:GeneralFormat" layerLengthUnits="bytes" representation="binary"/>,
+ <xs:element name="e1" dfdl:lengthKind="implicit">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="len" type="xs:int" dfdl:lengthKind="explicit" dfdl:length="4" dfdl:outputValueCalc="{ dfdl:contentLength(../x1, 'bytes') }"/>
+ <xs:element name="x1" dfdl:lengthKind="implicit">
+ <xs:complexType>
+ <xs:sequence dfdl:layerTransform="gzip" dfdl:layerLengthKind="explicit" dfdl:layerLength="{ ../len }">
+ <xs:element name="s1" type="xs:string" dfdl:lengthKind="delimited"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ <xs:element name="s2" type="xs:string" dfdl:lengthKind="delimited"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>,
+ elementFormDefault = "unqualified")
+
+ def makeGZIPLayer1Data() = {
+ val gzipData = makeGZIPData(text)
+ val dataLength = gzipData.length
+ val baos = new ByteArrayOutputStream()
+ val dos = new java.io.DataOutputStream(baos)
+ dos.writeInt(dataLength)
+ dos.write(gzipData)
+ dos.write("afterGzip".getBytes(StandardCharsets.UTF_8))
+ dos.close()
+ val data = baos.toByteArray()
+ (data, dataLength)
+ }
+
+ @Test def testGZIPLayer1() {
+ val sch = GZIPLayer1Schema
+ val (data, dataLength) = makeGZIPLayer1Data()
+ val infoset = <ex:e1 xmlns:ex={ example }><len>{ dataLength }</len><x1><s1>{ text }</s1></x1><s2>afterGzip</s2></ex:e1>
+ val (_, actual) = TestUtils.testBinary(sch, data, areTracing = false)
+ TestUtils.assertEqualsXMLElements(infoset, actual)
+
+ TestUtils.testUnparsingBinary(sch, infoset, data)
+ }
+
+ def makeB64GZIPSchema(term: String, layerTerm: String) = SchemaUtils.dfdlTestSchema(
+ <dfdl:format ref="tns:GeneralFormat" layerLengthUnits="bytes" representation="binary" layerEncoding="iso-8859-1"/>,
+ <xs:element name="e1" dfdl:lengthKind="implicit">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="s1" type="xs:string" dfdl:lengthKind="delimited" dfdl:terminator={ term }/>
+ <xs:sequence dfdl:layerTransform="base64_MIME" dfdl:layerLengthKind="boundaryMark" dfdl:layerBoundaryMark={ layerTerm }>
+ <xs:sequence>
+ <xs:element name="len" type="xs:int" dfdl:outputValueCalc="{ dfdl:contentLength(../x1, 'bytes') }"/>
+ <xs:element name="x1" dfdl:lengthKind="implicit">
+ <xs:complexType>
+ <xs:sequence dfdl:layerTransform="gzip" dfdl:layerLengthKind="explicit" dfdl:layerLength="{ ../len }">
+ <xs:element name="s2" type="xs:string" dfdl:lengthKind="delimited"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ </xs:sequence>
+ <xs:element name="s3" type="xs:string" dfdl:lengthKind="delimited"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>,
+ elementFormDefault = "unqualified")
+
+ def toB64(bytes: Array[Byte]) =
+ java.util.Base64.getMimeEncoder.encodeToString(bytes)
+
+ def makeB64GZIPData(term: String, layerTerm: String, before: String, after: String, text: String) = {
+ val gzipData = makeGZIPData(text)
+ val dataLength = gzipData.length
+ val baos = new ByteArrayOutputStream()
+ val dos = new java.io.DataOutputStream(baos)
+ dos.writeInt(dataLength) // 4 byte length of gzipped data
+ dos.write(gzipData)
+ dos.close()
+ val gzBytes = baos.toByteArray()
+ val b64Text = toB64(gzBytes) // encoded as base6
+ val baos2 = new ByteArrayOutputStream()
+ val dos2 = new java.io.DataOutputStream(baos2)
+
+ dos2.write(before.getBytes(StandardCharsets.UTF_8))
+ dos2.write(term.getBytes(StandardCharsets.UTF_8))
+ dos2.write(b64Text.getBytes("ascii")) // b64 text is always ascii.
+ dos2.write(layerTerm.getBytes("ascii"))
+ dos2.write(after.getBytes(StandardCharsets.UTF_8))
+ dos2.close()
+ (baos2.toByteArray(), dataLength)
+ }
+
+ @Test def testParseB64GZIPLayer1() {
+ val term = ";"
+ val layerTerm = "=_END_="
+ val sch = makeB64GZIPSchema(term, layerTerm)
+ val before = "beforeB64GZip"
+ val after = "afterB64GZip"
+ val (data, dataLength) = makeB64GZIPData(term, layerTerm, before, after, text)
+ val (_, actual) = TestUtils.testBinary(sch, data, areTracing = false)
+ val infoset = <ex:e1 xmlns:ex={ example }><s1>{ before }</s1><len>{ dataLength }</len><x1><s2>{ text }</s2></x1><s3>{ after }</s3></ex:e1>
+ TestUtils.assertEqualsXMLElements(infoset, actual)
+
+ TestUtils.testUnparsingBinary(sch, infoset, data)
+ }
+
+ val lineFoldLayer1Schema =
+ SchemaUtils.dfdlTestSchema(
+ <dfdl:format ref="tns:GeneralFormat"/>,
+ <xs:element name="e1" dfdl:lengthKind="implicit">
+ <xs:complexType>
+ <xs:sequence dfdl:layerTransform="lineFolded_IMF" dfdl:layerLengthKind="implicit">
+ <xs:element name="s1" type="xs:string" dfdl:lengthKind="delimited"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>, elementFormDefault = "qualified")
+
+ /**
+ * Has lines folded using IMF conventions.
+ *
+ * Notice use of the s"""...""" string interpolation. This interprets
+ * the escape sequences even though triple quote doesn't.
+ */
+ val ipsumLorem1 = s"""Lorem ipsum\r\n dolor sit amet"""
+
+ val ipsumLorem1Unfolded = s"""Lorem ipsum dolor sit amet"""
+
+ @Test def testParseLineFoldIMF1() {
+ val sch = lineFoldLayer1Schema
+ val data = ipsumLorem1
+ val infoset = <e1 xmlns={ example }><s1>{ ipsumLorem1Unfolded }</s1></e1>
+ val (_, actual) = TestUtils.testString(sch, data)
+ TestUtils.assertEqualsXMLElements(infoset, actual)
+ }
+
+ val ipsumLorem2 = s"""Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod\r\n tempor incididunt ut labore et dolore magna aliqua. Ut enim ad"""
+ ///////////////////// 123456789012345678901234567890123456789012345678901234567890123456789012 3 4567890123456789012345678901234567890123456789012345678901234567890
+ ///////////////////// 1 2 3 4 5 6 7 8
+ val ipsumLorem2Unfolded = s"""Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad"""
+
+ @Test def testUnparseLineFoldIMF1() {
+ val sch = lineFoldLayer1Schema
+ val data = ipsumLorem2
+ val infoset = <e1 xmlns={ example }><s1>{ ipsumLorem2Unfolded }</s1></e1>
+ val areTracing = false
+ TestUtils.testUnparsing(sch, infoset, data, areTracing)
+ }
+
+ val lineFoldLayer2Schema =
+ SchemaUtils.dfdlTestSchema(
+ <dfdl:format ref="tns:GeneralFormat"/>,
+ <xs:element name="e1" dfdl:lengthKind="implicit">
+ <xs:complexType>
+ <xs:sequence dfdl:layerTransform="lineFolded_IMF" dfdl:layerLengthKind="boundaryMark">
+ <xs:element name="s1" type="xs:string" dfdl:lengthKind="delimited"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>, elementFormDefault = "qualified")
+
+ /**
+ * Has lines folded using IMF conventions.
+ *
+ * Notice use of the s"""...""" string interpolation. This interprets
+ * the escape sequences even though triple quote doesn't.
+ */
+ val ipsumLorem3 = s"""Lorem ipsum\r\n dolor sit amet,\r\nconsectetur adipiscing elit"""
+
+ val ipsumLorem3Unfolded = s"""Lorem ipsum dolor sit amet,"""
+
+ @Test def testParseLineFoldIMF2() {
+ val sch = lineFoldLayer2Schema
+ val data = ipsumLorem3
+ val infoset = <e1 xmlns={ example }><s1>{ ipsumLorem3Unfolded }</s1></e1>
+ val (_, actual) = TestUtils.testString(sch, data)
+ TestUtils.assertEqualsXMLElements(infoset, actual)
+ }
+
+ val ipsumLorem4 = s"""Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod\r\n tempor incididunt\r\n"""
+ ///////////////////// 123456789012345678901234567890123456789012345678901234567890123456789012 3 4567890123456789012345678901234567890123456789012345678901234567890
+ ///////////////////// 1 2 3 4 5 6 7 8
+ val ipsumLorem4Unfolded = s"""Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt"""
+
+ @Test def testUnparseLineFoldIMF2() {
+ val sch = lineFoldLayer2Schema
+ val data = ipsumLorem4
+ val infoset = <e1 xmlns={ example }><s1>{ ipsumLorem4Unfolded }</s1></e1>
+ val areTracing = false
+ TestUtils.testUnparsing(sch, infoset, data, areTracing)
+ }
+
+ /**
+ * The length of the layer is constrained by surrounding explicit-length
+ * element.
+ */
+ val lineFoldLayer3Schema =
+ SchemaUtils.dfdlTestSchema(
+ <dfdl:format ref="tns:GeneralFormat"/>,
+ <xs:element name="e1" dfdl:lengthKind="explicit" dfdl:length="100">
+ <xs:complexType>
+ <xs:sequence dfdl:layerTransform="lineFolded_IMF" dfdl:layerLengthKind="implicit">
+ <xs:element name="s1" type="xs:string" dfdl:lengthKind="delimited"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>, elementFormDefault = "qualified")
+
+ val ipsumLorem5 = s"""Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod\r\n tempor incididunt ut labore et dolore magna aliqua."""
+ ///////////////////// 123456789012345678901234567890123456789012345678901234567890123456789012 3 4567890123456789012345678901234567890123456789012345678901234567890
+ ///////////////////// 1 2 3 4 5 6 7 8 9 A
+
+ val ipsumLorem5Unfolded = s"""Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labor"""
+
+ @Test def testParseLineFoldIMF3() {
+ val sch = lineFoldLayer3Schema
+ val data = ipsumLorem5
+ val infoset = <e1 xmlns={ example }><s1>{ ipsumLorem5Unfolded }</s1></e1>
+ val (_, actual) = TestUtils.testString(sch, data)
+ TestUtils.assertEqualsXMLElements(infoset, actual)
+ }
+
+ val ipsumLorem6 = s"""Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod\r\n tempor incididunt ut labor"""
+ ///////////////////// 123456789012345678901234567890123456789012345678901234567890123456789012 3 4567890123456789012345678901234567890123456789012345678901234567890
+ ///////////////////// 1 2 3 4 5 6 7 8 9 A
+
+ val ipsumLorem6Unfolded = s"""Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labor"""
+
+ @Test def testUnparseLineFoldIMF3() {
+ val sch = lineFoldLayer3Schema
+ val data = ipsumLorem6
+ val infoset = <e1 xmlns={ example }><s1>{ ipsumLorem6Unfolded }</s1></e1>
+ val areTracing = false
+ TestUtils.testUnparsing(sch, infoset, data, areTracing)
+ }
+}
diff --git a/daffodil-core/src/test/scala/org/apache/daffodil/util/TestUtils.scala b/daffodil-core/src/test/scala/org/apache/daffodil/util/TestUtils.scala
index 1d53189..a26aabf 100644
--- a/daffodil-core/src/test/scala/org/apache/daffodil/util/TestUtils.scala
+++ b/daffodil-core/src/test/scala/org/apache/daffodil/util/TestUtils.scala
@@ -83,9 +83,13 @@ object TestUtils {
runSchemaOnData(testSchema, Misc.stringToReadableByteChannel(data), isTracing)
}
- def testBinary(testSchema: Node, hexData: String, areTracing: Boolean = false) = {
+ def testBinary(testSchema: Node, hexData: String, areTracing: Boolean = false): (DFDL.ParseResult, Node) = {
val b = Misc.hex2Bytes(hexData)
- val rbc = Misc.byteArrayToReadableByteChannel(b)
+ testBinary(testSchema, b, areTracing)
+ }
+
+ def testBinary(testSchema: Node, data: Array[Byte], areTracing: Boolean): (DFDL.ParseResult, Node) = {
+ val rbc = Misc.byteArrayToReadableByteChannel(data)
runSchemaOnData(testSchema, rbc, areTracing)
}
@@ -127,21 +131,26 @@ object TestUtils {
actual.getDiagnostics
}
+ def throwDiagnostics(ds: Seq[Diagnostic]) {
+ if (ds.length == 1) throw (ds(0))
+ else {
+ val msgs = ds.map(_.getMessage()).mkString("\n")
+ throw new Exception(msgs)
+ }
+ }
+
def testUnparsingBinary(testSchema: scala.xml.Elem, infoset: Node, unparseTo: Array[Byte]) {
val compiler = Compiler()
val pf = compiler.compileNode(testSchema)
+ if (pf.isError) throwDiagnostics(pf.diagnostics)
val u = pf.onPath("/")
+ if (u.isError) throwDiagnostics(u.getDiagnostics)
val outputStream = new java.io.ByteArrayOutputStream()
val out = java.nio.channels.Channels.newChannel(outputStream)
val inputter = new ScalaXMLInfosetInputter(infoset)
val actual = u.unparse(inputter, out)
- if (actual.isProcessingError) {
- val msgs = actual.getDiagnostics.map(_.getMessage()).mkString("\n")
- throw new Exception(msgs)
- }
+ if (actual.isProcessingError) throwDiagnostics(actual.getDiagnostics)
val unparsed = outputStream.toByteArray()
- // System.err.println("parsed: " + infoset)
- // System.err.println("unparsed: " + unparsed)
out.close()
assertEquals(unparsed.length, unparseTo.length)
for (i <- 0 until unparsed.length) {
diff --git a/daffodil-io/src/main/scala/org/apache/daffodil/io/ByteBufferDataInputStream.scala b/daffodil-io/src/main/scala/org/apache/daffodil/io/ByteBufferDataInputStream.scala
index 68eac68..7ded5ff 100644
--- a/daffodil-io/src/main/scala/org/apache/daffodil/io/ByteBufferDataInputStream.scala
+++ b/daffodil-io/src/main/scala/org/apache/daffodil/io/ByteBufferDataInputStream.scala
@@ -23,7 +23,6 @@ import org.apache.daffodil.schema.annotation.props.gen.UTF16Width
import org.apache.daffodil.schema.annotation.props.gen.BitOrder
import org.apache.daffodil.schema.annotation.props.gen.ByteOrder
import java.nio.charset.CodingErrorAction
-import org.apache.commons.io.IOUtils
import java.io.ByteArrayOutputStream
import java.nio.ByteBuffer
import org.apache.daffodil.exceptions.Assert
@@ -91,7 +90,13 @@ object ByteBufferDataInputStream {
case _ => {
// copy the contents of the stream into an array of bytes
val bos = new ByteArrayOutputStream
- IOUtils.copy(in, bos)
+ var b: Int = 0
+ while ({
+ b = in.read()
+ b != -1
+ }) {
+ bos.write(b)
+ }
bos.flush()
bos.close()
in.close()
@@ -291,7 +296,10 @@ final class ByteBufferDataInputStream private (var data: ByteBuffer, initialBitP
// threadCheck()
// we always have a bitLimit in this implementation
Assert.invariant(st.maybeBitLimitOffset0b.isDefined)
- (data.limit << 3) + st.maybeBitLimitOffset0b.get
+ val dl = data.limit
+ val bitLimitOffset = st.maybeBitLimitOffset0b.get
+ val bitLimit = (dl << 3) + bitLimitOffset
+ bitLimit
}
override def setBitLimit0b(newBitLimit0b: MaybeULong): Boolean = {
@@ -359,7 +367,8 @@ final class ByteBufferDataInputStream private (var data: ByteBuffer, initialBitP
val bytesToFill = bitLengthFrom1 / 8
Assert.invariant(array.size >= bytesToFill)
- if (finfo.byteOrder == ByteOrder.BigEndian && finfo.bitOrder == BitOrder.MostSignificantBitFirst) {
+ if (bytesToFill == 1 || // 1 byte is super common case. We don't want to retrieve byteOrder nor bitOrder in this case
+ (finfo.byteOrder == ByteOrder.BigEndian && finfo.bitOrder == BitOrder.MostSignificantBitFirst)) {
// bits & bytes are already in order, read them straight into the array
data.get(array, 0, bytesToFill)
} else {
diff --git a/daffodil-io/src/main/scala/org/apache/daffodil/io/DataOutputStream.scala b/daffodil-io/src/main/scala/org/apache/daffodil/io/DataOutputStream.scala
index 8ee0077..53d7b10 100644
--- a/daffodil-io/src/main/scala/org/apache/daffodil/io/DataOutputStream.scala
+++ b/daffodil-io/src/main/scala/org/apache/daffodil/io/DataOutputStream.scala
@@ -78,7 +78,7 @@ import org.apache.daffodil.util.Logging
trait DataOutputStream extends DataStreamCommon
with Logging {
- def id: Int
+ def id: String
def relBitPos0b: ULong
diff --git a/daffodil-io/src/main/scala/org/apache/daffodil/io/DataOutputStreamImplMixin.scala b/daffodil-io/src/main/scala/org/apache/daffodil/io/DataOutputStreamImplMixin.scala
index 20212b6..72ed284 100644
--- a/daffodil-io/src/main/scala/org/apache/daffodil/io/DataOutputStreamImplMixin.scala
+++ b/daffodil-io/src/main/scala/org/apache/daffodil/io/DataOutputStreamImplMixin.scala
@@ -817,7 +817,8 @@ trait DataOutputStreamImplMixin extends DataStreamCommonState
final override def pastData(nBytesRequested: Int): ByteBuffer = {
Assert.usage(isReadable)
- if (!areDebugging) throw new IllegalStateException("Must be debugging.")
+ if (!areDebugging)
+ throw new IllegalStateException("Must be debugging.")
Assert.usage(nBytesRequested >= 0)
if (debugOutputStream == Nope) {
ByteBuffer.allocate(0)
diff --git a/daffodil-io/src/main/scala/org/apache/daffodil/io/DirectOrBufferedDataOutputStream.scala b/daffodil-io/src/main/scala/org/apache/daffodil/io/DirectOrBufferedDataOutputStream.scala
index a55392f..b07c0ef 100644
--- a/daffodil-io/src/main/scala/org/apache/daffodil/io/DirectOrBufferedDataOutputStream.scala
+++ b/daffodil-io/src/main/scala/org/apache/daffodil/io/DirectOrBufferedDataOutputStream.scala
@@ -96,11 +96,26 @@ final class DirectOrBufferedDataOutputStream private[io] (var splitFrom: DirectO
res
}
+ private val layerID: Int = synchronized {
+ if (splitFrom ne null) splitFrom.layerID
+ else {
+ val lid = DirectOrBufferedDataOutputStream.nextLayerID
+ DirectOrBufferedDataOutputStream.nextLayerID += 1
+ lid
+ }
+ }
+
/**
* Must be val, as split-from will get reset to null as streams
* are morphed into direct streams.
*/
- val id: Int = if (splitFrom == null) 0 else splitFrom.id + 1
+ private val splitID: Int = if (splitFrom == null) 0 else splitFrom.splitID + 1
+
+ /**
+ * id will be a N.M type identifier where N is the layer number,
+ * and M is the splitID.
+ */
+ lazy val id: String = layerID.toString + "." + splitID.toString
/**
* Two of these are equal if they are eq.
@@ -187,6 +202,10 @@ final class DirectOrBufferedDataOutputStream private[io] (var splitFrom: DirectO
*/
private var _following: Maybe[DirectOrBufferedDataOutputStream] = Nope
+ def lastInChain: DirectOrBufferedDataOutputStream =
+ if (_following.isEmpty) this
+ else _following.get.lastInChain
+
/**
* Provides a new buffered data output stream. Note that this must
* be completely configured (byteOrder, encoding, bitOrder, etc.)
@@ -305,6 +324,9 @@ final class DirectOrBufferedDataOutputStream private[io] (var splitFrom: DirectO
// zero out so we don't end up thinking it is still there
directStream.cst.setFragmentLastByte(0, 0)
}
+ // now flush/close the whole data output stream
+ // propagate the closed-ness by closing the underlying java stream.
+ directStream.getJavaOutputStream().close()
directStream.setDOSState(Uninitialized) // not just finished. We're dead now.
} else {
// the last stream we merged forward into was not finished.
@@ -700,6 +722,7 @@ class BitOrderChangeException(directDOS: DirectOrBufferedDataOutputStream, finfo
object DirectOrBufferedDataOutputStream {
+ var nextLayerID = 0
/**
* This is over here to be sure it isn't operating on other members
* of the object. This operates on the arguments only.
@@ -779,7 +802,8 @@ object DirectOrBufferedDataOutputStream {
}
/**
- * Factory for creating new ones
+ * Factory for creating new ones/
+ * Passing creator as null indicates no other stream created this one.
*/
def apply(jos: java.io.OutputStream, creator: DirectOrBufferedDataOutputStream) = {
val dbdos = new DirectOrBufferedDataOutputStream(creator)
diff --git a/daffodil-io/src/main/scala/org/apache/daffodil/io/LimitingJavaIinputStreams.scala b/daffodil-io/src/main/scala/org/apache/daffodil/io/LimitingJavaIinputStreams.scala
new file mode 100644
index 0000000..a169a48
--- /dev/null
+++ b/daffodil-io/src/main/scala/org/apache/daffodil/io/LimitingJavaIinputStreams.scala
@@ -0,0 +1,287 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.io
+
+import org.apache.daffodil.exceptions.Assert
+import java.util.regex.Pattern
+import java.util.Scanner
+import java.nio.charset.Charset
+import java.io.InputStream
+import java.io.FilterInputStream
+import java.io.InputStreamReader
+import java.nio.charset.StandardCharsets
+
+/**
+ * This class can be used with any InputStream to restrict what is
+ * read from it to N bytes.
+ *
+ * This can be used to forcibly stop consumption of data from a stream at
+ * a length obtained explicitly.
+ *
+ * Thread safety: This is inherently stateful - so not thread safe to use
+ * this object from more than one thread.
+ */
+class ExplicitLengthLimitingStream(in: InputStream, limit: Int)
+ extends FilterInputStream(in) {
+
+ private var numRemaining = limit
+
+ override def read(buf: Array[Byte], off: Int, len: Int) = {
+ Assert.invariant(numRemaining >= 0)
+ if (numRemaining == 0) -1
+ else if (len == 0) 0
+ else {
+ val requestSize = math.min(numRemaining, len)
+ val actualSize = in.read(buf, off, requestSize)
+ if (actualSize == -1)
+ numRemaining = 0
+ else
+ numRemaining -= actualSize
+ actualSize
+ }
+ }
+
+ private val readBuf = new Array[Byte](1)
+
+ override def read(): Int = {
+ readBuf(0) = 0
+ val n = read(readBuf, 0, 1)
+ if (n == -1)
+ -1
+ else {
+ Assert.invariant(n == 1)
+ val i = readBuf(0).toInt
+ val b = i & 0xFF
+ b
+ }
+ }
+}
+
+/**
+ * Can be used with any InputStream to restrict what is
+ * read from it to stop before a boundary mark string.
+ *
+ * The boundary mark string is exactly that, a string of characters. Not a
+ * regex, nor anything involving DFDL Character Entities or Character Class
+ * Entities. (No %WSP; no %NL; )
+ *
+ * This can be used to forcibly stop consumption of data from a stream at
+ * a length obtained from a delimiter.
+ *
+ * The boundary mark string is consumed from the underlying stream (if found), and
+ * the underlying stream is left positioned at the byte after the boundary mark
+ * string.
+ *
+ * Thread safety: This is inherently stateful - so not thread safe to use
+ * this object from more than one thread.
+ */
+object BoundaryMarkLimitingStream {
+
+ def apply(inputStream: InputStream, boundaryMark: String, charset: Charset,
+ targetChunkSize: Int = 32 * 1024) = {
+
+ Assert.usage(targetChunkSize >= 1)
+ Assert.usage(boundaryMark.length >= 1)
+
+ val boundaryMarkIn8859 = new String(boundaryMark.getBytes(charset), StandardCharsets.ISO_8859_1)
+
+ val quotedBoundaryMark = Pattern.quote(boundaryMarkIn8859) // in case pattern has non-regex-safe characters in it
+
+ val result = new RegexLimitingStream(inputStream, quotedBoundaryMark, boundaryMarkIn8859, charset, targetChunkSize)
+ result
+ }
+
+}
+
+class StreamIterator[T](s: Stream[T])
+ extends Iterator[T] {
+ private var str = s
+ override def hasNext = !str.isEmpty
+ override def next() = {
+ val res = str.head
+ str = str.tail
+ res
+ }
+}
+
+/**
+ * Can be used with any InputStream to restrict what is
+ * read from it to stop before a particular regex match.
+ *
+ * The regex must have a finite maximum length match string.
+ *
+ * This can be used to forcibly stop consumption of data from a stream at
+ * a length obtained from a delimiter that is described using a regex.
+ *
+ * The delimiter matching the regex is consumed from the underlying stream (if found), and
+ * the underlying stream is left positioned at the byte after the regex match
+ * string.
+ *
+ * IMPORTANT: The delimiter regex cannot contain any Capturing Groups!
+ * Use (?: ... ) which is non-capturing, instead of regular ( ... ).
+ * For example: this regex matches CRLF not followed by tab or space:
+ * {{{
+ * """\r\n(?!(?:\t|\ ))"""
+ * }}}
+ * Notice use of the ?: to avoid a capture group around the alternatives of tab or space.
+ *
+ * Thread safety: This is inherently stateful - so not thread safe to use
+ * this object from more than one thread.
+ */
+class RegexLimitingStream(inputStream: InputStream,
+ regexForDelimiter: String,
+ maximumLengthDelimiterExample: String,
+ charset: Charset,
+ targetChunkSize: Int = 32 * 1024)
+ extends InputStream {
+
+ Assert.usage(targetChunkSize >= 1)
+ Assert.usage(maximumLengthDelimiterExample.length >= 1)
+
+ private val in = inputStream
+
+ /**
+ * This only works because we lower the whole matching process to using iso-8859-1
+ * which is equivalent to raw bytes. That way when we read a char to get a match
+ * we know it consumes exactly one byte.
+ *
+ * This trick may be useful for dealing with DFDL's rawBytes feature.
+ * In principle, a delimiter in DFDL can be a mixture of characters and
+ * raw bytes. To match these, you have to lower the character parts to iso_8859_1
+ * in the way done here, and then combine with the raw bytes to make a string
+ * (of bytes) that can be matched pretending the data is iso_8859_1 data, when
+ * it really isn't.
+ *
+ * Example of this might be if a UTF_8 string was delimited by bytes that
+ * are illegal in UTF-8's encoding scheme. E.g., UTF-data delimited by say
+ * bytes 00 and FF. So this could be %#x00;%#rFF; The 00, or NUL is a legal
+ * UTF-8 code point. The FF is not.
+ */
+ private val maxDelimiterIn8859 = new String(maximumLengthDelimiterExample.getBytes(charset), StandardCharsets.ISO_8859_1)
+
+ private val maxDelimiterLength =
+ math.ceil(maxDelimiterIn8859.length *
+ charset.newEncoder().maxBytesPerChar()).toInt
+
+ private val chunkSize = math.max(targetChunkSize, maxDelimiterLength + 1)
+
+ /**
+ * This regex matches a chunk from zero to chunksize followed by boundaryMark, or
+ * anything from zero to chunksize.
+ * Group 1 is the chunk matched with group 2 containing the boundaryMark.
+ * Group 3 is the chunk matched if boundaryMark is not found.
+ */
+ private val regex = """([\s\S]{0,""" + chunkSize + """}?)(?=(""" + regexForDelimiter + """))|([\s\S]{0,""" + chunkSize + """})"""
+ private val pattern = Pattern.compile(regex)
+
+ /**
+ * The regex can match at most the chunkSize + the maxBoundaryMarkLength in size.
+ */
+ private val lookAheadMax = chunkSize + maxDelimiterLength
+
+ private lazy val charsIter = {
+ val cks = chunks
+ val streamChars = cks.flatten
+ val iter = new StreamIterator(streamChars)
+ iter
+ }
+
+ override def read(): Int = {
+ if (!charsIter.hasNext) -1
+ else charsIter.next().toInt
+ }
+
+ override def available(): Int =
+ if (charsIter.hasNext) 1 else 0
+
+ override def close() {
+ }
+
+ private var noMoreChunks = false
+ /**
+ * This lazy stream stuff might look like a lot of overhead, but
+ * consider that the overhead is once per chunk, so honestly the
+ * regex match is of more concern.
+ */
+ private def chunks: Stream[String] = {
+ if (noMoreChunks) Stream()
+ else {
+ in.mark(lookAheadMax)
+ //
+ // Unfortunately, we have to reconstruct these objects because they otherwise
+ // might pre-cache underlying data from the input and we need them to
+ // start from a precise byte location when the next scan begins.
+ //
+ val rdr = new InputStreamReader(in, StandardCharsets.ISO_8859_1)
+ val scanner = new Scanner(rdr)
+ val matchString = scanner.findWithinHorizon(pattern, 0)
+ val matchLength = checkScan(matchString, scanner)
+ //
+ // the trick is that the length of the matchString could be shorter than the
+ // number of characters (aka bytes) pulled from the input stream because in
+ // scanning, the scanner or reader might buffer up extra decoded characters that
+ // strictly speaking aren't needed to obtain the match.
+ //
+ // So we reset back to the start, and advance exactly that number of bytes.
+ //
+ // This only works because we have lowered everything to iso-8859-1 here.
+ // Decoding errors, and the complexities they create over how big the string
+ // is, vs. how many bytes were consumed... those can't happen with iso-8859-1.
+ //
+ in.reset() // might have to backup farther than the matchString length
+ in.skip(matchString.length + matchLength) // advance exactly the right number of bytes
+ if (matchLength > 0)
+ noMoreChunks = true
+ if (matchString.isEmpty())
+ Stream()
+ else
+ matchString #:: chunks
+ }
+ }
+
+ /**
+ * Thorough checking that we understand the behavior of our regex and
+ * scanner.
+ */
+ private def checkScan(matchString: String, scanner: Scanner) = {
+
+ Assert.invariant(matchString ne null); // worst case it matches 0 length and we get ""
+ //
+ // Just do some error checking to be absolutely sure we understand
+ // how the scanner works
+ //
+ val matcher = scanner.`match`()
+ val beforeDelimMatch = matcher.end(1) // maybe avoids allocating the string
+ val delimMatch = matcher.end(2)
+ val noDelimMatch = matcher.end(3)
+ val isFound = (beforeDelimMatch > -1)
+ val delimMatchLength =
+ if (isFound) {
+ Assert.invariant(delimMatch > -1)
+ Assert.invariant(noDelimMatch == -1)
+ matcher.end(2) - matcher.start(2)
+ } else {
+ Assert.invariant(delimMatch == -1)
+ Assert.invariant(noDelimMatch > -1)
+ 0
+ }
+ delimMatchLength
+ }
+
+}
+
diff --git a/daffodil-io/src/main/scala/org/apache/daffodil/processors/charset/USASCII6BitPackedDecoder.scala b/daffodil-io/src/main/scala/org/apache/daffodil/io/LimitingJavaOutputStreams.scala
similarity index 53%
copy from daffodil-io/src/main/scala/org/apache/daffodil/processors/charset/USASCII6BitPackedDecoder.scala
copy to daffodil-io/src/main/scala/org/apache/daffodil/io/LimitingJavaOutputStreams.scala
index 5f59f3b..2f28e53 100644
--- a/daffodil-io/src/main/scala/org/apache/daffodil/processors/charset/USASCII6BitPackedDecoder.scala
+++ b/daffodil-io/src/main/scala/org/apache/daffodil/io/LimitingJavaOutputStreams.scala
@@ -14,27 +14,25 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
+package org.apache.daffodil.io
-package org.apache.daffodil.processors.charset
+import java.nio.charset.Charset
-import org.apache.daffodil.schema.annotation.props.gen.BitOrder
+class LayerBoundaryMarkInsertingJavaOutputStream(jos: java.io.OutputStream, boundaryMark: String,
+ charset: Charset)
+ extends java.io.FilterOutputStream(jos) {
-/**
- * Some encodings are not byte-oriented.
- *
- * X-DFDL-US-ASCII-6-BIT-PACKED occupies only 6 bits with each
- * code unit.
- *
- */
+ private var closed = false
-object USASCII6BitPackedCharset
- extends NBitsWidth_BitsCharset("X-DFDL-US-ASCII-6-BIT-PACKED",
- """@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_ !"#$%&'()*+,-./0123456789:;<=>?""",
- 6, // width
- BitOrder.LeastSignificantBitFirst,
- 0x1F) { // replacement charCode for encoding of unmapped chars.
- //
- // Note: formula for computing string above is
- // def decodeString = ((0 to 63).map { charCode => (if (charCode <= 31) charCode + 64 else charCode).toChar }).mkString
-}
+ private val boundaryMarkBytes = boundaryMark.getBytes(charset)
+
+ override def close(): Unit = {
+ if (!closed) {
+ jos.write(boundaryMarkBytes)
+ jos.flush()
+ jos.close()
+ closed = true
+ }
+ }
+}
diff --git a/daffodil-io/src/main/scala/org/apache/daffodil/processors/charset/AIS_PAYLOAD_ARMORING.scala b/daffodil-io/src/main/scala/org/apache/daffodil/processors/charset/AIS_PAYLOAD_ARMORING.scala
new file mode 100644
index 0000000..1159044
--- /dev/null
+++ b/daffodil-io/src/main/scala/org/apache/daffodil/processors/charset/AIS_PAYLOAD_ARMORING.scala
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.daffodil.processors.charset
+
+import org.apache.daffodil.schema.annotation.props.gen.BitOrder
+
+/**
+ * Special purpose. This is not used for decoding anything.
+ * The encoder is used to convert strings using the characters
+ * allowed, into binary data using the AIS Payload Armoring
+ * described here:
+ *
+ * http://catb.org/gpsd/AIVDM.html#_aivdm_aivdo_payload_armoring
+ *
+ * To convert a string of length N, You will get 6N bits.
+ *
+ * The decoder can be used for unit testing, but the point of this class
+ * is to make the encoder available for use in un-doing the AIS Payload
+ * armoring.
+ *
+ * When encoding from 8-bit say, ascii, or iso-8859-1, this can only encode
+ * things that stay within the 64 allowed characters.
+ * dfdl:encodingErrorPolicy='error' would check this (once implemented), otherwise
+ * where this is used the checking needs to be done separately somehow.
+ */
+object AIS_PAYLOAD_ARMORING
+ extends NBitsWidth_BitsCharset("X-DAFFODIL-AIS-PAYLOAD-ARMORING",
+ """0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVW'abcdefghijklmnopqrstuvw""",
+ 6, // width
+ BitOrder.MostSignificantBitFirst,
+ 0x0) { // replacement charCode for encoding of unmapped chars. Should never be used.
+}
diff --git a/daffodil-io/src/main/scala/org/apache/daffodil/processors/charset/CharsetUtils.scala b/daffodil-io/src/main/scala/org/apache/daffodil/processors/charset/CharsetUtils.scala
index 38363f5..0c2393f 100644
--- a/daffodil-io/src/main/scala/org/apache/daffodil/processors/charset/CharsetUtils.scala
+++ b/daffodil-io/src/main/scala/org/apache/daffodil/processors/charset/CharsetUtils.scala
@@ -115,6 +115,10 @@ trait EncoderDecoderMixin
* except when adding a new not-seen-before encoder or decoder.
*/
+ //
+ // TODO: Cleanup - change below to use NonAllocatingMap to improve code style.
+ // And fix comment above.
+ //
private lazy val decoderCache = new java.util.HashMap[BitsCharset, DecoderInfo]
private lazy val encoderCache = new java.util.HashMap[BitsCharset, EncoderInfo]
diff --git a/daffodil-io/src/main/scala/org/apache/daffodil/processors/charset/DaffodilCharsetProvider.scala b/daffodil-io/src/main/scala/org/apache/daffodil/processors/charset/DaffodilCharsetProvider.scala
index faa0f18..b7d3955 100644
--- a/daffodil-io/src/main/scala/org/apache/daffodil/processors/charset/DaffodilCharsetProvider.scala
+++ b/daffodil-io/src/main/scala/org/apache/daffodil/processors/charset/DaffodilCharsetProvider.scala
@@ -66,7 +66,10 @@ object DaffodilCharsetProvider {
"X-DFDL-OCTAL-MSBF" -> OctalMSBF3BitCharset,
"X-DFDL-BITS-LSBF" -> X_DFDL_BITS_LSBF,
"X-DFDL-BITS-MSBF" -> X_DFDL_BITS_MSBF,
- "X-DFDL-6-BIT-DFI-264-DUI-001" -> X_DFDL_6_BIT_DFI_264_DUI_001 // needed for STANAG 5516/Link16
+ "X-DFDL-6-BIT-DFI-264-DUI-001" -> X_DFDL_6_BIT_DFI_264_DUI_001, // needed for STANAG 5516/Link16
+
+ "X-DFDL-US-ASCII-6-BIT-PACKED-LSB-FIRST" -> USASCII6BitPackedLSBFirstCharset, // name alias
+ "X-DFDL-US-ASCII-6-BIT-PACKED-MSB-FIRST" -> USASCII6BitPackedMSBFirstCharset // new. Not in DFDL v1.0. Used in AIS aka ITU M.1371-2
)
/**
diff --git a/daffodil-io/src/main/scala/org/apache/daffodil/processors/charset/USASCII6BitPackedDecoder.scala b/daffodil-io/src/main/scala/org/apache/daffodil/processors/charset/USASCII6BitPackedDecoder.scala
index 5f59f3b..4f68885 100644
--- a/daffodil-io/src/main/scala/org/apache/daffodil/processors/charset/USASCII6BitPackedDecoder.scala
+++ b/daffodil-io/src/main/scala/org/apache/daffodil/processors/charset/USASCII6BitPackedDecoder.scala
@@ -26,15 +26,26 @@ import org.apache.daffodil.schema.annotation.props.gen.BitOrder
* code unit.
*
*/
-
-object USASCII6BitPackedCharset
- extends NBitsWidth_BitsCharset("X-DFDL-US-ASCII-6-BIT-PACKED",
+private[charset] class USASCII6BitPacked(bitOrder: BitOrder, override val name: String)
+ extends NBitsWidth_BitsCharset(name,
"""@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_ !"#$%&'()*+,-./0123456789:;<=>?""",
6, // width
- BitOrder.LeastSignificantBitFirst,
+ bitOrder,
0x1F) { // replacement charCode for encoding of unmapped chars.
//
// Note: formula for computing string above is
// def decodeString = ((0 to 63).map { charCode => (if (charCode <= 31) charCode + 64 else charCode).toChar }).mkString
}
+object USASCII6BitPackedLSBFirstCharset
+ extends USASCII6BitPacked(BitOrder.LeastSignificantBitFirst,
+ "X-DFDL-US-ASCII-6-BIT-PACKED-LSB-FIRST")
+
+object USASCII6BitPackedCharset
+ extends USASCII6BitPacked(BitOrder.LeastSignificantBitFirst,
+ "X-DFDL-US-ASCII-6-BIT-PACKED") // just an alias for the the "LSB-FIRST variant.
+
+object USASCII6BitPackedMSBFirstCharset
+ extends USASCII6BitPacked(BitOrder.MostSignificantBitFirst,
+ "X-DFDL-US-ASCII-6-BIT-PACKED-MSB-FIRST")
+
diff --git a/daffodil-io/src/test/scala/org/apache/daffodil/io/TestAISPayloadArmoringEncoder.scala b/daffodil-io/src/test/scala/org/apache/daffodil/io/TestAISPayloadArmoringEncoder.scala
new file mode 100644
index 0000000..6fd378f
--- /dev/null
+++ b/daffodil-io/src/test/scala/org/apache/daffodil/io/TestAISPayloadArmoringEncoder.scala
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.io
+
+import java.nio.ByteBuffer
+import java.nio.CharBuffer
+import org.junit.Assert._
+import org.junit.Test
+import org.apache.daffodil.util.Misc
+import org.apache.daffodil.processors.charset.AIS_PAYLOAD_ARMORING
+
+class TestAISPayloadArmoringEncoder {
+
+ @Test def testAISArmoringDecoder_01(): Unit = {
+ val cs = AIS_PAYLOAD_ARMORING
+ val decoder = cs.newDecoder()
+ assertNotNull(decoder)
+
+ // val expected = """14eG;o@034o8sd<L9i:a;WF>062D"""
+ //
+ // Above example string from http://www.bosunsmate.org/ais/#bitvector
+ //
+ // Unfortunately, that string doesn't match the binary data. I.e., pretty
+ // sure the web site is wrong.
+ //
+ // Look at the fifth set of 6 bits which is 011100. In the above string
+ // they have ";" for this, but on the web site, the bit pattern for ";"
+ // is 001011. The character for the bits given is "L".
+ //
+ // The corrected expected string is below.
+ val expected = "14eGL:@000o8oQ'LMjOchmG@08HK"
+
+ val cb = CharBuffer.allocate(expected.length())
+ val bb = ByteBuffer.wrap(Misc.bits2Bytes(
+ """000001 000100 101101 010111
+ 011100 001010 010000 000000
+ 000000 000000 110111 001000
+ 110111 100001 101000 011100
+ 011101 110010 011111 101011
+ 110000 110101 010111 010000
+ 000000 001000 011000 011011"""))
+ val res = decoder.decode(bb, cb, false)
+ assertTrue(res.isUnderflow())
+ cb.flip()
+ val actual = cb.toString()
+ assertEquals(expected, actual)
+ }
+
+ @Test def testAISArmoringEncoder_01(): Unit = {
+ val cs = AIS_PAYLOAD_ARMORING
+ val encoder = cs.newEncoder()
+ assertNotNull(encoder)
+
+ // val actual = """14eG;o@034o8sd<L9i:a;WF>062D"""
+ //
+ // Above example string from http://www.bosunsmate.org/ais/#bitvector
+ //
+ // Unfortunately, that string doesn't match the binary data. I.e., pretty
+ // sure the web site is wrong.
+ //
+ // Look at the fifth set of 6 bits which is 011100. In the above string
+ // they have ";" for this, but on the web site, the bit pattern for ";"
+ // is 001011. The character for the bits given is "L".
+ //
+ // The corrected string is below.
+ val actual = "14eGL:@000o8oQ'LMjOchmG@08HK"
+
+ val cb = CharBuffer.wrap(actual)
+ val expected = ByteBuffer.wrap(Misc.bits2Bytes(
+ """000001 000100 101101 010111
+ 011100 001010 010000 000000
+ 000000 000000 110111 001000
+ 110111 100001 101000 011100
+ 011101 110010 011111 101011
+ 110000 110101 010111 010000
+ 000000 001000 011000 011011"""))
+ val bb = ByteBuffer.allocate(expected.limit())
+ val res = encoder.encode(cb, bb, false)
+ assertTrue(res.isUnderflow())
+ bb.flip()
+ assertEquals(expected.compareTo(bb), 0)
+ }
+}
diff --git a/daffodil-io/src/test/scala/org/apache/daffodil/layers/TestBase64.scala b/daffodil-io/src/test/scala/org/apache/daffodil/layers/TestBase64.scala
new file mode 100644
index 0000000..e0680b8
--- /dev/null
+++ b/daffodil-io/src/test/scala/org/apache/daffodil/layers/TestBase64.scala
@@ -0,0 +1,245 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.layers
+
+import junit.framework.Assert._
+import org.junit.Test
+import org.apache.daffodil.Implicits._
+import org.apache.commons.io.IOUtils
+import org.apache.daffodil.exceptions.Assert
+
+/**
+ * There are 3 variations of base64 we care about.
+ *
+ * Base64_RFC2045 (MIME)
+ * Base64_RFC4648
+ * Base64_RFC4648_URLSAFE
+ *
+ * When decoding from Base64, the CRLF vs. LF doesn't matter. However when
+ * encoding, the RFC specify that CRLF are inserted.
+ */
+class TestBase64 {
+ Assert.usage(scala.util.Properties.isJavaAtLeast("1.8"))
+
+ val text = """This is just some made up text that is intended to be
+a few lines long. If this had been real text, it would not have been quite
+so boring to read. Use of famous quotes or song lyrics or anything like that
+introduces copyright notice issues, so it is easier to simply make up
+a few lines of pointless text like this.""".replace("\n", " ")
+
+ val b64Text = """VGhpcyBpcyBqdXN0IHNvbWUgbWFkZSB1cCB0ZXh0IHRoYXQgaXMgaW50Z
+W5kZWQgdG8gYmUgYSBmZXcgbGluZXMgbG9uZy4gSWYgdGhpcyBoYWQgYmVlbiByZWFsIHRleHQsI
+Gl0IHdvdWxkIG5vdCBoYXZlIGJlZW4gcXVpdGUgc28gYm9yaW5nIHRvIHJlYWQuIFVzZSBvZiBmY
+W1vdXMgcXVvdGVzIG9yIHNvbmcgbHlyaWNzIG9yIGFueXRoaW5nIGxpa2UgdGhhdCBpbnRyb2R1Y
+2VzIGNvcHlyaWdodCBub3RpY2UgaXNzdWVzLCBzbyBpdCBpcyBlYXNpZXIgdG8gc2ltcGx5IG1ha
+2UgdXAgYSBmZXcgbGluZXMgb2YgcG9pbnRsZXNzIHRleHQgbGlrZSB0aGlzLg==
+""".replace("\n", "").sliding(76, 76).mkString("\r\n")
+
+ /**
+ * Same encoded data, but shot through with extra CRLFs
+ */
+ val b64TextExtraLFs = b64Text.split("Z").mkString("Z\r\n")
+
+ @Test def testBase64_01() {
+ val input = text
+ val expected = b64Text
+ val encoded = java.util.Base64.getMimeEncoder.encodeToString(input.getBytes("ascii"))
+ assertEquals(expected.length, encoded.length)
+ val pairs = expected zip encoded
+ var i = 0
+ var failed = false
+ pairs.foreach {
+ case (exp, act) => {
+ if (exp != act) {
+ println("differ at character %s (0-based). Expected '%s' got '%s'.".format(i, exp, act))
+ failed = true
+ }
+ i += 1
+ }
+ }
+ val textDecoded = new String(java.util.Base64.getMimeDecoder.decode(encoded))
+ assertEquals(input, textDecoded)
+ if (failed) fail()
+ }
+
+ @Test def testBase64_broken_data_01() {
+
+ val data = b64Text.tail
+
+ intercept[IllegalArgumentException] {
+ java.util.Base64.getMimeDecoder.decode(data)
+ }
+
+ }
+
+ @Test def testBase64_broken_data_02() {
+
+ val data = b64Text.dropRight(3)
+
+ intercept[IllegalArgumentException] {
+ println(new String(java.util.Base64.getMimeDecoder.decode(data)))
+ }
+
+ }
+
+ @Test def testBase64_tolerates_extra_CRLFs() {
+ val data = b64TextExtraLFs
+ val textDecoded = new String(java.util.Base64.getMimeDecoder.decode(data))
+ assertEquals(text, textDecoded)
+ }
+
+ @Test def testBase64_decode_consumes_final_CRLF() {
+
+ val data = b64Text ++ "\r\n" // add extra CRLF
+
+ val actual = new String(java.util.Base64.getMimeDecoder.decode(data))
+
+ assertEquals(text, actual)
+ }
+
+ @Test def testBase64_decode_consumes_final_LF() {
+
+ val data = b64Text ++ "\n" // add extra LF
+
+ val actual = new String(java.util.Base64.getMimeDecoder.decode(data))
+
+ assertEquals(text, actual)
+ }
+
+ /**
+ * Base64 decoder that decodes strings consumes any trailing CRLFs or LFs
+ * and ignores them.
+ */
+ @Test def testBase64_decode_consumes_final_LFLFLFLF() {
+
+ val data = b64Text ++ "\n\n\n\n" // add extra LF
+
+ val actual = new String(java.util.Base64.getMimeDecoder.decode(data))
+
+ assertEquals(text, actual)
+ }
+
+ /**
+ * This test shows us that if we wrap the base64 decoder around a
+ * java input stream, and read as much as we can from it, that it will stop
+ * at an equals sign, and tolerates additional equals signs even.
+ */
+ @Test def testBase64_decode_from_stream_consumes_nothing_extra() {
+
+ val additional = "====ABCD"
+ val data = b64Text ++ additional // add extra characters
+
+ val is = IOUtils.toInputStream(data, "ascii")
+ val b64 = java.util.Base64.getMimeDecoder().wrap(is)
+ val actual = IOUtils.toString(b64, "ascii")
+ assertEquals(text, actual)
+ b64.close()
+ val leftOver = IOUtils.toString(is, "ascii")
+
+ assertEquals(additional, leftOver)
+ }
+
+ /**
+ * This test shows us that if we wrap the base64 decoder around a
+ * java input stream, and read from an encoding that has no padding so
+ * no equals-signs on the end. Then it will NOT stop and will fail
+ * trying to read past that end.
+ *
+ * This tells us that base64 is, generally speaking, not self-delimiting.
+ */
+ @Test def testBase64_decode_from_stream_does_not_stop_by_itself() {
+
+ val data = "cGxlYXN1cmUu" ++ "=" // encoding of "pleasure."
+
+ val is = IOUtils.toInputStream(data, "ascii")
+ val b64 = java.util.Base64.getMimeDecoder().wrap(is)
+ val ex = intercept[java.io.IOException] {
+ IOUtils.toString(b64, "ascii")
+ }
+ val msg = ex.getMessage()
+ assertTrue(msg.toLowerCase().contains("illegal base64 ending sequence"))
+ }
+
+ /**
+ * Even if there is a LF there. And the line is shorter than the spec
+ * maximum.
+ *
+ * It behaves as if it removes all the line endings first, and then decodes.
+ */
+ @Test def testBase64_decode_from_stream_does_not_stop_by_itself2() {
+
+ val data = "cGxlYXN1cmUu" ++ "\n=" // encoding of "pleasure."
+
+ val is = IOUtils.toInputStream(data, "ascii")
+ val b64 = java.util.Base64.getMimeDecoder().wrap(is)
+ val ex = intercept[java.io.IOException] {
+ IOUtils.toString(b64, "ascii")
+ }
+ val msg = ex.getMessage()
+ assertTrue(msg.toLowerCase().contains("illegal base64 ending sequence"))
+ }
+
+ def compare(input: String, expected: String) = {
+ val encoded = java.util.Base64.getMimeEncoder.encodeToString(input.getBytes("ascii"))
+ assertEquals(expected.length, encoded.length)
+ val pairs = expected zip encoded
+ var i = 0
+ var failed = false
+ pairs.foreach {
+ case (exp, act) => {
+ if (exp != act) {
+ println("differ at character %s (0-based). Expected '%s' got '%s'.".format(i, exp, act))
+ failed = true
+ }
+ i += 1
+ }
+ }
+ val textDecoded = new String(java.util.Base64.getMimeDecoder.decode(encoded))
+ assertEquals(input, textDecoded)
+ if (failed) fail()
+ }
+
+ /**
+ * Zero-length string encodes to zero length string.
+ */
+ @Test def testBase64_0Byte() {
+ compare("", "")
+ }
+
+ /**
+ * If length is 1 mod 3, then there will be "==" after.
+ */
+ @Test def testBase64_1Byte() {
+ compare("Q", "UQ==")
+ }
+
+ /**
+ * If length is 2 mod 3, then there will be "=" after.
+ */
+ @Test def testBase64_2Byte() {
+ compare("QQ", "UVE=")
+ }
+
+ /**
+ * If length is 0 mod 3, then there will be no trailing characters.
+ */
+ @Test def testBase64_3Byte() {
+ compare("QQQ", "UVFR")
+ }
+
+}
diff --git a/daffodil-io/src/test/scala/org/apache/daffodil/layers/TestJavaIOStreams.scala b/daffodil-io/src/test/scala/org/apache/daffodil/layers/TestJavaIOStreams.scala
new file mode 100644
index 0000000..e8d659b
--- /dev/null
+++ b/daffodil-io/src/test/scala/org/apache/daffodil/layers/TestJavaIOStreams.scala
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.layers
+
+import junit.framework.Assert._
+import org.junit.Test
+import org.apache.commons.io.IOUtils
+import collection.JavaConverters._
+import org.apache.daffodil.exceptions.Assert
+import java.util.Scanner
+import java.nio.charset.StandardCharsets
+import java.io.BufferedInputStream
+import java.io.ByteArrayInputStream
+import java.io.ByteArrayOutputStream
+
+/**
+ * Characterizes the behavior of Java's java.io.InputStream and java.io.OutputStream
+ * that we depend upon to get layering to work properly for stream-oriented
+ * behaviors.
+ *
+ * For example, we depend on a principle called "precise consumption of bytes",
+ * that is, say I am reading input from a gzip stream, which is itself consuming
+ * from another input stream. We depend on the fact that the gzip stream, once it
+ * finishes producing data, it has not consumed any more data from the underlying
+ * input stream than is necessary to produce the gzip output.
+ *
+ * Under the right circumstances, and with some performance penalty (possibly substantial)
+ * we can work around misbehavior here and force the proper behavior.
+ *
+ * Purpose of these tests is to determine if this forcing structure is needed, and
+ * to insure that if the behavior of underlying Java I/O streams changes in such
+ * a way that our assumptions are no longer valid, then these tests break, rather than
+ * just some subtle failure in Daffodil that is hard to detect and debug.
+ */
+class TestJavaIOStreams {
+ Assert.usage(scala.util.Properties.isJavaAtLeast("1.8"))
+
+ val text = """Man is distinguished, not only by his reason, but by this singular passion from
+other animals, which is a lust of the mind, that by a perseverance of delight
+in the continued and indefatigable generation of knowledge, exceeds the short
+vehemence of any carnal pleasure.""".replace("\n", " ")
+
+ val b64Text = """TWFuIGlzIGRpc3Rpbmd1aXNoZWQsIG5vdCBvbmx5IGJ5IGhpcyByZWFzb24sIGJ1dCBieSB0aGlz
+IHNpbmd1bGFyIHBhc3Npb24gZnJvbSBvdGhlciBhbmltYWxzLCB3aGljaCBpcyBhIGx1c3Qgb2Yg
+dGhlIG1pbmQsIHRoYXQgYnkgYSBwZXJzZXZlcmFuY2Ugb2YgZGVsaWdodCBpbiB0aGUgY29udGlu
+dWVkIGFuZCBpbmRlZmF0aWdhYmxlIGdlbmVyYXRpb24gb2Yga25vd2xlZGdlLCBleGNlZWRzIHRo
+ZSBzaG9ydCB2ZWhlbWVuY2Ugb2YgYW55IGNhcm5hbCBwbGVhc3VyZS4="""
+
+ val zipped = {
+ val baos = new ByteArrayOutputStream()
+ val gzs = new java.util.zip.GZIPOutputStream(baos)
+ IOUtils.write(text, gzs, StandardCharsets.ISO_8859_1)
+ gzs.close()
+ val bytes = baos.toByteArray()
+ bytes
+ }
+
+ val additionalText = "This is text that shouldn't be read."
+
+ /**
+ * Insures that after a base64MimeDecoder is done decoding, it leaves the underlying
+ * stream exactly before the next byte.
+ *
+ * Alas, this depends on the base64 data ending with an "=" padding character.
+ * If the data happens to decode precisely (multiple of 3 characters long) without
+ * any needed padding, then the decoder doesn't know to stop.
+ */
+ @Test def testBase64MIMEDecoderWrapDoesNotPreBuffer() {
+ val inputStream = IOUtils.toInputStream(b64Text + additionalText, StandardCharsets.ISO_8859_1)
+ val expected = text
+ val decodedStream = java.util.Base64.getMimeDecoder().wrap(inputStream)
+ val lines = IOUtils.readLines(decodedStream, StandardCharsets.ISO_8859_1).asScala.toSeq
+ assertEquals(expected, lines(0))
+ val additionalLines = IOUtils.readLines(inputStream, StandardCharsets.ISO_8859_1).asScala.toSeq
+ assertEquals(1, additionalLines.length)
+ assertEquals(additionalText, additionalLines(0))
+ }
+
+ /**
+ * Insures that base64MimeDecoder detects end of base64 without reading ahead.
+ *
+ * Does this by putting two base64 regions back to back without intervening
+ * bytes. Again however, this only works because the base64 data happened
+ * to end with an "=". That won't necessarily be the case.
+ */
+ @Test def testBase64MIMEDecoderDetectsEndRobustly1() {
+ val inputStream = IOUtils.toInputStream(b64Text + b64Text, StandardCharsets.ISO_8859_1)
+ val expected = text
+ val decodedStream = java.util.Base64.getMimeDecoder().wrap(inputStream)
+ val lines = IOUtils.readLines(decodedStream, StandardCharsets.ISO_8859_1).asScala.toSeq
+ assertEquals(expected, lines(0))
+ val decodedStream2 = java.util.Base64.getMimeDecoder().wrap(inputStream)
+ val additionalLines = IOUtils.readLines(decodedStream2, StandardCharsets.ISO_8859_1).asScala.toSeq
+ assertEquals(expected, additionalLines(0))
+ }
+
+ @Test def testBase64ScanningForDelimiter1() {
+ val data = "cGxl" // encoding of "ple"
+ val terminator = ";"
+ val afterTerminator = "afterTerminator"
+ val is = IOUtils.toInputStream(data + terminator + afterTerminator, "ascii").asInstanceOf[ByteArrayInputStream]
+ val scanner = new Scanner(is, StandardCharsets.ISO_8859_1.name())
+ is.skip(3)
+ is.mark(2)
+ val matchString = scanner.findWithinHorizon("(.*?)(?=(\\Q;\\E))", 2)
+ is.reset()
+ assertEquals("l", matchString)
+ assertEquals(";", scanner.`match`().group(2))
+ }
+
+ /**
+ * Characterizes behavior of GZIP input stream in Java.
+ *
+ * It doesn't have precise ending behavior. It reads
+ * ahead at least two bytes beyond the data it needs.
+ */
+ @Test def testGZIPDecoderDoesPreBuffer1() {
+ val inputData = zipped ++ additionalText.getBytes(StandardCharsets.ISO_8859_1)
+ val inputStream = new ByteArrayInputStream(inputData)
+ val expected = text
+ //
+ // Even with a buffer size of 1, the gzip input stream still consumes beyond
+ // the end of the stream.
+ //
+ val gzipBufferSize = 1
+ val decodedStream = new java.util.zip.GZIPInputStream(inputStream, gzipBufferSize)
+ val lines = IOUtils.readLines(decodedStream, StandardCharsets.ISO_8859_1).asScala.toSeq
+ lines.foreach { println }
+ assertEquals(1, lines.length)
+ assertEquals(expected, lines(0))
+ val additionalLines = IOUtils.readLines(inputStream, StandardCharsets.ISO_8859_1).asScala.toSeq
+ assertEquals(1, additionalLines.length)
+ assertEquals(additionalText.drop(2), additionalLines(0))
+ }
+
+ /**
+ * Characterizes behavior of GZIP input stream in Java.
+ *
+ * Same as above, but uses a BufferedInputStream - shows that GZIP isn't going
+ * to reset the input back to immediately after the last required byte.
+ *
+ * Our conclusion is that it will require explicit length, or some way of isolating
+ * the length of the compressed data so that we can limit how many bytes the gzip
+ * stream can read.
+ */
+ @Test def testGZIPDecoderDoesPreBuffer2() {
+ val inputData = zipped ++ additionalText.getBytes(StandardCharsets.ISO_8859_1)
+ val rawInput = new ByteArrayInputStream(inputData)
+ val inputStream = new BufferedInputStream(rawInput)
+ val expected = text
+ //
+ // Even with a buffer size of 1, and consuming from a buffered input stream where
+ // it could, in principle, back up to push back the bytes it didn't need,
+ // the gzip input stream still consumes beyond
+ // the end of the data it actually needs.
+ //
+ val gzipBufferSize = 1
+ val decodedStream = new java.util.zip.GZIPInputStream(inputStream, gzipBufferSize)
+ val lines = IOUtils.readLines(decodedStream, StandardCharsets.ISO_8859_1).asScala.toSeq
+ lines.foreach { println }
+ assertEquals(1, lines.length)
+ assertEquals(expected, lines(0))
+ val additionalLines = IOUtils.readLines(inputStream, StandardCharsets.ISO_8859_1).asScala.toSeq
+ assertEquals(1, additionalLines.length)
+ assertEquals(additionalText.drop(2), additionalLines(0))
+ }
+
+}
+
diff --git a/daffodil-io/src/test/scala/org/apache/daffodil/layers/TestLimitingJavaIOStreams.scala b/daffodil-io/src/test/scala/org/apache/daffodil/layers/TestLimitingJavaIOStreams.scala
new file mode 100644
index 0000000..d5efd06
--- /dev/null
+++ b/daffodil-io/src/test/scala/org/apache/daffodil/layers/TestLimitingJavaIOStreams.scala
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.layers
+
+import junit.framework.Assert._
+import org.junit.Test
+import org.apache.commons.io.IOUtils
+import collection.JavaConverters._
+import org.apache.daffodil.exceptions.Assert
+import java.nio.charset.StandardCharsets
+import org.apache.daffodil.io.ExplicitLengthLimitingStream
+import org.apache.daffodil.io.BoundaryMarkLimitingStream
+import java.io.ByteArrayInputStream
+import java.io.ByteArrayOutputStream
+import org.apache.daffodil.io.RegexLimitingStream
+
+/**
+ * Tests our layering java io streams. These are supposed to implement
+ * a principle called "precise consumption of bytes",
+ * that is, say I am reading input from a gzip stream, which is itself consuming
+ * from another input stream. We depend on the fact that the gzip stream, once it
+ * finishes producing data, it has not consumed any more data from the underlying
+ * input stream than is necessary to produce the gzip output.
+ *
+ * The layering streams have to constrain the data available to these overlying
+ * layer streams in order to insure the precise consumptino of bytes principle is
+ * upheld, because the java io streams for Base64, gzip, etc. do internal buffering
+ * and such, so are not compatible with this principle by themselves.
+ *
+ */
+class TestLimitingJavaIOStreams {
+ Assert.usage(scala.util.Properties.isJavaAtLeast("1.8"))
+
+ val iso8859 = StandardCharsets.ISO_8859_1
+ val utf8 = StandardCharsets.ISO_8859_1
+
+ val text = """Man is distinguished, not only by his reason, but by this singular passion from
+other animals, which is a lust of the mind, that by a perseverance of delight
+in the continued and indefatigable generation of knowledge, exceeds the short
+vehemence of any carnal pleasure.""".replace("\n", " ")
+
+ val b64Text = """TWFuIGlzIGRpc3Rpbmd1aXNoZWQsIG5vdCBvbmx5IGJ5IGhpcyByZWFzb24sIGJ1dCBieSB0aGlz
+IHNpbmd1bGFyIHBhc3Npb24gZnJvbSBvdGhlciBhbmltYWxzLCB3aGljaCBpcyBhIGx1c3Qgb2Yg
+dGhlIG1pbmQsIHRoYXQgYnkgYSBwZXJzZXZlcmFuY2Ugb2YgZGVsaWdodCBpbiB0aGUgY29udGlu
+dWVkIGFuZCBpbmRlZmF0aWdhYmxlIGdlbmVyYXRpb24gb2Yga25vd2xlZGdlLCBleGNlZWRzIHRo
+ZSBzaG9ydCB2ZWhlbWVuY2Ugb2YgYW55IGNhcm5hbCBwbGVhc3VyZS4="""
+
+ val zipped = {
+ val baos = new ByteArrayOutputStream()
+ val gzs = new java.util.zip.GZIPOutputStream(baos)
+ IOUtils.write(text, gzs, iso8859)
+ gzs.close()
+ val bytes = baos.toByteArray()
+ bytes
+ }
+
+ val additionalText = "This is text that shouldn't be read."
+
+ @Test def testBase64DecodeFromDelimitedStream1() {
+ val data = "cGxlYXN1cmUu" // encoding of "pleasure."
+ val terminator = "terminator"
+ val afterTerminator = "afterTerminator"
+ val is = IOUtils.toInputStream(data + terminator + afterTerminator,
+ StandardCharsets.UTF_8).asInstanceOf[ByteArrayInputStream]
+ val delimitedStream =
+ BoundaryMarkLimitingStream(is, terminator, iso8859, targetChunkSize = 4)
+ val b64 = java.util.Base64.getMimeDecoder().wrap(delimitedStream)
+ val actualData = IOUtils.toString(b64, iso8859)
+ assertEquals("pleasure.", actualData)
+ // The input stream should have been advanced past the terminator
+ val actualAfterData = IOUtils.toString(is, iso8859)
+ assertEquals(afterTerminator, actualAfterData)
+ }
+
+ @Test def testBase64DecodeFromDelimitedStream2() {
+ val data = "cGxlYXN1cmUu" // encoding of "pleasure."
+ val terminator = ";;;"
+ val afterTerminator = "afterTerminator"
+ val is = IOUtils.toInputStream(data + terminator + afterTerminator, "ascii").asInstanceOf[ByteArrayInputStream]
+ val delimitedStream =
+ BoundaryMarkLimitingStream(is, terminator, iso8859, targetChunkSize = 1)
+ val b64 = java.util.Base64.getMimeDecoder().wrap(delimitedStream)
+ val actualData = IOUtils.toString(b64, iso8859)
+ assertEquals("pleasure.", actualData)
+ val actualAfterData = IOUtils.toString(is, iso8859)
+ assertEquals(afterTerminator, actualAfterData)
+ }
+
+ @Test def testBase64DecodeFromDelimitedStream3() {
+ val data = "cGxl" // encoding of "ple"
+ val terminator = ";"
+ val afterTerminator = "afterTerminator"
+ val is = IOUtils.toInputStream(data + terminator + afterTerminator, "ascii").asInstanceOf[ByteArrayInputStream]
+ val delimitedStream =
+ BoundaryMarkLimitingStream(is, terminator, iso8859, targetChunkSize = 1)
+ val b64 = java.util.Base64.getMimeDecoder().wrap(delimitedStream)
+ val actualData = IOUtils.toString(b64, iso8859)
+ assertEquals("ple", actualData)
+ val actualAfterData = IOUtils.toString(is, iso8859)
+ assertEquals(afterTerminator, actualAfterData)
+ }
+
+ /**
+ * Illustrates that if we clamp the gzip stream at a finite length,
+ * that we can then do precise consumption of bytes, as would be expected.
+ *
+ * So this is the technique needed for GZIP, since otherwise it reads too far ahead.
+ * Unlike Java 8 Base64, which stops when it self-detects the last byte of the
+ * encoded data, GZIP doesn't do that. It seems to always read past the end. This
+ * may be an artifact of the implementation, or inherent in the GZIP data format.
+ */
+ @Test def testGZIPDecoderWithLimit1() {
+ val inputData = zipped ++ additionalText.getBytes(iso8859)
+ val inputStream = new ByteArrayInputStream(inputData)
+ val limitedStream = new ExplicitLengthLimitingStream(inputStream,
+ zipped.length)
+ val expected = text
+
+ val decodedStream = new java.util.zip.GZIPInputStream(limitedStream, 5)
+ val lines = IOUtils.readLines(decodedStream, iso8859).asScala.toSeq
+ lines.foreach { println }
+ assertEquals(1, lines.length)
+ assertEquals(expected, lines(0))
+ val additionalLines = IOUtils.readLines(inputStream, iso8859).asScala.toSeq
+ assertEquals(1, additionalLines.length)
+ assertEquals(additionalText, additionalLines(0))
+ }
+
+ /**
+ * Test for lines of text containing \r\n\t or \r\n\x20 (CRLF + space)
+ * but terminated by a \r\n NOT followed by \t or space.
+ */
+ @Test def testRegexDelimStream1() = {
+ val beforeDelim = "12345\r\n\t67890\r\n\tabcde"
+ val delim = "\r\n"
+ val afterDelim = "fghij"
+ val inputString = beforeDelim + delim + afterDelim
+ val inputBytes = inputString.getBytes("utf-8")
+ val bais = new ByteArrayInputStream(inputBytes)
+ val rls = new RegexLimitingStream(bais, "\\r\\n(?!(?:\\t|\\ ))", "\r\n", utf8)
+ val baos = new ByteArrayOutputStream()
+ var c: Int = -1
+ while ({
+ c = rls.read()
+ c != -1
+ }) {
+ baos.write(c)
+ }
+ baos.close()
+ val actualBeforeDelim = new String(baos.toByteArray())
+ val afterBaos = new ByteArrayOutputStream()
+ while ({
+ c = bais.read()
+ c != -1
+ }) {
+ afterBaos.write(c)
+ }
+ afterBaos.close()
+ val actualAfterDelim = new String(afterBaos.toByteArray())
+ assertEquals(beforeDelim, actualBeforeDelim)
+ assertEquals(afterDelim, actualAfterDelim)
+
+ }
+
+ /**
+ * Shows that the match, if the delim regex isn't matched at all,
+ * contains the entire available ipput data.
+ */
+ @Test def testRegexDelimStream2() = {
+ val beforeDelim = "12345\r\n\t67890\r\n\tabcde"
+
+ val inputString = beforeDelim
+ val inputBytes = inputString.getBytes("utf-8")
+ val bais = new ByteArrayInputStream(inputBytes)
+ val rls = new RegexLimitingStream(bais, "\\r\\n(?!(?:\\t|\\ ))", "\r\n\t", utf8, 4)
+ val baos = new ByteArrayOutputStream()
+ var c: Int = -1
+ while ({
+ c = rls.read()
+ c != -1
+ }) {
+ baos.write(c)
+ }
+ baos.close()
+ val actualBeforeDelim = new String(baos.toByteArray())
+ assertEquals(beforeDelim, actualBeforeDelim)
+ }
+}
+
diff --git a/daffodil-lib/src/main/scala/org/apache/daffodil/cookers/Cookers.scala b/daffodil-lib/src/main/scala/org/apache/daffodil/cookers/Cookers.scala
index 63a88c1..ad3dfdc 100644
--- a/daffodil-lib/src/main/scala/org/apache/daffodil/cookers/Cookers.scala
+++ b/daffodil-lib/src/main/scala/org/apache/daffodil/cookers/Cookers.scala
@@ -80,3 +80,5 @@ object EncodingCooker extends UpperCaseToken()
object ChoiceDispatchKeyCooker extends StringLiteralNonEmptyNoCharClassEntitiesNoByteEntities()
object ChoiceBranchKeyCooker extends ListOfStringLiteralNonEmptyNoCharClassEntitiesNoByteEntities()
+
+object UpperCaseTokenCooker extends UpperCaseToken
diff --git a/daffodil-lib/src/main/scala/org/apache/daffodil/xml/XMLUtils.scala b/daffodil-lib/src/main/scala/org/apache/daffodil/xml/XMLUtils.scala
index c186212..2f6df38 100644
--- a/daffodil-lib/src/main/scala/org/apache/daffodil/xml/XMLUtils.scala
+++ b/daffodil-lib/src/main/scala/org/apache/daffodil/xml/XMLUtils.scala
@@ -651,7 +651,7 @@ object XMLUtils {
val newChildren: NodeSeq = children.flatMap { removeAttributes1(_, ns, Some(newScope)) }
// Important to merge adjacent text. Otherwise when comparing
- // two structuers that print out the same, they might not be equal
+ // two structures that print out the same, they might not be equal
// because they have different length lists of text nodes
//
// Ex: <foo>A</foo> creates an element containing TWO
@@ -660,6 +660,22 @@ object XMLUtils {
// Similarly <foo>abc<![CDATA[def]]>ghi</foo> has 3 child nodes.
// The middle one is PCData. The two around it are Text.
// Both Text and PCData are Atom[String].
+
+ // Note: as of 2018-04-30, Mike Beckerle said: I am unable to reproduce the above.
+ // The first example: <foo>A</foo>.child returns an array buffer with 1 child in it
+ // which is a Text node. The <foo>abc<![CDATA[def]]>ghi</foo> also has only
+ // one Text node. That said, this is from typing them at the scala shell.
+ //
+ // I suspect the above observations require that the scala.xml.parsing.ConstructingParser
+ // is used. We do use this, because while the regular XML loader coalesces
+ // text nodes well, but doesn't preserve whitespace for CDATA regions well. That's why we use the
+ // scala.xml.parser.ConstructingParser, which doesn't coalesce text nodes
+ // so well, and that's what motivates this explicit coalesce pass.
+ //
+ // See test test_scala_loader_cdata_bug - which characterizes the behavior
+ // that is problematic for us in the standard loader, and why we have to use
+ // the ConstructingParser.
+ //
val textMergedChildren = coalesceAdjacentTextNodes(newChildren)
val newPrefix = if (prefixInScope(prefix, newScope)) prefix else null
diff --git a/daffodil-propgen/src/main/resources/org/apache/daffodil/xsd/DFDL_part1_simpletypes.xsd b/daffodil-propgen/src/main/resources/org/apache/daffodil/xsd/DFDL_part1_simpletypes.xsd
index ec5d6bc..8ed5b82 100644
--- a/daffodil-propgen/src/main/resources/org/apache/daffodil/xsd/DFDL_part1_simpletypes.xsd
+++ b/daffodil-propgen/src/main/resources/org/apache/daffodil/xsd/DFDL_part1_simpletypes.xsd
@@ -671,4 +671,135 @@
</xsd:union>
</xsd:simpleType>
+ <xsd:simpleType name="LayerTransformType_Or_DFDLExpression">
+ <xsd:union>
+ <xsd:simpleType>
+ <xsd:restriction base="dfdl:DFDLExpression" />
+ </xsd:simpleType>
+ <xsd:simpleType>
+ <xsd:restriction base="dfdl:LayerTransformEnum" />
+ </xsd:simpleType>
+ </xsd:union>
+ </xsd:simpleType>
+
+ <xsd:simpleType name="LayerTransformEnum">
+ <xsd:restriction base="xsd:token">
+ <xsd:enumeration value="base64_MIME">
+ <xsd:annotation>
+ <xsd:documentation>
+ IETF RFC 2045
+
+ Max line length 76 characters.
+ </xsd:documentation>
+ </xsd:annotation>
+ </xsd:enumeration>
+ <xsd:enumeration value="base64">
+ <xsd:annotation>
+ <xsd:documentation>
+ IETF RFC 4648 - not the URL-SAFE version.
+
+ Must be strict about what is accepted.
+ Does not insert CRLFs. No line length limitation.
+
+ Other specifications refer to this RFC and impose
+ their own rules (as MIME does) about line lengths, and
+ characters allowed.
+ </xsd:documentation>
+ </xsd:annotation>
+ </xsd:enumeration>
+ <xsd:enumeration value="base64url">
+ <xsd:annotation>
+ <xsd:documentation>
+ IETF RFC 4648 - the URL-SAFE version.
+
+ The name base64url is suggested in the RFC as the name for this
+ encoding.
+ </xsd:documentation>
+ </xsd:annotation>
+ </xsd:enumeration>
+ <xsd:enumeration value="lineFolded_IMF">
+ <xsd:annotation>
+ <xsd:documentation><![CDATA[
+ IETF RFC 2822 Internet Message Format (IMF)
+
+ Each line of characters MUST be no more than
+ 998 characters, and SHOULD be no more than 78 characters, excluding
+ the CRLF.
+
+ Though structured field bodies are defined in such a way that
+ folding can take place between many of the lexical tokens (and even
+ within some of the lexical tokens), folding SHOULD be limited to
+ placing the CRLF at higher-level syntactic breaks. For instance, if
+ a field body is defined as comma-separated values, it is recommended
+ that folding occur after the comma separating the structured items in
+ preference to other places where the field could be folded, even if
+ it is allowed elsewhere.
+
+ Unfolding is accomplished by simply removing any CRLF
+ that is immediately followed by WSP.
+ ]]>
+ </xsd:documentation>
+ </xsd:annotation>
+ </xsd:enumeration>
+ <xsd:enumeration value="lineFolded_iCalendar">
+ <xsd:annotation>
+ <xsd:documentation><![CDATA[
+ IETF RFC 5545 Internet Calendaring and Scheduling (iCalendar)
+
+ Lines of text SHOULD NOT be longer than 75 octets, excluding the line
+ break. Long content lines SHOULD be split into a multiple line
+ representations using a line "folding" technique. That is, a long
+ line can be split between any two characters by inserting a CRLF
+ immediately followed by a single linear white-space character (i.e.,
+ SPACE or HTAB). Any sequence of CRLF followed immediately by a
+ single linear white-space character is ignored (i.e., removed) when
+ processing the content type.
+ ]]>
+ </xsd:documentation>
+ </xsd:annotation>
+ </xsd:enumeration>
+ <xsd:enumeration value="quotedPrintable">
+ <xsd:annotation>
+ <xsd:documentation>
+ IETF RFC 2045 Quoted Printable Content Transfer Encoding
+ </xsd:documentation>
+ </xsd:annotation>
+ </xsd:enumeration>
+ <xsd:enumeration value="ais">
+ <xsd:annotation>
+ <xsd:documentation>
+ Automatic Identification System - ITU-R M.1371-1
+ </xsd:documentation>
+ </xsd:annotation>
+ </xsd:enumeration>
+ <xsd:enumeration value="compress">
+ <xsd:annotation>
+ <xsd:documentation>
+ Lempel-Ziv-Welch compression per Unix 'compress' command.
+ </xsd:documentation>
+ </xsd:annotation>
+ </xsd:enumeration>
+ <xsd:enumeration value="gzip">
+ <xsd:annotation>
+ <xsd:documentation>
+ GZIP per https://www.ietf.org/rfc/rfc1952.txt
+ </xsd:documentation>
+ </xsd:annotation>
+ </xsd:enumeration>
+ </xsd:restriction>
+ </xsd:simpleType>
+
+ <xsd:simpleType name="LayerLengthUnitsEnum">
+ <xsd:restriction base="xsd:string">
+ <xsd:enumeration value="bytes" />
+ </xsd:restriction>
+ </xsd:simpleType>
+
+ <xsd:simpleType name="LayerLengthKindEnum">
+ <xsd:restriction base="xsd:string">
+ <xsd:enumeration value="explicit" />
+ <xsd:enumeration value="boundaryMark" />
+ <xsd:enumeration value="implicit" />
+ </xsd:restriction>
+ </xsd:simpleType>
</xsd:schema>
diff --git a/daffodil-propgen/src/main/resources/org/apache/daffodil/xsd/DFDL_part2_attributes.xsd b/daffodil-propgen/src/main/resources/org/apache/daffodil/xsd/DFDL_part2_attributes.xsd
index 38f1878..b1b8416 100644
--- a/daffodil-propgen/src/main/resources/org/apache/daffodil/xsd/DFDL_part2_attributes.xsd
+++ b/daffodil-propgen/src/main/resources/org/apache/daffodil/xsd/DFDL_part2_attributes.xsd
@@ -375,6 +375,30 @@
<xsd:attribute name="sequenceKind" type="dfdl:SequenceKindEnum" />
<xsd:attribute name="hiddenGroupRef" type="xsd:QName" />
</xsd:attributeGroup>
+
+ <xsd:attribute name="layerTransform"
+ type="dfdl:LayerTransformType_Or_DFDLExpression" />
+ <xsd:attribute name="layerEncoding"
+ type="dfdl:EncodingEnum_Or_DFDLExpression" />
+ <xsd:attribute name="layerLengthKind" type="dfdl:LayerLengthKindEnum" />
+ <xsd:attribute name="layerLength"
+ type="dfdl:DFDLNonNegativeInteger_Or_DFDLExpression" />
+ <xsd:attribute name="layerLengthUnits" type="dfdl:LayerLengthUnitsEnum" />
+ <xsd:attribute name="layerBoundaryMark"
+ type="dfdl:ListOfDFDLStringLiteral_Or_DFDLExpression" />
+
+ <xsd:attributeGroup name="LayeringAG">
+ <xsd:attribute name="layerTransform"
+ type="dfdl:LayerTransformType_Or_DFDLExpression" />
+ <xsd:attribute name="layerEncoding"
+ type="dfdl:EncodingEnum_Or_DFDLExpression" />
+ <xsd:attribute name="layerLengthKind" type="dfdl:LayerLengthKindEnum" />
+ <xsd:attribute name="layerLength"
+ type="dfdl:DFDLNonNegativeInteger_Or_DFDLExpression" />
+ <xsd:attribute name="layerLengthUnits" type="dfdl:LayerLengthUnitsEnum" />
+ <xsd:attribute name="layerBoundaryMark"
+ type="dfdl:ListOfDFDLStringLiteral_Or_DFDLExpression" />
+ </xsd:attributeGroup>
<!-- 14.2 Sequence Groups with Delimiters -->
<xsd:attribute name="separator"
@@ -558,6 +582,14 @@
<xsd:enumeration value="sequenceKind" />
<xsd:enumeration value="hiddenGroupRef" />
+
+ <xsd:enumeration value="layerTransform"/>
+ <xsd:enumeration value="layerEncoding" />
+ <xsd:enumeration value="layerLengthKind" />
+ <xsd:enumeration value="layerLength" />
+ <xsd:enumeration value="layerLengthUnits" />
+ <xsd:enumeration value="layerBoundaryMark" />
+
<xsd:enumeration value="initiatedContent" />
<xsd:enumeration value="separatorPosition" />
@@ -954,6 +986,7 @@
<xsd:attributeGroup name="GroupAGQualified">
<xsd:attributeGroup ref="dfdl:GroupCommonAGQualified" />
<xsd:attributeGroup ref="dfdl:SequenceAGQualified" />
+ <xsd:attributeGroup ref="dfdl:LayeringAGQualified" />
<xsd:attributeGroup ref="dfdl:ChoiceAGQualified" />
<xsd:attributeGroup ref="dfdl:SeparatorAGQualified" />
</xsd:attributeGroup>
@@ -977,4 +1010,17 @@
<xsd:attributeGroup ref="dfdl:BooleanBinaryAGQualified" />
</xsd:attributeGroup>
+ <xsd:attributeGroup name="LayeringAGQualified">
+ <xsd:attribute form="qualified" name="layerTransform"
+ type="dfdl:LayerTransformType_Or_DFDLExpression" />
+ <xsd:attribute form="qualified" name="layerEncoding"
+ type="dfdl:EncodingEnum_Or_DFDLExpression" />
+ <xsd:attribute form="qualified" name="layerLengthKind" type="dfdl:LayerLengthKindEnum" />
+ <xsd:attribute form="qualified" name="layerLength"
+ type="dfdl:DFDLNonNegativeInteger_Or_DFDLExpression" />
+ <xsd:attribute form="qualified" name="layerLengthUnits" type="dfdl:LayerLengthUnitsEnum" />
+ <xsd:attribute form="qualified" name="layerBoundaryMark"
+ type="dfdl:ListOfDFDLStringLiteral_Or_DFDLExpression" />
+ </xsd:attributeGroup>
+
</xsd:schema>
diff --git a/daffodil-propgen/src/main/resources/org/apache/daffodil/xsd/DFDL_part3_model.xsd b/daffodil-propgen/src/main/resources/org/apache/daffodil/xsd/DFDL_part3_model.xsd
index a1cafe5..e7cc041 100644
--- a/daffodil-propgen/src/main/resources/org/apache/daffodil/xsd/DFDL_part3_model.xsd
+++ b/daffodil-propgen/src/main/resources/org/apache/daffodil/xsd/DFDL_part3_model.xsd
@@ -247,6 +247,7 @@
<xsd:sequence />
<xsd:attributeGroup ref="dfdl:GroupCommonAG" />
<xsd:attributeGroup ref="dfdl:SequenceAG" />
+ <xsd:attributeGroup ref="dfdl:LayeringAG" />
<xsd:attributeGroup ref="dfdl:SeparatorAG" />
</xsd:extension>
</xsd:complexContent>
@@ -307,6 +308,7 @@
<xsd:attributeGroup name="GroupAG">
<xsd:attributeGroup ref="dfdl:GroupCommonAG" />
<xsd:attributeGroup ref="dfdl:SequenceAG" />
+ <xsd:attributeGroup ref="dfdl:LayeringAG" />
<xsd:attributeGroup ref="dfdl:ChoiceAG" />
<xsd:attributeGroup ref="dfdl:SeparatorAG" />
</xsd:attributeGroup>
diff --git a/daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/processors/unparsers/ElementKindUnparsers.scala b/daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/processors/unparsers/ElementKindUnparsers.scala
index 71ff9ca..08af746 100644
--- a/daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/processors/unparsers/ElementKindUnparsers.scala
+++ b/daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/processors/unparsers/ElementKindUnparsers.scala
@@ -50,7 +50,7 @@ class SequenceCombinatorUnparser(ctxt: ModelGroupRuntimeData, childUnparsers: Ve
extends CombinatorUnparser(ctxt)
with ToBriefXMLImpl {
- override lazy val runtimeDependencies = Nil
+ override lazy val runtimeDependencies: Seq[Evaluatable[AnyRef]] = Nil
override def nom = "Sequence"
diff --git a/daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/processors/unparsers/LayeredSequenceUnparser.scala b/daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/processors/unparsers/LayeredSequenceUnparser.scala
new file mode 100644
index 0000000..ee456e3
--- /dev/null
+++ b/daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/processors/unparsers/LayeredSequenceUnparser.scala
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.processors.unparsers
+
+import org.apache.daffodil.processors.{ LayerTransformerEv, ModelGroupRuntimeData }
+import org.apache.daffodil.io.DirectOrBufferedDataOutputStream
+
+class LayeredSequenceUnparser(ctxt: ModelGroupRuntimeData,
+ layerTransformerEv: LayerTransformerEv,
+ childUnparser: Unparser)
+ extends SequenceCombinatorUnparser(ctxt, Vector(childUnparser)) {
+
+ override lazy val runtimeDependencies = Seq(layerTransformerEv)
+
+ override def nom = "LayeredSequence"
+
+ override def unparse(state: UState): Unit = {
+ val layerTransformer = layerTransformerEv.evaluate(state)
+
+ val originalDOS = state.dataOutputStream // layer will output to the original, then finish it upon closing.
+
+ val newDOS = originalDOS.addBuffered // newDOS is where unparsers after this one returns will unparse into.
+
+ //
+ // FIXME: Cast should not be necessary
+ //
+ // New layerDOS is where the layer will unparse into. Ultimately anything written
+ // to layerDOS ends up, post transform, in originalDOS.
+ //
+ val layerDOS = layerTransformer.addLayer(originalDOS, state).asInstanceOf[DirectOrBufferedDataOutputStream]
+
+ // unparse the layer body into layerDOS
+ state.dataOutputStream = layerDOS
+ super.unparse(state)
+
+ // now we're done with the layer, so finalize the layer
+ layerDOS.lastInChain.setFinished(state)
+
+ // clean up resources - note however, that due to suspensions, the whole
+ // layer stack is potentially still needed, so not clear what can be
+ // cleaned up at this point.
+ //
+ layerTransformer.removeLayer(layerDOS, state)
+ state.dataOutputStream = newDOS
+ }
+
+}
+
diff --git a/daffodil-runtime1/src/main/scala/org/apache/daffodil/infoset/InfosetImpl.scala b/daffodil-runtime1/src/main/scala/org/apache/daffodil/infoset/InfosetImpl.scala
index 9aeee2c..1aaabeb 100644
--- a/daffodil-runtime1/src/main/scala/org/apache/daffodil/infoset/InfosetImpl.scala
+++ b/daffodil-runtime1/src/main/scala/org/apache/daffodil/infoset/InfosetImpl.scala
@@ -1332,6 +1332,8 @@ sealed class DIComplex(override val erd: ElementRuntimeData, val tunable: Daffod
}
val childNodes = new ArrayBuffer[DINode]
+ //
+ // TODO: Cleanup - Change below to use NonAllocatingMap to improve code style.
lazy val nameToChildNodeLookup = new HashMap[NamedQName, ArrayBuffer[DINode]]
override lazy val contents: IndexedSeq[DINode] = childNodes
diff --git a/daffodil-runtime1/src/main/scala/org/apache/daffodil/infoset/ScalaXMLInfosetInputter.scala b/daffodil-runtime1/src/main/scala/org/apache/daffodil/infoset/ScalaXMLInfosetInputter.scala
index 3329de3..a6dded6 100644
--- a/daffodil-runtime1/src/main/scala/org/apache/daffodil/infoset/ScalaXMLInfosetInputter.scala
+++ b/daffodil-runtime1/src/main/scala/org/apache/daffodil/infoset/ScalaXMLInfosetInputter.scala
@@ -27,6 +27,7 @@ import scala.xml.Text
import scala.xml.Node
import scala.xml.ProcInstr
import scala.xml.Comment
+import scala.xml.Atom
class ScalaXMLInfosetInputter(rootNode: Node)
extends InfosetInputter {
@@ -46,7 +47,7 @@ class ScalaXMLInfosetInputter(rootNode: Node)
* Start/EndDocument events.
*/
private val stack = {
- val s = new MStackOf[(Elem,Iterator[Node])]
+ val s = new MStackOf[(Elem, Iterator[Node])]
val iter = rootNode match {
case e: Elem => e.iterator
@@ -67,11 +68,11 @@ class ScalaXMLInfosetInputter(rootNode: Node)
}
override def getLocalName(): String = stack.top._1.label
-
+
override val supportsNamespaces = true
override def getNamespaceURI(): String = stack.top._1.namespace
-
+
override def getSimpleText(primType: NodeInfo.Kind): String = {
val text =
if (stack.top._2.hasNext) {
@@ -81,6 +82,18 @@ class ScalaXMLInfosetInputter(rootNode: Node)
}
val text = child match {
case t: Text => t.data
+ //
+ // Note: may be a bug in scala.xml library, but sometimes we are
+ // getting Atom[String] here, not Text. Some kinds of things we think
+ // of as text, like PCData nodes, are not derived from Text, but from
+ // Atom[String], so perhaps that is why.
+ //
+ // The above issue may be related to our use of the
+ // scala.xml.parsing.ConstructingParser which preserves PCData nodes
+ // well, and doesn't coalesce them into Text nodes incorrectly like
+ // the regular XML loader (in scala.xml 1.0.6) does.
+ //
+ case a: Atom[_] => a.data.toString()
case _ => throw new NonTextFoundInSimpleContentException(stack.top._1.label)
}
if (primType.isInstanceOf[NodeInfo.String.Kind]) {
@@ -100,7 +113,7 @@ class ScalaXMLInfosetInputter(rootNode: Node)
val nilAttrValueOpt = elem.attribute(XMLUtils.XSI_NAMESPACE, "nil")
val res =
if (nilAttrValueOpt.isEmpty) {
- MaybeBoolean.Nope
+ MaybeBoolean.Nope
} else {
val nilAttrValueSeq = nilAttrValueOpt.get
if (nilAttrValueSeq.length > 1) {
diff --git a/daffodil-runtime1/src/main/scala/org/apache/daffodil/layers/Base64Transformer.scala b/daffodil-runtime1/src/main/scala/org/apache/daffodil/layers/Base64Transformer.scala
new file mode 100644
index 0000000..01ad217
--- /dev/null
+++ b/daffodil-runtime1/src/main/scala/org/apache/daffodil/layers/Base64Transformer.scala
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.layers
+
+import org.apache.daffodil.schema.annotation.props.gen.LayerLengthKind
+import org.apache.daffodil.schema.annotation.props.gen.LayerLengthUnits
+import org.apache.daffodil.util.Maybe
+import org.apache.daffodil.processors.TermRuntimeData
+import org.apache.daffodil.processors.LayerLengthInBytesEv
+import org.apache.daffodil.processors.LayerBoundaryMarkEv
+import org.apache.daffodil.processors.LayerCharsetEv
+import org.apache.daffodil.io.BoundaryMarkLimitingStream
+import org.apache.daffodil.processors.parsers.PState
+import org.apache.daffodil.processors.charset.BitsCharsetWrappingJavaCharset
+import java.nio.charset.Charset
+import org.apache.daffodil.processors.charset.BitsCharset
+import org.apache.daffodil.exceptions.Assert
+import org.apache.daffodil.processors.unparsers.UState
+import org.apache.daffodil.io.LayerBoundaryMarkInsertingJavaOutputStream
+
+class Base64MIMETransformer(layerCharsetEv: LayerCharsetEv, layerBoundaryMarkEv: LayerBoundaryMarkEv)
+ extends LayerTransformer() {
+
+ override def wrapLayerDecoder(jis: java.io.InputStream): java.io.InputStream = {
+ val b64 = java.util.Base64.getMimeDecoder().wrap(jis)
+ b64
+ }
+
+ override def wrapLimitingStream(jis: java.io.InputStream, state: PState) = {
+ val layerCharset: BitsCharset = layerCharsetEv.evaluate(state)
+ val layerBoundaryMark = layerBoundaryMarkEv.evaluate(state)
+ val javaCharset: Charset = layerCharset match {
+ case jbcs: BitsCharsetWrappingJavaCharset => jbcs.javaCharset
+ case _ => Assert.invariantFailed("Not a java-compatible charset: " + layerCharset)
+ }
+ val s = BoundaryMarkLimitingStream(jis, layerBoundaryMark, javaCharset)
+ s
+ }
+
+ override protected def wrapLayerEncoder(jos: java.io.OutputStream): java.io.OutputStream = {
+ val b64 = java.util.Base64.getMimeEncoder().wrap(jos)
+ b64
+ }
+
+ override protected def wrapLimitingStream(jos: java.io.OutputStream, state: UState): java.io.OutputStream = {
+ val layerCharset: BitsCharset = layerCharsetEv.evaluate(state)
+ val layerBoundaryMark = layerBoundaryMarkEv.evaluate(state)
+ val javaCharset: Charset = layerCharset match {
+ case jbcs: BitsCharsetWrappingJavaCharset => jbcs.javaCharset
+ case _ => Assert.invariantFailed("Not a java-compatible charset: " + layerCharset)
+ }
+ val newJOS = new LayerBoundaryMarkInsertingJavaOutputStream(jos, layerBoundaryMark, javaCharset)
+ newJOS
+ }
+}
+
+object Base64MIMETransformerFactory
+ extends LayerTransformerFactory("base64_MIME") {
+
+ override def newInstance(maybeLayerCharsetEv: Maybe[LayerCharsetEv],
+ maybeLayerLengthKind: Maybe[LayerLengthKind],
+ maybeLayerLengthInBytesEv: Maybe[LayerLengthInBytesEv],
+ maybeLayerLengthUnits: Maybe[LayerLengthUnits],
+ maybeLayerBoundaryMarkEv: Maybe[LayerBoundaryMarkEv],
+ trd: TermRuntimeData): LayerTransformer = {
+
+ trd.schemaDefinitionUnless(scala.util.Properties.isJavaAtLeast("1.8"),
+ "Base64 layer support requires Java 8 (aka Java 1.8).")
+
+ trd.schemaDefinitionUnless(maybeLayerBoundaryMarkEv.isDefined,
+ "Property dfdl:layerBoundaryMark was not defined.")
+ trd.schemaDefinitionUnless(maybeLayerLengthKind.isEmpty ||
+ (maybeLayerLengthKind.get eq LayerLengthKind.BoundaryMark),
+ "Only dfdl:layerLengthKind 'boundaryMark' is supported, but '%s' was specified",
+ maybeLayerLengthKind.get.toString)
+
+ trd.schemaDefinitionUnless(maybeLayerCharsetEv.isDefined,
+ "Property dfdl:layerEncoding must be defined.")
+
+ val xformer = new Base64MIMETransformer(maybeLayerCharsetEv.get, maybeLayerBoundaryMarkEv.get)
+ xformer
+ }
+}
diff --git a/daffodil-runtime1/src/main/scala/org/apache/daffodil/layers/GZipTransformer.scala b/daffodil-runtime1/src/main/scala/org/apache/daffodil/layers/GZipTransformer.scala
new file mode 100644
index 0000000..1c8d4a5
--- /dev/null
+++ b/daffodil-runtime1/src/main/scala/org/apache/daffodil/layers/GZipTransformer.scala
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.layers
+
+import org.apache.daffodil.schema.annotation.props.gen.LayerLengthKind
+import org.apache.daffodil.schema.annotation.props.gen.LayerLengthUnits
+import org.apache.daffodil.util.Maybe
+import org.apache.daffodil.processors.TermRuntimeData
+import org.apache.daffodil.processors.LayerLengthInBytesEv
+import org.apache.daffodil.processors.LayerBoundaryMarkEv
+import org.apache.daffodil.processors.LayerCharsetEv
+import org.apache.daffodil.processors.parsers.PState
+import org.apache.daffodil.io.ExplicitLengthLimitingStream
+import org.apache.daffodil.processors.unparsers.UState
+
+class GZIPTransformer(layerLengthInBytesEv: LayerLengthInBytesEv)
+ extends LayerTransformer() {
+
+ override def wrapLayerDecoder(jis: java.io.InputStream) = {
+ val s = new java.util.zip.GZIPInputStream(jis)
+ s
+ }
+
+ override def wrapLimitingStream(jis: java.io.InputStream, state: PState) = {
+ val layerLengthInBytes: Int = layerLengthInBytesEv.evaluate(state).toInt
+ val s = new ExplicitLengthLimitingStream(jis, layerLengthInBytes)
+ s
+ }
+
+ override protected def wrapLayerEncoder(jos: java.io.OutputStream): java.io.OutputStream = {
+ val s = new java.util.zip.GZIPOutputStream(jos)
+ s
+ }
+
+ override protected def wrapLimitingStream(jis: java.io.OutputStream, state: UState): java.io.OutputStream = {
+ jis // just return jis. The way the length will be used/stored is by way of
+ // taking the content length of the enclosing element. That will measure the
+ // length relative to the "ultimate" data output stream.
+ }
+}
+
+object GZIPTransformerFactory
+ extends LayerTransformerFactory("gzip") {
+
+ override def newInstance(maybeLayerCharsetEv: Maybe[LayerCharsetEv],
+ maybeLayerLengthKind: Maybe[LayerLengthKind],
+ maybeLayerLengthInBytesEv: Maybe[LayerLengthInBytesEv],
+ maybeLayerLengthUnits: Maybe[LayerLengthUnits],
+ maybeLayerBoundaryMarkEv: Maybe[LayerBoundaryMarkEv],
+ trd: TermRuntimeData): LayerTransformer = {
+
+ val xformer = new GZIPTransformer(maybeLayerLengthInBytesEv.get)
+ xformer
+ }
+}
diff --git a/daffodil-runtime1/src/main/scala/org/apache/daffodil/layers/LayerTransformer.scala b/daffodil-runtime1/src/main/scala/org/apache/daffodil/layers/LayerTransformer.scala
new file mode 100644
index 0000000..a4a6bc2
--- /dev/null
+++ b/daffodil-runtime1/src/main/scala/org/apache/daffodil/layers/LayerTransformer.scala
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.layers
+
+import org.apache.daffodil.schema.annotation.props.gen.LayerLengthKind
+import org.apache.daffodil.schema.annotation.props.gen.LayerLengthUnits
+import org.apache.daffodil.processors.TermRuntimeData
+import org.apache.daffodil.processors.LayerLengthInBytesEv
+import org.apache.daffodil.processors.LayerBoundaryMarkEv
+import org.apache.daffodil.processors.LayerCharsetEv
+import java.util.HashMap
+import org.apache.daffodil.processors.ParseOrUnparseState
+import org.apache.daffodil.util.NonAllocatingMap
+import org.apache.daffodil.io.DataInputStream
+import org.apache.daffodil.io.DataOutputStream
+import org.apache.daffodil.io.ByteBufferDataInputStream
+import org.apache.daffodil.io.DataInputStream.Mark
+import org.apache.daffodil.util.Misc
+import org.apache.daffodil.exceptions.Assert
+import org.apache.daffodil.util.Maybe
+import org.apache.daffodil.util.Maybe.One
+import org.apache.daffodil.util.Maybe.Nope
+import org.apache.daffodil.io.FormatInfo
+import org.apache.daffodil.schema.annotation.props.gen.BitOrder
+import org.apache.daffodil.processors.parsers.PState
+import org.apache.daffodil.processors.unparsers.UState
+import org.apache.daffodil.io.DirectOrBufferedDataOutputStream
+import passera.unsigned.ULong
+
+/**
+ * Factory for a layer transformer.
+ *
+ * Responsible for digesting the various args, erroring if the wrong ones
+ * are specified, and ultimately constructing the LayerTransformer
+ * of the correct type with the parameters it needs.
+ */
+abstract class LayerTransformerFactory(nom: String)
+ extends Serializable {
+
+ val name = nom.toUpperCase()
+
+ def newInstance(maybeLayerCharsetEv: Maybe[LayerCharsetEv],
+ maybeLayerLengthKind: Maybe[LayerLengthKind],
+ maybeLayerLengthInBytesEv: Maybe[LayerLengthInBytesEv],
+ maybeLayerLengthUnits: Maybe[LayerLengthUnits],
+ maybeLayerBoundaryMarkEv: Maybe[LayerBoundaryMarkEv],
+ trd: TermRuntimeData): LayerTransformer
+}
+
+/**
+ * Transformers have factories. This lets you find the transformer factory
+ * by the name obtained from dfdl:layerTransform.
+ */
+object LayerTransformerFactory {
+
+ private lazy val transformerMap = new NonAllocatingMap(new HashMap[String, LayerTransformerFactory])
+
+ def register(factory: LayerTransformerFactory): Unit = {
+ transformerMap.put(factory.name, factory)
+ }
+ /**
+ * Given name, finds the factory for the transformer. SDE otherwise.
+ *
+ * The state is passed in order to provide diagnostic context if not found.
+ */
+ def find(name: String, state: ParseOrUnparseState): LayerTransformerFactory = {
+ val maybeFactory = transformerMap.get(name)
+ if (maybeFactory.isEmpty) {
+ val choices = transformerMap.keySet.mkString(", ")
+ state.SDE("The dfdl:layerTransform '%s' was not found. Available choices are: %s", name, choices)
+ } else {
+ maybeFactory.get
+ }
+ }
+
+ /**
+ * All transformers must be registered so they are available by name.
+ *
+ * It is possible to package a transformer in a separate jar also, but then
+ * someone has to register it by calling the register method.
+ *
+ * Transformers built into the primary Daffodil jars/packages should be
+ * registered here.
+ */
+ register(Base64MIMETransformerFactory)
+ register(GZIPTransformerFactory)
+ register(IMFLineFoldedTransformerFactory)
+ register(ICalendarLineFoldedTransformerFactory)
+}
+
+/**
+ * Shared functionality of all LayerTransformers.
+ */
+abstract class LayerTransformer()
+ extends Serializable {
+
+ protected def wrapLayerDecoder(jis: java.io.InputStream): java.io.InputStream
+
+ protected def wrapLimitingStream(jis: java.io.InputStream, state: PState): java.io.InputStream
+
+ def wrapJavaInputStream(s: DataInputStream, fInfo: FormatInfo): java.io.InputStream = {
+ new JavaIOInputStream(s, fInfo)
+ }
+
+ protected def wrapLayerEncoder(jos: java.io.OutputStream): java.io.OutputStream
+
+ protected def wrapLimitingStream(jis: java.io.OutputStream, state: UState): java.io.OutputStream
+
+ def wrapJavaOutputStream(s: DataOutputStream, fInfo: FormatInfo): java.io.OutputStream = {
+ new JavaIOOutputStream(s, fInfo)
+ }
+ /**
+ * Override these if we ever have transformers that don't have these
+ * requirements.
+ */
+ val mandatoryLayerAlignmentInBits: Int = 8
+ val mandatoryLengthUnit: LayerLengthUnits = LayerLengthUnits.Bytes
+
+ def addLayer(s: DataInputStream, state: PState): DataInputStream = {
+ val jis = wrapJavaInputStream(s, state)
+ val limitedJIS = wrapLimitingStream(jis, state)
+ val decodedInputStream = wrapLayerDecoder(limitedJIS)
+ // val str = Iterator.continually { decodedInputStream.read() }.takeWhile { _ != -1 }.toStream.mkString
+ // println(str)
+ // val bais = new ByteArrayInputStream(str.getBytes("ascii"))
+
+ val newDIS = ByteBufferDataInputStream(decodedInputStream, 0L)
+ newDIS.cst.setPriorBitOrder(BitOrder.MostSignificantBitFirst) // must initialize priorBitOrder
+ newDIS
+ }
+
+ def removeLayer(s: DataInputStream): Unit = {
+ // nothing for now
+ }
+
+ def addLayer(s: DataOutputStream, state: UState): DataOutputStream = {
+ val jos = wrapJavaOutputStream(s, state)
+ val limitedJOS = wrapLimitingStream(jos, state)
+ val encodedOutputStream = wrapLayerEncoder(limitedJOS)
+ val newDOS = DirectOrBufferedDataOutputStream(encodedOutputStream, null)
+ newDOS.setPriorBitOrder(BitOrder.MostSignificantBitFirst)
+ newDOS.setAbsStartingBitPos0b(ULong(0L))
+ newDOS
+ }
+
+ def removeLayer(s: DirectOrBufferedDataOutputStream, state: UState): Unit = {
+ //
+ // Because the layer may have suspensions that will write to it long after
+ // this unparser has left the stack, it is not clear we can do any
+ // cleanup of resources here.
+ //
+ }
+
+ /**
+ * These were very useful for debugging. Note that they stop with a ???
+ * error. That's because dumping streams changes their state.
+ *
+ * Keeping these around commented out, for now. While this feature is still
+ * new and may need further debugging.
+ */
+ // def dumpLayer(is: InputStream) {
+ // val str = Stream.continually(is.read).takeWhile(_ != -1).map(_.toChar).mkString
+ // println("dump length " + str.length + " = " + str)
+ // ???
+ // }
+ //
+ // def hexDumpLayer(is: InputStream) {
+ // val str = hexify(is)
+ // println("hex dump = " + str)
+ // ???
+ // }
+ //
+ // def hexify(is: InputStream): String =
+ // Stream.continually(is.read).takeWhile(_ != -1).map(x => "%02x".format(x.toInt)).mkString(" ")
+ //
+ // def hexDumpLayer(is: DataInputStream, finfo: FormatInfo) {
+ // val jis = new JavaIOInputStream(is, finfo)
+ // val str = hexify(jis)
+ // println("DIS hex dump = " + str)
+ // ???
+ // }
+}
+
+class JavaIOInputStream(s: DataInputStream, finfo: FormatInfo)
+ extends java.io.InputStream {
+
+ private lazy val id = Misc.getNameFromClass(this)
+
+ override def read(): Int = {
+ if (!s.isDefinedForLength(8)) -1
+ else {
+ val ul = s.getUnsignedLong(8, finfo)
+ val byte: Int = ul.toInt & 0xFF
+ byte
+ }
+ }
+
+ override def available(): Int = 1
+
+ override def close(): Unit = {
+ // do nothing
+ }
+
+ override def mark(readlimit: Int): Unit = {
+ Assert.usage(maybeSavedMark.isEmpty)
+ maybeSavedMark = One(s.mark(id))
+ }
+
+ private var maybeSavedMark: Maybe[Mark] = Nope
+
+ override def reset() {
+ Assert.usage(maybeSavedMark.isDefined)
+ s.reset(maybeSavedMark.get)
+ maybeSavedMark = Nope
+ }
+
+ override def markSupported() = true
+}
+
+class JavaIOOutputStream(dos: DataOutputStream, finfo: FormatInfo)
+ extends java.io.OutputStream {
+
+ private var closed = false
+
+ def nBytesWritten = nBytes
+
+ private var nBytes = 0
+
+ override def write(b: Int): Unit = {
+ val wasWritten = dos.putLong(b, 8, finfo)
+ if (wasWritten) nBytes += 1
+ }
+
+ override def close() {
+ if (!closed) {
+ dos.setFinished(finfo)
+ closed = true
+ }
+ }
+}
diff --git a/daffodil-runtime1/src/main/scala/org/apache/daffodil/layers/LineFoldedTransformer.scala b/daffodil-runtime1/src/main/scala/org/apache/daffodil/layers/LineFoldedTransformer.scala
new file mode 100644
index 0000000..7b659ec
--- /dev/null
+++ b/daffodil-runtime1/src/main/scala/org/apache/daffodil/layers/LineFoldedTransformer.scala
@@ -0,0 +1,474 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.layers
+
+import org.apache.daffodil.schema.annotation.props.gen.LayerLengthKind
+import org.apache.daffodil.schema.annotation.props.gen.LayerLengthUnits
+import org.apache.daffodil.util.Maybe
+import org.apache.daffodil.processors.TermRuntimeData
+import org.apache.daffodil.processors.LayerLengthInBytesEv
+import org.apache.daffodil.processors.LayerBoundaryMarkEv
+import org.apache.daffodil.processors.LayerCharsetEv
+import org.apache.daffodil.processors.parsers.PState
+import java.nio.charset.StandardCharsets
+import org.apache.daffodil.exceptions.Assert
+import org.apache.daffodil.processors.unparsers.UState
+import org.apache.daffodil.io.LayerBoundaryMarkInsertingJavaOutputStream
+import java.io.OutputStream
+import java.io.InputStream
+import org.apache.daffodil.exceptions.ThrowsSDE
+import org.apache.daffodil.schema.annotation.props.Enum
+import org.apache.daffodil.io.RegexLimitingStream
+
+/*
+ * This and related classes implement so called "line folding" from
+ * IETF RFC 2822 Internet Message Format (IMF), and IETF RFC 5545 iCalendar.
+ *
+ * There are multiple varieties of line folding, and it is important to
+ * be specific about which algorithm.
+ *
+ * For IMF, unfolding simply removes CRLFs if they are followed by a space or tab.
+ * The Folding is more complex however, as CRLFs can only be inserted before
+ * a space/tab that appears in the data. If the data has no spaces, then no
+ * folding is possible.
+ * If there are spaces/tabs, the one closest to position 78 is used unless it is
+ * followed by punctuation, in which case a prior space/tab (if it exists) is used.
+ * (This preference for spaces not followed by punctuation is optional, it is
+ * not required, but is preferred in the IMF RFC.)
+ *
+ * Note: folding is done by some systems in a manner that does not respect
+ * character boundaries - i.e., in utf-8, a multi-byte character sequence may be
+ * broken in the middle by insertion of a CRLF. Hence, unfolding initially treats
+ * the text as iso-8859-1, i.e., just bytes, and removes CRLFs, then subsequently
+ * re-interprets the bytes as the expected charset such as utf-8.
+ *
+ * IMF is supposed to be US-ASCII, but implementations have gone to 8-bit characters
+ * being preserved, so the above problem can occur.
+ *
+ * IMF has a maximum line length of 998 characters per line excluding the CRLF.
+ * The layer will fail (cause a parse error) if a line longer than this is encountered
+ * or constructed after unfolding. When unparsing, if a line longer than 998 cannot be
+ * folded due to no spaces/tabs being present in it, then it is an unparse error.
+ *
+ * Note that i/vCalendar, vCard, and MIME payloads held by IMF do not run into
+ * the IMF line length issues, in that they have their own line length limits that
+ * are smaller than those of IMF, and which do not require accomodation by having
+ * pre-existing spaces/tabs in the data. So such data *always* will be short
+ * enough lines.
+ *
+ * For vCard, iCalendar, and vCalendar, the maximum is 75 bytes plus the CRLF, for
+ * a total of 77. Folding is inserted by inserting CRLF + a space or tab. The
+ * CRLF and the following space or tab are removed to unfold. If data happened to
+ * contain a CRLF followed by a space or tab initially, then that will be lost when
+ * the data is parsed.
+ *
+ * For MIME, the maximum line length is 76.
+ */
+sealed trait LineFoldMode extends LineFoldMode.Value
+object LineFoldMode extends Enum[LineFoldMode] {
+
+ case object IMF extends LineFoldMode; forceConstruction(Left)
+ case object iCalendar extends LineFoldMode; forceConstruction(Right)
+
+ override def apply(name: String, context: ThrowsSDE): LineFoldMode = stringToEnum("lineFoldMode", name, context)
+}
+
+/**
+ * For line folded, the notion of "delimited" means that the element is a "line"
+ * that ends with CRLF, except that if it is long, it will be folded, which involves
+ * inserting/removing CRLF+Space (or CRLF+TAB). A CRLF not followed by space or tab
+ * is ALWAYS the actual "delimiter". There's no means of supplying a specific delimiter.
+ */
+class LineFoldedTransformerDelimited(mode: LineFoldMode)
+ extends LayerTransformer {
+
+ override protected def wrapLimitingStream(jis: java.io.InputStream, state: PState) = {
+ // regex means CRLF not followed by space or tab.
+ // NOTE: this regex cannot contain ANY capturing groups (per scaladoc on RegexLimitingStream)
+ val s = new RegexLimitingStream(jis, "\\r\\n(?!(?:\\t|\\ ))", "\r\n", StandardCharsets.ISO_8859_1)
+ s
+ }
+
+ override protected def wrapLimitingStream(jos: java.io.OutputStream, state: UState): java.io.OutputStream = {
+ //
+ // Q: How do we insert a CRLF "not followed by tab or space" when we don't
+ // control what follows?
+ // A: We don't. This is nature of the format. If what follows could begin
+ // with a space or tab, then the format can't use a line-folded layer.
+ //
+ val newJOS = new LayerBoundaryMarkInsertingJavaOutputStream(jos, "\r\n", StandardCharsets.ISO_8859_1)
+ newJOS
+ }
+
+ override protected def wrapLayerDecoder(jis: java.io.InputStream): java.io.InputStream = {
+ val s = new LineFoldedInputStream(mode, jis)
+ s
+ }
+ override protected def wrapLayerEncoder(jos: java.io.OutputStream): java.io.OutputStream = {
+ val s = new LineFoldedOutputStream(mode, jos)
+ s
+ }
+}
+
+/**
+ * For line folded, the 'implicit' length kind means that the region continues
+ * to end of data. At top level this would be the "whole file/stream" but this can
+ * also be used with a specified length enclosing element. This code cannot tell
+ * the difference.
+ */
+class LineFoldedTransformerImplicit(mode: LineFoldMode)
+ extends LayerTransformer {
+
+ override protected def wrapLimitingStream(jis: java.io.InputStream, state: PState) = {
+ jis // no limiting - just pull input until EOF.
+ }
+
+ override protected def wrapLimitingStream(jos: java.io.OutputStream, state: UState): java.io.OutputStream = {
+ jos // no limiting - just write output until EOF.
+ }
+
+ override protected def wrapLayerDecoder(jis: java.io.InputStream): java.io.InputStream = {
+ val s = new LineFoldedInputStream(mode, jis)
+ s
+ }
+ override protected def wrapLayerEncoder(jos: java.io.OutputStream): java.io.OutputStream = {
+ val s = new LineFoldedOutputStream(mode, jos)
+ s
+ }
+}
+
+sealed abstract class LineFoldedTransformerFactory(mode: LineFoldMode, name: String)
+ extends LayerTransformerFactory(name) {
+
+ override def newInstance(maybeLayerCharsetEv: Maybe[LayerCharsetEv],
+ maybeLayerLengthKind: Maybe[LayerLengthKind],
+ maybeLayerLengthInBytesEv: Maybe[LayerLengthInBytesEv],
+ maybeLayerLengthUnits: Maybe[LayerLengthUnits],
+ maybeLayerBoundaryMarkEv: Maybe[LayerBoundaryMarkEv],
+ trd: TermRuntimeData): LayerTransformer = {
+
+ trd.schemaDefinitionUnless(maybeLayerLengthKind.isDefined,
+ "The propert dfdl:layerLengthKind must be defined.")
+
+ val xformer =
+ maybeLayerLengthKind.get match {
+ case LayerLengthKind.BoundaryMark => {
+ new LineFoldedTransformerDelimited(mode)
+ }
+ case LayerLengthKind.Implicit => {
+ new LineFoldedTransformerImplicit(mode)
+ }
+ case x =>
+ trd.SDE("Property dfdl:layerLengthKind can only be 'implicit' or 'boundaryMark', but was '%s'",
+ x.toString)
+ }
+ xformer
+ }
+}
+
+object IMFLineFoldedTransformerFactory
+ extends LineFoldedTransformerFactory(LineFoldMode.IMF, "lineFolded_IMF")
+
+object ICalendarLineFoldedTransformerFactory
+ extends LineFoldedTransformerFactory(LineFoldMode.iCalendar, "lineFolded_iCalendar")
+
+/**
+ * Doesn't enforce 998 max line length limit.
+ *
+ * This is a state machine, so of course must be used only on a single thread.
+ */
+class LineFoldedInputStream(mode: LineFoldMode, jis: InputStream)
+ extends InputStream {
+
+ object State extends org.apache.daffodil.util.Enum {
+ abstract sealed trait Type extends EnumValueType
+
+ /**
+ * No state. Read a character, and if CR, go to state GotCR.
+ */
+ case object Start extends Type
+
+ /**
+ * Read another character and if LF go to state GotCRLF.
+ */
+ case object GotCR extends Type
+
+ /**
+ * Read another character and if SP/TAB then what we do depends on
+ * IMF or iCalendar mode.
+ *
+ * In iCalendar mode we just goto Start, and iterate
+ * again. effectively absorbing all the CR, LF, and the sp/tab.
+ *
+ * In IMF mode we change state to Start, but we return the sp/tab so that
+ * we've effectively absorbed the CRLF, but not the space/tab character.
+ */
+ case object GotCRLF extends Type
+
+ /**
+ * We have a single saved character. Return it, go to Start state
+ */
+ case object Buf1 extends Type
+
+ /**
+ * We have 2 saved characters. They must be a LF, then the next character.
+ * Return the LF and go to state Buf1.
+ */
+ case object Buf2 extends Type
+
+ /**
+ * Done. Always return -1, stay in state Done
+ */
+ case object Done extends Type
+ }
+
+ private var c: Int = -2
+ private var state: State.Type = State.Start
+
+ /**
+ * Assumes an ascii-family encoding, but reads it byte at a time regardless
+ * of the encoding. This enables it to handle data where a CRLF was inserted
+ * to limit line length, and that insertion broke up a multi-byte character.
+ *
+ * Does not detect errors such as isolated \r or isolated \n. Leaves those
+ * alone. Does not care if lines are in fact less than any limit in length.
+ *
+ * All this does is remove \r\n[\ \t], replacing with just the space or tab.(IMF)
+ * or replace with nothing (iCalendar).
+ *
+ */
+ override def read(): Int = {
+ import State._
+ while (state != Done) {
+ state match {
+ case Start => {
+ c = jis.read()
+ c match {
+ case -1 => {
+ state = Done
+ return -1
+ }
+ case '\r' => {
+ state = GotCR
+ }
+ case _ => {
+ // state stays Start
+ return c
+ }
+ }
+ }
+ case GotCR => {
+ c = jis.read()
+ c match {
+ case -1 => {
+ state = Done
+ return '\r'
+ }
+ case '\n' => {
+ state = GotCRLF
+ }
+ case _ => {
+ state = Buf1
+ return '\r'
+ }
+ }
+ }
+ case GotCRLF => {
+ c = jis.read()
+ c match {
+ case -1 => {
+ state = Buf2 // buffering up the LF
+ return '\r'
+ }
+ case ' ' | '\t' => {
+ if (mode eq LineFoldMode.IMF) {
+ state = Start // absorb the CR, LF, but not the sp/tab
+ return c // return the sp/tab
+ } else {
+ // iCalendar case
+ // we don't return a character, we go around again
+ // effectively we've absorbed the CR, LF, and the SP/TAB.
+ state = Start
+ }
+ }
+ case _ => {
+ // CRLF followed by other not sp/tab
+ state = Buf2
+ return '\r'
+ }
+ }
+ }
+ case Buf1 => {
+ state = Start
+ return c
+ }
+ case Buf2 => {
+ state = Buf1
+ return '\n'
+ }
+ case Done =>
+ Assert.invariantFailed("Done state not allowed.")
+ }
+ }
+ Assert.invariantFailed("No fall through to here.")
+ }
+}
+
+class LineFoldedOutputStream(mode: LineFoldMode, jos: OutputStream)
+ extends OutputStream {
+
+ private val (lineLength, breaker) = mode match {
+ case LineFoldMode.IMF => (78, "\r\n".getBytes("ascii"))
+ case LineFoldMode.iCalendar => (75, "\r\n ".getBytes("ascii")) // CRLF + a space.
+ }
+
+ private val line = new StringBuilder(lineLength + breaker.length)
+
+ private def lineBytes: Array[Byte] = line.map { _.toByte }.toArray
+
+ private def lastCharWas(c: Char) = {
+ if (line.length == 0) false
+ else line.last == c.toByte
+ }
+
+ private var closed = false
+
+ override def close(): Unit = {
+ if (!closed) {
+ if (line.length > 0) jos.write(lineBytes)
+ jos.close()
+ closed = true
+ }
+ }
+
+ override def write(bInt: Int): Unit = {
+ val b = bInt.toChar
+ Assert.usage(!closed)
+ Assert.invariant(line.length <= lineLength)
+ Assert.usage(b >= 0)
+ if (line.length < lineLength) {
+ // there's room for more on the line
+ b match {
+ case '\r' => line += b
+ case '\n' if !lastCharWas('\r') =>
+ {
+ // isolated \n. Output both \r and \n
+ // and flush the line
+ line += '\r'
+ line += '\n'
+ jos.write(lineBytes)
+ line.clear()
+ }
+ case '\n' if lastCharWas('\r') =>
+ {
+ // newline after a CR, regular CRLF case.
+ // add and output
+ line += '\n'
+ jos.write(lineBytes)
+ line.clear()
+ }
+ case c if lastCharWas('\r') => {
+ // isolated CR. Output CRLF
+ line += '\n'
+ jos.write(lineBytes)
+ line.clear()
+ line += c
+ }
+ case c => {
+ line += c
+ }
+ }
+ } else {
+ // buffer is full of non-newline-oriented characters
+ // this is a "too-long" line now with the incoming character.
+ //
+ // Now things get quite different for IMF vs. iCalendar
+ //
+ mode match {
+ case LineFoldMode.iCalendar =>
+ tooLongLineICalendar(b)
+ case LineFoldMode.IMF =>
+ tooLongLineIMF(b)
+ }
+ }
+ }
+
+ /**
+ * For iCalendar, we insert a CRLF+space right here, without worry about
+ * surrounding context characters.
+ */
+ private def tooLongLineICalendar(b: Char): Unit = {
+ line += '\r'
+ line += '\n'
+ if (mode eq LineFoldMode.iCalendar)
+ line += ' '
+ jos.write(lineBytes)
+ line.clear()
+ line += b
+ }
+
+ /**
+ * For IMF, we can only insert a CRLF if we can find a space or tab to
+ * put it before.
+ *
+ * If there is no space/tab we simply cannot break/wrap the line.
+ *
+ * This should be the latest sp/tab in the data so the line is wrapped as
+ * long as allowed.
+ *
+ * Ideally it would not be a space immediately before punctuation, as that
+ * would affect readability (for people) by starting the next line with a
+ * punctuation character. But this is, strictly speaking, unnecessary.
+ */
+ private def tooLongLineIMF(b: Char): Unit = {
+ var i = line.length - 1
+ var done = false
+ while (i > 0 && !done) {
+ val c = line(i)
+ c match {
+ case ' ' | '\t' => {
+ done = true
+ //
+ // i is the index of latest sp/tab
+ //
+ jos.write(lineBytes, 0, i)
+ jos.write(breaker)
+ //
+ // now the characters from i to the end of the line
+ // are part of the next line, including the sp/tab.
+ // we have to slide those down to positions 0 to n
+ // in the line.
+ //
+ line.delete(0, i) // slides remaining down.
+ line += b
+ }
+ case _ => // just keep iterating
+ }
+ i -= 1
+ }
+ if (!done) {
+ // We never found a space/tab. We can't break the line.
+ // So we output what we have, and keep going with this
+ // new character, and no line break.
+ jos.write(lineBytes)
+ line.clear()
+ line += b
+ }
+ }
+}
+
diff --git a/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/EvEncoding.scala b/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/EvEncoding.scala
index ba09f0d..184b790 100644
--- a/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/EvEncoding.scala
+++ b/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/EvEncoding.scala
@@ -48,7 +48,7 @@ import org.apache.daffodil.processors.charset.BitsCharsetWrappingJavaCharset
/**
* Encoding is a string, so there is no converter.
*/
-class EncodingEv(override val expr: CompiledExpression[String], trd: TermRuntimeData)
+abstract class EncodingEvBase(override val expr: CompiledExpression[String], trd: TermRuntimeData)
extends EvaluatableConvertedExpression[String, String](
expr,
EncodingCooker, // cooker insures upper-case and trimmed of whitespace.
@@ -74,7 +74,10 @@ class EncodingEv(override val expr: CompiledExpression[String], trd: TermRuntime
}
}
-final class CharsetEv(encodingEv: EncodingEv, val trd: TermRuntimeData)
+final class EncodingEv(expr: CompiledExpression[String], trd: TermRuntimeData)
+ extends EncodingEvBase(expr, trd)
+
+abstract class CharsetEvBase(encodingEv: EncodingEvBase, val trd: TermRuntimeData)
extends Evaluatable[BitsCharset](trd)
with InfosetCachedEvaluatable[BitsCharset] {
@@ -100,6 +103,9 @@ final class CharsetEv(encodingEv: EncodingEv, val trd: TermRuntimeData)
}
}
+final class CharsetEv(encodingEv: EncodingEv, trd: TermRuntimeData)
+ extends CharsetEvBase(encodingEv, trd)
+
class FillByteEv(fillByteRaw: String, charsetEv: CharsetEv, val trd: TermRuntimeData)
extends Evaluatable[Integer](trd)
with InfosetCachedEvaluatable[Integer] {
diff --git a/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/EvLayering.scala b/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/EvLayering.scala
new file mode 100644
index 0000000..483dd20
--- /dev/null
+++ b/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/EvLayering.scala
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.processors
+
+import org.apache.daffodil.cookers.UpperCaseTokenCooker
+import org.apache.daffodil.dsom._
+import java.lang.{ Long => JLong }
+import org.apache.daffodil.schema.annotation.props.gen.LayerLengthUnits
+import org.apache.daffodil.util.Maybe
+import org.apache.daffodil.schema.annotation.props.gen.LayerLengthKind
+import org.apache.daffodil.layers.LayerTransformer
+import org.apache.daffodil.layers.LayerTransformerFactory
+
+/*
+ * Layering-related Evaluatables
+ */
+final class LayerTransformEv(override val expr: CompiledExpression[String], trd: TermRuntimeData)
+ extends EvaluatableConvertedExpression[String, String](
+ expr,
+ UpperCaseTokenCooker, // cooker insures upper-case and trimmed of whitespace.
+ trd)
+ with NoCacheEvaluatable[String]
+
+final class LayerEncodingEv(override val expr: CompiledExpression[String], trd: TermRuntimeData)
+ extends EncodingEvBase(expr, trd)
+
+final class LayerCharsetEv(layerEncodingEv: LayerEncodingEv, override val trd: TermRuntimeData)
+ extends CharsetEvBase(layerEncodingEv, trd)
+
+final class LayerLengthInBytesEv(override val expr: CompiledExpression[JLong], override val rd: TermRuntimeData)
+ extends EvaluatableExpression[JLong](
+ expr,
+ rd)
+ with NoCacheEvaluatable[JLong] {
+ override lazy val runtimeDependencies = Nil
+
+ override def compute(state: State): JLong = {
+ val v: JLong = super.compute(state)
+ if (v < 0) {
+ state.SDE("dfdl:length expression result must be non-negative, but was: %d", v)
+ }
+ v
+ }
+}
+
+final class LayerBoundaryMarkEv(override val expr: CompiledExpression[String], override val rd: TermRuntimeData)
+ extends EvaluatableExpression[String](
+ expr,
+ rd)
+ with NoCacheEvaluatable[String] {
+ override lazy val runtimeDependencies = Nil
+}
+
+final class LayerTransformerEv(
+ layerTransformEv: LayerTransformEv,
+ maybeLayerCharsetEv: Maybe[LayerCharsetEv],
+ maybeLayerLengthKind: Maybe[LayerLengthKind],
+ maybeLayerLengthInBytesEv: Maybe[LayerLengthInBytesEv],
+ maybeLayerLengthUnits: Maybe[LayerLengthUnits],
+ maybeLayerBoundaryMarkEv: Maybe[LayerBoundaryMarkEv],
+ trd: TermRuntimeData)
+ extends Evaluatable[LayerTransformer](trd)
+ with NoCacheEvaluatable[LayerTransformer] {
+
+ override lazy val runtimeDependencies = layerTransformEv +:
+ (maybeLayerCharsetEv.toList ++
+ maybeLayerLengthInBytesEv.toList ++
+ maybeLayerBoundaryMarkEv.toList)
+
+ /**
+ * Finds the proper layer transformer and constructs it with its parameters
+ * as needed from the various layer properties.
+ */
+ override def compute(state: State): LayerTransformer = {
+ val layerTransform = layerTransformEv.evaluate(state)
+ val factory = LayerTransformerFactory.find(layerTransform, state)
+ val xformer = factory.newInstance(maybeLayerCharsetEv,
+ maybeLayerLengthKind,
+ maybeLayerLengthInBytesEv,
+ maybeLayerLengthUnits,
+ maybeLayerBoundaryMarkEv,
+ trd)
+ xformer
+ }
+}
+
diff --git a/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/ProcessorStateBases.scala b/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/ProcessorStateBases.scala
index 54dc27d..bb6da36 100644
--- a/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/ProcessorStateBases.scala
+++ b/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/ProcessorStateBases.scala
@@ -228,20 +228,30 @@ abstract class ParseOrUnparseState protected (
case erd: ElementRuntimeData => erd.maybeByteOrderEv.get.evaluate(this)
case mgrd: ModelGroupRuntimeData => {
//
- // Model Groups can't have byte order.
+ // Model Groups can't have the byteOrder property.
+ //
// However, I/O layer still requests it because alignment regions
// use skip, which ultimately uses getLong/putLong, which asks for
- // byteOrder.
+ // byteOrder. (Not any more for 1-byte case - mikeb)
//
// A model group DOES care about bit order for its alignment regions,
// and for the charset encoding of say, initiators or prefix separators.
// A bitOrder change requires that we check the new bitOrder against the
// byte order to insure compatibility. (byteOrder can be an expression),
- // so of necessity, we also need byte order. However, if byte order isn't defined
- // we can assume littleEndian since that works with all bit orders.
- // (Big endian only allows MSBF bit order)
+ // so of necessity, we also need byte order.
+ //
+ // Because the binary integer unparsers (unsignedLong in particular)
+ // dispatch on byte order first, then bit order (if littleEndian),
+ // we do have to provide the byte order that is consistent with the
+ // bit order - if the bit order is LSBF, we have to provide byte order
+ // of LittleEndian, otherwise we end up in BigEndian+MSBF code paths.
//
- ByteOrder.LittleEndian
+ val bo =
+ if (mgrd.defaultBitOrder eq BitOrder.LeastSignificantBitFirst)
+ ByteOrder.LittleEndian
+ else
+ ByteOrder.BigEndian
+ bo
}
case _ => Assert.usageError("byte order of non term: " + runtimeData)
}
diff --git a/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/Runtime.scala b/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/Runtime.scala
index 728da72..7504f2e 100644
--- a/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/Runtime.scala
+++ b/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/Runtime.scala
@@ -353,6 +353,17 @@ class DataProcessor(val ssrd: SchemaSetRuntimeData)
// LoggingDefaults.setLoggingLevel(LogLevel.Debug)
rootUnparser.unparse1(state)
+ // Restore invariant that there is always a processor.
+ // Later when suspensions get evaluated, there are still times when
+ // properties are read, and those require a processor to be set
+ // for model groups so that the RuntimeData can be found.
+ //
+ // For elements, the infoset node enables you to find the
+ // ElementRuntimeData, but for model groups there is no infoset node,
+ // so we need the Processor, which has a context which is the RD.
+ //
+ state.setProcessor(rootUnparser)
+
/* Verify that all stacks are empty */
Assert.invariant(state.arrayIndexStack.length == 1)
Assert.invariant(state.groupIndexStack.length == 1)
diff --git a/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/parsers/ElementKindParsers.scala b/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/parsers/ElementKindParsers.scala
index 1886ee8..fecd7d6 100644
--- a/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/parsers/ElementKindParsers.scala
+++ b/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/parsers/ElementKindParsers.scala
@@ -26,6 +26,7 @@ import org.apache.daffodil.processors.RuntimeData
import org.apache.daffodil.processors.Success
import org.apache.daffodil.util.LogLevel
import org.apache.daffodil.processors.TermRuntimeData
+import org.apache.daffodil.processors.Evaluatable
class ComplexTypeParser(rd: RuntimeData, bodyParser: Parser)
extends CombinatorParser(rd) {
@@ -123,7 +124,7 @@ class SequenceCombinatorParser(rd: TermRuntimeData, bodyParser: Parser)
extends CombinatorParser(rd) {
override def nom = "Sequence"
- override lazy val runtimeDependencies = Nil
+ override lazy val runtimeDependencies: Seq[Evaluatable[AnyRef]] = Nil
override lazy val childProcessors = Seq(bodyParser)
diff --git a/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/parsers/LayeredSequenceParser.scala b/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/parsers/LayeredSequenceParser.scala
new file mode 100644
index 0000000..f87c550
--- /dev/null
+++ b/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/parsers/LayeredSequenceParser.scala
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.processors.parsers
+
+import org.apache.daffodil.processors.TermRuntimeData
+import org.apache.daffodil.processors.LayerTransformerEv
+import org.apache.daffodil.io.ByteBufferDataInputStream
+
+class LayeredSequenceParser(rd: TermRuntimeData,
+ layerTransformerEv: LayerTransformerEv,
+ bodyParser: Parser)
+ extends SequenceCombinatorParser(rd, bodyParser) {
+ override def nom = "LayeredSequence"
+
+ override lazy val runtimeDependencies = Seq(layerTransformerEv)
+
+ override def parse(state: PState): Unit = {
+
+ val layerTransformer = layerTransformerEv.evaluate(state)
+ val savedDIS = state.dataInputStream
+
+ val isAligned = savedDIS.align(layerTransformer.mandatoryLayerAlignmentInBits, state)
+ if (!isAligned)
+ PE(state, "Unable to align to the mandatory layer alignment of %s(bits)",
+ layerTransformer.mandatoryLayerAlignmentInBits)
+
+ val newDIS = layerTransformer.addLayer(savedDIS, state)
+
+ //
+ // FIXME: Cast should not be needed
+ //
+ state.dataInputStream = newDIS.asInstanceOf[ByteBufferDataInputStream]
+
+ super.parse(state)
+
+ layerTransformer.removeLayer(newDIS)
+
+ state.dataInputStream = savedDIS
+ }
+}
diff --git a/daffodil-runtime1/src/test/scala/org/apache/daffodil/layers/TestLengthLimitedLineFoldingStreams.scala b/daffodil-runtime1/src/test/scala/org/apache/daffodil/layers/TestLengthLimitedLineFoldingStreams.scala
new file mode 100644
index 0000000..b8c034b
--- /dev/null
+++ b/daffodil-runtime1/src/test/scala/org/apache/daffodil/layers/TestLengthLimitedLineFoldingStreams.scala
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.layers
+
+import junit.framework.Assert._
+import java.io._
+import org.junit.Test
+import org.apache.daffodil.io.RegexLimitingStream
+import java.nio.charset.StandardCharsets
+
+class TestLengthLimitedLineFoldingStreams {
+
+ /**
+ * Has lines folded using IMF conventions.
+ *
+ * Notice use of the s"""...""" string interpolation. This interprets
+ * the escape sequences even though triple quote doesn't.
+ */
+ val ipsumLorem1 = s"""Lorem ipsum dolor sit amet, consectetur adipiscing elit,\r
+\tsed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad\r
+minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea\r
+\tcommodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit\r
+esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat\r
+\tnon proident, sunt in culpa qui officia deserunt mollit anim id est laborum."""
+
+ val ipsumLorem1UnfoldedFirstLine = s"""Lorem ipsum dolor sit amet, consectetur adipiscing elit,""" +
+ s"""\tsed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad"""
+
+ val iso8859 = StandardCharsets.ISO_8859_1
+
+ /**
+ * Shows that the regex will limit length to just the first line, but unfold will
+ * then apply and unfold to a longer line.
+ */
+ @Test def testLineFoldedIMFOneLine() = {
+ val dataString = ipsumLorem1
+ val bba = new ByteArrayInputStream(dataString.getBytes("utf-8"))
+ //
+ // regex is CRLF not followed by a tab or space.
+ //
+ val rls = new RegexLimitingStream(bba, "\\r\\n(?!(?:\\t|\\ ))", "\r\n", iso8859)
+ val lfs = new LineFoldedInputStream(LineFoldMode.IMF, rls)
+
+ val baos = new ByteArrayOutputStream()
+ var c: Int = -1
+ while ({
+ c = lfs.read()
+ c != -1
+ }) {
+ baos.write(c)
+ }
+ baos.close()
+ val resultString = new String(baos.toByteArray())
+ val expected = ipsumLorem1UnfoldedFirstLine
+ assertEquals(expected, resultString)
+ }
+
+ /**
+ * This has lines folded with iCalendar conventions \r\n\t (uses tabs).
+ * Because the \r\n\t will be removed, we have two \t consecutively so that the
+ * result has one \t still.
+ */
+ val ipsumLorem2 = s"""Lorem ipsum dolor sit amet, consectetur adipiscing elit,\r
+\t\tsed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad\r
+minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea\r
+\t\tcommodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit\r
+esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat\r
+\t\tnon proident, sunt in culpa qui officia deserunt mollit anim id est laborum."""
+
+ /**
+ * Shows that the regex will limit length to just the first line, but unfold will
+ * then apply and unfold to a longer line. (iCalendar conventions)
+ */
+ @Test def testLineFoldediCalendarOneLine() = {
+ val dataString = ipsumLorem2
+ val bba = new ByteArrayInputStream(dataString.getBytes("utf-8"))
+ //
+ // regex is CRLF not followed by a tab or space.
+ //
+ val rls = new RegexLimitingStream(bba, "\\r\\n(?!(?:\\t|\\ ))", "\r\n", iso8859)
+ val lfs = new LineFoldedInputStream(LineFoldMode.iCalendar, rls)
+
+ val baos = new ByteArrayOutputStream()
+ var c: Int = -1
+ while ({
+ c = lfs.read()
+ c != -1
+ }) {
+ baos.write(c)
+ }
+ baos.close()
+ val resultString = new String(baos.toByteArray())
+ val expected = ipsumLorem1UnfoldedFirstLine
+ assertEquals(expected, resultString)
+ }
+
+ /**
+ * All characters removed.
+ */
+ @Test def testLineFoldediCalendarNothing() = {
+ val dataString = "\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n "
+ val bba = new ByteArrayInputStream(dataString.getBytes("utf-8"))
+ //
+ // regex is CRLF not followed by a tab or space.
+ //
+ val rls = new RegexLimitingStream(bba, "\\r\\n(?!(?:\\t|\\ ))", "\r\n", iso8859)
+ val lfs = new LineFoldedInputStream(LineFoldMode.iCalendar, rls)
+
+ val baos = new ByteArrayOutputStream()
+ var c: Int = -1
+ while ({
+ c = lfs.read()
+ c != -1
+ }) {
+ baos.write(c)
+ }
+ baos.close()
+ val resultString = new String(baos.toByteArray())
+ val expected = ""
+ assertEquals(expected, resultString)
+ }
+
+ /**
+ * All characters except spaces removed.
+ */
+ @Test def testLineFoldedIMFNothing() = {
+ val dataString = "\r\n \r\n \r\n "
+ val bba = new ByteArrayInputStream(dataString.getBytes("utf-8"))
+ //
+ // regex is CRLF not followed by a tab or space.
+ //
+ val rls = new RegexLimitingStream(bba, "\\r\\n(?!(?:\\t|\\ ))", "\r\n", iso8859)
+ val lfs = new LineFoldedInputStream(LineFoldMode.IMF, rls)
+
+ val baos = new ByteArrayOutputStream()
+ var c: Int = -1
+ while ({
+ c = lfs.read()
+ c != -1
+ }) {
+ baos.write(c)
+ }
+ baos.close()
+ val resultString = new String(baos.toByteArray())
+ val expected = " "
+ assertEquals(expected, resultString)
+ }
+
+ /**
+ * Empty string doesn't trip things up. IMF
+ */
+ @Test def testLineFoldedEmptyIMF() = {
+ val dataString = ""
+ val bba = new ByteArrayInputStream(dataString.getBytes("utf-8"))
+ //
+ // regex is CRLF not followed by a tab or space.
+ //
+ val rls = new RegexLimitingStream(bba, "\\r\\n(?!(?:\\t|\\ ))", "\r\n", iso8859)
+ val lfs = new LineFoldedInputStream(LineFoldMode.IMF, rls)
+
+ val baos = new ByteArrayOutputStream()
+ var c: Int = -1
+ while ({
+ c = lfs.read()
+ c != -1
+ }) {
+ baos.write(c)
+ }
+ baos.close()
+ val resultString = new String(baos.toByteArray())
+ val expected = ""
+ assertEquals(expected, resultString)
+ }
+
+ /**
+ * Empty string doesn't trip things up. iCalendar
+ */
+ @Test def testLineFoldedEmptyICalendar() = {
+ val dataString = ""
+ val bba = new ByteArrayInputStream(dataString.getBytes("utf-8"))
+ //
+ // regex is CRLF not followed by a tab or space.
+ //
+ val rls = new RegexLimitingStream(bba, "\\r\\n(?!(?:\\t|\\ ))", "\r\n", iso8859)
+ val lfs = new LineFoldedInputStream(LineFoldMode.iCalendar, rls)
+
+ val baos = new ByteArrayOutputStream()
+ var c: Int = -1
+ while ({
+ c = lfs.read()
+ c != -1
+ }) {
+ baos.write(c)
+ }
+ baos.close()
+ val resultString = new String(baos.toByteArray())
+ val expected = ""
+ assertEquals(expected, resultString)
+ }
+
+}
diff --git a/daffodil-runtime1/src/test/scala/org/apache/daffodil/layers/TestLineFoldingStreams.scala b/daffodil-runtime1/src/test/scala/org/apache/daffodil/layers/TestLineFoldingStreams.scala
new file mode 100644
index 0000000..76e0213
--- /dev/null
+++ b/daffodil-runtime1/src/test/scala/org/apache/daffodil/layers/TestLineFoldingStreams.scala
@@ -0,0 +1,231 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.layers
+
+import junit.framework.Assert._
+import java.io._
+import org.junit.Test
+
+class TestLineFoldingStreams {
+
+ @Test def testLineFoldedOutputStream_iCal1() = {
+ val bba = new ByteArrayOutputStream()
+ val lfs = new LineFoldedOutputStream(LineFoldMode.iCalendar, bba)
+ val baos = new ByteArrayOutputStream()
+ val dataString = "Embedded\r\nIsolated NL\nwith isolated CR\rwith other" +
+ " very long lines of things that are too long for the usual" +
+ " line length which is 78-ish."
+ baos.write(dataString.getBytes("utf-8"))
+ lfs.write(baos.toByteArray())
+ lfs.close()
+ val resultString = new String(bba.toByteArray())
+ val expected = "Embedded\r\nIsolated NL\r\nwith isolated CR\r\nwith other" +
+ " very long lines of things that are too long for the usual" +
+ " line l\r\n ength which is 78-ish."
+ assertEquals(expected, resultString)
+ }
+
+ /**
+ * Boundary condition where the character that is 1 too big for the line is
+ * a CR.
+ */
+ @Test def testLineFoldedOutputStream_iCal2() = {
+ val bba = new ByteArrayOutputStream()
+ val lfs = new LineFoldedOutputStream(LineFoldMode.iCalendar, bba)
+ val baos = new ByteArrayOutputStream()
+ val dataString = "With other" +
+ " very long lines of things that are too long for the usual" +
+ " line l\r\nength which is 78-ish."
+ baos.write(dataString.getBytes("utf-8"))
+ lfs.write(baos.toByteArray())
+ lfs.close()
+ val resultString = new String(bba.toByteArray())
+ val expected = "With other" +
+ " very long lines of things that are too long for the usual" +
+ " line l\r\n \r\nength which is 78-ish."
+ assertEquals(expected, resultString)
+ }
+
+ /**
+ * Boundary condition where the character that is 1 too big for the line is
+ * a LF of CRLF.
+ */
+ @Test def testLineFoldedOutputStream_iCal3() = {
+ val bba = new ByteArrayOutputStream()
+ val lfs = new LineFoldedOutputStream(LineFoldMode.iCalendar, bba)
+ val baos = new ByteArrayOutputStream()
+ val dataString = "With other" +
+ " very long lines of things that are too long for the usual" +
+ " line \r\nlength which is 78-ish."
+ baos.write(dataString.getBytes("utf-8"))
+ lfs.write(baos.toByteArray())
+ lfs.close()
+ val resultString = new String(bba.toByteArray())
+ val expected = "With other" +
+ " very long lines of things that are too long for the usual" +
+ " line \r\r\n \nlength which is 78-ish."
+ assertEquals(expected, resultString)
+ }
+
+ /**
+ * Tests that folding backs up to the prior space/tab and inserts the CRLF
+ * before that space/tab. Also that isolated \r or \n are converted to CRLF,
+ * and existing CRLF are preserved.
+ */
+ @Test def testLineFoldedOutputStream_IMF1() = {
+ val bba = new ByteArrayOutputStream()
+ val lfs = new LineFoldedOutputStream(LineFoldMode.IMF, bba)
+ val baos = new ByteArrayOutputStream()
+ val dataString = "Embedded\r\nIsolated NL\nwith isolated CR\rwith other" +
+ " very long lines of things that are too long for the usual" +
+ " line length which is 78-ish."
+
+ baos.write(dataString.getBytes("utf-8"))
+ lfs.write(baos.toByteArray())
+ lfs.close()
+ val resultString = new String(bba.toByteArray())
+ val expected = "Embedded\r\nIsolated NL\r\nwith isolated CR\r\nwith other " +
+ "very long lines of things that are too long for the usual " +
+ "line\r\n length which is 78-ish."
+ assertEquals(expected, resultString)
+ }
+
+ /**
+ * Insures lines without tabs/spaces aren't folded, though isolated \r and \n
+ * are converted to CRLFs. There are no spaces in this data. All replaced by "_"
+ * (underscore), so by IMF folding rules, you can't fold the long part of this data.
+ */
+ @Test def testLineFoldedOutputStream_IMF2() = {
+ val bba = new ByteArrayOutputStream()
+ val lfs = new LineFoldedOutputStream(LineFoldMode.IMF, bba)
+ val baos = new ByteArrayOutputStream()
+ val dataString = "Embedded\r\nIsolated NL\nwith_isolated_CR\rwith_other" +
+ "_very_long_lines_of_things_that_are_too_long_for_the_usual" +
+ "_line_length_which_is_78-ish."
+
+ /**/ val ruler = "with_other_very_long_lines_of_things_that_are_too_long_for_the_usual_line_length_which_is_78-ish."
+ //// val roolr = "1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567"
+ //// val rool2 = " 1 2 3 4 5 6 7 8 9 "
+
+ baos.write(dataString.getBytes("utf-8"))
+ lfs.write(baos.toByteArray())
+ lfs.close()
+ val resultString = new String(bba.toByteArray())
+ val expected = "Embedded\r\nIsolated NL\r\nwith_isolated_CR\r\n" + ruler
+ assertEquals(expected, resultString)
+ }
+
+ /**
+ * CRLF on the boundary where it would fold, prevents folding.
+ */
+ @Test def testLineFoldedOutputStream_IMF3() = {
+ val bba = new ByteArrayOutputStream()
+ val lfs = new LineFoldedOutputStream(LineFoldMode.IMF, bba)
+ val baos = new ByteArrayOutputStream()
+ val dataString = "With_other_very_long_lines_of_things_that_are_too_long_for_the_usual" +
+ "_line\r\n length_which_is_78-ish."
+
+ /**/ val ruler = "With_other_very_long_lines_of_things_that_are_too_long_for_the_usual_line\r\n length_which_is_78-ish."
+ //// val roolr = "12345678901234567890123456789012345678901234567890123456789012345678901234 5 6789012345678901234567"
+ //// val rool2 = " 1 2 3 4 5 6 7 8 9 "
+
+ baos.write(dataString.getBytes("utf-8"))
+ lfs.write(baos.toByteArray())
+ lfs.close()
+ val resultString = new String(bba.toByteArray())
+ val expected = ruler
+ assertEquals(expected, resultString)
+ }
+
+ /**
+ * CRLF just past the boundary where it would fold, prevents folding.
+ */
+ @Test def testLineFoldedOutputStream_IMF4() = {
+ val bba = new ByteArrayOutputStream()
+ val lfs = new LineFoldedOutputStream(LineFoldMode.IMF, bba)
+ val baos = new ByteArrayOutputStream()
+ val dataString = "With_other_very_long_lines_of_things_that_are_too_long_for_the_usual" +
+ "_line \r\nlength_which_is_78-ish."
+
+ /**/ val ruler = "With_other_very_long_lines_of_things_that_are_too_long_for_the_usual_line \r\nlength_which_is_78-ish."
+ //// val roolr = "123456789012345678901234567890123456789012345678901234567890123456789012345 6 789012345678901234567"
+ //// val rool2 = " 1 2 3 4 5 6 7 8 9 "
+
+ baos.write(dataString.getBytes("utf-8"))
+ lfs.write(baos.toByteArray())
+ lfs.close()
+ val resultString = new String(bba.toByteArray())
+ val expected = ruler
+ assertEquals(expected, resultString)
+ }
+
+ @Test def testLineFoldedInputStreamTab1() = {
+ val dataString = "Here's data containing the line ending we care about\r\n\t followed by other text."
+ val bba = new ByteArrayInputStream(dataString.getBytes("utf-8"))
+ val lfs = new LineFoldedInputStream(LineFoldMode.iCalendar, bba)
+ val baos = new ByteArrayOutputStream()
+ var c: Int = -1
+ while ({
+ c = lfs.read()
+ c != -1
+ }) {
+ baos.write(c)
+ }
+ baos.close()
+ val resultString = new String(baos.toByteArray())
+ val expected = "Here's data containing the line ending we care about followed by other text."
+ assertEquals(expected, resultString)
+ }
+
+ @Test def testLineFoldedInputStreamIMFSpace1() = {
+ val dataString = "Here's data containing the line ending we care about\r\n followed by other text."
+ val bba = new ByteArrayInputStream(dataString.getBytes("utf-8"))
+ val lfs = new LineFoldedInputStream(LineFoldMode.IMF, bba)
+ val baos = new ByteArrayOutputStream()
+ var c: Int = -1
+ while ({
+ c = lfs.read()
+ c != -1
+ }) {
+ baos.write(c)
+ }
+ baos.close()
+ val resultString = new String(baos.toByteArray())
+ val expected = "Here's data containing the line ending we care about followed by other text."
+ assertEquals(expected, resultString)
+ }
+
+ @Test def testLineFoldedInputStreamCRLFInteractions1() = {
+ val dataString = "Foobar\r\n Quuxly"
+ val bba = new ByteArrayInputStream(dataString.getBytes("utf-8"))
+ val lfs = new LineFoldedInputStream(LineFoldMode.IMF, bba)
+ val baos = new ByteArrayOutputStream()
+ var c: Int = -1
+ while ({
+ c = lfs.read()
+ c != -1
+ }) {
+ baos.write(c)
+ }
+ baos.close()
+ val resultString = new String(baos.toByteArray())
+ val expected = "Foobar Quuxly"
+ assertEquals(expected, resultString)
+ }
+
+}
diff --git a/daffodil-test/src/test/resources/org/apache/daffodil/layers/layers.tdml b/daffodil-test/src/test/resources/org/apache/daffodil/layers/layers.tdml
new file mode 100644
index 0000000..c5027c7
--- /dev/null
+++ b/daffodil-test/src/test/resources/org/apache/daffodil/layers/layers.tdml
@@ -0,0 +1,257 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<tdml:testSuite xmlns:tdml="http://www.ibm.com/xmlns/dfdl/testData" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:fn="http://www.w3.org/2005/xpath-functions"
+ xmlns:ex="http://example.com" xmlns:tns="http://example.com" defaultRoundTrip="true">
+
+ <tdml:defineSchema name="s1" elementFormDefault="unqualified">
+ <dfdl:defineFormat name="general">
+ <dfdl:format ref="ex:GeneralFormat" />
+ </dfdl:defineFormat>
+ <dfdl:defineFormat name="compressed">
+ <dfdl:format ref="ex:general" layerTransform="gzip" layerLengthKind="explicit" layerLengthUnits="bytes" />
+ </dfdl:defineFormat>
+ <dfdl:format ref="ex:general" />
+
+ <xs:group name="compressedGroupContents">
+ <xs:sequence>
+ <xs:element name="text" type="xs:string" dfdl:lengthKind="delimited" />
+ </xs:sequence>
+ </xs:group>
+
+ <xs:element name="root">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="compressedPayloadLength" type="xs:int" dfdl:representation="binary"
+ dfdl:outputValueCalc='{ dfdl:contentLength(../compressedPayload, "bytes") }' />
+
+ <xs:element name="compressedPayload">
+ <xs:complexType>
+ <xs:sequence dfdl:ref="tns:compressed" dfdl:layerLength="{ ../compressedPayloadLength }">
+ <xs:group ref="tns:compressedGroupContents" />
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+
+ <xs:sequence>
+ <xs:annotation>
+ <xs:appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:assert>{ compressedPayloadLength eq dfdl:contentLength(compressedPayload, "bytes") }</dfdl:assert>
+ </xs:appinfo>
+ </xs:annotation>
+ </xs:sequence>
+ <xs:element name="after" type="xs:string" dfdl:lengthKind="delimited" />
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ </tdml:defineSchema>
+
+
+
+ <tdml:parserTestCase name="layers1" root="root" model="s1">
+ <tdml:document>
+ <tdml:documentPart type="byte"><![CDATA[
+ 000000D41F8B08000000000000004D904176C3200C44AF3207C8F33DBA6F0F40CCD8568391
+ 8B44D3DC3EC2C9A2EFB1013EF3357C6E6288F5DDCD61BA137BCA443FE0FC73F8967C5C4B75
+ D6CC0C575C8984857714A93414ADEB848F25D800B794036045632A67C605E2B86B2F19553D
+ 805FBE889F2ECE70E2AA4DEA3AA2E3519EF065842E58D2AEDD02530F8DB640832A8F26F3B9
+ 4DF511CA712437BE27ADDE34F739F8598F20D7CD875566460BEBB4CB10CAD989C9846D684D
+ F6A33CA2F9ED6CFEBF5DCC7168C4169ABDBEE46D139B9E8B9C8E093C010000616674657247
+ 7A6970]]>
+ </tdml:documentPart>
+ </tdml:document>
+ <tdml:infoset>
+ <tdml:dfdlInfoset>
+ <ex:root>
+ <compressedPayloadLength>212</compressedPayloadLength>
+ <compressedPayload>
+ <text><![CDATA[This is just some made up text that is intended to be a few lines long. If this had been real text, it would not have been quite so boring to read. Use of famous quotes or song lyrics or anything like that introduces copyright notice issues, so it is easier to simply make up a few lines of pointless text like this.]]></text>
+ </compressedPayload>
+ <after>afterGzip</after>
+ </ex:root>
+ </tdml:dfdlInfoset>
+ </tdml:infoset>
+ </tdml:parserTestCase>
+
+
+ <tdml:defineSchema name="s2" elementFormDefault="unqualified">
+ <dfdl:defineFormat name="general">
+ <dfdl:format ref="ex:GeneralFormat" lengthKind="delimited" outputNewLine="%CR;%LF;" />
+ </dfdl:defineFormat>
+ <dfdl:defineFormat name="base64">
+ <dfdl:format ref="ex:general" layerTransform="base64_MIME" layerLengthKind="boundaryMark" layerLengthUnits="bytes"
+ layerEncoding="iso-8859-1" />
+ </dfdl:defineFormat>
+ <dfdl:defineFormat name="folded">
+ <dfdl:format ref="ex:general" layerTransform="lineFolded_IMF" layerLengthKind="implicit" layerLengthUnits="bytes"
+ layerEncoding="iso-8859-1" />
+ </dfdl:defineFormat>
+ <dfdl:format ref="ex:general" />
+
+
+ <xs:element name="root" dfdl:lengthKind="implicit">
+ <xs:complexType>
+ <xs:sequence dfdl:ref="folded">
+ <xs:sequence>
+ <xs:element name="marker" dfdl:initiator="boundary=" type="xs:string" dfdl:terminator="%CR;%LF;" />
+ <xs:element name="contents" dfdl:lengthKind="implicit" dfdl:initiator="{ fn:concat('--', ../marker, '%CR;%LF;') }">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="comment" dfdl:initiator="Comment:%SP;" type="xs:string" dfdl:terminator="%CR;%LF;" />
+ <xs:element name="contentTransferEncoding" dfdl:initiator="Content-Transfer-Encoding:%SP;" type="xs:string"
+ dfdl:terminator="%CR;%LF;" />
+ <xs:element name="body" dfdl:lengthKind="implicit" dfdl:initiator="%CR;%LF;">
+ <xs:complexType>
+ <xs:choice dfdl:choiceDispatchKey="{ ../contentTransferEncoding }">
+ <xs:sequence dfdl:choiceBranchKey="base64">
+ <xs:sequence dfdl:ref="tns:base64"
+ dfdl:layerBoundaryMark="{
+ fn:concat(dfdl:decodeDFDLEntities('%CR;%LF;'),'--', ../../marker, '--')
+ }">
+ <xs:element name="value" type="xs:string" />
+ </xs:sequence>
+ </xs:sequence>
+ <!--
+ This is where other choice branches than base64 would go.
+ -->
+ </xs:choice>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+
+ </tdml:defineSchema>
+
+ <!-- useful rulers
+ 1 2 3 4 5 6 7 8
+12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
+ -->
+
+ <tdml:unparserTestCase name="layers2" root="root" model="s2" roundTrip="true">
+ <tdml:document>
+ <tdml:documentPart type="text" replaceDFDLEntities="true"><![CDATA[boundary=frontier%CR;
+--frontier%CR;
+Comment: This simulates a header field that is so long it will get folded%CR;
+ into multiple lines of text because it is too long and my job is at the%CR;
+ redundancy department is where I work.%CR;
+Content-Transfer-Encoding: base64%CR;
+%CR;
+TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFtZXQsIGNvbnNlY3RldHVyIGFkaXBpc2NpbmcgZWxpdCwg%CR;
+c2VkIGRvIGVpdXNtb2QgdGVtcG9yIGluY2lkaWR1bnQgdXQgbGFib3JlIGV0IGRvbG9yZSBtYWdu%CR;
+YSBhbGlxdWEuIFV0IGVuaW0gYWQ=%CR;
+--frontier--]]></tdml:documentPart>
+ </tdml:document>
+ <tdml:infoset>
+ <tdml:dfdlInfoset>
+ <ex:root>
+ <marker>frontier</marker>
+ <contents>
+ <comment><![CDATA[This simulates a header field that is so long it will get folded into multiple lines of text because it is too long and my job is at the redundancy department is where I work.]]></comment>
+ <contentTransferEncoding>base64</contentTransferEncoding>
+ <body>
+ <value><![CDATA[Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad]]></value>
+ </body>
+ </contents>
+ </ex:root>
+ </tdml:dfdlInfoset>
+ </tdml:infoset>
+ </tdml:unparserTestCase>
+
+ <tdml:defineSchema name="s3" elementFormDefault="unqualified">
+ <dfdl:defineFormat name="general">
+ <dfdl:format ref="ex:GeneralFormat" lengthKind="delimited" outputNewLine="%CR;%LF;" />
+ </dfdl:defineFormat>
+ <dfdl:defineFormat name="base64">
+ <dfdl:format ref="ex:general" layerTransform="base64_MIME" layerLengthKind="boundaryMark" layerLengthUnits="bytes"
+ layerEncoding="iso-8859-1" />
+ </dfdl:defineFormat>
+ <dfdl:defineFormat name="folded">
+ <dfdl:format ref="ex:general" layerTransform="lineFolded_IMF" layerLengthKind="implicit" layerLengthUnits="bytes"
+ layerEncoding="iso-8859-1" />
+ </dfdl:defineFormat>
+ <dfdl:format ref="ex:general" />
+
+
+ <xs:element name="root" dfdl:lengthKind="implicit">
+ <xs:complexType>
+ <xs:sequence dfdl:ref="folded" xmlns:foo="urn:Foo" foo:bar="shouldBeIgnored">
+ <xs:sequence>
+ <xs:element name="marker" dfdl:initiator="boundary=" type="xs:string" dfdl:terminator="%CR;%LF;" />
+ <xs:element name="nothing" type="xs:string" dfdl:initiator="xxx" />
+ </xs:sequence>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+
+ </tdml:defineSchema>
+
+ <tdml:parserTestCase name="layers3" root="root" model="s3" roundTrip="true">
+ <tdml:document>
+ <tdml:documentPart type="text" replaceDFDLEntities="true"><![CDATA[boundary=frontier%CR;
+xxx]]></tdml:documentPart>
+ </tdml:document>
+ <tdml:infoset>
+ <tdml:dfdlInfoset>
+ <ex:root>
+ <marker>frontier</marker>
+ <nothing />
+ </ex:root>
+ </tdml:dfdlInfoset>
+ </tdml:infoset>
+ </tdml:parserTestCase>
+
+ <tdml:defineSchema name="err1" elementFormDefault="unqualified">
+ <dfdl:defineFormat name="general">
+ <dfdl:format ref="ex:GeneralFormat" lengthKind="delimited" outputNewLine="%CR;%LF;" />
+ </dfdl:defineFormat>
+ <dfdl:defineFormat name="folded">
+ <dfdl:format ref="ex:general" layerTransform="lineFolded_IMF" layerLengthKind="implicit" layerLengthUnits="bytes"
+ layerEncoding="iso-8859-1" />
+ </dfdl:defineFormat>
+ <dfdl:format ref="ex:general" />
+
+
+ <xs:element name="root" dfdl:lengthKind="implicit">
+ <xs:complexType>
+ <xs:sequence dfdl:ref="folded" dfdl:separator="notAllowedInLayeredSequence"
+ xmlns:foo="urn:Foo" foo:bar="shouldBeIgnored">
+ <xs:sequence>
+ <xs:element name="marker" dfdl:initiator="boundary=" type="xs:string" dfdl:terminator="%CR;%LF;" />
+ <xs:element name="nothing" type="xs:string" dfdl:initiator="xxx" />
+ </xs:sequence>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+
+ </tdml:defineSchema>
+
+ <tdml:parserTestCase name="layersErr1" root="root" model="err1" roundTrip="true">
+ <tdml:document />
+ <tdml:errors>
+ <tdml:error>layerTransform</tdml:error>
+ <tdml:error>separator</tdml:error>
+ </tdml:errors>
+ </tdml:parserTestCase>
+
+</tdml:testSuite>
\ No newline at end of file
diff --git a/daffodil-io/src/main/scala/org/apache/daffodil/processors/charset/USASCII6BitPackedDecoder.scala b/daffodil-test/src/test/scala/org/apache/daffodil/layers/TestLayers.scala
similarity index 53%
copy from daffodil-io/src/main/scala/org/apache/daffodil/processors/charset/USASCII6BitPackedDecoder.scala
copy to daffodil-test/src/test/scala/org/apache/daffodil/layers/TestLayers.scala
index 5f59f3b..0f9752d 100644
--- a/daffodil-io/src/main/scala/org/apache/daffodil/processors/charset/USASCII6BitPackedDecoder.scala
+++ b/daffodil-test/src/test/scala/org/apache/daffodil/layers/TestLayers.scala
@@ -15,26 +15,32 @@
* limitations under the License.
*/
-package org.apache.daffodil.processors.charset
+package org.apache.daffodil.layers
-import org.apache.daffodil.schema.annotation.props.gen.BitOrder
-
-/**
- * Some encodings are not byte-oriented.
- *
- * X-DFDL-US-ASCII-6-BIT-PACKED occupies only 6 bits with each
- * code unit.
- *
+/* This section00 is for testing general features of DFDL that are
+ * not related to any specific requirement
*/
-object USASCII6BitPackedCharset
- extends NBitsWidth_BitsCharset("X-DFDL-US-ASCII-6-BIT-PACKED",
- """@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_ !"#$%&'()*+,-./0123456789:;<=>?""",
- 6, // width
- BitOrder.LeastSignificantBitFirst,
- 0x1F) { // replacement charCode for encoding of unmapped chars.
- //
- // Note: formula for computing string above is
- // def decodeString = ((0 to 63).map { charCode => (if (charCode <= 31) charCode + 64 else charCode).toChar }).mkString
+import org.junit.Test
+import org.apache.daffodil.tdml.Runner
+import org.junit.AfterClass
+
+object TestLayers {
+ lazy val testDir = "/org/apache/daffodil/layers/"
+ lazy val runner = Runner(testDir, "layers.tdml")
+
+ @AfterClass def shutDown() {
+ runner.reset
+ }
}
+class TestLayers {
+
+ import TestLayers._
+
+ @Test def test_layers1() { runner.runOneTest("layers1") }
+ @Test def test_layers2() { runner.runOneTest("layers2") }
+ @Test def test_layers3() { runner.runOneTest("layers3") }
+ @Test def test_layersErr1() { runner.runOneTest("layersErr1") }
+
+}
diff --git a/daffodil-test/src/test/scala/org/apache/daffodil/section07/external_variables/TestExternalVariablesNew.scala b/daffodil-test/src/test/scala/org/apache/daffodil/section07/external_variables/TestExternalVariablesNew.scala
index e304578..0204ed6 100644
--- a/daffodil-test/src/test/scala/org/apache/daffodil/section07/external_variables/TestExternalVariablesNew.scala
+++ b/daffodil-test/src/test/scala/org/apache/daffodil/section07/external_variables/TestExternalVariablesNew.scala
@@ -45,6 +45,6 @@ class TestExternalVariablesNew {
*/
@Test
def test_testNoRootUnnecessaryBinding(): Unit = {
- runner.trace.runOneTest("testNoRootUnnecessaryBinding")
+ runner.runOneTest("testNoRootUnnecessaryBinding")
}
}
--
To stop receiving notification emails like this one, please contact
mbeckerle@apache.org.