You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@daffodil.apache.org by mb...@apache.org on 2024/04/28 23:54:47 UTC
(daffodil) branch main updated: New Layer API & Improved Implementation
This is an automated email from the ASF dual-hosted git repository.
mbeckerle pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/daffodil.git
The following commit(s) were added to refs/heads/main by this push:
new fab8bf5d2 New Layer API & Improved Implementation
fab8bf5d2 is described below
commit fab8bf5d20b26a56912ab2a736a2bf5275f5a36a
Author: Michael Beckerle <mb...@apache.org>
AuthorDate: Fri Feb 23 15:17:47 2024 -0500
New Layer API & Improved Implementation
This commit fixes the remaining issues open about Daffodil's Layers
extension/plugins. Partial progress was in commit ff9ba4. This
finishes that work.
- DAFFODIL-2845 - Layer Improvements (umbrella ticket)
- DAFFODIL-2844 - LayerNotEnoughDataException ...
- DAFFODIL-2825 - Define Supported Layering API
- DAFFODIL-2841 - Layers do not support created SDE or PE/UE Errors
- DAFFODIL-2843 - How to create PE/UE from a layer
The layer API is all defined in Java clases and interfaces now. One
of the built-in layers (Gzip) is written in Java now.
There is only one layer property now: dfdlx:layer which takes a QName of
the layer class.
Layer classes are derived from
org.apache.daffodil.runtime1.layers.api.Layer
A layer a name and target namespace, and hence a QName is required to
identify it. This namespace is OWNED by the layer. All DFDL variables
defined in that namespace are either used to pass parameters to the
layer code, or receive results (such as a checksum) back from the
layer code. This is enforced. A layer that has no DFDL variables does
not have to define a DFDL schema that defines the layer's target
namespace, but any layer that uses DFDL variables *must* define a
schema with the layer's namespace as its target namespace, with the
variables declared in it (using dfdl:defineVariable).
There is also an abstract base class for defining checksum layers
called,
org.apache.daffodil.runtime1.layers.api.ChecksumLayer
All the layer properties about layer length kinds etc. are gone.
dfdlx:layerLengthKind='boundaryMark' and dfdlx:boundaryMark properties
are replaced by a layer named 'boundaryMark'.
dfdlx:layerLengthKind='explicit' and dfdlx:layerLength properties are
replaced by a layer named 'fixedLength'.
dfdlx:layerLengthKind='implicit' is replaced by just not using one of
the two above layers.
dfdlx:layerTransform is replaced by just dfdlx:layer and is now a
QName. Layers no longer have both transformers and length-limiters
mixed together.
There are two ways to do explicit layer limiting, that differ in an
important way.
If you use 'implicit' layer length (no restriction layer), and
constrain the layer length by a surrounding element of specified
length, then the layer length is controlled when parsing, but NOT
controlled when unparsing.
If you use the layer "fixedLength", that dictates the parse and
unparse length to be the value of the layerLength variable. Similarly
if you use a checksum layer built with the ChecksumLayer base class,
the length is controlled for both parsing and unparsing as it works
like the fixedLength layer.
Layer Variables:
Each variable is either a parameter passed to a special setter named
setLayerVariableParameters, or is a return variable which is populated
from a special getter. The special setter args and getters taken
together must use up all the DFDL variables defined in the layer's
namespace. (See the javadoc for the Layer base class.)
A special getter example might be:
int getLayerVariableResult_checksum() { .... }
This corresponds to, and returns the value that populates, the DFDL
variable named 'checksum' in the layer's namespace. That DFDL variable
must have a DFDL type corresponding to the int type (xs:unsignedShort or
xs:int).
Layers could have multiple result variables, but we have no actual
examples of more than one value being returned. (Returning multiple
variables is tested. We just have no real use cases for it at this
time.)
Layer Exception Handling:
Added setProcessingErrorException(...) method to Layer
This allows the layer to specify that if the layer throws specific
exceptions or runtime exceptions that they are converted into processing
errors. This eliminates most need for layers to contain try-catches.
Additional Fixes in this Commit:
Fixed gzip layer which was not constructing on Java 17 or 21 due to
static initializer differences of some sort.
Removed restriction that a layer can only have a single term inside it.
Added close() to InputSourceDataInputStream, as we weren't closing these.
They now implement java.io.Closeable so can use try-with-resources for auto
closing in Java.
There are a large number of simplifications in the "layering" of the various
layer-related implementation classes.
LayerRegistry is gone, merged into LayerFactory, which is gone, merged into
LayerDriver.
In the examples/tests for layering: Changed check digit bad digit style to
construct an always-invalid element instead of an assertion failure.
About Testing:
The testing is quite extensive now. There are tests for every way that
a user can goof up the definition of a layer class, and there are
tests for processingError, runtimeSchemaDefinitionError, and throwing
an Exception from every place a user-defined Layer could cause these.
Parse errors cause backtracking in all sensible cases. Nothing aborts.
Protecting every place that layer code is called from any sort of
error/throw requires quite a number of try/catch blocks. This may have
performance implications.
Add some fuzz testing around gzip layer
GZip layer converts IOExceptions to PE explicitly.
Fuzz tests show that gzip does a good job converting
various bad data issues into IOException, so
really that's the only one the GzipLayer has to deal with.
Changed boundaryMarkLayer and gzip layer and AIS (test) layer
to use setProcessingErrorException to handle IOExceptions.
(and the test SimpleBombOutLayer.scala used to drive negative tests.)
Added bug-related comments to other files about bugs discovered
while fuzz-testing of other non-daffodil layer schemas.
DEPRECATION/COMPATABILITY
This new Layer API is 100% incompatible with schemas or layer code
from Daffodil 3.6.0 and all prior versions of Daffodil. The layer
feature was just an experimental feature in Daffodil, so we reserved
the right to change it.
However, it is our hope that the new Layer API introduced by this commit
will prove to be stable and supportable indefinitely.
DAFFODIL-2845, DAFFODIL-2844, DAFFODIL-2825, DAFFODIL-2841, DAFFODIL-2843
---
DEVELOP.md | 2 +-
.../main/scala/org/apache/daffodil/cli/Main.scala | 250 ++--
... org.apache.daffodil.runtime1.layers.api.Layer} | 2 +-
.../org/apache/daffodil/layers/buggy.dfdl.xsd | 7 +-
.../apache/daffodil/layers/xsd/buggyLayer.dfdl.xsd | 13 +-
.../test/resources/test/cli/fixedLength.dfdl.xsd | 4 +-
.../daffodil/cli/cliTest/TestCLIParsing.scala | 2 +-
.../org/apache/daffodil/layers/BuggyLayer.scala | 32 +-
.../apache/daffodil/layers/BuggyTransformer.scala | 73 --
.../apache/daffodil/core/compiler/Compiler.scala | 3 +
.../apache/daffodil/core/dpath/Expression.scala | 7 +-
.../daffodil/core/dsom/DFDLFormatAnnotation.scala | 14 +-
.../org/apache/daffodil/core/dsom/ModelGroup.scala | 148 ++-
.../RawCommonRuntimeValuedPropertiesMixin.scala | 13 -
.../daffodil/core/dsom/RuntimePropertyMixins.scala | 92 +-
.../org/apache/daffodil/core/dsom/SchemaSet.scala | 3 +-
.../apache/daffodil/core/dsom/SequenceGroup.scala | 96 +-
.../core/grammar/SequenceGrammarMixin.scala | 12 +-
.../core/grammar/primitives/LayeredSequence.scala | 40 +-
.../daffodil/core/layers/LayerSchemaCompiler.scala | 73 ++
.../core/runtime1/SchemaSetRuntime1Mixin.scala | 18 +-
.../core/runtime1/SequenceTermRuntime1Mixin.scala | 2 +
.../core/runtime1/VariableRuntime1Mixin.scala | 11 +-
.../daffodil/core/api/TestMetadataWalking.scala | 8 +-
.../core/dpath/TestDFDLExpressionEvaluation.scala | 16 +-
.../daffodil/core/dsom/TestBinaryInput_01.scala | 6 +-
.../core/layers/TestBoundaryMarkLayer.scala | 134 ++
.../core/layers/TestFixedLengthLayer.scala | 154 +++
.../daffodil/core/layers/TestGzipErrors.scala | 285 +++++
.../apache/daffodil/core/layers/TestLayers.scala | 281 ++++-
.../org/apache/daffodil/core/util/FuzzData.scala | 235 ++++
.../org/apache/daffodil/core/util/TestUtils.scala | 19 +
...dil.io.processors.charset.BitsCharsetDefinition | 1 -
.../apache/daffodil/io/BoundaryMarkStreams.scala | 99 ++
.../org/apache/daffodil/io/DataInputStream.scala | 6 +
.../io/DirectOrBufferedDataOutputStream.scala | 135 +-
.../scala/org/apache/daffodil/io/InputSource.scala | 11 +
.../daffodil/io/InputSourceDataInputStream.scala | 8 +-
...treams.scala => RegexLimitingInputStream.scala} | 133 +-
.../io/StringDataInputStreamForUnparse.scala | 4 +
.../io/processors/charset/AISPayloadArmoring.scala | 52 -
.../io/processors/charset/BitsCharset.scala | 3 +
.../charset/BitsCharsetNonByteSize.scala | 4 +-
.../apache/daffodil/io/TestDataOutputStream.scala | 52 +-
.../apache/daffodil/io/TestDataOutputStream2.scala | 6 +-
.../apache/daffodil/io/TestDataOutputStream3.scala | 52 +-
.../apache/daffodil/io/TestDataOutputStream4.scala | 2 +-
.../org/apache/daffodil/layers/TestBase64.scala | 8 +-
.../apache/daffodil/layers/TestJavaIOStreams.scala | 5 +-
.../layers/TestLimitingJavaIOStreams.scala | 109 +-
.../japi/io/InputSourceDataInputStream.scala | 13 +-
.../daffodil/example/TestCustomDebuggerAPI.java | 12 +-
.../org/apache/daffodil/example/TestJavaAPI.java | 1083 ++++++++--------
.../daffodil/example/ValidatorApiExample.java | 46 +-
.../daffodil/example/ValidatorSpiExample.java | 18 +-
.../org/apache/daffodil/lib/api/Diagnostic.scala | 12 +-
.../lib/schema/annotation/props/ByHandMixins.scala | 11 +-
.../schema/annotation/props/PropertyScoping.scala | 20 +-
.../scala/org/apache/daffodil/lib/util/Misc.scala | 2 +-
.../org/apache/daffodil/lib/xml/Namespaces.scala | 2 +-
.../org/apache/daffodil/lib/xml/QNameBase.scala | 60 +-
.../org/apache/daffodil/lib/xml/XMLUtils.scala | 3 +
.../apache/daffodil/xsd/DFDL_part2_attributes.xsd | 4 +-
.../resources/org/apache/daffodil/xsd/dafext.xsd | 4 +-
.../resources/org/apache/daffodil/xsd/dfdlx.xsd | 114 +-
.../daffodil/propGen/PropertyGenerator.scala | 7 +-
... org.apache.daffodil.runtime1.layers.api.Layer} | 14 +-
.../daffodil/layers/xsd/base64_MIMELayer.dfdl.xsd | 32 +
.../daffodil/layers/xsd/boundaryMarkLayer.dfdl.xsd | 27 +-
.../daffodil/layers/xsd/byteSwapLayer.dfdl.xsd | 29 +-
.../daffodil/layers/xsd/fixedLengthLayer.dfdl.xsd | 26 +-
.../apache/daffodil/layers/xsd/gzipLayer.dfdl.xsd | 27 +-
.../daffodil/layers/xsd/lineFoldedLayer.dfdl.xsd | 26 +-
.../daffodil/layers/runtime1/Base64MimeLayer.scala | 27 +-
.../layers/runtime1/Base64Transformer.scala | 99 --
.../layers/runtime1/BoundaryMarkLayer.scala | 99 ++
...teSwapTransformer.scala => ByteSwapLayer.scala} | 85 +-
.../layers/runtime1/FixedLengthLayer.scala | 134 ++
.../{GZipTransformer.scala => GZipLayer.java} | 144 +--
.../daffodil/layers/runtime1/LineFoldMode.java | 23 +-
...Transformer.scala => LineFoldedLayerBase.scala} | 237 +---
.../daffodil/layers/runtime1/TestGzipLayer.scala | 59 +
.../TestLengthLimitedLineFoldingStreams.scala | 90 +-
.../runtime1/LayeredSequenceUnparser.scala | 86 +-
.../daffodil/runtime1/dpath/DPathRuntime.scala | 3 +-
.../apache/daffodil/runtime1/dpath/NodeInfo.scala | 31 +
.../org/apache/daffodil/runtime1/dsom/SDE.scala | 10 +-
.../daffodil/runtime1/infoset/InfosetWalker.scala | 4 +
.../runtime1/layers/ChecksumLayerBase.scala | 147 +++
.../daffodil/runtime1/layers/LayerCompiler.scala | 135 --
.../runtime1/layers/LayerCompilerRegistry.scala | 49 -
.../daffodil/runtime1/layers/LayerDriver.scala | 420 +++++++
.../layers/LayerNotEnoughDataException.java | 36 +
.../runtime1/layers/LayerProcessingException.java | 44 +
...TransformerFactory.scala => LayerRuntime.scala} | 32 +-
.../runtime1/layers/LayerRuntimeCompiler.scala | 279 ++++
.../runtime1/layers/LayerRuntimeData.scala | 45 +
.../runtime1/layers/LayerTransformer.scala | 453 -------
.../runtime1/layers/LayerUnexpectedException.java | 23 +-
.../daffodil/runtime1/layers/LayerUtils.scala | 93 ++
.../runtime1/layers/LayerVarsRuntime.scala | 137 ++
...sformerFactory.scala => ScalaLayerHelper.scala} | 20 +-
.../runtime1/layers/api/ChecksumLayer.java | 87 ++
.../apache/daffodil/runtime1/layers/api/Layer.java | 212 ++++
.../processors/DaffodilParseXMLReader.scala | 16 +-
.../runtime1/processors/DataProcessor.scala | 4 +-
.../daffodil/runtime1/processors/EvLayering.scala | 54 -
.../runtime1/processors/ProcessorStateBases.scala | 43 +
.../daffodil/runtime1/processors/RuntimeData.scala | 17 +-
.../runtime1/processors/SchemaSetRuntimeData.scala | 27 +
.../runtime1/processors/VariableMap1.scala | 66 +-
.../processors/dfa/TextDelimitedParser.scala | 181 +--
.../processors/parsers/LayeredSequenceParser.scala | 62 +-
.../runtime1/processors/parsers/PState.scala | 2 +
.../runtime1/processors/parsers/ParseErrors.scala | 4 +-
.../sapi/io/InputSourceDataInputStream.scala | 10 +-
.../daffodil/example/TestCustomDebuggerAPI.scala | 12 +-
.../org/apache/daffodil/example/TestScalaAPI.scala | 1000 +++++++--------
.../example/ValidatorExamplesSupport.scala | 5 +-
.../org/apache/daffodil/tdml/TDMLRunner.scala | 31 +-
.../processor/tdml/DaffodilTDMLDFDLProcessor.scala | 46 +-
... org.apache.daffodil.runtime1.layers.api.Layer} | 29 +-
.../org/apache/daffodil/layers/TestCheckDigit.tdml | 17 +-
.../apache/daffodil/layers/TestGzipFoldB64.tdml | 159 +++
.../daffodil/layers/{IPv4.tdml => TestIPv4.tdml} | 50 +-
.../org/apache/daffodil/layers/TestLayers.tdml | 242 ++++
.../org/apache/daffodil/layers/TestLayersBomb.tdml | 1327 ++++++++++++++++++++
.../org/apache/daffodil/layers/ais.dfdl.xsd | 151 +++
.../resources/org/apache/daffodil/layers/ais.tdml | 85 +-
...ckDigit.dfdl.xsd => exampleCheckDigit.dfdl.xsd} | 57 +-
.../daffodil/layers/exampleGzipLayer.dfdl.xsd | 82 ++
.../daffodil/layers/exampleGzipLayer2.dfdl.xsd | 132 ++
.../layers/{IPv4.dfdl.xsd => exampleIPv4.dfdl.xsd} | 40 +-
.../layers/exampleLineFoldedLayer.dfdl.xsd | 100 ++
...r.dfdl.xsd => exampleLineFoldedLayer2.dfdl.xsd} | 48 +-
...fdl.xsd => exampleLineFoldedLayer3Err.dfdl.xsd} | 50 +-
.../org/apache/daffodil/layers/layers.tdml | 379 ------
.../daffodil/layers/useAllTypesLayer.dfdl.xsd | 144 +++
.../daffodil/layers/useSimpleBombLayer.dfdl.xsd | 151 +++
.../daffodil/layers/useSimpleTestLayers.dfdl.xsd | 386 ++++++
.../daffodil/layers/xsd/AllTypesLayer.dfdl.xsd | 78 ++
.../daffodil/layers/xsd/IPv4ChecksumLayer.dfdl.xsd | 17 +-
...mLayer.dfdl.xsd => SimpleBombOutLayer.dfdl.xsd} | 30 +-
.../daffodil/layers/xsd/SimpleTestLayer.dfdl.xsd | 25 +-
.../layers/xsd/SimpleTestLayerNoIntVar.dfdl.xsd | 25 +-
.../layers/xsd/SimpleTestLayerNoStringVar.dfdl.xsd | 25 +-
.../daffodil/layers/xsd/checkDigitLayer.dfdl.xsd | 23 +-
...sformer.scala => AISPayloadArmoringLayer.scala} | 164 +--
.../daffodil/runtime1/layers/AllTypesLayer.scala | 128 ++
.../daffodil/runtime1/layers/CheckDigit.scala | 84 --
.../daffodil/runtime1/layers/CheckDigitLayer.scala | 114 ++
.../runtime1/layers/CheckDigitLayerCompiler.scala | 92 --
.../layers/IPv4ChecksumLayerCompiler.scala | 76 --
.../runtime1/layers/SimpleBombOutLayer.scala | 177 +++
.../daffodil/runtime1/layers/SimpleTestLayer.scala | 182 +++
.../layers}/TestAISPayloadArmoringEncoder.scala | 5 +-
.../daffodil/runtime1/layers/TestAISStreams.scala | 8 +-
.../daffodil/runtime1/layers/TestCheckDigit.scala | 225 +++-
.../{TestLayers.scala => TestGzipFoldB64.scala} | 12 +-
.../apache/daffodil/runtime1/layers/TestIPv4.scala | 2 +-
.../daffodil/runtime1/layers/TestLayers.scala | 35 +-
.../daffodil/runtime1/layers/TestLayers2.scala | 216 ++++
.../IPv4ChecksumLayer.scala} | 61 +-
163 files changed, 9907 insertions(+), 4786 deletions(-)
diff --git a/DEVELOP.md b/DEVELOP.md
index 37b5711af..4c8ad7f13 100644
--- a/DEVELOP.md
+++ b/DEVELOP.md
@@ -234,7 +234,7 @@ daffodil/
├── daffodil-macro-lib/ - Defines Daffodil's Scala 2.x macros
├── daffodil-propgen/ - Contains generators to generate more source code
├── daffodil-runtime1/ - Contains Daffodil's Scala parser
-├── daffodil-runtime1-layers/ - Contains Daffodil's Scala layer transformers
+├── daffodil-runtime1-layers/ - Contains Daffodil's Scala layers
├── daffodil-runtime1-unparser/ - Contains Daffodil's Scala unparser
├── daffodil-sapi/ - Contains Daffodil's Scala API
├── daffodil-schematron/
diff --git a/daffodil-cli/src/main/scala/org/apache/daffodil/cli/Main.scala b/daffodil-cli/src/main/scala/org/apache/daffodil/cli/Main.scala
index dcf8214b2..e7eb7494d 100644
--- a/daffodil-cli/src/main/scala/org/apache/daffodil/cli/Main.scala
+++ b/daffodil-cli/src/main/scala/org/apache/daffodil/cli/Main.scala
@@ -45,6 +45,7 @@ import org.apache.daffodil.core.dsom.ExpressionCompilers
import org.apache.daffodil.io.DataDumper
import org.apache.daffodil.io.FormatInfo
import org.apache.daffodil.io.InputSourceDataInputStream
+import org.apache.daffodil.lib.Implicits.using
import org.apache.daffodil.lib.api.DaffodilConfig
import org.apache.daffodil.lib.api.DaffodilConfigException
import org.apache.daffodil.lib.api.DaffodilTunables
@@ -70,7 +71,7 @@ import org.apache.daffodil.runtime1.debugger.DebuggerExitException
import org.apache.daffodil.runtime1.debugger.InteractiveDebugger
import org.apache.daffodil.runtime1.debugger.TraceDebuggerRunner
import org.apache.daffodil.runtime1.externalvars.ExternalVariablesLoader
-import org.apache.daffodil.runtime1.layers.LayerExecutionException
+import org.apache.daffodil.runtime1.layers.LayerFatalException
import org.apache.daffodil.runtime1.processors.DataLoc
import org.apache.daffodil.runtime1.processors.DataProcessor
import org.apache.daffodil.runtime1.processors.ExternalVariableException
@@ -1170,141 +1171,141 @@ class Main(
new FileInputStream(f)
}
}
- val inStream = InputSourceDataInputStream(input)
+ using(InputSourceDataInputStream(input)) { inStream =>
+ val output = parseOpts.output.toOption match {
+ case Some("-") | None => STDOUT
+ case Some(file) => new FileOutputStream(file)
+ }
- val output = parseOpts.output.toOption match {
- case Some("-") | None => STDOUT
- case Some(file) => new FileOutputStream(file)
- }
+ val infosetHandler = InfosetType.getInfosetHandler(
+ parseOpts.infosetType(),
+ processor,
+ parseOpts.schema.map(_.uri).toOption,
+ forPerformance = false,
+ )
- val infosetHandler = InfosetType.getInfosetHandler(
- parseOpts.infosetType(),
- processor,
- parseOpts.schema.map(_.uri).toOption,
- forPerformance = false,
- )
+ var lastParseBitPosition = 0L
+ var keepParsing = true
+ var exitCode = ExitCode.Success
- var lastParseBitPosition = 0L
- var keepParsing = true
- var exitCode = ExitCode.Success
+ while (keepParsing) {
+ val infosetResult =
+ Timer.getResult("parsing", infosetHandler.parse(inStream, output))
+ val parseResult = infosetResult.parseResult
- while (keepParsing) {
- val infosetResult =
- Timer.getResult("parsing", infosetHandler.parse(inStream, output))
- val parseResult = infosetResult.parseResult
+ val finfo = parseResult.resultState.asInstanceOf[FormatInfo]
+ val loc = parseResult.resultState.currentLocation.asInstanceOf[DataLoc]
+ displayDiagnostics(parseResult)
- val finfo = parseResult.resultState.asInstanceOf[FormatInfo]
- val loc = parseResult.resultState.currentLocation.asInstanceOf[DataLoc]
- displayDiagnostics(parseResult)
-
- if (parseResult.isProcessingError || parseResult.isValidationError) {
- keepParsing = false
- exitCode = ExitCode.ParseError
- } else {
- // Success. Some InfosetHandlers do not write the result to the output
- // stream when parsing (e.g. they just create Scala objects). Since we are
- // parsing, ask the InfosetHandler to serialize the result if it has an
- // implementation so that the user can see an XML represenation regardless
- // of the infoset type.
- infosetResult.write(output)
- output.flush()
-
- if (!inStream.hasData()) {
- // not even 1 more bit is available.
- // do not try to keep parsing, nothing left to parse
+ if (parseResult.isProcessingError || parseResult.isValidationError) {
keepParsing = false
+ exitCode = ExitCode.ParseError
} else {
- // There is more data available.
- if (parseOpts.stream.toOption.get) {
- // Streaming mode
- if (lastParseBitPosition == loc.bitPos0b) {
- // this parse consumed no data, that means this would get
- // stuck in an infinite loop if we kept trying to stream,
- // so we need to quit
+ // Success. Some InfosetHandlers do not write the result to the output
+ // stream when parsing (e.g. they just create Scala objects). Since we are
+ // parsing, ask the InfosetHandler to serialize the result if it has an
+ // implementation so that the user can see an XML represenation regardless
+ // of the infoset type.
+ infosetResult.write(output)
+ output.flush()
+
+ if (!inStream.hasData()) {
+ // not even 1 more bit is available.
+ // do not try to keep parsing, nothing left to parse
+ keepParsing = false
+ } else {
+ // There is more data available.
+ if (parseOpts.stream.toOption.get) {
+ // Streaming mode
+ if (lastParseBitPosition == loc.bitPos0b) {
+ // this parse consumed no data, that means this would get
+ // stuck in an infinite loop if we kept trying to stream,
+ // so we need to quit
+ val remainingBits =
+ if (loc.bitLimit0b.isDefined) {
+ (loc.bitLimit0b.get - loc.bitPos0b).toString
+ } else {
+ "at least " + (inStream.inputSource.knownBytesAvailable * 8)
+ }
+ Logger.log.error(
+ s"Left over data after consuming 0 bits while streaming. Stopped after consuming ${loc.bitPos0b} bit(s) with ${remainingBits} bit(s) remaining.",
+ )
+ keepParsing = false
+ exitCode = ExitCode.LeftOverData
+ } else {
+ // last parse did consume data, and we know there is more
+ // data to come, so try to parse again.
+ lastParseBitPosition = loc.bitPos0b
+ keepParsing = true
+ output.write(0) // NUL-byte separates streams
+ }
+ } else {
+ // not streaming mode, and there is more data available,
+ // so show left over data warning
+ val Dump = new DataDumper
+ val bitsAlreadyConsumed = (loc.bitPos0b % 8).toInt
+ val firstByteString = if (bitsAlreadyConsumed != 0) {
+ val bitsToDisplay = 8 - bitsAlreadyConsumed
+ val pbp = inStream.inputSource.position + 1
+ val firstByteBitArray = inStream.getByteArray(bitsToDisplay, finfo)
+ val fbs = firstByteBitArray(0).toBinaryString
+ .takeRight(8)
+ .reverse
+ .padTo(8, '0')
+ .reverse
+ val bits = if (finfo.bitOrder == BitOrder.MostSignificantBitFirst) {
+ "x" * bitsAlreadyConsumed + fbs.dropRight(bitsAlreadyConsumed)
+ } else {
+ fbs.takeRight(bitsToDisplay) + "x" * bitsAlreadyConsumed
+ }
+ val dumpString =
+ f"\nLeft over data starts with partial byte. Left over data (Binary) at byte $pbp is: (0b$bits)"
+ dumpString
+ } else ""
+ val curBytePosition1b = inStream.inputSource.position + 1
+ val bytesAvailable = inStream.inputSource.knownBytesAvailable
+ val bytesLimit = math.min(8, bytesAvailable).toInt
+ val destArray = new Array[Byte](bytesLimit)
+ val destArrayFilled = inStream.inputSource.get(destArray, 0, bytesLimit)
+ val dumpString =
+ if (destArrayFilled)
+ Dump
+ .dump(
+ Dump.TextOnly(Some("utf-8")),
+ 0,
+ destArray.length * 8,
+ ByteBuffer.wrap(destArray),
+ includeHeadingLine = false,
+ )
+ .mkString("\n")
+ else ""
+ val dataText =
+ if (destArrayFilled)
+ s"\nLeft over data (UTF-8) starting at byte ${curBytePosition1b} is: (${dumpString}...)"
+ else ""
+ val dataHex =
+ if (destArrayFilled)
+ s"\nLeft over data (Hex) starting at byte ${curBytePosition1b} is: (0x${destArray.map { a =>
+ f"$a%02x"
+ }.mkString}...)"
+ else ""
val remainingBits =
if (loc.bitLimit0b.isDefined) {
(loc.bitLimit0b.get - loc.bitPos0b).toString
} else {
- "at least " + (inStream.inputSource.knownBytesAvailable * 8)
+ "at least " + (bytesAvailable * 8)
}
- Logger.log.error(
- s"Left over data after consuming 0 bits while streaming. Stopped after consuming ${loc.bitPos0b} bit(s) with ${remainingBits} bit(s) remaining.",
- )
+ val leftOverDataMessage =
+ s"Left over data. Consumed ${loc.bitPos0b} bit(s) with ${remainingBits} bit(s) remaining." + firstByteString + dataHex + dataText
+ Logger.log.error(leftOverDataMessage)
keepParsing = false
exitCode = ExitCode.LeftOverData
- } else {
- // last parse did consume data, and we know there is more
- // data to come, so try to parse again.
- lastParseBitPosition = loc.bitPos0b
- keepParsing = true
- output.write(0) // NUL-byte separates streams
}
- } else {
- // not streaming mode, and there is more data available,
- // so show left over data warning
- val Dump = new DataDumper
- val bitsAlreadyConsumed = (loc.bitPos0b % 8).toInt
- val firstByteString = if (bitsAlreadyConsumed != 0) {
- val bitsToDisplay = 8 - bitsAlreadyConsumed
- val pbp = inStream.inputSource.position + 1
- val firstByteBitArray = inStream.getByteArray(bitsToDisplay, finfo)
- val fbs = firstByteBitArray(0).toBinaryString
- .takeRight(8)
- .reverse
- .padTo(8, '0')
- .reverse
- val bits = if (finfo.bitOrder == BitOrder.MostSignificantBitFirst) {
- "x" * bitsAlreadyConsumed + fbs.dropRight(bitsAlreadyConsumed)
- } else {
- fbs.takeRight(bitsToDisplay) + "x" * bitsAlreadyConsumed
- }
- val dumpString =
- f"\nLeft over data starts with partial byte. Left over data (Binary) at byte $pbp is: (0b$bits)"
- dumpString
- } else ""
- val curBytePosition1b = inStream.inputSource.position + 1
- val bytesAvailable = inStream.inputSource.knownBytesAvailable
- val bytesLimit = math.min(8, bytesAvailable).toInt
- val destArray = new Array[Byte](bytesLimit)
- val destArrayFilled = inStream.inputSource.get(destArray, 0, bytesLimit)
- val dumpString =
- if (destArrayFilled)
- Dump
- .dump(
- Dump.TextOnly(Some("utf-8")),
- 0,
- destArray.length * 8,
- ByteBuffer.wrap(destArray),
- includeHeadingLine = false,
- )
- .mkString("\n")
- else ""
- val dataText =
- if (destArrayFilled)
- s"\nLeft over data (UTF-8) starting at byte ${curBytePosition1b} is: (${dumpString}...)"
- else ""
- val dataHex =
- if (destArrayFilled)
- s"\nLeft over data (Hex) starting at byte ${curBytePosition1b} is: (0x${destArray.map { a =>
- f"$a%02x"
- }.mkString}...)"
- else ""
- val remainingBits =
- if (loc.bitLimit0b.isDefined) {
- (loc.bitLimit0b.get - loc.bitPos0b).toString
- } else {
- "at least " + (bytesAvailable * 8)
- }
- val leftOverDataMessage =
- s"Left over data. Consumed ${loc.bitPos0b} bit(s) with ${remainingBits} bit(s) remaining." + firstByteString + dataHex + dataText
- Logger.log.error(leftOverDataMessage)
- keepParsing = false
- exitCode = ExitCode.LeftOverData
}
}
}
+ exitCode
}
- exitCode
}
}
rc
@@ -1411,11 +1412,12 @@ class Main(
})
case Right(bytes) =>
Timer.getTimeResult({
- val input = InputSourceDataInputStream(bytes)
- val infosetResult =
- infosetHandler.parse(input, nullOutputStreamForParse)
- val parseResult = infosetResult.parseResult
- parseResult
+ using(InputSourceDataInputStream(bytes)) { input =>
+ val infosetResult =
+ infosetHandler.parse(input, nullOutputStreamForParse)
+ val parseResult = infosetResult.parseResult
+ parseResult
+ }
})
}
@@ -1993,8 +1995,8 @@ class Main(
Logger.log.error(e.message)
ExitCode.ConfigError
}
- case e: LayerExecutionException => {
- Logger.log.error(e.message, e)
+ case e: LayerFatalException => {
+ Logger.log.error(e.getMessage, e)
ExitCode.LayerExecutionError
}
case e: Exception => {
diff --git a/daffodil-cli/src/test/resources/META-INF/services/org.apache.daffodil.runtime1.layers.LayerCompiler b/daffodil-cli/src/test/resources/META-INF/services/org.apache.daffodil.runtime1.layers.api.Layer
similarity index 94%
rename from daffodil-cli/src/test/resources/META-INF/services/org.apache.daffodil.runtime1.layers.LayerCompiler
rename to daffodil-cli/src/test/resources/META-INF/services/org.apache.daffodil.runtime1.layers.api.Layer
index 51a19c3f0..541974133 100644
--- a/daffodil-cli/src/test/resources/META-INF/services/org.apache.daffodil.runtime1.layers.LayerCompiler
+++ b/daffodil-cli/src/test/resources/META-INF/services/org.apache.daffodil.runtime1.layers.api.Layer
@@ -13,4 +13,4 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-org.apache.daffodil.layers.BuggyLayerCompiler
+org.apache.daffodil.layers.BuggyLayer
diff --git a/daffodil-cli/src/test/resources/org/apache/daffodil/layers/buggy.dfdl.xsd b/daffodil-cli/src/test/resources/org/apache/daffodil/layers/buggy.dfdl.xsd
index eb67855fc..36384d772 100644
--- a/daffodil-cli/src/test/resources/org/apache/daffodil/layers/buggy.dfdl.xsd
+++ b/daffodil-cli/src/test/resources/org/apache/daffodil/layers/buggy.dfdl.xsd
@@ -21,11 +21,8 @@
xmlns="http://www.w3.org/2001/XMLSchema"
xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions"
- xmlns:fn="http://www.w3.org/2005/xpath-functions"
- xmlns:daf="urn:ogf:dfdl:2013:imp:daffodil.apache.org:2018:ext"
xmlns:buggy="urn:org.apache.daffodil.layers.buggy"
xmlns:ex="http://example.com"
- xmlns:tns="http://example.com"
targetNamespace="http://example.com">
<include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd" />
@@ -36,13 +33,13 @@
<annotation>
<appinfo source="http://www.ogf.org/dfdl/">
- <dfdl:format ref="tns:GeneralFormat" encoding="ascii" />
+ <dfdl:format ref="ex:GeneralFormat" encoding="ascii" />
</appinfo>
</annotation>
<element name="r">
<complexType>
- <sequence dfdl:ref="buggy:buggyFormat" dfdlx:layerLength="10">
+ <sequence dfdlx:layer="buggy:buggy">
<element name="value" type="xs:string" dfdl:lengthKind="explicit" dfdl:length="1"/>
</sequence>
</complexType>
diff --git a/daffodil-cli/src/test/resources/org/apache/daffodil/layers/xsd/buggyLayer.dfdl.xsd b/daffodil-cli/src/test/resources/org/apache/daffodil/layers/xsd/buggyLayer.dfdl.xsd
index 7ddbe4bce..f3e75b75e 100644
--- a/daffodil-cli/src/test/resources/org/apache/daffodil/layers/xsd/buggyLayer.dfdl.xsd
+++ b/daffodil-cli/src/test/resources/org/apache/daffodil/layers/xsd/buggyLayer.dfdl.xsd
@@ -28,17 +28,6 @@
<include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd" />
- <annotation>
- <appinfo source="http://www.ogf.org/dfdl/">
-
- <dfdl:format ref="buggy:GeneralFormat" />
-
- <dfdl:defineFormat name="buggyFormat">
- <dfdl:format dfdlx:layerTransform="buggy" dfdlx:layerLengthKind="explicit" dfdlx:layerLengthUnits="bytes"
- dfdlx:layerEncoding="ascii" />
- </dfdl:defineFormat>
-
- </appinfo>
- </annotation>
+ <!-- This layer has no parameters nor return variables -->
</schema>
diff --git a/daffodil-cli/src/test/resources/test/cli/fixedLength.dfdl.xsd b/daffodil-cli/src/test/resources/test/cli/fixedLength.dfdl.xsd
index 1e70c9342..6c59c8059 100644
--- a/daffodil-cli/src/test/resources/test/cli/fixedLength.dfdl.xsd
+++ b/daffodil-cli/src/test/resources/test/cli/fixedLength.dfdl.xsd
@@ -21,14 +21,14 @@
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
- xmlns:tns="http://www.example.org/example1/"
+ xmlns:ex="http://www.example.org/example1/"
targetNamespace="http://www.example.org/example1/">
<xs:include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
- <dfdl:format ref="tns:GeneralFormat"/>
+ <dfdl:format ref="ex:GeneralFormat"/>
</xs:appinfo>
</xs:annotation>
diff --git a/daffodil-cli/src/test/scala/org/apache/daffodil/cli/cliTest/TestCLIParsing.scala b/daffodil-cli/src/test/scala/org/apache/daffodil/cli/cliTest/TestCLIParsing.scala
index b68e32bf0..5bddb2871 100644
--- a/daffodil-cli/src/test/scala/org/apache/daffodil/cli/cliTest/TestCLIParsing.scala
+++ b/daffodil-cli/src/test/scala/org/apache/daffodil/cli/cliTest/TestCLIParsing.scala
@@ -939,7 +939,7 @@ class TestCLIParsing {
)
runCLI(args"parse -s $schema") { cli =>
cli.sendLine("0", inputDone = true)
- cli.expectErr("Unexpected exception in layer transformer 'buggy'")
+ cli.expectErr("bad input stream")
}(ExitCode.LayerExecutionError)
}
diff --git a/daffodil-io/src/main/scala/org/apache/daffodil/io/LimitingJavaOutputStreams.scala b/daffodil-cli/src/test/scala/org/apache/daffodil/layers/BuggyLayer.scala
similarity index 57%
copy from daffodil-io/src/main/scala/org/apache/daffodil/io/LimitingJavaOutputStreams.scala
copy to daffodil-cli/src/test/scala/org/apache/daffodil/layers/BuggyLayer.scala
index ed54a7662..677f689b4 100644
--- a/daffodil-io/src/main/scala/org/apache/daffodil/io/LimitingJavaOutputStreams.scala
+++ b/daffodil-cli/src/test/scala/org/apache/daffodil/layers/BuggyLayer.scala
@@ -14,27 +14,27 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.daffodil.io
-import java.nio.charset.Charset
+package org.apache.daffodil.layers
-class LayerBoundaryMarkInsertingJavaOutputStream(
- jos: java.io.OutputStream,
- boundaryMark: String,
- charset: Charset,
-) extends java.io.FilterOutputStream(jos) {
+import java.io.IOException
+import java.io.InputStream
+import java.io.OutputStream
- private var closed = false
+import org.apache.daffodil.runtime1.layers.api.Layer
- private val boundaryMarkBytes = boundaryMark.getBytes(charset)
+final class BuggyLayer extends Layer("buggy", "urn:org.apache.daffodil.layers.buggy") {
- override def close(): Unit = {
- if (!closed) {
- jos.write(boundaryMarkBytes)
- jos.flush()
- jos.close()
- closed = true
- }
+ override def wrapLayerInput(jis: InputStream): InputStream = new BuggyInputStream(jis)
+
+ override def wrapLayerOutput(jos: OutputStream): OutputStream = jos
+}
+
+final class BuggyInputStream(is: InputStream) extends InputStream {
+
+ def read(): Int = {
+ val b = is.read()
+ if (b != '0') b else throw new IOException("bad input stream")
}
}
diff --git a/daffodil-cli/src/test/scala/org/apache/daffodil/layers/BuggyTransformer.scala b/daffodil-cli/src/test/scala/org/apache/daffodil/layers/BuggyTransformer.scala
deleted file mode 100644
index d1b8d8231..000000000
--- a/daffodil-cli/src/test/scala/org/apache/daffodil/layers/BuggyTransformer.scala
+++ /dev/null
@@ -1,73 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.daffodil.layers
-
-import java.io.InputStream
-
-import org.apache.daffodil.runtime1.layers.LayerCompileInfo
-import org.apache.daffodil.runtime1.layers.LayerCompiler
-import org.apache.daffodil.runtime1.layers.LayerRuntimeInfo
-import org.apache.daffodil.runtime1.layers.LayerTransformer
-import org.apache.daffodil.runtime1.layers.LayerTransformerFactory
-import org.apache.daffodil.runtime1.processors.ParseOrUnparseState
-
-final class BuggyLayerCompiler extends LayerCompiler("buggy") {
-
- override def compileLayer(layerCompileInfo: LayerCompileInfo): BuggyTransformerFactory = {
- new BuggyTransformerFactory(name)
- }
-}
-
-final class BuggyTransformerFactory(name: String) extends LayerTransformerFactory(name) {
-
- override def newInstance(layerRuntimeInfo: LayerRuntimeInfo) = {
- new BuggyTransformer(name, layerRuntimeInfo)
- }
-}
-
-class BuggyTransformer(name: String, layerRuntimeInfo: LayerRuntimeInfo)
- extends LayerTransformer(name, layerRuntimeInfo) {
-
- override def wrapLayerDecoder(jis: InputStream) = {
- new BuggyInputStream(jis)
- }
-
- override def wrapLimitingStream(state: ParseOrUnparseState, jis: InputStream) = {
- jis
- }
-
- override protected def wrapLayerEncoder(jos: java.io.OutputStream): java.io.OutputStream = {
- jos
- }
-
- override protected def wrapLimitingStream(
- state: ParseOrUnparseState,
- jis: java.io.OutputStream,
- ) = {
- jis
- }
-}
-
-final class BuggyInputStream(is: InputStream) extends InputStream {
-
- def read(): Int = {
- val b = is.read()
- if (b != '0') b else throw new java.io.IOException("bad input stream")
- }
-
-}
diff --git a/daffodil-core/src/main/scala/org/apache/daffodil/core/compiler/Compiler.scala b/daffodil-core/src/main/scala/org/apache/daffodil/core/compiler/Compiler.scala
index 16365d68e..3c05ae0cf 100644
--- a/daffodil-core/src/main/scala/org/apache/daffodil/core/compiler/Compiler.scala
+++ b/daffodil-core/src/main/scala/org/apache/daffodil/core/compiler/Compiler.scala
@@ -273,6 +273,9 @@ class Compiler private (
val dpObj = objInput.readObject()
objInput.close()
val dp = dpObj.asInstanceOf[DataProcessor]
+ // must recompile the layers since their data structure in memory
+ // is not serializable
+ dp.ssrd.compileLayers()
dp
} catch {
case _: ZipException =>
diff --git a/daffodil-core/src/main/scala/org/apache/daffodil/core/dpath/Expression.scala b/daffodil-core/src/main/scala/org/apache/daffodil/core/dpath/Expression.scala
index 4bfbf177e..a8034a510 100644
--- a/daffodil-core/src/main/scala/org/apache/daffodil/core/dpath/Expression.scala
+++ b/daffodil-core/src/main/scala/org/apache/daffodil/core/dpath/Expression.scala
@@ -215,7 +215,7 @@ abstract class Expression extends OOLAGHostImpl() with BasicComponent {
*/
def inherentType: NodeInfo.Kind
- def resolveRef(qnameString: String) = {
+ def resolveRef(qnameString: String): RefQName = {
QName
.resolveRef(qnameString, namespaces, tunable.unqualifiedPathStepPolicy)
.recover { case _: Throwable =>
@@ -1363,10 +1363,9 @@ case class VariableRef(val qnameString: String) extends PrimaryExpression(Nil) {
}
override def text = "$" + qnameString
- lazy val theQName: GlobalQName = {
+ lazy val theQName: RefQName = {
val refQ = resolveRef(qnameString)
- val global = QName.createGlobal(refQ.local, refQ.namespace, namespaces)
- global
+ refQ
}
lazy val vrd = compileInfo.variableMap
diff --git a/daffodil-core/src/main/scala/org/apache/daffodil/core/dsom/DFDLFormatAnnotation.scala b/daffodil-core/src/main/scala/org/apache/daffodil/core/dsom/DFDLFormatAnnotation.scala
index 07a198395..183683771 100644
--- a/daffodil-core/src/main/scala/org/apache/daffodil/core/dsom/DFDLFormatAnnotation.scala
+++ b/daffodil-core/src/main/scala/org/apache/daffodil/core/dsom/DFDLFormatAnnotation.scala
@@ -32,8 +32,8 @@ case class DeprecatedProperty(namespace: NS, property: String, replacement: Stri
object DeprecatedProperty {
- private val deprecatedProperties: Seq[DeprecatedProperty] = Seq(
- DeprecatedProperty(XMLUtils.DFDL_NAMESPACE, "layerTransform", "dfdlx:layerTransform"),
+ private def deprecatedProperties: Seq[DeprecatedProperty] = Seq(
+ DeprecatedProperty(XMLUtils.DFDL_NAMESPACE, "layerTransform", "dfdlx:layer"),
DeprecatedProperty(XMLUtils.DFDL_NAMESPACE, "layerEncoding", "dfdlx:layerEncoding"),
DeprecatedProperty(XMLUtils.DFDL_NAMESPACE, "layerLengthKind", "dfdlx:layerLengthKind"),
DeprecatedProperty(XMLUtils.DFDL_NAMESPACE, "layerLength", "dfdlx:layerLength"),
@@ -52,11 +52,15 @@ object DeprecatedProperty {
DeprecatedProperty(XMLUtils.EXT_NS_NCSA, "parseUnparsePolicy", "dfdlx:parseUnparsePolicy"),
)
+ // introduced to eliminate linear search per property of the above list, which is expected
+ // to fail every time, so every property lookup scans the whole thing.
+ private lazy val deprecatedPropertyMap = deprecatedProperties.map { dp =>
+ ((dp.property, dp.namespace), dp)
+ }.toMap
+
def warnIfDeprecated(propertyName: String, propertyNS: NS, context: SchemaComponent): Unit = {
- val deprecation = deprecatedProperties.find { dp =>
- dp.property == propertyName && dp.namespace == propertyNS
- }
+ val deprecation = deprecatedPropertyMap.get((propertyName, propertyNS))
if (deprecation.isDefined) {
val warnID = propertyNS match {
diff --git a/daffodil-core/src/main/scala/org/apache/daffodil/core/dsom/ModelGroup.scala b/daffodil-core/src/main/scala/org/apache/daffodil/core/dsom/ModelGroup.scala
index 729ce1388..1637f14d2 100644
--- a/daffodil-core/src/main/scala/org/apache/daffodil/core/dsom/ModelGroup.scala
+++ b/daffodil-core/src/main/scala/org/apache/daffodil/core/dsom/ModelGroup.scala
@@ -18,8 +18,11 @@
package org.apache.daffodil.core.dsom
import java.lang.{ Integer => JInt }
+import scala.xml.Comment
+import scala.xml.Elem
import scala.xml.Node
import scala.xml.NodeSeq.seqToNodeSeq
+import scala.xml.Text
import org.apache.daffodil.core.dsom.walker.ModelGroupView
import org.apache.daffodil.core.grammar.ModelGroupGrammarMixin
@@ -33,6 +36,9 @@ import org.apache.daffodil.lib.schema.annotation.props.gen.YesNo
*
* Takes care of detecting group references, and constructing the
* proper SequenceGroupRef or ChoiceGroupRef object.
+ *
+ * If you want to do source xml to source xml rewriting transformations
+ * (like macro expansions) do them here or in TermFactory
*/
object ModelGroupFactory {
@@ -65,60 +71,144 @@ object ModelGroupFactory {
}
private def makeModelGroup(
- child: Node,
+ xmlNode: Node,
lexicalParent: SchemaComponent,
position: JInt,
isHidden: Boolean,
nodesAlreadyTrying: Set[Node],
): ModelGroup = {
- val childModelGroup: ModelGroup = child match {
- case <sequence>{_*}</sequence> => {
- // throwaway just so we can grab local properties to check for hidden, without
- // special-case code
- val seq = LocalSequence(child, lexicalParent, position)
- if (seq.hiddenGroupRefOption.isDefined) {
- // explicit check here, because we're about to discard this, and construct
- // a SequenceGroupRef or ChoiceGroupRef, with isHidden = true. So this
- // temporary sequence will be discarded after this.
- seq.checkHiddenGroupRefHasNoChildren
- //
- // construct the group ref XML, then recursively process that,
- // but set flag so it will be hidden.
- //
- val hgrXML = seq.hiddenGroupRefXML
- ModelGroupFactory(
- hgrXML,
- lexicalParent,
- position,
- isHidden = true,
- nodesAlreadyTrying,
- )
- } else {
- LocalSequence(child, lexicalParent, position)
- }
+ val childModelGroup: ModelGroup = xmlNode match {
+
+ case seqXML: Elem if seqXML.label == "sequence" => {
+ // seq is a throwaway just so we can grab local properties to check for things
+ // that require rewriting without special-case code
+ val seq = LocalSequence(xmlNode, lexicalParent, position)
+
+ //
+ // Check sequences for rewriting situations
+ //
+ if (seq.hiddenGroupRefOption.isDefined)
+ rewriteHiddenGroup(seq, lexicalParent, position, isHidden, nodesAlreadyTrying)
+ else if (
+ seq.isLayered &&
+ (seq.termChildren.length > 1 || seq.termChildren.head.isArray)
+ )
+ rewriteLayeredSequence(seq, lexicalParent, position, isHidden, nodesAlreadyTrying)
+ else
+ LocalSequence(xmlNode, lexicalParent, position) // No rewriting the sequence.
}
- case <choice>{_*}</choice> => Choice(child, lexicalParent, position)
+ case <choice>{_*}</choice> => Choice(xmlNode, lexicalParent, position)
case <group>{_*}</group> => {
val pos: Int = lexicalParent match {
case ct: ComplexTypeBase => 1
case mg: ModelGroup => position
case gd: GlobalGroupDef => position
}
- val groupRef = GroupRefFactory(child, lexicalParent, pos, isHidden)
+ val groupRef = GroupRefFactory(xmlNode, lexicalParent, pos, isHidden)
groupRef.asModelGroup
}
case _ => {
+ // $COVERAGE-OFF$
Assert.invariantFailed(
- "Unrecognized construct %s should be handled by caller.".format(child),
+ "Unrecognized construct %s should be handled by caller.".format(xmlNode),
)
+ // $COVERAGE_ON$
}
}
childModelGroup
}
+
+ /**
+ * Rewrite of XML for layered sequence with multiple children or an array child.
+ * This rewrites the schema so that the layered sequence has one non-array child only.
+ *
+ * @param seq The LocalSequence to be rewritten.
+ * @param lexicalParent The SchemaComponent representing the lexical parent.
+ * @param position The position of the layered sequence within its enclosing model group
+ * @param isHidden Indicates whether the sequence is hidden or not.
+ * @param nodesAlreadyTrying A set of nodes that have already been tried.
+ * @return The rewritten ModelGroup.
+ */
+ private def rewriteLayeredSequence(
+ seq: LocalSequence,
+ lexicalParent: SchemaComponent,
+ position: JInt,
+ isHidden: Boolean,
+ nodesAlreadyTrying: Set[Node],
+ ): ModelGroup = {
+ val realChildren: Seq[Node] = seq.apparentXMLChildren.filterNot {
+ // strip away all the trash nodes
+ case _: Text => true // whitespace nodes
+ case _: Comment => true
+ case _ => false
+ }
+ val newXML = realChildren match {
+ // If an annotation is first, that stays on the outer sequence
+ case Seq(annotation @ <annotation>{_*}</annotation>, terms @ _*) =>
+ <sequence dfdlx:layer={seq.xml.attribute("layer")}>
+ {annotation}
+ <sequence>
+ {terms}
+ </sequence>
+ </sequence>.copy(scope = seq.xml.scope) % seq.xml.attributes
+ case terms =>
+ // there is no annotation, so all children just go inside the inner sequence.
+ <sequence dfdlx:layer={seq.xml.attribute("layer")}>
+ <sequence>
+ {terms}
+ </sequence>
+ </sequence>.copy(scope = seq.xml.scope) % seq.xml.attributes
+ }
+ makeModelGroup(
+ newXML,
+ lexicalParent,
+ position,
+ isHidden,
+ nodesAlreadyTrying,
+ )
+ }
+
+ /**
+ * Rewrites a sequence with dfdl:hiddenGroupRef into a group reference + the isHidden flag.
+ *
+ * @param seq The local sequence containing the hidden group.
+ * @param lexicalParent The parent schema component of the hidden group.
+ * @param position The position of the hidden group within the sequence.
+ * @param isHidden Flag indicating whether the hidden group should be hidden.
+ * @param nodesAlreadyTrying Set of nodes that have already been processed.
+ * @return The processed hidden group.
+ */
+ private def rewriteHiddenGroup(
+ seq: LocalSequence,
+ lexicalParent: SchemaComponent,
+ position: JInt,
+ isHidden: Boolean,
+ nodesAlreadyTrying: Set[Node],
+ ) = {
+ // explicit check here, because we're about to discard this, and construct
+ // a SequenceGroupRef or ChoiceGroupRef, with isHidden = true. So this
+ // temporary sequence will be discarded after this.
+ seq.checkHiddenGroupRefHasNoChildren
+ //
+ // construct the group ref XML, then recursively process that,
+ // but set flag so it will be hidden.
+ //
+ val hgrXML = seq.hiddenGroupRefXML
+ ModelGroupFactory(
+ hgrXML,
+ lexicalParent,
+ position,
+ isHidden = true,
+ nodesAlreadyTrying,
+ )
+ }
}
/**
* Factory for Terms
+ *
+ * If you want to do source XML to source XML rewrite transformations, do them
+ * here or in ModelGroupFactory.
*/
object TermFactory {
diff --git a/daffodil-core/src/main/scala/org/apache/daffodil/core/dsom/RawCommonRuntimeValuedPropertiesMixin.scala b/daffodil-core/src/main/scala/org/apache/daffodil/core/dsom/RawCommonRuntimeValuedPropertiesMixin.scala
index 265208ad8..7a1d48681 100644
--- a/daffodil-core/src/main/scala/org/apache/daffodil/core/dsom/RawCommonRuntimeValuedPropertiesMixin.scala
+++ b/daffodil-core/src/main/scala/org/apache/daffodil/core/dsom/RawCommonRuntimeValuedPropertiesMixin.scala
@@ -67,19 +67,6 @@ trait RawSequenceRuntimeValuedPropertiesMixin extends RawDelimitedRuntimeValuedP
protected final lazy val separatorRaw = requireProperty(optionSeparatorRaw)
}
-trait RawLayeringRuntimeValuedPropertiesMixin extends PropertyMixin {
- protected final lazy val optionLayerEncodingRaw =
- findPropertyOption("layerEncoding", expressionAllowed = true)
- protected final lazy val layerEncodingRaw = requireProperty(optionLayerEncodingRaw)
- protected final lazy val optionLayerLengthRaw =
- findPropertyOption("layerLength", expressionAllowed = true)
- protected final lazy val layerLengthRaw = requireProperty(optionLayerLengthRaw)
- protected final lazy val optionLayerBoundaryMarkRaw =
- findPropertyOption("layerBoundaryMark", expressionAllowed = true)
- protected final lazy val layerBoundaryMarkRaw = requireProperty(optionLayerBoundaryMarkRaw)
-
-}
-
trait RawEscapeSchemeRuntimeValuedPropertiesMixin extends PropertyMixin {
// package private because used in unit test
final lazy val optionEscapeCharacterRaw =
diff --git a/daffodil-core/src/main/scala/org/apache/daffodil/core/dsom/RuntimePropertyMixins.scala b/daffodil-core/src/main/scala/org/apache/daffodil/core/dsom/RuntimePropertyMixins.scala
index e04bca9c5..0869c04f3 100644
--- a/daffodil-core/src/main/scala/org/apache/daffodil/core/dsom/RuntimePropertyMixins.scala
+++ b/daffodil-core/src/main/scala/org/apache/daffodil/core/dsom/RuntimePropertyMixins.scala
@@ -51,10 +51,6 @@ import org.apache.daffodil.runtime1.processors.FillByteEv
import org.apache.daffodil.runtime1.processors.ImplicitLengthEv
import org.apache.daffodil.runtime1.processors.InitiatorParseEv
import org.apache.daffodil.runtime1.processors.InitiatorUnparseEv
-import org.apache.daffodil.runtime1.processors.LayerBoundaryMarkEv
-import org.apache.daffodil.runtime1.processors.LayerCharsetEv
-import org.apache.daffodil.runtime1.processors.LayerEncodingEv
-import org.apache.daffodil.runtime1.processors.LayerLengthEv
import org.apache.daffodil.runtime1.processors.LengthEv
import org.apache.daffodil.runtime1.processors.LengthInBitsEv
import org.apache.daffodil.runtime1.processors.MinLengthInBitsEv
@@ -335,8 +331,8 @@ trait ElementRuntimeValuedPropertiesMixin
private lazy val implicitLengthEv: LengthEv = {
Assert.usage(repElement.lengthKind eq LengthKind.Implicit)
- import Representation._
import NodeInfo._
+ import Representation._
lazy val maxLengthLong = repElement.maxLength.longValueExact
val ev = (repElement.impliedRepresentation, repElement.typeDef.typeNode) match {
case (Text, String) => new ImplicitLengthEv(maxLengthLong, eci)
@@ -393,8 +389,8 @@ trait ElementRuntimeValuedPropertiesMixin
(repElement.lengthKind eq LengthKind.Implicit) || (repElement.lengthKind eq LengthKind.Explicit),
)
import LengthKind._
- import Representation._
import NodeInfo._
+ import Representation._
val (units: LengthUnits, lenEv: LengthEv) =
(
repElement.lengthKind,
@@ -438,8 +434,8 @@ trait ElementRuntimeValuedPropertiesMixin
protected lazy val (minLenUnits: LengthUnits, minLen: Long) = {
import LengthKind._
- import Representation._
import NodeInfo._
+ import Representation._
lazy val maxLengthLong = maxLength.longValueExact
lazy val minLengthLong = minLength.longValueExact
val res: (LengthUnits, Long) =
@@ -823,88 +819,6 @@ trait SequenceRuntimeValuedPropertiesMixin
}
-trait LayeringRuntimeValuedPropertiesMixin extends RawLayeringRuntimeValuedPropertiesMixin {
- decl: SequenceTermBase =>
-
- private lazy val layerEncodingExpr = {
- val qn = this.qNameForProperty("layerEncoding")
- ExpressionCompilers.String.compileProperty(
- qn,
- NodeInfo.NonEmptyString,
- layerEncodingRaw,
- decl,
- dpathCompileInfo,
- )
- }
-
- private final lazy val layerEncodingEv = {
- if (maybeLayerEncodingEv.isEmpty) layerEncodingRaw // must be defined
- maybeLayerEncodingEv.get
- }
-
- private final lazy val maybeLayerEncodingEv = {
- if (optionLayerEncodingRaw.isDefined) {
- val ev = new LayerEncodingEv(layerEncodingExpr, tci)
- ev.compile(tunable)
- One(ev)
- } else {
- Nope
- }
- }
-
- final lazy val maybeLayerCharsetEv =
- if (optionLayerEncodingRaw.isDefined) {
- val ev = new LayerCharsetEv(layerEncodingEv, tci)
- ev.compile(tunable)
- One(ev)
- } else
- Nope
-
- private lazy val layerLengthExpr = {
- val qn = this.qNameForProperty("layerLength")
- ExpressionCompilers.JLong.compileProperty(
- qn,
- NodeInfo.Long,
- layerLengthRaw,
- decl,
- dpathCompileInfo,
- )
- }
-
- final lazy val maybeLayerLengthEv = {
- if (optionLayerLengthRaw.isDefined) {
- layerLengthUnits
- val ev = new LayerLengthEv(layerLengthExpr, tci)
- ev.compile(tunable)
- One(ev)
- } else {
- Nope
- }
- }
-
- private lazy val layerBoundaryMarkExpr = {
- val qn = this.qNameForProperty("layerBoundaryMark")
- ExpressionCompilers.String.compileProperty(
- qn,
- NodeInfo.String,
- layerBoundaryMarkRaw,
- decl,
- dpathCompileInfo,
- )
- }
-
- final lazy val maybeLayerBoundaryMarkEv = {
- if (optionLayerBoundaryMarkRaw.isDefined) {
- val ev = new LayerBoundaryMarkEv(layerBoundaryMarkExpr, tci)
- ev.compile(tunable)
- One(ev)
- } else {
- Nope
- }
- }
-
-}
-
trait SimpleTypeRuntimeValuedPropertiesMixin
extends DFDLSimpleTypeMixin
with RawSimpleTypeRuntimeValuedPropertiesMixin
diff --git a/daffodil-core/src/main/scala/org/apache/daffodil/core/dsom/SchemaSet.scala b/daffodil-core/src/main/scala/org/apache/daffodil/core/dsom/SchemaSet.scala
index 696f0268d..3e2d03d3e 100644
--- a/daffodil-core/src/main/scala/org/apache/daffodil/core/dsom/SchemaSet.scala
+++ b/daffodil-core/src/main/scala/org/apache/daffodil/core/dsom/SchemaSet.scala
@@ -685,7 +685,8 @@ final class SchemaSet private (
startingGlobalComponents,
)
- lazy val allSchemaComponents = allSchemaComponentsSet.toIndexedSeq
+ lazy val allSchemaComponents: Seq[SchemaComponent] = allSchemaComponentsSet.toIndexedSeq
+
}
object TransitiveClosureSchemaComponents {
diff --git a/daffodil-core/src/main/scala/org/apache/daffodil/core/dsom/SequenceGroup.scala b/daffodil-core/src/main/scala/org/apache/daffodil/core/dsom/SequenceGroup.scala
index 32281a0b3..af1f3bbfe 100644
--- a/daffodil-core/src/main/scala/org/apache/daffodil/core/dsom/SequenceGroup.scala
+++ b/daffodil-core/src/main/scala/org/apache/daffodil/core/dsom/SequenceGroup.scala
@@ -30,14 +30,12 @@ import org.apache.daffodil.lib.schema.annotation.props.Found
import org.apache.daffodil.lib.schema.annotation.props.PropertyLookupResult
import org.apache.daffodil.lib.schema.annotation.props.SeparatorSuppressionPolicy
import org.apache.daffodil.lib.schema.annotation.props.SeparatorSuppressionPolicyMixin
-import org.apache.daffodil.lib.schema.annotation.props.gen.LayerLengthUnits
import org.apache.daffodil.lib.schema.annotation.props.gen.OccursCountKind
import org.apache.daffodil.lib.schema.annotation.props.gen.SeparatorPosition
import org.apache.daffodil.lib.schema.annotation.props.gen.SequenceKind
-import org.apache.daffodil.lib.schema.annotation.props.gen.Sequence_AnnotationMixin
+import org.apache.daffodil.lib.xml.RefQName
import org.apache.daffodil.lib.xml.XMLUtils
-import org.apache.daffodil.runtime1.layers.LayerCompiler
-import org.apache.daffodil.runtime1.layers.LayerCompilerRegistry
+import org.apache.daffodil.runtime1.layers.LayerRuntimeData
import org.apache.daffodil.runtime1.processors.SeparatorParseEv
import org.apache.daffodil.runtime1.processors.SeparatorUnparseEv
@@ -60,13 +58,11 @@ abstract class SequenceTermBase(
def separatorPosition: SeparatorPosition
- def isLayered: Boolean
-
def separatorParseEv: SeparatorParseEv
def separatorUnparseEv: SeparatorUnparseEv
- def layerLengthUnits: LayerLengthUnits
+ def layerOption: Option[RefQName]
def isOrdered: Boolean
@@ -75,6 +71,23 @@ abstract class SequenceTermBase(
*/
def isHidden: Boolean = false
+ def isLayered: Boolean = layerOption.isDefined
+
+ lazy val optionLayerRuntimeData: Option[LayerRuntimeData] =
+ layerOption.map { layerProperty: RefQName =>
+ val layerVars = variableMap.vrds.filter { vrd =>
+ vrd.globalQName.namespace == layerProperty.namespace
+ }
+ val qNameToVRD = layerVars.map { vrd => (vrd.globalQName.local, vrd) }.toMap
+ val sfl = schemaFileLocation
+ new LayerRuntimeData(
+ layerProperty,
+ schemaFileLocation = sfl,
+ qNameToVRD,
+ this.sequenceRuntimeData,
+ )
+ }
+
}
/**
@@ -84,10 +97,8 @@ abstract class SequenceTermBase(
*/
abstract class SequenceGroupTermBase(xml: Node, lexicalParent: SchemaComponent, position: Int)
extends SequenceTermBase(xml, Option(lexicalParent), position)
- with Sequence_AnnotationMixin
with SequenceRuntimeValuedPropertiesMixin
- with SeparatorSuppressionPolicyMixin
- with LayeringRuntimeValuedPropertiesMixin {
+ with SeparatorSuppressionPolicyMixin {
requiredEvaluationsIfActivated(checkIfValidUnorderedSequence)
requiredEvaluationsIfActivated(checkIfNonEmptyAndDiscrimsOrAsserts)
@@ -115,19 +126,19 @@ abstract class SequenceGroupTermBase(xml: Node, lexicalParent: SchemaComponent,
* we know a separator MUST appear in the data stream.
*/
private lazy val infixAndTwoOrMoreStaticallyRequiredInstances =
- hasInfixSep && (representedMembers.filter { m =>
+ hasInfixSep && (representedMembers.count { m =>
val res = m.hasStaticallyRequiredOccurrencesInDataRepresentation
res
- }.length >= 2)
+ } >= 2)
private lazy val sepAndArrayaWithTwoOrMoreStaticallyRequiredInstances =
- hasSeparator && (representedMembers.filter { m =>
+ hasSeparator && representedMembers.exists { m =>
val res = m.hasStaticallyRequiredOccurrencesInDataRepresentation && (m match {
case e: ElementBase => e.minOccurs >= 2
case _ => false
})
res
- }.length > 0)
+ }
final override lazy val hasKnownRequiredSyntax = LV('hasKnownRequiredSyntax) {
if (hasFraming) true
@@ -153,9 +164,9 @@ abstract class SequenceGroupTermBase(xml: Node, lexicalParent: SchemaComponent,
protected final lazy val checkIfNonEmptyAndDiscrimsOrAsserts: Unit = {
val msg = "Counterintuitive placement detected. Wrap the discriminator or assert " +
"in an empty sequence to evaluate before the contents."
- if (groupMembers.size > 0 && discriminatorStatements.size > 0)
+ if (groupMembers.nonEmpty && discriminatorStatements.nonEmpty)
SDW(WarnID.DiscouragedDiscriminatorPlacement, msg)
- if (groupMembers.size > 0 && assertStatements.size > 0)
+ if (groupMembers.nonEmpty && assertStatements.nonEmpty)
SDW(WarnID.DiscouragedAssertPlacement, msg)
}
@@ -221,36 +232,6 @@ abstract class SequenceGroupTermBase(xml: Node, lexicalParent: SchemaComponent,
case SequenceKind.Unordered => false
}
- private val layeredSequenceAllowedProps = Set(
- "ref",
- "layerTransform",
- "layerEncoding",
- "layerLengthKind",
- "layerLength",
- "layerLengthUnits",
- "layerBoundaryMark",
- )
-
- private lazy val optionLayerTransform = findPropertyOption("layerTransform").toOption
-
- final lazy val layerCompiler: LayerCompiler = {
-
- // need to check that only layering properties are specified
- val localProps = this.formatAnnotation.justThisOneProperties
- val localKeys = localProps.keySet
- val disallowedKeys = localKeys.filterNot(k => layeredSequenceAllowedProps.contains(k))
- if (disallowedKeys.size > 0)
- SDE(
- "Sequence has dfdlx:layerTransform specified, so cannot have non-layering properties: %s",
- disallowedKeys.mkString(", "),
- )
-
- optionLayerTransform.map { xformName =>
- LayerCompilerRegistry.find(xformName, this)
- }.get
- }
-
- final def isLayered = optionLayerTransform.isDefined
}
/**
@@ -266,7 +247,8 @@ trait SequenceDefMixin
def groupMembersNotShared: Seq[Term]
- protected final def isMyFormatAnnotation(a: DFDLAnnotation) = a.isInstanceOf[DFDLSequence]
+ protected final def isMyFormatAnnotation(a: DFDLAnnotation): Boolean =
+ a.isInstanceOf[DFDLSequence]
protected final def annotationFactory(node: Node): Option[DFDLAnnotation] = {
node match {
@@ -277,9 +259,9 @@ trait SequenceDefMixin
protected def emptyFormatFactory = new DFDLSequence(newDFDLAnnotationXML("sequence"), this)
- final lazy val <sequence>{apparentXMLChildren @ _*}</sequence> = (xml \\ "sequence")(0)
+ final lazy val apparentXMLChildren = (xml \\ "sequence").head.child
- def xmlChildren = apparentXMLChildren
+ def xmlChildren: Seq[Node] = apparentXMLChildren
// The dfdl:hiddenGroupRef property cannot be scoped, nor defaulted. It's really a special
// attribute, not a format property in the usual sense.
@@ -291,7 +273,7 @@ trait SequenceDefMixin
override lazy val optReferredToComponent = None
- lazy val checkHiddenGroupRefHasNoChildren = {
+ lazy val checkHiddenGroupRefHasNoChildren: Unit = {
if (hiddenGroupRefOption.isDefined) {
val axc = apparentXMLChildren
val len = axc.length
@@ -312,7 +294,7 @@ trait SequenceDefMixin
}
object LocalSequence {
- def apply(xmlArg: Node, lexicalParent: SchemaComponent, position: Int) = {
+ def apply(xmlArg: Node, lexicalParent: SchemaComponent, position: Int): LocalSequence = {
val ls = new LocalSequence(xmlArg, lexicalParent, position)
ls.initialize()
ls
@@ -328,7 +310,7 @@ final class LocalSequence private (xmlArg: Node, lexicalParent: SchemaComponent,
with SequenceView
object ChoiceBranchImpliedSequence {
- def apply(rawGM: Term) = {
+ def apply(rawGM: Term): ChoiceBranchImpliedSequence = {
val cbis = new ChoiceBranchImpliedSequence(rawGM)
cbis.initialize()
cbis
@@ -350,7 +332,7 @@ final class ChoiceBranchImpliedSequence private (rawGM: Term)
with SequenceDefMixin
with ChoiceBranchImpliedSequenceRuntime1Mixin {
- override final protected lazy val groupMembersDef: Seq[Term] = Seq(rawGM)
+ override protected lazy val groupMembersDef: Seq[Term] = Seq(rawGM)
override def separatorSuppressionPolicy: SeparatorSuppressionPolicy =
SeparatorSuppressionPolicy.TrailingEmptyStrict
@@ -360,6 +342,7 @@ final class ChoiceBranchImpliedSequence private (rawGM: Term)
override def separatorPosition: SeparatorPosition = SeparatorPosition.Infix
override def isLayered: Boolean = false
+ override def layerOption: Option[RefQName] = None
override lazy val hasSeparator = false
override lazy val hasTerminator = false
@@ -371,9 +354,6 @@ final class ChoiceBranchImpliedSequence private (rawGM: Term)
override def separatorUnparseEv: SeparatorUnparseEv =
Assert.usageError("Not to be called on choice branches.")
- override def layerLengthUnits: LayerLengthUnits =
- Assert.usageError("Not to be called for choice branches.")
-
override def isOrdered = true
override def findPropertyOption(
@@ -387,9 +367,9 @@ final class ChoiceBranchImpliedSequence private (rawGM: Term)
*/
override lazy val nonDefaultPropertySources: Seq[ChainPropProvider] = Seq()
- final override def xmlChildren: Seq[scala.xml.Node] = Seq(xml)
+ override def xmlChildren: Seq[scala.xml.Node] = Seq(xml)
// Members declared in Term
- def hasKnownRequiredSyntax: Boolean = groupMembers(0).hasKnownRequiredSyntax
+ def hasKnownRequiredSyntax: Boolean = groupMembers.head.hasKnownRequiredSyntax
}
diff --git a/daffodil-core/src/main/scala/org/apache/daffodil/core/grammar/SequenceGrammarMixin.scala b/daffodil-core/src/main/scala/org/apache/daffodil/core/grammar/SequenceGrammarMixin.scala
index 5d8e25f70..319201100 100644
--- a/daffodil-core/src/main/scala/org/apache/daffodil/core/grammar/SequenceGrammarMixin.scala
+++ b/daffodil-core/src/main/scala/org/apache/daffodil/core/grammar/SequenceGrammarMixin.scala
@@ -42,20 +42,14 @@ trait SequenceGrammarMixin extends GrammarMixin with SequenceTermRuntime1Mixin {
}
private lazy val layerContent = {
- schemaDefinitionUnless(
- groupMembers.length == 1,
- "Layered sequence can have only 1 child term. %s were found: %s",
- groupMembers.length,
- groupMembers.mkString(", "),
- )
- val term = groupMembers(0)
- schemaDefinitionWhen(term.isArray, "Layered sequence body cannot be an array.")
this match {
- case sgtb: SequenceGroupTermBase =>
+ case sgtb: SequenceGroupTermBase => {
+ val term = groupMembers.head
LayeredSequence(
sgtb,
new ScalarOrderedSequenceChild(this, term, 1),
) // We use 1-based indexing for children.
+ }
// $COVERAGE-OFF$
case _ =>
Assert.invariantFailed(
diff --git a/daffodil-core/src/main/scala/org/apache/daffodil/core/grammar/primitives/LayeredSequence.scala b/daffodil-core/src/main/scala/org/apache/daffodil/core/grammar/primitives/LayeredSequence.scala
index 44b038ad8..4bd824376 100644
--- a/daffodil-core/src/main/scala/org/apache/daffodil/core/grammar/primitives/LayeredSequence.scala
+++ b/daffodil-core/src/main/scala/org/apache/daffodil/core/grammar/primitives/LayeredSequence.scala
@@ -19,10 +19,8 @@ package org.apache.daffodil.core.grammar.primitives
import org.apache.daffodil.core.dsom._
import org.apache.daffodil.core.grammar.Terminal
-import org.apache.daffodil.lib.util.Maybe
+import org.apache.daffodil.core.layers.LayerSchemaCompiler
import org.apache.daffodil.lib.util.Misc
-import org.apache.daffodil.runtime1.layers.LayerCompileInfo
-import org.apache.daffodil.runtime1.layers.LayerRuntimeInfo
import org.apache.daffodil.runtime1.processors.parsers.LayeredSequenceParser
import org.apache.daffodil.runtime1.processors.parsers.{ Parser => DaffodilParser }
import org.apache.daffodil.runtime1.processors.unparsers.{ Unparser => DaffodilUnparser }
@@ -34,41 +32,19 @@ case class LayeredSequence(sq: SequenceGroupTermBase, bodyTerm: SequenceChild)
private val srd = sq.sequenceRuntimeData
private val trd = bodyTerm.termRuntimeData
- val layerCompileInfo = new LayerCompileInfo(
- sq,
- new LayerRuntimeInfo(
- sq.sequenceRuntimeData,
- sq.maybeLayerCharsetEv,
- Maybe.toMaybe(sq.optionLayerLengthKind),
- sq.maybeLayerLengthEv,
- Maybe.toMaybe(sq.optionLayerLengthUnits),
- sq.maybeLayerBoundaryMarkEv,
- ),
- )
+ LayerSchemaCompiler.compile(sq) // static checking
- private val layerTransformerFactory = {
- val layerCompiler = sq.layerCompiler
- //
- // this layer compilation step allows each layer to check
- // the layering properties at schema-compilation time.
- //
- layerCompiler.compileLayer(layerCompileInfo)
- }
-
- override def toString() =
+ override def toString: String =
"<" + Misc.getNameFromClass(this) + ">" +
bodyTerm.toString() +
"</" + Misc.getNameFromClass(this) + ">"
- lazy val bodyParser = bodyTerm.parser
- lazy val bodyUnparser = bodyTerm.unparser
-
- lazy val layerRuntimeInfo = layerCompileInfo.layerRuntimeInfo
+ private lazy val bodyParser = bodyTerm.parser
+ private lazy val bodyUnparser = bodyTerm.unparser
override lazy val parser: DaffodilParser =
- new LayeredSequenceParser(srd, layerTransformerFactory, layerRuntimeInfo, bodyParser)
+ new LayeredSequenceParser(srd, bodyParser)
- override lazy val unparser: DaffodilUnparser = {
- new LayeredSequenceUnparser(srd, layerTransformerFactory, layerRuntimeInfo, bodyUnparser)
- }
+ override lazy val unparser: DaffodilUnparser =
+ new LayeredSequenceUnparser(srd, bodyUnparser)
}
diff --git a/daffodil-core/src/main/scala/org/apache/daffodil/core/layers/LayerSchemaCompiler.scala b/daffodil-core/src/main/scala/org/apache/daffodil/core/layers/LayerSchemaCompiler.scala
new file mode 100644
index 000000000..42aecac1f
--- /dev/null
+++ b/daffodil-core/src/main/scala/org/apache/daffodil/core/layers/LayerSchemaCompiler.scala
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.daffodil.core.layers
+
+import org.apache.daffodil.core.dsom.SequenceGroupTermBase
+import org.apache.daffodil.lib.exceptions.Assert
+
+object LayerSchemaCompiler {
+
+ /**
+ * Compiles the layer.
+ *
+ * This is mostly checking for errant use of DFDL properties that
+ * can't be used on layers.
+ *
+ * The real compilation - creating runtime data structures based
+ * on constructor signatures and DFDL variables that are in the
+ * layer's namespace, that is called here, but it is part of
+ * the daffodil runtime because that step has to be carried out
+ * *also* at runtime to verify that the dynamically loaded layer
+ * classes are compatible with the variables and their definitions.
+ *
+ * This compilation produces no output object, as the LayerRuntimeData
+ * already contains everything needed.
+ */
+ def compile(sq: SequenceGroupTermBase): Unit = {
+ val lc = new LayerSchemaCompiler(sq)
+ lc.compile
+ }
+}
+
+private class LayerSchemaCompiler private (sq: SequenceGroupTermBase) {
+
+ Assert.usage(sq.isLayered)
+
+ private def lrd = sq.optionLayerRuntimeData.get
+
+ private val layeredSequenceAllowedProps = Seq("layer") // no other properties are allowed.
+
+ private def checkOnlyAllowedProperties(): Unit = {
+ // need to check that only layering properties are specified
+ val localProps = sq.formatAnnotation.justThisOneProperties
+ val localKeys = localProps.keySet
+ val disallowedKeys = localKeys.filterNot(k => layeredSequenceAllowedProps.contains(k))
+ if (disallowedKeys.nonEmpty)
+ sq.SDE(
+ "Sequence has dfdlx:layer specified, so cannot have non-layering properties: %s",
+ disallowedKeys.mkString(", "),
+ )
+ }
+
+ lazy val compile: Unit = {
+ checkOnlyAllowedProperties()
+ //
+ // Note: does not check layer vars against layer code setters/getters.
+ // That checking is done elsewhere.
+ }
+
+}
diff --git a/daffodil-core/src/main/scala/org/apache/daffodil/core/runtime1/SchemaSetRuntime1Mixin.scala b/daffodil-core/src/main/scala/org/apache/daffodil/core/runtime1/SchemaSetRuntime1Mixin.scala
index 4e9ce52b1..37d4bdfe3 100644
--- a/daffodil-core/src/main/scala/org/apache/daffodil/core/runtime1/SchemaSetRuntime1Mixin.scala
+++ b/daffodil-core/src/main/scala/org/apache/daffodil/core/runtime1/SchemaSetRuntime1Mixin.scala
@@ -18,8 +18,11 @@
package org.apache.daffodil.core.runtime1
import org.apache.daffodil.core.dsom.SchemaSet
+import org.apache.daffodil.core.dsom.SequenceTermBase
import org.apache.daffodil.lib.util.Logger
import org.apache.daffodil.runtime1.api.DFDL
+import org.apache.daffodil.runtime1.layers.LayerRuntimeCompiler
+import org.apache.daffodil.runtime1.layers.LayerRuntimeData
import org.apache.daffodil.runtime1.processors.DataProcessor
import org.apache.daffodil.runtime1.processors.Processor
import org.apache.daffodil.runtime1.processors.SchemaSetRuntimeData
@@ -32,6 +35,7 @@ trait SchemaSetRuntime1Mixin {
requiredEvaluationsAlways(parser)
requiredEvaluationsAlways(unparser)
+ requiredEvaluationsAlways(allLayers)
requiredEvaluationsAlways(root.elementRuntimeData.initialize)
override lazy val variableMap: VariableMap = LV('variableMap) {
@@ -53,6 +57,18 @@ trait SchemaSetRuntime1Mixin {
unp
}.value
+ private lazy val layerRuntimeCompiler = new LayerRuntimeCompiler
+
+ lazy val allLayers: Seq[LayerRuntimeData] = LV('allLayers) {
+ val lrds: Seq[LayerRuntimeData] = self.allSchemaComponents
+ .collect {
+ case stb: SequenceTermBase if (stb.isLayered) => stb
+ }
+ .map { _.optionLayerRuntimeData.get }
+ layerRuntimeCompiler.compileAll(lrds) // for checking only. We're not saving this.
+ lrds
+ }.value
+
def onPath(xpath: String): DFDL.DataProcessor = {
checkNotError()
if (xpath != "/")
@@ -65,7 +81,7 @@ trait SchemaSetRuntime1Mixin {
val p = if (!root.isError) parser else null
val u = if (!root.isError) unparser else null
val ssrd =
- new SchemaSetRuntimeData(p, u, rootERD, variableMap)
+ new SchemaSetRuntimeData(p, u, rootERD, variableMap, allLayers, layerRuntimeCompiler)
if (root.numComponents > root.numUniqueComponents)
Logger.log.debug(
s"Compiler: component counts: unique ${root.numUniqueComponents}, actual ${root.numComponents}.",
diff --git a/daffodil-core/src/main/scala/org/apache/daffodil/core/runtime1/SequenceTermRuntime1Mixin.scala b/daffodil-core/src/main/scala/org/apache/daffodil/core/runtime1/SequenceTermRuntime1Mixin.scala
index fd074134a..d260270c9 100644
--- a/daffodil-core/src/main/scala/org/apache/daffodil/core/runtime1/SequenceTermRuntime1Mixin.scala
+++ b/daffodil-core/src/main/scala/org/apache/daffodil/core/runtime1/SequenceTermRuntime1Mixin.scala
@@ -50,6 +50,7 @@ trait SequenceTermRuntime1Mixin { self: SequenceTermBase =>
maybeCheckByteAndBitOrderEv,
maybeCheckBitOrderAndCharsetEv,
isHidden,
+ Delay('maybeLayerRuntimeData, this, Maybe.toMaybe(optionLayerRuntimeData)),
)
}
@@ -84,6 +85,7 @@ trait ChoiceBranchImpliedSequenceRuntime1Mixin { self: ChoiceBranchImpliedSequen
Maybe.Nope,
Maybe.Nope,
isHidden = false,
+ Delay('maybeLayerRuntimeData, this, Maybe.Nope),
)
}
diff --git a/daffodil-core/src/main/scala/org/apache/daffodil/core/runtime1/VariableRuntime1Mixin.scala b/daffodil-core/src/main/scala/org/apache/daffodil/core/runtime1/VariableRuntime1Mixin.scala
index 6b0250f36..712493916 100644
--- a/daffodil-core/src/main/scala/org/apache/daffodil/core/runtime1/VariableRuntime1Mixin.scala
+++ b/daffodil-core/src/main/scala/org/apache/daffodil/core/runtime1/VariableRuntime1Mixin.scala
@@ -34,6 +34,7 @@ trait DFDLDefineVariableRuntime1Mixin { self: DFDLDefineVariable =>
requiredEvaluationsAlways(variableRuntimeData.initialize)
final lazy val variableRuntimeData = {
+ val index = this.schemaSet.allDefinedVariables.indexOf(this)
val vrd = new VariableRuntimeData(
this.schemaFileLocation,
this.diagnosticDebugName,
@@ -46,7 +47,7 @@ trait DFDLDefineVariableRuntime1Mixin { self: DFDLDefineVariable =>
this.namedQName.asInstanceOf[GlobalQName],
this.primType,
this.tunable.unqualifiedPathStepPolicy,
- this.schemaSet.allDefinedVariables.indexOf(this),
+ vmapIndex = index,
)
vrd
}
@@ -86,6 +87,7 @@ trait DFDLNewVariableInstanceRuntime1Mixin { self: DFDLNewVariableInstance =>
* point to this location instead of the original definition
*/
final override lazy val variableRuntimeData = {
+ val globalVRD = defv.variableRuntimeData
val vrd = new VariableRuntimeData(
this.schemaFileLocation,
this.diagnosticDebugName,
@@ -95,10 +97,13 @@ trait DFDLNewVariableInstanceRuntime1Mixin { self: DFDLNewVariableInstance =>
defv.direction,
Delay('maybeDefaultValueExpr2, this, maybeDefaultValueExpr),
defv.typeQName,
- defv.namedQName.asInstanceOf[GlobalQName],
+ globalVRD.globalQName, // important that this is the exact same globalQName object as is used by the globalVRD.
defv.primType,
this.tunable.unqualifiedPathStepPolicy,
- this.schemaSet.allDefinedVariables.indexOf(defv),
+ // This is a really important invariant. The index of the NVI's VRD
+ // must be identical to that of the global VRD for this variable,
+ // So we take it from there.
+ vmapIndex = globalVRD.vmapIndex,
)
vrd
}
diff --git a/daffodil-core/src/test/scala/org/apache/daffodil/core/api/TestMetadataWalking.scala b/daffodil-core/src/test/scala/org/apache/daffodil/core/api/TestMetadataWalking.scala
index 83fc30088..63764deaa 100644
--- a/daffodil-core/src/test/scala/org/apache/daffodil/core/api/TestMetadataWalking.scala
+++ b/daffodil-core/src/test/scala/org/apache/daffodil/core/api/TestMetadataWalking.scala
@@ -22,6 +22,7 @@ import scala.xml.Elem
import org.apache.daffodil.core.util.TestUtils
import org.apache.daffodil.io.InputSourceDataInputStream
+import org.apache.daffodil.lib.Implicits.using
import org.apache.daffodil.lib.util._
import org.apache.daffodil.runtime1.api.ChoiceMetadata
import org.apache.daffodil.runtime1.api.ComplexElementMetadata
@@ -55,9 +56,10 @@ class TestMetadataWalking {
def parseAndWalkData(dp: DataProcessor, infosetOutputter: InfosetOutputter)(
data: Array[Byte],
): ParseResult = {
- val isdis = InputSourceDataInputStream(data)
- val res = dp.parse(isdis, infosetOutputter)
- res
+ using(InputSourceDataInputStream(data)) { isdis =>
+ val res = dp.parse(isdis, infosetOutputter)
+ res
+ }
}
class GatherMetadata extends MetadataHandler {
diff --git a/daffodil-core/src/test/scala/org/apache/daffodil/core/dpath/TestDFDLExpressionEvaluation.scala b/daffodil-core/src/test/scala/org/apache/daffodil/core/dpath/TestDFDLExpressionEvaluation.scala
index 1fd22f51f..eb86c1f40 100644
--- a/daffodil-core/src/test/scala/org/apache/daffodil/core/dpath/TestDFDLExpressionEvaluation.scala
+++ b/daffodil-core/src/test/scala/org/apache/daffodil/core/dpath/TestDFDLExpressionEvaluation.scala
@@ -17,6 +17,7 @@
package org.apache.daffodil.core.dpath
+import java.nio.ByteBuffer
import scala.util.parsing.combinator.Parsers
import org.apache.daffodil.core.compiler._
@@ -62,13 +63,14 @@ class TestDFDLExpressionEvaluation extends Parsers {
val compiledExpr = exprCompiler.compile(expr)
val doc = infosetRootElem.parent.asInstanceOf[InfosetDocument]
- val dis = InputSourceDataInputStream(
- java.nio.ByteBuffer.allocate(0),
- ) // fake. Zero bits available.
- val outputter = new NullInfosetOutputter()
- val pstate = PState.createInitialPState(doc, erd, dis, outputter, dp, areDebugging = false)
- val result = compiledExpr.evaluate(pstate)
- body(result)
+ // Below, dis is fake. Zero bits available.
+ using(InputSourceDataInputStream(ByteBuffer.allocate(0))) { dis =>
+ val outputter = new NullInfosetOutputter()
+ val pstate =
+ PState.createInitialPState(doc, erd, dis, outputter, dp, areDebugging = false)
+ val result = compiledExpr.evaluate(pstate)
+ body(result)
+ }
}
@Test def test_a() = {
diff --git a/daffodil-core/src/test/scala/org/apache/daffodil/core/dsom/TestBinaryInput_01.scala b/daffodil-core/src/test/scala/org/apache/daffodil/core/dsom/TestBinaryInput_01.scala
index c88a0d3c9..24d59473e 100644
--- a/daffodil-core/src/test/scala/org/apache/daffodil/core/dsom/TestBinaryInput_01.scala
+++ b/daffodil-core/src/test/scala/org/apache/daffodil/core/dsom/TestBinaryInput_01.scala
@@ -39,7 +39,11 @@ class TestBinaryInput_01 {
var startOver: DataInputStream.Mark = null
var dis: DataInputStream = null
- def fromBytes(ba: Array[Byte], byteOrd: ByteOrder, bitOrd: BitOrder) = {
+ def fromBytes(
+ ba: Array[Byte],
+ byteOrd: ByteOrder,
+ bitOrd: BitOrder,
+ ): (DataInputStream, FakeFormatInfo) = {
dis = InputSourceDataInputStream(ba)
val finfo = new FakeFormatInfo(bitOrd, byteOrd)
startOver = dis.mark("TestBinaryInput_01")
diff --git a/daffodil-core/src/test/scala/org/apache/daffodil/core/layers/TestBoundaryMarkLayer.scala b/daffodil-core/src/test/scala/org/apache/daffodil/core/layers/TestBoundaryMarkLayer.scala
new file mode 100644
index 000000000..8b448bc0c
--- /dev/null
+++ b/daffodil-core/src/test/scala/org/apache/daffodil/core/layers/TestBoundaryMarkLayer.scala
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.core.layers
+
+import scala.xml.Elem
+
+import org.apache.daffodil.core.util.TestUtils
+import org.apache.daffodil.lib.Implicits.intercept
+import org.apache.daffodil.lib.util._
+import org.apache.daffodil.lib.xml.XMLUtils
+import org.apache.daffodil.runtime1.processors.parsers.ParseError
+import org.apache.daffodil.runtime1.processors.unparsers.UnparseError
+
+import org.junit.Assert.assertTrue
+import org.junit.Test
+
+class TestBoundaryMarkLayer {
+ val example = XMLUtils.EXAMPLE_NAMESPACE
+
+ def bm1Schema(bm: String, encoding: String = "iso-8859-1"): Elem =
+ SchemaUtils.dfdlTestSchema(
+ <xs:include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+ <xs:import namespace="urn:org.apache.daffodil.layers.boundaryMark"
+ schemaLocation="/org/apache/daffodil/layers/xsd/boundaryMarkLayer.dfdl.xsd"/>,
+ <dfdl:format ref="tns:GeneralFormat" lengthKind="delimited"/>
+ <dfdl:defineVariable name="bmark" type="xs:string" external="true" defaultValue={
+ bm
+ }/>,
+ <xs:element name="e1" dfdl:lengthKind="implicit"
+ xmlns:bm="urn:org.apache.daffodil.layers.boundaryMark">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:sequence dfdlx:layer="bm:boundaryMark">
+ <xs:annotation>
+ <xs:appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:newVariableInstance ref="bm:boundaryMark" defaultValue="{ $ex:bmark }"/>
+ <dfdl:newVariableInstance ref="bm:layerEncoding" defaultValue={encoding}/>
+ </xs:appinfo>
+ </xs:annotation>
+ <xs:element name="s1" type="xs:string"/>
+ </xs:sequence>
+ <xs:element name="after" type="xs:string"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>,
+ elementFormDefault = "unqualified",
+ )
+
+ val abcInfoset =
+ <ex:e1 xmlns:ex={example}>
+ <s1>AB</s1>
+ <after>C</after>
+ </ex:e1>
+
+ @Test def testBoundaryMarkLayerOk1(): Unit = {
+ val sch = bm1Schema(";")
+ val data = "AB;C"
+ val infoset = abcInfoset
+
+ val (_, actual) = TestUtils.testString(sch, data, areTracing = false)
+ TestUtils.assertEqualsXMLElements(infoset, actual)
+
+ TestUtils.testUnparsing(sch, infoset, data, areTracing = false)
+ }
+
+ @Test def testBoundaryMarkBadEmptyString1(): Unit = {
+ val sch = bm1Schema("")
+ val data = "AB;C"
+ val infoset = abcInfoset
+
+ val e = intercept[ParseError] {
+ TestUtils.testString(sch, data, areTracing = false)
+ }
+ val msg = e.getMessage.toLowerCase()
+ assertTrue(msg.contains("may not be empty string"))
+
+ val f = intercept[UnparseError] {
+ TestUtils.testUnparsing(sch, infoset, data, areTracing = false)
+ }
+ val m = f.getMessage.toLowerCase
+ assertTrue(m.contains("may not be empty string"))
+ }
+
+ @Test def testBoundaryMarkBadTooLongString1(): Unit = {
+ val sch = bm1Schema("!" * 32768)
+ val data = "AB;C"
+ val infoset = abcInfoset
+
+ val e = intercept[ParseError] {
+ TestUtils.testString(sch, data, areTracing = false)
+ }
+ val msg = e.getMessage.toLowerCase()
+ assertTrue(msg.contains("length may not be greater than the limit"))
+
+ val f = intercept[UnparseError] {
+ TestUtils.testUnparsing(sch, infoset, data, areTracing = false)
+ }
+ val m = f.getMessage.toLowerCase
+ assertTrue(m.contains("length may not be greater than the limit"))
+ }
+
+ @Test def testBoundaryMarkBadLayerEncoding1(): Unit = {
+ val sch = bm1Schema(";", "notACharsetName")
+ val data = "AB;C"
+ val infoset = abcInfoset
+
+ val e = intercept[ParseError] {
+ TestUtils.testString(sch, data, areTracing = false)
+ }
+ val msg = e.getMessage
+ assertTrue(msg.contains("notACharsetName"))
+
+ val f = intercept[UnparseError] {
+ TestUtils.testUnparsing(sch, infoset, data, areTracing = false)
+ }
+ val m = f.getMessage
+ assertTrue(m.contains("notACharsetName"))
+ }
+}
diff --git a/daffodil-core/src/test/scala/org/apache/daffodil/core/layers/TestFixedLengthLayer.scala b/daffodil-core/src/test/scala/org/apache/daffodil/core/layers/TestFixedLengthLayer.scala
new file mode 100644
index 000000000..14d99438b
--- /dev/null
+++ b/daffodil-core/src/test/scala/org/apache/daffodil/core/layers/TestFixedLengthLayer.scala
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.core.layers
+
+import scala.xml.Elem
+
+import org.apache.daffodil.core.util.TestUtils
+import org.apache.daffodil.lib.Implicits.intercept
+import org.apache.daffodil.lib.util._
+import org.apache.daffodil.lib.xml.XMLUtils
+import org.apache.daffodil.runtime1.processors.parsers.ParseError
+import org.apache.daffodil.runtime1.processors.unparsers.UnparseError
+
+import org.junit.Assert.assertTrue
+import org.junit.Test
+
+class TestFixedLengthLayer {
+ val example = XMLUtils.EXAMPLE_NAMESPACE
+
+ def fl1Schema(flLayerLen: Int): Elem =
+ SchemaUtils.dfdlTestSchema(
+ <xs:include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+ <xs:import namespace="urn:org.apache.daffodil.layers.fixedLength"
+ schemaLocation="/org/apache/daffodil/layers/xsd/fixedLengthLayer.dfdl.xsd"/>,
+ <dfdl:format ref="tns:GeneralFormat" lengthKind="delimited"/>
+ <dfdl:defineVariable name="flLayerLen" type="xs:int" external="true" defaultValue={
+ flLayerLen.toString
+ }/>,
+ <xs:element name="e1" dfdl:lengthKind="implicit"
+ xmlns:fl="urn:org.apache.daffodil.layers.fixedLength">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:sequence dfdlx:layer="fl:fixedLength">
+ <xs:annotation><xs:appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:newVariableInstance ref="fl:fixedLength" defaultValue="{ $ex:flLayerLen }"/>
+ </xs:appinfo></xs:annotation>
+ <xs:element name="s1" type="xs:string" />
+ </xs:sequence>
+ <xs:element name="after" type="xs:string"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>,
+ elementFormDefault = "unqualified",
+ )
+
+ val abcInfoset =
+ <ex:e1 xmlns:ex={example}>
+ <s1>AB</s1>
+ <after>C</after>
+ </ex:e1>
+
+ @Test def testfixedLengthLayerOk1(): Unit = {
+ val sch = fl1Schema(2)
+ val data = "ABC"
+ val infoset = abcInfoset
+
+ val (_, actual) = TestUtils.testString(sch, data, areTracing = false)
+ TestUtils.assertEqualsXMLElements(infoset, actual)
+
+ TestUtils.testUnparsing(sch, infoset, data, areTracing = false)
+ }
+
+ @Test def testfixedLengthLayerBadNegLen1(): Unit = {
+ val sch = fl1Schema(-1)
+ val data = "ABC"
+ val infoset = abcInfoset
+
+ val e = intercept[ParseError] {
+ TestUtils.testString(sch, data, areTracing = false)
+ }
+ val msg = e.getMessage.toLowerCase()
+ assertTrue(msg.contains("cannot convert '-1'"))
+
+ val f = intercept[UnparseError] {
+ TestUtils.testUnparsing(sch, infoset, data, areTracing = false)
+ }
+ val m = f.getMessage.toLowerCase
+ assertTrue(m.contains("cannot convert '-1'"))
+ }
+
+ @Test def testfixedLengthLayerBadTooBigLen1(): Unit = {
+ val sch = fl1Schema(65536)
+ val data = "ABC"
+ val infoset = abcInfoset
+
+ val e = intercept[ParseError] {
+ TestUtils.testString(sch, data, areTracing = false)
+ }
+ val msg = e.getMessage.toLowerCase()
+ assertTrue(msg.contains("65536 is above the maximum"))
+
+ val f = intercept[UnparseError] {
+ TestUtils.testUnparsing(sch, infoset, data, areTracing = false)
+ }
+ val m = f.getMessage.toLowerCase
+ assertTrue(m.contains("65536 is above the maximum"))
+ }
+
+ @Test def testfixedLengthLayerBadParseDataShort1(): Unit = {
+ val sch = fl1Schema(2)
+ val data = "A"
+ val infoset = abcInfoset
+
+ val e = intercept[ParseError] {
+ TestUtils.testString(sch, data, areTracing = false)
+ }
+ val msg = e.getMessage.toLowerCase()
+ assertTrue(msg.contains("insufficient data"))
+ }
+
+ @Test def testfixedLengthLayerBadUnparseDataShort1(): Unit = {
+ val sch = fl1Schema(2)
+ val data = "A"
+ val infoset = <ex:e1 xmlns:ex={example}>
+ <s1>A</s1>
+ <after>D</after>
+ </ex:e1>
+
+ val e = intercept[UnparseError] {
+ TestUtils.testUnparsing(sch, infoset, data, areTracing = false)
+ }
+ val msg = e.getMessage.toLowerCase()
+ assertTrue(msg.contains("insufficient output data"))
+ }
+
+ @Test def testfixedLengthLayerBadUnparseDataLong1(): Unit = {
+ val sch = fl1Schema(2)
+ val data = "ABC"
+ val infoset = <ex:e1 xmlns:ex={example}>
+ <s1>ABC</s1>
+ <after>D</after>
+ </ex:e1>
+ val f = intercept[UnparseError] {
+ TestUtils.testUnparsing(sch, infoset, data, areTracing = false)
+ }
+ val m = f.getMessage.toLowerCase
+ assertTrue(m.contains("exceeded fixed layer length of 2"))
+ }
+}
diff --git a/daffodil-core/src/test/scala/org/apache/daffodil/core/layers/TestGzipErrors.scala b/daffodil-core/src/test/scala/org/apache/daffodil/core/layers/TestGzipErrors.scala
new file mode 100644
index 000000000..b6a1dab7d
--- /dev/null
+++ b/daffodil-core/src/test/scala/org/apache/daffodil/core/layers/TestGzipErrors.scala
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.core.layers
+
+import java.io.ByteArrayOutputStream
+import java.nio.charset.StandardCharsets
+import scala.xml.Elem
+
+import org.apache.daffodil.core.util.FuzzOneBits
+import org.apache.daffodil.core.util.FuzzRandomByteRuns
+import org.apache.daffodil.core.util.FuzzRandomSingleByte
+import org.apache.daffodil.core.util.FuzzZeroBits
+import org.apache.daffodil.core.util.LayerParseTester
+import org.apache.daffodil.core.util.TestUtils
+import org.apache.daffodil.lib.util._
+import org.apache.daffodil.lib.xml.XMLUtils
+
+import org.apache.commons.io.IOUtils
+import org.junit.Test
+
+class TestGzipErrors {
+
+ val example = XMLUtils.EXAMPLE_NAMESPACE
+
+ val text =
+ """This is just some made up text that is intended to be
+a few lines long. If this had been real text, it would not have been quite
+so boring to read. Use of famous quotes or song lyrics or anything like that
+introduces copyright notice issues, so it is easier to simply make up
+a few lines of pointless text like this.""".replace("\r\n", "\n").replace("\n", " ")
+
+ val GZIPLayer1Schema =
+ SchemaUtils.dfdlTestSchema(
+ <xs:include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+ <xs:import namespace="urn:org.apache.daffodil.layers.gzip"
+ schemaLocation="/org/apache/daffodil/layers/xsd/gzipLayer.dfdl.xsd"/>,
+ <dfdl:format ref="tns:GeneralFormat" representation="binary"/>,
+ <xs:element name="root">
+ <xs:complexType>
+ <xs:choice>
+ <xs:element name="e1" dfdl:lengthKind="implicit"
+ xmlns:gz="urn:org.apache.daffodil.layers.gzip">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="len" type="xs:int" dfdl:lengthKind="explicit" dfdl:length="4"
+ dfdl:outputValueCalc="{ dfdl:contentLength(../x1/data, 'bytes') }"/>
+ <xs:element name="x1" dfdl:lengthKind="explicit" dfdl:length="{ ../len }">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="data">
+ <xs:complexType>
+ <xs:sequence dfdlx:layer="gz:gzip">
+ <xs:element name="s1" type="xs:string" dfdl:lengthKind="delimited"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ <xs:element name="s2" type="xs:string" dfdl:lengthKind="delimited"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ <xs:element name="hex" type="xs:hexBinary" dfdl:lengthKind="delimited" dfdl:encoding="iso-8859-1"/>
+ </xs:choice>
+ </xs:complexType>
+ </xs:element>,
+ elementFormDefault = "unqualified",
+ )
+
+ def makeGZIPData(text: String) = {
+ val baos = new ByteArrayOutputStream()
+ val gzos = new java.util.zip.GZIPOutputStream(baos)
+ IOUtils.write(text, gzos, StandardCharsets.UTF_8)
+ gzos.close()
+ val data = baos.toByteArray()
+ // Java 16+ sets the 9th byte to 0xFF, but previous Java versions set the
+ // value to 0x00. Daffodil always unparses with 0xFF regardless of Java
+ // version, so force the gzip data to 0xFF to make sure tests round trip
+ data(9) = 0xff.toByte
+ data
+ }
+
+ def makeGZIPLayer1Data() = {
+ val gzipData = makeGZIPData(text)
+ val dataLength = gzipData.length
+ val baos = new ByteArrayOutputStream()
+ val dos = new java.io.DataOutputStream(baos)
+ dos.writeInt(dataLength)
+ dos.write(gzipData)
+ dos.write("afterGzip".getBytes(StandardCharsets.UTF_8))
+ dos.close()
+ val data = baos.toByteArray()
+ (data, dataLength)
+ }
+
+ @Test def testGZIPLayerOk1(): Unit = {
+ val sch = GZIPLayer1Schema
+ val (data, _) = makeGZIPLayer1Data()
+ val infoset =
+ <ex:root xmlns:ex={example}>
+ <e1>
+ <len>{212}</len>
+ <x1>
+ <data>
+ <s1>{text}</s1>
+ </data>
+ </x1>
+ <s2>afterGzip</s2>
+ </e1>
+ </ex:root>
+ val (_, actual) = TestUtils.testBinary(sch, data, areTracing = false)
+ TestUtils.assertEqualsXMLElements(infoset, actual)
+ }
+
+ @Test def testGZIPLayerErr1(): Unit = {
+ val sch = GZIPLayer1Schema
+ val (data, _) = makeGZIPLayer1Data()
+ // clobber half the data
+ ((data.length / 2) to data.length - 1).foreach { i => data(i) = 0 }
+ // This causes the gzip input stream to throw an IOException
+ val infoset =
+ <ex:root xmlns:ex={example}>
+ <hex xmlns=""><![CDATA[000000D41F8B08000000000000FF4D904176C3200C44AF3207C8F33DBA6F0F40CCD85683918B44D3DC3EC2C9A2EFB1013EF3357C6E6288F5DDCD61BA137BCA443FE0FC73F8967C5C4B75D6CC0C575C8984857714A93414ADEB848F25D800B794036045632A67C605E2B86B2F19553D800000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000]]></hex>
+ </ex:root>
+ val (_, actual) = TestUtils.testBinary(sch, data, areTracing = false)
+ TestUtils.assertEqualsXMLElements(infoset, actual)
+ }
+
+ @Test def testGZIPLayerErr2(): Unit = {
+ val sch = GZIPLayer1Schema
+ val (data, _) = makeGZIPLayer1Data()
+ // clobber last 10% of the data with 0xFF bytes.
+ ((data.length - (data.length / 10)) to data.length - 1).foreach { i => data(i) = -1 }
+ // This causes the gzip input stream to throw an IOException
+ val infoset =
+ <ex:root xmlns:ex={example}>
+ <hex xmlns=""><![CDATA[000000D41F8B08000000000000FF4D904176C3200C44AF3207C8F33DBA6F0F40CCD85683918B44D3DC3EC2C9A2EFB1013EF3357C6E6288F5DDCD61BA137BCA443FE0FC73F8967C5C4B75D6CC0C575C8984857714A93414ADEB848F25D800B794036045632A67C605E2B86B2F19553D805FBE889F2ECE70E2AA4DEA3AA2E3519EF065842E58D2AEDD02530F8DB640832A8F26F3B94DF511CA712437BE27ADDE34F739F8598F20D7CD875566460BEBB4CB10CAD989C9846D684DF6A33CA2F9ED6CFEBF5DCC7168C4169ABDBEFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF]]></hex>
+ </ex:root>
+ val (_, actual) = TestUtils.testBinary(sch, data, areTracing = false)
+ TestUtils.assertEqualsXMLElements(infoset, actual)
+ }
+
+ val GZIPLayerErrSchema =
+ SchemaUtils.dfdlTestSchema(
+ <xs:include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+ <xs:import namespace="urn:org.apache.daffodil.layers.gzip"
+ schemaLocation="/org/apache/daffodil/layers/xsd/gzipLayer.dfdl.xsd"/>,
+ <dfdl:format ref="tns:GeneralFormat" representation="binary"/>,
+ <xs:element name="root">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="e1" dfdl:lengthKind="implicit"
+ xmlns:gz="urn:org.apache.daffodil.layers.gzip">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="len" type="xs:unsignedInt" dfdl:lengthKind="explicit" dfdl:length="4"
+ dfdl:outputValueCalc="{ dfdl:contentLength(../x1/data, 'bytes') }"/>
+ <xs:element name="x1" dfdl:lengthKind="explicit" dfdl:length="{ ../len }">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="data">
+ <xs:complexType>
+ <xs:sequence dfdlx:layer="gz:gzip">
+ <xs:element name="s1" type="xs:string" dfdl:lengthKind="delimited"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ <xs:element name="s2" type="xs:string" dfdl:lengthKind="delimited"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>,
+ elementFormDefault = "unqualified",
+ )
+
+ /**
+ * These are parse errors due to IOExceptions that occur.
+ */
+ val excludes: Seq[String] = Seq(
+ "EOFException",
+ "Corrupt GZIP trailer",
+ "Unexpected end of ZLIB",
+ "invalid distance too far back",
+ "Not in GZIP format",
+ "invalid literal/lengths set",
+ "invalid bit length repeat",
+ "invalid code lengths set",
+ "invalid code -- missing end-of-block",
+ "invalid distances set",
+ "Unsupported compression method",
+ "Insufficient bits in data",
+ "too many length or distance symbols",
+ "invalid stored block lengths",
+ "invalid block type",
+ "Corrupt GZIP header",
+ "invalid distance code",
+ "invalid literal/length code",
+ )
+
+ val expected = <root xmlns="http://example.com">
+ <e1 xmlns="">
+ <len>212</len> <x1>
+ <data>
+ <s1>This is just some made up text that is intended to be a few lines long. If this had been real text, it would not have been quite so boring to read. Use of famous quotes or song lyrics or anything like that introduces copyright notice issues, so it is easier to simply make up a few lines of pointless text like this.</s1>
+ </data>
+ </x1> <s2>afterGzip</s2>
+ </e1>
+ </root>
+
+ // fuzz0 - writes runs of 0 bytes of random length in random locations
+ def fuzz0(nTrials: Int, sch: Elem, data: Array[Byte], expected: Elem): Unit = {
+ var shouldFail = false
+ val fuzzer = new FuzzZeroBits(data, 4, 0)
+ val tester = new LayerParseTester(sch, data, expected, fuzzer, excludes)
+ tester.run(nTrials)
+ }
+
+ // fuzz1 - writes runs of 0xFF (all 1 bits) bytes of random length in random locations
+ def fuzz1(nTrials: Int, sch: Elem, data: Array[Byte], expected: Elem): Unit = {
+ var shouldFail = false
+ val fuzzer = new FuzzOneBits(data, 4, 0)
+ val tester = new LayerParseTester(sch, data, expected, fuzzer, excludes)
+ tester.run(nTrials)
+ }
+
+ // fuzz2 only modifies 1 byte at a time in the data with a random (but always different) byte
+ def fuzz2(nTrials: Int, sch: Elem, data: Array[Byte], expected: Elem): Unit = {
+ val fuzzer =
+ new FuzzRandomSingleByte(data, 4, 9) // leave first 4 bytes and last 9 bytes alone.
+ val tester = new LayerParseTester(sch, data, expected, fuzzer, excludes)
+ tester.run(nTrials)
+ }
+
+ // fuzz3 modifies runs of bytes of random length.
+ def fuzz3(nTrials: Int, sch: Elem, data: Array[Byte], expected: Elem): Unit = {
+ var shouldFail = false
+ val fuzzer = new FuzzRandomByteRuns(data, 4, 0)
+ val tester = new LayerParseTester(sch, data, expected, fuzzer, excludes)
+ tester.run(nTrials)
+ }
+
+ // Can be run for 100,000 trials - 11 cases fuzzed data did not stop the parse
+ @Test def testGZIPLayerFuzz0(): Unit = {
+ fuzz0(10, sch = GZIPLayerErrSchema, data = makeGZIPLayer1Data()._1, expected)
+ }
+
+ // Can be run for 100,000 trials - 11 cases fuzzed data did not stop the parse
+ @Test def testGZIPLayerFuzz1(): Unit = {
+ fuzz1(10, sch = GZIPLayerErrSchema, data = makeGZIPLayer1Data()._1, expected)
+ }
+
+ // Can be run for 100,000 trials
+ @Test def testGZIPLayerFuzz2(): Unit = {
+ fuzz2(10, sch = GZIPLayerErrSchema, data = makeGZIPLayer1Data()._1, expected)
+ }
+
+ // This has been run for 100K trials
+ // which only takes a few seconds, and
+ // all errors thrown were converted to parse errors
+ // so that means they were all IOExceptions from the gzip layer.
+ @Test def testGZIPLayerFuzz3(): Unit =
+ fuzz3(10, sch = GZIPLayerErrSchema, data = makeGZIPLayer1Data()._1, expected)
+
+}
diff --git a/daffodil-core/src/test/scala/org/apache/daffodil/core/layers/TestLayers.scala b/daffodil-core/src/test/scala/org/apache/daffodil/core/layers/TestLayers.scala
index 18c9d130c..74b5dcc71 100644
--- a/daffodil-core/src/test/scala/org/apache/daffodil/core/layers/TestLayers.scala
+++ b/daffodil-core/src/test/scala/org/apache/daffodil/core/layers/TestLayers.scala
@@ -37,13 +37,48 @@ class TestLayers {
val B64Layer1Schema =
SchemaUtils.dfdlTestSchema(
- <xs:include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>,
+ <xs:include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+ <xs:import namespace="urn:org.apache.daffodil.layers.boundaryMark"
+ schemaLocation="/org/apache/daffodil/layers/xsd/boundaryMarkLayer.dfdl.xsd"/>
+ <xs:import namespace="urn:org.apache.daffodil.layers.base64_MIME"
+ schemaLocation="/org/apache/daffodil/layers/xsd/base64_MIMELayer.dfdl.xsd"/>,
<dfdl:format ref="tns:GeneralFormat"/>,
- <xs:element name="e1" dfdl:lengthKind="implicit">
+ <xs:element name="e1" dfdl:lengthKind="implicit"
+ xmlns:bm="urn:org.apache.daffodil.layers.boundaryMark"
+ xmlns:b64="urn:org.apache.daffodil.layers.base64_MIME">
<xs:complexType>
- <xs:sequence dfdlx:layerTransform="base64_MIME" dfdlx:layerLengthKind="boundaryMark" dfdlx:layerBoundaryMark="!" dfdlx:layerEncoding="iso-8859-1">
- <xs:element name="s1" type="xs:string" dfdl:lengthKind="explicit" dfdl:length="3"/>
- </xs:sequence>
+ <xs:sequence dfdlx:layer="bm:boundaryMark">
+ <!--
+ notice that the newVariableInstance bindings can be on the sequence
+ that has the layer reference on it. There need not be yet another
+ sequence wrapped around the outside.
+ -->
+ <xs:annotation><xs:appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:newVariableInstance ref="bm:layerEncoding" defaultValue="ascii"/>
+ <dfdl:newVariableInstance ref="bm:boundaryMark" defaultValue="!"/>
+ </xs:appinfo></xs:annotation>
+ <xs:sequence dfdlx:layer="b64:base64_MIME" >
+ <xs:element name="s1" type="xs:string" dfdl:lengthKind="explicit" dfdl:length="3"/>
+ <!--
+ this sequence tests that the contents of a layered sequence can be more than 1 child
+ when there is no annotation starting the sequence.
+ -->
+ <xs:sequence>
+ <xs:annotation><xs:appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:assert test='{ $bm:layerEncoding eq "ascii" }'/>
+ </xs:appinfo></xs:annotation>
+ </xs:sequence>
+ </xs:sequence>
+ <xs:sequence>
+ <!--
+ this sequence tests that the contents of a layered sequence can be more than 1 child
+ and the sequence can furthermore begin with an annotation block
+ -->
+ <xs:annotation><xs:appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:assert test='{ $bm:layerEncoding eq "ascii" }'/>
+ </xs:appinfo></xs:annotation>
+ </xs:sequence>
+ </xs:sequence>
</xs:complexType>
</xs:element>,
elementFormDefault = "unqualified",
@@ -62,12 +97,24 @@ class TestLayers {
val B64Layer2Schema =
SchemaUtils.dfdlTestSchema(
- <xs:include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>,
+ <xs:include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+ <xs:import namespace="urn:org.apache.daffodil.layers.boundaryMark"
+ schemaLocation="/org/apache/daffodil/layers/xsd/boundaryMarkLayer.dfdl.xsd"/>
+ <xs:import namespace="urn:org.apache.daffodil.layers.base64_MIME"
+ schemaLocation="/org/apache/daffodil/layers/xsd/base64_MIMELayer.dfdl.xsd"/>,
<dfdl:format ref="tns:GeneralFormat" lengthKind='delimited'/>,
- <xs:element name="e1" dfdl:lengthKind="implicit">
+ <xs:element name="e1" dfdl:lengthKind="implicit"
+ xmlns:bm="urn:org.apache.daffodil.layers.boundaryMark"
+ xmlns:b64="urn:org.apache.daffodil.layers.base64_MIME">
<xs:complexType>
- <xs:sequence dfdlx:layerTransform="base64_MIME" dfdlx:layerLengthKind="boundaryMark" dfdlx:layerBoundaryMark="!" dfdlx:layerEncoding="iso-8859-1">
- <xs:element name="s1" type="xs:string"/>
+ <xs:sequence dfdlx:layer="bm:boundaryMark">
+ <xs:annotation><xs:appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:newVariableInstance ref="bm:layerEncoding" defaultValue="iso-8859-1"/>
+ <dfdl:newVariableInstance ref="bm:boundaryMark" defaultValue="!"/>
+ </xs:appinfo></xs:annotation>
+ <xs:sequence dfdlx:layer="b64:base64_MIME">
+ <xs:element name="s1" type="xs:string"/>
+ </xs:sequence>
</xs:sequence>
</xs:complexType>
</xs:element>,
@@ -87,13 +134,25 @@ class TestLayers {
val B64Layer3Schema =
SchemaUtils.dfdlTestSchema(
- <xs:include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>,
+ <xs:include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+ <xs:import namespace="urn:org.apache.daffodil.layers.boundaryMark"
+ schemaLocation="/org/apache/daffodil/layers/xsd/boundaryMarkLayer.dfdl.xsd"/>
+ <xs:import namespace="urn:org.apache.daffodil.layers.base64_MIME"
+ schemaLocation="/org/apache/daffodil/layers/xsd/base64_MIMELayer.dfdl.xsd"/>,
<dfdl:format ref="tns:GeneralFormat" lengthKind='delimited'/>,
- <xs:element name="e1" dfdl:lengthKind="implicit">
+ <xs:element name="e1" dfdl:lengthKind="implicit"
+ xmlns:bm="urn:org.apache.daffodil.layers.boundaryMark"
+ xmlns:b64="urn:org.apache.daffodil.layers.base64_MIME">
<xs:complexType>
<xs:sequence>
- <xs:sequence dfdlx:layerTransform="base64_MIME" dfdlx:layerLengthKind="boundaryMark" dfdlx:layerBoundaryMark="!" dfdlx:layerEncoding="iso-8859-1">
- <xs:element name="s1" type="xs:string"/>
+ <xs:sequence dfdlx:layer="bm:boundaryMark">
+ <xs:annotation><xs:appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:newVariableInstance ref="bm:layerEncoding" defaultValue="iso-8859-1"/>
+ <dfdl:newVariableInstance ref="bm:boundaryMark" defaultValue="!"/>
+ </xs:appinfo></xs:annotation>
+ <xs:sequence dfdlx:layer="b64:base64_MIME">
+ <xs:element name="s1" type="xs:string"/>
+ </xs:sequence>
</xs:sequence>
<xs:element name="s2" type="xs:string"/>
</xs:sequence>
@@ -141,18 +200,64 @@ a few lines of pointless text like this.""".replace("\r\n", "\n").replace("\n",
assertEquals(text, textBack)
}
+ val GZIPLayer0Schema =
+ SchemaUtils.dfdlTestSchema(
+ <xs:include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+ <xs:import namespace="urn:org.apache.daffodil.layers.gzip"
+ schemaLocation="/org/apache/daffodil/layers/xsd/gzipLayer.dfdl.xsd"/>,
+ <dfdl:format ref="ex:GeneralFormat" representation="binary"/>,
+ <xs:element name="e1" dfdl:lengthKind="implicit"
+ xmlns:gz="urn:org.apache.daffodil.layers.gzip">
+ <xs:complexType>
+ <xs:sequence dfdlx:layer="gz:gzip">
+ <xs:element name="s1" type="xs:string" dfdl:lengthKind="delimited"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>,
+ elementFormDefault = "unqualified",
+ )
+
+ def makeGZIPLayer0Data() = {
+ val gzipData: Array[Byte] = makeGZIPData(text)
+ gzipData
+ }
+
+ @Test
+ def testGZIPLayer0(): Unit = {
+ val sch = GZIPLayer0Schema
+ val data = makeGZIPLayer0Data()
+ val infoset = <ex:e1 xmlns:ex={example}><s1>{
+ text
+ }</s1></ex:e1>
+ val (_, actual) = TestUtils.testBinary(sch, data, areTracing = false)
+ TestUtils.assertEqualsXMLElements(infoset, actual)
+
+ TestUtils.testUnparsingBinary(sch, infoset, data)
+ }
+
+ // TODO: use prefixed length kind to simplify this example.
val GZIPLayer1Schema =
SchemaUtils.dfdlTestSchema(
- <xs:include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>,
- <dfdl:format ref="tns:GeneralFormat" dfdlx:layerLengthUnits="bytes" representation="binary"/>,
- <xs:element name="e1" dfdl:lengthKind="implicit">
+ <xs:include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+ <xs:import namespace="urn:org.apache.daffodil.layers.gzip"
+ schemaLocation="/org/apache/daffodil/layers/xsd/gzipLayer.dfdl.xsd"/>,
+ <dfdl:format ref="tns:GeneralFormat" representation="binary"/>,
+ <xs:element name="e1" dfdl:lengthKind="implicit"
+ xmlns:gz="urn:org.apache.daffodil.layers.gzip">
<xs:complexType>
<xs:sequence>
- <xs:element name="len" type="xs:int" dfdl:lengthKind="explicit" dfdl:length="4" dfdl:outputValueCalc="{ dfdl:contentLength(../x1, 'bytes') }"/>
- <xs:element name="x1" dfdl:lengthKind="implicit">
+ <xs:element name="len" type="xs:int" dfdl:lengthKind="explicit" dfdl:length="4"
+ dfdl:outputValueCalc="{ dfdl:contentLength(../x1/data, 'bytes') }"/>
+ <xs:element name="x1" dfdl:lengthKind="explicit" dfdl:length="{ ../len }">
<xs:complexType>
- <xs:sequence dfdlx:layerTransform="gzip" dfdlx:layerLengthKind="explicit" dfdlx:layerLength="{ ../len }">
- <xs:element name="s1" type="xs:string" dfdl:lengthKind="delimited"/>
+ <xs:sequence>
+ <xs:element name="data">
+ <xs:complexType>
+ <xs:sequence dfdlx:layer="gz:gzip">
+ <xs:element name="s1" type="xs:string" dfdl:lengthKind="delimited"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
@@ -179,9 +284,9 @@ a few lines of pointless text like this.""".replace("\r\n", "\n").replace("\n",
@Test def testGZIPLayer1(): Unit = {
val sch = GZIPLayer1Schema
val (data, dataLength) = makeGZIPLayer1Data()
- val infoset = <ex:e1 xmlns:ex={example}><len>{dataLength}</len><x1><s1>{
+ val infoset = <ex:e1 xmlns:ex={example}><len>{dataLength}</len><x1><data><s1>{
text
- }</s1></x1><s2>afterGzip</s2></ex:e1>
+ }</s1></data></x1><s2>afterGzip</s2></ex:e1>
val (_, actual) = TestUtils.testBinary(sch, data, areTracing = false)
TestUtils.assertEqualsXMLElements(infoset, actual)
@@ -189,28 +294,46 @@ a few lines of pointless text like this.""".replace("\r\n", "\n").replace("\n",
}
def makeB64GZIPSchema(term: String, layerTerm: String) = SchemaUtils.dfdlTestSchema(
- <xs:include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>,
- <dfdl:format ref="tns:GeneralFormat" dfdlx:layerLengthUnits="bytes" representation="binary" dfdlx:layerEncoding="iso-8859-1"/>,
- <xs:element name="e1" dfdl:lengthKind="implicit">
+ <xs:include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+ <xs:import namespace="urn:org.apache.daffodil.layers.boundaryMark"
+ schemaLocation="/org/apache/daffodil/layers/xsd/boundaryMarkLayer.dfdl.xsd"/>
+ <xs:import namespace="urn:org.apache.daffodil.layers.gzip"
+ schemaLocation="/org/apache/daffodil/layers/xsd/gzipLayer.dfdl.xsd"/>
+ <xs:import namespace="urn:org.apache.daffodil.layers.base64_MIME"
+ schemaLocation="/org/apache/daffodil/layers/xsd/base64_MIMELayer.dfdl.xsd"/>,
+ <dfdl:format ref="tns:GeneralFormat" representation="binary"/>,
+ <xs:element name="e1" dfdl:lengthKind="implicit"
+ xmlns:bm="urn:org.apache.daffodil.layers.boundaryMark"
+ xmlns:b64="urn:org.apache.daffodil.layers.base64_MIME"
+ xmlns:gz="urn:org.apache.daffodil.layers.gzip">
<xs:complexType>
<xs:sequence>
- <xs:element name="s1" type="xs:string" dfdl:lengthKind="delimited" dfdl:terminator={
- term
- }/>
- <xs:sequence dfdlx:layerTransform="base64_MIME" dfdlx:layerLengthKind="boundaryMark" dfdlx:layerBoundaryMark={
- layerTerm
- }>
- <xs:sequence>
- <xs:element name="len" type="xs:int" dfdl:outputValueCalc="{ dfdl:contentLength(../x1, 'bytes') }"/>
- <xs:element name="x1" dfdl:lengthKind="implicit">
- <xs:complexType>
- <xs:sequence dfdlx:layerTransform="gzip" dfdlx:layerLengthKind="explicit" dfdlx:layerLength="{ ../len }">
- <xs:element name="s2" type="xs:string" dfdl:lengthKind="delimited"/>
- </xs:sequence>
- </xs:complexType>
- </xs:element>
+ <xs:element name="s1" type="xs:string" dfdl:lengthKind="delimited"
+ dfdl:terminator={term}/>
+ <xs:sequence dfdlx:layer="bm:boundaryMark">
+ <xs:annotation><xs:appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:newVariableInstance ref="bm:layerEncoding" defaultValue="iso-8859-1"/>
+ <dfdl:newVariableInstance ref="bm:boundaryMark" defaultValue={layerTerm}/>
+ </xs:appinfo></xs:annotation>
+ <xs:sequence dfdlx:layer="b64:base64_MIME">
+ <xs:sequence>
+ <xs:element name="len" type="xs:int" dfdl:outputValueCalc="{ dfdl:contentLength(../x1/data, 'bytes') }"/>
+ <xs:element name="x1" dfdl:lengthKind="explicit" dfdl:length="{ ../len }">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="data">
+ <xs:complexType>
+ <xs:sequence dfdlx:layer="gz:gzip">
+ <xs:element name="s2" type="xs:string" dfdl:lengthKind="delimited"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ </xs:sequence>
</xs:sequence>
- </xs:sequence>
<xs:element name="s3" type="xs:string" dfdl:lengthKind="delimited"/>
</xs:sequence>
</xs:complexType>
@@ -257,9 +380,11 @@ a few lines of pointless text like this.""".replace("\r\n", "\n").replace("\n",
val after = "afterB64GZip"
val (data, dataLength) = makeB64GZIPData(term, layerTerm, before, after, text)
val (_, actual) = TestUtils.testBinary(sch, data, areTracing = false)
- val infoset = <ex:e1 xmlns:ex={example}><s1>{before}</s1><len>{dataLength}</len><x1><s2>{
+ val infoset = <ex:e1 xmlns:ex={example}><s1>{before}</s1><len>{
+ dataLength
+ }</len><x1><data><s2>{
text
- }</s2></x1><s3>{after}</s3></ex:e1>
+ }</s2></data></x1><s3>{after}</s3></ex:e1>
TestUtils.assertEqualsXMLElements(infoset, actual)
TestUtils.testUnparsingBinary(sch, infoset, data)
@@ -267,11 +392,14 @@ a few lines of pointless text like this.""".replace("\r\n", "\n").replace("\n",
val lineFoldLayer1Schema =
SchemaUtils.dfdlTestSchema(
- <xs:include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>,
+ <xs:include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+ <xs:import namespace="urn:org.apache.daffodil.layers.lineFolded"
+ schemaLocation="/org/apache/daffodil/layers/xsd/lineFoldedLayer.dfdl.xsd"/>,
<dfdl:format ref="tns:GeneralFormat"/>,
- <xs:element name="e1" dfdl:lengthKind="implicit">
+ <xs:element name="e1" dfdl:lengthKind="implicit"
+ xmlns:lf="urn:org.apache.daffodil.layers.lineFolded">
<xs:complexType>
- <xs:sequence dfdlx:layerTransform="lineFolded_IMF" dfdlx:layerLengthKind="implicit">
+ <xs:sequence dfdlx:layer="lf:lineFolded_IMF">
<xs:element name="s1" type="xs:string" dfdl:lengthKind="delimited"/>
</xs:sequence>
</xs:complexType>
@@ -299,8 +427,8 @@ a few lines of pointless text like this.""".replace("\r\n", "\n").replace("\n",
val ipsumLorem2 =
s"""Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod\r\n tempor incididunt ut labore et dolore magna aliqua. Ut enim ad"""
- ///////////////////// 123456789012345678901234567890123456789012345678901234567890123456789012 3 4567890123456789012345678901234567890123456789012345678901234567890
- ///////////////////// 1 2 3 4 5 6 7 8
+ ///// 123456789012345678901234567890123456789012345678901234567890123456789012 3 4567890123456789012345678901234567890123456789012345678901234567890
+ ///// 1 2 3 4 5 6 7 8
val ipsumLorem2Unfolded =
s"""Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad"""
@@ -314,11 +442,14 @@ a few lines of pointless text like this.""".replace("\r\n", "\n").replace("\n",
val lineFoldLayer2Schema =
SchemaUtils.dfdlTestSchema(
- <xs:include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>,
+ <xs:include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+ <xs:import namespace="urn:org.apache.daffodil.layers.lineFolded"
+ schemaLocation="/org/apache/daffodil/layers/xsd/lineFoldedLayer.dfdl.xsd"/>,
<dfdl:format ref="tns:GeneralFormat"/>,
- <xs:element name="e1" dfdl:lengthKind="implicit">
+ <xs:element name="e1" dfdl:lengthKind="implicit"
+ xmlns:lf="urn:org.apache.daffodil.layers.lineFolded">
<xs:complexType>
- <xs:sequence dfdlx:layerTransform="lineFolded_IMF" dfdlx:layerLengthKind="implicit">
+ <xs:sequence dfdlx:layer="lf:lineFolded_IMF">
<xs:element name="s1" type="xs:string" dfdl:lengthKind="delimited"/>
</xs:sequence>
</xs:complexType>
@@ -346,8 +477,8 @@ a few lines of pointless text like this.""".replace("\r\n", "\n").replace("\n",
val ipsumLorem4 =
s"""Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod\r\n tempor incididunt\r\n"""
- ///////////////////// 123456789012345678901234567890123456789012345678901234567890123456789012 3 4567890123456789012345678901234567890123456789012345678901234567890
- ///////////////////// 1 2 3 4 5 6 7 8
+ ///// 123456789012345678901234567890123456789012345678901234567890123456789012 3 4567890123456789012345678901234567890123456789012345678901234567890
+ ///// 1 2 3 4 5 6 7 8
val ipsumLorem4Unfolded =
s"""Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt\n"""
@@ -365,11 +496,14 @@ a few lines of pointless text like this.""".replace("\r\n", "\n").replace("\n",
*/
val lineFoldLayer3Schema =
SchemaUtils.dfdlTestSchema(
- <xs:include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>,
+ <xs:include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+ <xs:import namespace="urn:org.apache.daffodil.layers.lineFolded"
+ schemaLocation="/org/apache/daffodil/layers/xsd/lineFoldedLayer.dfdl.xsd"/>,
<dfdl:format ref="tns:GeneralFormat"/>,
- <xs:element name="e1" dfdl:lengthKind="explicit" dfdl:length="100">
+ <xs:element name="e1" dfdl:lengthKind="explicit" dfdl:length="100"
+ xmlns:lf="urn:org.apache.daffodil.layers.lineFolded">
<xs:complexType>
- <xs:sequence dfdlx:layerTransform="lineFolded_IMF" dfdlx:layerLengthKind="implicit">
+ <xs:sequence dfdlx:layer="lf:lineFolded_IMF">
<xs:element name="s1" type="xs:string" dfdl:lengthKind="delimited"/>
</xs:sequence>
</xs:complexType>
@@ -379,8 +513,8 @@ a few lines of pointless text like this.""".replace("\r\n", "\n").replace("\n",
val ipsumLorem5 =
s"""Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod\r\n tempor incididunt ut labore et dolore magna aliqua."""
- ///////////////////// 123456789012345678901234567890123456789012345678901234567890123456789012 3 4567890123456789012345678901234567890123456789012345678901234567890
- ///////////////////// 1 2 3 4 5 6 7 8 9 A
+ ///// 123456789012345678901234567890123456789012345678901234567890123456789012 3 4567890123456789012345678901234567890123456789012345678901234567890
+ ///// 1 2 3 4 5 6 7 8 9 A
val ipsumLorem5Unfolded =
s"""Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labor"""
@@ -395,8 +529,8 @@ a few lines of pointless text like this.""".replace("\r\n", "\n").replace("\n",
val ipsumLorem6 =
s"""Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod\r\n tempor incididunt ut labor"""
- ///////////////////// 123456789012345678901234567890123456789012345678901234567890123456789012 3 4567890123456789012345678901234567890123456789012345678901234567890
- ///////////////////// 1 2 3 4 5 6 7 8 9 A
+ ///// 123456789012345678901234567890123456789012345678901234567890123456789012 3 4567890123456789012345678901 2 34567890123456789012345678901234567890
+ ///// 1 2 3 4 5 6 7 8 9 A
val ipsumLorem6Unfolded =
s"""Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labor"""
@@ -417,21 +551,32 @@ a few lines of pointless text like this.""".replace("\r\n", "\n").replace("\n",
val le32BitSchema =
SchemaUtils.dfdlTestSchema(
- <xs:include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>,
+ <xs:include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+ <xs:import namespace="urn:org.apache.daffodil.layers.fixedLength"
+ schemaLocation="/org/apache/daffodil/layers/xsd/fixedLengthLayer.dfdl.xsd"/>
+ <xs:import namespace="urn:org.apache.daffodil.layers.byteSwap"
+ schemaLocation="/org/apache/daffodil/layers/xsd/byteSwapLayer.dfdl.xsd"/>,
<dfdl:format ref="tns:GeneralFormat" bitOrder="leastSignificantBitFirst"
byteOrder="littleEndian" alignmentUnits="bits" alignment="1"
lengthKind="explicit" lengthUnits="bits"/>,
- <xs:element name="e1" dfdl:lengthKind="implicit">
+ <xs:element name="e1" dfdl:lengthKind="implicit"
+ xmlns:fl="urn:org.apache.daffodil.layers.fixedLength"
+ xmlns:bs="urn:org.apache.daffodil.layers.byteSwap">
<xs:complexType>
<xs:sequence>
<xs:element name="s0" type="xs:hexBinary" dfdl:length="4" dfdl:byteOrder="bigEndian" dfdl:bitOrder="mostSignificantBitFirst"/>
<xs:element name="s1" type="xs:hexBinary" dfdl:length="4" dfdl:byteOrder="bigEndian" dfdl:bitOrder="mostSignificantBitFirst"/>
- <xs:sequence dfdlx:layerTransform="fourbyteswap" dfdlx:layerLengthKind="explicit" dfdlx:layerLengthUnits="bytes" dfdlx:layerLength="6">
- <xs:sequence>
- <xs:element name="s2" type="xs:hexBinary" dfdl:length="4"/>
- <xs:element name="s3" type="xs:hexBinary" dfdl:length="24"/>
- <xs:element name="s4" type="xs:hexBinary" dfdl:length="8"/>
- <xs:element name="s5" type="xs:hexBinary" dfdl:length="12"/>
+ <xs:sequence dfdlx:layer="fl:fixedLength">
+ <xs:annotation><xs:appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:newVariableInstance ref="fl:fixedLength" defaultValue="6"/>
+ </xs:appinfo></xs:annotation>
+ <xs:sequence dfdlx:layer="bs:fourbyteswap">
+ <xs:sequence>
+ <xs:element name="s2" type="xs:hexBinary" dfdl:length="4"/>
+ <xs:element name="s3" type="xs:hexBinary" dfdl:length="24"/>
+ <xs:element name="s4" type="xs:hexBinary" dfdl:length="8"/>
+ <xs:element name="s5" type="xs:hexBinary" dfdl:length="12"/>
+ </xs:sequence>
</xs:sequence>
</xs:sequence>
<xs:element name="s6" type="xs:hexBinary" dfdl:length="4" dfdl:byteOrder="bigEndian" dfdl:bitOrder="mostSignificantBitFirst"/>
diff --git a/daffodil-core/src/test/scala/org/apache/daffodil/core/util/FuzzData.scala b/daffodil-core/src/test/scala/org/apache/daffodil/core/util/FuzzData.scala
new file mode 100644
index 000000000..d80053015
--- /dev/null
+++ b/daffodil-core/src/test/scala/org/apache/daffodil/core/util/FuzzData.scala
@@ -0,0 +1,235 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.daffodil.core.util
+
+import java.io.ByteArrayInputStream
+import scala.collection.mutable
+import scala.util.Random
+import scala.xml.Elem
+import scala.xml.Node
+
+import org.apache.daffodil.core.util.TestUtils.getAMessage
+import org.apache.daffodil.lib.exceptions.Assert
+import org.apache.daffodil.lib.xml.XMLUtils.XMLDifferenceException
+import org.apache.daffodil.runtime1.processors.parsers.ParseError
+
+import org.junit.Assert.fail
+
+/**
+ * Utility base class to support fuzz testing. Given a piece of data,
+ * it fuzzes it (without modifying the original data) each time next() is
+ * called.
+ *
+ * The kind of fuzzing, what region within the data, how much data (run length),
+ * and what it is overwritten with, is determined by the parameters to
+ * the constructor, and the implementations of the runLength and fuzzOneByte
+ * methods.
+ * @param data a byte array of data to created fuzzed instances of.
+ * @param offset where in the data to start the fuzzed region
+ * @param endOffset where, measured back from the end, we want to protect from fuzzing
+ * @param seed random number generator seed. Choosing different values for this
+ * will produce different fuzzed data, but it should be perfectly
+ * repeatable. Defaults to a specific value so that tests will be
+ * repeatable.
+ */
+abstract class FuzzData(
+ val data: Array[Byte],
+ offset: Int, // allows you not to clobber a prefix of the data
+ endOffset: Int, // number of bytes at the end to not clobber
+ seed: Int = 1, // specify to get varied, but repeatable results
+) extends Iterator[Array[Byte]] {
+
+ protected val r = new Random(seed)
+ protected val dataLen = data.length
+ private val originalData = data.clone()
+
+ /**
+ * Override this to generate a random run length, or to specify
+ * a specific run length. Must be > 0.
+ * @return an integer greater than 0, and less than than
+ * the size of the region to fuzz.
+ */
+ def runLength: Int
+
+ override def hasNext() = true // we can keep fuzzing forever
+
+ override def next(): Array[Byte] = {
+ val data = originalData.clone()
+ val limit = dataLen - endOffset
+ val regionLen = limit - offset
+ val runLen = runLength
+ val regionRunLocations = regionLen - runLen
+ val startOfRun = r.nextInt(regionRunLocations)
+ (startOfRun to (startOfRun + runLen)).foreach { i =>
+ fuzzOneByte(data, i)
+ }
+ data
+ }
+
+ /**
+ * Modifies one byte of the data array at the given position
+ * @param data the byte array to modify
+ * @param i zero-based position of the byte to modify
+ */
+ def fuzzOneByte(data: Array[Byte], i: Int): Unit
+}
+
+class FuzzZeroBits(data: Array[Byte], offset: Int, endOffset: Int)
+ extends FuzzData(data, offset, endOffset) {
+
+ override def runLength: Int = r.nextInt(dataLen - (offset + endOffset) - 1)
+
+ override def fuzzOneByte(data: Array[Byte], i: Int): Unit = data(i) = 0
+}
+
+class FuzzOneBits(data: Array[Byte], offset: Int, endOffset: Int)
+ extends FuzzData(data, offset, endOffset) {
+
+ override def runLength: Int = r.nextInt(dataLen - (offset + endOffset) - 1)
+
+ override def fuzzOneByte(data: Array[Byte], i: Int): Unit = data(i) = 0xff.toByte
+}
+
+class FuzzRandomSingleByte(data: Array[Byte], offset: Int, endOffset: Int)
+ extends FuzzData(data, offset, endOffset) {
+
+ override def runLength: Int = 1
+
+ override def fuzzOneByte(data: Array[Byte], i: Int): Unit = {
+ var current: Byte = data(i)
+ var newValue: Byte = current
+ // make sure they are different so we're definitely changing a byte value
+ while (current == newValue) {
+ newValue = ((current ^ r.nextInt(256)) & 0xff).toByte
+ }
+ data(i) = newValue
+ }
+}
+
+class FuzzRandomByteRuns(data: Array[Byte], offset: Int, endOffset: Int)
+ extends FuzzRandomSingleByte(data, offset, endOffset) {
+ override def runLength = r.nextInt(dataLen - (offset + endOffset) - 1)
+}
+
+/**
+ * Parses fuzzed data with a DFDL schema. Gathers unexpected throw classes
+ * (things thrown other than parse errors), and also gathers unexpected
+ * successes where the fuzzed data did not cause the parse to fail.
+ * @param sch - scala.xml.Node which is the schema to test
+ * @param data - data which will be fuzzed to create many variations
+ * @param expected - expected XML if the data parses successfully
+ * @param fuzzer - a FuzzData object
+ */
+class FuzzParseTester(
+ sch: Elem,
+ data: Array[Byte],
+ expected: Elem,
+ fuzzer: FuzzData,
+) {
+
+ private val p = TestUtils.compileSchema(sch)
+
+ val unexpectedThrowClasses = new mutable.HashSet[Class[_ <: Throwable]]()
+
+ val okParses = new mutable.HashSet[(Node, Array[Byte], XMLDifferenceException)]
+
+ protected def handleParseError(p: ParseError, testData: Array[Byte], i: Int) = {
+ // do nothing by default
+ }
+
+ protected def handleThrow(t: Throwable): Unit = {
+ unexpectedThrowClasses += t.getClass
+ }
+
+ protected def handleUnexpectedSuccess(
+ actual: Node,
+ testData: Array[Byte],
+ diff: XMLDifferenceException,
+ ): Unit = {
+ // we didn't get a throw from this fuzzed data
+ // but the XML output was not what was expected either
+ // save up the first N such.
+ if (okParses.size <= 10)
+ okParses += ((actual, testData, diff)) // extra parens because of varargs stuff ??
+ }
+
+ def run(nTrials: Int): Unit = {
+ Assert.usage(nTrials >= 1)
+ (1 to nTrials).foreach { i =>
+ val testData = fuzzer.next()
+ val bais = new ByteArrayInputStream(testData)
+ val (pr, node) =
+ try {
+ TestUtils.runDataProcessorOnInputStream(p, bais, areTracing = false)
+ } catch {
+ case p: ParseError => {
+ handleParseError(p, testData, i)
+ (null, null)
+ }
+ case t: Throwable => {
+ handleThrow(t)
+ (null, null)
+ }
+ }
+ if (pr ne null) {
+ try {
+ TestUtils.assertEqualsXMLElements(expected, node)
+ } catch {
+ case e: XMLDifferenceException =>
+ handleUnexpectedSuccess(node, testData, e)
+ }
+ }
+ }
+ }
+
+}
+
+class LayerParseTester(
+ sch: Elem,
+ data: Array[Byte],
+ expected: Elem,
+ fuzzer: FuzzData,
+ excludes: Seq[String],
+) extends FuzzParseTester(sch, data, expected, fuzzer) {
+
+ var shouldFail = false
+
+ override def handleParseError(p: ParseError, data: Array[Byte], nth: Int): Unit = {
+ val msg = getAMessage(p)
+ if (excludes.exists { msg.contains(_) }) {
+ // we know about this one
+ } else {
+ // new parse error type
+ println(s"trial $nth produced $p")
+ shouldFail = true
+ }
+ }
+
+ override def run(nTrials: Int) = {
+ super.run(nTrials)
+ if (unexpectedThrowClasses.size > 0) {
+ unexpectedThrowClasses.foreach { c =>
+ println(s"Class ${c.getSimpleName} was thrown")
+ shouldFail = true
+ }
+ }
+ if (okParses.size > 0)
+ println(s"Parse succeeded on fuzzed data ${okParses.size} times.")
+ if (shouldFail)
+ fail("errors were found")
+ }
+}
diff --git a/daffodil-core/src/test/scala/org/apache/daffodil/core/util/TestUtils.scala b/daffodil-core/src/test/scala/org/apache/daffodil/core/util/TestUtils.scala
index 2c205fe2a..358705754 100644
--- a/daffodil-core/src/test/scala/org/apache/daffodil/core/util/TestUtils.scala
+++ b/daffodil-core/src/test/scala/org/apache/daffodil/core/util/TestUtils.scala
@@ -319,6 +319,25 @@ object TestUtils {
val theTry = Timer.getResult(compileAndSave(compiler, schemaSource, nullChannel))
theTry.get
}
+
+ /**
+ * Gets some sort of a message from a throwable. Includes messages from all
+ * classes in the cause chain. If there is no message, just uses the class name.
+ * @param cause
+ * @return
+ */
+ def getAMessage(cause: Throwable): String = {
+ val n1 = Misc.getNameFromClass(cause)
+ val m = cause.getMessage
+ val c = cause.getCause
+ val s = (m, c) match {
+ case (null, null) => Misc.getNameFromClass(cause)
+ case (null, c) => getAMessage(c)
+ case (m, null) => m
+ case (m, c) => m + " " + getAMessage(c)
+ }
+ n1 + s"($s)"
+ }
}
/**
diff --git a/daffodil-io/src/main/resources/META-INF/services/org.apache.daffodil.io.processors.charset.BitsCharsetDefinition b/daffodil-io/src/main/resources/META-INF/services/org.apache.daffodil.io.processors.charset.BitsCharsetDefinition
index 759b2b8ab..b8cd30ad5 100644
--- a/daffodil-io/src/main/resources/META-INF/services/org.apache.daffodil.io.processors.charset.BitsCharsetDefinition
+++ b/daffodil-io/src/main/resources/META-INF/services/org.apache.daffodil.io.processors.charset.BitsCharsetDefinition
@@ -24,7 +24,6 @@ org.apache.daffodil.io.processors.charset.BitsCharset5BitPackedLSBFDefinition
org.apache.daffodil.io.processors.charset.BitsCharset6BitDFI264DUI001Definition
org.apache.daffodil.io.processors.charset.BitsCharset6BitDFI311DUI002Definition
org.apache.daffodil.io.processors.charset.BitsCharset6BitICAOAircraftIDDefinition
-org.apache.daffodil.io.processors.charset.BitsCharsetAISPayloadArmoringDefinition
org.apache.daffodil.io.processors.charset.BitsCharsetBase4LSBFDefinition
org.apache.daffodil.io.processors.charset.BitsCharsetBase4MSBFDefinition
org.apache.daffodil.io.processors.charset.BitsCharsetBinaryLSBFDefinition
diff --git a/daffodil-io/src/main/scala/org/apache/daffodil/io/BoundaryMarkStreams.scala b/daffodil-io/src/main/scala/org/apache/daffodil/io/BoundaryMarkStreams.scala
new file mode 100644
index 000000000..21209f26d
--- /dev/null
+++ b/daffodil-io/src/main/scala/org/apache/daffodil/io/BoundaryMarkStreams.scala
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.daffodil.io
+
+import java.io.FilterOutputStream
+import java.io.InputStream
+import java.nio.charset.Charset
+import java.nio.charset.StandardCharsets
+import java.util.regex.Pattern
+
+import org.apache.daffodil.lib.exceptions.Assert
+
+/**
+ * Can be used with any InputStream to restrict what is
+ * read from it to stop before a boundary mark string.
+ *
+ * The boundary mark string is exactly that, a string of characters. Not a
+ * regex, nor anything involving DFDL Character Entities or Character Class
+ * Entities. (No %WSP; no %NL; )
+ *
+ * This can be used to forcibly stop consumption of data from a stream at
+ * a length obtained from a delimiter.
+ *
+ * The boundary mark string is consumed from the underlying stream (if found), and
+ * the underlying stream is left positioned at the byte after the boundary mark
+ * string.
+ *
+ * Thread safety: This is inherently stateful - so not thread safe to use
+ * this object from more than one thread.
+ */
+class BoundaryMarkLimitingInputStream(
+ inputStream: InputStream,
+ boundaryMark: String,
+ charset: Charset,
+ targetChunkSize: Int = 32 * 1024,
+) extends InputStream {
+
+ Assert.usage(targetChunkSize >= 1)
+ Assert.usage(boundaryMark.nonEmpty)
+
+ private lazy val boundaryMarkIn8859 =
+ new String(boundaryMark.getBytes(charset), StandardCharsets.ISO_8859_1)
+
+ private lazy val quotedBoundaryMark =
+ Pattern.quote(boundaryMarkIn8859) // in case pattern has non-regex-safe characters in it
+
+ private lazy val delegateStream = new RegexLimitingInputStream(
+ inputStream,
+ quotedBoundaryMark,
+ boundaryMarkIn8859,
+ charset,
+ targetChunkSize,
+ )
+
+ override def read(): Int = delegateStream.read()
+
+}
+
+/**
+ * A FilterOutputStream that inserts a boundary mark at the end of the layer's output stream.
+ *
+ * @param jos The underlying Java OutputStream.
+ * @param boundaryMark The boundary mark to be inserted at the end of the stream.
+ * @param charset The character set used to encode the boundary mark.
+ */
+class BoundaryMarkInsertingJavaOutputStream(
+ jos: java.io.OutputStream,
+ boundaryMark: String,
+ charset: Charset,
+) extends FilterOutputStream(jos) {
+
+ private var closed = false
+
+ private lazy val boundaryMarkBytes = boundaryMark.getBytes(charset)
+
+ override def close(): Unit = {
+ if (!closed) {
+ jos.write(boundaryMarkBytes)
+ jos.flush()
+ jos.close()
+ closed = true
+ }
+ }
+
+}
diff --git a/daffodil-io/src/main/scala/org/apache/daffodil/io/DataInputStream.scala b/daffodil-io/src/main/scala/org/apache/daffodil/io/DataInputStream.scala
index 9f7f8b87e..413874369 100644
--- a/daffodil-io/src/main/scala/org/apache/daffodil/io/DataInputStream.scala
+++ b/daffodil-io/src/main/scala/org/apache/daffodil/io/DataInputStream.scala
@@ -550,4 +550,10 @@ trait DataInputStream extends DataStreamCommon {
*/
final def withBitLengthLimit(lengthLimitInBits: Long)(body: => Unit): Boolean =
macro IOMacros.withBitLengthLimitMacroForInput
+
+ /**
+ * Closes any underlying I/O streams/channels that are part of the implementation
+ * of this and frees related resources.
+ */
+ def close(): Unit
}
diff --git a/daffodil-io/src/main/scala/org/apache/daffodil/io/DirectOrBufferedDataOutputStream.scala b/daffodil-io/src/main/scala/org/apache/daffodil/io/DirectOrBufferedDataOutputStream.scala
index 566aab17f..549ab17f9 100644
--- a/daffodil-io/src/main/scala/org/apache/daffodil/io/DirectOrBufferedDataOutputStream.scala
+++ b/daffodil-io/src/main/scala/org/apache/daffodil/io/DirectOrBufferedDataOutputStream.scala
@@ -18,7 +18,9 @@
package org.apache.daffodil.io
import java.io.File
+import java.io.OutputStream
import java.nio.file.Path
+import scala.annotation.tailrec
import org.apache.daffodil.lib.equality._
import org.apache.daffodil.lib.exceptions.Assert
@@ -36,11 +38,11 @@ import passera.unsigned.ULong
* This simple extension just gives us a public method for access to the underlying byte array.
* That way we don't have to make a copy just to access the bytes.
*/
-class ByteArrayOutputStreamWithGetBuf() extends java.io.ByteArrayOutputStream {
+class ByteArrayOutputStreamWithGetBuf extends java.io.ByteArrayOutputStream {
def setBuf(bufArg: Array[Byte]): Unit = {
buf = bufArg
}
- def getBuf = buf
+ def getBuf: Array[Byte] = buf
}
private[io] class ByteArrayOrFileOutputStream(
@@ -53,7 +55,7 @@ private[io] class ByteArrayOrFileOutputStream(
// This will be the file object that can represent either a generated
// temporary file or an existing blob file
- var maybeFile: Maybe[File] = {
+ private var maybeFile: Maybe[File] = {
if (maybeExistingFile.isDefined)
Maybe(maybeExistingFile.get.toFile)
else
@@ -63,7 +65,7 @@ private[io] class ByteArrayOrFileOutputStream(
// Keep track of whether or not this is a temporary file. If it is an existing
// file, i.e. a blob file, we do not want to delete it as that is handled
// elsewhere.
- lazy val isTempFile = (!maybeExistingFile.isDefined && isFile)
+ lazy val isTempFile: Boolean = !maybeExistingFile.isDefined && isFile
// This is only used to determine if we need to switch to a file based output
// stream
@@ -90,12 +92,12 @@ private[io] class ByteArrayOrFileOutputStream(
throw new FileIOException(
"Unable to create temporary file in %s: %s".format(
tempDirPath.getPath,
- e.getMessage(),
+ e.getMessage,
),
)
}
val newStream: java.io.OutputStream = new java.io.FileOutputStream(maybeFile.get)
- stream.flush
+ stream.flush()
stream.asInstanceOf[ByteArrayOutputStreamWithGetBuf].writeTo(newStream)
stream = newStream
isFile = true
@@ -103,40 +105,40 @@ private[io] class ByteArrayOrFileOutputStream(
nBytes += lengthInBytes
}
- override def write(b: Array[Byte]) = {
+ override def write(b: Array[Byte]): Unit = {
checkBuffer(b.length)
stream.write(b)
}
- override def write(b: Array[Byte], off: Int, len: Int) = {
+ override def write(b: Array[Byte], off: Int, len: Int): Unit = {
checkBuffer(len)
stream.write(b, off, len)
}
- override def write(b: Int) = {
+ override def write(b: Int): Unit = {
checkBuffer(1)
stream.write(b)
}
- override def close() = stream.close()
+ override def close(): Unit = stream.close()
- def getBuf() = {
+ def getBuf: Array[Byte] = {
Assert.usage(!isFile, "Attempted to call getBuf on FileOutputStream")
stream.asInstanceOf[ByteArrayOutputStreamWithGetBuf].getBuf
}
def getFile: File = maybeFile.get
- def hexDump: String = {
+ private def hexDump: String = {
if (isFile)
maybeFile.get.toString
else
- (0 to (nBytes.toInt - 1))
- .map { i => "%2x".format(getBuf()(i).toInt & 0xff) }
+ (0 until nBytes.toInt)
+ .map { i => "%2x".format(getBuf(i).toInt & 0xff) }
.mkString(".")
}
- override def toString = hexDump
+ override def toString: String = hexDump
}
/**
@@ -162,7 +164,7 @@ private[io] class ByteArrayOrFileOutputStream(
* buffer. When direct, all output goes into a "real" DataOutputStream.
*
* The isLayer parameter defines whether or not this instance originated from a
- * layer or not. This is important to specify because this class is reponsible
+ * layer or not. This is important to specify because this class is responsible
* for closing the associated Java OutputStream, ultimately being written to
* the underlying underlying DataOutputStream. However, if the DataOutputStream
* is not related to a layer, that means the associated Java OutputStream came
@@ -192,7 +194,7 @@ class DirectOrBufferedDataOutputStream private[io] (
val maybeExistingFile: Maybe[Path],
) extends DataOutputStreamImplMixin {
- type ThisType = DirectOrBufferedDataOutputStream
+ private type ThisType = DirectOrBufferedDataOutputStream
override def putULong(
unsignedLong: ULong,
@@ -203,7 +205,11 @@ class DirectOrBufferedDataOutputStream private[io] (
res
}
- override def putLong(signedLong: Long, bitLengthFrom1To64: Int, finfo: FormatInfo) = {
+ override def putLong(
+ signedLong: Long,
+ bitLengthFrom1To64: Int,
+ finfo: FormatInfo,
+ ): Boolean = {
val res = putLongChecked(signedLong.longValue, bitLengthFrom1To64, finfo)
res
}
@@ -233,12 +239,12 @@ class DirectOrBufferedDataOutputStream private[io] (
* Two of these are equal if they are eq.
* This matters because we compare them to see if we are making forward progress
*/
- override def equals(other: Any) = AnyRef.equals(other)
+ override def equals(other: Any): Boolean = AnyRef.equals(other)
- override def hashCode() = AnyRef.hashCode()
+ override def hashCode(): Int = AnyRef.hashCode()
- override def toString = {
- lazy val buf = bufferingJOS.getBuf()
+ override def toString: String = {
+ lazy val buf = bufferingJOS.getBuf
lazy val max16ByteArray = buf.slice(0, 16)
lazy val upTo16BytesInHex = Misc.bytes2Hex(max16ByteArray)
val toDisplay = "DOS(id=" + id + ", " + dosState +
@@ -281,14 +287,14 @@ class DirectOrBufferedDataOutputStream private[io] (
*
* If reused, this must be reset.
*/
- protected val bufferingJOS =
+ private val bufferingJOS =
new ByteArrayOrFileOutputStream(maxBufferSizeInBytes, tempDirPath, maybeExistingFile)
/**
* Switched to point a either the buffering or direct java output stream in order
* to change modes from buffering to direct (and back if these objects get reused.)
*/
- protected var _javaOutputStream: java.io.OutputStream = bufferingJOS
+ private var _javaOutputStream: java.io.OutputStream = bufferingJOS
final def isBuffering: Boolean = {
val res = getJavaOutputStream()._eq_(bufferingJOS)
@@ -303,7 +309,7 @@ class DirectOrBufferedDataOutputStream private[io] (
) // these are born buffering, and evolve into direct.
}
- override def getJavaOutputStream() = {
+ override def getJavaOutputStream(): OutputStream = {
Assert.usage(_javaOutputStream ne null)
_javaOutputStream
}
@@ -317,7 +323,8 @@ class DirectOrBufferedDataOutputStream private[io] (
override def maybeNextInChain: Maybe[DataOutputStream] = _following
- def lastInChain: DirectOrBufferedDataOutputStream =
+ @tailrec
+ private def lastInChain: DirectOrBufferedDataOutputStream =
if (_following.isEmpty) this
else _following.get.lastInChain
@@ -361,7 +368,7 @@ class DirectOrBufferedDataOutputStream private[io] (
finfo: FormatInfo,
): DirectOrBufferedDataOutputStream = {
- // create a special buffered blob data outputstream and split the current
+ // create a special buffered blob data output stream and split the current
// DOS to it. When this normal DOS is finished, the blob DOS will handle
// delivering the blob data to it without loading the whole blob into
// memory all at once.
@@ -423,7 +430,7 @@ class DirectOrBufferedDataOutputStream private[io] (
Assert.usage(isBuffering)
Assert.usage(oldDirectDOS.isDirect)
- setJavaOutputStream(oldDirectDOS.getJavaOutputStream)
+ setJavaOutputStream(oldDirectDOS.getJavaOutputStream())
Assert.invariant(isDirect)
this.setAbsStartingBitPos0b(ULong(0))
@@ -485,7 +492,7 @@ class DirectOrBufferedDataOutputStream private[io] (
first.isFinished // continue until AFTER we merge forward into the first non-finished successor
Assert.invariant(first.isBuffering)
- Logger.log.debug(s"merging direct DOS ${directStream} into DOS ${first}")
+ Logger.log.debug(s"merging direct DOS $directStream into DOS $first")
val dabp = directStream.maybeAbsBitPos0b.getULong
if (first.maybeAbsStartingBitPos0b.isEmpty) {
first.setAbsStartingBitPos0b(dabp)
@@ -503,7 +510,7 @@ class DirectOrBufferedDataOutputStream private[io] (
first.convertToDirect(directStream) // first is now the direct stream
directStream.setDOSState(Uninitialized) // old direct stream is now dead
directStream = first // long live the new direct stream!
- Logger.log.debug(s"New direct DOS ${directStream}")
+ Logger.log.debug(s"New direct DOS $directStream")
}
if (directStream._following.isDefined) {
@@ -537,7 +544,8 @@ class DirectOrBufferedDataOutputStream private[io] (
// OutputStream from a user and they are responsible for closing it.
directStream.getJavaOutputStream().flush()
if (directStream.isLayer) {
- directStream.getJavaOutputStream().close()
+ val jos: OutputStream = directStream.getJavaOutputStream()
+ jos.close()
}
directStream.setDOSState(Uninitialized) // not just finished. We're dead now.
} else {
@@ -615,7 +623,7 @@ class DirectOrBufferedDataOutputStream private[io] (
if (pmabp.isDefined) {
val pabp = pmabp.getULong
this.setAbsStartingBitPos0b(pabp)
- Logger.log.debug(s"for ${this} propagated absolute starting bit pos ${pabp}")
+ Logger.log.debug(s"for $this propagated absolute starting bit pos $pabp")
super.maybeAbsBitPos0b // will get the right value this time.
} else {
// prior doesn't have an abs bit pos.
@@ -673,15 +681,7 @@ class DirectOrBufferedDataOutputStream private[io] (
bits = (bits << shift1) >>> shift1
nBitsRemaining = bitLengthFrom1To64 - nBitsOfFragToBeFilled
- if (nFragBitsAfter == 8) {
- // we filled the entire frag byte. Write it out, then zero it
- realStream.write(newFragByte.toByte)
- setFragmentLastByte(0, 0)
- } else {
- // we did not fill up the frag byte. We added bits to it (at least 1), but
- // it's not filled up yet.
- setFragmentLastByte(newFragByte.toInt, nFragBitsAfter)
- }
+ handleFragmentLastByte(nFragBitsAfter, newFragByte, realStream)
}
// at this point we have bits and nBitsRemaining
@@ -715,6 +715,22 @@ class DirectOrBufferedDataOutputStream private[io] (
}
}
+ private def handleFragmentLastByte(
+ nFragBitsAfter: Int,
+ newFragByte: Int,
+ realStream: OutputStream,
+ ): Unit = {
+ if (nFragBitsAfter == 8) {
+ // we filled the entire frag byte. Write it out, then zero it
+ realStream.write(newFragByte.toByte)
+ setFragmentLastByte(0, 0)
+ } else {
+ // we did not fill up the frag byte. We added bits to it (at least 1), but
+ // it's not filled up yet.
+ setFragmentLastByte(newFragByte, nFragBitsAfter)
+ }
+ }
+
final override protected def putLong_LE_MSBFirst(
signedLong: Long,
bitLengthFrom1To64: Int,
@@ -733,7 +749,7 @@ class DirectOrBufferedDataOutputStream private[io] (
// that byte, since we're storing data MSBF.
//
val nWholeBytesAtStart = bitLengthFrom1To64 / 8
- val nUsedBitsLastByte = (bitLengthFrom1To64 % 8)
+ val nUsedBitsLastByte = bitLengthFrom1To64 % 8
val nUnusedBitsLastByte = if (nUsedBitsLastByte == 0) 0 else 8 - nUsedBitsLastByte
val indexOfLastByteLE = nWholeBytesAtStart - (if (nUnusedBitsLastByte > 0) 0 else 1)
@@ -772,21 +788,12 @@ class DirectOrBufferedDataOutputStream private[io] (
val newFragBitsMask = (0x80.toByte >> (nBitsOfFragToBeFilled - 1)) & 0xff
val LSByte = unionByteBuffer.get(0)
val bitsToGoIntoFragInPosition =
- (((LSByte & newFragBitsMask) & 0xff) >>> fragmentLastByteLimit).toInt
+ ((LSByte & newFragBitsMask) & 0xff) >>> fragmentLastByteLimit
val newFragByte =
Bits.asUnsignedByte((fragmentLastByte | bitsToGoIntoFragInPosition).toByte)
- Assert.invariant(newFragByte <= 255 && newFragByte >= 0)
- if (nFragBitsAfter == 8) {
- // we filled the entire frag byte. Write it out, then zero it
- realStream.write(newFragByte.toByte)
- setFragmentLastByte(0, 0)
- } else {
- // we did not fill up the frag byte. We added bits to it (at least 1), but
- // it's not filled up yet.
- setFragmentLastByte(newFragByte, nFragBitsAfter)
- }
+ handleFragmentLastByte(nFragBitsAfter, newFragByte, realStream)
//
// Now we have to remove the bits that went into the
@@ -867,15 +874,7 @@ class DirectOrBufferedDataOutputStream private[io] (
bits = bits >>> nBitsOfFragToBeFilled
nBitsRemaining = bitLengthFrom1To64 - nBitsOfFragToBeFilled
- if (nFragBitsAfter == 8) {
- // we filled the entire frag byte. Write it out, then zero it
- realStream.write(newFragByte.toByte)
- setFragmentLastByte(0, 0)
- } else {
- // we did not fill up the frag byte. We added bits to it (at least 1), but
- // it's not filled up yet.
- setFragmentLastByte(newFragByte, nFragBitsAfter)
- }
+ handleFragmentLastByte(nFragBitsAfter, newFragByte, realStream)
}
// at this point we have bits and nBitsRemaining
@@ -888,7 +887,7 @@ class DirectOrBufferedDataOutputStream private[io] (
Assert.invariant(fragmentLastByteLimit == 0) // there is no frag byte.
val nWholeBytes = nBitsRemaining / 8
val nFragBits = nBitsRemaining % 8
- val fragUsedBitsMask = ((1 << nFragBits) - 1)
+ val fragUsedBitsMask = (1 << nFragBits) - 1
var shiftedBits = bits
@@ -919,12 +918,12 @@ class DirectOrBufferedDataOutputStream private[io] (
val bufferNBits =
this.relBitPos0b // don't have to subtract a starting offset. It's always zero in buffered case.
val bufOS = this.bufferingJOS
- val nBytes = (bufferNBits / 8)
+ val nBytes = bufferNBits / 8
val nFragBits = this.fragmentLastByteLimit
val wholeBytesWritten = {
if (bufOS.isFile) {
- bufOS.close
+ bufOS.close()
val nBitsPut =
try {
directDOS.putFile(bufOS.getFile.toPath, bufferNBits.toLong, chunkSizeInBytes, finfo)
@@ -1009,7 +1008,7 @@ class DirectOrBufferedDataOutputStream private[io] (
class BitOrderChangeException(directDOS: DirectOrBufferedDataOutputStream, finfo: FormatInfo)
extends Exception {
- override def getMessage() = {
+ override def getMessage: String = {
"Data output stream %s with bitOrder '%s' which is not on a byte boundary (%s bits past last byte boundary), cannot be populated with bitOrder '%s'."
.format(
directDOS,
@@ -1024,7 +1023,7 @@ class FileIOException(message: String) extends Exception(message)
object DirectOrBufferedDataOutputStream {
- var nextLayerID = 0
+ private var nextLayerID = 0
/**
* This is over here to be sure it isn't operating on other members
@@ -1096,7 +1095,7 @@ object DirectOrBufferedDataOutputStream {
maxBufferSizeInBytes: Long,
tempDirPath: File,
maybeExistingFile: Maybe[Path] = Maybe.Nope,
- ) = {
+ ): DirectOrBufferedDataOutputStream = {
val dbdos = new DirectOrBufferedDataOutputStream(
creator,
isLayer,
diff --git a/daffodil-io/src/main/scala/org/apache/daffodil/io/InputSource.scala b/daffodil-io/src/main/scala/org/apache/daffodil/io/InputSource.scala
index 141049ecd..d13e4079e 100644
--- a/daffodil-io/src/main/scala/org/apache/daffodil/io/InputSource.scala
+++ b/daffodil-io/src/main/scala/org/apache/daffodil/io/InputSource.scala
@@ -187,6 +187,11 @@ abstract class InputSource {
*/
def compact(): Unit
+ /**
+ * Release resources. Cascades the close call to any underlying input streams.
+ */
+ def close(): Unit
+
private var _debugging = false
protected var _isValid = true
@@ -221,6 +226,8 @@ class BucketingInputSource(
val bytes = new Array[Byte](bucketSize)
}
+ override def close(): Unit = inputStream.close()
+
/**
* The maximum number of buckets that can be cached in the buckets array. Note
* that because we periodically remove buckets from the buckets array by
@@ -614,4 +621,8 @@ class ByteBufferInputSource(byteBuffer: ByteBuffer) extends InputSource {
* For a byte buffer, this is always true.
*/
override def hasReachedEndOfData = true
+
+ override def close(): Unit = {
+ // do nothing. No resources to release.
+ }
}
diff --git a/daffodil-io/src/main/scala/org/apache/daffodil/io/InputSourceDataInputStream.scala b/daffodil-io/src/main/scala/org/apache/daffodil/io/InputSourceDataInputStream.scala
index e745d348f..b1e2c396b 100644
--- a/daffodil-io/src/main/scala/org/apache/daffodil/io/InputSourceDataInputStream.scala
+++ b/daffodil-io/src/main/scala/org/apache/daffodil/io/InputSourceDataInputStream.scala
@@ -94,7 +94,8 @@ private[io] class MarkPool() extends Pool[MarkState] {
* Underlying representation is an InputSource containing all input data.
*/
final class InputSourceDataInputStream private (val inputSource: InputSource)
- extends DataInputStreamImplMixin {
+ extends DataInputStreamImplMixin
+ with java.io.Closeable {
import DataInputStream._
@@ -111,6 +112,11 @@ final class InputSourceDataInputStream private (val inputSource: InputSource)
val markStack = new MStackOf[MarkState]
val markPool = new MarkPool()
+ /**
+ * Creators of this class are responsible to call close when done.
+ */
+ override def close(): Unit = inputSource.close()
+
@inline
override final def bitPos0b: Long = cst.bitPos0b
diff --git a/daffodil-io/src/main/scala/org/apache/daffodil/io/LimitingJavaIinputStreams.scala b/daffodil-io/src/main/scala/org/apache/daffodil/io/RegexLimitingInputStream.scala
similarity index 66%
rename from daffodil-io/src/main/scala/org/apache/daffodil/io/LimitingJavaIinputStreams.scala
rename to daffodil-io/src/main/scala/org/apache/daffodil/io/RegexLimitingInputStream.scala
index f007fc128..19dce083e 100644
--- a/daffodil-io/src/main/scala/org/apache/daffodil/io/LimitingJavaIinputStreams.scala
+++ b/daffodil-io/src/main/scala/org/apache/daffodil/io/RegexLimitingInputStream.scala
@@ -17,7 +17,6 @@
package org.apache.daffodil.io
-import java.io.FilterInputStream
import java.io.InputStream
import java.io.InputStreamReader
import java.nio.charset.Charset
@@ -27,109 +26,6 @@ import java.util.regex.Pattern
import org.apache.daffodil.lib.exceptions.Assert
-/**
- * This class can be used with any InputStream to restrict what is
- * read from it to N bytes.
- *
- * This can be used to forcibly stop consumption of data from a stream at
- * a length obtained explicitly.
- *
- * Thread safety: This is inherently stateful - so not thread safe to use
- * this object from more than one thread.
- */
-class ExplicitLengthLimitingStream(in: InputStream, limit: Long) extends FilterInputStream(in) {
-
- private var numRemaining = limit
-
- override def read(buf: Array[Byte], off: Int, len: Int): Int = {
- Assert.invariant(numRemaining >= 0)
- if (numRemaining == 0) -1
- else if (len == 0) 0
- else {
- val requestSize = math.min(numRemaining, len).toInt
- val actualSize = in.read(buf, off, requestSize)
- if (actualSize == -1)
- numRemaining = 0
- else
- numRemaining -= actualSize
- actualSize
- }
- }
-
- private val readBuf = new Array[Byte](1)
-
- override def read(): Int = {
- readBuf(0) = 0
- val n = read(readBuf, 0, 1)
- if (n == -1)
- -1
- else {
- Assert.invariant(n == 1)
- val i = readBuf(0).toInt
- val b = i & 0xff
- b
- }
- }
-}
-
-/**
- * Can be used with any InputStream to restrict what is
- * read from it to stop before a boundary mark string.
- *
- * The boundary mark string is exactly that, a string of characters. Not a
- * regex, nor anything involving DFDL Character Entities or Character Class
- * Entities. (No %WSP; no %NL; )
- *
- * This can be used to forcibly stop consumption of data from a stream at
- * a length obtained from a delimiter.
- *
- * The boundary mark string is consumed from the underlying stream (if found), and
- * the underlying stream is left positioned at the byte after the boundary mark
- * string.
- *
- * Thread safety: This is inherently stateful - so not thread safe to use
- * this object from more than one thread.
- */
-object BoundaryMarkLimitingStream {
-
- def apply(
- inputStream: InputStream,
- boundaryMark: String,
- charset: Charset,
- targetChunkSize: Int = 32 * 1024,
- ): InputStream = {
-
- Assert.usage(targetChunkSize >= 1)
- Assert.usage(boundaryMark.length >= 1)
-
- val boundaryMarkIn8859 =
- new String(boundaryMark.getBytes(charset), StandardCharsets.ISO_8859_1)
-
- val quotedBoundaryMark =
- Pattern.quote(boundaryMarkIn8859) // in case pattern has non-regex-safe characters in it
-
- val result = new RegexLimitingStream(
- inputStream,
- quotedBoundaryMark,
- boundaryMarkIn8859,
- charset,
- targetChunkSize,
- )
- result
- }
-
-}
-
-class StreamIterator[T](s: Stream[T]) extends Iterator[T] {
- private var str = s
- override def hasNext = !str.isEmpty
- override def next() = {
- val res = str.head
- str = str.tail
- res
- }
-}
-
/**
* Can be used with any InputStream to restrict what is
* read from it to stop before a particular regex match.
@@ -154,16 +50,16 @@ class StreamIterator[T](s: Stream[T]) extends Iterator[T] {
* Thread safety: This is inherently stateful - so not thread safe to use
* this object from more than one thread.
*/
-class RegexLimitingStream(
+class RegexLimitingInputStream(
inputStream: InputStream,
regexForDelimiter: String,
maximumLengthDelimiterExample: String,
charset: Charset,
- targetChunkSize: Int = 32 * 1024,
+ targetChunkSize: Int = 32 * 1024, // TODO: should this be smaller? Limited by a tunable?
) extends InputStream {
Assert.usage(targetChunkSize >= 1)
- Assert.usage(maximumLengthDelimiterExample.length >= 1)
+ Assert.usage(maximumLengthDelimiterExample.nonEmpty)
private val in = inputStream
@@ -215,7 +111,7 @@ class RegexLimitingStream(
private lazy val charsIter = {
val cks = chunks
val streamChars = cks.flatten
- val iter = new StreamIterator(streamChars)
+ val iter = streamChars.iterator
iter
}
@@ -249,10 +145,10 @@ class RegexLimitingStream(
//
val rdr = new InputStreamReader(in, StandardCharsets.ISO_8859_1)
val scanner = new Scanner(rdr)
- val matchString = scanner.findWithinHorizon(pattern, 0)
- val matchLength = checkScan(matchString, scanner)
+ val beforeMatchString = scanner.findWithinHorizon(pattern, 0)
+ val delimMatchLength = checkScan(beforeMatchString, scanner)
//
- // the trick is that the length of the matchString could be shorter than the
+ // the trick is that the length of the beforeMatchString could be shorter than the
// number of characters (aka bytes) pulled from the input stream because in
// scanning, the scanner or reader might buffer up extra decoded characters that
// strictly speaking aren't needed to obtain the match.
@@ -263,14 +159,19 @@ class RegexLimitingStream(
// Decoding errors, and the complexities they create over how big the string
// is, vs. how many bytes were consumed... those can't happen with iso-8859-1.
//
- in.reset() // might have to backup farther than the matchString length
- in.skip(matchString.length + matchLength) // advance exactly the right number of bytes
- if (matchLength > 0)
+ // We need to position our stream exactly after the characters that are terminated by the match
+ // being found.
+ in.reset()
+ in.skip(
+ beforeMatchString.length + delimMatchLength,
+ ) // advance exactly the right number of bytes (past the regex match)
+ if (delimMatchLength > 0)
noMoreChunks = true
- if (matchString.isEmpty())
+ if (beforeMatchString.isEmpty)
Stream()
else
- matchString #:: chunks
+ // lazy list construction. chunks will not be recursively evaluated unless demanded
+ beforeMatchString #:: chunks
}
}
diff --git a/daffodil-io/src/main/scala/org/apache/daffodil/io/StringDataInputStreamForUnparse.scala b/daffodil-io/src/main/scala/org/apache/daffodil/io/StringDataInputStreamForUnparse.scala
index 55eda3c63..513905a60 100644
--- a/daffodil-io/src/main/scala/org/apache/daffodil/io/StringDataInputStreamForUnparse.scala
+++ b/daffodil-io/src/main/scala/org/apache/daffodil/io/StringDataInputStreamForUnparse.scala
@@ -51,6 +51,10 @@ final class StringDataInputStreamForUnparse extends DataInputStreamImplMixin {
dis.asIteratorChar
}
+ override def close(): Unit = {
+ if (dis != null) dis.close()
+ }
+
override def bitLimit0b = dis.bitLimit0b
override def bitPos0b: Long = dis.bitPos0b
override def discard(mark: DataInputStream.Mark): Unit = dis.discard(mark)
diff --git a/daffodil-io/src/main/scala/org/apache/daffodil/io/processors/charset/AISPayloadArmoring.scala b/daffodil-io/src/main/scala/org/apache/daffodil/io/processors/charset/AISPayloadArmoring.scala
deleted file mode 100644
index 3b2715205..000000000
--- a/daffodil-io/src/main/scala/org/apache/daffodil/io/processors/charset/AISPayloadArmoring.scala
+++ /dev/null
@@ -1,52 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.daffodil.io.processors.charset
-
-import org.apache.daffodil.lib.schema.annotation.props.gen.BitOrder
-
-/**
- * Special purpose. This is not used for decoding anything.
- * The encoder is used to convert strings using the characters
- * allowed, into binary data using the AIS Payload Armoring
- * described here:
- *
- * http://catb.org/gpsd/AIVDM.html#_aivdm_aivdo_payload_armoring
- *
- * To convert a string of length N bytes, You will get 6N bits.
- *
- * The decoder can be used for unit testing, but the point of this class
- * is to make the encoder available for use in un-doing the AIS Payload
- * armoring when parsing, and performing this armoring when unparsing.
- *
- * When encoding from 8-bit say, ascii, or iso-8859-1, this can only encode
- * things that stay within the 64 allowed characters.
- * dfdl:encodingErrorPolicy='error' would check this (once implemented), otherwise
- * where this is used the checking needs to be done separately somehow.
- */
-object BitsCharsetAISPayloadArmoring
- extends {
- override val name = "X-DAFFODIL-AIS-PAYLOAD-ARMORING"
- override val bitWidthOfACodeUnit = 6
- override val decodeString =
- """0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVW'abcdefghijklmnopqrstuvw"""
- override val replacementCharCode = 0x30
- override val requiredBitOrder = BitOrder.MostSignificantBitFirst
- }
- with BitsCharsetNonByteSize
-
-final class BitsCharsetAISPayloadArmoringDefinition
- extends BitsCharsetDefinition(BitsCharsetAISPayloadArmoring)
diff --git a/daffodil-io/src/main/scala/org/apache/daffodil/io/processors/charset/BitsCharset.scala b/daffodil-io/src/main/scala/org/apache/daffodil/io/processors/charset/BitsCharset.scala
index 17f9fbcc0..45453a1ec 100644
--- a/daffodil-io/src/main/scala/org/apache/daffodil/io/processors/charset/BitsCharset.scala
+++ b/daffodil-io/src/main/scala/org/apache/daffodil/io/processors/charset/BitsCharset.scala
@@ -69,6 +69,9 @@ trait BitsCharset extends Serializable {
*/
def isEbcdicFamily(): Boolean = false
+ /**
+ * @return the width, in bits of a character, or Nope if the character set is variable width.
+ */
def maybeFixedWidth: MaybeInt
final def padCharWidthInBits = {
diff --git a/daffodil-io/src/main/scala/org/apache/daffodil/io/processors/charset/BitsCharsetNonByteSize.scala b/daffodil-io/src/main/scala/org/apache/daffodil/io/processors/charset/BitsCharsetNonByteSize.scala
index 04bb1aec3..41750453d 100644
--- a/daffodil-io/src/main/scala/org/apache/daffodil/io/processors/charset/BitsCharsetNonByteSize.scala
+++ b/daffodil-io/src/main/scala/org/apache/daffodil/io/processors/charset/BitsCharsetNonByteSize.scala
@@ -117,12 +117,12 @@ trait BitsCharsetNonByteSize extends BitsCharset {
}
/**
- * Hyjack a JavaCharsetEncoder to drive the encodeLoop.
+ * Hijack a JavaCharsetEncoder to drive the encodeLoop.
*
* This avoids us reimplementing all the error handling and flush/end logic.
*
* TODO: Similar to our decoders, we should create custom encoders. Then we
- * wouldn't need all this complex code related to proxying java charsets.
+ * wouldn't need all this complex code related to proxying java charsets.
*/
protected final class ProxyJavaCharsetEncoder(
cs: JavaCharset,
diff --git a/daffodil-io/src/test/scala/org/apache/daffodil/io/TestDataOutputStream.scala b/daffodil-io/src/test/scala/org/apache/daffodil/io/TestDataOutputStream.scala
index 293f3e5ef..24ec6434b 100644
--- a/daffodil-io/src/test/scala/org/apache/daffodil/io/TestDataOutputStream.scala
+++ b/daffodil-io/src/test/scala/org/apache/daffodil/io/TestDataOutputStream.scala
@@ -54,7 +54,7 @@ class TestDataOutputStream {
out.putLong(-1L, 32, beFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(-1, buf(0))
assertEquals(-1, buf(1))
@@ -74,7 +74,7 @@ class TestDataOutputStream {
out.setFinished(beFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0x80.toByte, buf(0))
if (buf.length > 1)
@@ -90,7 +90,7 @@ class TestDataOutputStream {
out.setFinished(beFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0xc0.toByte, buf(0))
if (buf.length > 1)
@@ -106,7 +106,7 @@ class TestDataOutputStream {
out.setFinished(beFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0x4a.toByte, buf(0))
if (buf.length > 1)
@@ -122,7 +122,7 @@ class TestDataOutputStream {
out.setFinished(beFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0xa5.toByte, buf(0))
if (buf.length > 1)
@@ -138,7 +138,7 @@ class TestDataOutputStream {
out.setFinished(beFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0xd2.toByte, buf(0))
assertEquals(0x80.toByte, buf(1))
@@ -155,7 +155,7 @@ class TestDataOutputStream {
out.setFinished(beFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
val res = 0x25a5a5a5a5a5a5a5L << 1
var i = 0
@@ -178,7 +178,7 @@ class TestDataOutputStream {
out.setFinished(beFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
val res = 0xa5a5a5a5a5a5a5a5L
var i = 0
@@ -209,7 +209,7 @@ class TestDataOutputStream {
direct.setFinished(beFinfo)
out.setFinished(beFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(-1, buf(0))
assertEquals(-1, buf(1))
@@ -232,7 +232,7 @@ class TestDataOutputStream {
direct.setFinished(beFinfo)
out.setFinished(beFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0x80.toByte, buf(0))
if (buf.length > 1)
@@ -251,7 +251,7 @@ class TestDataOutputStream {
direct.setFinished(beFinfo)
out.setFinished(beFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0xc0.toByte, buf(0))
if (buf.length > 1)
@@ -270,7 +270,7 @@ class TestDataOutputStream {
direct.setFinished(beFinfo)
out.setFinished(beFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0x4a.toByte, buf(0))
if (buf.length > 1)
@@ -290,7 +290,7 @@ class TestDataOutputStream {
direct.setFinished(beFinfo)
out.setFinished(beFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0xa5.toByte, buf(0))
if (buf.length > 1)
@@ -309,7 +309,7 @@ class TestDataOutputStream {
direct.setFinished(beFinfo)
out.setFinished(beFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0xd2.toByte, buf(0))
assertEquals(0x80.toByte, buf(1))
@@ -329,7 +329,7 @@ class TestDataOutputStream {
direct.setFinished(beFinfo)
out.setFinished(beFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
val res = 0x25a5a5a5a5a5a5a5L << 1
var i = 0
@@ -355,7 +355,7 @@ class TestDataOutputStream {
direct.setFinished(beFinfo)
out.setFinished(beFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
val res = 0xa5a5a5a5a5a5a5a5L
var i = 0
@@ -387,7 +387,7 @@ class TestDataOutputStream {
direct.setFinished(beFinfo)
out.setFinished(beFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(-1, buf(0))
assertEquals(-1, buf(1))
@@ -415,7 +415,7 @@ class TestDataOutputStream {
direct.setFinished(beFinfo)
out.setFinished(beFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0xc0.toByte, buf(0))
if (buf.length > 1)
@@ -435,7 +435,7 @@ class TestDataOutputStream {
direct.setFinished(beFinfo)
out.setFinished(beFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0xb0.toByte, buf(0))
if (buf.length > 1)
@@ -455,7 +455,7 @@ class TestDataOutputStream {
direct.setFinished(beFinfo)
out.setFinished(beFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0x4a.toByte, buf(0))
assertEquals(0x94.toByte, buf(1))
@@ -476,7 +476,7 @@ class TestDataOutputStream {
direct.setFinished(beFinfo)
out.setFinished(beFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0xa5.toByte, buf(0))
assertEquals(0xa5.toByte, buf(1))
@@ -497,7 +497,7 @@ class TestDataOutputStream {
direct.setFinished(beFinfo)
out.setFinished(beFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0x77.toByte, buf(0))
assertEquals(0xab.toByte, buf(1))
@@ -519,7 +519,7 @@ class TestDataOutputStream {
direct.setFinished(beFinfo)
out.setFinished(beFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
val res = 0xa5a5a5a5a5a5a5a5L
var i = 0
@@ -546,7 +546,7 @@ class TestDataOutputStream {
direct.setFinished(beFinfo)
out.setFinished(beFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
val res1 = 0x4b4b4b4b4b4b4b4aL
var i = 0
@@ -582,7 +582,7 @@ class TestDataOutputStream {
direct.setFinished(beFinfo)
out.setFinished(beFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
val res1 = 0xa5a5a5a5a5a5a5a5L
var i = 0
@@ -615,7 +615,7 @@ class TestDataOutputStream {
out.setFinished(beFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0x50.toByte, buf(0))
if (buf.length > 1)
diff --git a/daffodil-io/src/test/scala/org/apache/daffodil/io/TestDataOutputStream2.scala b/daffodil-io/src/test/scala/org/apache/daffodil/io/TestDataOutputStream2.scala
index a096e7e07..61322cfc2 100644
--- a/daffodil-io/src/test/scala/org/apache/daffodil/io/TestDataOutputStream2.scala
+++ b/daffodil-io/src/test/scala/org/apache/daffodil/io/TestDataOutputStream2.scala
@@ -52,7 +52,7 @@ class TestDataOutputStream2 {
out.setFinished(beFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0xbe.toByte, buf(0))
assertEquals(0xef.toByte, buf(1))
@@ -80,7 +80,7 @@ class TestDataOutputStream2 {
out.setFinished(beFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0xbe.toByte, buf(0))
assertEquals(0x80.toByte, buf(1))
@@ -108,7 +108,7 @@ class TestDataOutputStream2 {
out.setFinished(beFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0xbe.toByte, buf(0))
assertEquals(0xee.toByte, buf(1))
diff --git a/daffodil-io/src/test/scala/org/apache/daffodil/io/TestDataOutputStream3.scala b/daffodil-io/src/test/scala/org/apache/daffodil/io/TestDataOutputStream3.scala
index b974572c7..5fbc22130 100644
--- a/daffodil-io/src/test/scala/org/apache/daffodil/io/TestDataOutputStream3.scala
+++ b/daffodil-io/src/test/scala/org/apache/daffodil/io/TestDataOutputStream3.scala
@@ -57,7 +57,7 @@ class TestDataOutputStream3 {
out.putLong(-1L, 32, lsbfFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(-1, buf(0))
assertEquals(-1, buf(1))
@@ -77,7 +77,7 @@ class TestDataOutputStream3 {
out.setFinished(lsbfFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0x01.toByte, buf(0))
if (buf.length > 1)
@@ -93,7 +93,7 @@ class TestDataOutputStream3 {
out.setFinished(lsbfFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0x03.toByte, buf(0))
if (buf.length > 1)
@@ -109,7 +109,7 @@ class TestDataOutputStream3 {
out.setFinished(lsbfFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0x25.toByte, buf(0))
if (buf.length > 1)
@@ -125,7 +125,7 @@ class TestDataOutputStream3 {
out.setFinished(lsbfFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0xa5.toByte, buf(0))
if (buf.length > 1)
@@ -141,7 +141,7 @@ class TestDataOutputStream3 {
out.setFinished(lsbfFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0xa5.toByte, buf(0))
assertEquals(0x01.toByte, buf(1))
@@ -158,7 +158,7 @@ class TestDataOutputStream3 {
out.setFinished(lsbfFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
val res = 0x25a5a5a5a5a5a5a5L
var i = 0
@@ -181,7 +181,7 @@ class TestDataOutputStream3 {
out.setFinished(lsbfFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
val res = 0xa5a5a5a5a5a5a5a5L
var i = 0
@@ -211,7 +211,7 @@ class TestDataOutputStream3 {
direct.setFinished(lsbfFinfo)
out.setFinished(lsbfFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(-1, buf(0))
assertEquals(-1, buf(1))
@@ -233,7 +233,7 @@ class TestDataOutputStream3 {
direct.setFinished(lsbfFinfo)
out.setFinished(lsbfFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0x01.toByte, buf(0))
if (buf.length > 1)
@@ -251,7 +251,7 @@ class TestDataOutputStream3 {
direct.setFinished(lsbfFinfo)
out.setFinished(lsbfFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0x03.toByte, buf(0))
if (buf.length > 1)
@@ -269,7 +269,7 @@ class TestDataOutputStream3 {
direct.setFinished(lsbfFinfo)
out.setFinished(lsbfFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0x25.toByte, buf(0))
if (buf.length > 1)
@@ -287,7 +287,7 @@ class TestDataOutputStream3 {
direct.setFinished(lsbfFinfo)
out.setFinished(lsbfFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0xa5.toByte, buf(0))
if (buf.length > 1)
@@ -305,7 +305,7 @@ class TestDataOutputStream3 {
direct.setFinished(lsbfFinfo)
out.setFinished(lsbfFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0xa5.toByte, buf(0))
assertEquals(0x01.toByte, buf(1))
@@ -324,7 +324,7 @@ class TestDataOutputStream3 {
direct.setFinished(lsbfFinfo)
out.setFinished(lsbfFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
val res = 0x25a5a5a5a5a5a5a5L
var i = 0
@@ -349,7 +349,7 @@ class TestDataOutputStream3 {
direct.setFinished(lsbfFinfo)
out.setFinished(lsbfFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
val res = 0xa5a5a5a5a5a5a5a5L
var i = 0
@@ -380,7 +380,7 @@ class TestDataOutputStream3 {
direct.setFinished(lsbfFinfo)
out.setFinished(lsbfFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(-1, buf(0))
assertEquals(-1, buf(1))
@@ -408,7 +408,7 @@ class TestDataOutputStream3 {
direct.setFinished(lsbfFinfo)
out.setFinished(lsbfFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0x03.toByte, buf(0))
if (buf.length > 1)
@@ -428,7 +428,7 @@ class TestDataOutputStream3 {
direct.setFinished(lsbfFinfo)
out.setFinished(lsbfFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0x0e.toByte, buf(0))
if (buf.length > 1)
@@ -448,7 +448,7 @@ class TestDataOutputStream3 {
direct.setFinished(lsbfFinfo)
out.setFinished(lsbfFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0xa5.toByte, buf(0))
assertEquals(0x12.toByte, buf(1))
@@ -468,7 +468,7 @@ class TestDataOutputStream3 {
direct.setFinished(lsbfFinfo)
out.setFinished(lsbfFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0xa5.toByte, buf(0))
assertEquals(0xa5.toByte, buf(1))
@@ -489,7 +489,7 @@ class TestDataOutputStream3 {
direct.setFinished(lsbfFinfo)
out.setFinished(lsbfFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0xef.toByte, buf(0))
assertEquals(0x5a.toByte, buf(1))
@@ -511,7 +511,7 @@ class TestDataOutputStream3 {
direct.setFinished(lsbfFinfo)
out.setFinished(lsbfFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
val res = (0xa5a5a5a5a5a5a5a5L << 1) + 1
var i = 0
@@ -538,7 +538,7 @@ class TestDataOutputStream3 {
direct.setFinished(lsbfFinfo)
out.setFinished(lsbfFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
val res1 = 0xa5a5a5a5a5a5a5a5L
var i = 0
@@ -573,7 +573,7 @@ class TestDataOutputStream3 {
direct.setFinished(lsbfFinfo)
out.setFinished(lsbfFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
val res1 = 0xa5a5a5a5a5a5a5a5L
var i = 0
@@ -606,7 +606,7 @@ class TestDataOutputStream3 {
out.setFinished(lsbfFinfo)
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0x05.toByte, buf(0))
if (buf.length > 1)
diff --git a/daffodil-io/src/test/scala/org/apache/daffodil/io/TestDataOutputStream4.scala b/daffodil-io/src/test/scala/org/apache/daffodil/io/TestDataOutputStream4.scala
index e3e779f90..70ec09c52 100644
--- a/daffodil-io/src/test/scala/org/apache/daffodil/io/TestDataOutputStream4.scala
+++ b/daffodil-io/src/test/scala/org/apache/daffodil/io/TestDataOutputStream4.scala
@@ -74,7 +74,7 @@ class TestDataOutputStream4 {
}
private def checkResults(baos: ByteArrayOrFileOutputStream): Unit = {
- val buf = baos.getBuf()
+ val buf = baos.getBuf
assertEquals(0xb4.toByte, buf(0))
assertEquals(0xb4.toByte, buf(1))
diff --git a/daffodil-io/src/test/scala/org/apache/daffodil/layers/TestBase64.scala b/daffodil-io/src/test/scala/org/apache/daffodil/layers/TestBase64.scala
index d65dd87a6..200d3a7e5 100644
--- a/daffodil-io/src/test/scala/org/apache/daffodil/layers/TestBase64.scala
+++ b/daffodil-io/src/test/scala/org/apache/daffodil/layers/TestBase64.scala
@@ -37,11 +37,13 @@ import org.junit.Test
class TestBase64 {
Assert.usage(scala.util.Properties.isJavaAtLeast("1.8"))
+ val crlf = "\r\n"
+
val text = """This is just some made up text that is intended to be
a few lines long. If this had been real text, it would not have been quite
so boring to read. Use of famous quotes or song lyrics or anything like that
introduces copyright notice issues, so it is easier to simply make up
-a few lines of pointless text like this.""".replace("\r\n", "\n").replace("\n", " ")
+a few lines of pointless text like this.""".replace(crlf, "\n").replace("\n", " ")
val b64Text = """VGhpcyBpcyBqdXN0IHNvbWUgbWFkZSB1cCB0ZXh0IHRoYXQgaXMgaW50Z
W5kZWQgdG8gYmUgYSBmZXcgbGluZXMgbG9uZy4gSWYgdGhpcyBoYWQgYmVlbiByZWFsIHRleHQsI
@@ -49,7 +51,7 @@ Gl0IHdvdWxkIG5vdCBoYXZlIGJlZW4gcXVpdGUgc28gYm9yaW5nIHRvIHJlYWQuIFVzZSBvZiBmY
W1vdXMgcXVvdGVzIG9yIHNvbmcgbHlyaWNzIG9yIGFueXRoaW5nIGxpa2UgdGhhdCBpbnRyb2R1Y
2VzIGNvcHlyaWdodCBub3RpY2UgaXNzdWVzLCBzbyBpdCBpcyBlYXNpZXIgdG8gc2ltcGx5IG1ha
2UgdXAgYSBmZXcgbGluZXMgb2YgcG9pbnRsZXNzIHRleHQgbGlrZSB0aGlzLg==
-""".replace("\r\n", "\n").replace("\n", "").sliding(76, 76).mkString("\r\n")
+""".replace(crlf, "\n").replace("\n", "").sliding(76, 76).mkString(crlf)
/**
* Same encoded data, but shot through with extra CRLFs
@@ -105,7 +107,7 @@ W1vdXMgcXVvdGVzIG9yIHNvbmcgbHlyaWNzIG9yIGFueXRoaW5nIGxpa2UgdGhhdCBpbnRyb2R1Y
@Test def testBase64_decode_consumes_final_CRLF(): Unit = {
- val data = b64Text ++ "\r\n" // add extra CRLF
+ val data = b64Text ++ crlf // add extra CRLF
val actual = new String(java.util.Base64.getMimeDecoder.decode(data))
diff --git a/daffodil-io/src/test/scala/org/apache/daffodil/layers/TestJavaIOStreams.scala b/daffodil-io/src/test/scala/org/apache/daffodil/layers/TestJavaIOStreams.scala
index eba14b049..f7f804832 100644
--- a/daffodil-io/src/test/scala/org/apache/daffodil/layers/TestJavaIOStreams.scala
+++ b/daffodil-io/src/test/scala/org/apache/daffodil/layers/TestJavaIOStreams.scala
@@ -52,12 +52,15 @@ import org.junit.Test
class TestJavaIOStreams {
Assert.usage(scala.util.Properties.isJavaAtLeast("1.8"))
+ val crlfRegex = "\\r\\n(?!(?:\\t|\\ ))"
+ val crlf = "\r\n"
+
val text = """Daffodil is an open source implementation of the DFDL
specification that uses these DFDL schemas to parse fixed format data into an
infoset, which is most commonly represented as either XML or JSON. This
allows the use of well-established XML or JSON technologies and libraries to
consume, inspect, and manipulate fixed format data in existing solutions."""
- .replace("\r\n", "\n")
+ .replace(crlf, "\n")
.replace("\n", " ")
val b64Text = """RGFmZm9kaWwgaXMgYW4gb3BlbiBzb3VyY2UgaW1wbGVtZW50YXRpb24gb2YgdGhlIERGREwgc3Bl
diff --git a/daffodil-io/src/test/scala/org/apache/daffodil/layers/TestLimitingJavaIOStreams.scala b/daffodil-io/src/test/scala/org/apache/daffodil/layers/TestLimitingJavaIOStreams.scala
index 5560bfa25..4dc47d7e3 100644
--- a/daffodil-io/src/test/scala/org/apache/daffodil/layers/TestLimitingJavaIOStreams.scala
+++ b/daffodil-io/src/test/scala/org/apache/daffodil/layers/TestLimitingJavaIOStreams.scala
@@ -21,12 +21,10 @@ import java.io.ByteArrayInputStream
import java.io.ByteArrayOutputStream
import java.nio.charset.StandardCharsets
-import org.apache.daffodil.io.BoundaryMarkLimitingStream
-import org.apache.daffodil.io.ExplicitLengthLimitingStream
-import org.apache.daffodil.io.RegexLimitingStream
+import org.apache.daffodil.io.BoundaryMarkLimitingInputStream
+import org.apache.daffodil.io.RegexLimitingInputStream
import org.apache.daffodil.lib.exceptions.Assert
-import collection.JavaConverters._
import org.apache.commons.io.IOUtils
import org.junit.Assert._
import org.junit.Test
@@ -51,12 +49,15 @@ class TestLimitingJavaIOStreams {
val iso8859 = StandardCharsets.ISO_8859_1
val utf8 = StandardCharsets.ISO_8859_1
+ val crlfRegex = "\\r\\n(?!(?:\\t|\\ ))"
+ val crlf = "\r\n"
+
val text = """Daffodil is an open source implementation of the DFDL
specification that uses these DFDL schemas to parse fixed format data into an
infoset, which is most commonly represented as either XML or JSON. This
allows the use of well-established XML or JSON technologies and libraries to
consume, inspect, and manipulate fixed format data in existing solutions."""
- .replace("\r\n", "\n")
+ .replace(crlf, "\n")
.replace("\n", " ")
val b64Text = """RGFmZm9kaWwgaXMgYW4gb3BlbiBzb3VyY2UgaW1wbGVtZW50YXRpb24gb2YgdGhlIERGREwgc3Bl
@@ -86,7 +87,7 @@ ZyBzb2x1dGlvbnMuCg=="""
.toInputStream(data + terminator + afterTerminator, StandardCharsets.UTF_8)
.asInstanceOf[ByteArrayInputStream]
val delimitedStream =
- BoundaryMarkLimitingStream(is, terminator, iso8859, targetChunkSize = 4)
+ new BoundaryMarkLimitingInputStream(is, terminator, iso8859, targetChunkSize = 4)
val b64 = java.util.Base64.getMimeDecoder().wrap(delimitedStream)
val actualData = IOUtils.toString(b64, iso8859)
assertEquals("solutions.", actualData)
@@ -103,7 +104,7 @@ ZyBzb2x1dGlvbnMuCg=="""
.toInputStream(data + terminator + afterTerminator, "ascii")
.asInstanceOf[ByteArrayInputStream]
val delimitedStream =
- BoundaryMarkLimitingStream(is, terminator, iso8859, targetChunkSize = 1)
+ new BoundaryMarkLimitingInputStream(is, terminator, iso8859, targetChunkSize = 1)
val b64 = java.util.Base64.getMimeDecoder().wrap(delimitedStream)
val actualData = IOUtils.toString(b64, iso8859)
assertEquals("solutions.", actualData)
@@ -119,7 +120,7 @@ ZyBzb2x1dGlvbnMuCg=="""
.toInputStream(data + terminator + afterTerminator, "ascii")
.asInstanceOf[ByteArrayInputStream]
val delimitedStream =
- BoundaryMarkLimitingStream(is, terminator, iso8859, targetChunkSize = 1)
+ new BoundaryMarkLimitingInputStream(is, terminator, iso8859, targetChunkSize = 1)
val b64 = java.util.Base64.getMimeDecoder().wrap(delimitedStream)
val actualData = IOUtils.toString(b64, iso8859)
assertEquals("ple", actualData)
@@ -127,87 +128,63 @@ ZyBzb2x1dGlvbnMuCg=="""
assertEquals(afterTerminator, actualAfterData)
}
- /**
- * Illustrates that if we clamp the gzip stream at a finite length,
- * that we can then do precise consumption of bytes, as would be expected.
- *
- * So this is the technique needed for GZIP, since otherwise it reads too far ahead.
- * Unlike Java 8 Base64, which stops when it self-detects the last byte of the
- * encoded data, GZIP doesn't do that. It seems to always read past the end. This
- * may be an artifact of the implementation, or inherent in the GZIP data format.
- */
- @Test def testGZIPDecoderWithLimit1(): Unit = {
- val inputData = zipped ++ additionalText.getBytes(iso8859)
- val inputStream = new ByteArrayInputStream(inputData)
- val limitedStream = new ExplicitLengthLimitingStream(inputStream, zipped.length)
- val expected = text
-
- val decodedStream = new java.util.zip.GZIPInputStream(limitedStream, 5)
- val lines = IOUtils.readLines(decodedStream, iso8859).asScala.toSeq
- assertEquals(1, lines.length)
- assertEquals(expected, lines(0))
- val additionalLines = IOUtils.readLines(inputStream, iso8859).asScala.toSeq
- assertEquals(1, additionalLines.length)
- assertEquals(additionalText, additionalLines(0))
- }
-
/**
* Test for lines of text containing \r\n\t or \r\n\x20 (CRLF + space)
* but terminated by a \r\n NOT followed by \t or space.
*/
@Test def testRegexDelimStream1() = {
val beforeDelim = "12345\r\n\t67890\r\n\tabcde"
- val delim = "\r\n"
+ val delim = crlf
val afterDelim = "fghij"
val inputString = beforeDelim + delim + afterDelim
val inputBytes = inputString.getBytes("utf-8")
val bais = new ByteArrayInputStream(inputBytes)
- val rls = new RegexLimitingStream(bais, "\\r\\n(?!(?:\\t|\\ ))", "\r\n", utf8)
- val baos = new ByteArrayOutputStream()
- var c: Int = -1
- while ({
- c = rls.read()
- c != -1
- }) {
- baos.write(c)
- }
- baos.close()
- val actualBeforeDelim = new String(baos.toByteArray())
- val afterBaos = new ByteArrayOutputStream()
- while ({
- c = bais.read()
- c != -1
- }) {
- afterBaos.write(c)
- }
- afterBaos.close()
- val actualAfterDelim = new String(afterBaos.toByteArray())
+ val rls = new RegexLimitingInputStream(bais, crlfRegex, crlf, utf8)
+ val actualBeforeDelim = IOUtils.toString(rls, utf8)
+ // after the regex match the bais stream should be positioned immediately after the \r\n delim match.
+ val actualAfterDelim = IOUtils.toString(bais, utf8)
assertEquals(beforeDelim, actualBeforeDelim)
assertEquals(afterDelim, actualAfterDelim)
-
}
/**
* Shows that the match, if the delim regex isn't matched at all,
- * contains the entire available ipput data.
+ * contains the entire available input data.
*/
@Test def testRegexDelimStream2() = {
val beforeDelim = "12345\r\n\t67890\r\n\tabcde"
+ val inputString = beforeDelim
+ val inputBytes = inputString.getBytes("utf-8")
+ val bais = new ByteArrayInputStream(inputBytes)
+ val rls = new RegexLimitingInputStream(bais, crlfRegex, "\r\n\t", utf8)
+ val actualBeforeDelim = IOUtils.toString(rls, utf8)
+ assertEquals(beforeDelim, actualBeforeDelim)
+ }
+ /**
+ * Shows that the match if the delim is immediately followed by end-of-data
+ * that also matches.
+ */
+ @Test def testRegexDelimStream3() = {
+ val beforeDelim = "12345\r\n\t67890\r\n\tabcde"
+ val inputString = beforeDelim + crlf
+ val inputBytes = inputString.getBytes("utf-8")
+ val bais = new ByteArrayInputStream(inputBytes)
+ val rls = new RegexLimitingInputStream(bais, crlfRegex, "\r\n\t", utf8)
+ val actualBeforeDelim = IOUtils.toString(rls, utf8)
+ assertEquals(beforeDelim, actualBeforeDelim)
+ }
+
+ /**
+ * Shows that the match if the delim is never found
+ */
+ @Test def testRegexDelimStream4() = {
+ val beforeDelim = "1234567890tabcde"
val inputString = beforeDelim
val inputBytes = inputString.getBytes("utf-8")
val bais = new ByteArrayInputStream(inputBytes)
- val rls = new RegexLimitingStream(bais, "\\r\\n(?!(?:\\t|\\ ))", "\r\n\t", utf8, 4)
- val baos = new ByteArrayOutputStream()
- var c: Int = -1
- while ({
- c = rls.read()
- c != -1
- }) {
- baos.write(c)
- }
- baos.close()
- val actualBeforeDelim = new String(baos.toByteArray())
+ val rls = new RegexLimitingInputStream(bais, crlfRegex, "\r\n\t", utf8)
+ val actualBeforeDelim = IOUtils.toString(rls, utf8)
assertEquals(beforeDelim, actualBeforeDelim)
}
}
diff --git a/daffodil-japi/src/main/scala/org/apache/daffodil/japi/io/InputSourceDataInputStream.scala b/daffodil-japi/src/main/scala/org/apache/daffodil/japi/io/InputSourceDataInputStream.scala
index 040872ca2..166dff26b 100644
--- a/daffodil-japi/src/main/scala/org/apache/daffodil/japi/io/InputSourceDataInputStream.scala
+++ b/daffodil-japi/src/main/scala/org/apache/daffodil/japi/io/InputSourceDataInputStream.scala
@@ -30,7 +30,7 @@ import org.apache.daffodil.io.{ InputSourceDataInputStream => SInputSourceDataIn
*/
class InputSourceDataInputStream private[japi] (
private[japi] val dis: SInputSourceDataInputStream,
-) {
+) extends java.io.Closeable {
/**
* Create an InputSourceDataInputStream from a java.io.InputStream
@@ -66,4 +66,15 @@ class InputSourceDataInputStream private[japi] (
* been consumed or some data is left-over.
*/
def hasData(): Boolean = dis.isDefinedForLength(1)
+
+ /**
+ * Closes the underlying resource.
+ *
+ * This method is used to close the underlying resource associated with the current instance.
+ * It is typically used to release any system resources that have been acquired during
+ * the lifespan of the instance.
+ *
+ * throws IOException if an I/O error occurs during the close operation.
+ */
+ def close(): Unit = dis.close()
}
diff --git a/daffodil-japi/src/test/java/org/apache/daffodil/example/TestCustomDebuggerAPI.java b/daffodil-japi/src/test/java/org/apache/daffodil/example/TestCustomDebuggerAPI.java
index 06ec05cea..29360875a 100644
--- a/daffodil-japi/src/test/java/org/apache/daffodil/example/TestCustomDebuggerAPI.java
+++ b/daffodil-japi/src/test/java/org/apache/daffodil/example/TestCustomDebuggerAPI.java
@@ -32,6 +32,7 @@ import org.junit.Test;
import java.io.IOException;
import java.net.URI;
+import static java.util.Objects.nonNull;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
@@ -69,11 +70,12 @@ public class TestCustomDebuggerAPI {
String file = Misc.getRequiredResource("/test/japi/myData2.dat").toURL().getFile();
java.io.FileInputStream fis = new java.io.FileInputStream(file);
- InputSourceDataInputStream dis = new InputSourceDataInputStream(fis);
- ParseResult res = dp.parse(dis, new NullInfosetOutputter());
+ try (InputSourceDataInputStream dis = new InputSourceDataInputStream(fis)) {
+ ParseResult res = dp.parse(dis, new NullInfosetOutputter());
- assertEquals(6, dbg.nodes);
- assertTrue(dbg.inited);
- assertTrue(dbg.finished);
+ assertEquals(6, dbg.nodes);
+ assertTrue(dbg.inited);
+ assertTrue(dbg.finished);
+ }
}
}
diff --git a/daffodil-japi/src/test/java/org/apache/daffodil/example/TestJavaAPI.java b/daffodil-japi/src/test/java/org/apache/daffodil/example/TestJavaAPI.java
index 1ad7b38b8..70439586f 100644
--- a/daffodil-japi/src/test/java/org/apache/daffodil/example/TestJavaAPI.java
+++ b/daffodil-japi/src/test/java/org/apache/daffodil/example/TestJavaAPI.java
@@ -113,7 +113,7 @@ public class TestJavaAPI {
result = Daffodil.compiler().reload(Channels.newChannel(bais));
} catch (InvalidParserException e) {
fail("Unable to reload data processor");
- }
+ }
return result;
}
@@ -131,22 +131,23 @@ public class TestJavaAPI {
java.io.File file = getResource("/test/japi/myData.dat");
java.io.FileInputStream fis = new java.io.FileInputStream(file);
- InputSourceDataInputStream dis = new InputSourceDataInputStream(fis);
- JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
- ParseResult res = dp.parse(dis, outputter);
- boolean err = res.isError();
- assertFalse(err);
- assertTrue(debugger.lines.size() > 0);
- assertTrue(debugger.lines.contains("----------------------------------------------------------------- 1\n"));
- assertTrue(debugger.getCommand().equals("trace"));
-
- java.io.ByteArrayOutputStream bos = new java.io.ByteArrayOutputStream();
- java.nio.channels.WritableByteChannel wbc = java.nio.channels.Channels.newChannel(bos);
- JDOMInfosetInputter inputter = new JDOMInfosetInputter(outputter.getResult());
- UnparseResult res2 = dp.unparse(inputter, wbc);
- err = res2.isError();
- assertFalse(err);
- assertEquals("42", bos.toString());
+ try (InputSourceDataInputStream dis = new InputSourceDataInputStream(fis)) {
+ JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
+ ParseResult res = dp.parse(dis, outputter);
+ boolean err = res.isError();
+ assertFalse(err);
+ assertTrue(debugger.lines.size() > 0);
+ assertTrue(debugger.lines.contains("----------------------------------------------------------------- 1\n"));
+ assertTrue(debugger.getCommand().equals("trace"));
+
+ java.io.ByteArrayOutputStream bos = new java.io.ByteArrayOutputStream();
+ java.nio.channels.WritableByteChannel wbc = java.nio.channels.Channels.newChannel(bos);
+ JDOMInfosetInputter inputter = new JDOMInfosetInputter(outputter.getResult());
+ UnparseResult res2 = dp.unparse(inputter, wbc);
+ err = res2.isError();
+ assertFalse(err);
+ assertEquals("42", bos.toString());
+ }
}
// This is a duplicate of test testJavaAPI1 that serializes the parser
@@ -178,22 +179,23 @@ public class TestJavaAPI {
// and byte buffer.
byte[] ba = FileUtils.readFileToByteArray(data);
ByteBuffer bb = ByteBuffer.wrap(ba);
- InputSourceDataInputStream dis = new InputSourceDataInputStream(bb);
- JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
- ParseResult res = parser.parse(dis, outputter);
- boolean err = res.isError();
- assertFalse(err);
- assertTrue(debugger.lines.size() > 0);
- assertTrue(debugger.lines.contains("----------------------------------------------------------------- 1\n"));
- assertTrue(debugger.getCommand().equals("trace"));
-
- java.io.ByteArrayOutputStream bos = new java.io.ByteArrayOutputStream();
- java.nio.channels.WritableByteChannel wbc = java.nio.channels.Channels.newChannel(bos);
- JDOMInfosetInputter inputter = new JDOMInfosetInputter(outputter.getResult());
- UnparseResult res2 = dp.unparse(inputter, wbc);
- err = res2.isError();
- assertFalse(err);
- assertEquals("42", bos.toString());
+ try (InputSourceDataInputStream dis = new InputSourceDataInputStream(bb)) {
+ JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
+ ParseResult res = parser.parse(dis, outputter);
+ boolean err = res.isError();
+ assertFalse(err);
+ assertTrue(debugger.lines.size() > 0);
+ assertTrue(debugger.lines.contains("----------------------------------------------------------------- 1\n"));
+ assertTrue(debugger.getCommand().equals("trace"));
+
+ java.io.ByteArrayOutputStream bos = new java.io.ByteArrayOutputStream();
+ java.nio.channels.WritableByteChannel wbc = java.nio.channels.Channels.newChannel(bos);
+ JDOMInfosetInputter inputter = new JDOMInfosetInputter(outputter.getResult());
+ UnparseResult res2 = dp.unparse(inputter, wbc);
+ err = res2.isError();
+ assertFalse(err);
+ assertEquals("42", bos.toString());
+ }
}
// This is a duplicate of test testJavaAPI1 that serializes the parser
@@ -240,29 +242,30 @@ public class TestJavaAPI {
// the constructor for creating an InputSourceDataInputStream from a byte array
// and byte buffer.
byte[] ba = FileUtils.readFileToByteArray(file);
- InputSourceDataInputStream dis = new InputSourceDataInputStream(ba);
- JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
- ParseResult res = dp.parse(dis, outputter);
-
- // TODO: NEED a java friendly way to get the status of the outputter.
- // Scala enums don't work well
- // assertTrue(outputter.getStatus() != Status.DONE); // This is a hack
- // so that Java doesn't have to know about Nope/Maybe, need to figure
- // out better api that is Java compatible
- assertTrue(res.isError());
- java.util.List<Diagnostic> diags = res.getDiagnostics();
- assertEquals(1, diags.size());
- Diagnostic d = diags.get(0);
- // System.err.println(d.getMessage());
- assertTrue(d.getMessage().contains("int"));
- assertTrue(d.getMessage().contains("Not an int"));
- assertTrue(d.getDataLocations().toString().contains("10"));
- java.util.List<LocationInSchemaFile> locs = d.getLocationsInSchemaFiles();
- assertEquals(1, locs.size());
- LocationInSchemaFile loc = locs.get(0);
- assertTrue(loc.toString().contains("mySchema1.dfdl.xsd"));
- // above is mySchema1 as it reports the element ref location,
- // not element decl.
+ try (InputSourceDataInputStream dis = new InputSourceDataInputStream(ba)) {
+ JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
+ ParseResult res = dp.parse(dis, outputter);
+
+ // TODO: NEED a java friendly way to get the status of the outputter.
+ // Scala enums don't work well
+ // assertTrue(outputter.getStatus() != Status.DONE); // This is a hack
+ // so that Java doesn't have to know about Nope/Maybe, need to figure
+ // out better api that is Java compatible
+ assertTrue(res.isError());
+ java.util.List<Diagnostic> diags = res.getDiagnostics();
+ assertEquals(1, diags.size());
+ Diagnostic d = diags.get(0);
+ // System.err.println(d.getMessage());
+ assertTrue(d.getMessage().contains("int"));
+ assertTrue(d.getMessage().contains("Not an int"));
+ assertTrue(d.getDataLocations().toString().contains("10"));
+ java.util.List<LocationInSchemaFile> locs = d.getLocationsInSchemaFiles();
+ assertEquals(1, locs.size());
+ LocationInSchemaFile loc = locs.get(0);
+ assertTrue(loc.toString().contains("mySchema1.dfdl.xsd"));
+ // above is mySchema1 as it reports the element ref location,
+ // not element decl.
+ }
}
/**
@@ -282,21 +285,22 @@ public class TestJavaAPI {
java.io.File file = getResource("/test/japi/myData16.dat");
java.io.FileInputStream fis = new java.io.FileInputStream(file);
- InputSourceDataInputStream dis = new InputSourceDataInputStream(fis);
- JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
- ParseResult res = dp.parse(dis, outputter);
- boolean err = res.isError();
- assertFalse(err);
- assertEquals(2, res.location().bytePos1b());
- assertEquals(9, res.location().bitPos1b());
-
- java.io.ByteArrayOutputStream bos = new java.io.ByteArrayOutputStream();
- java.nio.channels.WritableByteChannel wbc = java.nio.channels.Channels.newChannel(bos);
- JDOMInfosetInputter inputter = new JDOMInfosetInputter(outputter.getResult());
- UnparseResult res2 = dp.unparse(inputter, wbc);
- err = res2.isError();
- assertFalse(err);
- assertEquals("9", bos.toString());
+ try (InputSourceDataInputStream dis = new InputSourceDataInputStream(fis)) {
+ JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
+ ParseResult res = dp.parse(dis, outputter);
+ boolean err = res.isError();
+ assertFalse(err);
+ assertEquals(2, res.location().bytePos1b());
+ assertEquals(9, res.location().bitPos1b());
+
+ java.io.ByteArrayOutputStream bos = new java.io.ByteArrayOutputStream();
+ java.nio.channels.WritableByteChannel wbc = java.nio.channels.Channels.newChannel(bos);
+ JDOMInfosetInputter inputter = new JDOMInfosetInputter(outputter.getResult());
+ UnparseResult res2 = dp.unparse(inputter, wbc);
+ err = res2.isError();
+ assertFalse(err);
+ assertEquals("9", bos.toString());
+ }
}
// This is a duplicate of test testJavaAPI3 that serializes the parser
@@ -322,21 +326,22 @@ public class TestJavaAPI {
java.io.File file = getResource("/test/japi/myData16.dat");
java.io.FileInputStream fis = new java.io.FileInputStream(file);
- InputSourceDataInputStream dis = new InputSourceDataInputStream(fis);
- JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
- ParseResult res = parser.parse(dis, outputter);
- boolean err = res.isError();
- assertFalse(err);
- assertEquals(2, res.location().bytePos1b());
- assertEquals(9, res.location().bitPos1b());
-
- java.io.ByteArrayOutputStream bos = new java.io.ByteArrayOutputStream();
- java.nio.channels.WritableByteChannel wbc = java.nio.channels.Channels.newChannel(bos);
- JDOMInfosetInputter inputter = new JDOMInfosetInputter(outputter.getResult());
- UnparseResult res2 = dp.unparse(inputter, wbc);
- err = res2.isError();
- assertFalse(err);
- assertEquals("9", bos.toString());
+ try (InputSourceDataInputStream dis = new InputSourceDataInputStream(fis)) {
+ JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
+ ParseResult res = parser.parse(dis, outputter);
+ boolean err = res.isError();
+ assertFalse(err);
+ assertEquals(2, res.location().bytePos1b());
+ assertEquals(9, res.location().bitPos1b());
+
+ java.io.ByteArrayOutputStream bos = new java.io.ByteArrayOutputStream();
+ java.nio.channels.WritableByteChannel wbc = java.nio.channels.Channels.newChannel(bos);
+ JDOMInfosetInputter inputter = new JDOMInfosetInputter(outputter.getResult());
+ UnparseResult res2 = dp.unparse(inputter, wbc);
+ err = res2.isError();
+ assertFalse(err);
+ assertEquals("9", bos.toString());
+ }
}
@Test
@@ -350,21 +355,22 @@ public class TestJavaAPI {
java.io.File file = getResource("/test/japi/myData2.dat");
java.io.FileInputStream fis = new java.io.FileInputStream(file);
- InputSourceDataInputStream dis = new InputSourceDataInputStream(fis);
- JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
- ParseResult res = dp.parse(dis, outputter);
- boolean err = res.isError();
- assertFalse(err);
- assertEquals(5, res.location().bytePos1b());
- assertEquals(33, res.location().bitPos1b());
-
- java.io.ByteArrayOutputStream bos = new java.io.ByteArrayOutputStream();
- java.nio.channels.WritableByteChannel wbc = java.nio.channels.Channels.newChannel(bos);
- JDOMInfosetInputter inputter = new JDOMInfosetInputter(outputter.getResult());
- UnparseResult res2 = dp.unparse(inputter, wbc);
- err = res2.isError();
- assertFalse(err);
- assertEquals("data", bos.toString());
+ try (InputSourceDataInputStream dis = new InputSourceDataInputStream(fis)) {
+ JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
+ ParseResult res = dp.parse(dis, outputter);
+ boolean err = res.isError();
+ assertFalse(err);
+ assertEquals(5, res.location().bytePos1b());
+ assertEquals(33, res.location().bitPos1b());
+
+ java.io.ByteArrayOutputStream bos = new java.io.ByteArrayOutputStream();
+ java.nio.channels.WritableByteChannel wbc = java.nio.channels.Channels.newChannel(bos);
+ JDOMInfosetInputter inputter = new JDOMInfosetInputter(outputter.getResult());
+ UnparseResult res2 = dp.unparse(inputter, wbc);
+ err = res2.isError();
+ assertFalse(err);
+ assertEquals("data", bos.toString());
+ }
}
@Test
@@ -376,23 +382,24 @@ public class TestJavaAPI {
dp = reserializeDataProcessor(dp);
java.io.File file = getResource("/test/japi/myData3.dat"); // contains 5
- // bytes
+ // bytes
java.io.FileInputStream fis = new java.io.FileInputStream(file);
- InputSourceDataInputStream dis = new InputSourceDataInputStream(fis);
- JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
- ParseResult res = dp.parse(dis, outputter);
- boolean err = res.isError();
- assertFalse(err);
- assertEquals(5, res.location().bytePos1b());
- assertEquals(33, res.location().bitPos1b());
-
- java.io.ByteArrayOutputStream bos = new java.io.ByteArrayOutputStream();
- java.nio.channels.WritableByteChannel wbc = java.nio.channels.Channels.newChannel(bos);
- JDOMInfosetInputter inputter = new JDOMInfosetInputter(outputter.getResult());
- UnparseResult res2 = dp.unparse(inputter, wbc);
- err = res2.isError();
- assertFalse(err);
- assertEquals("data", bos.toString());
+ try (InputSourceDataInputStream dis = new InputSourceDataInputStream(fis)) {
+ JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
+ ParseResult res = dp.parse(dis, outputter);
+ boolean err = res.isError();
+ assertFalse(err);
+ assertEquals(5, res.location().bytePos1b());
+ assertEquals(33, res.location().bitPos1b());
+
+ java.io.ByteArrayOutputStream bos = new java.io.ByteArrayOutputStream();
+ java.nio.channels.WritableByteChannel wbc = java.nio.channels.Channels.newChannel(bos);
+ JDOMInfosetInputter inputter = new JDOMInfosetInputter(outputter.getResult());
+ UnparseResult res2 = dp.unparse(inputter, wbc);
+ err = res2.isError();
+ assertFalse(err);
+ assertEquals("data", bos.toString());
+ }
}
/***
@@ -437,19 +444,20 @@ public class TestJavaAPI {
java.io.File file = getResource("/test/japi/01very_simple.txt");
java.io.FileInputStream fis = new java.io.FileInputStream(file);
- InputSourceDataInputStream dis = new InputSourceDataInputStream(fis);
- JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
- ParseResult res = dp.parse(dis, outputter);
- boolean err = res.isError();
- assertFalse(err);
-
- java.io.ByteArrayOutputStream bos = new java.io.ByteArrayOutputStream();
- java.nio.channels.WritableByteChannel wbc = java.nio.channels.Channels.newChannel(bos);
- JDOMInfosetInputter inputter = new JDOMInfosetInputter(outputter.getResult());
- UnparseResult res2 = dp.unparse(inputter, wbc);
- err = res2.isError();
- assertFalse(err);
- assertTrue(bos.toString().contains("Return-Path: <bo...@smith.com>"));
+ try (InputSourceDataInputStream dis = new InputSourceDataInputStream(fis)) {
+ JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
+ ParseResult res = dp.parse(dis, outputter);
+ boolean err = res.isError();
+ assertFalse(err);
+
+ java.io.ByteArrayOutputStream bos = new java.io.ByteArrayOutputStream();
+ java.nio.channels.WritableByteChannel wbc = java.nio.channels.Channels.newChannel(bos);
+ JDOMInfosetInputter inputter = new JDOMInfosetInputter(outputter.getResult());
+ UnparseResult res2 = dp.unparse(inputter, wbc);
+ err = res2.isError();
+ assertFalse(err);
+ assertTrue(bos.toString().contains("Return-Path: <bo...@smith.com>"));
+ }
}
/**
@@ -470,19 +478,20 @@ public class TestJavaAPI {
java.io.File file = getResource("/test/japi/01very_simple.txt");
java.io.FileInputStream fis = new java.io.FileInputStream(file);
- InputSourceDataInputStream dis = new InputSourceDataInputStream(fis);
- JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
- ParseResult res = dp.parse(dis, outputter);
- boolean err = res.isError();
- assertFalse(err);
-
- java.io.ByteArrayOutputStream bos = new java.io.ByteArrayOutputStream();
- java.nio.channels.WritableByteChannel wbc = java.nio.channels.Channels.newChannel(bos);
- JDOMInfosetInputter inputter = new JDOMInfosetInputter(outputter.getResult());
- UnparseResult res2 = dp.unparse(inputter, wbc);
- err = res2.isError();
- assertFalse(err);
- assertTrue(bos.toString().contains("Return-Path: <bo...@smith.com>"));
+ try (InputSourceDataInputStream dis = new InputSourceDataInputStream(fis)) {
+ JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
+ ParseResult res = dp.parse(dis, outputter);
+ boolean err = res.isError();
+ assertFalse(err);
+
+ java.io.ByteArrayOutputStream bos = new java.io.ByteArrayOutputStream();
+ java.nio.channels.WritableByteChannel wbc = java.nio.channels.Channels.newChannel(bos);
+ JDOMInfosetInputter inputter = new JDOMInfosetInputter(outputter.getResult());
+ UnparseResult res2 = dp.unparse(inputter, wbc);
+ err = res2.isError();
+ assertFalse(err);
+ assertTrue(bos.toString().contains("Return-Path: <bo...@smith.com>"));
+ }
}
/**
@@ -499,31 +508,32 @@ public class TestJavaAPI {
java.io.File file = getResource("/test/japi/01very_simple.txt");
java.io.FileInputStream fis = new java.io.FileInputStream(file);
- InputSourceDataInputStream dis = new InputSourceDataInputStream(fis);
- JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
- ParseResult res = dp.parse(dis, outputter);
- boolean err = res.isError();
- assertFalse(err);
-
- org.jdom2.Document doc1 = outputter.getResult();
-
- java.io.ByteArrayOutputStream bos1 = new java.io.ByteArrayOutputStream();
- java.nio.channels.WritableByteChannel wbc1 = java.nio.channels.Channels.newChannel(bos1);
- JDOMInfosetInputter inputter1 = new JDOMInfosetInputter(doc1);
- UnparseResult res2 = dp.unparse(inputter1, wbc1);
- err = res2.isError();
- assertFalse(err);
- assertTrue(bos1.toString().contains("Return-Path: <bo...@smith.com>"));
-
- org.jdom2.Document doc2 = outputter.getResult();
-
- java.io.ByteArrayOutputStream bos2 = new java.io.ByteArrayOutputStream();
- java.nio.channels.WritableByteChannel wbc2 = java.nio.channels.Channels.newChannel(bos2);
- JDOMInfosetInputter inputter2 = new JDOMInfosetInputter(doc2);
- UnparseResult res3 = dp.unparse(inputter2, wbc2);
- err = res3.isError();
- assertFalse(err);
- assertTrue(bos2.toString().contains("Return-Path: <bo...@smith.com>"));
+ try (InputSourceDataInputStream dis = new InputSourceDataInputStream(fis)) {
+ JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
+ ParseResult res = dp.parse(dis, outputter);
+ boolean err = res.isError();
+ assertFalse(err);
+
+ org.jdom2.Document doc1 = outputter.getResult();
+
+ java.io.ByteArrayOutputStream bos1 = new java.io.ByteArrayOutputStream();
+ java.nio.channels.WritableByteChannel wbc1 = java.nio.channels.Channels.newChannel(bos1);
+ JDOMInfosetInputter inputter1 = new JDOMInfosetInputter(doc1);
+ UnparseResult res2 = dp.unparse(inputter1, wbc1);
+ err = res2.isError();
+ assertFalse(err);
+ assertTrue(bos1.toString().contains("Return-Path: <bo...@smith.com>"));
+
+ org.jdom2.Document doc2 = outputter.getResult();
+
+ java.io.ByteArrayOutputStream bos2 = new java.io.ByteArrayOutputStream();
+ java.nio.channels.WritableByteChannel wbc2 = java.nio.channels.Channels.newChannel(bos2);
+ JDOMInfosetInputter inputter2 = new JDOMInfosetInputter(doc2);
+ UnparseResult res3 = dp.unparse(inputter2, wbc2);
+ err = res3.isError();
+ assertFalse(err);
+ assertTrue(bos2.toString().contains("Return-Path: <bo...@smith.com>"));
+ }
}
/**
@@ -540,17 +550,18 @@ public class TestJavaAPI {
java.io.File file = getResource("/test/japi/myData4.dat");
java.io.FileInputStream fis = new java.io.FileInputStream(file);
- InputSourceDataInputStream dis = new InputSourceDataInputStream(fis);
- JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
- ParseResult res = dp.parse(dis, outputter);
- boolean err = res.isError();
- assertFalse(err);
- org.jdom2.Document doc = outputter.getResult();
- org.jdom2.output.XMLOutputter xo = new org.jdom2.output.XMLOutputter();
- xo.setFormat(Format.getPrettyFormat());
- org.jdom2.Element rootNode = doc.getRootElement();
- org.jdom2.Element hidden = rootNode.getChild("hiddenElement", rootNode.getNamespace());
- assertTrue(null == hidden);
+ try (InputSourceDataInputStream dis = new InputSourceDataInputStream(fis)) {
+ JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
+ ParseResult res = dp.parse(dis, outputter);
+ boolean err = res.isError();
+ assertFalse(err);
+ org.jdom2.Document doc = outputter.getResult();
+ org.jdom2.output.XMLOutputter xo = new org.jdom2.output.XMLOutputter();
+ xo.setFormat(Format.getPrettyFormat());
+ org.jdom2.Element rootNode = doc.getRootElement();
+ org.jdom2.Element hidden = rootNode.getChild("hiddenElement", rootNode.getNamespace());
+ assertTrue(null == hidden);
+ }
}
/**
@@ -567,30 +578,31 @@ public class TestJavaAPI {
java.io.File file = getResource("/test/japi/myData5.dat");
java.io.FileInputStream fis = new java.io.FileInputStream(file);
- InputSourceDataInputStream dis = new InputSourceDataInputStream(fis);
- JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
- ParseResult res = dp.parse(dis, outputter);
- boolean err = res.isError();
- assertFalse(err);
- org.jdom2.Document doc = outputter.getResult();
- org.jdom2.output.XMLOutputter xo = new org.jdom2.output.XMLOutputter();
- xo.setFormat(Format.getPrettyFormat());
- // xo.output(doc, System.out);
- org.jdom2.Element rootNode = doc.getRootElement();
- org.jdom2.Element elementGroup = rootNode.getChild("elementGroup", null); // local
- // element
- // names
- // are
- // unqualified
- assertTrue(null != elementGroup);
- org.jdom2.Element groupE2 = elementGroup.getChild("e2", null);
- assertTrue(null != groupE2);
- org.jdom2.Element groupE3 = elementGroup.getChild("e3", null);
- assertTrue(null != groupE3);
- org.jdom2.Element rootE2 = rootNode.getChild("e2", null);
- assertTrue(null == rootE2);
- org.jdom2.Element rootE3 = rootNode.getChild("e3", null);
- assertTrue(null == rootE3);
+ try (InputSourceDataInputStream dis = new InputSourceDataInputStream(fis)) {
+ JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
+ ParseResult res = dp.parse(dis, outputter);
+ boolean err = res.isError();
+ assertFalse(err);
+ org.jdom2.Document doc = outputter.getResult();
+ org.jdom2.output.XMLOutputter xo = new org.jdom2.output.XMLOutputter();
+ xo.setFormat(Format.getPrettyFormat());
+ // xo.output(doc, System.out);
+ org.jdom2.Element rootNode = doc.getRootElement();
+ org.jdom2.Element elementGroup = rootNode.getChild("elementGroup", null); // local
+ // element
+ // names
+ // are
+ // unqualified
+ assertTrue(null != elementGroup);
+ org.jdom2.Element groupE2 = elementGroup.getChild("e2", null);
+ assertTrue(null != groupE2);
+ org.jdom2.Element groupE3 = elementGroup.getChild("e3", null);
+ assertTrue(null != groupE3);
+ org.jdom2.Element rootE2 = rootNode.getChild("e2", null);
+ assertTrue(null == rootE2);
+ org.jdom2.Element rootE3 = rootNode.getChild("e3", null);
+ assertTrue(null == rootE3);
+ }
}
@Test
@@ -609,14 +621,15 @@ public class TestJavaAPI {
java.io.File file = getResource("/test/japi/myData.dat");
java.io.FileInputStream fis = new java.io.FileInputStream(file);
- InputSourceDataInputStream dis = new InputSourceDataInputStream(fis);
- JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
- ParseResult res = dp.parse(dis, outputter);
- boolean err = res.isError();
- assertFalse(err);
+ try (InputSourceDataInputStream dis = new InputSourceDataInputStream(fis)) {
+ JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
+ ParseResult res = dp.parse(dis, outputter);
+ boolean err = res.isError();
+ assertFalse(err);
- assertTrue(debugger.lines.size() > 0);
- assertTrue(debugger.lines.contains("----------------------------------------------------------------- 1\n"));
+ assertTrue(debugger.lines.size() > 0);
+ assertTrue(debugger.lines.contains("----------------------------------------------------------------- 1\n"));
+ }
}
@Test
@@ -639,19 +652,20 @@ public class TestJavaAPI {
java.io.File file = getResource("/test/japi/myData.dat");
java.io.FileInputStream fis = new java.io.FileInputStream(file);
- InputSourceDataInputStream dis = new InputSourceDataInputStream(fis);
- JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
- ParseResult res = dp.parse(dis, outputter);
- boolean err = res.isError();
- assertFalse(err);
- org.jdom2.Document doc = outputter.getResult();
- org.jdom2.output.XMLOutputter xo = new org.jdom2.output.XMLOutputter();
- xo.setFormat(Format.getPrettyFormat());
- String docString = xo.outputString(doc);
- boolean containsVar1 = docString.contains("var1Value");
- boolean containsVar1Value = docString.contains("externallySet");
- assertTrue(containsVar1);
- assertTrue(containsVar1Value);
+ try (InputSourceDataInputStream dis = new InputSourceDataInputStream(fis)) {
+ JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
+ ParseResult res = dp.parse(dis, outputter);
+ boolean err = res.isError();
+ assertFalse(err);
+ org.jdom2.Document doc = outputter.getResult();
+ org.jdom2.output.XMLOutputter xo = new org.jdom2.output.XMLOutputter();
+ xo.setFormat(Format.getPrettyFormat());
+ String docString = xo.outputString(doc);
+ boolean containsVar1 = docString.contains("var1Value");
+ boolean containsVar1Value = docString.contains("externallySet");
+ assertTrue(containsVar1);
+ assertTrue(containsVar1Value);
+ }
}
@Test
@@ -673,22 +687,23 @@ public class TestJavaAPI {
java.io.File file = getResource("/test/japi/myData.dat");
java.io.FileInputStream fis = new java.io.FileInputStream(file);
- InputSourceDataInputStream dis = new InputSourceDataInputStream(fis);
- JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
- ParseResult res = dp.parse(dis, outputter);
- boolean err = res.isError();
- assertFalse(err);
- org.jdom2.Document doc = outputter.getResult();
- org.jdom2.output.XMLOutputter xo = new org.jdom2.output.XMLOutputter();
- xo.setFormat(Format.getPrettyFormat());
- String docString = xo.outputString(doc);
- boolean containsVar1 = docString.contains("var1Value");
- boolean containsVar1Value = docString.contains("externallySet");
- assertTrue(containsVar1);
- assertTrue(containsVar1Value);
-
- assertTrue(debugger.lines.size() > 0);
- assertTrue(debugger.lines.contains("----------------------------------------------------------------- 1\n"));
+ try (InputSourceDataInputStream dis = new InputSourceDataInputStream(fis)) {
+ JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
+ ParseResult res = dp.parse(dis, outputter);
+ boolean err = res.isError();
+ assertFalse(err);
+ org.jdom2.Document doc = outputter.getResult();
+ org.jdom2.output.XMLOutputter xo = new org.jdom2.output.XMLOutputter();
+ xo.setFormat(Format.getPrettyFormat());
+ String docString = xo.outputString(doc);
+ boolean containsVar1 = docString.contains("var1Value");
+ boolean containsVar1Value = docString.contains("externallySet");
+ assertTrue(containsVar1);
+ assertTrue(containsVar1Value);
+
+ assertTrue(debugger.lines.size() > 0);
+ assertTrue(debugger.lines.contains("----------------------------------------------------------------- 1\n"));
+ }
}
@Test
@@ -736,19 +751,20 @@ public class TestJavaAPI {
java.io.File file = getResource("/test/japi/myData.dat");
java.io.FileInputStream fis = new java.io.FileInputStream(file);
- InputSourceDataInputStream dis = new InputSourceDataInputStream(fis);
- JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
- ParseResult res = dp.parse(dis, outputter);
- assertTrue(res.isError());
- assertFalse(res.isProcessingError());
- assertTrue(res.isValidationError());
-
- java.util.List<Diagnostic> diags = res.getDiagnostics();
- assertEquals(1, diags.size());
- Diagnostic d = diags.get(0);
- assertTrue(d.getMessage().contains("maxInclusive"));
- assertTrue(d.getMessage().contains("e2"));
- assertTrue(d.getMessage().contains("20"));
+ try (InputSourceDataInputStream dis = new InputSourceDataInputStream(fis)) {
+ JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
+ ParseResult res = dp.parse(dis, outputter);
+ assertTrue(res.isError());
+ assertFalse(res.isProcessingError());
+ assertTrue(res.isValidationError());
+
+ java.util.List<Diagnostic> diags = res.getDiagnostics();
+ assertEquals(1, diags.size());
+ Diagnostic d = diags.get(0);
+ assertTrue(d.getMessage().contains("maxInclusive"));
+ assertTrue(d.getMessage().contains("e2"));
+ assertTrue(d.getMessage().contains("20"));
+ }
}
@Test
@@ -762,104 +778,108 @@ public class TestJavaAPI {
java.io.File file = getResource("/test/japi/myData.dat");
java.io.FileInputStream fis = new java.io.FileInputStream(file);
- InputSourceDataInputStream dis = new InputSourceDataInputStream(fis);
- JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
- ParseResult res = dp.parse(dis, outputter);
- assertTrue(res.isError());
- assertFalse(res.isProcessingError());
- assertTrue(res.isValidationError());
- long actualLength = res.location().bytePos1b() - 1;
- assertEquals(file.length(), actualLength);
-
- java.util.List<Diagnostic> diags = res.getDiagnostics();
- assertEquals(3, diags.size());
- Diagnostic d0 = diags.get(0);
- Diagnostic d1 = diags.get(1);
- Diagnostic d2 = diags.get(2);
-
- assertTrue(d0.getMessage().contains("42"));
- assertTrue(d0.getMessage().contains("e2"));
- assertTrue(d0.getMessage().contains("not valid"));
-
- assertTrue(d1.getMessage().contains("42"));
- assertTrue(d1.getMessage().contains("maxInclusive"));
- assertTrue(d1.getMessage().contains("20"));
-
- assertTrue(d2.getMessage().contains("maxInclusive"));
- assertTrue(d2.getMessage().contains("e2"));
- assertTrue(d2.getMessage().contains("20"));
+ try (InputSourceDataInputStream dis = new InputSourceDataInputStream(fis)) {
+ JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
+ ParseResult res = dp.parse(dis, outputter);
+ assertTrue(res.isError());
+ assertFalse(res.isProcessingError());
+ assertTrue(res.isValidationError());
+ long actualLength = res.location().bytePos1b() - 1;
+ assertEquals(file.length(), actualLength);
+
+ java.util.List<Diagnostic> diags = res.getDiagnostics();
+ assertEquals(3, diags.size());
+ Diagnostic d0 = diags.get(0);
+ Diagnostic d1 = diags.get(1);
+ Diagnostic d2 = diags.get(2);
+
+ assertTrue(d0.getMessage().contains("42"));
+ assertTrue(d0.getMessage().contains("e2"));
+ assertTrue(d0.getMessage().contains("not valid"));
+
+ assertTrue(d1.getMessage().contains("42"));
+ assertTrue(d1.getMessage().contains("maxInclusive"));
+ assertTrue(d1.getMessage().contains("20"));
+
+ assertTrue(d2.getMessage().contains("maxInclusive"));
+ assertTrue(d2.getMessage().contains("e2"));
+ assertTrue(d2.getMessage().contains("20"));
+ }
}
@Test
public void testJavaAPI18() throws IOException, ClassNotFoundException {
- // Demonstrate that we can use the API to continue a parse where we left off
- org.apache.daffodil.japi.Compiler c = Daffodil.compiler();
-
- java.io.File schemaFile = getResource("/test/japi/mySchema3.dfdl.xsd");
- ProcessorFactory pf = c.compileFile(schemaFile,"e4", null);
- DataProcessor dp = pf.onPath("/");
- dp = reserializeDataProcessor(dp);
-
- java.io.File file = getResource("/test/japi/myData2.dat");
- java.io.FileInputStream fis = new java.io.FileInputStream(file);
- InputSourceDataInputStream input = new InputSourceDataInputStream(fis);
-
- JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
- ParseResult res = null;
- boolean err = false;
-
- res = dp.parse(input, outputter);
- err = res.isError();
- assertFalse(err);
- assertEquals(5, res.location().bytePos1b());
- assertEquals("data", outputter.getResult().getRootElement().getText());
-
- outputter.reset();
- res = dp.parse(input, outputter);
- err = res.isError();
- assertFalse(err);
- assertEquals(9, res.location().bytePos1b());
- assertEquals("left", outputter.getResult().getRootElement().getText());
-
- outputter.reset();
- res = dp.parse(input, outputter);
- err = res.isError();
- assertFalse(err);
- assertFalse(input.hasData());
- assertEquals(13, res.location().bytePos1b());
- assertEquals("over", outputter.getResult().getRootElement().getText());
+ // Demonstrate that we can use the API to continue a parse where we left off
+ org.apache.daffodil.japi.Compiler c = Daffodil.compiler();
+
+ java.io.File schemaFile = getResource("/test/japi/mySchema3.dfdl.xsd");
+ ProcessorFactory pf = c.compileFile(schemaFile,"e4", null);
+ DataProcessor dp = pf.onPath("/");
+ dp = reserializeDataProcessor(dp);
+
+ java.io.File file = getResource("/test/japi/myData2.dat");
+ java.io.FileInputStream fis = new java.io.FileInputStream(file);
+ try (InputSourceDataInputStream input = new InputSourceDataInputStream(fis)) {
+
+ JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
+ ParseResult res = null;
+ boolean err = false;
+
+ res = dp.parse(input, outputter);
+ err = res.isError();
+ assertFalse(err);
+ assertEquals(5, res.location().bytePos1b());
+ assertEquals("data", outputter.getResult().getRootElement().getText());
+
+ outputter.reset();
+ res = dp.parse(input, outputter);
+ err = res.isError();
+ assertFalse(err);
+ assertEquals(9, res.location().bytePos1b());
+ assertEquals("left", outputter.getResult().getRootElement().getText());
+
+ outputter.reset();
+ res = dp.parse(input, outputter);
+ err = res.isError();
+ assertFalse(err);
+ assertFalse(input.hasData());
+ assertEquals(13, res.location().bytePos1b());
+ assertEquals("over", outputter.getResult().getRootElement().getText());
+ }
}
@Test
public void testJavaAPI19() throws IOException, ClassNotFoundException {
- // Demonstrate that we cannot use the API to continue a parse with an invalid InputSource
- // ie. after a runtime SDE. This test needs to be run with an input file larger than 256MB
- org.apache.daffodil.japi.Compiler c = Daffodil.compiler();
-
- java.io.File schemaFile = getResource("/test/japi/ambig_elt.dfdl.xsd");
- ProcessorFactory pf = c.compileFile(schemaFile, "root", null);
- DataProcessor dp = pf.onPath("/");
- dp = reserializeDataProcessor(dp);
-
- java.io.File file = getResource("/test/japi/myData19.dat");
- java.io.FileInputStream fis = new java.io.FileInputStream(file);
- InputSourceDataInputStream input = new InputSourceDataInputStream(fis);
-
- JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
- ParseResult res = null;
- boolean err = false;
-
- res = dp.parse(input, outputter);
- err = res.isError();
- assertTrue(err);
-
- outputter.reset();
- try {
- res = dp.parse(input, outputter);
- } catch (Exception e) {
- assertTrue(e.getMessage().contains("Usage error"));
- assertTrue(e.getMessage().contains("invalid input source"));
- }
+ // Demonstrate that we cannot use the API to continue a parse with an invalid InputSource
+ // ie. after a runtime SDE. This test needs to be run with an input file larger than 256MB
+ org.apache.daffodil.japi.Compiler c = Daffodil.compiler();
+
+ java.io.File schemaFile = getResource("/test/japi/ambig_elt.dfdl.xsd");
+ ProcessorFactory pf = c.compileFile(schemaFile, "root", null);
+ DataProcessor dp = pf.onPath("/");
+ dp = reserializeDataProcessor(dp);
+
+ java.io.File file = getResource("/test/japi/myData19.dat");
+ java.io.FileInputStream fis = new java.io.FileInputStream(file);
+ try (InputSourceDataInputStream input = new InputSourceDataInputStream(fis)) {
+
+ JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
+ ParseResult res = null;
+ boolean err = false;
+
+ res = dp.parse(input, outputter);
+ err = res.isError();
+ assertTrue(err);
+
+ outputter.reset();
+ try {
+ res = dp.parse(input, outputter);
+
+ } catch (Exception e) {
+ assertTrue(e.getMessage().contains("Usage error"));
+ assertTrue(e.getMessage().contains("invalid input source"));
+ }
+ }
}
@Test
@@ -875,65 +895,66 @@ public class TestJavaAPI {
java.io.File file = getResource("/test/japi/myData.dat");
java.io.FileInputStream fisDP = new java.io.FileInputStream(file);
java.io.FileInputStream fisSAX = new java.io.FileInputStream(file);
- InputSourceDataInputStream disDP = new InputSourceDataInputStream(fisDP);
- InputSourceDataInputStream disSAX = new InputSourceDataInputStream(fisSAX);
- ByteArrayOutputStream xmlBos = new ByteArrayOutputStream();
- XMLTextInfosetOutputter outputter = new XMLTextInfosetOutputter(xmlBos, true);
- ParseResult res = dp.parse(disDP, outputter);
- String infosetDPString = xmlBos.toString();
-
- org.jdom2.input.sax.SAXHandler contentHandler = new org.jdom2.input.sax.SAXHandler();
- SAXErrorHandlerForJAPITest errorHandler = new SAXErrorHandlerForJAPITest();
- // since SAXHandler uses a blank prefix when the below isn't set to true, it introduces
- // an undesired no-prefixed xmlns mapping
- parseXMLReader.setFeature(SAX_NAMESPACE_PREFIXES_FEATURE, true);
- parseXMLReader.setContentHandler(contentHandler);
- parseXMLReader.setErrorHandler(errorHandler);
- parseXMLReader.setProperty(DaffodilParseXMLReader.DAFFODIL_SAX_URN_BLOBDIRECTORY(),
- Paths.get(System.getProperty("java.io.tmpdir")));
- parseXMLReader.setProperty(DaffodilParseXMLReader.DAFFODIL_SAX_URN_BLOBPREFIX(),
- "daffodil-sapi-");
- parseXMLReader.setProperty(DaffodilParseXMLReader.DAFFODIL_SAX_URN_BLOBSUFFIX(),
- ".sax.blob");
- parseXMLReader.parse(disSAX);
- ParseResult resSAX = (ParseResult) parseXMLReader.getProperty(DaffodilParseXMLReader.DAFFODIL_SAX_URN_PARSERESULT());
- boolean err = errorHandler.isError();
- ArrayList<Diagnostic> diags = errorHandler.getDiagnostics();
- Format pretty = Format.getPrettyFormat().setLineSeparator(System.getProperty("line.separator"));
- String infosetSAXString = new org.jdom2.output.XMLOutputter(pretty).outputString(contentHandler.getDocument());
-
- assertFalse(err);
- assertTrue(diags.isEmpty());
- assertEquals(infosetDPString, infosetSAXString);
-
- // test unparse
- ByteArrayOutputStream unparseBos = new java.io.ByteArrayOutputStream();
- WritableByteChannel wbc = java.nio.channels.Channels.newChannel(unparseBos);
-
- // prep for SAX unparse
- DaffodilUnparseContentHandler unparseContentHandler = dp.newContentHandlerInstance(wbc);
- try {
- org.xml.sax.XMLReader unparseXMLReader = javax.xml.parsers.SAXParserFactory.newInstance()
- .newSAXParser().getXMLReader();
- setSecureDefaults(unparseXMLReader);
- unparseXMLReader.setContentHandler(unparseContentHandler);
- unparseXMLReader.setErrorHandler(errorHandler);
- unparseXMLReader.setFeature(SAX_NAMESPACES_FEATURE, true);
- unparseXMLReader.setFeature(SAX_NAMESPACE_PREFIXES_FEATURE, true);
- ByteArrayInputStream is = new ByteArrayInputStream(infosetSAXString.getBytes());
- // kickstart unparse
- unparseXMLReader.parse(new org.xml.sax.InputSource(is));
- } catch (javax.xml.parsers.ParserConfigurationException | org.xml.sax.SAXException e) {
- fail("Error: " + e);
- }
+ try (InputSourceDataInputStream disDP = new InputSourceDataInputStream(fisDP)) {
+ InputSourceDataInputStream disSAX = new InputSourceDataInputStream(fisSAX);
+ ByteArrayOutputStream xmlBos = new ByteArrayOutputStream();
+ XMLTextInfosetOutputter outputter = new XMLTextInfosetOutputter(xmlBos, true);
+ ParseResult res = dp.parse(disDP, outputter);
+ String infosetDPString = xmlBos.toString();
+
+ org.jdom2.input.sax.SAXHandler contentHandler = new org.jdom2.input.sax.SAXHandler();
+ SAXErrorHandlerForJAPITest errorHandler = new SAXErrorHandlerForJAPITest();
+ // since SAXHandler uses a blank prefix when the below isn't set to true, it introduces
+ // an undesired no-prefixed xmlns mapping
+ parseXMLReader.setFeature(SAX_NAMESPACE_PREFIXES_FEATURE, true);
+ parseXMLReader.setContentHandler(contentHandler);
+ parseXMLReader.setErrorHandler(errorHandler);
+ parseXMLReader.setProperty(DaffodilParseXMLReader.DAFFODIL_SAX_URN_BLOBDIRECTORY(),
+ Paths.get(System.getProperty("java.io.tmpdir")));
+ parseXMLReader.setProperty(DaffodilParseXMLReader.DAFFODIL_SAX_URN_BLOBPREFIX(),
+ "daffodil-sapi-");
+ parseXMLReader.setProperty(DaffodilParseXMLReader.DAFFODIL_SAX_URN_BLOBSUFFIX(),
+ ".sax.blob");
+ parseXMLReader.parse(disSAX);
+ ParseResult resSAX = (ParseResult) parseXMLReader.getProperty(DaffodilParseXMLReader.DAFFODIL_SAX_URN_PARSERESULT());
+ boolean err = errorHandler.isError();
+ ArrayList<Diagnostic> diags = errorHandler.getDiagnostics();
+ Format pretty = Format.getPrettyFormat().setLineSeparator(System.getProperty("line.separator"));
+ String infosetSAXString = new org.jdom2.output.XMLOutputter(pretty).outputString(contentHandler.getDocument());
+
+ assertFalse(err);
+ assertTrue(diags.isEmpty());
+ assertEquals(infosetDPString, infosetSAXString);
+
+ // test unparse
+ ByteArrayOutputStream unparseBos = new java.io.ByteArrayOutputStream();
+ WritableByteChannel wbc = java.nio.channels.Channels.newChannel(unparseBos);
+
+ // prep for SAX unparse
+ DaffodilUnparseContentHandler unparseContentHandler = dp.newContentHandlerInstance(wbc);
+ try {
+ org.xml.sax.XMLReader unparseXMLReader = javax.xml.parsers.SAXParserFactory.newInstance()
+ .newSAXParser().getXMLReader();
+ setSecureDefaults(unparseXMLReader);
+ unparseXMLReader.setContentHandler(unparseContentHandler);
+ unparseXMLReader.setErrorHandler(errorHandler);
+ unparseXMLReader.setFeature(SAX_NAMESPACES_FEATURE, true);
+ unparseXMLReader.setFeature(SAX_NAMESPACE_PREFIXES_FEATURE, true);
+ ByteArrayInputStream is = new ByteArrayInputStream(infosetSAXString.getBytes());
+ // kickstart unparse
+ unparseXMLReader.parse(new org.xml.sax.InputSource(is));
+ } catch (javax.xml.parsers.ParserConfigurationException | org.xml.sax.SAXException e) {
+ fail("Error: " + e);
+ }
- UnparseResult saxUr = unparseContentHandler.getUnparseResult();
- wbc.close();
+ UnparseResult saxUr = unparseContentHandler.getUnparseResult();
+ wbc.close();
- boolean saxErr = saxUr.isError();
- assertFalse(saxErr);
- assertTrue(saxUr.getDiagnostics().isEmpty());
- assertEquals("42", unparseBos.toString());
+ boolean saxErr = saxUr.isError();
+ assertFalse(saxErr);
+ assertTrue(saxUr.getDiagnostics().isEmpty());
+ assertEquals("42", unparseBos.toString());
+ }
}
@Test
@@ -948,30 +969,31 @@ public class TestJavaAPI {
java.io.File file = getResource("/test/japi/myDataBroken.dat");
java.io.FileInputStream fis = new java.io.FileInputStream(file);
- InputSourceDataInputStream dis = new InputSourceDataInputStream(fis);
-
- org.jdom2.input.sax.SAXHandler contentHandler = new org.jdom2.input.sax.SAXHandler();
- SAXErrorHandlerForJAPITest errorHandler = new SAXErrorHandlerForJAPITest();
- parseXMLReader.setContentHandler(contentHandler);
- parseXMLReader.setErrorHandler(errorHandler);
- parseXMLReader.setProperty(DaffodilParseXMLReader.DAFFODIL_SAX_URN_BLOBDIRECTORY(),
- Paths.get(System.getProperty("java.io.tmpdir")));
- parseXMLReader.setProperty(DaffodilParseXMLReader.DAFFODIL_SAX_URN_BLOBPREFIX(), "daffodil-sapi-");
- parseXMLReader.setProperty(DaffodilParseXMLReader.DAFFODIL_SAX_URN_BLOBSUFFIX(), ".sax.blob");
- parseXMLReader.parse(dis);
- boolean err = errorHandler.isError();
- ArrayList<Diagnostic> diags = errorHandler.getDiagnostics();
-
- assertTrue(err);
- assertEquals(1, diags.size());
- Diagnostic d = diags.get(0);
- assertTrue(d.getMessage().contains("int"));
- assertTrue(d.getMessage().contains("Not an int"));
- assertTrue(d.getDataLocations().toString().contains("10"));
- java.util.List<LocationInSchemaFile> locs = d.getLocationsInSchemaFiles();
- assertEquals(1, locs.size());
- LocationInSchemaFile loc = locs.get(0);
- assertTrue(loc.toString().contains("mySchema1.dfdl.xsd"));
+ try (InputSourceDataInputStream dis = new InputSourceDataInputStream(fis)) {
+
+ org.jdom2.input.sax.SAXHandler contentHandler = new org.jdom2.input.sax.SAXHandler();
+ SAXErrorHandlerForJAPITest errorHandler = new SAXErrorHandlerForJAPITest();
+ parseXMLReader.setContentHandler(contentHandler);
+ parseXMLReader.setErrorHandler(errorHandler);
+ parseXMLReader.setProperty(DaffodilParseXMLReader.DAFFODIL_SAX_URN_BLOBDIRECTORY(),
+ Paths.get(System.getProperty("java.io.tmpdir")));
+ parseXMLReader.setProperty(DaffodilParseXMLReader.DAFFODIL_SAX_URN_BLOBPREFIX(), "daffodil-sapi-");
+ parseXMLReader.setProperty(DaffodilParseXMLReader.DAFFODIL_SAX_URN_BLOBSUFFIX(), ".sax.blob");
+ parseXMLReader.parse(dis);
+ boolean err = errorHandler.isError();
+ ArrayList<Diagnostic> diags = errorHandler.getDiagnostics();
+
+ assertTrue(err);
+ assertEquals(1, diags.size());
+ Diagnostic d = diags.get(0);
+ assertTrue(d.getMessage().contains("int"));
+ assertTrue(d.getMessage().contains("Not an int"));
+ assertTrue(d.getDataLocations().toString().contains("10"));
+ java.util.List<LocationInSchemaFile> locs = d.getLocationsInSchemaFiles();
+ assertEquals(1, locs.size());
+ LocationInSchemaFile loc = locs.get(0);
+ assertTrue(loc.toString().contains("mySchema1.dfdl.xsd"));
+ }
}
@Test
@@ -996,22 +1018,23 @@ public class TestJavaAPI {
java.io.File file = getResource("/test/japi/myData.dat");
java.io.FileInputStream fis = new java.io.FileInputStream(file);
- InputSourceDataInputStream dis = new InputSourceDataInputStream(fis);
- JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
- ParseResult res = dp.parse(dis, outputter);
- boolean err = res.isError();
- assertFalse(err);
- org.jdom2.Document doc = outputter.getResult();
- org.jdom2.output.XMLOutputter xo = new org.jdom2.output.XMLOutputter();
- xo.setFormat(Format.getPrettyFormat());
- String docString = xo.outputString(doc);
- boolean containsVar1 = docString.contains("var1Value");
- boolean containsVar1Value = docString.contains("var1ValueFromMap");
- assertTrue(containsVar1);
- assertTrue(containsVar1Value);
-
- assertTrue(debugger.lines.size() > 0);
- assertTrue(debugger.lines.contains("----------------------------------------------------------------- 1\n"));
+ try (InputSourceDataInputStream dis = new InputSourceDataInputStream(fis)) {
+ JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
+ ParseResult res = dp.parse(dis, outputter);
+ boolean err = res.isError();
+ assertFalse(err);
+ org.jdom2.Document doc = outputter.getResult();
+ org.jdom2.output.XMLOutputter xo = new org.jdom2.output.XMLOutputter();
+ xo.setFormat(Format.getPrettyFormat());
+ String docString = xo.outputString(doc);
+ boolean containsVar1 = docString.contains("var1Value");
+ boolean containsVar1Value = docString.contains("var1ValueFromMap");
+ assertTrue(containsVar1);
+ assertTrue(containsVar1Value);
+
+ assertTrue(debugger.lines.size() > 0);
+ assertTrue(debugger.lines.contains("----------------------------------------------------------------- 1\n"));
+ }
}
@Test
@@ -1139,15 +1162,16 @@ public class TestJavaAPI {
{
byte[] ba = {};
ByteBuffer bb = ByteBuffer.wrap(ba);
- InputSourceDataInputStream dis = new InputSourceDataInputStream(bb);
- JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
- ParseResult res = dp.parse(dis, outputter);
- assertFalse(res.isError());
- org.jdom2.Document doc = outputter.getResult();
- org.jdom2.output.XMLOutputter xo = new org.jdom2.output.XMLOutputter();
- xo.setFormat(Format.getPrettyFormat());
- String docString = xo.outputString(doc);
- assertTrue(docString.contains("<ex1var>200</ex1var>"));
+ try (InputSourceDataInputStream dis = new InputSourceDataInputStream(bb)) {
+ JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
+ ParseResult res = dp.parse(dis, outputter);
+ assertFalse(res.isError());
+ org.jdom2.Document doc = outputter.getResult();
+ org.jdom2.output.XMLOutputter xo = new org.jdom2.output.XMLOutputter();
+ xo.setFormat(Format.getPrettyFormat());
+ String docString = xo.outputString(doc);
+ assertTrue(docString.contains("<ex1var>200</ex1var>"));
+ }
}
// can set an external variable after a parse
@@ -1159,15 +1183,16 @@ public class TestJavaAPI {
{
byte[] ba = {};
ByteBuffer bb = ByteBuffer.wrap(ba);
- InputSourceDataInputStream dis = new InputSourceDataInputStream(bb);
- JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
- ParseResult res = dp.parse(dis, outputter);
- assertFalse(res.isError());
- org.jdom2.Document doc = outputter.getResult();
- org.jdom2.output.XMLOutputter xo = new org.jdom2.output.XMLOutputter();
- xo.setFormat(Format.getPrettyFormat());
- String docString = xo.outputString(doc);
- assertTrue(docString.contains("<ex1var>300</ex1var>"));
+ try (InputSourceDataInputStream dis = new InputSourceDataInputStream(bb)) {
+ JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
+ ParseResult res = dp.parse(dis, outputter);
+ assertFalse(res.isError());
+ org.jdom2.Document doc = outputter.getResult();
+ org.jdom2.output.XMLOutputter xo = new org.jdom2.output.XMLOutputter();
+ xo.setFormat(Format.getPrettyFormat());
+ String docString = xo.outputString(doc);
+ assertTrue(docString.contains("<ex1var>300</ex1var>"));
+ }
}
}
@@ -1177,12 +1202,12 @@ public class TestJavaAPI {
String expectedData = "42";
TestInfosetEvent expectedEvents[] = {
- TestInfosetEvent.startDocument(),
- TestInfosetEvent.startComplex("e1", "http://example.com"),
- TestInfosetEvent.startSimple("e2", "http://example.com", expectedData),
- TestInfosetEvent.endSimple("e2", "http://example.com"),
- TestInfosetEvent.endComplex("e1", "http://example.com"),
- TestInfosetEvent.endDocument()
+ TestInfosetEvent.startDocument(),
+ TestInfosetEvent.startComplex("e1", "http://example.com"),
+ TestInfosetEvent.startSimple("e2", "http://example.com", expectedData),
+ TestInfosetEvent.endSimple("e2", "http://example.com"),
+ TestInfosetEvent.endComplex("e1", "http://example.com"),
+ TestInfosetEvent.endDocument()
};
org.apache.daffodil.japi.Compiler c = Daffodil.compiler();
@@ -1192,20 +1217,21 @@ public class TestJavaAPI {
java.io.File file = getResource("/test/japi/myData.dat");
java.io.FileInputStream fis = new java.io.FileInputStream(file);
- InputSourceDataInputStream dis = new InputSourceDataInputStream(fis);
- TestInfosetOutputter outputter = new TestInfosetOutputter();
- ParseResult pr = dp.parse(dis, outputter);
+ try (InputSourceDataInputStream dis = new InputSourceDataInputStream(fis)) {
+ TestInfosetOutputter outputter = new TestInfosetOutputter();
+ ParseResult pr = dp.parse(dis, outputter);
- assertFalse(pr.isError());
- assertArrayEquals(expectedEvents, outputter.events.toArray());
+ assertFalse(pr.isError());
+ assertArrayEquals(expectedEvents, outputter.events.toArray());
- java.io.ByteArrayOutputStream bos = new java.io.ByteArrayOutputStream();
- java.nio.channels.WritableByteChannel wbc = java.nio.channels.Channels.newChannel(bos);
- TestInfosetInputter inputter = new TestInfosetInputter(expectedEvents);
- UnparseResult ur = dp.unparse(inputter, wbc);
+ java.io.ByteArrayOutputStream bos = new java.io.ByteArrayOutputStream();
+ java.nio.channels.WritableByteChannel wbc = java.nio.channels.Channels.newChannel(bos);
+ TestInfosetInputter inputter = new TestInfosetInputter(expectedEvents);
+ UnparseResult ur = dp.unparse(inputter, wbc);
- assertFalse(ur.isError());
- assertEquals(expectedData, bos.toString());
+ assertFalse(ur.isError());
+ assertEquals(expectedData, bos.toString());
+ }
}
@Test
@@ -1287,7 +1313,7 @@ public class TestJavaAPI {
}
public void doXMLTextEscapeStyleTest(String expect, String data, String schemaType)
- throws IOException, ClassNotFoundException {
+ throws IOException, ClassNotFoundException {
org.apache.daffodil.japi.Compiler c = Daffodil.compiler();
java.io.File schemaFile = getResource("/test/japi/mySchemaCDATA.dfdl.xsd");
@@ -1295,19 +1321,20 @@ public class TestJavaAPI {
DataProcessor dp = pf.onPath("/");
ByteArrayInputStream is = new ByteArrayInputStream(data.getBytes(StandardCharsets.UTF_8));
- InputSourceDataInputStream input = new InputSourceDataInputStream(is);
- ByteArrayOutputStream bosDP = new ByteArrayOutputStream();
- XMLTextInfosetOutputter outputter = new XMLTextInfosetOutputter(bosDP, true, XMLTextEscapeStyle.CDATA);
- ParseResult res = dp.parse(input, outputter);
- boolean err = res.isError();
-
- String infosetDPString = bosDP.toString();
- int start = infosetDPString.indexOf(".com\">") + 6;
- int end = infosetDPString.indexOf("</tns");
- String value = infosetDPString.substring(start, end);
-
- assertFalse(err);
- assertEquals(expect, value);
+ try (InputSourceDataInputStream input = new InputSourceDataInputStream(is)) {
+ ByteArrayOutputStream bosDP = new ByteArrayOutputStream();
+ XMLTextInfosetOutputter outputter = new XMLTextInfosetOutputter(bosDP, true, XMLTextEscapeStyle.CDATA);
+ ParseResult res = dp.parse(input, outputter);
+ boolean err = res.isError();
+
+ String infosetDPString = bosDP.toString();
+ int start = infosetDPString.indexOf(".com\">") + 6;
+ int end = infosetDPString.indexOf("</tns");
+ String value = infosetDPString.substring(start, end);
+
+ assertFalse(err);
+ assertEquals(expect, value);
+ }
}
@Test
@@ -1318,31 +1345,32 @@ public class TestJavaAPI {
DataProcessor dp = pf.onPath("/");
dp = dp.withValidationMode(ValidationMode.Full);
- byte[] data = new byte[] { 0x00, 0x00, 0x00, 0x04, 0x01, 0x02, 0x03, 0x04 };
+ byte[] data = new byte[]{0x00, 0x00, 0x00, 0x04, 0x01, 0x02, 0x03, 0x04};
ByteArrayInputStream bis = new ByteArrayInputStream(data);
- InputSourceDataInputStream input = new InputSourceDataInputStream(data);
-
- Path blobRoot = Paths.get(System.getProperty("java.io.tmpdir"), "daffodil", "japi");
- Files.createDirectories(blobRoot);
- Path blobDir = Files.createTempDirectory(blobRoot, "blob-");
-
- ByteArrayOutputStream bos = new ByteArrayOutputStream();
- XMLTextInfosetOutputter output = new XMLTextInfosetOutputter(bos, true);
- output.setBlobAttributes(blobDir, "pre-", ".suf");
-
- ParseResult res = dp.parse(input, output);
- List<Path> blobPaths = JavaConverters.seqAsJavaList(output.getBlobPaths());
-
- try {
- assertFalse(res.isError());
- assertTrue(blobPaths.size() == 1);
- assertTrue(blobPaths.get(0).toString().contains("blob-"));
- assertTrue(blobPaths.get(0).toString().contains("pre-"));
- assertTrue(blobPaths.get(0).toString().contains(".suf"));
- } finally {
- Iterator<Path> pathIter = blobPaths.iterator();
- while (pathIter.hasNext()) Files.delete(pathIter.next());
- Files.delete(blobDir);
+ try (InputSourceDataInputStream input = new InputSourceDataInputStream(data)) {
+
+ Path blobRoot = Paths.get(System.getProperty("java.io.tmpdir"), "daffodil", "japi");
+ Files.createDirectories(blobRoot);
+ Path blobDir = Files.createTempDirectory(blobRoot, "blob-");
+
+ ByteArrayOutputStream bos = new ByteArrayOutputStream();
+ XMLTextInfosetOutputter output = new XMLTextInfosetOutputter(bos, true);
+ output.setBlobAttributes(blobDir, "pre-", ".suf");
+
+ ParseResult res = dp.parse(input, output);
+ List<Path> blobPaths = JavaConverters.seqAsJavaList(output.getBlobPaths());
+
+ try {
+ assertFalse(res.isError());
+ assertTrue(blobPaths.size() == 1);
+ assertTrue(blobPaths.get(0).toString().contains("blob-"));
+ assertTrue(blobPaths.get(0).toString().contains("pre-"));
+ assertTrue(blobPaths.get(0).toString().contains(".suf"));
+ } finally {
+ Iterator<Path> pathIter = blobPaths.iterator();
+ while (pathIter.hasNext()) Files.delete(pathIter.next());
+ Files.delete(blobDir);
+ }
}
}
@@ -1362,21 +1390,22 @@ public class TestJavaAPI {
java.io.File file = getResource("/test/japi/myData16.dat");
java.io.FileInputStream fis = new java.io.FileInputStream(file);
- InputSourceDataInputStream dis = new InputSourceDataInputStream(fis);
- JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
- ParseResult res = dp.parse(dis, outputter);
- boolean err = res.isError();
- assertFalse(err);
- assertEquals(5, res.location().bytePos1b());
- assertEquals(33, res.location().bitPos1b());
-
- java.io.ByteArrayOutputStream bos = new java.io.ByteArrayOutputStream();
- java.nio.channels.WritableByteChannel wbc = java.nio.channels.Channels.newChannel(bos);
- JDOMInfosetInputter inputter = new JDOMInfosetInputter(outputter.getResult());
- UnparseResult res2 = dp.unparse(inputter, wbc);
- err = res2.isError();
- assertFalse(err);
- assertEquals("9100", bos.toString());
+ try (InputSourceDataInputStream dis = new InputSourceDataInputStream(fis)) {
+ JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
+ ParseResult res = dp.parse(dis, outputter);
+ boolean err = res.isError();
+ assertFalse(err);
+ assertEquals(5, res.location().bytePos1b());
+ assertEquals(33, res.location().bitPos1b());
+
+ java.io.ByteArrayOutputStream bos = new java.io.ByteArrayOutputStream();
+ java.nio.channels.WritableByteChannel wbc = java.nio.channels.Channels.newChannel(bos);
+ JDOMInfosetInputter inputter = new JDOMInfosetInputter(outputter.getResult());
+ UnparseResult res2 = dp.unparse(inputter, wbc);
+ err = res2.isError();
+ assertFalse(err);
+ assertEquals("9100", bos.toString());
+ }
}
/***
diff --git a/daffodil-japi/src/test/java/org/apache/daffodil/example/ValidatorApiExample.java b/daffodil-japi/src/test/java/org/apache/daffodil/example/ValidatorApiExample.java
index 222c615e1..d029ffe99 100644
--- a/daffodil-japi/src/test/java/org/apache/daffodil/example/ValidatorApiExample.java
+++ b/daffodil-japi/src/test/java/org/apache/daffodil/example/ValidatorApiExample.java
@@ -49,11 +49,11 @@ public class ValidatorApiExample {
java.io.File file = getResource("/test/japi/myData5.dat");
java.io.FileInputStream fis = new java.io.FileInputStream(file);
- InputSourceDataInputStream dis = new InputSourceDataInputStream(fis);
- JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
- ParseResult res = dp.parse(dis, outputter);
-
- assertFalse(res.isValidationError());
+ try (InputSourceDataInputStream dis = new InputSourceDataInputStream(fis)) {
+ JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
+ ParseResult res = dp.parse(dis, outputter);
+ assertFalse(res.isValidationError());
+ }
}
@Test
@@ -65,11 +65,12 @@ public class ValidatorApiExample {
java.io.File file = getResource("/test/japi/myData.dat");
java.io.FileInputStream fis = new java.io.FileInputStream(file);
- InputSourceDataInputStream dis = new InputSourceDataInputStream(fis);
- JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
- ParseResult res = dp.parse(dis, outputter);
+ try (InputSourceDataInputStream dis = new InputSourceDataInputStream(fis)) {
+ JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
+ ParseResult res = dp.parse(dis, outputter);
- assertTrue(res.isValidationError());
+ assertTrue(res.isValidationError());
+ }
}
@Test // Verifies that Daffodil-2456 is a false report. Not a bug.
@@ -83,21 +84,22 @@ public class ValidatorApiExample {
DataProcessor dp = pf.onPath("/").withValidationMode(ValidationMode.Full);
java.io.InputStream fis = new ByteArrayInputStream("0".getBytes());
- InputSourceDataInputStream dis = new InputSourceDataInputStream(fis);
- JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
- ParseResult res = dp.parse(dis, outputter);
+ try (InputSourceDataInputStream dis = new InputSourceDataInputStream(fis)) {
+ JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
+ ParseResult res = dp.parse(dis, outputter);
- assertTrue(res.isValidationError());
+ assertTrue(res.isValidationError());
- for (Diagnostic d: res.getDiagnostics()) {
- // doublecheck - all the errors are validation errors.
- // System.err.println(d.getMessage());
- assertTrue(d.getMessage().contains("Validation Error"));
+ for (Diagnostic d : res.getDiagnostics()) {
+ // doublecheck - all the errors are validation errors.
+ // System.err.println(d.getMessage());
+ assertTrue(d.getMessage().contains("Validation Error"));
+ }
+ assertTrue(res.isError());
+ // XMLOutputter xout = new XMLOutputter();
+ // xout.output(outputter.getResult(), System.out);
+ // Now get the well formed, though invalid result.
+ assertNotNull(outputter.getResult()); // Daffodil-2456 said this fails. It doesn't. Not reproducible.
}
- assertTrue(res.isError());
- // XMLOutputter xout = new XMLOutputter();
- // xout.output(outputter.getResult(), System.out);
- // Now get the well formed, though invalid result.
- assertNotNull(outputter.getResult()); // Daffodil-2456 said this fails. It doesn't. Not reproducible.
}
}
diff --git a/daffodil-japi/src/test/java/org/apache/daffodil/example/ValidatorSpiExample.java b/daffodil-japi/src/test/java/org/apache/daffodil/example/ValidatorSpiExample.java
index 37cb837a2..aee183f0d 100644
--- a/daffodil-japi/src/test/java/org/apache/daffodil/example/ValidatorSpiExample.java
+++ b/daffodil-japi/src/test/java/org/apache/daffodil/example/ValidatorSpiExample.java
@@ -54,11 +54,12 @@ public class ValidatorSpiExample {
java.io.File file = getResource("/test/japi/myData5.dat");
java.io.FileInputStream fis = new java.io.FileInputStream(file);
- InputSourceDataInputStream dis = new InputSourceDataInputStream(fis);
- JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
- ParseResult res = dp.parse(dis, outputter);
+ try (InputSourceDataInputStream dis = new InputSourceDataInputStream(fis)) {
+ JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
+ ParseResult res = dp.parse(dis, outputter);
- assertFalse(res.isValidationError());
+ assertFalse(res.isValidationError());
+ }
}
@Test
@@ -73,10 +74,11 @@ public class ValidatorSpiExample {
java.io.File file = getResource("/test/japi/myData.dat");
java.io.FileInputStream fis = new java.io.FileInputStream(file);
- InputSourceDataInputStream dis = new InputSourceDataInputStream(fis);
- JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
- ParseResult res = dp.parse(dis, outputter);
+ try (InputSourceDataInputStream dis = new InputSourceDataInputStream(fis)) {
+ JDOMInfosetOutputter outputter = new JDOMInfosetOutputter();
+ ParseResult res = dp.parse(dis, outputter);
- assertTrue(res.isValidationError());
+ assertTrue(res.isValidationError());
+ }
}
}
diff --git a/daffodil-lib/src/main/scala/org/apache/daffodil/lib/api/Diagnostic.scala b/daffodil-lib/src/main/scala/org/apache/daffodil/lib/api/Diagnostic.scala
index c3d98b16a..2f178d887 100644
--- a/daffodil-lib/src/main/scala/org/apache/daffodil/lib/api/Diagnostic.scala
+++ b/daffodil-lib/src/main/scala/org/apache/daffodil/lib/api/Diagnostic.scala
@@ -57,8 +57,8 @@ abstract class Diagnostic protected (
private val maybeFormatString: Maybe[String],
private val args: Any*,
) extends Exception(
- null,
- (if (maybeCause.isDefined) maybeCause.get else null),
+ maybeFormatString.orNull,
+ maybeCause.orNull,
isThick,
isThick,
) {
@@ -165,13 +165,6 @@ abstract class Diagnostic protected (
final def getSomeCause: Some[Throwable] = Misc.getSomeCause(this)
final def getSomeMessage: Some[String] = Misc.getSomeMessage(this)
- private def init(): Unit = {
- Assert.invariant(maybeCause.isDefined ^ maybeFormatString.isDefined)
- Assert.invariant(
- maybeCause.isEmpty || args.length == 0,
- ) // if there is a cause, there can't be args.
- }
-
private def schemaLocationsString = {
val strings = getLocationsInSchemaFiles.map { _.locationDescription }
val res =
@@ -225,7 +218,6 @@ abstract class Diagnostic protected (
}
private lazy val message = {
- init
val res = modeName + " " + errorOrWarning + ": " +
(if (maybeCause.isDefined) msgCausedBy else msgString) +
componentText +
diff --git a/daffodil-lib/src/main/scala/org/apache/daffodil/lib/schema/annotation/props/ByHandMixins.scala b/daffodil-lib/src/main/scala/org/apache/daffodil/lib/schema/annotation/props/ByHandMixins.scala
index 4e32d8067..90ccd6d65 100644
--- a/daffodil-lib/src/main/scala/org/apache/daffodil/lib/schema/annotation/props/ByHandMixins.scala
+++ b/daffodil-lib/src/main/scala/org/apache/daffodil/lib/schema/annotation/props/ByHandMixins.scala
@@ -418,7 +418,7 @@ object BinaryBooleanFalseRepType {
if (i < 0)
element.schemaDefinitionError(
- "For property 'binaryBooleanFalseRep', value must be a non-negative integer. Found: %d",
+ "For property 'binaryBooleanFalseRep', value must be a non-negative integer. Found: %s",
i,
)
ULong(i)
@@ -538,12 +538,3 @@ trait EmptyElementParsePolicyMixin extends PropertyMixin {
}
}
-
-/**
- * Code generated by the property generator for the dfdl layering and dfdlx
- * layering properties results in duplicates. To resolve this, the dfdlx schema
- * defines the traits as Ext*, but the property generator is told to not
- * generated traits for these. So we must create the by hand.
- */
-trait ExtLayeringAGMixin
-trait ExtLayeringAGQualifiedMixin
diff --git a/daffodil-lib/src/main/scala/org/apache/daffodil/lib/schema/annotation/props/PropertyScoping.scala b/daffodil-lib/src/main/scala/org/apache/daffodil/lib/schema/annotation/props/PropertyScoping.scala
index 8bcbc04e2..0dbcff481 100644
--- a/daffodil-lib/src/main/scala/org/apache/daffodil/lib/schema/annotation/props/PropertyScoping.scala
+++ b/daffodil-lib/src/main/scala/org/apache/daffodil/lib/schema/annotation/props/PropertyScoping.scala
@@ -62,6 +62,15 @@ sealed abstract class PropertyLookupResult(val pname: String) extends Serializab
}
}
+/**
+ * Represents a successful property lookup result.
+ *
+ * @constructor Create a new Found instance with the specified value, location, property name, and format information.
+ * @param value The value associated with the property.
+ * @param location The lookup location where the property was found.
+ * @param pname The name of the property.
+ * @param isFromDefaultFormat Indicates whether the property was found in the default format.
+ */
case class Found(
value: String,
location: LookupLocation,
@@ -71,13 +80,22 @@ case class Found(
override def isDefined = true
}
+/**
+ * Represents a property lookup result when the property is not found.
+ *
+ * @param localWhereLooked The sequence of lookup locations where the property was searched locally.
+ * @param defaultWhereLooked The sequence of lookup locations where the property was searched in the default format.
+ * @param pname The name of the property that was searched.
+ */
case class NotFound(
localWhereLooked: Seq[LookupLocation],
defaultWhereLooked: Seq[LookupLocation],
override val pname: String,
) extends PropertyLookupResult(pname) {
override def isDefined = false
- override def isFromDefaultFormat = Assert.usageError("Not meaningful for NotFound.")
+ // $COVERAGE-OFF$
+ override def isFromDefaultFormat: Boolean = Assert.usageError("Not meaningful for NotFound.")
+ // $COVERAGE-ON$
}
/**
diff --git a/daffodil-lib/src/main/scala/org/apache/daffodil/lib/util/Misc.scala b/daffodil-lib/src/main/scala/org/apache/daffodil/lib/util/Misc.scala
index 50ab159ef..9af804bf1 100644
--- a/daffodil-lib/src/main/scala/org/apache/daffodil/lib/util/Misc.scala
+++ b/daffodil-lib/src/main/scala/org/apache/daffodil/lib/util/Misc.scala
@@ -707,7 +707,7 @@ object Misc {
case (null, c) => getSomeMessage(c).get
case (m, c) => {
val Some(cmsg) = getSomeMessage(c)
- cmsg + " (within " + m + ")"
+ cmsg + " (within " + th.getClass.getSimpleName + " " + m + ")"
}
}
Some(res)
diff --git a/daffodil-lib/src/main/scala/org/apache/daffodil/lib/xml/Namespaces.scala b/daffodil-lib/src/main/scala/org/apache/daffodil/lib/xml/Namespaces.scala
index 7c9d75f42..d5c40343f 100644
--- a/daffodil-lib/src/main/scala/org/apache/daffodil/lib/xml/Namespaces.scala
+++ b/daffodil-lib/src/main/scala/org/apache/daffodil/lib/xml/Namespaces.scala
@@ -128,7 +128,7 @@ sealed class NS protected (uriArg: URI) extends Serializable { // protected cons
override def equals(other: Any): Boolean = {
if (this eq other.asInstanceOf[AnyRef]) return true
Assert.invariant(
- this.toString != other.toString,
+ this.toString != other.toString, // these do not allocate strings each time they are called.
) // this fails if the cache isn't being used
false
}
diff --git a/daffodil-lib/src/main/scala/org/apache/daffodil/lib/xml/QNameBase.scala b/daffodil-lib/src/main/scala/org/apache/daffodil/lib/xml/QNameBase.scala
index b20eb1fa7..0a00348ff 100644
--- a/daffodil-lib/src/main/scala/org/apache/daffodil/lib/xml/QNameBase.scala
+++ b/daffodil-lib/src/main/scala/org/apache/daffodil/lib/xml/QNameBase.scala
@@ -206,6 +206,25 @@ object QName {
def createGlobal(name: String, targetNamespace: NS, scope: scala.xml.NamespaceBinding) = {
GlobalQName(optPrefix(scope, targetNamespace), name, targetNamespace)
}
+
+ /**
+ * Printing utility for QNames. Allows you control over the individual components.
+ * @param prefix if defined, will be included in printed representation
+ * @param local the local name (NCName)
+ * @param namespace an NS object
+ * @return
+ */
+ def toPrettyString(prefix: Option[String], local: String, namespace: NS): String = {
+ (prefix, local, namespace) match {
+ case (Some(pre), local, NoNamespace) =>
+ Assert.invariantFailed("QName has prefix, but NoNamespace")
+ case (Some(pre), local, UnspecifiedNamespace) => pre + ":" + local
+ case (None, local, NoNamespace) => "{}" + local
+ case (None, local, UnspecifiedNamespace) => local
+ case (None, local, ns) => "{" + ns + "}" + local
+ case (Some(pre), local, ns) => pre + ":" + local
+ }
+ }
}
protected sealed abstract class QNameSyntaxExceptionBase(
@@ -268,17 +287,7 @@ trait QNameBase extends Serializable {
*
* Incorrectly defined names are not tolerated.
*/
- def toPrettyString: String = {
- (prefix, local, namespace) match {
- case (Some(pre), local, NoNamespace) =>
- Assert.invariantFailed("QName has prefix, but NoNamespace")
- case (Some(pre), local, UnspecifiedNamespace) => pre + ":" + local
- case (None, local, NoNamespace) => "{}" + local
- case (None, local, UnspecifiedNamespace) => local
- case (None, local, ns) => "{" + ns + "}" + local
- case (Some(pre), local, ns) => pre + ":" + local
- }
- }
+ def toPrettyString: String = QName.toPrettyString(prefix, local, namespace)
/**
* displays all components that are available.
@@ -364,7 +373,7 @@ sealed abstract class NamedQName(prefix: Option[String], local: String, namespac
Assert.usage(!namespace.isUnspecified)
}
- def toRefQName = RefQName(prefix, local, namespace)
+ lazy val toRefQName = RefQName(prefix, local, namespace)
}
/**
@@ -408,19 +417,15 @@ final case class LocalDeclQName(prefix: Option[String], local: String, namespace
final case class GlobalQName(prefix: Option[String], local: String, namespace: NS)
extends NamedQName(prefix, local, namespace) {
+ /**
+ * This gets called a lot, even in runtime assertions, so let's make this
+ * as fast as possible.
+ * @param other
+ * @tparam Q
+ * @return
+ */
override def matches[Q <: QNameBase](other: Q): Boolean = {
- other match {
- // StepQNames match against global names in the case of a path
- // step that refers to an element that is defined in its
- // group, via an element reference.
- case StepQName(_, `local`, `namespace`) => true // exact match
- case StepQName(_, _, _) => false
- // RefQNames match against global names in element references,
- // group references, type references (i.e., type="..."), etc.
- case RefQName(_, `local`, `namespace`) => true // exact match
- case RefQName(_, _, _) => false
- case _ => Assert.usageError("other must be a StepQName or RefQName")
- }
+ other.local == this.local && other.namespace == this.namespace
}
}
@@ -445,8 +450,8 @@ final case class RefQName(prefix: Option[String], local: String, namespace: NS)
}
}
- def toStepQName = StepQName(prefix, local, namespace)
- def toGlobalQName = GlobalQName(prefix, local, namespace)
+ lazy val toStepQName = StepQName(prefix, local, namespace)
+ lazy val toGlobalQName = GlobalQName(prefix, local, namespace)
}
/**
@@ -512,7 +517,8 @@ protected trait RefQNameFactoryBase[T] {
}
val ns = (prefix, optURI) match {
case (None, None) => NoNamespace
- case (Some(pre), None) => throw new QNameUndefinedPrefixException(pre)
+ case (Some(pre), None) =>
+ throw new QNameUndefinedPrefixException(pre)
case (_, Some(ns)) => NS(ns)
}
val res = constructor(prefix, local, ns)
diff --git a/daffodil-lib/src/main/scala/org/apache/daffodil/lib/xml/XMLUtils.scala b/daffodil-lib/src/main/scala/org/apache/daffodil/lib/xml/XMLUtils.scala
index 860f0086b..c8188c912 100644
--- a/daffodil-lib/src/main/scala/org/apache/daffodil/lib/xml/XMLUtils.scala
+++ b/daffodil-lib/src/main/scala/org/apache/daffodil/lib/xml/XMLUtils.scala
@@ -153,6 +153,9 @@ object XMLUtils {
list
}
+ // FIXME: DAFFODIL-2883 - this needs checkForExistingPUA to be false so that data
+ // which contains unicode PUA characters doesn't cause an SDE. Needs to be either
+ // accepted or optionally cause a ParseError.
private val remapXMLToPUA =
new RemapXMLIllegalCharToPUA(checkForExistingPUA = true, replaceCRWithLF = true)
diff --git a/daffodil-propgen/src/main/resources/org/apache/daffodil/xsd/DFDL_part2_attributes.xsd b/daffodil-propgen/src/main/resources/org/apache/daffodil/xsd/DFDL_part2_attributes.xsd
index 8b3b3018a..fdba1f30f 100644
--- a/daffodil-propgen/src/main/resources/org/apache/daffodil/xsd/DFDL_part2_attributes.xsd
+++ b/daffodil-propgen/src/main/resources/org/apache/daffodil/xsd/DFDL_part2_attributes.xsd
@@ -102,7 +102,9 @@
<xsd:attribute ref="daf:parseUnparsePolicy"/> <!-- backwards compatibility -->
<xsd:attribute ref="dfdlx:parseUnparsePolicy"/>
<xsd:attribute name="emptyElementParsePolicy" type="dfdl:EmptyElementParsePolicyEnum"/>
- <xsd:attribute ref="dfdlx:emptyElementParsePolicy"/> <!-- deprecated -->
+
+ <!-- deprecated -->
+ <xsd:attribute ref="dfdlx:emptyElementParsePolicy"/>
<xsd:attribute name="encoding"
type="dfdl:EncodingEnum_Or_DFDLExpression" />
diff --git a/daffodil-propgen/src/main/resources/org/apache/daffodil/xsd/dafext.xsd b/daffodil-propgen/src/main/resources/org/apache/daffodil/xsd/dafext.xsd
index 41912d473..3d89c5661 100644
--- a/daffodil-propgen/src/main/resources/org/apache/daffodil/xsd/dafext.xsd
+++ b/daffodil-propgen/src/main/resources/org/apache/daffodil/xsd/dafext.xsd
@@ -632,7 +632,9 @@
<xs:restriction base="xs:string">
<xs:enumeration value="treatAsEmpty" />
<xs:enumeration value="treatAsAbsent" />
- <xs:enumeration value="treatAsMissing" /> <!-- deprecated -->
+
+ <!-- deprecated -->
+ <xs:enumeration value="treatAsMissing" />
</xs:restriction>
</xs:simpleType>
diff --git a/daffodil-propgen/src/main/resources/org/apache/daffodil/xsd/dfdlx.xsd b/daffodil-propgen/src/main/resources/org/apache/daffodil/xsd/dfdlx.xsd
index 5c32dd9bc..7e4fa6620 100644
--- a/daffodil-propgen/src/main/resources/org/apache/daffodil/xsd/dfdlx.xsd
+++ b/daffodil-propgen/src/main/resources/org/apache/daffodil/xsd/dfdlx.xsd
@@ -37,7 +37,10 @@
<xs:restriction base="xs:string">
<xs:enumeration value="dfdlx:alignmentKind" />
<xs:enumeration value="dfdlx:choiceBranchKeyRanges" />
- <xs:enumeration value="dfdlx:emptyElementParsePolicy"/> <!-- deprecated -->
+
+ <!-- deprecated -->
+ <xs:enumeration value="dfdlx:emptyElementParsePolicy"/>
+
<xs:enumeration value="dfdlx:objectKind"/>
<xs:enumeration value="dfdlx:parseUnparsePolicy"/>
<xs:enumeration value="dfdlx:repType"/>
@@ -93,11 +96,14 @@
</xs:restriction>
</xs:simpleType>
- <xs:attribute name="emptyElementParsePolicy" type="dfdlx:EmptyElementParsePolicyEnum"/> <!-- deprecated -->
+ <!-- deprecated -->
+ <xs:attribute name="emptyElementParsePolicy" type="dfdlx:EmptyElementParsePolicyEnum"/>
<xs:simpleType name="EmptyElementParsePolicyEnum">
<xs:restriction base="xs:string">
<xs:enumeration value="treatAsEmpty" />
- <xs:enumeration value="treatAsMissing" /> <!-- deprecated -->
+
+ <!-- deprecated -->
+ <xs:enumeration value="treatAsMissing" />
</xs:restriction>
</xs:simpleType>
@@ -122,109 +128,9 @@
</xs:attributeGroup>
<xs:attributeGroup name="ExtLayeringAGQualified">
- <xs:attribute form="qualified" name="layerTransform" type="dfdlx:LayerTransformType" />
- <xs:attribute form="qualified" name="layerEncoding" type="dfdl:EncodingEnum_Or_DFDLExpression" />
- <xs:attribute form="qualified" name="layerLengthKind" type="dfdlx:LayerLengthKindEnum" />
- <xs:attribute form="qualified" name="layerLength" type="dfdl:DFDLNonNegativeInteger_Or_DFDLExpression" />
- <xs:attribute form="qualified" name="layerLengthUnits" type="dfdlx:LayerLengthUnitsEnum" />
- <xs:attribute form="qualified" name="layerBoundaryMark" type="dfdl:ListOfDFDLStringLiteral_Or_DFDLExpression" />
+ <xs:attribute form="qualified" name="layer" type="xsd:QName" use='optional'/>
</xs:attributeGroup>
-
- <xs:simpleType name="LayerTransformType">
- <xs:union>
- <xs:simpleType>
- <xs:restriction base="dfdlx:LayerTransformEnum" />
- </xs:simpleType>
- <xs:simpleType>
- <xs:restriction base="xs:NCName" /><!-- for dynamically loaded layers -->
- </xs:simpleType>
- </xs:union>
- </xs:simpleType>
-
- <xs:simpleType name="LayerTransformEnum">
- <xs:restriction base="xs:token">
- <xs:enumeration value="fourbyteswap">
- <xs:annotation>
- <xs:documentation>
- Swap bytes in 32 bit words.
- </xs:documentation>
- </xs:annotation>
- </xs:enumeration>
- <xs:enumeration value="base64_MIME">
- <xs:annotation>
- <xs:documentation>
- IETF RFC 2045
-
- Max line length 76 characters.
- </xs:documentation>
- </xs:annotation>
- </xs:enumeration>
- <xs:enumeration value="lineFolded_IMF">
- <xs:annotation>
- <xs:documentation><![CDATA[
- IETF RFC 2822 Internet Message Format (IMF)
-
- Each line of characters MUST be no more than
- 998 characters, and SHOULD be no more than 78 characters, excluding
- the CRLF.
-
- Though structured field bodies are defined in such a way that
- folding can take place between many of the lexical tokens (and even
- within some of the lexical tokens), folding SHOULD be limited to
- placing the CRLF at higher-level syntactic breaks. For instance, if
- a field body is defined as comma-separated values, it is recommended
- that folding occur after the comma separating the structured items in
- preference to other places where the field could be folded, even if
- it is allowed elsewhere.
-
- Unfolding is accomplished by simply removing any CRLF
- that is immediately followed by WSP.
- ]]>
- </xs:documentation>
- </xs:annotation>
- </xs:enumeration>
- <xs:enumeration value="lineFolded_iCalendar">
- <xs:annotation>
- <xs:documentation><![CDATA[
- IETF RFC 5545 Internet Calendaring and Scheduling (iCalendar)
-
- Lines of text SHOULD NOT be longer than 75 octets, excluding the line
- break. Long content lines SHOULD be split into a multiple line
- representations using a line "folding" technique. That is, a long
- line can be split between any two characters by inserting a CRLF
- immediately followed by a single linear white-space character (i.e.,
- SPACE or HTAB). Any sequence of CRLF followed immediately by a
- single linear white-space character is ignored (i.e., removed) when
- processing the content type.
- ]]>
- </xs:documentation>
- </xs:annotation>
- </xs:enumeration>
- <xs:enumeration value="gzip">
- <xs:annotation>
- <xs:documentation>
- GZIP per https://www.ietf.org/rfc/rfc1952.txt
- </xs:documentation>
- </xs:annotation>
- </xs:enumeration>
- </xs:restriction>
- </xs:simpleType>
-
- <xs:simpleType name="LayerLengthUnitsEnum">
- <xs:restriction base="xs:string">
- <xs:enumeration value="bytes" />
- </xs:restriction>
- </xs:simpleType>
-
- <xs:simpleType name="LayerLengthKindEnum">
- <xs:restriction base="xs:string">
- <xs:enumeration value="explicit" />
- <xs:enumeration value="boundaryMark" />
- <xs:enumeration value="implicit" />
- </xs:restriction>
- </xs:simpleType>
-
<xs:attributeGroup name="ObjectKindAG">
<xs:attribute form="qualified" name="objectKind" type="dfdlx:ObjectKindType" />
</xs:attributeGroup>
diff --git a/daffodil-propgen/src/main/scala/org/apache/daffodil/propGen/PropertyGenerator.scala b/daffodil-propgen/src/main/scala/org/apache/daffodil/propGen/PropertyGenerator.scala
index 031509275..b2715745a 100644
--- a/daffodil-propgen/src/main/scala/org/apache/daffodil/propGen/PropertyGenerator.scala
+++ b/daffodil-propgen/src/main/scala/org/apache/daffodil/propGen/PropertyGenerator.scala
@@ -563,10 +563,15 @@ object Currency {
""" def EUR = convertToTYPE(findProperty("EUR").value)
"""
} else {
- """ def EUR = {
+ """
+ def EUR = {
val cp = findProperty("EUR")
convertToTYPE(cp.value, cp.location)
}
+ def EUROption = findPropertyOption("EUR") match {
+ case Found(s, ll, _, _) => Some(convertToTYPE(s, ll))
+ case _ => None
+ }
"""
}
diff --git a/daffodil-runtime1-layers/src/main/resources/META-INF/services/org.apache.daffodil.runtime1.layers.LayerCompiler b/daffodil-runtime1-layers/src/main/resources/META-INF/services/org.apache.daffodil.runtime1.layers.api.Layer
similarity index 66%
rename from daffodil-runtime1-layers/src/main/resources/META-INF/services/org.apache.daffodil.runtime1.layers.LayerCompiler
rename to daffodil-runtime1-layers/src/main/resources/META-INF/services/org.apache.daffodil.runtime1.layers.api.Layer
index 9e8ae649e..e6f3a5922 100644
--- a/daffodil-runtime1-layers/src/main/resources/META-INF/services/org.apache.daffodil.runtime1.layers.LayerCompiler
+++ b/daffodil-runtime1-layers/src/main/resources/META-INF/services/org.apache.daffodil.runtime1.layers.api.Layer
@@ -15,9 +15,11 @@
#
# These are the layers that are built-in to Daffodil's runtime1.
#
-org.apache.daffodil.layers.runtime1.Base64MIMELayerCompiler
-org.apache.daffodil.layers.runtime1.FourByteSwapLayerCompiler
-org.apache.daffodil.layers.runtime1.GZIPLayerCompiler
-org.apache.daffodil.layers.runtime1.LineFoldedIMFLayerCompiler
-org.apache.daffodil.layers.runtime1.LineFoldedICalendarLayerCompiler
-
+org.apache.daffodil.layers.runtime1.Base64MimeLayer
+org.apache.daffodil.layers.runtime1.FourByteSwapLayer
+org.apache.daffodil.layers.runtime1.TwoByteSwapLayer
+org.apache.daffodil.layers.runtime1.GZipLayer
+org.apache.daffodil.layers.runtime1.LineFoldedIMFLayer
+org.apache.daffodil.layers.runtime1.LineFoldedICalendarLayer
+org.apache.daffodil.layers.runtime1.BoundaryMarkLayer
+org.apache.daffodil.layers.runtime1.FixedLengthLayer
diff --git a/daffodil-runtime1-layers/src/main/resources/org/apache/daffodil/layers/xsd/base64_MIMELayer.dfdl.xsd b/daffodil-runtime1-layers/src/main/resources/org/apache/daffodil/layers/xsd/base64_MIMELayer.dfdl.xsd
new file mode 100644
index 000000000..675c3a83b
--- /dev/null
+++ b/daffodil-runtime1-layers/src/main/resources/org/apache/daffodil/layers/xsd/base64_MIMELayer.dfdl.xsd
@@ -0,0 +1,32 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<schema
+ xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns="http://www.w3.org/2001/XMLSchema"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions"
+ xmlns:b64="urn:org.apache.daffodil.layers.base64_MIME"
+ targetNamespace="urn:org.apache.daffodil.layers.base64_MIME">
+
+ <!--
+ This layer does not define any variables for parameters or results
+
+ The layer name is base64_MIME and it lives in the targetNamespace defined above.
+ -->
+
+</schema>
\ No newline at end of file
diff --git a/daffodil-cli/src/test/resources/org/apache/daffodil/layers/xsd/buggyLayer.dfdl.xsd b/daffodil-runtime1-layers/src/main/resources/org/apache/daffodil/layers/xsd/boundaryMarkLayer.dfdl.xsd
similarity index 51%
copy from daffodil-cli/src/test/resources/org/apache/daffodil/layers/xsd/buggyLayer.dfdl.xsd
copy to daffodil-runtime1-layers/src/main/resources/org/apache/daffodil/layers/xsd/boundaryMarkLayer.dfdl.xsd
index 7ddbe4bce..187bc95e1 100644
--- a/daffodil-cli/src/test/resources/org/apache/daffodil/layers/xsd/buggyLayer.dfdl.xsd
+++ b/daffodil-runtime1-layers/src/main/resources/org/apache/daffodil/layers/xsd/boundaryMarkLayer.dfdl.xsd
@@ -16,29 +16,22 @@
limitations under the License.
-->
<schema
- xmlns:xs="http://www.w3.org/2001/XMLSchema"
- xmlns="http://www.w3.org/2001/XMLSchema"
- xmlns:fn="http://www.w3.org/2005/xpath-functions"
- xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
- xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions"
- xmlns:buggy="urn:org.apache.daffodil.layers.buggy"
- xmlns:tns="urn:org.apache.daffodil.layers.buggy"
- targetNamespace="urn:org.apache.daffodil.layers.buggy"
- elementFormDefault="unqualified">
-
- <include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd" />
+ xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns="http://www.w3.org/2001/XMLSchema"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions"
+ xmlns:bm="urn:org.apache.daffodil.layers.boundaryMark"
+ targetNamespace="urn:org.apache.daffodil.layers.boundaryMark">
<annotation>
<appinfo source="http://www.ogf.org/dfdl/">
- <dfdl:format ref="buggy:GeneralFormat" />
+ <!-- These variables are both parameters to the boundaryMark layer -->
- <dfdl:defineFormat name="buggyFormat">
- <dfdl:format dfdlx:layerTransform="buggy" dfdlx:layerLengthKind="explicit" dfdlx:layerLengthUnits="bytes"
- dfdlx:layerEncoding="ascii" />
- </dfdl:defineFormat>
+ <dfdl:defineVariable name="boundaryMark" type="xs:string"/>
+ <dfdl:defineVariable name="layerEncoding" type="xs:string"/>
</appinfo>
</annotation>
-</schema>
+</schema>
\ No newline at end of file
diff --git a/daffodil-cli/src/test/resources/org/apache/daffodil/layers/xsd/buggyLayer.dfdl.xsd b/daffodil-runtime1-layers/src/main/resources/org/apache/daffodil/layers/xsd/byteSwapLayer.dfdl.xsd
similarity index 51%
copy from daffodil-cli/src/test/resources/org/apache/daffodil/layers/xsd/buggyLayer.dfdl.xsd
copy to daffodil-runtime1-layers/src/main/resources/org/apache/daffodil/layers/xsd/byteSwapLayer.dfdl.xsd
index 7ddbe4bce..958b0a39c 100644
--- a/daffodil-cli/src/test/resources/org/apache/daffodil/layers/xsd/buggyLayer.dfdl.xsd
+++ b/daffodil-runtime1-layers/src/main/resources/org/apache/daffodil/layers/xsd/byteSwapLayer.dfdl.xsd
@@ -16,29 +16,22 @@
limitations under the License.
-->
<schema
- xmlns:xs="http://www.w3.org/2001/XMLSchema"
- xmlns="http://www.w3.org/2001/XMLSchema"
- xmlns:fn="http://www.w3.org/2005/xpath-functions"
- xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
- xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions"
- xmlns:buggy="urn:org.apache.daffodil.layers.buggy"
- xmlns:tns="urn:org.apache.daffodil.layers.buggy"
- targetNamespace="urn:org.apache.daffodil.layers.buggy"
- elementFormDefault="unqualified">
-
- <include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd" />
+ xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns="http://www.w3.org/2001/XMLSchema"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions"
+ xmlns:bswap="urn:org.apache.daffodil.layers.byteSwap"
+ targetNamespace="urn:org.apache.daffodil.layers.byteSwap">
<annotation>
<appinfo source="http://www.ogf.org/dfdl/">
- <dfdl:format ref="buggy:GeneralFormat" />
-
- <dfdl:defineFormat name="buggyFormat">
- <dfdl:format dfdlx:layerTransform="buggy" dfdlx:layerLengthKind="explicit" dfdlx:layerLengthUnits="bytes"
- dfdlx:layerEncoding="ascii" />
- </dfdl:defineFormat>
+ <!--
+ The twobyteswap and fourbyteswap layers have no parameters nor
+ return variables
+ -->
</appinfo>
</annotation>
-</schema>
+</schema>
\ No newline at end of file
diff --git a/daffodil-cli/src/test/resources/org/apache/daffodil/layers/xsd/buggyLayer.dfdl.xsd b/daffodil-runtime1-layers/src/main/resources/org/apache/daffodil/layers/xsd/fixedLengthLayer.dfdl.xsd
similarity index 51%
copy from daffodil-cli/src/test/resources/org/apache/daffodil/layers/xsd/buggyLayer.dfdl.xsd
copy to daffodil-runtime1-layers/src/main/resources/org/apache/daffodil/layers/xsd/fixedLengthLayer.dfdl.xsd
index 7ddbe4bce..1fe072399 100644
--- a/daffodil-cli/src/test/resources/org/apache/daffodil/layers/xsd/buggyLayer.dfdl.xsd
+++ b/daffodil-runtime1-layers/src/main/resources/org/apache/daffodil/layers/xsd/fixedLengthLayer.dfdl.xsd
@@ -16,29 +16,21 @@
limitations under the License.
-->
<schema
- xmlns:xs="http://www.w3.org/2001/XMLSchema"
- xmlns="http://www.w3.org/2001/XMLSchema"
- xmlns:fn="http://www.w3.org/2005/xpath-functions"
- xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
- xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions"
- xmlns:buggy="urn:org.apache.daffodil.layers.buggy"
- xmlns:tns="urn:org.apache.daffodil.layers.buggy"
- targetNamespace="urn:org.apache.daffodil.layers.buggy"
- elementFormDefault="unqualified">
-
- <include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd" />
+ xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns="http://www.w3.org/2001/XMLSchema"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions"
+ xmlns:fl="urn:org.apache.daffodil.layers.fixedLength"
+ targetNamespace="urn:org.apache.daffodil.layers.fixedLength">
<annotation>
<appinfo source="http://www.ogf.org/dfdl/">
- <dfdl:format ref="buggy:GeneralFormat" />
+ <!-- the fixedLength layer takes one parameter variable -->
- <dfdl:defineFormat name="buggyFormat">
- <dfdl:format dfdlx:layerTransform="buggy" dfdlx:layerLengthKind="explicit" dfdlx:layerLengthUnits="bytes"
- dfdlx:layerEncoding="ascii" />
- </dfdl:defineFormat>
+ <dfdl:defineVariable name="fixedLength" type="xs:unsignedInt"/>
</appinfo>
</annotation>
-</schema>
+</schema>
\ No newline at end of file
diff --git a/daffodil-cli/src/test/resources/org/apache/daffodil/layers/xsd/buggyLayer.dfdl.xsd b/daffodil-runtime1-layers/src/main/resources/org/apache/daffodil/layers/xsd/gzipLayer.dfdl.xsd
similarity index 51%
copy from daffodil-cli/src/test/resources/org/apache/daffodil/layers/xsd/buggyLayer.dfdl.xsd
copy to daffodil-runtime1-layers/src/main/resources/org/apache/daffodil/layers/xsd/gzipLayer.dfdl.xsd
index 7ddbe4bce..30aca2ae0 100644
--- a/daffodil-cli/src/test/resources/org/apache/daffodil/layers/xsd/buggyLayer.dfdl.xsd
+++ b/daffodil-runtime1-layers/src/main/resources/org/apache/daffodil/layers/xsd/gzipLayer.dfdl.xsd
@@ -16,29 +16,20 @@
limitations under the License.
-->
<schema
- xmlns:xs="http://www.w3.org/2001/XMLSchema"
- xmlns="http://www.w3.org/2001/XMLSchema"
- xmlns:fn="http://www.w3.org/2005/xpath-functions"
- xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
- xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions"
- xmlns:buggy="urn:org.apache.daffodil.layers.buggy"
- xmlns:tns="urn:org.apache.daffodil.layers.buggy"
- targetNamespace="urn:org.apache.daffodil.layers.buggy"
- elementFormDefault="unqualified">
-
- <include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd" />
+ xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns="http://www.w3.org/2001/XMLSchema"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions"
+ xmlns:gz="urn:org.apache.daffodil.layers.gzip"
+ targetNamespace="urn:org.apache.daffodil.layers.gzip">
<annotation>
<appinfo source="http://www.ogf.org/dfdl/">
- <dfdl:format ref="buggy:GeneralFormat" />
-
- <dfdl:defineFormat name="buggyFormat">
- <dfdl:format dfdlx:layerTransform="buggy" dfdlx:layerLengthKind="explicit" dfdlx:layerLengthUnits="bytes"
- dfdlx:layerEncoding="ascii" />
- </dfdl:defineFormat>
+ <!-- the gzip layer has no parameter variables
+ nor return result variables. -->
</appinfo>
</annotation>
-</schema>
+</schema>
\ No newline at end of file
diff --git a/daffodil-cli/src/test/resources/org/apache/daffodil/layers/xsd/buggyLayer.dfdl.xsd b/daffodil-runtime1-layers/src/main/resources/org/apache/daffodil/layers/xsd/lineFoldedLayer.dfdl.xsd
similarity index 51%
copy from daffodil-cli/src/test/resources/org/apache/daffodil/layers/xsd/buggyLayer.dfdl.xsd
copy to daffodil-runtime1-layers/src/main/resources/org/apache/daffodil/layers/xsd/lineFoldedLayer.dfdl.xsd
index 7ddbe4bce..4f6153ff7 100644
--- a/daffodil-cli/src/test/resources/org/apache/daffodil/layers/xsd/buggyLayer.dfdl.xsd
+++ b/daffodil-runtime1-layers/src/main/resources/org/apache/daffodil/layers/xsd/lineFoldedLayer.dfdl.xsd
@@ -16,29 +16,21 @@
limitations under the License.
-->
<schema
- xmlns:xs="http://www.w3.org/2001/XMLSchema"
- xmlns="http://www.w3.org/2001/XMLSchema"
- xmlns:fn="http://www.w3.org/2005/xpath-functions"
- xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
- xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions"
- xmlns:buggy="urn:org.apache.daffodil.layers.buggy"
- xmlns:tns="urn:org.apache.daffodil.layers.buggy"
- targetNamespace="urn:org.apache.daffodil.layers.buggy"
- elementFormDefault="unqualified">
-
- <include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd" />
+ xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns="http://www.w3.org/2001/XMLSchema"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions"
+ xmlns:lf="urn:org.apache.daffodil.layers.lineFolded"
+ targetNamespace="urn:org.apache.daffodil.layers.lineFolded">
<annotation>
<appinfo source="http://www.ogf.org/dfdl/">
- <dfdl:format ref="buggy:GeneralFormat" />
+ <!-- lineFolded layers do not have any DFDL variables for parameter or results
- <dfdl:defineFormat name="buggyFormat">
- <dfdl:format dfdlx:layerTransform="buggy" dfdlx:layerLengthKind="explicit" dfdlx:layerLengthUnits="bytes"
- dfdlx:layerEncoding="ascii" />
- </dfdl:defineFormat>
+ The two layers are named lineFolded_IMF and lineFolded_iCalendar -->
</appinfo>
</annotation>
-</schema>
+</schema>
\ No newline at end of file
diff --git a/daffodil-io/src/main/scala/org/apache/daffodil/io/LimitingJavaOutputStreams.scala b/daffodil-runtime1-layers/src/main/scala/org/apache/daffodil/layers/runtime1/Base64MimeLayer.scala
similarity index 62%
copy from daffodil-io/src/main/scala/org/apache/daffodil/io/LimitingJavaOutputStreams.scala
copy to daffodil-runtime1-layers/src/main/scala/org/apache/daffodil/layers/runtime1/Base64MimeLayer.scala
index ed54a7662..dddff7490 100644
--- a/daffodil-io/src/main/scala/org/apache/daffodil/io/LimitingJavaOutputStreams.scala
+++ b/daffodil-runtime1-layers/src/main/scala/org/apache/daffodil/layers/runtime1/Base64MimeLayer.scala
@@ -14,27 +14,20 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.daffodil.io
-import java.nio.charset.Charset
+package org.apache.daffodil.layers.runtime1
-class LayerBoundaryMarkInsertingJavaOutputStream(
- jos: java.io.OutputStream,
- boundaryMark: String,
- charset: Charset,
-) extends java.io.FilterOutputStream(jos) {
+import java.io.InputStream
+import java.io.OutputStream
- private var closed = false
+import org.apache.daffodil.runtime1.layers.api.Layer
- private val boundaryMarkBytes = boundaryMark.getBytes(charset)
+final class Base64MimeLayer
+ extends Layer("base64_MIME", "urn:org.apache.daffodil.layers.base64_MIME") {
- override def close(): Unit = {
- if (!closed) {
- jos.write(boundaryMarkBytes)
- jos.flush()
- jos.close()
- closed = true
- }
- }
+ override def wrapLayerOutput(jos: OutputStream): OutputStream =
+ java.util.Base64.getMimeEncoder().wrap(jos)
+ override def wrapLayerInput(jis: InputStream): InputStream =
+ java.util.Base64.getMimeDecoder.wrap(jis)
}
diff --git a/daffodil-runtime1-layers/src/main/scala/org/apache/daffodil/layers/runtime1/Base64Transformer.scala b/daffodil-runtime1-layers/src/main/scala/org/apache/daffodil/layers/runtime1/Base64Transformer.scala
deleted file mode 100644
index 38a05ca68..000000000
--- a/daffodil-runtime1-layers/src/main/scala/org/apache/daffodil/layers/runtime1/Base64Transformer.scala
+++ /dev/null
@@ -1,99 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.daffodil.layers.runtime1
-
-import org.apache.daffodil.io.BoundaryMarkLimitingStream
-import org.apache.daffodil.io.LayerBoundaryMarkInsertingJavaOutputStream
-import org.apache.daffodil.lib.schema.annotation.props.gen.LayerLengthKind
-import org.apache.daffodil.runtime1.layers._
-import org.apache.daffodil.runtime1.processors.ParseOrUnparseState
-
-final class Base64MIMELayerCompiler extends LayerCompiler("base64_MIME") {
-
- override def compileLayer(
- layerCompileInfo: LayerCompileInfo,
- ): Base64MIMETransformerFactory = {
-
- layerCompileInfo.SDEUnless(
- scala.util.Properties.isJavaAtLeast("1.8"),
- "Base64 layer support requires Java 8 (aka Java 1.8).",
- )
-
- layerCompileInfo.SDEUnless(
- layerCompileInfo.optLayerBoundaryMarkOptConstantValue.isDefined,
- "Property dfdlx:layerBoundaryMark was not defined.",
- )
- layerCompileInfo.SDEUnless(
- layerCompileInfo.optLayerLengthKind.isEmpty ||
- (layerCompileInfo.optLayerLengthKind.get eq LayerLengthKind.BoundaryMark),
- "Only dfdlx:layerLengthKind 'boundaryMark' is supported, but '%s' was specified",
- layerCompileInfo.optLayerLengthKind.get.toString,
- )
-
- layerCompileInfo.SDEUnless(
- layerCompileInfo.optLayerJavaCharsetOptConstantValue match {
- case Some(Some(_)) => true
- case _ => false
- },
- "Property dfdlx:layerEncoding must be defined.",
- )
-
- val xformer = new Base64MIMETransformerFactory(name)
- xformer
- }
-}
-
-final class Base64MIMETransformerFactory(name: String) extends LayerTransformerFactory(name) {
-
- override def newInstance(layerRuntimeInfo: LayerRuntimeInfo) = {
- val xformer = new Base64MIMETransformer(name, layerRuntimeInfo)
- xformer
- }
-}
-
-final class Base64MIMETransformer(name: String, layerRuntimeInfo: LayerRuntimeInfo)
- extends LayerTransformer(name, layerRuntimeInfo) {
-
- override def wrapLayerDecoder(jis: java.io.InputStream): java.io.InputStream = {
- val b64 = java.util.Base64.getMimeDecoder().wrap(jis)
- b64
- }
-
- override def wrapLimitingStream(state: ParseOrUnparseState, jis: java.io.InputStream) = {
- val javaCharset = layerRuntimeInfo.optLayerCharset(state).get
- val layerBoundaryMark = layerRuntimeInfo.optLayerBoundaryMark(state).get
- val s = BoundaryMarkLimitingStream(jis, layerBoundaryMark, javaCharset)
- s
- }
-
- override protected def wrapLayerEncoder(jos: java.io.OutputStream): java.io.OutputStream = {
- val b64 = java.util.Base64.getMimeEncoder().wrap(jos)
- b64
- }
-
- override protected def wrapLimitingStream(
- state: ParseOrUnparseState,
- jos: java.io.OutputStream,
- ) = {
- val javaCharset = layerRuntimeInfo.optLayerCharset(state).get
- val layerBoundaryMark = layerRuntimeInfo.optLayerBoundaryMark(state).get
- val newJOS =
- new LayerBoundaryMarkInsertingJavaOutputStream(jos, layerBoundaryMark, javaCharset)
- newJOS
- }
-}
diff --git a/daffodil-runtime1-layers/src/main/scala/org/apache/daffodil/layers/runtime1/BoundaryMarkLayer.scala b/daffodil-runtime1-layers/src/main/scala/org/apache/daffodil/layers/runtime1/BoundaryMarkLayer.scala
new file mode 100644
index 000000000..c4c63f8dc
--- /dev/null
+++ b/daffodil-runtime1-layers/src/main/scala/org/apache/daffodil/layers/runtime1/BoundaryMarkLayer.scala
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.daffodil.layers.runtime1
+
+import java.io.InputStream
+import java.io.OutputStream
+import java.nio.charset.Charset
+import java.nio.charset.IllegalCharsetNameException
+import java.nio.charset.UnsupportedCharsetException
+
+import org.apache.daffodil.io.BoundaryMarkInsertingJavaOutputStream
+import org.apache.daffodil.io.BoundaryMarkLimitingInputStream
+import org.apache.daffodil.runtime1.layers.ScalaLayerHelper
+import org.apache.daffodil.runtime1.layers.api.Layer
+
+/**
+ * A layer which isolates text data by way of a boundary mark string.
+ * This is like a terminating delimiter of a DFDL element or sequence/choice group,
+ * except that there is no escaping mechanism, so the data cannot in any way
+ * contain the boundary mark string, and elements within the layer can have
+ * any dfdl:lengthKind, but this does not affect the search for the boundary mark.
+ *
+ * For example, when using an element with dfdl:lengthKind="delimited" and a dfdl:terminator="END",
+ * if that element contains a child element with dfdl:lengthKind="explicit", then the
+ * search for the "END" terminator is suspended for the length of the child element, and
+ * that search resumes after the child element's length has been parsed.
+ *
+ * In contrast to this, if a boundary mark layer is used with the boundaryMark variable
+ * bound to "END", then the data stream is decoded as characters in the charset encoding
+ * given by the layerEncoding variable, and the layer continues until the "END" is found.
+ * The dfdl:lengthKind of any child element enclosed within the layer, or even the lengths
+ * of other layers found within the scope of this boundary mark layer are not considered and do not
+ * disrupt the search for the boundary mark string.
+ *
+ * This layer does not populate any DFDL variables with results.
+ *
+ * This layer defines two DFDL variables which provide required parameters.
+ */
+final class BoundaryMarkLayer
+ extends Layer("boundaryMark", "urn:org.apache.daffodil.layers.boundaryMark")
+ with ScalaLayerHelper {
+
+ private var boundaryMark: String = _
+ private var charset: Charset = _
+
+ private val maxBoundaryMarkLength: Int = Short.MaxValue
+
+ // initialization. These exceptions get converted to processing errors
+ setProcessingErrorException(classOf[IllegalCharsetNameException])
+ setProcessingErrorException(classOf[UnsupportedCharsetException])
+
+ /**
+ * Provides the layer's parameter variables to the layer.
+ *
+ * @param boundaryMark a string which is the boundary marker. Searching for this string is
+ * done without any notion of escaping. When parsing the data in the
+ * layer is up to but not including that of the boundary mark string,
+ * which is removed from the data-stream on parsing, and inserted into
+ * the data stream on unparsing.
+ * @param layerEncoding a string which is the name of the character set encoding used to
+ * decode characters during the search for the boundary mark, and used
+ * to encode characters when inserting the boundary mark when unparsing.
+ */
+ private[layers] def setLayerVariableParameters(
+ boundaryMark: String,
+ layerEncoding: String,
+ ): Unit = {
+ this.boundaryMark = boundaryMark
+ if (boundaryMark.isEmpty)
+ procError("The boundaryMark variable value may not be empty string.")
+ if (boundaryMark.length > maxBoundaryMarkLength)
+ procError(
+ s"The boundaryMark string length may not be greater than the limit: $maxBoundaryMarkLength",
+ )
+ this.charset = Charset.forName(layerEncoding)
+ }
+
+ override def wrapLayerInput(jis: InputStream): InputStream = {
+ new BoundaryMarkLimitingInputStream(jis, boundaryMark, charset)
+ }
+
+ override def wrapLayerOutput(jos: OutputStream): OutputStream =
+ new BoundaryMarkInsertingJavaOutputStream(jos, boundaryMark, charset)
+
+}
diff --git a/daffodil-runtime1-layers/src/main/scala/org/apache/daffodil/layers/runtime1/ByteSwapTransformer.scala b/daffodil-runtime1-layers/src/main/scala/org/apache/daffodil/layers/runtime1/ByteSwapLayer.scala
similarity index 62%
rename from daffodil-runtime1-layers/src/main/scala/org/apache/daffodil/layers/runtime1/ByteSwapTransformer.scala
rename to daffodil-runtime1-layers/src/main/scala/org/apache/daffodil/layers/runtime1/ByteSwapLayer.scala
index b89a43ae9..387022288 100644
--- a/daffodil-runtime1-layers/src/main/scala/org/apache/daffodil/layers/runtime1/ByteSwapTransformer.scala
+++ b/daffodil-runtime1-layers/src/main/scala/org/apache/daffodil/layers/runtime1/ByteSwapLayer.scala
@@ -22,53 +22,29 @@ import java.io.OutputStream
import java.util.ArrayDeque
import java.util.Deque
-import org.apache.daffodil.io.ExplicitLengthLimitingStream
import org.apache.daffodil.lib.exceptions.Assert
-import org.apache.daffodil.lib.schema.annotation.props.gen.LayerLengthKind
-import org.apache.daffodil.runtime1.layers._
-import org.apache.daffodil.runtime1.processors.ParseOrUnparseState
-
-final class FourByteSwapLayerCompiler extends LayerCompiler("fourbyteswap") {
-
- override def compileLayer(layerCompileInfo: LayerCompileInfo): LayerTransformerFactory = {
-
- layerCompileInfo.optLayerLengthKind match {
- case Some(LayerLengthKind.Explicit) => // ok
- case None =>
- layerCompileInfo.SDE(
- "The property dfdlx:layerLengthKind must be defined and have value 'explicit'.",
- )
- case Some(other) =>
- layerCompileInfo.SDE(
- "The property dfdlx:layerLengthKind must be 'explicit' but was '$other'.",
- )
- }
+import org.apache.daffodil.runtime1.layers.api.Layer
- layerCompileInfo.SDEUnless(
- layerCompileInfo.optLayerLengthOptConstantValue.isDefined,
- "The property dfdlx:layerLength must be defined.",
- )
+final class TwoByteSwapLayer extends ByteSwap("twobyteswap", 2)
- val xformer = new ByteSwapTransformerFactory(4, name)
- xformer
- }
-}
+final class FourByteSwapLayer extends ByteSwap("fourbyteswap", 4)
-/**
- * LayerTransformerFactory for ByteSwapTransformer instances.
- *
- * Requires that layerLengthKind="explicit".
- */
-final class ByteSwapTransformerFactory(wordsize: Int, name: String)
- extends LayerTransformerFactory(name) {
+abstract class ByteSwap(name: String, count: Int)
+ extends Layer(name, "urn:org.apache.daffodil.layers.byteSwap") {
- override def newInstance(layerRuntimeInfo: LayerRuntimeInfo) =
- new ByteSwapTransformer(wordsize, name, layerRuntimeInfo)
+ override def wrapLayerOutput(jos: OutputStream): OutputStream =
+ new ByteSwapOutputStream(count, jos)
+
+ override def wrapLayerInput(jis: InputStream): InputStream =
+ new ByteSwapInputStream(count, jis)
}
/**
* An input stream wrapper that re-orders bytes according to wordsize.
*
+ * This is streaming - does not require buffering up the data. So can be used on
+ * very large data objects.
+ *
* Bytes within the wrapped input stream are re-ordered wordsize bytes at a time.
* For example, if the wrapped input stream contains 10 bytes and wordsize is
* 4, then the bytes from the wrapped input stream are returned in the
@@ -153,6 +129,8 @@ class ByteSwapInputStream(wordsize: Int, jis: InputStream) extends InputStream {
/**
* An output stream wrapper that re-orders bytes according to wordsize.
*
+ * This is streaming, so is suitable for use on very large data.
+ *
* Bytes sent to the wrapped output stream are re-ordered wordsize bytes at a
* time. For example, if 10 bytes are written and wordsize is 4, then the
* bytes are written to the wrapped output stream in the
@@ -184,36 +162,3 @@ class ByteSwapOutputStream(wordsize: Int, jos: OutputStream) extends OutputStrea
}
}
}
-
-/**
- * A LayerTransformer that re-orders bytes for the over/underlying layer.
- */
-class ByteSwapTransformer(wordsize: Int, name: String, layerRuntimeInfo: LayerRuntimeInfo)
- extends LayerTransformer(name, layerRuntimeInfo) {
-
- override def wrapLayerDecoder(jis: java.io.InputStream) = {
- val s = new ByteSwapInputStream(wordsize, jis)
- s
- }
-
- override def wrapLimitingStream(state: ParseOrUnparseState, jis: java.io.InputStream) = {
- val layerLengthInBytes = layerRuntimeInfo.optLayerLength(state).get
-
- val s = new ExplicitLengthLimitingStream(jis, layerLengthInBytes)
- s
- }
-
- override protected def wrapLayerEncoder(jos: java.io.OutputStream): java.io.OutputStream = {
- val s = new ByteSwapOutputStream(wordsize, jos)
- s
- }
-
- override protected def wrapLimitingStream(
- state: ParseOrUnparseState,
- jos: java.io.OutputStream,
- ) = {
- jos // just return jos. The way the length will be used/stored is by way of
- // taking the content length of the enclosing element. That will measure the
- // length relative to the "ultimate" data output stream.
- }
-}
diff --git a/daffodil-runtime1-layers/src/main/scala/org/apache/daffodil/layers/runtime1/FixedLengthLayer.scala b/daffodil-runtime1-layers/src/main/scala/org/apache/daffodil/layers/runtime1/FixedLengthLayer.scala
new file mode 100644
index 000000000..cb4954276
--- /dev/null
+++ b/daffodil-runtime1-layers/src/main/scala/org/apache/daffodil/layers/runtime1/FixedLengthLayer.scala
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.daffodil.layers.runtime1
+
+import java.io.ByteArrayInputStream
+import java.io.ByteArrayOutputStream
+import java.io.InputStream
+import java.io.OutputStream
+import java.nio.ByteBuffer
+
+import org.apache.daffodil.lib.exceptions.Assert
+import org.apache.daffodil.runtime1.layers.LayerNotEnoughDataException
+import org.apache.daffodil.runtime1.layers.api.Layer
+
+import org.apache.commons.io.IOUtils
+
+/**
+ * Suitable only for small sections of data, not large data streams or whole files.
+ * See the maxFixedLength value defined herein for the maximum.
+ *
+ * The entire fixed length region of the data will be pulled into a byte buffer in memory.
+ *
+ * TODO: Someday, enhance to make this streaming.
+ *
+ * One DFDL Variable is a parameter
+ * - fixedLength - an unsignedInt giving the fixed length of this layer.
+ * This length is enforced on both parsing and unparsing the layer.
+ * There are no output/result DFDL variables from this layer.
+ */
+final class FixedLengthLayer
+ extends Layer("fixedLength", "urn:org.apache.daffodil.layers.fixedLength") {
+
+ private var fixedLength: Int = -1
+
+ private def maxFixedLength = Short.MaxValue
+
+ /**
+ * Captures the fixedLength DFDL variable value and saves it.
+ *
+ * Also validates whether it is in acceptable range. Because the fixedLength variable
+ * can be populated from an expression that consumes input data, it's possible for the
+ * fixedLength to be out of range as a result of speculative parsing down
+ * an incorrect path. Hence, it is a processing error, not a schema definition error
+ * if the fixedLength variable value is out of range.
+ *
+ * @param fixedLength matches the name of the DFDL variable which will supply this value.
+ */
+ private[layers] def setLayerVariableParameters(fixedLength: Long): Unit = {
+ this.fixedLength = fixedLength.toInt
+ Assert.invariant(fixedLength >= 0) // variable is unsignedInt, so this can't be negative
+ if (fixedLength > maxFixedLength)
+ processingError(
+ s"fixedLength value of $fixedLength is above the maximum of $maxFixedLength.",
+ )
+ }
+
+ override def wrapLayerInput(jis: InputStream): InputStream = {
+ new FixedLengthInputStream(jis)
+ }
+
+ override def wrapLayerOutput(jos: OutputStream): OutputStream = {
+ new FixedLengthOutputStream(jos)
+ }
+
+ class FixedLengthInputStream(jis: InputStream) extends InputStream {
+
+ private lazy val bais = {
+ val ba = new Array[Byte](fixedLength.toInt)
+ val nRead = IOUtils.read(jis, ba)
+ if (nRead < fixedLength)
+ processingError(new LayerNotEnoughDataException(fixedLength, nRead))
+ val buf = ByteBuffer.wrap(ba)
+ new ByteArrayInputStream(ba)
+ }
+
+ override def read(): Int = bais.read()
+ }
+
+ class FixedLengthOutputStream(jos: OutputStream) extends OutputStream {
+
+ private lazy val baos = new ByteArrayOutputStream(fixedLength)
+
+ private var count: Long = 0
+
+ override def write(b: Int): Unit = {
+ baos.write(b)
+ count += 1
+ if (count == fixedLength) {
+ // we can auto-close it in this case
+ close()
+ } else if (count > fixedLength) {
+ //
+ // This could happen if the layer logically unparses as one of two choice branches where they
+ // are supposed to be all the same length, but one is in fact longer than expected by the bufLen.
+ processingError(
+ new IndexOutOfBoundsException(
+ s"Written data amount exceeded fixed layer length of $fixedLength.",
+ ),
+ )
+ } else {
+ // ok. We're still accumulating data
+ }
+ }
+
+ private var isOpen: Boolean = true
+
+ override def close(): Unit = {
+ if (isOpen) { // allow multiple closes
+ isOpen = false
+ val ba = baos.toByteArray
+ val baLen = ba.length
+ if (baLen != fixedLength)
+ processingError(
+ s"Insufficient output data for fixed-length layer. Needed $fixedLength bytes, but only $baLen were unparsed.",
+ )
+ jos.write(ba)
+ }
+ }
+ }
+}
diff --git a/daffodil-runtime1-layers/src/main/scala/org/apache/daffodil/layers/runtime1/GZipTransformer.scala b/daffodil-runtime1-layers/src/main/scala/org/apache/daffodil/layers/runtime1/GZipLayer.java
similarity index 53%
rename from daffodil-runtime1-layers/src/main/scala/org/apache/daffodil/layers/runtime1/GZipTransformer.scala
rename to daffodil-runtime1-layers/src/main/scala/org/apache/daffodil/layers/runtime1/GZipLayer.java
index 4d68dd997..09b3b5096 100644
--- a/daffodil-runtime1-layers/src/main/scala/org/apache/daffodil/layers/runtime1/GZipTransformer.scala
+++ b/daffodil-runtime1-layers/src/main/scala/org/apache/daffodil/layers/runtime1/GZipLayer.java
@@ -15,78 +15,57 @@
* limitations under the License.
*/
-package org.apache.daffodil.layers.runtime1
-
-import org.apache.daffodil.io.ExplicitLengthLimitingStream
-import org.apache.daffodil.lib.schema.annotation.props.gen.LayerLengthKind
-import org.apache.daffodil.runtime1.layers._
-import org.apache.daffodil.runtime1.processors.ParseOrUnparseState
-
-final class GZIPLayerCompiler extends LayerCompiler("gzip") {
-
- override def compileLayer(layerCompileInfo: LayerCompileInfo): GZIPTransformerFactory = {
-
- layerCompileInfo.SDEUnless(
- layerCompileInfo.optLayerLengthKind.isEmpty ||
- (layerCompileInfo.optLayerLengthKind.get eq LayerLengthKind.Explicit),
- "Only dfdlx:layerLengthKind 'explicit' is supported, but '%s' was specified",
- layerCompileInfo.optLayerLengthKind.get.toString,
- )
-
- val xformer = new GZIPTransformerFactory(name)
- xformer
- }
-}
-
-final class GZIPTransformerFactory(name: String) extends LayerTransformerFactory(name) {
-
- override def newInstance(layerRuntimeInfo: LayerRuntimeInfo) = {
- val xformer = new GZIPTransformer(name, layerRuntimeInfo)
- xformer
- }
-}
-
-class GZIPTransformer(name: String, layerRuntimeInfo: LayerRuntimeInfo)
- extends LayerTransformer(name, layerRuntimeInfo) {
-
- override def wrapLayerDecoder(jis: java.io.InputStream) = {
- val s = new java.util.zip.GZIPInputStream(jis)
- s
+package org.apache.daffodil.layers.runtime1;
+
+import org.apache.daffodil.runtime1.layers.api.Layer;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.Objects;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+public final class GZipLayer extends Layer {
+
+ private static Boolean fixNeeded = null;
+
+ public static boolean fixIsNeeded() {
+ if (Objects.isNull(fixNeeded)) {
+ // prior to java 16
+ String versionString = System.getProperty("java.version");
+
+ // Extract the major version using string manipulation
+ int majorVersion;
+ if (versionString.startsWith("1.8")) {
+ majorVersion = 8;
+ } else {
+ String[] parts = versionString.split("\\.");
+ assert(parts.length > 0);
+ majorVersion = Integer.parseInt(parts[0]);
+ }
+ fixNeeded = (majorVersion < 16);
+ }
+ return fixNeeded;
}
- override def wrapLimitingStream(state: ParseOrUnparseState, jis: java.io.InputStream) = {
- val layerLengthInBytes = layerRuntimeInfo.optLayerLength(state).get
- val s = new ExplicitLengthLimitingStream(jis, layerLengthInBytes)
- s
+ public GZipLayer() {
+ super("gzip", "urn:org.apache.daffodil.layers.gzip");
+ // convert IOExceptions to Processing Errors
+ setProcessingErrorException(IOException.class);
}
- override protected def wrapLayerEncoder(jos: java.io.OutputStream): java.io.OutputStream = {
- val s = GZIPFixedOutputStream(jos)
- s
+ @Override
+ public InputStream wrapLayerInput(InputStream jis) throws Exception {
+ return new GZIPInputStream(jis);
}
- override protected def wrapLimitingStream(
- state: ParseOrUnparseState,
- jis: java.io.OutputStream,
- ) = {
- jis // just return jis. The way the length will be used/stored is by way of
- // taking the content length of the enclosing element. That will measure the
- // length relative to the "ultimate" data output stream.
+ @Override
+ public OutputStream wrapLayerOutput(OutputStream jos) throws Exception {
+ OutputStream fixedOS = fixIsNeeded() ? new GZIPFixedOutputStream(jos) : jos;
+ return new GZIPOutputStream(fixedOS);
}
-}
-object GZIPFixedOutputStream {
-
- private val fixIsNeeded = !scala.util.Properties.isJavaAtLeast("16")
-
- /**
- * Create a GZIPOutputStream that, if necessary, proxies writes through an
- * OutputStream that fixes inconsistencies between Java versions
- */
- def apply(os: java.io.OutputStream) = {
- val fixedOS = if (fixIsNeeded) new GZIPFixedOutputStream(os) else os
- new java.util.zip.GZIPOutputStream(fixedOS)
- }
}
/**
@@ -103,25 +82,35 @@ object GZIPFixedOutputStream {
* GZIPFixedOutputStream, this will always write a value of 255, making all Java
* versions prior to 16 consistent with Java 16+ behavior.
*/
-class GZIPFixedOutputStream private (os: java.io.OutputStream) extends java.io.OutputStream {
+class GZIPFixedOutputStream extends OutputStream {
+
+ private final OutputStream os;
+
+ public GZIPFixedOutputStream(OutputStream os) {
+ this.os = os;
+ }
/**
* The next byte position that byte will be written to. If this is negative,
* that means we have already fixed the output and everything should just
* pass straight through.
*/
- private var bytePosition = 0
+ private int bytePosition = 0;
+
+ @Override
+ public void close() throws IOException { os.close(); }
- override def close(): Unit = os.close()
- override def flush(): Unit = os.flush()
+ @Override
+ public void flush() throws IOException { os.flush(); }
- override def write(b: Array[Byte], off: Int, len: Int): Unit = {
+ @Override
+ public void write(byte[] b, int off, int len) throws IOException {
if (bytePosition < 0) {
// The bad byte has been fixed, pass all writes directly through to the
// underlying OutputStream. This may be more efficient than the default
// OutputStream write() function, which writes the bytes from this array
// one at a time
- os.write(b, off, len)
+ os.write(b, off, len);
} else {
// The bad byte has not been fixed yet. Unless a newer version of Java
// has made changes, the GZIPOutputStreamm will have passed in a 10 byte
@@ -130,27 +119,28 @@ class GZIPFixedOutputStream private (os: java.io.OutputStream) extends java.io.O
// bytes one at a time and will call the write(int) method that will fix
// that byte. Calling write() one at a time is maybe inefficient but for
// such a small array it should not have a noticeable effect.
- super.write(b, off, len)
+ super.write(b, off, len);
}
}
- override def write(b: Int): Unit = {
+ @Override
+ public void write(int b) throws IOException {
if (bytePosition < 0) {
// The bad byte has already been fixed, simply pass this byte through to
// the underlying OutputStream
- os.write(b)
+ os.write(b);
} else if (bytePosition < 9) {
// The bad byte has not been fixed, and we haven't reached it yet, simply
// pass this byte through and increment our byte position
- os.write(b)
- bytePosition += 1
+ os.write(b);
+ bytePosition += 1;
} else if (bytePosition == 9) {
// This is the bad byte, it is a 0 on some Java versions. Write 255
// instead of to match Java 16+ behavior. Also, set bytePosition to -1 to
// signify that we have fixed the bad byte and that all other writes
// should just pass directly to the underlying OutputStream
- os.write(255)
- bytePosition = -1
+ os.write(255);
+ bytePosition = -1;
}
}
}
diff --git a/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/LayerTransformerFactory.scala b/daffodil-runtime1-layers/src/main/scala/org/apache/daffodil/layers/runtime1/LineFoldMode.java
similarity index 63%
copy from daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/LayerTransformerFactory.scala
copy to daffodil-runtime1-layers/src/main/scala/org/apache/daffodil/layers/runtime1/LineFoldMode.java
index 2364fa1ba..98b82e1ed 100644
--- a/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/LayerTransformerFactory.scala
+++ b/daffodil-runtime1-layers/src/main/scala/org/apache/daffodil/layers/runtime1/LineFoldMode.java
@@ -14,18 +14,19 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
+package org.apache.daffodil.layers.runtime1;
-package org.apache.daffodil.runtime1.layers
+public enum LineFoldMode {
+ /**
+ * IMF - Internet Mail Format
+ */
+ IMF("lineFolded_IMF"),
+ /**
+ * iCalendar
+ */
+ iCalendar("lineFolded_iCalendar");
-/**
- * Factory for a layer transformer.
- *
- * This is the serialized object which is saved as part of a processor.
- * It constructs the layer transformer at runtime when newInstance() is called.
- *
- * This must be implmented as part of implementation of a daffodil layer.
- */
-abstract class LayerTransformerFactory(val name: String) extends Serializable {
+ public final String dfdlName;
+ private LineFoldMode(final String dfdlName) { this.dfdlName = dfdlName; }
- def newInstance(layerRuntimeInfo: LayerRuntimeInfo): LayerTransformer
}
diff --git a/daffodil-runtime1-layers/src/main/scala/org/apache/daffodil/layers/runtime1/LineFoldedTransformer.scala b/daffodil-runtime1-layers/src/main/scala/org/apache/daffodil/layers/runtime1/LineFoldedLayerBase.scala
similarity index 63%
rename from daffodil-runtime1-layers/src/main/scala/org/apache/daffodil/layers/runtime1/LineFoldedTransformer.scala
rename to daffodil-runtime1-layers/src/main/scala/org/apache/daffodil/layers/runtime1/LineFoldedLayerBase.scala
index 8e3c92289..1230f0d0b 100644
--- a/daffodil-runtime1-layers/src/main/scala/org/apache/daffodil/layers/runtime1/LineFoldedTransformer.scala
+++ b/daffodil-runtime1-layers/src/main/scala/org/apache/daffodil/layers/runtime1/LineFoldedLayerBase.scala
@@ -19,17 +19,9 @@ package org.apache.daffodil.layers.runtime1
import java.io.InputStream
import java.io.OutputStream
-import java.nio.charset.StandardCharsets
-import org.apache.daffodil.io.LayerBoundaryMarkInsertingJavaOutputStream
-import org.apache.daffodil.io.RegexLimitingStream
import org.apache.daffodil.lib.exceptions.Assert
-import org.apache.daffodil.lib.exceptions.ThrowsSDE
-import org.apache.daffodil.lib.schema.annotation.props.Enum
-import org.apache.daffodil.lib.schema.annotation.props.gen.LayerLengthKind
-import org.apache.daffodil.runtime1.layers._
-import org.apache.daffodil.runtime1.processors.ParseOrUnparseState
-
+import org.apache.daffodil.runtime1.layers.api.Layer
/*
* This and related classes implement so called "line folding" from
* IETF RFC 2822 Internet Message Format (IMF), and IETF RFC 5545 iCalendar.
@@ -56,13 +48,13 @@ import org.apache.daffodil.runtime1.processors.ParseOrUnparseState
* being preserved, so the above problem can occur.
*
* IMF has a maximum line length of 998 characters per line excluding the CRLF.
- * The layer will fail (cause a parse error) if a line longer than this is encountered
+ * The layer should fail (cause a parse error) if a line longer than this is encountered
* or constructed after unfolding. When unparsing, if a line longer than 998 cannot be
* folded due to no spaces/tabs being present in it, then it is an unparse error.
*
* Note that i/vCalendar, vCard, and MIME payloads held by IMF do not run into
* the IMF line length issues, in that they have their own line length limits that
- * are smaller than those of IMF, and which do not require accomodation by having
+ * are smaller than those of IMF, and which do not require accommodation by having
* pre-existing spaces/tabs in the data. So such data *always* will be short
* enough lines.
*
@@ -73,170 +65,42 @@ import org.apache.daffodil.runtime1.processors.ParseOrUnparseState
* the data is parsed.
*
* For MIME, the maximum line length is 76.
+ *
+ * This layer would normally be used in a context of either an enclosing element
+ * which bounds its length (dfdl:lengthKind 'explicit', 'prefixed', or 'pattern')
+ * or an enclosing layer that bounds its length (ex: boundaryMark).
+ *
+ * This layer can interact badly with surrounding elements of dfdl:lengthKind 'pattern'
+ * if the pattern is, for example ".*?\\r\\n(?!(?:\\t|\\ ))" which is anything up to
+ * and including a CRLF not followed by a space or tab. The problem is that this layer
+ * converts isolated \n or \r into \r\n, and if this just happens to be followed by a
+ * non space/tab character this will have inserted an end-of-data in the middle of the
+ * data.
*/
-sealed abstract class LineFoldedLayerCompiler(mode: LineFoldMode)
- extends LayerCompiler(mode.transformName) {
-
- override def compileLayer(
- layerCompileInfo: LayerCompileInfo,
- ): LineFoldedTransformerFactory = {
-
- layerCompileInfo.optLayerLengthKind match {
- case Some(LayerLengthKind.BoundaryMark) =>
- layerCompileInfo.SDEUnless(
- layerCompileInfo.optLayerBoundaryMarkOptConstantValue.isDefined,
- "Property dfdlx:layerBoundaryMark was not defined.",
- )
- case Some(LayerLengthKind.Implicit) => // ok
- case Some(other) =>
- layerCompileInfo.SDE(
- s"Property dfdlx:layerLengthKind can only be 'implicit' or 'boundaryMark', but was '$other'.",
- )
- case None =>
- layerCompileInfo.SDE(
- s"Property dfdlx:layerLengthKind must be 'implicit' or 'boundaryMark'.",
- )
- }
-
- //
- // This layer assumes an ascii-family encoding.
- //
-
- val xformer =
- new LineFoldedTransformerFactory(mode, layerCompileInfo.optLayerLengthKind.get)
- xformer
- }
-}
-
-final class LineFoldedIMFLayerCompiler extends LineFoldedLayerCompiler(LineFoldMode.IMF)
-final class LineFoldedICalendarLayerCompiler
- extends LineFoldedLayerCompiler(LineFoldMode.iCalendar)
-
-final class LineFoldedTransformerFactory(mode: LineFoldMode, layerLengthKind: LayerLengthKind)
- extends LayerTransformerFactory(mode.transformName) {
-
- override def newInstance(layerRuntimeInfo: LayerRuntimeInfo): LayerTransformer = {
- val xformer =
- layerLengthKind match {
- case LayerLengthKind.BoundaryMark => {
- new LineFoldedTransformerDelimited(mode, layerRuntimeInfo)
- }
- case LayerLengthKind.Implicit => {
- new LineFoldedTransformerImplicit(mode, layerRuntimeInfo)
- }
- case _ =>
- Assert.invariantFailed(
- "Should already have checked that it is only one of BoundaryMark or Implicit",
- )
- }
- xformer
- }
-}
-
-sealed trait LineFoldMode extends LineFoldMode.Value {
- def transformName: String = s"lineFolded_${this.toString}"
-}
-
-object LineFoldMode extends Enum[LineFoldMode] {
-
- case object IMF extends LineFoldMode
- case object iCalendar extends LineFoldMode
- override lazy val values = Array(IMF, iCalendar)
-
- override def apply(name: String, context: ThrowsSDE): LineFoldMode =
- stringToEnum("lineFoldMode", name, context)
-}
-
-/**
- * For line folded, the notion of "delimited" means that the element is a "line"
- * that ends with CRLF, except that if it is long, it will be folded, which involves
- * inserting/removing CRLF+Space (or CRLF+TAB). A CRLF not followed by space or tab
- * is ALWAYS the actual "delimiter". There's no means of supplying a specific delimiter.
- */
-final class LineFoldedTransformerDelimited(
- mode: LineFoldMode,
- layerRuntimeInfo: LayerRuntimeInfo,
-) extends LayerTransformer(mode.transformName, layerRuntimeInfo) {
-
- override protected def wrapLimitingStream(
- state: ParseOrUnparseState,
- jis: java.io.InputStream,
- ) = {
- // regex means CRLF not followed by space or tab.
- // NOTE: this regex cannot contain ANY capturing groups (per scaladoc on RegexLimitingStream)
- val s =
- new RegexLimitingStream(jis, "\\r\\n(?!(?:\\t|\\ ))", "\r\n", StandardCharsets.ISO_8859_1)
- s
- }
+sealed abstract class LineFoldedLayerBase(mode: LineFoldMode)
+ extends Layer(mode.dfdlName, "urn:org.apache.daffodil.layers.lineFolded") {
- override protected def wrapLimitingStream(
- state: ParseOrUnparseState,
- jos: java.io.OutputStream,
- ) = {
- //
- // Q: How do we insert a CRLF "not followed by tab or space" when we don't
- // control what follows?
- // A: We don't. This is nature of the format. If what follows could begin
- // with a space or tab, then the format can't use a line-folded layer.
- //
- val newJOS =
- new LayerBoundaryMarkInsertingJavaOutputStream(jos, "\r\n", StandardCharsets.ISO_8859_1)
- newJOS
- }
+ override def wrapLayerInput(jis: InputStream): InputStream =
+ new LineFoldedInputStream(mode, jis)
- override protected def wrapLayerDecoder(jis: java.io.InputStream): java.io.InputStream = {
- val s = new LineFoldedInputStream(mode, jis)
- s
- }
- override protected def wrapLayerEncoder(jos: java.io.OutputStream): java.io.OutputStream = {
- val s = new LineFoldedOutputStream(mode, jos)
- s
- }
+ override def wrapLayerOutput(jos: OutputStream): OutputStream =
+ new LineFoldedOutputStream(mode, jos)
}
-/**
- * For line folded, the 'implicit' length kind means that the region continues
- * to end of data. At top level this would be the "whole file/stream" but this can
- * also be used with a specified length enclosing element. This code cannot tell
- * the difference.
- */
-class LineFoldedTransformerImplicit(mode: LineFoldMode, layerRuntimeInfo: LayerRuntimeInfo)
- extends LayerTransformer(mode.transformName, layerRuntimeInfo) {
-
- override protected def wrapLimitingStream(
- state: ParseOrUnparseState,
- jis: java.io.InputStream,
- ) = {
- jis // no limiting - just pull input until EOF.
- }
+final class LineFoldedIMFLayer extends LineFoldedLayerBase(LineFoldMode.IMF)
- override protected def wrapLimitingStream(
- state: ParseOrUnparseState,
- jos: java.io.OutputStream,
- ) = {
- jos // no limiting - just write output until EOF.
- }
-
- override protected def wrapLayerDecoder(jis: java.io.InputStream): java.io.InputStream = {
- val s = new LineFoldedInputStream(mode, jis)
- s
- }
- override protected def wrapLayerEncoder(jos: java.io.OutputStream): java.io.OutputStream = {
- val s = new LineFoldedOutputStream(mode, jos)
- s
- }
-}
+final class LineFoldedICalendarLayer extends LineFoldedLayerBase(LineFoldMode.iCalendar)
/**
- * Doesn't enforce 998 max line length limit.
+ * TODO: Doesn't enforce 998 max line length limit.
*
* This is a state machine, so of course must be used only on a single thread.
*/
class LineFoldedInputStream(mode: LineFoldMode, jis: InputStream) extends InputStream {
object State extends org.apache.daffodil.lib.util.Enum {
- abstract sealed trait Type extends EnumValueType
+ sealed trait Type extends EnumValueType
/**
* No state. Read a character, and if CR, go to state GotCR.
@@ -285,8 +149,18 @@ class LineFoldedInputStream(mode: LineFoldMode, jis: InputStream) extends InputS
* of the encoding. This enables it to handle data where a CRLF was inserted
* to limit line length, and that insertion broke up a multi-byte character.
*
+ * This will work for UTF-8 because it's multi-byte sequences do not reuse
+ * bytes that could be mistaken for \r or \n.
+ *
+ * This is unsafe for UTF-16BE or other multi-byte encodings.
+ * For example: the UTF-16 codepoint 0x0D0A is a character
+ * (MALAYALAM LETTER UU) 'ഊ' which if read a byte at a time looks exactly
+ * like a CRLF.
+ *
* Does not detect errors such as isolated \r or isolated \n. Leaves those
- * alone. Does not care if lines are in fact less than any limit in length.
+ * alone.
+ *
+ * TODO: Does not implement line-length limit of 998.
*
* All this does is remove \r\n[\ \t], replacing with just the space or tab.(IMF)
* or replace with nothing (iCalendar).
@@ -333,7 +207,7 @@ class LineFoldedInputStream(mode: LineFoldMode, jis: InputStream) extends InputS
c = jis.read()
c match {
case ' ' | '\t' => {
- if (mode eq LineFoldMode.IMF) {
+ if (mode == LineFoldMode.IMF) {
state = Start // absorb the CR, LF, but not the sp/tab
return c // return the sp/tab
} else {
@@ -345,9 +219,10 @@ class LineFoldedInputStream(mode: LineFoldMode, jis: InputStream) extends InputS
}
case _ => {
// CRLF followed by other not sp/tab, or end of data.
- // Buffer up to LF.
+ // Note: This can happen when layer length is just implicit because the
+ // layer does not end at the first \r\n that is not followed by tab/space.
state = Buf2
- return '\r'
+ return '\r' // We are preserving the CR in this case (and the LF next call)
}
}
}
@@ -367,8 +242,30 @@ class LineFoldedInputStream(mode: LineFoldMode, jis: InputStream) extends InputS
}
}
+/*
+ * TODO: Currently this normalizes isolated \n and \r to \r\n, which is not symmetric
+ * with the LineFoldedInputStream class. This needs to preserve isolated \n and \r.
+ * |
+ * Or, alternatively, the LineFoldedInputStream should remove any line-ending followed by
+ * a space or tab, whether it is \r\n or just \r or \n.
+ * |
+ * (We should follow the RFC
+ * details unless there is evidence of de-facto data that uses line-folding with just \n,
+ * not \r\n.)
+ */
class LineFoldedOutputStream(mode: LineFoldMode, jos: OutputStream) extends OutputStream {
-
+ //
+ // TODO: To work properly this state machine needs more than just the line buffer as state. It needs to
+ // suspend the decision about what to do with isolated \r and \n until it sees the next character after (if any).
+ // Consider just an isolated \n. If write(\n) is the last call, then no subsequent character exists,
+ // but we can't know what do do with the \n until the close() happens.
+ // At that point we know there is no following character, and the \n can be
+ // converted to a \r\n. But if the write(\n) call is followed by a write of a space or tab, then we
+ // output the \r\n and the space or tab character.
+ // If the write(\n) call is followed by a write(x) where x is not tab or space, then we must output
+ // \r\n and a tab or space so that this does NOT become an artificial end-of-data situation.
+ // (Similar treatment is needed for isolated \r.)
+ //
private val (lineLength, breaker) = mode match {
case LineFoldMode.IMF => (78, "\r\n".getBytes("ascii"))
case LineFoldMode.iCalendar => (75, "\r\n ".getBytes("ascii")) // CRLF + a space.
@@ -394,10 +291,10 @@ class LineFoldedOutputStream(mode: LineFoldMode, jos: OutputStream) extends Outp
}
override def write(bInt: Int): Unit = {
- val b = bInt.toChar
Assert.usage(!closed)
Assert.invariant(line.length <= lineLength)
- Assert.usage(b >= 0)
+ Assert.usage(bInt >= 0)
+ val b = bInt.toChar
if (line.length < lineLength) {
// there's room for more on the line
b match {
@@ -450,8 +347,7 @@ class LineFoldedOutputStream(mode: LineFoldMode, jos: OutputStream) extends Outp
private def tooLongLineICalendar(b: Char): Unit = {
line += '\r'
line += '\n'
- if (mode eq LineFoldMode.iCalendar)
- line += ' '
+ line += ' '
jos.write(lineBytes)
line.clear()
line += b
@@ -469,6 +365,7 @@ class LineFoldedOutputStream(mode: LineFoldMode, jos: OutputStream) extends Outp
* Ideally it would not be a space immediately before punctuation, as that
* would affect readability (for people) by starting the next line with a
* punctuation character. But this is, strictly speaking, unnecessary.
+ * (It is recommended by the RFC however, so it is likely expected behavior.)
*/
private def tooLongLineIMF(b: Char): Unit = {
var i = line.length - 1
@@ -477,6 +374,8 @@ class LineFoldedOutputStream(mode: LineFoldMode, jos: OutputStream) extends Outp
val c = line(i)
c match {
case ' ' | '\t' => {
+ // TODO: Implement check for punctuation immediately after the space or tab
+ // as that makes this an undesirable place for a line-ending to be inserted.
done = true
//
// i is the index of latest sp/tab
diff --git a/daffodil-runtime1-layers/src/test/scala/org/apache/daffodil/layers/runtime1/TestGzipLayer.scala b/daffodil-runtime1-layers/src/test/scala/org/apache/daffodil/layers/runtime1/TestGzipLayer.scala
new file mode 100644
index 000000000..95a40fc46
--- /dev/null
+++ b/daffodil-runtime1-layers/src/test/scala/org/apache/daffodil/layers/runtime1/TestGzipLayer.scala
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.layers.runtime1
+
+import java.io.ByteArrayOutputStream
+import java.util.zip.GZIPOutputStream
+
+import org.apache.commons.io.IOUtils
+import org.junit.Assert._
+import org.junit.Test
+
+class TestGzipLayer {
+
+ val gl = new GZipLayer // default constructor has to work.
+
+ @Test def testConstruction() = {
+ assertNotNull(gl)
+ }
+
+ /**
+ * Tests that our gzip fixer fixes up byte 9 of the gzip header
+ * so that it has 255 in it for java versions prior to Java 16
+ */
+ @Test def testByte9() = {
+ val baos = new ByteArrayOutputStream()
+ val baosRaw = new ByteArrayOutputStream()
+ val gzos = gl.wrapLayerOutput(baos)
+ val gzosRaw = new GZIPOutputStream(baosRaw)
+ val data = "testing 1, 2, 3"
+ IOUtils.write(data, gzos, "ascii")
+ IOUtils.write(data, gzosRaw, "ascii")
+ gzos.close()
+ gzosRaw.close()
+ val byte9 = baos.toByteArray.apply(9)
+ val byte9Raw = baosRaw.toByteArray.apply(9)
+ if (GZipLayer.fixIsNeeded()) {
+ assertEquals(0, byte9Raw.toInt & 0xff)
+ assertEquals(255, byte9.toInt & 0xff)
+ } else {
+ assertEquals(255, byte9Raw.toInt & 0xff)
+ assertEquals(255, byte9.toInt & 0xff)
+ }
+ }
+}
diff --git a/daffodil-runtime1-layers/src/test/scala/org/apache/daffodil/layers/runtime1/TestLengthLimitedLineFoldingStreams.scala b/daffodil-runtime1-layers/src/test/scala/org/apache/daffodil/layers/runtime1/TestLengthLimitedLineFoldingStreams.scala
index b7d0bd278..c6451102a 100644
--- a/daffodil-runtime1-layers/src/test/scala/org/apache/daffodil/layers/runtime1/TestLengthLimitedLineFoldingStreams.scala
+++ b/daffodil-runtime1-layers/src/test/scala/org/apache/daffodil/layers/runtime1/TestLengthLimitedLineFoldingStreams.scala
@@ -20,13 +20,17 @@ package org.apache.daffodil.layers.runtime1
import java.io._
import java.nio.charset.StandardCharsets
-import org.apache.daffodil.io.RegexLimitingStream
+import org.apache.daffodil.io.RegexLimitingInputStream
+import org.apache.commons.io.IOUtils
import org.junit.Assert._
import org.junit.Test
class TestLengthLimitedLineFoldingStreams {
+ val crlfRegex = "\\r\\n(?!(?:\\t|\\ ))"
+ val crlf = "\r\n"
+
/**
* Has lines folded using IMF conventions.
*
@@ -59,19 +63,9 @@ esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat\r
//
// regex is CRLF not followed by a tab or space.
//
- val rls = new RegexLimitingStream(bba, "\\r\\n(?!(?:\\t|\\ ))", "\r\n", iso8859)
+ val rls = new RegexLimitingInputStream(bba, crlfRegex, "\r\n", iso8859)
val lfs = new LineFoldedInputStream(LineFoldMode.IMF, rls)
-
- val baos = new ByteArrayOutputStream()
- var c: Int = -1
- while ({
- c = lfs.read()
- c != -1
- }) {
- baos.write(c)
- }
- baos.close()
- val resultString = new String(baos.toByteArray())
+ val resultString = IOUtils.toString(lfs, iso8859)
val expected = ipsumLorem1UnfoldedFirstLine
assertEquals(expected, resultString)
}
@@ -101,19 +95,9 @@ esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat\r
//
// regex is CRLF not followed by a tab or space.
//
- val rls = new RegexLimitingStream(bba, "\\r\\n(?!(?:\\t|\\ ))", "\r\n", iso8859)
+ val rls = new RegexLimitingInputStream(bba, crlfRegex, "\r\n", iso8859)
val lfs = new LineFoldedInputStream(LineFoldMode.iCalendar, rls)
-
- val baos = new ByteArrayOutputStream()
- var c: Int = -1
- while ({
- c = lfs.read()
- c != -1
- }) {
- baos.write(c)
- }
- baos.close()
- val resultString = new String(baos.toByteArray())
+ val resultString = IOUtils.toString(lfs, iso8859)
val expected = ipsumLorem1UnfoldedFirstLine
assertEquals(expected, resultString)
}
@@ -127,19 +111,9 @@ esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat\r
//
// regex is CRLF not followed by a tab or space.
//
- val rls = new RegexLimitingStream(bba, "\\r\\n(?!(?:\\t|\\ ))", "\r\n", iso8859)
+ val rls = new RegexLimitingInputStream(bba, crlfRegex, "\r\n", iso8859)
val lfs = new LineFoldedInputStream(LineFoldMode.iCalendar, rls)
-
- val baos = new ByteArrayOutputStream()
- var c: Int = -1
- while ({
- c = lfs.read()
- c != -1
- }) {
- baos.write(c)
- }
- baos.close()
- val resultString = new String(baos.toByteArray())
+ val resultString = IOUtils.toString(lfs, iso8859)
val expected = ""
assertEquals(expected, resultString)
}
@@ -153,19 +127,9 @@ esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat\r
//
// regex is CRLF not followed by a tab or space.
//
- val rls = new RegexLimitingStream(bba, "\\r\\n(?!(?:\\t|\\ ))", "\r\n", iso8859)
+ val rls = new RegexLimitingInputStream(bba, crlfRegex, "\r\n", iso8859)
val lfs = new LineFoldedInputStream(LineFoldMode.IMF, rls)
-
- val baos = new ByteArrayOutputStream()
- var c: Int = -1
- while ({
- c = lfs.read()
- c != -1
- }) {
- baos.write(c)
- }
- baos.close()
- val resultString = new String(baos.toByteArray())
+ val resultString = IOUtils.toString(lfs, iso8859)
val expected = " "
assertEquals(expected, resultString)
}
@@ -179,19 +143,9 @@ esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat\r
//
// regex is CRLF not followed by a tab or space.
//
- val rls = new RegexLimitingStream(bba, "\\r\\n(?!(?:\\t|\\ ))", "\r\n", iso8859)
+ val rls = new RegexLimitingInputStream(bba, crlfRegex, "\r\n", iso8859)
val lfs = new LineFoldedInputStream(LineFoldMode.IMF, rls)
-
- val baos = new ByteArrayOutputStream()
- var c: Int = -1
- while ({
- c = lfs.read()
- c != -1
- }) {
- baos.write(c)
- }
- baos.close()
- val resultString = new String(baos.toByteArray())
+ val resultString = IOUtils.toString(lfs, iso8859)
val expected = ""
assertEquals(expected, resultString)
}
@@ -205,19 +159,9 @@ esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat\r
//
// regex is CRLF not followed by a tab or space.
//
- val rls = new RegexLimitingStream(bba, "\\r\\n(?!(?:\\t|\\ ))", "\r\n", iso8859)
+ val rls = new RegexLimitingInputStream(bba, crlfRegex, "\r\n", iso8859)
val lfs = new LineFoldedInputStream(LineFoldMode.iCalendar, rls)
-
- val baos = new ByteArrayOutputStream()
- var c: Int = -1
- while ({
- c = lfs.read()
- c != -1
- }) {
- baos.write(c)
- }
- baos.close()
- val resultString = new String(baos.toByteArray())
+ val resultString = IOUtils.toString(lfs, iso8859)
val expected = ""
assertEquals(expected, resultString)
}
diff --git a/daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/unparsers/runtime1/LayeredSequenceUnparser.scala b/daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/unparsers/runtime1/LayeredSequenceUnparser.scala
index d3863b149..b5aaaacd3 100644
--- a/daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/unparsers/runtime1/LayeredSequenceUnparser.scala
+++ b/daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/unparsers/runtime1/LayeredSequenceUnparser.scala
@@ -17,25 +17,18 @@
package org.apache.daffodil.unparsers.runtime1
-import org.apache.daffodil.runtime1.layers.LayerRuntimeInfo
-import org.apache.daffodil.runtime1.layers.LayerTransformerFactory
+import org.apache.daffodil.runtime1.layers.LayerDriver
import org.apache.daffodil.runtime1.processors.SequenceRuntimeData
import org.apache.daffodil.runtime1.processors.unparsers._
class LayeredSequenceUnparser(
ctxt: SequenceRuntimeData,
- layerTransformerFactory: LayerTransformerFactory,
- layerRuntimeInfo: LayerRuntimeInfo,
childUnparser: SequenceChildUnparser,
) extends OrderedUnseparatedSequenceUnparser(ctxt, Seq(childUnparser)) {
- override lazy val runtimeDependencies =
- layerRuntimeInfo.evaluatables.toVector
-
override def nom = "LayeredSequence"
override def unparse(state: UState): Unit = {
- val layerTransformer = layerTransformerFactory.newInstance(layerRuntimeInfo)
val originalDOS = state.dataOutputStream
@@ -52,10 +45,14 @@ class LayeredSequenceUnparser(
// since the flushing of the layer might be delayed due to suspensions. By
// getting an immutable state, we ensure that the flushing of the layer
// occurs with the state at this point.
+ //
+ // TODO: we're not unparsing here, just writing bytes, so perhaps we do not
+ // need this cloned state? Everything in layers is byte-centric, so there is
+ // no issue of fragment bytes.
val formatInfoPre = state.asInstanceOf[UStateMain].cloneForSuspension(layerUnderlyingDOS)
// mark the original DOS as finished--no more data will be unparsed to it.
- // If known, this will carry bit position forward to the layerUnerlyingDOS,
+ // If known, this will carry bit position forward to the layerUnderlyingDOS,
// or could even make that DOS direct. Note that it is important to use the
// cloned state from above. This is because the setFinished function stores
// the formatInfo (as finishedFormatInfo) to be used when delivering
@@ -66,37 +63,46 @@ class LayeredSequenceUnparser(
// create a new DOS where unparsers following this layer will unparse
val layerFollowingDOS = layerUnderlyingDOS.addBuffered()
-
- // New layerDOS is where the layer will unparse into. Ultimately anything written
- // to layerDOS ends up, post transform, in layerUnderlyingDOS
- val layerDOS = layerTransformer.addLayer(layerUnderlyingDOS, state, formatInfoPre)
-
- // unparse the layer body into layerDOS
- state.dataOutputStream = layerDOS
- super.unparse(state)
- layerTransformer.endLayerForUnparse(state)
-
- // now we're done unparsing the layer, so finalize the last DOS in the
- // chain. Note that there might be suspensions so some parts of the
- // layerDOS chain may not be finished. When those suspensions are all
- // finished, the layerDOS content will be written to the
- // layerUnderlyingDOS, which will subsequently be finished. Like above, it
- // is important to pass an immutable UState into the setFinished function
- // because that state is stored in the DOS (as finishedFormatInfo) and its
- // use may be delayed to after the actual UState has been changed. The
- // UState has almost certainly changed since the last cloneForSuspension
- // call, so we need a new clone to finish this DOS.
- val layerDOSLast = layerDOS.lastInChain
- val formatInfoPost = state.asInstanceOf[UStateMain].cloneForSuspension(layerDOSLast)
- layerDOSLast.setFinished(formatInfoPost)
-
- // clean up resources - note however, that due to suspensions, the whole
- // layer stack is potentially still needed, so not clear what can be
- // cleaned up at this point.
- layerTransformer.removeLayer(layerDOS, state)
-
- // reset the state so subsequent unparsers write to the following DOS
- state.dataOutputStream = layerFollowingDOS
+ var layerDriver: LayerDriver = null
+ try {
+ layerDriver = LayerDriver(formatInfoPre, ctxt.layerRuntimeData)
+ layerDriver.init() // could fail if layer setter code causes errors
+
+ // New layerDOS is where the layer will unparse into. Ultimately anything written
+ // to layerDOS ends up, post transform, in layerUnderlyingDOS
+ val layerDOS = layerDriver.addOutputLayer(layerUnderlyingDOS)
+
+ // unparse the layer body into layerDOS
+ state.dataOutputStream = layerDOS
+ super.unparse(state)
+ // now we're done unparsing the layer recursively.
+ // While doing that unparsing, the data output stream may have been split, so the
+ // DOS in the state may no longer be the layerDOS.
+ //
+ // However, it is when whatever DOS is in the state at this point, that, when that
+ // DOS is consolidated and written out, that is when the layer is finished
+ // and the wrap-up of the layer (such as writing output variables) can occur.
+ //
+ val endOfLayerUnparseDOS = state.dataOutputStream
+ val formatInfoPost =
+ state.asInstanceOf[UStateMain].cloneForSuspension(endOfLayerUnparseDOS)
+
+ // setFinished on this end-of-layer-unparse data-output-stream ensures
+ // that the layerDOS gets close() called on it.
+ endOfLayerUnparseDOS.setFinished(formatInfoPost)
+
+ // clean up resources - note however, that due to suspensions, the whole
+ // layer stack is potentially still needed, so
+ // nothing can be cleaned up at this point.
+ } catch {
+ case t: Throwable if (layerDriver ne null) => layerDriver.handleThrowable(t)
+ case t: Throwable => LayerDriver.handleThrowableWithoutLayer(t)
+ // otherwise we have no layer driver, so we were unable to load the layer.
+ // just let that propagate.
+ } finally {
+ // reset the state so subsequent unparsers write to the following DOS
+ state.dataOutputStream = layerFollowingDOS
+ }
}
}
diff --git a/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/dpath/DPathRuntime.scala b/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/dpath/DPathRuntime.scala
index de4443e5c..5ffde2121 100644
--- a/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/dpath/DPathRuntime.scala
+++ b/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/dpath/DPathRuntime.scala
@@ -193,7 +193,8 @@ abstract class RecipeOpWithSubRecipes(recipes: List[CompiledDPath]) extends Reci
case class VRef(vrd: VariableRuntimeData, context: ThrowsSDE) extends RecipeOp {
override def run(dstate: DState): Unit = {
- if (dstate.parseOrUnparseState.isEmpty) throw new VariableHasNoValue(vrd.globalQName, vrd)
+ if (dstate.parseOrUnparseState.isEmpty)
+ throw new VariableHasNoValue(vrd.globalQName, vrd)
val value = dstate.parseOrUnparseState.get.getVariable(vrd, context)
dstate.setCurrentValue(value)
}
diff --git a/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/dpath/NodeInfo.scala b/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/dpath/NodeInfo.scala
index 478daa63a..097fd8a10 100644
--- a/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/dpath/NodeInfo.scala
+++ b/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/dpath/NodeInfo.scala
@@ -18,12 +18,14 @@
package org.apache.daffodil.runtime1.dpath
import java.io.Serializable
+import java.lang.{ Boolean => JBoolean }
import java.lang.{ Byte => JByte }
import java.lang.{ Double => JDouble }
import java.lang.{ Float => JFloat }
import java.lang.{ Integer => JInt }
import java.lang.{ Long => JLong }
import java.lang.{ Short => JShort }
+import java.lang.{ String => JString }
import java.math.{ BigDecimal => JBigDecimal }
import java.math.{ BigInteger => JBigInt }
import java.net.URI
@@ -62,6 +64,8 @@ import org.apache.daffodil.runtime1.infoset.DataValue.DataValueShort
import org.apache.daffodil.runtime1.infoset.DataValue.DataValueTime
import org.apache.daffodil.runtime1.infoset.DataValue.DataValueURI
+import com.ibm.icu.util.{ Calendar => ICUCalendar }
+
/**
* We need to have a data structure that lets us represent a type, and
* its relationship (conversion, subtyping) to other types.
@@ -932,6 +936,33 @@ object NodeInfo extends Enum {
DFDLTimeConversion.fromXMLString(s)
}
}
+
+ def toJavaType(dfdlType: DFDLPrimType): Class[_] = {
+ dfdlType match {
+ case DFDLPrimType.String => classOf[JString]
+ case DFDLPrimType.Int => classOf[JInt]
+ case DFDLPrimType.Byte => classOf[JByte]
+ case DFDLPrimType.Short => classOf[JShort]
+ case DFDLPrimType.Long => classOf[JLong]
+ case DFDLPrimType.Integer => classOf[JBigInt]
+ case DFDLPrimType.Decimal => classOf[JBigDecimal]
+ case DFDLPrimType.UnsignedInt => classOf[JLong]
+ case DFDLPrimType.UnsignedByte => classOf[JShort]
+ case DFDLPrimType.UnsignedShort => classOf[JInt]
+ case DFDLPrimType.UnsignedLong => classOf[JBigInt]
+ case DFDLPrimType.NonNegativeInteger => classOf[JBigInt]
+ case DFDLPrimType.Double => classOf[JDouble]
+ case DFDLPrimType.Float => classOf[JFloat]
+ case DFDLPrimType.HexBinary => classOf[Array[Byte]]
+ case DFDLPrimType.AnyURI => classOf[java.net.URI]
+ case DFDLPrimType.Boolean => classOf[JBoolean]
+ case DFDLPrimType.DateTime => classOf[ICUCalendar]
+ case DFDLPrimType.Date => classOf[ICUCalendar]
+ case DFDLPrimType.Time => classOf[ICUCalendar]
+ }
+ }
+
+ def toJavaTypeString(dfdlType: DFDLPrimType): String = toJavaType(dfdlType).toString
}
//
diff --git a/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/dsom/SDE.scala b/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/dsom/SDE.scala
index a0898dca8..2e47840ab 100644
--- a/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/dsom/SDE.scala
+++ b/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/dsom/SDE.scala
@@ -59,8 +59,8 @@ class RelativePathPastRootError(schemaContext: SchemaFileLocation, kind: String,
class RuntimeSchemaDefinitionError(
schemaContext: SchemaFileLocation,
runtimeContext: ParseOrUnparseState,
- causedBy: Maybe[Throwable],
- fmtString: Maybe[String],
+ causedBy: Throwable,
+ fmtString: String,
args: Any*,
) extends SchemaDefinitionDiagnosticBase(
Maybe(schemaContext),
@@ -69,8 +69,8 @@ class RuntimeSchemaDefinitionError(
case _ => Maybe(runtimeContext)
}),
None,
- causedBy,
- fmtString,
+ Maybe(causedBy),
+ Maybe(fmtString),
args: _*,
) {
@@ -83,7 +83,7 @@ class RuntimeSchemaDefinitionError(
fmtString: String,
args: Any*,
) =
- this(schemaContext, runtimeContext, Nope, Maybe(fmtString), args: _*)
+ this(schemaContext, runtimeContext, null, fmtString, args: _*)
}
class RuntimeSchemaDefinitionWarning(
diff --git a/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/infoset/InfosetWalker.scala b/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/infoset/InfosetWalker.scala
index c091327b7..50b7d3125 100644
--- a/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/infoset/InfosetWalker.scala
+++ b/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/infoset/InfosetWalker.scala
@@ -453,6 +453,10 @@ class InfosetWalker private (
outputterFunc
} catch {
case e: Exception => {
+ // FIXME: DAFFODIL-2884 This escalates a parser data exception to an SDE
+ // Which breaks if string-as-xml encounters a string that is malformed XML.
+ // We get the error thrown by the xml parser here outside of parsing, which is
+ // too late.
val cause = e.getCause
val msg = if (cause == null) e.toString else cause.toString
context.SDE("Failed to %s: %s", desc, msg)
diff --git a/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/ChecksumLayerBase.scala b/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/ChecksumLayerBase.scala
new file mode 100644
index 000000000..820a9ba0c
--- /dev/null
+++ b/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/ChecksumLayerBase.scala
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.daffodil.runtime1.layers
+
+import java.io.ByteArrayInputStream
+import java.io.ByteArrayOutputStream
+import java.io.InputStream
+import java.io.OutputStream
+import java.nio.ByteBuffer
+
+import org.apache.daffodil.lib.exceptions.Assert
+import org.apache.daffodil.runtime1.layers.api.Layer
+
+import org.apache.commons.io.IOUtils
+
+/**
+ * A base class for ChecksumLayer - hides details from the API user who is deriving a
+ * concrete class from ChecksumLayer.
+ *
+ * Suitable only for checksums computed over small sections of data, not large data streams or whole files.
+ */
+abstract class ChecksumLayerBase(
+ localName: String,
+ targetNamespace: String,
+) extends Layer(localName, targetNamespace) {
+
+ private var checksum: Int = -1
+
+ private var length: Int = -1
+ private var isInitialized: Boolean = false
+
+ final def getLength: Int = length
+
+ /**
+ * Must be called to specify the length, or it is a fatal API usage error.
+ * @param len The fixed length over which the checksum is computed.
+ */
+ final protected def setLength(len: Int): Unit = {
+ this.length = len
+ if (len < 0) processingError(s"The layer length is negative: $len.")
+ isInitialized = true
+ }
+
+ final protected def getChecksum: Int = checksum
+ final private[layers] def setChecksum(checksum: Int): Unit = { this.checksum = checksum }
+
+ def compute(
+ isUnparse: Boolean,
+ byteBuffer: ByteBuffer,
+ ): Int
+
+ private def checkInitialized() = {
+ if (!isInitialized) {
+ // this is a RuntimeException, so will be fatal whether parsing or unparsing.
+ throw new IllegalStateException(
+ "ChecksumLayer API usage: setLength method was never called.",
+ )
+ }
+ }
+
+ final override def wrapLayerInput(jis: InputStream): InputStream = {
+ checkInitialized()
+ new ChecksumDecoderInputStream(this, jis)
+ }
+
+ final override def wrapLayerOutput(jos: OutputStream): OutputStream = {
+ checkInitialized()
+ new ChecksumEncoderOutputStream(this, jos)
+ }
+}
+
+class ChecksumDecoderInputStream(
+ layer: ChecksumLayerBase,
+ jis: InputStream,
+) extends InputStream {
+
+ private def doubleCheckLength(actualDataLen: Int): Unit = {
+ val neededLen = layer.getLength
+ if (neededLen > actualDataLen)
+ layer.processingError(new LayerNotEnoughDataException(neededLen, actualDataLen))
+ }
+
+ private lazy val bais = {
+ val ba = new Array[Byte](layer.getLength)
+ val nRead = IOUtils.read(jis, ba)
+ doubleCheckLength(nRead)
+ val buf = ByteBuffer.wrap(ba)
+ layer.setChecksum(layer.compute(isUnparse = false, buf))
+ new ByteArrayInputStream(ba)
+ }
+
+ override def read(): Int = bais.read()
+}
+
+class ChecksumEncoderOutputStream(
+ layer: ChecksumLayerBase,
+ jos: OutputStream,
+) extends OutputStream {
+
+ private lazy val baos = new ByteArrayOutputStream(layer.getLength)
+
+ private var count: Long = 0
+
+ override def write(b: Int): Unit = {
+ baos.write(b)
+ count += 1
+ if (count > layer.getLength) {
+ //
+ // This could happen if the layer logically unparses as one of two choice branches where they
+ // are supposed to be all the same length, but one is in fact longer than expected by the bufLen.
+ layer.processingError(
+ new IndexOutOfBoundsException(
+ s"Written data amount exceeded fixed layer length of ${layer.getLength}.",
+ ),
+ )
+ }
+ }
+
+ private var isOpen: Boolean = true
+
+ override def close(): Unit = {
+ if (isOpen) { // allow multiple closes
+ isOpen = false
+ val ba = baos.toByteArray
+ val baLen = ba.length
+ Assert.invariant(baLen <= layer.getLength)
+ val buf = ByteBuffer.wrap(ba)
+ layer.setChecksum(layer.compute(isUnparse = true, buf))
+ jos.write(ba)
+ jos.flush()
+ }
+ }
+}
diff --git a/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/LayerCompiler.scala b/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/LayerCompiler.scala
deleted file mode 100644
index 00fd36d8b..000000000
--- a/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/LayerCompiler.scala
+++ /dev/null
@@ -1,135 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.daffodil.runtime1.layers
-
-import org.apache.daffodil.io.processors.charset.BitsCharsetJava
-import org.apache.daffodil.io.processors.charset.BitsCharsetNonByteSize
-import org.apache.daffodil.lib.api.WarnID
-import org.apache.daffodil.lib.schema.annotation.props.gen.LayerLengthKind
-import org.apache.daffodil.lib.schema.annotation.props.gen.LayerLengthUnits
-import org.apache.daffodil.lib.xml.NS
-import org.apache.daffodil.lib.xml.RefQName
-import org.apache.daffodil.runtime1.dpath.NodeInfo.PrimType
-import org.apache.daffodil.runtime1.dsom.ImplementsThrowsOrSavesSDE
-import org.apache.daffodil.runtime1.processors.SequenceRuntimeData
-import org.apache.daffodil.runtime1.processors.VariableRuntimeData
-
-/**
- * Must be implemented by all layers.
- *
- * These are the classes which must be dynamically loaded in order to add a layer implementation
- * to Daffodil.
- *
- * These instances are NOT serialized as part of a saved processor. The compileLayer method
- * is called and the resulting LayerTransformerFactory is the serialized object.
- */
-abstract class LayerCompiler(nom: String) {
-
- def name() = nom
-
- /**
- * Performs all schema-compile-time checking for the layer parameters, and constructs
- * a LayerTransformerFactory which is the serialized runtime object that becomes part of the
- * processor.
- * @param layerCompileInfo Schema-compilation time information about the layer properties.
- * @return
- */
- def compileLayer(layerCompileInfo: LayerCompileInfo): LayerTransformerFactory
-
-}
-
-/**
- * Provides access to DFDL schema compile-time information about the layer properties.
- *
- * Allows reporting of schema definition errors and warnings at schema compile time.
- */
-final class LayerCompileInfo(
- sequence: ImplementsThrowsOrSavesSDE,
- val layerRuntimeInfo: LayerRuntimeInfo,
-) {
-
- private def lri = layerRuntimeInfo
- private def srd: SequenceRuntimeData = lri.runtimeData
-
- def getVariableRuntimeData(
- prefix: String,
- namespace: String,
- localName: String,
- primType: PrimType,
- ): VariableRuntimeData = {
- val varNamespace = NS(namespace)
- val qName = RefQName(Some(prefix), localName, varNamespace).toGlobalQName
- val vrd = srd.variableMap.getVariableRuntimeData(qName).getOrElse {
- srd.SDE("Variable '%s' is not defined.", qName.toExtendedSyntax)
- }
- srd.schemaDefinitionUnless(
- vrd.primType == primType,
- "Variable '%s' is not of type '%s'.",
- qName.toExtendedSyntax,
- primType,
- )
- vrd
- }
-
- /**
- * If defined, and the value is a compile-time constant then this will be Some(Some(Charset))
- * If defined, and the value is non-constant or not a regular JVM charset, then this will be Some(None)
- * If undefined, the value is None
- */
- def optLayerJavaCharsetOptConstantValue: Option[Option[java.nio.charset.Charset]] = {
- if (lri.maybeLayerCharsetEv.isEmpty) None
- else
- lri.maybeLayerCharsetEv.get.optConstant.map {
- case java: BitsCharsetJava => Some(java.javaCharset)
- case _: BitsCharsetNonByteSize => None
- }
- }
-
- def optLayerLengthKind: Option[LayerLengthKind] = {
- lri.maybeLayerLengthKind.toScalaOption
- }
-
- def optLayerLengthOptConstantValue: Option[Option[Long]] = {
- if (lri.maybeLayerLengthEv.isEmpty) None
- else
- Some(lri.maybeLayerLengthEv.get.optConstant.map {
- _.toLong
- })
- }
-
- def optLayerLengthUnits: Option[LayerLengthUnits] = {
- lri.maybeLayerLengthUnits.toScalaOption
- }
-
- def optLayerBoundaryMarkOptConstantValue: Option[Option[String]] = {
- if (lri.maybeLayerBoundaryMarkEv.isEmpty) None
- else Some(lri.maybeLayerBoundaryMarkEv.get.optConstant)
- }
-
- def schemaDefinitionError(message: String, args: Any*): Nothing = {
- sequence.schemaDefinitionError(message, args: _*)
- }
-
- def schemaDefinitionWarning(message: String, args: Any*): Unit = {
- sequence.SDW(WarnID.LayerCompileWarning, message, args: _*)
- }
-
- def SDEUnless(test: Boolean, message: String, args: Any*): Unit =
- if (!test) SDE(message, args: _*)
-
- def SDE(message: String, args: Any*): Nothing = schemaDefinitionError(message, args: _*)
-}
diff --git a/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/LayerCompilerRegistry.scala b/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/LayerCompilerRegistry.scala
deleted file mode 100644
index a7e64bf72..000000000
--- a/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/LayerCompilerRegistry.scala
+++ /dev/null
@@ -1,49 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.daffodil.runtime1.layers
-
-import org.apache.daffodil.lib.exceptions.ThrowsSDE
-import org.apache.daffodil.lib.util.SimpleNamedServiceLoader
-
-/**
- * Transformers have factories. This lets you find the transformer factory
- * by the name obtained from dfdlx:layerTransform.
- */
-object LayerCompilerRegistry {
-
- private lazy val layerCompilerMap: Map[String, LayerCompiler] =
- SimpleNamedServiceLoader.loadClass[LayerCompiler](classOf[LayerCompiler])
-
- /**
- * Given name, finds the factory for the transformer. SDE otherwise.
- *
- * The state is passed in order to provide diagnostic context if not found.
- */
- def find(name: String, context: ThrowsSDE): LayerCompiler = {
- val maybeCompiler: Option[LayerCompiler] = layerCompilerMap.get(name)
- if (maybeCompiler.isEmpty) {
- val choices = layerCompilerMap.keySet.mkString(", ")
- context.SDE(
- "The dfdlx:layerTransform '%s' was not found. Available choices are: %s",
- name,
- choices,
- )
- } else {
- maybeCompiler.get
- }
- }
-}
diff --git a/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/LayerDriver.scala b/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/LayerDriver.scala
new file mode 100644
index 000000000..e23d6acc0
--- /dev/null
+++ b/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/LayerDriver.scala
@@ -0,0 +1,420 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime1.layers
+
+import java.io.FilterInputStream
+import java.io.InputStream
+import java.io.OutputStream
+import scala.collection.convert.ImplicitConversions.`collection AsScalaIterable`
+
+import org.apache.daffodil.io.DataInputStream.Mark
+import org.apache.daffodil.io.DataOutputStream
+import org.apache.daffodil.io.DirectOrBufferedDataOutputStream
+import org.apache.daffodil.io.InputSourceDataInputStream
+import org.apache.daffodil.lib.exceptions.Assert
+import org.apache.daffodil.lib.exceptions.UnsuppressableException
+import org.apache.daffodil.lib.schema.annotation.props.gen.BitOrder
+import org.apache.daffodil.lib.util.Maybe
+import org.apache.daffodil.lib.util.Maybe.Nope
+import org.apache.daffodil.lib.util.Maybe.One
+import org.apache.daffodil.lib.util.Misc
+import org.apache.daffodil.runtime1.dsom.RuntimeSchemaDefinitionError
+import org.apache.daffodil.runtime1.layers.api.Layer
+import org.apache.daffodil.runtime1.processors.ParseOrUnparseState
+import org.apache.daffodil.runtime1.processors.ProcessingError
+
+import passera.unsigned.ULong
+
+/**
+ * Mandatory factory to create and initialize the layer driver object.
+ */
+object LayerDriver {
+ def apply(state: ParseOrUnparseState, lrd: LayerRuntimeData): LayerDriver = {
+ val lvr =
+ state.dataProc.get.ssrd.getLayerVarsRuntime(lrd) // could fail if layer not SPI loaded
+ val lr = new LayerRuntime(state, lrd, lvr) // can't fail
+ val newLayer = lvr.constructInstance() // can't fail
+ newLayer.setLayerRuntime(lr) // can't fail
+ val instance = new LayerDriver(newLayer) // can't fail
+ instance
+ }
+
+ def handleThrowableWithoutLayer(t: Throwable) = {
+ t match {
+ case u: UnsuppressableException => throw t
+ case pe: ProcessingError => throw pe // already a PE. Do not further mess with it.
+ case rsde: RuntimeSchemaDefinitionError =>
+ throw rsde
+ case re: RuntimeException =>
+ throw new LayerFatalException(re)
+ case e: Exception =>
+ throw new LayerFatalException(e)
+ }
+ }
+}
+
+/**
+ * Driving mechanism that incorporates a layer at runtime into the I/O streams
+ * to transform the data stream and optionally receive parameter values from,
+ * and populate result values back into DFDL variables.
+ *
+ * A layer driver is created at runtime as part of a single parse/unparse call.
+ * Hence, they can be stateful without causing thread-safety issues.
+ *
+ * @param layer The Layer instance to parameterize, drive, and get results back from.
+ */
+class LayerDriver private (val layer: Layer) {
+
+ def init(): Unit = {
+ try {
+ // timing here is subtle.
+ // This could throw, but the steps after this are necessary
+ // in order to properly handle the thrown exception
+ //
+ layer.getLayerRuntime.layerVarsRuntime.callParamSetterWithParameterVars(this.layer)
+ } finally {
+ //
+ // because we want to deal with RuntimeException as if it was NOT a subclass of
+ // Exception, we first divide up the classes as to whether they are RuntimeExceptions or not.
+ val (runtimeExceptionSubClasses, regularExceptionSubclasses) =
+ layer.getProcessingErrorExceptions.partition { c =>
+ classOf[RuntimeException].isAssignableFrom(c)
+ }
+
+ setLayerThrowHandler(new LayerThrowHandler() {
+ override def handleThrow(t: Throwable): Unit = {
+ val tc = t.getClass
+ val isRuntimeException = classOf[RuntimeException].isAssignableFrom(tc)
+ if (isRuntimeException) {
+ if (runtimeExceptionSubClasses.exists { ec => ec.isAssignableFrom(tc) })
+ layer.processingError(t)
+ } else {
+ if (regularExceptionSubclasses.exists { ec => ec.isAssignableFrom(tc) })
+ layer.processingError(t)
+ }
+ }
+ })
+ }
+ }
+
+ def layerRuntime: LayerRuntime = layer.getLayerRuntime
+
+ private def layerVarsRuntime = layerRuntime.layerVarsRuntime
+
+ private def wrapJavaInputStream(s: InputSourceDataInputStream) =
+ new JavaIOInputStream(s, layer)
+
+ private def wrapJavaOutputStream(s: DataOutputStream) =
+ new JavaIOOutputStream(s, layer)
+
+ /**
+ * Override these if we ever have transformers that don't have these
+ * requirements.
+ */
+ val mandatoryLayerAlignmentInBits: Int = 8
+
+ final def addInputLayer(s: InputSourceDataInputStream): InputSourceDataInputStream = {
+ val jis = wrapJavaInputStream(s)
+ // This wrapLayerInput could fail via a throw. It's calling layer code.
+ val wrapped = layer.wrapLayerInput(jis)
+ // This assured close thing is just in case the user's layer stream doesn't propagate
+ // the close. That would be incorrect of it, but we want to tolerate that not happening.
+ val decodedInputStream = new ThrowProtectedAssuredCloseInputStream(wrapped, jis)
+ val newDIS = InputSourceDataInputStream(decodedInputStream)
+ // must initialize priorBitOrder
+ newDIS.cst.setPriorBitOrder(BitOrder.MostSignificantBitFirst)
+ newDIS.setDebugging(s.areDebugging)
+ newDIS
+ }
+
+ /**
+ * Parsing works as a tree traversal, so when the parser unwinds from
+ * parsing the layer we can invoke this to handle cleanups, and
+ * finalization issues like assigning the result variables
+ */
+ final def removeInputLayer(s: InputSourceDataInputStream): Unit =
+ try {
+ layerVarsRuntime.callGettersToPopulateResultVars(layer) // could throw.
+ } finally {
+ s.close()
+ }
+
+ /**
+ * Note that there is addOutputLayer, but there is no remove output layer because
+ * unparsing is very asynchronous due to suspensions. The unwinding of this
+ * layering for unparsing can only happen in the close() of the layer stream.
+ */
+ final def addOutputLayer(s: DataOutputStream): DirectOrBufferedDataOutputStream = {
+ val jos = wrapJavaOutputStream(s)
+ val wrappedStream = layer.wrapLayerOutput(jos)
+ val encodedOutputStream = new ThrowProtectedAssuredCloseOutputStream(wrappedStream, jos)
+ val newDOS = DirectOrBufferedDataOutputStream(
+ encodedOutputStream,
+ null,
+ isLayer = true,
+ s.chunkSizeInBytes,
+ s.maxBufferSizeInBytes,
+ s.tempDirPath,
+ )
+ newDOS.setPriorBitOrder(BitOrder.MostSignificantBitFirst)
+ newDOS.setAbsStartingBitPos0b(ULong(0L))
+ newDOS.setDebugging(s.areDebugging)
+ newDOS
+ }
+
+ def handleThrowable(t: Throwable) = {
+ t match {
+ case u: UnsuppressableException => throw t
+ case pe: ProcessingError => throw pe // already a PE. Do not further mess with it.
+ case _ =>
+ // if the layer supplies a handler, then
+ // we invoke it.
+ val layerHandler = getLayerThrowHandler
+ if (layerHandler ne null)
+ layerHandler.handleThrow(t)
+ // if we get here, the layer handler didn't exist or
+ // it didn't handle the throwable.
+ LayerDriver.handleThrowableWithoutLayer(t)
+ }
+ }
+
+ private final class ThrowProtectedAssuredCloseInputStream(
+ stream: InputStream,
+ inner: InputStream,
+ ) extends FilterInputStream(stream) {
+
+ override def read(): Int = {
+ try {
+ super.read()
+ } catch {
+ case t: Throwable => handleThrowable(t)
+ }
+ }
+
+ override def read(b: Array[Byte], off: Int, len: Int): Int = {
+ try {
+ super.read(b, off, len)
+ } catch {
+ case t: Throwable => handleThrowable(t)
+ }
+ }
+
+ override def close(): Unit = {
+ try {
+ stream.close()
+ inner.close()
+ } catch {
+ case t: Throwable => handleThrowable(t)
+ }
+ }
+ }
+
+ /**
+ * Besides making sure the inner stream gets closed, this also
+ * provides handling of exceptions thrown by the layer-author's code
+ * (or things that that code calls).
+ *
+ * This is needed because unparsing can involve suspensions, so these
+ * calls can happen long after the call stack of unparsers is over; hence,
+ * any nested try/catches around the actual unparser calls may not be
+ * surrounding these actions.
+ *
+ * To get sensible behavior even if these throws happen late due to
+ * suspensions, we have to encapsulate here with our own throw/catches.
+ *
+ * @param stream the stream we're encapsulating
+ * @param inner the stream we're assuring will be closed even if a close on the primary stream doesn't propagate the stream
+ */
+ private final class ThrowProtectedAssuredCloseOutputStream(
+ stream: OutputStream,
+ inner: OutputStream,
+ ) extends OutputStream {
+
+ /**
+ * We hope this isn't get used because a try-catch around every individual write of a byte
+ * is inefficient certainly.
+ *
+ * @param b
+ */
+ override def write(b: Int): Unit = {
+ try {
+ stream.write(b)
+ } catch {
+ case t: Throwable =>
+ handleThrowable(t)
+ }
+ }
+
+ /**
+ * We hope this is the way most writes are done and with fairly large buffers to amortize the
+ * cost of the try/catch logic over many bytes of output.
+ *
+ * @param b
+ * @param off
+ * @param len
+ */
+ override def write(b: Array[Byte], off: Int, len: Int): Unit = {
+ try {
+ stream.write(b, off, len)
+ } catch {
+ case t: Throwable =>
+ handleThrowable(t)
+ }
+ }
+
+ override def close(): Unit = {
+ try {
+ stream.close()
+ inner.close()
+ } catch {
+ case t: Throwable =>
+ handleThrowable(t)
+ }
+ }
+ }
+
+ private var lth: LayerThrowHandler = null
+
+ trait LayerThrowHandler {
+ def handleThrow(t: Throwable): Unit
+ }
+
+ /**
+ * If you set a layer throw handler, then if the layer code throws
+ * anything, the handler is invoked and can convert specific kinds of
+ * exceptions into processing errors or runtime SDEs.
+ *
+ * If the handler does not throw or call processingError or
+ * runtimeSchemaDefinitionError, then the throw is treated as
+ * if there was no handler, and propagates generally as a fatal error.
+ *
+ * @param lth the throw handler
+ */
+ def setLayerThrowHandler(lth: LayerThrowHandler): Unit = {
+ this.lth = lth
+ }
+
+ def getLayerThrowHandler: LayerThrowHandler = {
+ return this.lth
+ }
+
+}
+
+/**
+ * This is used to encapsulate runtime exceptions that are thrown out of layer code.
+ */
+class LayerFatalException(msg: String, cause: Throwable) extends RuntimeException(msg, cause) {
+ def this(msg: String) = this(msg, null)
+
+ /**
+ * When constructed with just a cause, we also pass a message if the cause provides one
+ * because the SLF4J logging system doesn't synthesize a message from the cause if only
+ * a cause is provided.
+ * @param cause
+ */
+ def this(cause: Throwable) =
+ this(
+ {
+ if (cause.getMessage == null)
+ Misc.getNameFromClass(cause)
+ else
+ cause.getMessage
+ },
+ cause,
+ )
+}
+
+/**
+ * Turns Daffodil's bits-capable InputSourceDataInputStream objects into ordinary
+ * java InputStream API for byte-oriented reads.
+ *
+ * @param s the Daffodil InputSourceDataInputStream
+ * @return a java.io.InputStream which when read from, delegates to the InputSourceDataInputStream
+ */
+class JavaIOInputStream(s: InputSourceDataInputStream, layer: Layer) extends InputStream {
+ private def finfo = layer.getLayerRuntime.finfo
+ private lazy val id = Misc.getNameFromClass(this)
+
+ override def read(): Int = {
+ if (!s.isDefinedForLength(8)) -1
+ else {
+ val ul = s.getUnsignedLong(8, finfo)
+ val byte: Int = ul.toInt & 0xff
+ byte
+ }
+ }
+
+ override def available(): Int = 0
+
+ override def close(): Unit = {
+ // do nothing. Important. We do not want the close to cascade beyond here.
+ }
+
+ /**
+ * @param readlimit Ignored. The limits of daffodil's input system are specified
+ * elsewhere. See BucketingInputSource in the daffodil-io module.
+ */
+ override def mark(readlimit: Int): Unit = {
+ maybeSavedMark = One(s.mark(id))
+ }
+
+ private var maybeSavedMark: Maybe[Mark] = Nope
+
+ override def reset(): Unit = {
+ Assert.usage(maybeSavedMark.isDefined)
+ s.reset(maybeSavedMark.get)
+ maybeSavedMark = Nope
+ }
+
+ override def markSupported() = true
+}
+
+/**
+ * Turns a Daffodil bits-capable DataOutputStream into an ordinary java.io.OutputStream.
+ *
+ * @param dos The DataOutputStream to write the data to.
+ */
+class JavaIOOutputStream(dos: DataOutputStream, layer: Layer) extends OutputStream {
+
+ private def lr = layer.getLayerRuntime
+ private def lvr = lr.layerVarsRuntime
+ private def finfo = lr.finfo
+
+ private var closed = false
+
+ private var nBytes = 0
+
+ override def write(b: Int): Unit = {
+ val wasWritten = dos.putLong(b, 8, finfo)
+ if (wasWritten) nBytes += 1
+ }
+
+ override def close(): Unit = {
+ if (!closed) {
+ dos.setFinished(finfo)
+ //
+ // This is where we can reliably wrap-up the layer processing.
+ //
+ try {
+ lvr.callGettersToPopulateResultVars(layer)
+ } finally {
+ closed = true
+ }
+ }
+ }
+}
diff --git a/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/LayerNotEnoughDataException.java b/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/LayerNotEnoughDataException.java
new file mode 100644
index 000000000..de55ef799
--- /dev/null
+++ b/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/LayerNotEnoughDataException.java
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.daffodil.runtime1.layers;
+
+/**
+ * LayerNotEnoughDataException is a custom exception class that represents an exception that occurs
+ * when there is insufficient data for a layer in a program.
+ */
+public class LayerNotEnoughDataException extends LayerProcessingException {
+
+ /**
+ * Creates a new instance of LayerNotEnoughDataException with the specified
+ * number of needed bytes and available bytes.
+ *
+ * @param numNeededBytes the number of bytes needed by the layer
+ * @param numAvailableBytes the number of bytes available for the layer
+ */
+ public LayerNotEnoughDataException(int numNeededBytes, int numAvailableBytes) {
+ super("Insufficient data. Needed " + numNeededBytes +
+ " bytes, but only " + numAvailableBytes + " were available.");
+ }
+}
diff --git a/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/LayerProcessingException.java b/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/LayerProcessingException.java
new file mode 100644
index 000000000..c3b72ab90
--- /dev/null
+++ b/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/LayerProcessingException.java
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.daffodil.runtime1.layers;
+
+import org.apache.daffodil.lib.util.Misc;
+
+import java.util.Objects;
+
+/**
+ * LayerProcessingException represents an exception that can occur during the usage of layers in a program.
+ */
+public class LayerProcessingException extends Exception {
+
+ public LayerProcessingException(String msg, Throwable cause) {
+ super(msg, cause);
+ if (Objects.isNull(msg)) Objects.requireNonNull(cause);
+ if (Objects.isNull(cause)) Objects.requireNonNull(msg);
+ }
+
+ public LayerProcessingException(String msg) { this(msg, null); }
+
+ /**
+ * When creating an exception with just a cause, we also synthesize some sort of
+ * message, because otherwise SLF4J doesn't do so, and you get out errors like
+ * '[error] null' because you didn't supply a message.
+ * @param cause the cause of this processing exception
+ */
+ public LayerProcessingException(Throwable cause) {
+ this((cause.getMessage()) == null ? Misc.getNameFromClass(cause) : cause.getMessage(), cause); }
+}
diff --git a/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/LayerTransformerFactory.scala b/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/LayerRuntime.scala
similarity index 51%
copy from daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/LayerTransformerFactory.scala
copy to daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/LayerRuntime.scala
index 2364fa1ba..817dbd2a0 100644
--- a/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/LayerTransformerFactory.scala
+++ b/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/LayerRuntime.scala
@@ -14,18 +14,32 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-
package org.apache.daffodil.runtime1.layers
+import org.apache.daffodil.io.FormatInfo
+import org.apache.daffodil.runtime1.processors.ParseOrUnparseState
+
/**
- * Factory for a layer transformer.
- *
- * This is the serialized object which is saved as part of a processor.
- * It constructs the layer transformer at runtime when newInstance() is called.
- *
- * This must be implmented as part of implementation of a daffodil layer.
+ * Glues together the state and the static layer runtime data (created by the schema compiler)
+ * and the vars runtime (which is created at the start of runtime)
*/
-abstract class LayerTransformerFactory(val name: String) extends Serializable {
+class LayerRuntime(
+ val state: ParseOrUnparseState,
+ val layerRuntimeData: LayerRuntimeData,
+ val layerVarsRuntime: LayerVarsRuntime,
+) {
+
+ final def finfo: FormatInfo = state
+
+ def processingError(cause: Throwable): Nothing =
+ state.toss(state.toProcessingError(cause))
+
+ def processingError(msg: String): Nothing =
+ state.toss(state.toProcessingError(msg))
+
+ def runtimeSchemaDefinitionError(msg: String): Nothing =
+ state.SDE(msg)
- def newInstance(layerRuntimeInfo: LayerRuntimeInfo): LayerTransformer
+ def runtimeSchemaDefinitionError(cause: Throwable): Nothing =
+ state.SDE(cause)
}
diff --git a/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/LayerRuntimeCompiler.scala b/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/LayerRuntimeCompiler.scala
new file mode 100644
index 000000000..6609858df
--- /dev/null
+++ b/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/LayerRuntimeCompiler.scala
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.daffodil.runtime1.layers
+
+import java.lang.reflect.Method
+import java.lang.{ Boolean => JBoolean }
+import java.lang.{ Byte => JByte }
+import java.lang.{ Double => JDouble }
+import java.lang.{ Float => JFloat }
+import java.lang.{ Integer => JInt }
+import java.lang.{ Long => JLong }
+import java.lang.{ Short => JShort }
+import scala.collection.immutable.ListSet
+import scala.collection.mutable
+
+import org.apache.daffodil.lib.exceptions.Assert
+import org.apache.daffodil.lib.util.SimpleNamedServiceLoader
+import org.apache.daffodil.runtime1.dpath.NodeInfo.PrimType
+import org.apache.daffodil.runtime1.layers.api.Layer
+import org.apache.daffodil.runtime1.processors.VariableRuntimeData
+
+class LayerRuntimeCompiler {
+
+ /**
+ * This prefix is appended onto the method names of getters that return values from the layer.
+ * These getters are called and the corresponding variables are set to have those values.
+ */
+ private def varResultPrefix = "getLayerVariableResult_"
+ private def varParamSetter = "setLayerVariableParameters"
+
+ /** cache that maps spiName of layer to the LayerVarsRuntime */
+ private[layers] lazy val alreadyCheckedLayers =
+ new mutable.LinkedHashMap[String, LayerVarsRuntime]()
+
+ /**
+ * Computes the things that can be computed once-only to make calling the constructor
+ * and passing the parameter variables faster at runtime. This also does all the
+ * checking that the constructor has an argument for each variable of properly matching type, and
+ * has a getter for each result variable, again returning the proper type.
+ *
+ * This is called at schema compile time to ensure the layer code is properly defined and matches
+ * the layer variable definitions.
+ *
+ * It is called again at runtime after the Layer class is loaded by the SPI
+ * to ensure that the loaded layer class constructor signature at least matches the layer
+ * variables defined in the schema.
+ *
+ * @param lrd the layer static schema compiler-determined information
+ * @param protoLayer the layer instance allocated by the SPI loader (zero-arg constructed)
+ * @return the runtime structure used to construct, optionally provide DFDL variable values
+ * as parameters, initialize, optinally get results which are stored-back to DFDL variables.
+ */
+ private def computeLayerVarsRuntime(
+ lrd: LayerRuntimeData,
+ protoLayer: Layer,
+ ): LayerVarsRuntime = {
+ val optLayerVarsRuntime = alreadyCheckedLayers.get(protoLayer.name())
+ optLayerVarsRuntime.getOrElse {
+ val c = protoLayer.getClass
+ val constructor = c.getConstructor() // cannot fail because SPI loader would have failed
+ val allMethods = c.getMethods
+ val optParamSetter = allMethods.find { _.getName == varParamSetter }
+ val allVarResultGetters =
+ ListSet(allMethods.filter { m =>
+ val nom = m.getName
+ nom.startsWith(varResultPrefix)
+ }.toSeq: _*)
+
+ if (lrd.qNameToVRD.isEmpty && optParamSetter.isEmpty && allVarResultGetters.isEmpty) {
+ // there are no vars, so no setter, and no getter(s). We're done
+ new LayerVarsRuntime(constructor, None, Nil, Nil)
+ } else {
+ val allLayerVRDs = ListSet(lrd.qNameToVRD.toSeq.map(_._2): _*)
+ // there is either a params setter, a result getter, or both.
+ val paramVRDs: Seq[VariableRuntimeData] =
+ optParamSetter.toSeq.flatMap { paramSetter =>
+ // there is a param setter with args that are supposed to correspond to bound vars
+ // it is allowed for it to have zero args. Then it's purpose is just initialization, but it
+ // must still get called.
+ val params = paramSetter.getParameters.toSeq
+ val paramTypes = paramSetter.getParameterTypes.toSeq
+ val pVRDs: Seq[VariableRuntimeData] = (params.zip(paramTypes)).map { case (p, pt) =>
+ val vrd = lrd.qNameToVRD.getOrElse(
+ p.getName,
+ lrd.context.SDE(
+ s"No layer DFDL variable named '${p.getName}' was found in namespace ${lrd.namespace}.",
+ ),
+ )
+ val vrdClass = PrimType.toJavaType(vrd.primType.dfdlType)
+ lrd.context.schemaDefinitionUnless(
+ compatibleTypes(vrdClass, pt),
+ s"""Layer setter argument ${vrd.globalQName.local} and the corresponding
+ |Layer DFDL variable have differing types: ${pt.getName}
+ | and ${vrdClass.getName} respectively.""".stripMargin,
+ )
+ vrd
+ }
+
+ // Now we deal with the result getters and the corresponding vars
+ //
+ pVRDs
+ }
+ val nParams = paramVRDs.length
+ val nVars = lrd.qNameToVRD.size
+ val returnVRDs = allLayerVRDs -- paramVRDs // set subtraction
+
+ // each returnVRD needs to have a corresponding getter method
+ val returnVRDNames = returnVRDs.map(_.globalQName.local)
+ val resultGettersNames = allVarResultGetters.map(_.getName.replace(varResultPrefix, ""))
+ val nResultGetters = resultGettersNames.size
+
+ val javaParamSetterArgs =
+ allLayerVRDs
+ .map { vrd =>
+ s"type: ${PrimType.toJavaTypeString(vrd.primType.dfdlType)} name: ${vrd.globalQName.local}"
+ }
+ .mkString(", ")
+
+ lrd.context.schemaDefinitionUnless(
+ nParams + nResultGetters == nVars,
+ s"""Layer class $c does not have a setter with arguments for each of the layer's variables.
+ | It should have a setter named $varParamSetter with an argument for each layer parameter, in any order, such as
+ | ($javaParamSetterArgs), and a getter for remaining layer variables, named with a specific
+ | name prefix like: ' $varResultPrefix '.""".stripMargin,
+ )
+
+ val returnVRDsWithoutGetters = returnVRDNames -- resultGettersNames
+ val resultGettersWithoutVRDs = resultGettersNames -- returnVRDNames
+ lrd.context.schemaDefinitionUnless(
+ returnVRDsWithoutGetters.isEmpty,
+ s"""The layer variables ${returnVRDsWithoutGetters.mkString(
+ ",",
+ )} have no corresponding getters.""",
+ )
+ lrd.context.schemaDefinitionUnless(
+ resultGettersWithoutVRDs.isEmpty, {
+ val getterFullNames = returnVRDsWithoutGetters.map { vname =>
+ this.varResultPrefix + vname
+ }
+ s"""The getters ${getterFullNames.mkString(
+ ",",
+ )} have no corresponding layer variables."""
+ },
+ )
+ // at this point we know each variable that was not a parameter of the setter
+ // has a getter with matching name.
+ val resultVarPairs = resultGettersNames.map { rgn: String =>
+ val getter: Method =
+ allVarResultGetters
+ .find { g: Method => g.getName == varResultPrefix + rgn }
+ .getOrElse {
+ Assert.invariantFailed("no getter for getter name.")
+ }
+ val vrd = returnVRDs.find { vrd => vrd.globalQName.local == rgn }.getOrElse {
+ Assert.invariantFailed("no vrd for getter name.")
+ }
+ (vrd, getter)
+ }
+ resultVarPairs.foreach { case (vrd, getter) =>
+ val vrdClass = PrimType.toJavaType(vrd.primType.dfdlType)
+ val gt = getter.getReturnType
+ lrd.context.schemaDefinitionUnless(
+ compatibleTypes(vrdClass, gt),
+ s"""Layer return variable ${vrd.globalQName.local} and the corresponding
+ |Layer getter have differing types: ${vrdClass.getName}
+ | and ${gt.getName} respectively.""".stripMargin,
+ )
+ lrd.context.schemaDefinitionUnless(
+ getter.getParameterCount == 0,
+ s"""Layer return variable getter ${getter.getName} must have no arguments.""",
+ )
+ }
+ val lrv =
+ new LayerVarsRuntime(constructor, optParamSetter, paramVRDs, resultVarPairs.toSeq)
+ alreadyCheckedLayers.put(lrd.spiName, lrv)
+ lrv
+ }
+ }
+ }
+
+ /**
+ * Tolerates both boxed and unboxed flavors of the primitive types
+ * @param vrdClass - layer variable's type as a class object
+ * @param pt - setter arg type, or getter result type from reflection
+ * @return true if the types are compatible meaning a DFDL variable can supply the parameter, or receive the result.
+ */
+ private def compatibleTypes(vrdClass: Class[_], pt: Class[_]): Boolean = {
+ vrdClass == pt || compatibleBoxedPrimType(vrdClass, pt)
+ }
+
+ private val LongPrimClass = classOf[Long]
+ private val IntPrimClass = classOf[Int]
+ private val ShortPrimClass = classOf[Short]
+ private val BytePrimClass = classOf[Byte]
+ private val FloatPrimClass = classOf[Float]
+ private val DoublePrimClass = classOf[Double]
+ private val BooleanPrimClass = classOf[Boolean]
+
+ private def compatibleBoxedPrimType(vrdClass: Class[_], pt: Class[_]): Boolean = {
+ pt match {
+ case LongPrimClass => vrdClass == classOf[JLong]
+ case IntPrimClass => vrdClass == classOf[JInt]
+ case ShortPrimClass => vrdClass == classOf[JShort]
+ case BytePrimClass => vrdClass == classOf[JByte]
+ case FloatPrimClass => vrdClass == classOf[JFloat]
+ case DoublePrimClass => vrdClass == classOf[JDouble]
+ case BooleanPrimClass => vrdClass == classOf[JBoolean]
+ case _ => false
+ }
+ }
+
+ private lazy val layerMap: Map[String, Layer] =
+ SimpleNamedServiceLoader.loadClass[Layer](classOf[Layer])
+
+ private def findSPILayer(spiName: String, lrd: LayerRuntimeData): Layer = {
+ val optLayer: Option[Layer] = layerMap.get(spiName)
+ optLayer.getOrElse {
+ val choices = layerMap.keySet.mkString(", ")
+ lrd.context.SDE(
+ "The dfdlx:layer '%s' was not found. Available choices are: %s",
+ spiName,
+ choices,
+ )
+ }
+ }
+
+ private def compile(layerRuntimeData: LayerRuntimeData): LayerVarsRuntime = {
+ val optLayerVarsRuntime = alreadyCheckedLayers.get(layerRuntimeData.spiName)
+ optLayerVarsRuntime.getOrElse {
+ val layer = findSPILayer(layerRuntimeData.spiName, layerRuntimeData)
+ val res = computeLayerVarsRuntime(layerRuntimeData, layer)
+ alreadyCheckedLayers.put(layerRuntimeData.spiName, res)
+ res
+ }
+ }
+
+ /**
+ * Called twice. Once during schema compilation time (for when the compiled schema
+ * is immediately used), once when a compiled schema is reloaded.
+ *
+ * The two calls are necessary because the resulting structure, which is needed at runtime,
+ * is not serializable, so can't be saved from schema compilation time as part of the
+ * runtime data.
+ *
+ * Any errors that occur are schema definition errors.
+ *
+ * Note that user-written layer code is not run at this time, but method signatures
+ * of setters/getters are checked against the DFDL variables defined in
+ * the layer's target namespace.
+ */
+ def compileAll(lrds: Seq[LayerRuntimeData]): Unit = {
+ lrds.foreach { compile(_) }
+ }
+
+ /**
+ * Requires that the argument is for a layer that has already been runtime compiled.
+ * @param lrd
+ * @return the LayerVarsRuntime object for the layer.
+ */
+ def getLayerVarsRuntime(lrd: LayerRuntimeData): LayerVarsRuntime = {
+ alreadyCheckedLayers.get(lrd.spiName).getOrElse {
+ Assert.invariantFailed("layer was not already compiled.")
+ }
+ }
+}
diff --git a/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/LayerRuntimeData.scala b/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/LayerRuntimeData.scala
new file mode 100644
index 000000000..c08980de0
--- /dev/null
+++ b/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/LayerRuntimeData.scala
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.daffodil.runtime1.layers
+
+import org.apache.daffodil.lib.exceptions.SchemaFileLocation
+import org.apache.daffodil.lib.exceptions.ThrowsSDE
+import org.apache.daffodil.lib.xml.NS
+import org.apache.daffodil.lib.xml.RefQName
+import org.apache.daffodil.runtime1.processors.VariableRuntimeData
+
+/**
+ * Like ElementRuntimeData, SequenceRuntimeData, etc. this class
+ * contains the static information created by the schema compilation
+ * process that is _used_ by the runtime system.
+ *
+ */
+class LayerRuntimeData(
+ val layerQName: RefQName,
+ val schemaFileLocation: SchemaFileLocation,
+ val qNameToVRD: Map[String, VariableRuntimeData],
+ val context: ThrowsSDE,
+) extends Serializable {
+
+ final def localName: String = layerQName.local
+ final def namespace: NS = layerQName.namespace
+
+ final def toss(th: Throwable): Nothing =
+ throw th // good place for a breakpoint
+
+ lazy val spiName: String = LayerUtils.spiName(localName, namespace)
+}
diff --git a/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/LayerTransformer.scala b/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/LayerTransformer.scala
deleted file mode 100644
index 3d8439425..000000000
--- a/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/layers/LayerTransformer.scala
+++ /dev/null
@@ -1,453 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.daffodil.runtime1.layers
-
-import java.io.ByteArrayInputStream
-import java.io.EOFException
-import java.io.InputStream
-import java.io.OutputStream
-import java.nio.ByteBuffer
-import java.nio.charset.Charset
-
-import org.apache.daffodil.io.DataInputStream.Mark
-import org.apache.daffodil.io.DataOutputStream
-import org.apache.daffodil.io.DirectOrBufferedDataOutputStream
-import org.apache.daffodil.io.FormatInfo
-import org.apache.daffodil.io.InputSourceDataInputStream
-import org.apache.daffodil.io.processors.charset.BitsCharsetJava
-import org.apache.daffodil.lib.api.DataLocation
-import org.apache.daffodil.lib.api.ThinDiagnostic
-import org.apache.daffodil.lib.exceptions.Assert
-import org.apache.daffodil.lib.exceptions.SchemaFileLocation
-import org.apache.daffodil.lib.schema.annotation.props.gen.BitOrder
-import org.apache.daffodil.lib.schema.annotation.props.gen.LayerLengthKind
-import org.apache.daffodil.lib.schema.annotation.props.gen.LayerLengthUnits
-import org.apache.daffodil.lib.util.ByteBufferOutputStream
-import org.apache.daffodil.lib.util.Maybe
-import org.apache.daffodil.lib.util.Maybe.Nope
-import org.apache.daffodil.lib.util.Maybe.One
-import org.apache.daffodil.lib.util.Misc
-import org.apache.daffodil.runtime1.infoset.DataValue
-import org.apache.daffodil.runtime1.processors.Evaluatable
-import org.apache.daffodil.runtime1.processors.LayerBoundaryMarkEv
-import org.apache.daffodil.runtime1.processors.LayerCharsetEv
-import org.apache.daffodil.runtime1.processors.LayerLengthEv
-import org.apache.daffodil.runtime1.processors.ParseOrUnparseState
-import org.apache.daffodil.runtime1.processors.SequenceRuntimeData
-import org.apache.daffodil.runtime1.processors.SuspendableOperation
-import org.apache.daffodil.runtime1.processors.VariableRuntimeData
-import org.apache.daffodil.runtime1.processors.parsers.PState
-import org.apache.daffodil.runtime1.processors.unparsers.UState
-
-import org.apache.commons.io.IOUtils
-import passera.unsigned.ULong
-
-/**
- * Shared functionality of all LayerTransformers.
- *
- * A layer transformer is created at runtime as part of a single parse/unparse call.
- * Hence, they can be stateful without causing thread-safety issues.
- */
-abstract class LayerTransformer(val layerName: String, layerRuntimeInfo: LayerRuntimeInfo) {
-
- protected def wrapLayerDecoder(jis: InputStream): InputStream
-
- protected def wrapLimitingStream(state: ParseOrUnparseState, jis: InputStream): InputStream
-
- final def wrapJavaInputStream(
- s: InputSourceDataInputStream,
- fInfo: FormatInfo,
- ): InputStream = {
- new JavaIOInputStream(s, fInfo)
- }
-
- protected def wrapLayerEncoder(jos: OutputStream): OutputStream
-
- protected def wrapLimitingStream(state: ParseOrUnparseState, jis: OutputStream): OutputStream
-
... 9518 lines suppressed ...