You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@daffodil.apache.org by GitBox <gi...@apache.org> on 2018/12/15 05:38:56 UTC
[GitHub] mbeckerle opened a new pull request #158: Daffodil 1080 sequences
and separators - preliminary review
mbeckerle opened a new pull request #158: Daffodil 1080 sequences and separators - preliminary review
URL: https://github.com/apache/incubator-daffodil/pull/158
This is a checkpoint in work to improve Sequences and Separators.
I do not intend to merge this branch; however, review would be helpful perhaps.
There are some difficult issues here.
Tests that distinguish proper function in Daffodil, proper function in
IBM DFDL, failures of each, and differences between them added.
The tests that fail (either implementation) are in scala-debug.
Parsing is working better than unparsing. Unparsing is failing in a
number
of important scenarios still.
There is much duplication of test-case logic in order to provide
distinct parse/unparse and daffodil/ibm variants of a test.
In many cases there are 4 copies of a test. Ex: for a test named "foo":
fooN_Np_daf
fooN_Np_ibm
fooN_Nu_daf
fooN_Nu_ibm
In these cases the test document data, and test infoset are IDENTICAL
AND SHOULD BE KEPT THAT WAY. Eventually as we improve things we would
hope
to collapse these back to a single, portable and round-trip test.
(alternatively, TDML Runner features allowing a single test to be
expressed by drawing in infoset and document from shared definitions,
would be helpful. That way one could have 4 tests, but have them
explicitly share the same data document definition and infoset
definition.
You can do this today, but only by putting the infoset and document part
into external files. I'd like to keep this self contained.)
For now this split up allow us to have tests which work only some of
the 4 possible combinations, and to split those up over scala vs.
scala-debug.
The "_ibm" tests are to understand and characterize the behavior of
that implementation that we need to interoperate with, because in some
cases this behavior is NOT conforming to current DFDL specification
due to well known bugs and limitations.
== Affect of changes so far on general function/regression tests ==
1 regression in daffodil-test-ibm1 - AX000 fails. Complex situation at
first glance.
1 regression in daffodil-tdml-processor tests - the 'threePass' test
doesn't work because it was depending on an empty element being created
that is now suppressed. See below discussion.
36 regressions in daffodil-test.
A substantial number of these are due to empty elements that we were
creating, but no longer are, and some are nilled elements we used to
create but no longer are.
There are some other failure scenarios also however.
Right now the code does too much suppression of elements having zero
length. They are not supposed to be suppressed if ZL is the nil
representation nor if ZL is the "empty string" or "empty hexBinary"
representation.
Daffodil should implement the correct DFDL-spec-compliant behavior,
and did prior to these changes. In order to get interop with IBM DFDL
on some of the new EDI-style tests, I changed behavior for string and
hexbinary elements so that they are more often suppressed as elements.
However, IBM DFDL does not implement this functionality according to
the DFDL spec, so interoperability with IBM DFDL will require a flag
that changes this behavior. Many of these changes were added to DFDL
spec several years ago before the 2014 revision of DFDL v1.0. But
IBM's DFDL implementation was already in operation, so they have a
legacy of behavior they'll have to eventually adjust. This big set of
subtle revisions were associated with the DFDL Workgroup "Action 140"
work/action item. Basically IBM DFDL did not implement (some ? or all
of) the Action 140 changes.
So we either need a flag, or alternatively, 36 tests have to be made
Daffodil specific - which may be preferable. This depends on whether
the behavior is required to make "real" schemas work portably, or just
little test cases.
DAFFODIL-1080
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services