You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@daffodil.apache.org by Mike Beckerle <mb...@apache.org> on 2023/06/05 22:07:08 UTC
[DISCUSS] time to create daffodil 3.5.0 release?
All critical and blocker type bugs have been fixed including a couple quite
challenging ones in the last few weeks.
I think it is time to create an RC for 3.5.0.
Please discuss through 6pm EDT.US on Thursday June 8. (72 hours) and if
favorable we'll figure out who gets to create the RC.
-mikeb
Mike Beckerle
Apache Daffodil PMC | daffodil.apache.org
OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl
Owl Cyber Defense | www.owlcyberdefense.com
RE: [DISCUSS] time to create daffodil 3.5.0 release?
Posted by "Interrante, John A (GE Aerospace, US)" <Jo...@ge.com>.
I would like to throw in another daffodil-codegen-c change before we release 3.5.0/4.0.0. The change has already been in use locally for a while; I just hadn't gotten around to writing a unit test, creating a JIRA issue number, and submitting a PR yet. The unit test is done and I'll submit the PR very soon.
John
-----Original Message-----
From: Mike Beckerle <mb...@apache.org>
Sent: Monday, June 5, 2023 6:07 PM
To: dev@daffodil.apache.org
Subject: EXT: [DISCUSS] time to create daffodil 3.5.0 release?
WARNING: This email originated from outside of GE. Please validate the sender's email address before clicking on links or attachments as they may not be safe.
All critical and blocker type bugs have been fixed including a couple quite challenging ones in the last few weeks.
I think it is time to create an RC for 3.5.0.
Please discuss through 6pm EDT.US on Thursday June 8. (72 hours) and if favorable we'll figure out who gets to create the RC.
-mikeb
Mike Beckerle
Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl
Owl Cyber Defense | www.owlcyberdefense.com
Re: [DISCUSS] time to create daffodil 3.5.0 release?
Posted by Steve Lawrence <sl...@apache.org>.
Actually, I've since found two bugs related to the IBM DFDL crosstester
that I think should be fixed:
DAFFODIL-2820: GeneralFormatPortable isn't actually portable due to
emptyElementParsePolicy
https://issues.apache.org/jira/browse/DAFFODIL-2820:
DAFFODIL-2813: Schematron URIResolver uses Classpath rather than
Daffodil Resolver
https://issues.apache.org/jira/browse/DAFFODIL-2813
The first is needed to do any IBM DFDL cross testing with schemas that
use GeneralFormat (which is alot). I have a simple fix for that, but it
has a regression tha requires the second to be fixed. I hope to have a
fix and get them into code review today.
- Steve
On 2023-06-12 09:57 AM, Mike Beckerle wrote:
> Steve, thanks so much for going through this.
>
> I would go with your suggestion that it seems like we should stick with
> 3.5.0 for the release numbering.
>
> There have been no objections to the creation of the release within the 72+
> hours we've had this thread open, so we will proceed now to create RC1 and
> bring it for a vote.
>
> For our next feature release, we need to improve and firm up the API for
> all Daffodil features people are using (layering specifically) so that we
> can support it longer term.
>
> On Fri, Jun 9, 2023 at 8:33 AM Steve Lawrence <sl...@apache.org> wrote:
>
>> I have pretty much completed updating the regression suite to work with
>> the latest 3.5.0 snapshot. Just a few to go, but they are variants of
>> schemas I have fixed, so I do no expect any additional regressions.
>> Below is a summary of the changes that were needed. Note that many of
>> the schema projecs have not been updated since much older versions of
>> Daffodil, like 2.6.0 and 3.1.0, so some changes were required that are
>> not specific to 3.5.0. Those are listed at towards the end.
>>
>> * DAFFODIL-1704: A number of schemas broke because we now ensure that
>> binary specified lengths are not larger than the associated types. I
>> think this is considered a DFDL schema bug and is a good regression that
>> schema authors should fix.
>>
>> * DAFFODIL-2683: Changes to OSGI broke a number of test suits. There are
>> a relatively small number of classes that affect the tests and the
>> needed changes are very simple. The relevant classes are:
>>
>> org.apache.daffodil.core.compiler.Compiler
>> org.apache.daffodil.lib.api.URISchemaSource
>> org.apache.daffodil.lib.api.WithDiagnostics
>> org.apache.daffodil.lib.exceptions.Abort
>> org.apache.daffodil.lib.exceptions.Assert
>> org.apache.daffodil.lib.schema.annotation.props.gen.LayerLengthKind
>> org.apache.daffodil.lib.schema.annotation.props.gen.LayerLengthUnits
>> org.apache.daffodil.lib.util.Misc
>> org.apache.daffodil.lib.xml.XMLUtils
>> org.apache.daffodil.runtime1.dpath.NodeInfo.PrimType
>> org.apache.daffodil.runtime1.layers.ByteBufferExplicitLengthLayerTransform
>> org.apache.daffodil.runtime1.layers.LayerCompileInfo
>> org.apache.daffodil.runtime1.layers.LayerCompiler
>> org.apache.daffodil.runtime1.layers.LayerRuntimeInfo
>> org.apache.daffodil.runtime1.layers.LayerTransformerFactory
>> org.apache.daffodil.runtime1.processors.ParseOrUnparseState
>> org.apache.daffodil.runtime1.processors.VariableRuntimeData
>> org.apache.daffodil.runtime1.processors.parsers.ParseError
>>
>> Note that no changes are needed for the TDML runner or Java/Scala API
>> classes, which I sort of consider the "public" API which tries harder to
>> maintain backwards compatibility.
>>
>> Some of these, like the Compiler/Diagnostics/etc should probably just be
>> updated to the the more stable Scala/Java API. Others are related to
>> custom layers, but I consider that an experimental API that is still
>> subject to change. Considering this and the ease of changes (only import
>> lines needed to change), while it did cause regressions, I'm not sure it
>> rises to the level of a 4.0.0 release. If we required actual code
>> changes, or broke the TDMLRunner or Scala/Java API's which are more
>> "public", I think that would be a stronger argument.
>>
>> * DAFFODIL-2158: We changed behavior of textNumberCheckPolicy="strict"
>> so that the pattern is used to determine if a decimal point must exist
>> or not. For example, if text numbers are integers without a decimal, the
>> pattern should not contain a decimal. I believe this was just a bug in
>> Daffodil, and schema authors should update their schemas to either use a
>> lax policy or use a pattern that correctly matches their data.
>>
>> * DAFFODIL-2818: A regression related to parsing negative zero was found
>> and fixed. This bug was created and fixed in 3.5.0.
>>
>> * DAFFODIL-2607: Diagnostics were modified/improved when an enumeration
>> key was not found, which some tests checked for. The tests either need
>> to be less specific or check for the new error message. I don't think we
>> make a guarantee that diagnostics will not change.
>>
>> ====
>>
>> The following changes were required due to Scala/Java API changes, but
>> those went through a deprecation process and were removed prior to
>> 3.5.0. Because it was prior to 3.5.0, I have not looked into exactly
>> which prior release the changes occurred:
>>
>> * If validation errors are created by a parse but no validation errors
>> are listed in a TDML test cases, the TDML test fails. I am actually not
>> sure if this is a change to Daffodil behavior or if depedent schema
>> update just included new validation checks that now create errors. This
>> may be a schema regression and not a Daffodil regression.
>>
>> * Some formats had custom tests that use an old Scala/Java API before
>> the introduction of InputSourceInputDataStream. This must be updated to
>> use the new API.
>>
>> * The setValidateDFDLSchemas function was removed from the Compiler
>> class, which a few classes used to enable schema validation. Schemas are
>> now always validated and the Java/Scala API does not have a way to
>> disable this.
>>
>> * The setDistinguishedRootNode function was removed from the Compiler
>> class. Instead the root node should be passed as a parameter to the
>> compileSource/File function, or withDistinguishedRootNode should be used
>> on the ProcessorFactory.
>>
>> * The ParseResult.isAtEnd function was removed from the ParseResult
>> class. Instead, the InputSourceDataInputStream.hasData function should
>> be used.
>>
>> * A number of test suites still use the old tdml.DFDLTestSuite instead
>> of the new tdml.Runner.
>>
>>
>>
>>
>> On 2023-06-06 07:42 PM, Mike Beckerle wrote:
>>> Topic: Should this be version 4.0.0, not 3.5.0?
>>>
>>> The OGSI-package-name-changes have broken many schemas because the test
>>> rigs used Daffodil internals, and some deprecated things were finally
>>> removed in the current 3.5.0 snapshot.
>>>
>>> But as a result, these schemas, many of which are non-public, but
>>> nevertheless can be counted, a long list, which used to work, don't work
>>> any more.
>>> It's possible there are false-negatives here. Nevertheless it is a long
>>> list.
>>>
>>> [fail] (04/62) dfdl-bmp
>>> [fail] (11/62) dfdl-examples-hexwords
>>> [fail] (12/62) dfdl-examples-self-descriptive-data
>>> [fail] (13/62) dfdl-examples-xslt-csv
>>> [fail] (22/62) dfdl-imf-icalendar-cse
>>> [fail] (23/62) dfdl-ipf-demo
>>> [fail] (29/62) dfdl-jpeg2000
>>> [fail] (30/62) dfdl-jreap
>>> [fail] (34/62) dfdl-mil-std-2045
>>> [fail] (38/62) dfdl-nitf
>>> [fail] (42/62) dfdl-plc4x-s7
>>> [fail] (43/62) dfdl-png
>>> [fail] (46/62) dfdl-raster
>>> [fail] (49/62) dfdl-stanag-5516
>>> [fail] (50/62) dfdl-stanag-5516-with-nact
>>> [fail] (53/62) dfdl-usmtf
>>> [fail] (54/62) dfdl-usmtf-generic
>>> [fail] (57/62) dfdl-vmf
>>> [fail] (59/62) dfdl-vmf-spock
>>> [fail] (61/62) ibm-dfdl-crosstester
>>>
>>> This suggests to me that user-created schemas have some fairly high
>>> likelihood of being clobbered.
>>>
>>> We changed the packages of many things. We can claim these were internal
>>> things, but there are many schemas that depend on Daffodil internal
>> things
>>> now. This needs to get fixed so we can torque the internals more without
>>> breaking things, but right now... things are broken.
>>>
>>> Every schema using layers, for example as we never hardened the API for
>>> layers, so as it has evolved, schemas have broken.
>>>
>>>
>>>
>>> On Mon, Jun 5, 2023 at 6:07 PM Mike Beckerle <mb...@apache.org>
>> wrote:
>>>
>>>>
>>>> All critical and blocker type bugs have been fixed including a couple
>>>> quite challenging ones in the last few weeks.
>>>>
>>>> I think it is time to create an RC for 3.5.0.
>>>>
>>>> Please discuss through 6pm EDT.US on Thursday June 8. (72 hours) and if
>>>> favorable we'll figure out who gets to create the RC.
>>>>
>>>> -mikeb
>>>>
>>>> Mike Beckerle
>>>> Apache Daffodil PMC | daffodil.apache.org
>>>> OGF DFDL Workgroup Co-Chair |
>> www.ogf.org/ogf/doku.php/standards/dfdl/dfdl
>>>> Owl Cyber Defense | www.owlcyberdefense.com
>>>>
>>>>
>>>>
>>>
>>
>>
>
Re: [DISCUSS] time to create daffodil 3.5.0 release?
Posted by Mike Beckerle <mb...@apache.org>.
Steve, thanks so much for going through this.
I would go with your suggestion that it seems like we should stick with
3.5.0 for the release numbering.
There have been no objections to the creation of the release within the 72+
hours we've had this thread open, so we will proceed now to create RC1 and
bring it for a vote.
For our next feature release, we need to improve and firm up the API for
all Daffodil features people are using (layering specifically) so that we
can support it longer term.
On Fri, Jun 9, 2023 at 8:33 AM Steve Lawrence <sl...@apache.org> wrote:
> I have pretty much completed updating the regression suite to work with
> the latest 3.5.0 snapshot. Just a few to go, but they are variants of
> schemas I have fixed, so I do no expect any additional regressions.
> Below is a summary of the changes that were needed. Note that many of
> the schema projecs have not been updated since much older versions of
> Daffodil, like 2.6.0 and 3.1.0, so some changes were required that are
> not specific to 3.5.0. Those are listed at towards the end.
>
> * DAFFODIL-1704: A number of schemas broke because we now ensure that
> binary specified lengths are not larger than the associated types. I
> think this is considered a DFDL schema bug and is a good regression that
> schema authors should fix.
>
> * DAFFODIL-2683: Changes to OSGI broke a number of test suits. There are
> a relatively small number of classes that affect the tests and the
> needed changes are very simple. The relevant classes are:
>
> org.apache.daffodil.core.compiler.Compiler
> org.apache.daffodil.lib.api.URISchemaSource
> org.apache.daffodil.lib.api.WithDiagnostics
> org.apache.daffodil.lib.exceptions.Abort
> org.apache.daffodil.lib.exceptions.Assert
> org.apache.daffodil.lib.schema.annotation.props.gen.LayerLengthKind
> org.apache.daffodil.lib.schema.annotation.props.gen.LayerLengthUnits
> org.apache.daffodil.lib.util.Misc
> org.apache.daffodil.lib.xml.XMLUtils
> org.apache.daffodil.runtime1.dpath.NodeInfo.PrimType
> org.apache.daffodil.runtime1.layers.ByteBufferExplicitLengthLayerTransform
> org.apache.daffodil.runtime1.layers.LayerCompileInfo
> org.apache.daffodil.runtime1.layers.LayerCompiler
> org.apache.daffodil.runtime1.layers.LayerRuntimeInfo
> org.apache.daffodil.runtime1.layers.LayerTransformerFactory
> org.apache.daffodil.runtime1.processors.ParseOrUnparseState
> org.apache.daffodil.runtime1.processors.VariableRuntimeData
> org.apache.daffodil.runtime1.processors.parsers.ParseError
>
> Note that no changes are needed for the TDML runner or Java/Scala API
> classes, which I sort of consider the "public" API which tries harder to
> maintain backwards compatibility.
>
> Some of these, like the Compiler/Diagnostics/etc should probably just be
> updated to the the more stable Scala/Java API. Others are related to
> custom layers, but I consider that an experimental API that is still
> subject to change. Considering this and the ease of changes (only import
> lines needed to change), while it did cause regressions, I'm not sure it
> rises to the level of a 4.0.0 release. If we required actual code
> changes, or broke the TDMLRunner or Scala/Java API's which are more
> "public", I think that would be a stronger argument.
>
> * DAFFODIL-2158: We changed behavior of textNumberCheckPolicy="strict"
> so that the pattern is used to determine if a decimal point must exist
> or not. For example, if text numbers are integers without a decimal, the
> pattern should not contain a decimal. I believe this was just a bug in
> Daffodil, and schema authors should update their schemas to either use a
> lax policy or use a pattern that correctly matches their data.
>
> * DAFFODIL-2818: A regression related to parsing negative zero was found
> and fixed. This bug was created and fixed in 3.5.0.
>
> * DAFFODIL-2607: Diagnostics were modified/improved when an enumeration
> key was not found, which some tests checked for. The tests either need
> to be less specific or check for the new error message. I don't think we
> make a guarantee that diagnostics will not change.
>
> ====
>
> The following changes were required due to Scala/Java API changes, but
> those went through a deprecation process and were removed prior to
> 3.5.0. Because it was prior to 3.5.0, I have not looked into exactly
> which prior release the changes occurred:
>
> * If validation errors are created by a parse but no validation errors
> are listed in a TDML test cases, the TDML test fails. I am actually not
> sure if this is a change to Daffodil behavior or if depedent schema
> update just included new validation checks that now create errors. This
> may be a schema regression and not a Daffodil regression.
>
> * Some formats had custom tests that use an old Scala/Java API before
> the introduction of InputSourceInputDataStream. This must be updated to
> use the new API.
>
> * The setValidateDFDLSchemas function was removed from the Compiler
> class, which a few classes used to enable schema validation. Schemas are
> now always validated and the Java/Scala API does not have a way to
> disable this.
>
> * The setDistinguishedRootNode function was removed from the Compiler
> class. Instead the root node should be passed as a parameter to the
> compileSource/File function, or withDistinguishedRootNode should be used
> on the ProcessorFactory.
>
> * The ParseResult.isAtEnd function was removed from the ParseResult
> class. Instead, the InputSourceDataInputStream.hasData function should
> be used.
>
> * A number of test suites still use the old tdml.DFDLTestSuite instead
> of the new tdml.Runner.
>
>
>
>
> On 2023-06-06 07:42 PM, Mike Beckerle wrote:
> > Topic: Should this be version 4.0.0, not 3.5.0?
> >
> > The OGSI-package-name-changes have broken many schemas because the test
> > rigs used Daffodil internals, and some deprecated things were finally
> > removed in the current 3.5.0 snapshot.
> >
> > But as a result, these schemas, many of which are non-public, but
> > nevertheless can be counted, a long list, which used to work, don't work
> > any more.
> > It's possible there are false-negatives here. Nevertheless it is a long
> > list.
> >
> > [fail] (04/62) dfdl-bmp
> > [fail] (11/62) dfdl-examples-hexwords
> > [fail] (12/62) dfdl-examples-self-descriptive-data
> > [fail] (13/62) dfdl-examples-xslt-csv
> > [fail] (22/62) dfdl-imf-icalendar-cse
> > [fail] (23/62) dfdl-ipf-demo
> > [fail] (29/62) dfdl-jpeg2000
> > [fail] (30/62) dfdl-jreap
> > [fail] (34/62) dfdl-mil-std-2045
> > [fail] (38/62) dfdl-nitf
> > [fail] (42/62) dfdl-plc4x-s7
> > [fail] (43/62) dfdl-png
> > [fail] (46/62) dfdl-raster
> > [fail] (49/62) dfdl-stanag-5516
> > [fail] (50/62) dfdl-stanag-5516-with-nact
> > [fail] (53/62) dfdl-usmtf
> > [fail] (54/62) dfdl-usmtf-generic
> > [fail] (57/62) dfdl-vmf
> > [fail] (59/62) dfdl-vmf-spock
> > [fail] (61/62) ibm-dfdl-crosstester
> >
> > This suggests to me that user-created schemas have some fairly high
> > likelihood of being clobbered.
> >
> > We changed the packages of many things. We can claim these were internal
> > things, but there are many schemas that depend on Daffodil internal
> things
> > now. This needs to get fixed so we can torque the internals more without
> > breaking things, but right now... things are broken.
> >
> > Every schema using layers, for example as we never hardened the API for
> > layers, so as it has evolved, schemas have broken.
> >
> >
> >
> > On Mon, Jun 5, 2023 at 6:07 PM Mike Beckerle <mb...@apache.org>
> wrote:
> >
> >>
> >> All critical and blocker type bugs have been fixed including a couple
> >> quite challenging ones in the last few weeks.
> >>
> >> I think it is time to create an RC for 3.5.0.
> >>
> >> Please discuss through 6pm EDT.US on Thursday June 8. (72 hours) and if
> >> favorable we'll figure out who gets to create the RC.
> >>
> >> -mikeb
> >>
> >> Mike Beckerle
> >> Apache Daffodil PMC | daffodil.apache.org
> >> OGF DFDL Workgroup Co-Chair |
> www.ogf.org/ogf/doku.php/standards/dfdl/dfdl
> >> Owl Cyber Defense | www.owlcyberdefense.com
> >>
> >>
> >>
> >
>
>
Re: [DISCUSS] time to create daffodil 3.5.0 release?
Posted by Steve Lawrence <sl...@apache.org>.
I have pretty much completed updating the regression suite to work with
the latest 3.5.0 snapshot. Just a few to go, but they are variants of
schemas I have fixed, so I do no expect any additional regressions.
Below is a summary of the changes that were needed. Note that many of
the schema projecs have not been updated since much older versions of
Daffodil, like 2.6.0 and 3.1.0, so some changes were required that are
not specific to 3.5.0. Those are listed at towards the end.
* DAFFODIL-1704: A number of schemas broke because we now ensure that
binary specified lengths are not larger than the associated types. I
think this is considered a DFDL schema bug and is a good regression that
schema authors should fix.
* DAFFODIL-2683: Changes to OSGI broke a number of test suits. There are
a relatively small number of classes that affect the tests and the
needed changes are very simple. The relevant classes are:
org.apache.daffodil.core.compiler.Compiler
org.apache.daffodil.lib.api.URISchemaSource
org.apache.daffodil.lib.api.WithDiagnostics
org.apache.daffodil.lib.exceptions.Abort
org.apache.daffodil.lib.exceptions.Assert
org.apache.daffodil.lib.schema.annotation.props.gen.LayerLengthKind
org.apache.daffodil.lib.schema.annotation.props.gen.LayerLengthUnits
org.apache.daffodil.lib.util.Misc
org.apache.daffodil.lib.xml.XMLUtils
org.apache.daffodil.runtime1.dpath.NodeInfo.PrimType
org.apache.daffodil.runtime1.layers.ByteBufferExplicitLengthLayerTransform
org.apache.daffodil.runtime1.layers.LayerCompileInfo
org.apache.daffodil.runtime1.layers.LayerCompiler
org.apache.daffodil.runtime1.layers.LayerRuntimeInfo
org.apache.daffodil.runtime1.layers.LayerTransformerFactory
org.apache.daffodil.runtime1.processors.ParseOrUnparseState
org.apache.daffodil.runtime1.processors.VariableRuntimeData
org.apache.daffodil.runtime1.processors.parsers.ParseError
Note that no changes are needed for the TDML runner or Java/Scala API
classes, which I sort of consider the "public" API which tries harder to
maintain backwards compatibility.
Some of these, like the Compiler/Diagnostics/etc should probably just be
updated to the the more stable Scala/Java API. Others are related to
custom layers, but I consider that an experimental API that is still
subject to change. Considering this and the ease of changes (only import
lines needed to change), while it did cause regressions, I'm not sure it
rises to the level of a 4.0.0 release. If we required actual code
changes, or broke the TDMLRunner or Scala/Java API's which are more
"public", I think that would be a stronger argument.
* DAFFODIL-2158: We changed behavior of textNumberCheckPolicy="strict"
so that the pattern is used to determine if a decimal point must exist
or not. For example, if text numbers are integers without a decimal, the
pattern should not contain a decimal. I believe this was just a bug in
Daffodil, and schema authors should update their schemas to either use a
lax policy or use a pattern that correctly matches their data.
* DAFFODIL-2818: A regression related to parsing negative zero was found
and fixed. This bug was created and fixed in 3.5.0.
* DAFFODIL-2607: Diagnostics were modified/improved when an enumeration
key was not found, which some tests checked for. The tests either need
to be less specific or check for the new error message. I don't think we
make a guarantee that diagnostics will not change.
====
The following changes were required due to Scala/Java API changes, but
those went through a deprecation process and were removed prior to
3.5.0. Because it was prior to 3.5.0, I have not looked into exactly
which prior release the changes occurred:
* If validation errors are created by a parse but no validation errors
are listed in a TDML test cases, the TDML test fails. I am actually not
sure if this is a change to Daffodil behavior or if depedent schema
update just included new validation checks that now create errors. This
may be a schema regression and not a Daffodil regression.
* Some formats had custom tests that use an old Scala/Java API before
the introduction of InputSourceInputDataStream. This must be updated to
use the new API.
* The setValidateDFDLSchemas function was removed from the Compiler
class, which a few classes used to enable schema validation. Schemas are
now always validated and the Java/Scala API does not have a way to
disable this.
* The setDistinguishedRootNode function was removed from the Compiler
class. Instead the root node should be passed as a parameter to the
compileSource/File function, or withDistinguishedRootNode should be used
on the ProcessorFactory.
* The ParseResult.isAtEnd function was removed from the ParseResult
class. Instead, the InputSourceDataInputStream.hasData function should
be used.
* A number of test suites still use the old tdml.DFDLTestSuite instead
of the new tdml.Runner.
On 2023-06-06 07:42 PM, Mike Beckerle wrote:
> Topic: Should this be version 4.0.0, not 3.5.0?
>
> The OGSI-package-name-changes have broken many schemas because the test
> rigs used Daffodil internals, and some deprecated things were finally
> removed in the current 3.5.0 snapshot.
>
> But as a result, these schemas, many of which are non-public, but
> nevertheless can be counted, a long list, which used to work, don't work
> any more.
> It's possible there are false-negatives here. Nevertheless it is a long
> list.
>
> [fail] (04/62) dfdl-bmp
> [fail] (11/62) dfdl-examples-hexwords
> [fail] (12/62) dfdl-examples-self-descriptive-data
> [fail] (13/62) dfdl-examples-xslt-csv
> [fail] (22/62) dfdl-imf-icalendar-cse
> [fail] (23/62) dfdl-ipf-demo
> [fail] (29/62) dfdl-jpeg2000
> [fail] (30/62) dfdl-jreap
> [fail] (34/62) dfdl-mil-std-2045
> [fail] (38/62) dfdl-nitf
> [fail] (42/62) dfdl-plc4x-s7
> [fail] (43/62) dfdl-png
> [fail] (46/62) dfdl-raster
> [fail] (49/62) dfdl-stanag-5516
> [fail] (50/62) dfdl-stanag-5516-with-nact
> [fail] (53/62) dfdl-usmtf
> [fail] (54/62) dfdl-usmtf-generic
> [fail] (57/62) dfdl-vmf
> [fail] (59/62) dfdl-vmf-spock
> [fail] (61/62) ibm-dfdl-crosstester
>
> This suggests to me that user-created schemas have some fairly high
> likelihood of being clobbered.
>
> We changed the packages of many things. We can claim these were internal
> things, but there are many schemas that depend on Daffodil internal things
> now. This needs to get fixed so we can torque the internals more without
> breaking things, but right now... things are broken.
>
> Every schema using layers, for example as we never hardened the API for
> layers, so as it has evolved, schemas have broken.
>
>
>
> On Mon, Jun 5, 2023 at 6:07 PM Mike Beckerle <mb...@apache.org> wrote:
>
>>
>> All critical and blocker type bugs have been fixed including a couple
>> quite challenging ones in the last few weeks.
>>
>> I think it is time to create an RC for 3.5.0.
>>
>> Please discuss through 6pm EDT.US on Thursday June 8. (72 hours) and if
>> favorable we'll figure out who gets to create the RC.
>>
>> -mikeb
>>
>> Mike Beckerle
>> Apache Daffodil PMC | daffodil.apache.org
>> OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl
>> Owl Cyber Defense | www.owlcyberdefense.com
>>
>>
>>
>
Re: [DISCUSS] time to create daffodil 3.5.0 release?
Posted by Steve Lawrence <sl...@apache.org>.
I'm currently going through the regression suite and making changes to
get all the tests to pass.
I think one problem is many of these schemas haven't been update in a
real long time, in some cases since daffodil 2.6.0. Some of the things
that have been removed went through a reasonable deprecation process and
are very minor changes now hat they've been removed.
I'll send a report of all the changes that I find to get the schemas
working with 3.5.0 and see if that helps determine if 4.0.0 is more
appropriate or 3.5.0 is fine.
On 2023-06-06 07:42 PM, Mike Beckerle wrote:
> Topic: Should this be version 4.0.0, not 3.5.0?
>
> The OGSI-package-name-changes have broken many schemas because the test
> rigs used Daffodil internals, and some deprecated things were finally
> removed in the current 3.5.0 snapshot.
>
> But as a result, these schemas, many of which are non-public, but
> nevertheless can be counted, a long list, which used to work, don't work
> any more.
> It's possible there are false-negatives here. Nevertheless it is a long
> list.
>
> [fail] (04/62) dfdl-bmp
> [fail] (11/62) dfdl-examples-hexwords
> [fail] (12/62) dfdl-examples-self-descriptive-data
> [fail] (13/62) dfdl-examples-xslt-csv
> [fail] (22/62) dfdl-imf-icalendar-cse
> [fail] (23/62) dfdl-ipf-demo
> [fail] (29/62) dfdl-jpeg2000
> [fail] (30/62) dfdl-jreap
> [fail] (34/62) dfdl-mil-std-2045
> [fail] (38/62) dfdl-nitf
> [fail] (42/62) dfdl-plc4x-s7
> [fail] (43/62) dfdl-png
> [fail] (46/62) dfdl-raster
> [fail] (49/62) dfdl-stanag-5516
> [fail] (50/62) dfdl-stanag-5516-with-nact
> [fail] (53/62) dfdl-usmtf
> [fail] (54/62) dfdl-usmtf-generic
> [fail] (57/62) dfdl-vmf
> [fail] (59/62) dfdl-vmf-spock
> [fail] (61/62) ibm-dfdl-crosstester
>
> This suggests to me that user-created schemas have some fairly high
> likelihood of being clobbered.
>
> We changed the packages of many things. We can claim these were internal
> things, but there are many schemas that depend on Daffodil internal things
> now. This needs to get fixed so we can torque the internals more without
> breaking things, but right now... things are broken.
>
> Every schema using layers, for example as we never hardened the API for
> layers, so as it has evolved, schemas have broken.
>
>
>
> On Mon, Jun 5, 2023 at 6:07 PM Mike Beckerle <mb...@apache.org> wrote:
>
>>
>> All critical and blocker type bugs have been fixed including a couple
>> quite challenging ones in the last few weeks.
>>
>> I think it is time to create an RC for 3.5.0.
>>
>> Please discuss through 6pm EDT.US on Thursday June 8. (72 hours) and if
>> favorable we'll figure out who gets to create the RC.
>>
>> -mikeb
>>
>> Mike Beckerle
>> Apache Daffodil PMC | daffodil.apache.org
>> OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl
>> Owl Cyber Defense | www.owlcyberdefense.com
>>
>>
>>
>
Re: [DISCUSS] time to create daffodil 3.5.0 release?
Posted by Mike Beckerle <mb...@apache.org>.
Topic: Should this be version 4.0.0, not 3.5.0?
The OGSI-package-name-changes have broken many schemas because the test
rigs used Daffodil internals, and some deprecated things were finally
removed in the current 3.5.0 snapshot.
But as a result, these schemas, many of which are non-public, but
nevertheless can be counted, a long list, which used to work, don't work
any more.
It's possible there are false-negatives here. Nevertheless it is a long
list.
[fail] (04/62) dfdl-bmp
[fail] (11/62) dfdl-examples-hexwords
[fail] (12/62) dfdl-examples-self-descriptive-data
[fail] (13/62) dfdl-examples-xslt-csv
[fail] (22/62) dfdl-imf-icalendar-cse
[fail] (23/62) dfdl-ipf-demo
[fail] (29/62) dfdl-jpeg2000
[fail] (30/62) dfdl-jreap
[fail] (34/62) dfdl-mil-std-2045
[fail] (38/62) dfdl-nitf
[fail] (42/62) dfdl-plc4x-s7
[fail] (43/62) dfdl-png
[fail] (46/62) dfdl-raster
[fail] (49/62) dfdl-stanag-5516
[fail] (50/62) dfdl-stanag-5516-with-nact
[fail] (53/62) dfdl-usmtf
[fail] (54/62) dfdl-usmtf-generic
[fail] (57/62) dfdl-vmf
[fail] (59/62) dfdl-vmf-spock
[fail] (61/62) ibm-dfdl-crosstester
This suggests to me that user-created schemas have some fairly high
likelihood of being clobbered.
We changed the packages of many things. We can claim these were internal
things, but there are many schemas that depend on Daffodil internal things
now. This needs to get fixed so we can torque the internals more without
breaking things, but right now... things are broken.
Every schema using layers, for example as we never hardened the API for
layers, so as it has evolved, schemas have broken.
On Mon, Jun 5, 2023 at 6:07 PM Mike Beckerle <mb...@apache.org> wrote:
>
> All critical and blocker type bugs have been fixed including a couple
> quite challenging ones in the last few weeks.
>
> I think it is time to create an RC for 3.5.0.
>
> Please discuss through 6pm EDT.US on Thursday June 8. (72 hours) and if
> favorable we'll figure out who gets to create the RC.
>
> -mikeb
>
> Mike Beckerle
> Apache Daffodil PMC | daffodil.apache.org
> OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl
> Owl Cyber Defense | www.owlcyberdefense.com
>
>
>