You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@daffodil.apache.org by Mike Beckerle <mb...@apache.org> on 2022/03/02 19:12:29 UTC
Re: [DISCUSS] need to release Daffodil 3.3.0
I've marked all the alignment/cyclic-deadlock regressions as blockers for
3.3.0 along with the "hammer" to
just turn off alignment.
The fixing suggested in the thread here may be the fix, or the "hammer"
fix, but the regressions on unparsing have to be addressed in 3.3.0, i.e.,
asap, before we can release it.
I think other things we "almost" got working, like prefixed length fixes
(of various bugs) could wait for a later release.
There are numerous user projects I know about that are depending on 3.3.0
coming out quite soon now, without regressions.
On Wed, Feb 23, 2022 at 11:19 AM Steve Lawrence <sl...@apache.org>
wrote:
> I assume this is caused by alignment regions not getting optimized out
> with the recent changes to the alignment algorithm. It's now more
> correct, but it's more pessimistic.
>
> A hammer to just disable alignment might be a reasonable solution, but
> I'd be concerned there are alignment regions that are needed, it's not
> usually obvious, especially in complex schemas.
>
> I think the main change that causes regions to fail to optimize out is
> that we can't optimize out alignment related to global declarations
> because we don't know the alignment of the references.
>
> I added comments in https://issues.apache.org/jira/browse/DAFFODIL-2626
> that discuss this issue, and a potential fixe. I believe we just need to
> require that alignment of global decl's to be the same as their
> references. I hope that this would allow more optimization of alignment
> regions. One issue was raised about global complexType's, who's
> alignment only comes from the references, with no information on the
> declaration. So that also causes issues with this approach.
>
> I think implementing one or both of these options as tunables might help
> improve the alignment issue and would be reasonable to get in 3.3.0.
>
>
> On 2/23/22 11:08 AM, Mike Beckerle wrote:
> > So, we seem to be seeing a lot of regressions in various DFDL schemas
> like
> > most recently NITF, previously PNG.
> >
> > What if users run into this in their own DFDL schemas?
> >
> > These are showing unparser deadlocks due to cyclic relationships. At one
> > time we discussed adding a "big hammer" property or tunable that simply
> > turns off alignment, as a workaround for all these sorts of alignment
> > issues. I am wondering if we will need that so that users can work around
> > these alignment issues in their schemas.
> >
> > Changing these schemas for 3.3.0 compatibility is highly undesirable (as
> > was done for PNG), even if the changes are backward compatible.
> >
> > (Though if the schemas are actually incorrect in some way that we're now
> > detecting more effectively, that is the right fix.)
> >
> >
> >
> > On Tue, Feb 22, 2022 at 11:38 AM Interrante, John A (GE Research, US) <
> > John.Interrante@ge.com> wrote:
> >
> >> +1
> >>
> >> I personally have no blocker or urgent issues that must be fixed before
> >> the next release (only some things I will need to start working on in
> the C
> >> backend, "Runtime 2," to handle some more complicated schemas).
> >>
> >> How will the roadmap for upcoming releases (
> >>
> https://cwiki.apache.org/confluence/display/DAFFODIL/Roadmap+for+Upcoming+Releases
> )
> >> change as a result of 3.3.0 being released asap?
> >>
> >> John
> >>
> >> -----Original Message-----
> >> From: Mike Beckerle <mb...@apache.org>
> >> Sent: Tuesday, February 22, 2022 10:44 AM
> >> To: dev@daffodil.apache.org
> >> Subject: EXT: [DISCUSS] need to release Daffodil 3.3.0
> >>
> >> WARNING: This email originated from outside of GE. Please validate the
> >> sender's email address before clicking on links or attachments as they
> may
> >> not be safe.
> >>
> >> A number of people are asking for 3.3.0, with its many bug fixes, to be
> >> released asap.
> >>
> >> Are there any remaining issues that must be fixed before this release?
> >>
> >> Otherwise I'd like to suggest we release 3.3.0.
> >>
> >
>
>
Re: [DISCUSS] need to release Daffodil 3.3.0
Posted by Steve Lawrence <sl...@apache.org>.
It looks like your "interran" account has access. Not sure which one you
use, but I also added your "jinterrante" account as an admin.
On 3/18/22 12:31 PM, Interrante, John A (GE Research, US) wrote:
> Steve, I think the same JIRA problem (no permission) exists on Confluence too. I've looked carefully but I can't find an Edit link in the Release Workflow page:
>
> https://cwiki.apache.org/confluence/display/DAFFODIL/Release+Workflow
>
> John
>
> P.S. I apologize for addressing only PMC members in my vote email. Everyone is welcome to vote, not only PMC members. (I forgot that there are non-PMC members on the dev list.)
>
> -----Original Message-----
> From: Steve Lawrence <sl...@apache.org>
> Sent: Friday, March 18, 2022 4:41 AM
> To: dev@daffodil.apache.org
> Subject: EXT: Re: [DISCUSS] need to release Daffodil 3.3.0
>
> WARNING: This email originated from outside of GE. Please validate the sender's email address before clicking on links or attachments as they may not be safe.
>
> I think I've added you (and all other PMC members) to the appropriate roles to manage Jira. You should be able to bulk edit and add new releases now.
>
> Yes, that release notes page looks like a much better approach to creating the site information. Could you update the release workflow with how to get to this page? I can't seem to find it.
>
> Also, just noticed maybe even better: if you click Configure Release Notes and select "Text", the "Edit/Copy Release Notes" box at the bottom has something almost perfect for the website. Just needs a tweak to use the %jira macro.
>
>
> On 3/17/22 10:03 PM, Interrante, John A (GE Research, US) wrote:
>> I don't think I have enough permission to move issues in JIRA or even create a new unreleased version. The Bulk Edit tool won't let me move issues (it says I don't have permission). I poked around in the JIRA UI and found the list of releases (https://issues.apache.org/jira/projects/DAFFODIL?selectedItem=com.atlassian.jira.jira-projects-plugin:release-page&status=released-unreleased), but I can't find a way in the UI to create 3.4.0 as a new unreleased version or mark 3.3.0 as a released version.
>>
>> Incidentally, I found a way to generate release notes automatically using JIRA itself. Go to https://issues.apache.org/jira/projects/DAFFODIL/versions/12350900 and click the "Release Notes" link below the red "Create" button. Then you end up with a nice list of issues (https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12321622&version=12350900) which looks almost perfect (unfortunately the list includes 8 unresolved issues that need to be moved to 3.4.0 once 3.4.0 is created and available for use). I think we can simplify the release workflow by using this "Release Notes", copying the list of issues, editing the issue numbers to {% jira XXXX %} format, and editing the category titles.
>>
>> John
>>
>> -----Original Message-----
>> From: Interrante, John A (GE Research, US) <Jo...@ge.com>
>> Sent: Thursday, March 17, 2022 10:19 AM
>> To: mbeckerle@apache.org
>> Cc: dev@daffodil.apache.org
>> Subject: EXT: [DISCUSS] need to release Daffodil 3.3.0
>>
>> Please review and approve Prepare for Release 3.3.0 by tuxji · Pull Request #775 · apache/daffodil (github.com)<https://github.com/apache/daffodil/pull/775>.
>>
>> Also, please help improve the release notes by reviewing Prepare for Release 3.3.0 by tuxji · Pull Request #82 · apache/daffodil-site (github.com)<https://github.com/apache/daffodil-site/pull/82>.
>>
>> From: Mike Beckerle <mb...@apache.org>
>> Sent: Thursday, March 17, 2022 6:54 AM
>> To: Interrante, John A (GE Research, US) <Jo...@ge.com>
>> Cc: dev@daffodil.apache.org
>> Subject: EXT: Re: [DISCUSS] need to release Daffodil 3.3.0
>>
>> Nobody else has stepped forward, so I think you "win the prize" for this release John.
>>
>> On Tue, Mar 15, 2022 at 3:57 PM Interrante, John A (GE Research, US) <Jo...@ge.com>> wrote:
>> I can volunteer to be release manager if no one else wants to and you don't need the rc soon (I won't have time to prepare a rc until Thursday evening and it's my first time so it would take more time).
>>
>> John
>>
>> -----Original Message-----
>> From: Mike Beckerle
>> <mb...@apache.org>>
>> Sent: Tuesday, March 15, 2022 1:59 PM
>> To: dev@daffodil.apache.org<ma...@daffodil.apache.org>
>> Subject: EXT: Re: [DISCUSS] need to release Daffodil 3.3.0
>>
>> Ok, all the critical issues for 3.3.0 are fixed now.
>>
>> Any volunteers for release manager?
>>
>> On Mon, Mar 14, 2022 at 7:24 PM Mike Beckerle <mb...@apache.org>> wrote:
>>
>>> This was an easy fix.
>>>
>>> Once this gets merged I believe we have direction to move forward
>>> with
>>> 3.3.0 release process.
>>>
>>> Who would like to be release manager?
>>>
>>>
>>> On Mon, Mar 14, 2022 at 6:06 PM Mike Beckerle
>>> <mb...@apache.org>>
>>> wrote:
>>>
>>>> So the 3 blocking issues have been fixed/merged.
>>>>
>>>> However, of several bugs found over the last few days, this one is
>>>> quite
>>>> problematic:
>>>>
>>>> https://issues.apache.org/jira/browse/DAFFODIL-2673
>>>>
>>>> I am going to try to fix this in the next day, and if it proves to
>>>> be harder than that, postpone it until post 3.3.0.
>>>>
>>>>
>>>>
>>>> On Fri, Mar 11, 2022 at 10:37 AM Steve Lawrence
>>>> <sl...@apache.org>>
>>>> wrote:
>>>>
>>>>> Agreed. +1 to start the release processes once these are resolved.
>>>>>
>>>>> On 3/10/22 6:05 PM, Interrante, John A (GE Research, US) wrote:
>>>>>> Fixing these 3 tickets seems sufficient to me.
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Mike Beckerle
>>>>>> <mb...@apache.org>>
>>>>>> Sent: Wednesday, March 9, 2022 6:00 PM
>>>>>> To: dev@daffodil.apache.org<ma...@daffodil.apache.org>
>>>>>> Subject: EXT: Re: [DISCUSS] need to release Daffodil 3.3.0
>>>>>>
>>>>>> Once the revert/fix for regressions is merged, I think we're down
>>>>>> to
>>>>> just these 3 tickets as really critical for Daffodil 3.3.0 release:
>>>>>>
>>>>>> DAFFODIL-2652
>>>>>> <https://issues.apache.org/jira/browse/DAFFODIL-2652> -
>>>>> Ability to disable all alignment
>>>>>>
>>>>>> Given the number of outstanding bugs associated with unparser
>>>>>> deadlock
>>>>> and alignment I think this feature is an important hedge allowing
>>>>> progress to be made by schema authors even if they run into these
>>>>> unparser/alignment related issues (like DAFFODIL-2662 or
>>>>> DAFFODIL-2666 which are hard to fix, and I think we don't want to
>>>>> hold back the release for those fixes because they will take a
>>>>> while.)
>>>>>>
>>>>>> DAFFODIL-2650
>>>>>> <https://issues.apache.org/jira/browse/DAFFODIL-2650> -
>>>>> using config file with cli parse or save parser causes backtrace
>>>>>>
>>>>>> Major usability issue when dealing with DFDL schemas that issue
>>>>>> many
>>>>> warnings.
>>>>>>
>>>>>> DAFFODIL-2267
>>>>>> <https://issues.apache.org/jira/browse/DAFFODIL-2267> -
>>>>> Warnings emitted on pre-compiled parsers
>>>>>>
>>>>>> Major usability issue when dealing with DFDL schemas that issue
>>>>>> many
>>>>> warnings in deployed Daffodil-based applications. Clutters the log
>>>>> with too many things users have to know can be ignored.
>>>>>>
>>>>>> I will say these latter 2 bugs have been a huge pain in the neck
>>>>>> to me
>>>>> of late in debugging efforts associated with some DFDL schema work.
>>>>> They just so clutter the output that you really can't see what is
>>>>> going on sometimes.
>>>>>>
>>>>>> Thoughts? Is fixing these enough for us to do a release of 3.3.0 ?
>>>>>>
>>>>>> On Wed, Mar 9, 2022 at 2:06 PM Mike Beckerle
>>>>>> <mb...@apache.org>>
>>>>> wrote:
>>>>>>
>>>>>>> I just opened a PR which reverts a change which fixed a bug
>>>>>>> (DAFFODIL-2626), but caused a number of regressions detected only
>>>>>>> by other DFDL schemas such as NITF. (DAFFODIL-2666 and
>>>>>>> DAFFODIL-2662 are regressions it caused.)
>>>>>>>
>>>>>>> The original bug is preferable to these regressions.
>>>>>>>
>>>>>>> This will get us closer to a releasable 3.3.0.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Mar 2, 2022 at 2:12 PM Mike Beckerle
>>>>>>> <mb...@apache.org>>
>>>>> wrote:
>>>>>>>
>>>>>>>> I've marked all the alignment/cyclic-deadlock regressions as
>>>>>>>> blockers for
>>>>>>>> 3.3.0 along with the "hammer" to just turn off alignment.
>>>>>>>>
>>>>>>>> The fixing suggested in the thread here may be the fix, or the
>>>>> "hammer"
>>>>>>>> fix, but the regressions on unparsing have to be addressed in
>>>>>>>> 3.3.0, i.e., asap, before we can release it.
>>>>>>>>
>>>>>>>> I think other things we "almost" got working, like prefixed
>>>>>>>> length fixes (of various bugs) could wait for a later release.
>>>>>>>>
>>>>>>>> There are numerous user projects I know about that are depending
>>>>>>>> on
>>>>>>>> 3.3.0 coming out quite soon now, without regressions.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Feb 23, 2022 at 11:19 AM Steve Lawrence
>>>>>>>> <sl...@apache.org>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I assume this is caused by alignment regions not getting
>>>>>>>>> optimized out with the recent changes to the alignment
>>>>>>>>> algorithm. It's now more correct, but it's more pessimistic.
>>>>>>>>>
>>>>>>>>> A hammer to just disable alignment might be a reasonable
>>>>>>>>> solution, but I'd be concerned there are alignment regions that
>>>>>>>>> are needed, it's not usually obvious, especially in complex schemas.
>>>>>>>>>
>>>>>>>>> I think the main change that causes regions to fail to optimize
>>>>>>>>> out is that we can't optimize out alignment related to global
>>>>>>>>> declarations because we don't know the alignment of the references.
>>>>>>>>>
>>>>>>>>> I added comments in
>>>>>>>>> https://issues.apache.org/jira/browse/DAFFODIL-2626
>>>>>>>>> that discuss this issue, and a potential fixe. I believe we
>>>>>>>>> just need to require that alignment of global decl's to be the
>>>>>>>>> same as their references. I hope that this would allow more
>>>>>>>>> optimization of alignment regions. One issue was raised about
>>>>>>>>> global complexType's, who's alignment only comes from the
>>>>>>>>> references, with no information on the declaration. So that also causes issues with this approach.
>>>>>>>>>
>>>>>>>>> I think implementing one or both of these options as tunables
>>>>>>>>> might help improve the alignment issue and would be reasonable
>>>>>>>>> to get in
>>>>> 3.3.0.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 2/23/22 11:08 AM, Mike Beckerle wrote:
>>>>>>>>>> So, we seem to be seeing a lot of regressions in various DFDL
>>>>>>>>>> schemas
>>>>>>>>> like
>>>>>>>>>> most recently NITF, previously PNG.
>>>>>>>>>>
>>>>>>>>>> What if users run into this in their own DFDL schemas?
>>>>>>>>>>
>>>>>>>>>> These are showing unparser deadlocks due to cyclic relationships.
>>>>>>>>>> At
>>>>>>>>> one
>>>>>>>>>> time we discussed adding a "big hammer" property or tunable
>>>>>>>>>> that simply turns off alignment, as a workaround for all these
>>>>>>>>>> sorts of alignment issues. I am wondering if we will need that
>>>>>>>>>> so that users can work
>>>>>>>>> around
>>>>>>>>>> these alignment issues in their schemas.
>>>>>>>>>>
>>>>>>>>>> Changing these schemas for 3.3.0 compatibility is highly
>>>>>>>>>> undesirable
>>>>>>>>> (as
>>>>>>>>>> was done for PNG), even if the changes are backward compatible.
>>>>>>>>>>
>>>>>>>>>> (Though if the schemas are actually incorrect in some way that
>>>>>>>>>> we're
>>>>>>>>> now
>>>>>>>>>> detecting more effectively, that is the right fix.)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Feb 22, 2022 at 11:38 AM Interrante, John A (GE
>>>>>>>>>> Research,
>>>>>>>>>> US) < John.Interrante@ge.com<ma...@ge.com>> wrote:
>>>>>>>>>>
>>>>>>>>>>> +1
>>>>>>>>>>>
>>>>>>>>>>> I personally have no blocker or urgent issues that must be
>>>>>>>>>>> fixed
>>>>>>>>> before
>>>>>>>>>>> the next release (only some things I will need to start
>>>>>>>>>>> working on in
>>>>>>>>> the C
>>>>>>>>>>> backend, "Runtime 2," to handle some more complicated schemas).
>>>>>>>>>>>
>>>>>>>>>>> How will the roadmap for upcoming releases (
>>>>>>>>>>>
>>>>>>>>>
>>>>> https://cwiki.apache.org/confluence/display/DAFFODIL/Roadmap+for+Up
>>>>> c
>>>>>>>>> oming+Releases
>>>>>>>>> )
>>>>>>>>>>> change as a result of 3.3.0 being released asap?
>>>>>>>>>>>
>>>>>>>>>>> John
>>>>>>>>>>>
>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>> From: Mike Beckerle
>>>>>>>>>>> <mb...@apache.org>>
>>>>>>>>>>> Sent: Tuesday, February 22, 2022 10:44 AM
>>>>>>>>>>> To: dev@daffodil.apache.org<ma...@daffodil.apache.org>
>>>>>>>>>>> Subject: EXT: [DISCUSS] need to release Daffodil 3.3.0
>>>>>>>>>>>
>>>>>>>>>>> WARNING: This email originated from outside of GE. Please
>>>>>>>>>>> validate the sender's email address before clicking on links
>>>>>>>>>>> or attachments as
>>>>>>>>> they may
>>>>>>>>>>> not be safe.
>>>>>>>>>>>
>>>>>>>>>>> A number of people are asking for 3.3.0, with its many bug
>>>>>>>>>>> fixes, to
>>>>>>>>> be
>>>>>>>>>>> released asap.
>>>>>>>>>>>
>>>>>>>>>>> Are there any remaining issues that must be fixed before this
>>>>> release?
>>>>>>>>>>>
>>>>>>>>>>> Otherwise I'd like to suggest we release 3.3.0.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>
>>>>>
>
[DISCUSS] need to release Daffodil 3.3.0
Posted by "Interrante, John A (GE Research, US)" <Jo...@ge.com>.
Steve, I think the same JIRA problem (no permission) exists on Confluence too. I've looked carefully but I can't find an Edit link in the Release Workflow page:
https://cwiki.apache.org/confluence/display/DAFFODIL/Release+Workflow
John
P.S. I apologize for addressing only PMC members in my vote email. Everyone is welcome to vote, not only PMC members. (I forgot that there are non-PMC members on the dev list.)
-----Original Message-----
From: Steve Lawrence <sl...@apache.org>
Sent: Friday, March 18, 2022 4:41 AM
To: dev@daffodil.apache.org
Subject: EXT: Re: [DISCUSS] need to release Daffodil 3.3.0
WARNING: This email originated from outside of GE. Please validate the sender's email address before clicking on links or attachments as they may not be safe.
I think I've added you (and all other PMC members) to the appropriate roles to manage Jira. You should be able to bulk edit and add new releases now.
Yes, that release notes page looks like a much better approach to creating the site information. Could you update the release workflow with how to get to this page? I can't seem to find it.
Also, just noticed maybe even better: if you click Configure Release Notes and select "Text", the "Edit/Copy Release Notes" box at the bottom has something almost perfect for the website. Just needs a tweak to use the %jira macro.
On 3/17/22 10:03 PM, Interrante, John A (GE Research, US) wrote:
> I don't think I have enough permission to move issues in JIRA or even create a new unreleased version. The Bulk Edit tool won't let me move issues (it says I don't have permission). I poked around in the JIRA UI and found the list of releases (https://issues.apache.org/jira/projects/DAFFODIL?selectedItem=com.atlassian.jira.jira-projects-plugin:release-page&status=released-unreleased), but I can't find a way in the UI to create 3.4.0 as a new unreleased version or mark 3.3.0 as a released version.
>
> Incidentally, I found a way to generate release notes automatically using JIRA itself. Go to https://issues.apache.org/jira/projects/DAFFODIL/versions/12350900 and click the "Release Notes" link below the red "Create" button. Then you end up with a nice list of issues (https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12321622&version=12350900) which looks almost perfect (unfortunately the list includes 8 unresolved issues that need to be moved to 3.4.0 once 3.4.0 is created and available for use). I think we can simplify the release workflow by using this "Release Notes", copying the list of issues, editing the issue numbers to {% jira XXXX %} format, and editing the category titles.
>
> John
>
> -----Original Message-----
> From: Interrante, John A (GE Research, US) <Jo...@ge.com>
> Sent: Thursday, March 17, 2022 10:19 AM
> To: mbeckerle@apache.org
> Cc: dev@daffodil.apache.org
> Subject: EXT: [DISCUSS] need to release Daffodil 3.3.0
>
> Please review and approve Prepare for Release 3.3.0 by tuxji · Pull Request #775 · apache/daffodil (github.com)<https://github.com/apache/daffodil/pull/775>.
>
> Also, please help improve the release notes by reviewing Prepare for Release 3.3.0 by tuxji · Pull Request #82 · apache/daffodil-site (github.com)<https://github.com/apache/daffodil-site/pull/82>.
>
> From: Mike Beckerle <mb...@apache.org>
> Sent: Thursday, March 17, 2022 6:54 AM
> To: Interrante, John A (GE Research, US) <Jo...@ge.com>
> Cc: dev@daffodil.apache.org
> Subject: EXT: Re: [DISCUSS] need to release Daffodil 3.3.0
>
> Nobody else has stepped forward, so I think you "win the prize" for this release John.
>
> On Tue, Mar 15, 2022 at 3:57 PM Interrante, John A (GE Research, US) <Jo...@ge.com>> wrote:
> I can volunteer to be release manager if no one else wants to and you don't need the rc soon (I won't have time to prepare a rc until Thursday evening and it's my first time so it would take more time).
>
> John
>
> -----Original Message-----
> From: Mike Beckerle
> <mb...@apache.org>>
> Sent: Tuesday, March 15, 2022 1:59 PM
> To: dev@daffodil.apache.org<ma...@daffodil.apache.org>
> Subject: EXT: Re: [DISCUSS] need to release Daffodil 3.3.0
>
> Ok, all the critical issues for 3.3.0 are fixed now.
>
> Any volunteers for release manager?
>
> On Mon, Mar 14, 2022 at 7:24 PM Mike Beckerle <mb...@apache.org>> wrote:
>
>> This was an easy fix.
>>
>> Once this gets merged I believe we have direction to move forward
>> with
>> 3.3.0 release process.
>>
>> Who would like to be release manager?
>>
>>
>> On Mon, Mar 14, 2022 at 6:06 PM Mike Beckerle
>> <mb...@apache.org>>
>> wrote:
>>
>>> So the 3 blocking issues have been fixed/merged.
>>>
>>> However, of several bugs found over the last few days, this one is
>>> quite
>>> problematic:
>>>
>>> https://issues.apache.org/jira/browse/DAFFODIL-2673
>>>
>>> I am going to try to fix this in the next day, and if it proves to
>>> be harder than that, postpone it until post 3.3.0.
>>>
>>>
>>>
>>> On Fri, Mar 11, 2022 at 10:37 AM Steve Lawrence
>>> <sl...@apache.org>>
>>> wrote:
>>>
>>>> Agreed. +1 to start the release processes once these are resolved.
>>>>
>>>> On 3/10/22 6:05 PM, Interrante, John A (GE Research, US) wrote:
>>>>> Fixing these 3 tickets seems sufficient to me.
>>>>>
>>>>> -----Original Message-----
>>>>> From: Mike Beckerle
>>>>> <mb...@apache.org>>
>>>>> Sent: Wednesday, March 9, 2022 6:00 PM
>>>>> To: dev@daffodil.apache.org<ma...@daffodil.apache.org>
>>>>> Subject: EXT: Re: [DISCUSS] need to release Daffodil 3.3.0
>>>>>
>>>>> Once the revert/fix for regressions is merged, I think we're down
>>>>> to
>>>> just these 3 tickets as really critical for Daffodil 3.3.0 release:
>>>>>
>>>>> DAFFODIL-2652
>>>>> <https://issues.apache.org/jira/browse/DAFFODIL-2652> -
>>>> Ability to disable all alignment
>>>>>
>>>>> Given the number of outstanding bugs associated with unparser
>>>>> deadlock
>>>> and alignment I think this feature is an important hedge allowing
>>>> progress to be made by schema authors even if they run into these
>>>> unparser/alignment related issues (like DAFFODIL-2662 or
>>>> DAFFODIL-2666 which are hard to fix, and I think we don't want to
>>>> hold back the release for those fixes because they will take a
>>>> while.)
>>>>>
>>>>> DAFFODIL-2650
>>>>> <https://issues.apache.org/jira/browse/DAFFODIL-2650> -
>>>> using config file with cli parse or save parser causes backtrace
>>>>>
>>>>> Major usability issue when dealing with DFDL schemas that issue
>>>>> many
>>>> warnings.
>>>>>
>>>>> DAFFODIL-2267
>>>>> <https://issues.apache.org/jira/browse/DAFFODIL-2267> -
>>>> Warnings emitted on pre-compiled parsers
>>>>>
>>>>> Major usability issue when dealing with DFDL schemas that issue
>>>>> many
>>>> warnings in deployed Daffodil-based applications. Clutters the log
>>>> with too many things users have to know can be ignored.
>>>>>
>>>>> I will say these latter 2 bugs have been a huge pain in the neck
>>>>> to me
>>>> of late in debugging efforts associated with some DFDL schema work.
>>>> They just so clutter the output that you really can't see what is
>>>> going on sometimes.
>>>>>
>>>>> Thoughts? Is fixing these enough for us to do a release of 3.3.0 ?
>>>>>
>>>>> On Wed, Mar 9, 2022 at 2:06 PM Mike Beckerle
>>>>> <mb...@apache.org>>
>>>> wrote:
>>>>>
>>>>>> I just opened a PR which reverts a change which fixed a bug
>>>>>> (DAFFODIL-2626), but caused a number of regressions detected only
>>>>>> by other DFDL schemas such as NITF. (DAFFODIL-2666 and
>>>>>> DAFFODIL-2662 are regressions it caused.)
>>>>>>
>>>>>> The original bug is preferable to these regressions.
>>>>>>
>>>>>> This will get us closer to a releasable 3.3.0.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Mar 2, 2022 at 2:12 PM Mike Beckerle
>>>>>> <mb...@apache.org>>
>>>> wrote:
>>>>>>
>>>>>>> I've marked all the alignment/cyclic-deadlock regressions as
>>>>>>> blockers for
>>>>>>> 3.3.0 along with the "hammer" to just turn off alignment.
>>>>>>>
>>>>>>> The fixing suggested in the thread here may be the fix, or the
>>>> "hammer"
>>>>>>> fix, but the regressions on unparsing have to be addressed in
>>>>>>> 3.3.0, i.e., asap, before we can release it.
>>>>>>>
>>>>>>> I think other things we "almost" got working, like prefixed
>>>>>>> length fixes (of various bugs) could wait for a later release.
>>>>>>>
>>>>>>> There are numerous user projects I know about that are depending
>>>>>>> on
>>>>>>> 3.3.0 coming out quite soon now, without regressions.
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Feb 23, 2022 at 11:19 AM Steve Lawrence
>>>>>>> <sl...@apache.org>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I assume this is caused by alignment regions not getting
>>>>>>>> optimized out with the recent changes to the alignment
>>>>>>>> algorithm. It's now more correct, but it's more pessimistic.
>>>>>>>>
>>>>>>>> A hammer to just disable alignment might be a reasonable
>>>>>>>> solution, but I'd be concerned there are alignment regions that
>>>>>>>> are needed, it's not usually obvious, especially in complex schemas.
>>>>>>>>
>>>>>>>> I think the main change that causes regions to fail to optimize
>>>>>>>> out is that we can't optimize out alignment related to global
>>>>>>>> declarations because we don't know the alignment of the references.
>>>>>>>>
>>>>>>>> I added comments in
>>>>>>>> https://issues.apache.org/jira/browse/DAFFODIL-2626
>>>>>>>> that discuss this issue, and a potential fixe. I believe we
>>>>>>>> just need to require that alignment of global decl's to be the
>>>>>>>> same as their references. I hope that this would allow more
>>>>>>>> optimization of alignment regions. One issue was raised about
>>>>>>>> global complexType's, who's alignment only comes from the
>>>>>>>> references, with no information on the declaration. So that also causes issues with this approach.
>>>>>>>>
>>>>>>>> I think implementing one or both of these options as tunables
>>>>>>>> might help improve the alignment issue and would be reasonable
>>>>>>>> to get in
>>>> 3.3.0.
>>>>>>>>
>>>>>>>>
>>>>>>>> On 2/23/22 11:08 AM, Mike Beckerle wrote:
>>>>>>>>> So, we seem to be seeing a lot of regressions in various DFDL
>>>>>>>>> schemas
>>>>>>>> like
>>>>>>>>> most recently NITF, previously PNG.
>>>>>>>>>
>>>>>>>>> What if users run into this in their own DFDL schemas?
>>>>>>>>>
>>>>>>>>> These are showing unparser deadlocks due to cyclic relationships.
>>>>>>>>> At
>>>>>>>> one
>>>>>>>>> time we discussed adding a "big hammer" property or tunable
>>>>>>>>> that simply turns off alignment, as a workaround for all these
>>>>>>>>> sorts of alignment issues. I am wondering if we will need that
>>>>>>>>> so that users can work
>>>>>>>> around
>>>>>>>>> these alignment issues in their schemas.
>>>>>>>>>
>>>>>>>>> Changing these schemas for 3.3.0 compatibility is highly
>>>>>>>>> undesirable
>>>>>>>> (as
>>>>>>>>> was done for PNG), even if the changes are backward compatible.
>>>>>>>>>
>>>>>>>>> (Though if the schemas are actually incorrect in some way that
>>>>>>>>> we're
>>>>>>>> now
>>>>>>>>> detecting more effectively, that is the right fix.)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Feb 22, 2022 at 11:38 AM Interrante, John A (GE
>>>>>>>>> Research,
>>>>>>>>> US) < John.Interrante@ge.com<ma...@ge.com>> wrote:
>>>>>>>>>
>>>>>>>>>> +1
>>>>>>>>>>
>>>>>>>>>> I personally have no blocker or urgent issues that must be
>>>>>>>>>> fixed
>>>>>>>> before
>>>>>>>>>> the next release (only some things I will need to start
>>>>>>>>>> working on in
>>>>>>>> the C
>>>>>>>>>> backend, "Runtime 2," to handle some more complicated schemas).
>>>>>>>>>>
>>>>>>>>>> How will the roadmap for upcoming releases (
>>>>>>>>>>
>>>>>>>>
>>>> https://cwiki.apache.org/confluence/display/DAFFODIL/Roadmap+for+Up
>>>> c
>>>>>>>> oming+Releases
>>>>>>>> )
>>>>>>>>>> change as a result of 3.3.0 being released asap?
>>>>>>>>>>
>>>>>>>>>> John
>>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Mike Beckerle
>>>>>>>>>> <mb...@apache.org>>
>>>>>>>>>> Sent: Tuesday, February 22, 2022 10:44 AM
>>>>>>>>>> To: dev@daffodil.apache.org<ma...@daffodil.apache.org>
>>>>>>>>>> Subject: EXT: [DISCUSS] need to release Daffodil 3.3.0
>>>>>>>>>>
>>>>>>>>>> WARNING: This email originated from outside of GE. Please
>>>>>>>>>> validate the sender's email address before clicking on links
>>>>>>>>>> or attachments as
>>>>>>>> they may
>>>>>>>>>> not be safe.
>>>>>>>>>>
>>>>>>>>>> A number of people are asking for 3.3.0, with its many bug
>>>>>>>>>> fixes, to
>>>>>>>> be
>>>>>>>>>> released asap.
>>>>>>>>>>
>>>>>>>>>> Are there any remaining issues that must be fixed before this
>>>> release?
>>>>>>>>>>
>>>>>>>>>> Otherwise I'd like to suggest we release 3.3.0.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>
>>>>
Re: [DISCUSS] need to release Daffodil 3.3.0
Posted by Steve Lawrence <sl...@apache.org>.
I think I've added you (and all other PMC members) to the appropriate
roles to manage Jira. You should be able to bulk edit and add new
releases now.
Yes, that release notes page looks like a much better approach to
creating the site information. Could you update the release workflow
with how to get to this page? I can't seem to find it.
Also, just noticed maybe even better: if you click Configure Release
Notes and select "Text", the "Edit/Copy Release Notes" box at the bottom
has something almost perfect for the website. Just needs a tweak to use
the %jira macro.
On 3/17/22 10:03 PM, Interrante, John A (GE Research, US) wrote:
> I don't think I have enough permission to move issues in JIRA or even create a new unreleased version. The Bulk Edit tool won't let me move issues (it says I don't have permission). I poked around in the JIRA UI and found the list of releases (https://issues.apache.org/jira/projects/DAFFODIL?selectedItem=com.atlassian.jira.jira-projects-plugin:release-page&status=released-unreleased), but I can't find a way in the UI to create 3.4.0 as a new unreleased version or mark 3.3.0 as a released version.
>
> Incidentally, I found a way to generate release notes automatically using JIRA itself. Go to https://issues.apache.org/jira/projects/DAFFODIL/versions/12350900 and click the "Release Notes" link below the red "Create" button. Then you end up with a nice list of issues (https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12321622&version=12350900) which looks almost perfect (unfortunately the list includes 8 unresolved issues that need to be moved to 3.4.0 once 3.4.0 is created and available for use). I think we can simplify the release workflow by using this "Release Notes", copying the list of issues, editing the issue numbers to {% jira XXXX %} format, and editing the category titles.
>
> John
>
> -----Original Message-----
> From: Interrante, John A (GE Research, US) <Jo...@ge.com>
> Sent: Thursday, March 17, 2022 10:19 AM
> To: mbeckerle@apache.org
> Cc: dev@daffodil.apache.org
> Subject: EXT: [DISCUSS] need to release Daffodil 3.3.0
>
> Please review and approve Prepare for Release 3.3.0 by tuxji · Pull Request #775 · apache/daffodil (github.com)<https://github.com/apache/daffodil/pull/775>.
>
> Also, please help improve the release notes by reviewing Prepare for Release 3.3.0 by tuxji · Pull Request #82 · apache/daffodil-site (github.com)<https://github.com/apache/daffodil-site/pull/82>.
>
> From: Mike Beckerle <mb...@apache.org>
> Sent: Thursday, March 17, 2022 6:54 AM
> To: Interrante, John A (GE Research, US) <Jo...@ge.com>
> Cc: dev@daffodil.apache.org
> Subject: EXT: Re: [DISCUSS] need to release Daffodil 3.3.0
>
> Nobody else has stepped forward, so I think you "win the prize" for this release John.
>
> On Tue, Mar 15, 2022 at 3:57 PM Interrante, John A (GE Research, US) <Jo...@ge.com>> wrote:
> I can volunteer to be release manager if no one else wants to and you don't need the rc soon (I won't have time to prepare a rc until Thursday evening and it's my first time so it would take more time).
>
> John
>
> -----Original Message-----
> From: Mike Beckerle <mb...@apache.org>>
> Sent: Tuesday, March 15, 2022 1:59 PM
> To: dev@daffodil.apache.org<ma...@daffodil.apache.org>
> Subject: EXT: Re: [DISCUSS] need to release Daffodil 3.3.0
>
> Ok, all the critical issues for 3.3.0 are fixed now.
>
> Any volunteers for release manager?
>
> On Mon, Mar 14, 2022 at 7:24 PM Mike Beckerle <mb...@apache.org>> wrote:
>
>> This was an easy fix.
>>
>> Once this gets merged I believe we have direction to move forward with
>> 3.3.0 release process.
>>
>> Who would like to be release manager?
>>
>>
>> On Mon, Mar 14, 2022 at 6:06 PM Mike Beckerle
>> <mb...@apache.org>>
>> wrote:
>>
>>> So the 3 blocking issues have been fixed/merged.
>>>
>>> However, of several bugs found over the last few days, this one is
>>> quite
>>> problematic:
>>>
>>> https://issues.apache.org/jira/browse/DAFFODIL-2673
>>>
>>> I am going to try to fix this in the next day, and if it proves to be
>>> harder than that, postpone it until post 3.3.0.
>>>
>>>
>>>
>>> On Fri, Mar 11, 2022 at 10:37 AM Steve Lawrence
>>> <sl...@apache.org>>
>>> wrote:
>>>
>>>> Agreed. +1 to start the release processes once these are resolved.
>>>>
>>>> On 3/10/22 6:05 PM, Interrante, John A (GE Research, US) wrote:
>>>>> Fixing these 3 tickets seems sufficient to me.
>>>>>
>>>>> -----Original Message-----
>>>>> From: Mike Beckerle
>>>>> <mb...@apache.org>>
>>>>> Sent: Wednesday, March 9, 2022 6:00 PM
>>>>> To: dev@daffodil.apache.org<ma...@daffodil.apache.org>
>>>>> Subject: EXT: Re: [DISCUSS] need to release Daffodil 3.3.0
>>>>>
>>>>> Once the revert/fix for regressions is merged, I think we're down
>>>>> to
>>>> just these 3 tickets as really critical for Daffodil 3.3.0 release:
>>>>>
>>>>> DAFFODIL-2652
>>>>> <https://issues.apache.org/jira/browse/DAFFODIL-2652> -
>>>> Ability to disable all alignment
>>>>>
>>>>> Given the number of outstanding bugs associated with unparser
>>>>> deadlock
>>>> and alignment I think this feature is an important hedge allowing
>>>> progress to be made by schema authors even if they run into these
>>>> unparser/alignment related issues (like DAFFODIL-2662 or
>>>> DAFFODIL-2666 which are hard to fix, and I think we don't want to
>>>> hold back the release for those fixes because they will take a
>>>> while.)
>>>>>
>>>>> DAFFODIL-2650
>>>>> <https://issues.apache.org/jira/browse/DAFFODIL-2650> -
>>>> using config file with cli parse or save parser causes backtrace
>>>>>
>>>>> Major usability issue when dealing with DFDL schemas that issue
>>>>> many
>>>> warnings.
>>>>>
>>>>> DAFFODIL-2267
>>>>> <https://issues.apache.org/jira/browse/DAFFODIL-2267> -
>>>> Warnings emitted on pre-compiled parsers
>>>>>
>>>>> Major usability issue when dealing with DFDL schemas that issue
>>>>> many
>>>> warnings in deployed Daffodil-based applications. Clutters the log
>>>> with too many things users have to know can be ignored.
>>>>>
>>>>> I will say these latter 2 bugs have been a huge pain in the neck
>>>>> to me
>>>> of late in debugging efforts associated with some DFDL schema work.
>>>> They just so clutter the output that you really can't see what is
>>>> going on sometimes.
>>>>>
>>>>> Thoughts? Is fixing these enough for us to do a release of 3.3.0 ?
>>>>>
>>>>> On Wed, Mar 9, 2022 at 2:06 PM Mike Beckerle
>>>>> <mb...@apache.org>>
>>>> wrote:
>>>>>
>>>>>> I just opened a PR which reverts a change which fixed a bug
>>>>>> (DAFFODIL-2626), but caused a number of regressions detected only
>>>>>> by other DFDL schemas such as NITF. (DAFFODIL-2666 and
>>>>>> DAFFODIL-2662 are regressions it caused.)
>>>>>>
>>>>>> The original bug is preferable to these regressions.
>>>>>>
>>>>>> This will get us closer to a releasable 3.3.0.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Mar 2, 2022 at 2:12 PM Mike Beckerle
>>>>>> <mb...@apache.org>>
>>>> wrote:
>>>>>>
>>>>>>> I've marked all the alignment/cyclic-deadlock regressions as
>>>>>>> blockers for
>>>>>>> 3.3.0 along with the "hammer" to just turn off alignment.
>>>>>>>
>>>>>>> The fixing suggested in the thread here may be the fix, or the
>>>> "hammer"
>>>>>>> fix, but the regressions on unparsing have to be addressed in
>>>>>>> 3.3.0, i.e., asap, before we can release it.
>>>>>>>
>>>>>>> I think other things we "almost" got working, like prefixed
>>>>>>> length fixes (of various bugs) could wait for a later release.
>>>>>>>
>>>>>>> There are numerous user projects I know about that are depending
>>>>>>> on
>>>>>>> 3.3.0 coming out quite soon now, without regressions.
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Feb 23, 2022 at 11:19 AM Steve Lawrence
>>>>>>> <sl...@apache.org>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I assume this is caused by alignment regions not getting
>>>>>>>> optimized out with the recent changes to the alignment
>>>>>>>> algorithm. It's now more correct, but it's more pessimistic.
>>>>>>>>
>>>>>>>> A hammer to just disable alignment might be a reasonable
>>>>>>>> solution, but I'd be concerned there are alignment regions that
>>>>>>>> are needed, it's not usually obvious, especially in complex schemas.
>>>>>>>>
>>>>>>>> I think the main change that causes regions to fail to optimize
>>>>>>>> out is that we can't optimize out alignment related to global
>>>>>>>> declarations because we don't know the alignment of the references.
>>>>>>>>
>>>>>>>> I added comments in
>>>>>>>> https://issues.apache.org/jira/browse/DAFFODIL-2626
>>>>>>>> that discuss this issue, and a potential fixe. I believe we
>>>>>>>> just need to require that alignment of global decl's to be the
>>>>>>>> same as their references. I hope that this would allow more
>>>>>>>> optimization of alignment regions. One issue was raised about
>>>>>>>> global complexType's, who's alignment only comes from the
>>>>>>>> references, with no information on the declaration. So that also causes issues with this approach.
>>>>>>>>
>>>>>>>> I think implementing one or both of these options as tunables
>>>>>>>> might help improve the alignment issue and would be reasonable
>>>>>>>> to get in
>>>> 3.3.0.
>>>>>>>>
>>>>>>>>
>>>>>>>> On 2/23/22 11:08 AM, Mike Beckerle wrote:
>>>>>>>>> So, we seem to be seeing a lot of regressions in various DFDL
>>>>>>>>> schemas
>>>>>>>> like
>>>>>>>>> most recently NITF, previously PNG.
>>>>>>>>>
>>>>>>>>> What if users run into this in their own DFDL schemas?
>>>>>>>>>
>>>>>>>>> These are showing unparser deadlocks due to cyclic relationships.
>>>>>>>>> At
>>>>>>>> one
>>>>>>>>> time we discussed adding a "big hammer" property or tunable
>>>>>>>>> that simply turns off alignment, as a workaround for all these
>>>>>>>>> sorts of alignment issues. I am wondering if we will need that
>>>>>>>>> so that users can work
>>>>>>>> around
>>>>>>>>> these alignment issues in their schemas.
>>>>>>>>>
>>>>>>>>> Changing these schemas for 3.3.0 compatibility is highly
>>>>>>>>> undesirable
>>>>>>>> (as
>>>>>>>>> was done for PNG), even if the changes are backward compatible.
>>>>>>>>>
>>>>>>>>> (Though if the schemas are actually incorrect in some way that
>>>>>>>>> we're
>>>>>>>> now
>>>>>>>>> detecting more effectively, that is the right fix.)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Feb 22, 2022 at 11:38 AM Interrante, John A (GE
>>>>>>>>> Research,
>>>>>>>>> US) < John.Interrante@ge.com<ma...@ge.com>> wrote:
>>>>>>>>>
>>>>>>>>>> +1
>>>>>>>>>>
>>>>>>>>>> I personally have no blocker or urgent issues that must be
>>>>>>>>>> fixed
>>>>>>>> before
>>>>>>>>>> the next release (only some things I will need to start
>>>>>>>>>> working on in
>>>>>>>> the C
>>>>>>>>>> backend, "Runtime 2," to handle some more complicated schemas).
>>>>>>>>>>
>>>>>>>>>> How will the roadmap for upcoming releases (
>>>>>>>>>>
>>>>>>>>
>>>> https://cwiki.apache.org/confluence/display/DAFFODIL/Roadmap+for+Upc
>>>>>>>> oming+Releases
>>>>>>>> )
>>>>>>>>>> change as a result of 3.3.0 being released asap?
>>>>>>>>>>
>>>>>>>>>> John
>>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Mike Beckerle
>>>>>>>>>> <mb...@apache.org>>
>>>>>>>>>> Sent: Tuesday, February 22, 2022 10:44 AM
>>>>>>>>>> To: dev@daffodil.apache.org<ma...@daffodil.apache.org>
>>>>>>>>>> Subject: EXT: [DISCUSS] need to release Daffodil 3.3.0
>>>>>>>>>>
>>>>>>>>>> WARNING: This email originated from outside of GE. Please
>>>>>>>>>> validate the sender's email address before clicking on links
>>>>>>>>>> or attachments as
>>>>>>>> they may
>>>>>>>>>> not be safe.
>>>>>>>>>>
>>>>>>>>>> A number of people are asking for 3.3.0, with its many bug
>>>>>>>>>> fixes, to
>>>>>>>> be
>>>>>>>>>> released asap.
>>>>>>>>>>
>>>>>>>>>> Are there any remaining issues that must be fixed before this
>>>> release?
>>>>>>>>>>
>>>>>>>>>> Otherwise I'd like to suggest we release 3.3.0.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>
>>>>
RE: [DISCUSS] need to release Daffodil 3.3.0
Posted by "Interrante, John A (GE Research, US)" <Jo...@ge.com>.
I don't think I have enough permission to move issues in JIRA or even create a new unreleased version. The Bulk Edit tool won't let me move issues (it says I don't have permission). I poked around in the JIRA UI and found the list of releases (https://issues.apache.org/jira/projects/DAFFODIL?selectedItem=com.atlassian.jira.jira-projects-plugin:release-page&status=released-unreleased), but I can't find a way in the UI to create 3.4.0 as a new unreleased version or mark 3.3.0 as a released version.
Incidentally, I found a way to generate release notes automatically using JIRA itself. Go to https://issues.apache.org/jira/projects/DAFFODIL/versions/12350900 and click the "Release Notes" link below the red "Create" button. Then you end up with a nice list of issues (https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12321622&version=12350900) which looks almost perfect (unfortunately the list includes 8 unresolved issues that need to be moved to 3.4.0 once 3.4.0 is created and available for use). I think we can simplify the release workflow by using this "Release Notes", copying the list of issues, editing the issue numbers to {% jira XXXX %} format, and editing the category titles.
John
-----Original Message-----
From: Interrante, John A (GE Research, US) <Jo...@ge.com>
Sent: Thursday, March 17, 2022 10:19 AM
To: mbeckerle@apache.org
Cc: dev@daffodil.apache.org
Subject: EXT: [DISCUSS] need to release Daffodil 3.3.0
Please review and approve Prepare for Release 3.3.0 by tuxji · Pull Request #775 · apache/daffodil (github.com)<https://github.com/apache/daffodil/pull/775>.
Also, please help improve the release notes by reviewing Prepare for Release 3.3.0 by tuxji · Pull Request #82 · apache/daffodil-site (github.com)<https://github.com/apache/daffodil-site/pull/82>.
From: Mike Beckerle <mb...@apache.org>
Sent: Thursday, March 17, 2022 6:54 AM
To: Interrante, John A (GE Research, US) <Jo...@ge.com>
Cc: dev@daffodil.apache.org
Subject: EXT: Re: [DISCUSS] need to release Daffodil 3.3.0
Nobody else has stepped forward, so I think you "win the prize" for this release John.
On Tue, Mar 15, 2022 at 3:57 PM Interrante, John A (GE Research, US) <Jo...@ge.com>> wrote:
I can volunteer to be release manager if no one else wants to and you don't need the rc soon (I won't have time to prepare a rc until Thursday evening and it's my first time so it would take more time).
John
-----Original Message-----
From: Mike Beckerle <mb...@apache.org>>
Sent: Tuesday, March 15, 2022 1:59 PM
To: dev@daffodil.apache.org<ma...@daffodil.apache.org>
Subject: EXT: Re: [DISCUSS] need to release Daffodil 3.3.0
Ok, all the critical issues for 3.3.0 are fixed now.
Any volunteers for release manager?
On Mon, Mar 14, 2022 at 7:24 PM Mike Beckerle <mb...@apache.org>> wrote:
> This was an easy fix.
>
> Once this gets merged I believe we have direction to move forward with
> 3.3.0 release process.
>
> Who would like to be release manager?
>
>
> On Mon, Mar 14, 2022 at 6:06 PM Mike Beckerle
> <mb...@apache.org>>
> wrote:
>
>> So the 3 blocking issues have been fixed/merged.
>>
>> However, of several bugs found over the last few days, this one is
>> quite
>> problematic:
>>
>> https://issues.apache.org/jira/browse/DAFFODIL-2673
>>
>> I am going to try to fix this in the next day, and if it proves to be
>> harder than that, postpone it until post 3.3.0.
>>
>>
>>
>> On Fri, Mar 11, 2022 at 10:37 AM Steve Lawrence
>> <sl...@apache.org>>
>> wrote:
>>
>>> Agreed. +1 to start the release processes once these are resolved.
>>>
>>> On 3/10/22 6:05 PM, Interrante, John A (GE Research, US) wrote:
>>> > Fixing these 3 tickets seems sufficient to me.
>>> >
>>> > -----Original Message-----
>>> > From: Mike Beckerle
>>> > <mb...@apache.org>>
>>> > Sent: Wednesday, March 9, 2022 6:00 PM
>>> > To: dev@daffodil.apache.org<ma...@daffodil.apache.org>
>>> > Subject: EXT: Re: [DISCUSS] need to release Daffodil 3.3.0
>>> >
>>> > Once the revert/fix for regressions is merged, I think we're down
>>> > to
>>> just these 3 tickets as really critical for Daffodil 3.3.0 release:
>>> >
>>> > DAFFODIL-2652
>>> > <https://issues.apache.org/jira/browse/DAFFODIL-2652> -
>>> Ability to disable all alignment
>>> >
>>> > Given the number of outstanding bugs associated with unparser
>>> > deadlock
>>> and alignment I think this feature is an important hedge allowing
>>> progress to be made by schema authors even if they run into these
>>> unparser/alignment related issues (like DAFFODIL-2662 or
>>> DAFFODIL-2666 which are hard to fix, and I think we don't want to
>>> hold back the release for those fixes because they will take a
>>> while.)
>>> >
>>> > DAFFODIL-2650
>>> > <https://issues.apache.org/jira/browse/DAFFODIL-2650> -
>>> using config file with cli parse or save parser causes backtrace
>>> >
>>> > Major usability issue when dealing with DFDL schemas that issue
>>> > many
>>> warnings.
>>> >
>>> > DAFFODIL-2267
>>> > <https://issues.apache.org/jira/browse/DAFFODIL-2267> -
>>> Warnings emitted on pre-compiled parsers
>>> >
>>> > Major usability issue when dealing with DFDL schemas that issue
>>> > many
>>> warnings in deployed Daffodil-based applications. Clutters the log
>>> with too many things users have to know can be ignored.
>>> >
>>> > I will say these latter 2 bugs have been a huge pain in the neck
>>> > to me
>>> of late in debugging efforts associated with some DFDL schema work.
>>> They just so clutter the output that you really can't see what is
>>> going on sometimes.
>>> >
>>> > Thoughts? Is fixing these enough for us to do a release of 3.3.0 ?
>>> >
>>> > On Wed, Mar 9, 2022 at 2:06 PM Mike Beckerle
>>> > <mb...@apache.org>>
>>> wrote:
>>> >
>>> >> I just opened a PR which reverts a change which fixed a bug
>>> >> (DAFFODIL-2626), but caused a number of regressions detected only
>>> >> by other DFDL schemas such as NITF. (DAFFODIL-2666 and
>>> >> DAFFODIL-2662 are regressions it caused.)
>>> >>
>>> >> The original bug is preferable to these regressions.
>>> >>
>>> >> This will get us closer to a releasable 3.3.0.
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> On Wed, Mar 2, 2022 at 2:12 PM Mike Beckerle
>>> >> <mb...@apache.org>>
>>> wrote:
>>> >>
>>> >>> I've marked all the alignment/cyclic-deadlock regressions as
>>> >>> blockers for
>>> >>> 3.3.0 along with the "hammer" to just turn off alignment.
>>> >>>
>>> >>> The fixing suggested in the thread here may be the fix, or the
>>> "hammer"
>>> >>> fix, but the regressions on unparsing have to be addressed in
>>> >>> 3.3.0, i.e., asap, before we can release it.
>>> >>>
>>> >>> I think other things we "almost" got working, like prefixed
>>> >>> length fixes (of various bugs) could wait for a later release.
>>> >>>
>>> >>> There are numerous user projects I know about that are depending
>>> >>> on
>>> >>> 3.3.0 coming out quite soon now, without regressions.
>>> >>>
>>> >>>
>>> >>> On Wed, Feb 23, 2022 at 11:19 AM Steve Lawrence
>>> >>> <sl...@apache.org>>
>>> >>> wrote:
>>> >>>
>>> >>>> I assume this is caused by alignment regions not getting
>>> >>>> optimized out with the recent changes to the alignment
>>> >>>> algorithm. It's now more correct, but it's more pessimistic.
>>> >>>>
>>> >>>> A hammer to just disable alignment might be a reasonable
>>> >>>> solution, but I'd be concerned there are alignment regions that
>>> >>>> are needed, it's not usually obvious, especially in complex schemas.
>>> >>>>
>>> >>>> I think the main change that causes regions to fail to optimize
>>> >>>> out is that we can't optimize out alignment related to global
>>> >>>> declarations because we don't know the alignment of the references.
>>> >>>>
>>> >>>> I added comments in
>>> >>>> https://issues.apache.org/jira/browse/DAFFODIL-2626
>>> >>>> that discuss this issue, and a potential fixe. I believe we
>>> >>>> just need to require that alignment of global decl's to be the
>>> >>>> same as their references. I hope that this would allow more
>>> >>>> optimization of alignment regions. One issue was raised about
>>> >>>> global complexType's, who's alignment only comes from the
>>> >>>> references, with no information on the declaration. So that also causes issues with this approach.
>>> >>>>
>>> >>>> I think implementing one or both of these options as tunables
>>> >>>> might help improve the alignment issue and would be reasonable
>>> >>>> to get in
>>> 3.3.0.
>>> >>>>
>>> >>>>
>>> >>>> On 2/23/22 11:08 AM, Mike Beckerle wrote:
>>> >>>>> So, we seem to be seeing a lot of regressions in various DFDL
>>> >>>>> schemas
>>> >>>> like
>>> >>>>> most recently NITF, previously PNG.
>>> >>>>>
>>> >>>>> What if users run into this in their own DFDL schemas?
>>> >>>>>
>>> >>>>> These are showing unparser deadlocks due to cyclic relationships.
>>> >>>>> At
>>> >>>> one
>>> >>>>> time we discussed adding a "big hammer" property or tunable
>>> >>>>> that simply turns off alignment, as a workaround for all these
>>> >>>>> sorts of alignment issues. I am wondering if we will need that
>>> >>>>> so that users can work
>>> >>>> around
>>> >>>>> these alignment issues in their schemas.
>>> >>>>>
>>> >>>>> Changing these schemas for 3.3.0 compatibility is highly
>>> >>>>> undesirable
>>> >>>> (as
>>> >>>>> was done for PNG), even if the changes are backward compatible.
>>> >>>>>
>>> >>>>> (Though if the schemas are actually incorrect in some way that
>>> >>>>> we're
>>> >>>> now
>>> >>>>> detecting more effectively, that is the right fix.)
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> On Tue, Feb 22, 2022 at 11:38 AM Interrante, John A (GE
>>> >>>>> Research,
>>> >>>>> US) < John.Interrante@ge.com<ma...@ge.com>> wrote:
>>> >>>>>
>>> >>>>>> +1
>>> >>>>>>
>>> >>>>>> I personally have no blocker or urgent issues that must be
>>> >>>>>> fixed
>>> >>>> before
>>> >>>>>> the next release (only some things I will need to start
>>> >>>>>> working on in
>>> >>>> the C
>>> >>>>>> backend, "Runtime 2," to handle some more complicated schemas).
>>> >>>>>>
>>> >>>>>> How will the roadmap for upcoming releases (
>>> >>>>>>
>>> >>>>
>>> https://cwiki.apache.org/confluence/display/DAFFODIL/Roadmap+for+Upc
>>> >>>> oming+Releases
>>> >>>> )
>>> >>>>>> change as a result of 3.3.0 being released asap?
>>> >>>>>>
>>> >>>>>> John
>>> >>>>>>
>>> >>>>>> -----Original Message-----
>>> >>>>>> From: Mike Beckerle
>>> >>>>>> <mb...@apache.org>>
>>> >>>>>> Sent: Tuesday, February 22, 2022 10:44 AM
>>> >>>>>> To: dev@daffodil.apache.org<ma...@daffodil.apache.org>
>>> >>>>>> Subject: EXT: [DISCUSS] need to release Daffodil 3.3.0
>>> >>>>>>
>>> >>>>>> WARNING: This email originated from outside of GE. Please
>>> >>>>>> validate the sender's email address before clicking on links
>>> >>>>>> or attachments as
>>> >>>> they may
>>> >>>>>> not be safe.
>>> >>>>>>
>>> >>>>>> A number of people are asking for 3.3.0, with its many bug
>>> >>>>>> fixes, to
>>> >>>> be
>>> >>>>>> released asap.
>>> >>>>>>
>>> >>>>>> Are there any remaining issues that must be fixed before this
>>> release?
>>> >>>>>>
>>> >>>>>> Otherwise I'd like to suggest we release 3.3.0.
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>>>
>>>
>>>
[DISCUSS] need to release Daffodil 3.3.0
Posted by "Interrante, John A (GE Research, US)" <Jo...@ge.com>.
Please review and approve Prepare for Release 3.3.0 by tuxji · Pull Request #775 · apache/daffodil (github.com)<https://github.com/apache/daffodil/pull/775>.
Also, please help improve the release notes by reviewing Prepare for Release 3.3.0 by tuxji · Pull Request #82 · apache/daffodil-site (github.com)<https://github.com/apache/daffodil-site/pull/82>.
From: Mike Beckerle <mb...@apache.org>
Sent: Thursday, March 17, 2022 6:54 AM
To: Interrante, John A (GE Research, US) <Jo...@ge.com>
Cc: dev@daffodil.apache.org
Subject: EXT: Re: [DISCUSS] need to release Daffodil 3.3.0
WARNING: This email originated from outside of GE. Please validate the sender's email address before clicking on links or attachments as they may not be safe.
Nobody else has stepped forward, so I think you "win the prize" for this release John.
On Tue, Mar 15, 2022 at 3:57 PM Interrante, John A (GE Research, US) <Jo...@ge.com>> wrote:
I can volunteer to be release manager if no one else wants to and you don't need the rc soon (I won't have time to prepare a rc until Thursday evening and it's my first time so it would take more time).
John
-----Original Message-----
From: Mike Beckerle <mb...@apache.org>>
Sent: Tuesday, March 15, 2022 1:59 PM
To: dev@daffodil.apache.org<ma...@daffodil.apache.org>
Subject: EXT: Re: [DISCUSS] need to release Daffodil 3.3.0
WARNING: This email originated from outside of GE. Please validate the sender's email address before clicking on links or attachments as they may not be safe.
Ok, all the critical issues for 3.3.0 are fixed now.
Any volunteers for release manager?
On Mon, Mar 14, 2022 at 7:24 PM Mike Beckerle <mb...@apache.org>> wrote:
> This was an easy fix.
>
> Once this gets merged I believe we have direction to move forward with
> 3.3.0 release process.
>
> Who would like to be release manager?
>
>
> On Mon, Mar 14, 2022 at 6:06 PM Mike Beckerle <mb...@apache.org>>
> wrote:
>
>> So the 3 blocking issues have been fixed/merged.
>>
>> However, of several bugs found over the last few days, this one is
>> quite
>> problematic:
>>
>> https://issues.apache.org/jira/browse/DAFFODIL-2673
>>
>> I am going to try to fix this in the next day, and if it proves to be
>> harder than that, postpone it until post 3.3.0.
>>
>>
>>
>> On Fri, Mar 11, 2022 at 10:37 AM Steve Lawrence
>> <sl...@apache.org>>
>> wrote:
>>
>>> Agreed. +1 to start the release processes once these are resolved.
>>>
>>> On 3/10/22 6:05 PM, Interrante, John A (GE Research, US) wrote:
>>> > Fixing these 3 tickets seems sufficient to me.
>>> >
>>> > -----Original Message-----
>>> > From: Mike Beckerle <mb...@apache.org>>
>>> > Sent: Wednesday, March 9, 2022 6:00 PM
>>> > To: dev@daffodil.apache.org<ma...@daffodil.apache.org>
>>> > Subject: EXT: Re: [DISCUSS] need to release Daffodil 3.3.0
>>> >
>>> > Once the revert/fix for regressions is merged, I think we're down
>>> > to
>>> just these 3 tickets as really critical for Daffodil 3.3.0 release:
>>> >
>>> > DAFFODIL-2652
>>> > <https://issues.apache.org/jira/browse/DAFFODIL-2652> -
>>> Ability to disable all alignment
>>> >
>>> > Given the number of outstanding bugs associated with unparser
>>> > deadlock
>>> and alignment I think this feature is an important hedge allowing
>>> progress to be made by schema authors even if they run into these
>>> unparser/alignment related issues (like DAFFODIL-2662 or
>>> DAFFODIL-2666 which are hard to fix, and I think we don't want to
>>> hold back the release for those fixes because they will take a
>>> while.)
>>> >
>>> > DAFFODIL-2650
>>> > <https://issues.apache.org/jira/browse/DAFFODIL-2650> -
>>> using config file with cli parse or save parser causes backtrace
>>> >
>>> > Major usability issue when dealing with DFDL schemas that issue
>>> > many
>>> warnings.
>>> >
>>> > DAFFODIL-2267
>>> > <https://issues.apache.org/jira/browse/DAFFODIL-2267> -
>>> Warnings emitted on pre-compiled parsers
>>> >
>>> > Major usability issue when dealing with DFDL schemas that issue
>>> > many
>>> warnings in deployed Daffodil-based applications. Clutters the log
>>> with too many things users have to know can be ignored.
>>> >
>>> > I will say these latter 2 bugs have been a huge pain in the neck
>>> > to me
>>> of late in debugging efforts associated with some DFDL schema work.
>>> They just so clutter the output that you really can't see what is
>>> going on sometimes.
>>> >
>>> > Thoughts? Is fixing these enough for us to do a release of 3.3.0 ?
>>> >
>>> > On Wed, Mar 9, 2022 at 2:06 PM Mike Beckerle
>>> > <mb...@apache.org>>
>>> wrote:
>>> >
>>> >> I just opened a PR which reverts a change which fixed a bug
>>> >> (DAFFODIL-2626), but caused a number of regressions detected only
>>> >> by other DFDL schemas such as NITF. (DAFFODIL-2666 and
>>> >> DAFFODIL-2662 are regressions it caused.)
>>> >>
>>> >> The original bug is preferable to these regressions.
>>> >>
>>> >> This will get us closer to a releasable 3.3.0.
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> On Wed, Mar 2, 2022 at 2:12 PM Mike Beckerle
>>> >> <mb...@apache.org>>
>>> wrote:
>>> >>
>>> >>> I've marked all the alignment/cyclic-deadlock regressions as
>>> >>> blockers for
>>> >>> 3.3.0 along with the "hammer" to just turn off alignment.
>>> >>>
>>> >>> The fixing suggested in the thread here may be the fix, or the
>>> "hammer"
>>> >>> fix, but the regressions on unparsing have to be addressed in
>>> >>> 3.3.0, i.e., asap, before we can release it.
>>> >>>
>>> >>> I think other things we "almost" got working, like prefixed
>>> >>> length fixes (of various bugs) could wait for a later release.
>>> >>>
>>> >>> There are numerous user projects I know about that are depending
>>> >>> on
>>> >>> 3.3.0 coming out quite soon now, without regressions.
>>> >>>
>>> >>>
>>> >>> On Wed, Feb 23, 2022 at 11:19 AM Steve Lawrence
>>> >>> <sl...@apache.org>>
>>> >>> wrote:
>>> >>>
>>> >>>> I assume this is caused by alignment regions not getting
>>> >>>> optimized out with the recent changes to the alignment
>>> >>>> algorithm. It's now more correct, but it's more pessimistic.
>>> >>>>
>>> >>>> A hammer to just disable alignment might be a reasonable
>>> >>>> solution, but I'd be concerned there are alignment regions that
>>> >>>> are needed, it's not usually obvious, especially in complex schemas.
>>> >>>>
>>> >>>> I think the main change that causes regions to fail to optimize
>>> >>>> out is that we can't optimize out alignment related to global
>>> >>>> declarations because we don't know the alignment of the references.
>>> >>>>
>>> >>>> I added comments in
>>> >>>> https://issues.apache.org/jira/browse/DAFFODIL-2626
>>> >>>> that discuss this issue, and a potential fixe. I believe we
>>> >>>> just need to require that alignment of global decl's to be the
>>> >>>> same as their references. I hope that this would allow more
>>> >>>> optimization of alignment regions. One issue was raised about
>>> >>>> global complexType's, who's alignment only comes from the
>>> >>>> references, with no information on the declaration. So that also causes issues with this approach.
>>> >>>>
>>> >>>> I think implementing one or both of these options as tunables
>>> >>>> might help improve the alignment issue and would be reasonable
>>> >>>> to get in
>>> 3.3.0.
>>> >>>>
>>> >>>>
>>> >>>> On 2/23/22 11:08 AM, Mike Beckerle wrote:
>>> >>>>> So, we seem to be seeing a lot of regressions in various DFDL
>>> >>>>> schemas
>>> >>>> like
>>> >>>>> most recently NITF, previously PNG.
>>> >>>>>
>>> >>>>> What if users run into this in their own DFDL schemas?
>>> >>>>>
>>> >>>>> These are showing unparser deadlocks due to cyclic relationships.
>>> >>>>> At
>>> >>>> one
>>> >>>>> time we discussed adding a "big hammer" property or tunable
>>> >>>>> that simply turns off alignment, as a workaround for all these
>>> >>>>> sorts of alignment issues. I am wondering if we will need that
>>> >>>>> so that users can work
>>> >>>> around
>>> >>>>> these alignment issues in their schemas.
>>> >>>>>
>>> >>>>> Changing these schemas for 3.3.0 compatibility is highly
>>> >>>>> undesirable
>>> >>>> (as
>>> >>>>> was done for PNG), even if the changes are backward compatible.
>>> >>>>>
>>> >>>>> (Though if the schemas are actually incorrect in some way that
>>> >>>>> we're
>>> >>>> now
>>> >>>>> detecting more effectively, that is the right fix.)
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> On Tue, Feb 22, 2022 at 11:38 AM Interrante, John A (GE
>>> >>>>> Research,
>>> >>>>> US) < John.Interrante@ge.com<ma...@ge.com>> wrote:
>>> >>>>>
>>> >>>>>> +1
>>> >>>>>>
>>> >>>>>> I personally have no blocker or urgent issues that must be
>>> >>>>>> fixed
>>> >>>> before
>>> >>>>>> the next release (only some things I will need to start
>>> >>>>>> working on in
>>> >>>> the C
>>> >>>>>> backend, "Runtime 2," to handle some more complicated schemas).
>>> >>>>>>
>>> >>>>>> How will the roadmap for upcoming releases (
>>> >>>>>>
>>> >>>>
>>> https://cwiki.apache.org/confluence/display/DAFFODIL/Roadmap+for+Upc
>>> >>>> oming+Releases
>>> >>>> )
>>> >>>>>> change as a result of 3.3.0 being released asap?
>>> >>>>>>
>>> >>>>>> John
>>> >>>>>>
>>> >>>>>> -----Original Message-----
>>> >>>>>> From: Mike Beckerle <mb...@apache.org>>
>>> >>>>>> Sent: Tuesday, February 22, 2022 10:44 AM
>>> >>>>>> To: dev@daffodil.apache.org<ma...@daffodil.apache.org>
>>> >>>>>> Subject: EXT: [DISCUSS] need to release Daffodil 3.3.0
>>> >>>>>>
>>> >>>>>> WARNING: This email originated from outside of GE. Please
>>> >>>>>> validate the sender's email address before clicking on links
>>> >>>>>> or attachments as
>>> >>>> they may
>>> >>>>>> not be safe.
>>> >>>>>>
>>> >>>>>> A number of people are asking for 3.3.0, with its many bug
>>> >>>>>> fixes, to
>>> >>>> be
>>> >>>>>> released asap.
>>> >>>>>>
>>> >>>>>> Are there any remaining issues that must be fixed before this
>>> release?
>>> >>>>>>
>>> >>>>>> Otherwise I'd like to suggest we release 3.3.0.
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>>>
>>>
>>>
Re: [DISCUSS] need to release Daffodil 3.3.0
Posted by Mike Beckerle <mb...@apache.org>.
Nobody else has stepped forward, so I think you "win the prize" for this
release John.
On Tue, Mar 15, 2022 at 3:57 PM Interrante, John A (GE Research, US) <
John.Interrante@ge.com> wrote:
> I can volunteer to be release manager if no one else wants to and you
> don't need the rc soon (I won't have time to prepare a rc until Thursday
> evening and it's my first time so it would take more time).
>
> John
>
> -----Original Message-----
> From: Mike Beckerle <mb...@apache.org>
> Sent: Tuesday, March 15, 2022 1:59 PM
> To: dev@daffodil.apache.org
> Subject: EXT: Re: [DISCUSS] need to release Daffodil 3.3.0
>
> WARNING: This email originated from outside of GE. Please validate the
> sender's email address before clicking on links or attachments as they may
> not be safe.
>
> Ok, all the critical issues for 3.3.0 are fixed now.
>
> Any volunteers for release manager?
>
> On Mon, Mar 14, 2022 at 7:24 PM Mike Beckerle <mb...@apache.org>
> wrote:
>
> > This was an easy fix.
> >
> > Once this gets merged I believe we have direction to move forward with
> > 3.3.0 release process.
> >
> > Who would like to be release manager?
> >
> >
> > On Mon, Mar 14, 2022 at 6:06 PM Mike Beckerle <mb...@apache.org>
> > wrote:
> >
> >> So the 3 blocking issues have been fixed/merged.
> >>
> >> However, of several bugs found over the last few days, this one is
> >> quite
> >> problematic:
> >>
> >> https://issues.apache.org/jira/browse/DAFFODIL-2673
> >>
> >> I am going to try to fix this in the next day, and if it proves to be
> >> harder than that, postpone it until post 3.3.0.
> >>
> >>
> >>
> >> On Fri, Mar 11, 2022 at 10:37 AM Steve Lawrence
> >> <sl...@apache.org>
> >> wrote:
> >>
> >>> Agreed. +1 to start the release processes once these are resolved.
> >>>
> >>> On 3/10/22 6:05 PM, Interrante, John A (GE Research, US) wrote:
> >>> > Fixing these 3 tickets seems sufficient to me.
> >>> >
> >>> > -----Original Message-----
> >>> > From: Mike Beckerle <mb...@apache.org>
> >>> > Sent: Wednesday, March 9, 2022 6:00 PM
> >>> > To: dev@daffodil.apache.org
> >>> > Subject: EXT: Re: [DISCUSS] need to release Daffodil 3.3.0
> >>> >
> >>> > Once the revert/fix for regressions is merged, I think we're down
> >>> > to
> >>> just these 3 tickets as really critical for Daffodil 3.3.0 release:
> >>> >
> >>> > DAFFODIL-2652
> >>> > <https://issues.apache.org/jira/browse/DAFFODIL-2652> -
> >>> Ability to disable all alignment
> >>> >
> >>> > Given the number of outstanding bugs associated with unparser
> >>> > deadlock
> >>> and alignment I think this feature is an important hedge allowing
> >>> progress to be made by schema authors even if they run into these
> >>> unparser/alignment related issues (like DAFFODIL-2662 or
> >>> DAFFODIL-2666 which are hard to fix, and I think we don't want to
> >>> hold back the release for those fixes because they will take a
> >>> while.)
> >>> >
> >>> > DAFFODIL-2650
> >>> > <https://issues.apache.org/jira/browse/DAFFODIL-2650> -
> >>> using config file with cli parse or save parser causes backtrace
> >>> >
> >>> > Major usability issue when dealing with DFDL schemas that issue
> >>> > many
> >>> warnings.
> >>> >
> >>> > DAFFODIL-2267
> >>> > <https://issues.apache.org/jira/browse/DAFFODIL-2267> -
> >>> Warnings emitted on pre-compiled parsers
> >>> >
> >>> > Major usability issue when dealing with DFDL schemas that issue
> >>> > many
> >>> warnings in deployed Daffodil-based applications. Clutters the log
> >>> with too many things users have to know can be ignored.
> >>> >
> >>> > I will say these latter 2 bugs have been a huge pain in the neck
> >>> > to me
> >>> of late in debugging efforts associated with some DFDL schema work.
> >>> They just so clutter the output that you really can't see what is
> >>> going on sometimes.
> >>> >
> >>> > Thoughts? Is fixing these enough for us to do a release of 3.3.0 ?
> >>> >
> >>> > On Wed, Mar 9, 2022 at 2:06 PM Mike Beckerle
> >>> > <mb...@apache.org>
> >>> wrote:
> >>> >
> >>> >> I just opened a PR which reverts a change which fixed a bug
> >>> >> (DAFFODIL-2626), but caused a number of regressions detected only
> >>> >> by other DFDL schemas such as NITF. (DAFFODIL-2666 and
> >>> >> DAFFODIL-2662 are regressions it caused.)
> >>> >>
> >>> >> The original bug is preferable to these regressions.
> >>> >>
> >>> >> This will get us closer to a releasable 3.3.0.
> >>> >>
> >>> >>
> >>> >>
> >>> >>
> >>> >>
> >>> >> On Wed, Mar 2, 2022 at 2:12 PM Mike Beckerle
> >>> >> <mb...@apache.org>
> >>> wrote:
> >>> >>
> >>> >>> I've marked all the alignment/cyclic-deadlock regressions as
> >>> >>> blockers for
> >>> >>> 3.3.0 along with the "hammer" to just turn off alignment.
> >>> >>>
> >>> >>> The fixing suggested in the thread here may be the fix, or the
> >>> "hammer"
> >>> >>> fix, but the regressions on unparsing have to be addressed in
> >>> >>> 3.3.0, i.e., asap, before we can release it.
> >>> >>>
> >>> >>> I think other things we "almost" got working, like prefixed
> >>> >>> length fixes (of various bugs) could wait for a later release.
> >>> >>>
> >>> >>> There are numerous user projects I know about that are depending
> >>> >>> on
> >>> >>> 3.3.0 coming out quite soon now, without regressions.
> >>> >>>
> >>> >>>
> >>> >>> On Wed, Feb 23, 2022 at 11:19 AM Steve Lawrence
> >>> >>> <sl...@apache.org>
> >>> >>> wrote:
> >>> >>>
> >>> >>>> I assume this is caused by alignment regions not getting
> >>> >>>> optimized out with the recent changes to the alignment
> >>> >>>> algorithm. It's now more correct, but it's more pessimistic.
> >>> >>>>
> >>> >>>> A hammer to just disable alignment might be a reasonable
> >>> >>>> solution, but I'd be concerned there are alignment regions that
> >>> >>>> are needed, it's not usually obvious, especially in complex
> schemas.
> >>> >>>>
> >>> >>>> I think the main change that causes regions to fail to optimize
> >>> >>>> out is that we can't optimize out alignment related to global
> >>> >>>> declarations because we don't know the alignment of the
> references.
> >>> >>>>
> >>> >>>> I added comments in
> >>> >>>> https://issues.apache.org/jira/browse/DAFFODIL-2626
> >>> >>>> that discuss this issue, and a potential fixe. I believe we
> >>> >>>> just need to require that alignment of global decl's to be the
> >>> >>>> same as their references. I hope that this would allow more
> >>> >>>> optimization of alignment regions. One issue was raised about
> >>> >>>> global complexType's, who's alignment only comes from the
> >>> >>>> references, with no information on the declaration. So that also
> causes issues with this approach.
> >>> >>>>
> >>> >>>> I think implementing one or both of these options as tunables
> >>> >>>> might help improve the alignment issue and would be reasonable
> >>> >>>> to get in
> >>> 3.3.0.
> >>> >>>>
> >>> >>>>
> >>> >>>> On 2/23/22 11:08 AM, Mike Beckerle wrote:
> >>> >>>>> So, we seem to be seeing a lot of regressions in various DFDL
> >>> >>>>> schemas
> >>> >>>> like
> >>> >>>>> most recently NITF, previously PNG.
> >>> >>>>>
> >>> >>>>> What if users run into this in their own DFDL schemas?
> >>> >>>>>
> >>> >>>>> These are showing unparser deadlocks due to cyclic relationships.
> >>> >>>>> At
> >>> >>>> one
> >>> >>>>> time we discussed adding a "big hammer" property or tunable
> >>> >>>>> that simply turns off alignment, as a workaround for all these
> >>> >>>>> sorts of alignment issues. I am wondering if we will need that
> >>> >>>>> so that users can work
> >>> >>>> around
> >>> >>>>> these alignment issues in their schemas.
> >>> >>>>>
> >>> >>>>> Changing these schemas for 3.3.0 compatibility is highly
> >>> >>>>> undesirable
> >>> >>>> (as
> >>> >>>>> was done for PNG), even if the changes are backward compatible.
> >>> >>>>>
> >>> >>>>> (Though if the schemas are actually incorrect in some way that
> >>> >>>>> we're
> >>> >>>> now
> >>> >>>>> detecting more effectively, that is the right fix.)
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> >>> >>>>> On Tue, Feb 22, 2022 at 11:38 AM Interrante, John A (GE
> >>> >>>>> Research,
> >>> >>>>> US) < John.Interrante@ge.com> wrote:
> >>> >>>>>
> >>> >>>>>> +1
> >>> >>>>>>
> >>> >>>>>> I personally have no blocker or urgent issues that must be
> >>> >>>>>> fixed
> >>> >>>> before
> >>> >>>>>> the next release (only some things I will need to start
> >>> >>>>>> working on in
> >>> >>>> the C
> >>> >>>>>> backend, "Runtime 2," to handle some more complicated schemas).
> >>> >>>>>>
> >>> >>>>>> How will the roadmap for upcoming releases (
> >>> >>>>>>
> >>> >>>>
> >>> https://cwiki.apache.org/confluence/display/DAFFODIL/Roadmap+for+Upc
> >>> >>>> oming+Releases
> >>> >>>> )
> >>> >>>>>> change as a result of 3.3.0 being released asap?
> >>> >>>>>>
> >>> >>>>>> John
> >>> >>>>>>
> >>> >>>>>> -----Original Message-----
> >>> >>>>>> From: Mike Beckerle <mb...@apache.org>
> >>> >>>>>> Sent: Tuesday, February 22, 2022 10:44 AM
> >>> >>>>>> To: dev@daffodil.apache.org
> >>> >>>>>> Subject: EXT: [DISCUSS] need to release Daffodil 3.3.0
> >>> >>>>>>
> >>> >>>>>> WARNING: This email originated from outside of GE. Please
> >>> >>>>>> validate the sender's email address before clicking on links
> >>> >>>>>> or attachments as
> >>> >>>> they may
> >>> >>>>>> not be safe.
> >>> >>>>>>
> >>> >>>>>> A number of people are asking for 3.3.0, with its many bug
> >>> >>>>>> fixes, to
> >>> >>>> be
> >>> >>>>>> released asap.
> >>> >>>>>>
> >>> >>>>>> Are there any remaining issues that must be fixed before this
> >>> release?
> >>> >>>>>>
> >>> >>>>>> Otherwise I'd like to suggest we release 3.3.0.
> >>> >>>>>>
> >>> >>>>>
> >>> >>>>
> >>> >>>>
> >>>
> >>>
>
[DISCUSS] need to release Daffodil 3.3.0
Posted by "Interrante, John A (GE Research, US)" <Jo...@ge.com>.
I can volunteer to be release manager if no one else wants to and you don't need the rc soon (I won't have time to prepare a rc until Thursday evening and it's my first time so it would take more time).
John
-----Original Message-----
From: Mike Beckerle <mb...@apache.org>
Sent: Tuesday, March 15, 2022 1:59 PM
To: dev@daffodil.apache.org
Subject: EXT: Re: [DISCUSS] need to release Daffodil 3.3.0
WARNING: This email originated from outside of GE. Please validate the sender's email address before clicking on links or attachments as they may not be safe.
Ok, all the critical issues for 3.3.0 are fixed now.
Any volunteers for release manager?
On Mon, Mar 14, 2022 at 7:24 PM Mike Beckerle <mb...@apache.org> wrote:
> This was an easy fix.
>
> Once this gets merged I believe we have direction to move forward with
> 3.3.0 release process.
>
> Who would like to be release manager?
>
>
> On Mon, Mar 14, 2022 at 6:06 PM Mike Beckerle <mb...@apache.org>
> wrote:
>
>> So the 3 blocking issues have been fixed/merged.
>>
>> However, of several bugs found over the last few days, this one is
>> quite
>> problematic:
>>
>> https://issues.apache.org/jira/browse/DAFFODIL-2673
>>
>> I am going to try to fix this in the next day, and if it proves to be
>> harder than that, postpone it until post 3.3.0.
>>
>>
>>
>> On Fri, Mar 11, 2022 at 10:37 AM Steve Lawrence
>> <sl...@apache.org>
>> wrote:
>>
>>> Agreed. +1 to start the release processes once these are resolved.
>>>
>>> On 3/10/22 6:05 PM, Interrante, John A (GE Research, US) wrote:
>>> > Fixing these 3 tickets seems sufficient to me.
>>> >
>>> > -----Original Message-----
>>> > From: Mike Beckerle <mb...@apache.org>
>>> > Sent: Wednesday, March 9, 2022 6:00 PM
>>> > To: dev@daffodil.apache.org
>>> > Subject: EXT: Re: [DISCUSS] need to release Daffodil 3.3.0
>>> >
>>> > Once the revert/fix for regressions is merged, I think we're down
>>> > to
>>> just these 3 tickets as really critical for Daffodil 3.3.0 release:
>>> >
>>> > DAFFODIL-2652
>>> > <https://issues.apache.org/jira/browse/DAFFODIL-2652> -
>>> Ability to disable all alignment
>>> >
>>> > Given the number of outstanding bugs associated with unparser
>>> > deadlock
>>> and alignment I think this feature is an important hedge allowing
>>> progress to be made by schema authors even if they run into these
>>> unparser/alignment related issues (like DAFFODIL-2662 or
>>> DAFFODIL-2666 which are hard to fix, and I think we don't want to
>>> hold back the release for those fixes because they will take a
>>> while.)
>>> >
>>> > DAFFODIL-2650
>>> > <https://issues.apache.org/jira/browse/DAFFODIL-2650> -
>>> using config file with cli parse or save parser causes backtrace
>>> >
>>> > Major usability issue when dealing with DFDL schemas that issue
>>> > many
>>> warnings.
>>> >
>>> > DAFFODIL-2267
>>> > <https://issues.apache.org/jira/browse/DAFFODIL-2267> -
>>> Warnings emitted on pre-compiled parsers
>>> >
>>> > Major usability issue when dealing with DFDL schemas that issue
>>> > many
>>> warnings in deployed Daffodil-based applications. Clutters the log
>>> with too many things users have to know can be ignored.
>>> >
>>> > I will say these latter 2 bugs have been a huge pain in the neck
>>> > to me
>>> of late in debugging efforts associated with some DFDL schema work.
>>> They just so clutter the output that you really can't see what is
>>> going on sometimes.
>>> >
>>> > Thoughts? Is fixing these enough for us to do a release of 3.3.0 ?
>>> >
>>> > On Wed, Mar 9, 2022 at 2:06 PM Mike Beckerle
>>> > <mb...@apache.org>
>>> wrote:
>>> >
>>> >> I just opened a PR which reverts a change which fixed a bug
>>> >> (DAFFODIL-2626), but caused a number of regressions detected only
>>> >> by other DFDL schemas such as NITF. (DAFFODIL-2666 and
>>> >> DAFFODIL-2662 are regressions it caused.)
>>> >>
>>> >> The original bug is preferable to these regressions.
>>> >>
>>> >> This will get us closer to a releasable 3.3.0.
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> On Wed, Mar 2, 2022 at 2:12 PM Mike Beckerle
>>> >> <mb...@apache.org>
>>> wrote:
>>> >>
>>> >>> I've marked all the alignment/cyclic-deadlock regressions as
>>> >>> blockers for
>>> >>> 3.3.0 along with the "hammer" to just turn off alignment.
>>> >>>
>>> >>> The fixing suggested in the thread here may be the fix, or the
>>> "hammer"
>>> >>> fix, but the regressions on unparsing have to be addressed in
>>> >>> 3.3.0, i.e., asap, before we can release it.
>>> >>>
>>> >>> I think other things we "almost" got working, like prefixed
>>> >>> length fixes (of various bugs) could wait for a later release.
>>> >>>
>>> >>> There are numerous user projects I know about that are depending
>>> >>> on
>>> >>> 3.3.0 coming out quite soon now, without regressions.
>>> >>>
>>> >>>
>>> >>> On Wed, Feb 23, 2022 at 11:19 AM Steve Lawrence
>>> >>> <sl...@apache.org>
>>> >>> wrote:
>>> >>>
>>> >>>> I assume this is caused by alignment regions not getting
>>> >>>> optimized out with the recent changes to the alignment
>>> >>>> algorithm. It's now more correct, but it's more pessimistic.
>>> >>>>
>>> >>>> A hammer to just disable alignment might be a reasonable
>>> >>>> solution, but I'd be concerned there are alignment regions that
>>> >>>> are needed, it's not usually obvious, especially in complex schemas.
>>> >>>>
>>> >>>> I think the main change that causes regions to fail to optimize
>>> >>>> out is that we can't optimize out alignment related to global
>>> >>>> declarations because we don't know the alignment of the references.
>>> >>>>
>>> >>>> I added comments in
>>> >>>> https://issues.apache.org/jira/browse/DAFFODIL-2626
>>> >>>> that discuss this issue, and a potential fixe. I believe we
>>> >>>> just need to require that alignment of global decl's to be the
>>> >>>> same as their references. I hope that this would allow more
>>> >>>> optimization of alignment regions. One issue was raised about
>>> >>>> global complexType's, who's alignment only comes from the
>>> >>>> references, with no information on the declaration. So that also causes issues with this approach.
>>> >>>>
>>> >>>> I think implementing one or both of these options as tunables
>>> >>>> might help improve the alignment issue and would be reasonable
>>> >>>> to get in
>>> 3.3.0.
>>> >>>>
>>> >>>>
>>> >>>> On 2/23/22 11:08 AM, Mike Beckerle wrote:
>>> >>>>> So, we seem to be seeing a lot of regressions in various DFDL
>>> >>>>> schemas
>>> >>>> like
>>> >>>>> most recently NITF, previously PNG.
>>> >>>>>
>>> >>>>> What if users run into this in their own DFDL schemas?
>>> >>>>>
>>> >>>>> These are showing unparser deadlocks due to cyclic relationships.
>>> >>>>> At
>>> >>>> one
>>> >>>>> time we discussed adding a "big hammer" property or tunable
>>> >>>>> that simply turns off alignment, as a workaround for all these
>>> >>>>> sorts of alignment issues. I am wondering if we will need that
>>> >>>>> so that users can work
>>> >>>> around
>>> >>>>> these alignment issues in their schemas.
>>> >>>>>
>>> >>>>> Changing these schemas for 3.3.0 compatibility is highly
>>> >>>>> undesirable
>>> >>>> (as
>>> >>>>> was done for PNG), even if the changes are backward compatible.
>>> >>>>>
>>> >>>>> (Though if the schemas are actually incorrect in some way that
>>> >>>>> we're
>>> >>>> now
>>> >>>>> detecting more effectively, that is the right fix.)
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> On Tue, Feb 22, 2022 at 11:38 AM Interrante, John A (GE
>>> >>>>> Research,
>>> >>>>> US) < John.Interrante@ge.com> wrote:
>>> >>>>>
>>> >>>>>> +1
>>> >>>>>>
>>> >>>>>> I personally have no blocker or urgent issues that must be
>>> >>>>>> fixed
>>> >>>> before
>>> >>>>>> the next release (only some things I will need to start
>>> >>>>>> working on in
>>> >>>> the C
>>> >>>>>> backend, "Runtime 2," to handle some more complicated schemas).
>>> >>>>>>
>>> >>>>>> How will the roadmap for upcoming releases (
>>> >>>>>>
>>> >>>>
>>> https://cwiki.apache.org/confluence/display/DAFFODIL/Roadmap+for+Upc
>>> >>>> oming+Releases
>>> >>>> )
>>> >>>>>> change as a result of 3.3.0 being released asap?
>>> >>>>>>
>>> >>>>>> John
>>> >>>>>>
>>> >>>>>> -----Original Message-----
>>> >>>>>> From: Mike Beckerle <mb...@apache.org>
>>> >>>>>> Sent: Tuesday, February 22, 2022 10:44 AM
>>> >>>>>> To: dev@daffodil.apache.org
>>> >>>>>> Subject: EXT: [DISCUSS] need to release Daffodil 3.3.0
>>> >>>>>>
>>> >>>>>> WARNING: This email originated from outside of GE. Please
>>> >>>>>> validate the sender's email address before clicking on links
>>> >>>>>> or attachments as
>>> >>>> they may
>>> >>>>>> not be safe.
>>> >>>>>>
>>> >>>>>> A number of people are asking for 3.3.0, with its many bug
>>> >>>>>> fixes, to
>>> >>>> be
>>> >>>>>> released asap.
>>> >>>>>>
>>> >>>>>> Are there any remaining issues that must be fixed before this
>>> release?
>>> >>>>>>
>>> >>>>>> Otherwise I'd like to suggest we release 3.3.0.
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>>>
>>>
>>>
Re: [DISCUSS] need to release Daffodil 3.3.0
Posted by Mike Beckerle <mb...@apache.org>.
Ok, all the critical issues for 3.3.0 are fixed now.
Any volunteers for release manager?
On Mon, Mar 14, 2022 at 7:24 PM Mike Beckerle <mb...@apache.org> wrote:
> This was an easy fix.
>
> Once this gets merged I believe we have direction to move forward with
> 3.3.0 release process.
>
> Who would like to be release manager?
>
>
> On Mon, Mar 14, 2022 at 6:06 PM Mike Beckerle <mb...@apache.org>
> wrote:
>
>> So the 3 blocking issues have been fixed/merged.
>>
>> However, of several bugs found over the last few days, this one is quite
>> problematic:
>>
>> https://issues.apache.org/jira/browse/DAFFODIL-2673
>>
>> I am going to try to fix this in the next day, and if it proves to be
>> harder than that, postpone it until post 3.3.0.
>>
>>
>>
>> On Fri, Mar 11, 2022 at 10:37 AM Steve Lawrence <sl...@apache.org>
>> wrote:
>>
>>> Agreed. +1 to start the release processes once these are resolved.
>>>
>>> On 3/10/22 6:05 PM, Interrante, John A (GE Research, US) wrote:
>>> > Fixing these 3 tickets seems sufficient to me.
>>> >
>>> > -----Original Message-----
>>> > From: Mike Beckerle <mb...@apache.org>
>>> > Sent: Wednesday, March 9, 2022 6:00 PM
>>> > To: dev@daffodil.apache.org
>>> > Subject: EXT: Re: [DISCUSS] need to release Daffodil 3.3.0
>>> >
>>> > Once the revert/fix for regressions is merged, I think we're down to
>>> just these 3 tickets as really critical for Daffodil 3.3.0 release:
>>> >
>>> > DAFFODIL-2652 <https://issues.apache.org/jira/browse/DAFFODIL-2652> -
>>> Ability to disable all alignment
>>> >
>>> > Given the number of outstanding bugs associated with unparser deadlock
>>> and alignment I think this feature is an important hedge allowing progress
>>> to be made by schema authors even if they run into these unparser/alignment
>>> related issues (like DAFFODIL-2662 or DAFFODIL-2666 which are hard to fix,
>>> and I think we don't want to hold back the release for those fixes because
>>> they will take a while.)
>>> >
>>> > DAFFODIL-2650 <https://issues.apache.org/jira/browse/DAFFODIL-2650> -
>>> using config file with cli parse or save parser causes backtrace
>>> >
>>> > Major usability issue when dealing with DFDL schemas that issue many
>>> warnings.
>>> >
>>> > DAFFODIL-2267 <https://issues.apache.org/jira/browse/DAFFODIL-2267> -
>>> Warnings emitted on pre-compiled parsers
>>> >
>>> > Major usability issue when dealing with DFDL schemas that issue many
>>> warnings in deployed Daffodil-based applications. Clutters the log with too
>>> many things users have to know can be ignored.
>>> >
>>> > I will say these latter 2 bugs have been a huge pain in the neck to me
>>> of late in debugging efforts associated with some DFDL schema work. They
>>> just so clutter the output that you really can't see what is going on
>>> sometimes.
>>> >
>>> > Thoughts? Is fixing these enough for us to do a release of 3.3.0 ?
>>> >
>>> > On Wed, Mar 9, 2022 at 2:06 PM Mike Beckerle <mb...@apache.org>
>>> wrote:
>>> >
>>> >> I just opened a PR which reverts a change which fixed a bug
>>> >> (DAFFODIL-2626), but caused a number of regressions detected only by
>>> >> other DFDL schemas such as NITF. (DAFFODIL-2666 and DAFFODIL-2662 are
>>> >> regressions it caused.)
>>> >>
>>> >> The original bug is preferable to these regressions.
>>> >>
>>> >> This will get us closer to a releasable 3.3.0.
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> On Wed, Mar 2, 2022 at 2:12 PM Mike Beckerle <mb...@apache.org>
>>> wrote:
>>> >>
>>> >>> I've marked all the alignment/cyclic-deadlock regressions as blockers
>>> >>> for
>>> >>> 3.3.0 along with the "hammer" to
>>> >>> just turn off alignment.
>>> >>>
>>> >>> The fixing suggested in the thread here may be the fix, or the
>>> "hammer"
>>> >>> fix, but the regressions on unparsing have to be addressed in 3.3.0,
>>> >>> i.e., asap, before we can release it.
>>> >>>
>>> >>> I think other things we "almost" got working, like prefixed length
>>> >>> fixes (of various bugs) could wait for a later release.
>>> >>>
>>> >>> There are numerous user projects I know about that are depending on
>>> >>> 3.3.0 coming out quite soon now, without regressions.
>>> >>>
>>> >>>
>>> >>> On Wed, Feb 23, 2022 at 11:19 AM Steve Lawrence
>>> >>> <sl...@apache.org>
>>> >>> wrote:
>>> >>>
>>> >>>> I assume this is caused by alignment regions not getting optimized
>>> >>>> out with the recent changes to the alignment algorithm. It's now
>>> >>>> more correct, but it's more pessimistic.
>>> >>>>
>>> >>>> A hammer to just disable alignment might be a reasonable solution,
>>> >>>> but I'd be concerned there are alignment regions that are needed,
>>> >>>> it's not usually obvious, especially in complex schemas.
>>> >>>>
>>> >>>> I think the main change that causes regions to fail to optimize out
>>> >>>> is that we can't optimize out alignment related to global
>>> >>>> declarations because we don't know the alignment of the references.
>>> >>>>
>>> >>>> I added comments in
>>> >>>> https://issues.apache.org/jira/browse/DAFFODIL-2626
>>> >>>> that discuss this issue, and a potential fixe. I believe we just
>>> >>>> need to require that alignment of global decl's to be the same as
>>> >>>> their references. I hope that this would allow more optimization of
>>> >>>> alignment regions. One issue was raised about global complexType's,
>>> >>>> who's alignment only comes from the references, with no information
>>> >>>> on the declaration. So that also causes issues with this approach.
>>> >>>>
>>> >>>> I think implementing one or both of these options as tunables might
>>> >>>> help improve the alignment issue and would be reasonable to get in
>>> 3.3.0.
>>> >>>>
>>> >>>>
>>> >>>> On 2/23/22 11:08 AM, Mike Beckerle wrote:
>>> >>>>> So, we seem to be seeing a lot of regressions in various DFDL
>>> >>>>> schemas
>>> >>>> like
>>> >>>>> most recently NITF, previously PNG.
>>> >>>>>
>>> >>>>> What if users run into this in their own DFDL schemas?
>>> >>>>>
>>> >>>>> These are showing unparser deadlocks due to cyclic relationships.
>>> >>>>> At
>>> >>>> one
>>> >>>>> time we discussed adding a "big hammer" property or tunable that
>>> >>>>> simply turns off alignment, as a workaround for all these sorts of
>>> >>>>> alignment issues. I am wondering if we will need that so that
>>> >>>>> users can work
>>> >>>> around
>>> >>>>> these alignment issues in their schemas.
>>> >>>>>
>>> >>>>> Changing these schemas for 3.3.0 compatibility is highly
>>> >>>>> undesirable
>>> >>>> (as
>>> >>>>> was done for PNG), even if the changes are backward compatible.
>>> >>>>>
>>> >>>>> (Though if the schemas are actually incorrect in some way that
>>> >>>>> we're
>>> >>>> now
>>> >>>>> detecting more effectively, that is the right fix.)
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> On Tue, Feb 22, 2022 at 11:38 AM Interrante, John A (GE Research,
>>> >>>>> US) < John.Interrante@ge.com> wrote:
>>> >>>>>
>>> >>>>>> +1
>>> >>>>>>
>>> >>>>>> I personally have no blocker or urgent issues that must be fixed
>>> >>>> before
>>> >>>>>> the next release (only some things I will need to start working
>>> >>>>>> on in
>>> >>>> the C
>>> >>>>>> backend, "Runtime 2," to handle some more complicated schemas).
>>> >>>>>>
>>> >>>>>> How will the roadmap for upcoming releases (
>>> >>>>>>
>>> >>>>
>>> https://cwiki.apache.org/confluence/display/DAFFODIL/Roadmap+for+Upc
>>> >>>> oming+Releases
>>> >>>> )
>>> >>>>>> change as a result of 3.3.0 being released asap?
>>> >>>>>>
>>> >>>>>> John
>>> >>>>>>
>>> >>>>>> -----Original Message-----
>>> >>>>>> From: Mike Beckerle <mb...@apache.org>
>>> >>>>>> Sent: Tuesday, February 22, 2022 10:44 AM
>>> >>>>>> To: dev@daffodil.apache.org
>>> >>>>>> Subject: EXT: [DISCUSS] need to release Daffodil 3.3.0
>>> >>>>>>
>>> >>>>>> WARNING: This email originated from outside of GE. Please
>>> >>>>>> validate the sender's email address before clicking on links or
>>> >>>>>> attachments as
>>> >>>> they may
>>> >>>>>> not be safe.
>>> >>>>>>
>>> >>>>>> A number of people are asking for 3.3.0, with its many bug fixes,
>>> >>>>>> to
>>> >>>> be
>>> >>>>>> released asap.
>>> >>>>>>
>>> >>>>>> Are there any remaining issues that must be fixed before this
>>> release?
>>> >>>>>>
>>> >>>>>> Otherwise I'd like to suggest we release 3.3.0.
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>>>
>>>
>>>
Re: [DISCUSS] need to release Daffodil 3.3.0
Posted by Mike Beckerle <mb...@apache.org>.
This was an easy fix.
Once this gets merged I believe we have direction to move forward with
3.3.0 release process.
Who would like to be release manager?
On Mon, Mar 14, 2022 at 6:06 PM Mike Beckerle <mb...@apache.org> wrote:
> So the 3 blocking issues have been fixed/merged.
>
> However, of several bugs found over the last few days, this one is quite
> problematic:
>
> https://issues.apache.org/jira/browse/DAFFODIL-2673
>
> I am going to try to fix this in the next day, and if it proves to be
> harder than that, postpone it until post 3.3.0.
>
>
>
> On Fri, Mar 11, 2022 at 10:37 AM Steve Lawrence <sl...@apache.org>
> wrote:
>
>> Agreed. +1 to start the release processes once these are resolved.
>>
>> On 3/10/22 6:05 PM, Interrante, John A (GE Research, US) wrote:
>> > Fixing these 3 tickets seems sufficient to me.
>> >
>> > -----Original Message-----
>> > From: Mike Beckerle <mb...@apache.org>
>> > Sent: Wednesday, March 9, 2022 6:00 PM
>> > To: dev@daffodil.apache.org
>> > Subject: EXT: Re: [DISCUSS] need to release Daffodil 3.3.0
>> >
>> > Once the revert/fix for regressions is merged, I think we're down to
>> just these 3 tickets as really critical for Daffodil 3.3.0 release:
>> >
>> > DAFFODIL-2652 <https://issues.apache.org/jira/browse/DAFFODIL-2652> -
>> Ability to disable all alignment
>> >
>> > Given the number of outstanding bugs associated with unparser deadlock
>> and alignment I think this feature is an important hedge allowing progress
>> to be made by schema authors even if they run into these unparser/alignment
>> related issues (like DAFFODIL-2662 or DAFFODIL-2666 which are hard to fix,
>> and I think we don't want to hold back the release for those fixes because
>> they will take a while.)
>> >
>> > DAFFODIL-2650 <https://issues.apache.org/jira/browse/DAFFODIL-2650> -
>> using config file with cli parse or save parser causes backtrace
>> >
>> > Major usability issue when dealing with DFDL schemas that issue many
>> warnings.
>> >
>> > DAFFODIL-2267 <https://issues.apache.org/jira/browse/DAFFODIL-2267> -
>> Warnings emitted on pre-compiled parsers
>> >
>> > Major usability issue when dealing with DFDL schemas that issue many
>> warnings in deployed Daffodil-based applications. Clutters the log with too
>> many things users have to know can be ignored.
>> >
>> > I will say these latter 2 bugs have been a huge pain in the neck to me
>> of late in debugging efforts associated with some DFDL schema work. They
>> just so clutter the output that you really can't see what is going on
>> sometimes.
>> >
>> > Thoughts? Is fixing these enough for us to do a release of 3.3.0 ?
>> >
>> > On Wed, Mar 9, 2022 at 2:06 PM Mike Beckerle <mb...@apache.org>
>> wrote:
>> >
>> >> I just opened a PR which reverts a change which fixed a bug
>> >> (DAFFODIL-2626), but caused a number of regressions detected only by
>> >> other DFDL schemas such as NITF. (DAFFODIL-2666 and DAFFODIL-2662 are
>> >> regressions it caused.)
>> >>
>> >> The original bug is preferable to these regressions.
>> >>
>> >> This will get us closer to a releasable 3.3.0.
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On Wed, Mar 2, 2022 at 2:12 PM Mike Beckerle <mb...@apache.org>
>> wrote:
>> >>
>> >>> I've marked all the alignment/cyclic-deadlock regressions as blockers
>> >>> for
>> >>> 3.3.0 along with the "hammer" to
>> >>> just turn off alignment.
>> >>>
>> >>> The fixing suggested in the thread here may be the fix, or the
>> "hammer"
>> >>> fix, but the regressions on unparsing have to be addressed in 3.3.0,
>> >>> i.e., asap, before we can release it.
>> >>>
>> >>> I think other things we "almost" got working, like prefixed length
>> >>> fixes (of various bugs) could wait for a later release.
>> >>>
>> >>> There are numerous user projects I know about that are depending on
>> >>> 3.3.0 coming out quite soon now, without regressions.
>> >>>
>> >>>
>> >>> On Wed, Feb 23, 2022 at 11:19 AM Steve Lawrence
>> >>> <sl...@apache.org>
>> >>> wrote:
>> >>>
>> >>>> I assume this is caused by alignment regions not getting optimized
>> >>>> out with the recent changes to the alignment algorithm. It's now
>> >>>> more correct, but it's more pessimistic.
>> >>>>
>> >>>> A hammer to just disable alignment might be a reasonable solution,
>> >>>> but I'd be concerned there are alignment regions that are needed,
>> >>>> it's not usually obvious, especially in complex schemas.
>> >>>>
>> >>>> I think the main change that causes regions to fail to optimize out
>> >>>> is that we can't optimize out alignment related to global
>> >>>> declarations because we don't know the alignment of the references.
>> >>>>
>> >>>> I added comments in
>> >>>> https://issues.apache.org/jira/browse/DAFFODIL-2626
>> >>>> that discuss this issue, and a potential fixe. I believe we just
>> >>>> need to require that alignment of global decl's to be the same as
>> >>>> their references. I hope that this would allow more optimization of
>> >>>> alignment regions. One issue was raised about global complexType's,
>> >>>> who's alignment only comes from the references, with no information
>> >>>> on the declaration. So that also causes issues with this approach.
>> >>>>
>> >>>> I think implementing one or both of these options as tunables might
>> >>>> help improve the alignment issue and would be reasonable to get in
>> 3.3.0.
>> >>>>
>> >>>>
>> >>>> On 2/23/22 11:08 AM, Mike Beckerle wrote:
>> >>>>> So, we seem to be seeing a lot of regressions in various DFDL
>> >>>>> schemas
>> >>>> like
>> >>>>> most recently NITF, previously PNG.
>> >>>>>
>> >>>>> What if users run into this in their own DFDL schemas?
>> >>>>>
>> >>>>> These are showing unparser deadlocks due to cyclic relationships.
>> >>>>> At
>> >>>> one
>> >>>>> time we discussed adding a "big hammer" property or tunable that
>> >>>>> simply turns off alignment, as a workaround for all these sorts of
>> >>>>> alignment issues. I am wondering if we will need that so that
>> >>>>> users can work
>> >>>> around
>> >>>>> these alignment issues in their schemas.
>> >>>>>
>> >>>>> Changing these schemas for 3.3.0 compatibility is highly
>> >>>>> undesirable
>> >>>> (as
>> >>>>> was done for PNG), even if the changes are backward compatible.
>> >>>>>
>> >>>>> (Though if the schemas are actually incorrect in some way that
>> >>>>> we're
>> >>>> now
>> >>>>> detecting more effectively, that is the right fix.)
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> On Tue, Feb 22, 2022 at 11:38 AM Interrante, John A (GE Research,
>> >>>>> US) < John.Interrante@ge.com> wrote:
>> >>>>>
>> >>>>>> +1
>> >>>>>>
>> >>>>>> I personally have no blocker or urgent issues that must be fixed
>> >>>> before
>> >>>>>> the next release (only some things I will need to start working
>> >>>>>> on in
>> >>>> the C
>> >>>>>> backend, "Runtime 2," to handle some more complicated schemas).
>> >>>>>>
>> >>>>>> How will the roadmap for upcoming releases (
>> >>>>>>
>> >>>> https://cwiki.apache.org/confluence/display/DAFFODIL/Roadmap+for+Upc
>> >>>> oming+Releases
>> >>>> )
>> >>>>>> change as a result of 3.3.0 being released asap?
>> >>>>>>
>> >>>>>> John
>> >>>>>>
>> >>>>>> -----Original Message-----
>> >>>>>> From: Mike Beckerle <mb...@apache.org>
>> >>>>>> Sent: Tuesday, February 22, 2022 10:44 AM
>> >>>>>> To: dev@daffodil.apache.org
>> >>>>>> Subject: EXT: [DISCUSS] need to release Daffodil 3.3.0
>> >>>>>>
>> >>>>>> WARNING: This email originated from outside of GE. Please
>> >>>>>> validate the sender's email address before clicking on links or
>> >>>>>> attachments as
>> >>>> they may
>> >>>>>> not be safe.
>> >>>>>>
>> >>>>>> A number of people are asking for 3.3.0, with its many bug fixes,
>> >>>>>> to
>> >>>> be
>> >>>>>> released asap.
>> >>>>>>
>> >>>>>> Are there any remaining issues that must be fixed before this
>> release?
>> >>>>>>
>> >>>>>> Otherwise I'd like to suggest we release 3.3.0.
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>>
>>
>>
Re: [DISCUSS] need to release Daffodil 3.3.0
Posted by Mike Beckerle <mb...@apache.org>.
So the 3 blocking issues have been fixed/merged.
However, of several bugs found over the last few days, this one is quite
problematic:
https://issues.apache.org/jira/browse/DAFFODIL-2673
I am going to try to fix this in the next day, and if it proves to be
harder than that, postpone it until post 3.3.0.
On Fri, Mar 11, 2022 at 10:37 AM Steve Lawrence <sl...@apache.org>
wrote:
> Agreed. +1 to start the release processes once these are resolved.
>
> On 3/10/22 6:05 PM, Interrante, John A (GE Research, US) wrote:
> > Fixing these 3 tickets seems sufficient to me.
> >
> > -----Original Message-----
> > From: Mike Beckerle <mb...@apache.org>
> > Sent: Wednesday, March 9, 2022 6:00 PM
> > To: dev@daffodil.apache.org
> > Subject: EXT: Re: [DISCUSS] need to release Daffodil 3.3.0
> >
> > Once the revert/fix for regressions is merged, I think we're down to
> just these 3 tickets as really critical for Daffodil 3.3.0 release:
> >
> > DAFFODIL-2652 <https://issues.apache.org/jira/browse/DAFFODIL-2652> -
> Ability to disable all alignment
> >
> > Given the number of outstanding bugs associated with unparser deadlock
> and alignment I think this feature is an important hedge allowing progress
> to be made by schema authors even if they run into these unparser/alignment
> related issues (like DAFFODIL-2662 or DAFFODIL-2666 which are hard to fix,
> and I think we don't want to hold back the release for those fixes because
> they will take a while.)
> >
> > DAFFODIL-2650 <https://issues.apache.org/jira/browse/DAFFODIL-2650> -
> using config file with cli parse or save parser causes backtrace
> >
> > Major usability issue when dealing with DFDL schemas that issue many
> warnings.
> >
> > DAFFODIL-2267 <https://issues.apache.org/jira/browse/DAFFODIL-2267> -
> Warnings emitted on pre-compiled parsers
> >
> > Major usability issue when dealing with DFDL schemas that issue many
> warnings in deployed Daffodil-based applications. Clutters the log with too
> many things users have to know can be ignored.
> >
> > I will say these latter 2 bugs have been a huge pain in the neck to me
> of late in debugging efforts associated with some DFDL schema work. They
> just so clutter the output that you really can't see what is going on
> sometimes.
> >
> > Thoughts? Is fixing these enough for us to do a release of 3.3.0 ?
> >
> > On Wed, Mar 9, 2022 at 2:06 PM Mike Beckerle <mb...@apache.org>
> wrote:
> >
> >> I just opened a PR which reverts a change which fixed a bug
> >> (DAFFODIL-2626), but caused a number of regressions detected only by
> >> other DFDL schemas such as NITF. (DAFFODIL-2666 and DAFFODIL-2662 are
> >> regressions it caused.)
> >>
> >> The original bug is preferable to these regressions.
> >>
> >> This will get us closer to a releasable 3.3.0.
> >>
> >>
> >>
> >>
> >>
> >> On Wed, Mar 2, 2022 at 2:12 PM Mike Beckerle <mb...@apache.org>
> wrote:
> >>
> >>> I've marked all the alignment/cyclic-deadlock regressions as blockers
> >>> for
> >>> 3.3.0 along with the "hammer" to
> >>> just turn off alignment.
> >>>
> >>> The fixing suggested in the thread here may be the fix, or the "hammer"
> >>> fix, but the regressions on unparsing have to be addressed in 3.3.0,
> >>> i.e., asap, before we can release it.
> >>>
> >>> I think other things we "almost" got working, like prefixed length
> >>> fixes (of various bugs) could wait for a later release.
> >>>
> >>> There are numerous user projects I know about that are depending on
> >>> 3.3.0 coming out quite soon now, without regressions.
> >>>
> >>>
> >>> On Wed, Feb 23, 2022 at 11:19 AM Steve Lawrence
> >>> <sl...@apache.org>
> >>> wrote:
> >>>
> >>>> I assume this is caused by alignment regions not getting optimized
> >>>> out with the recent changes to the alignment algorithm. It's now
> >>>> more correct, but it's more pessimistic.
> >>>>
> >>>> A hammer to just disable alignment might be a reasonable solution,
> >>>> but I'd be concerned there are alignment regions that are needed,
> >>>> it's not usually obvious, especially in complex schemas.
> >>>>
> >>>> I think the main change that causes regions to fail to optimize out
> >>>> is that we can't optimize out alignment related to global
> >>>> declarations because we don't know the alignment of the references.
> >>>>
> >>>> I added comments in
> >>>> https://issues.apache.org/jira/browse/DAFFODIL-2626
> >>>> that discuss this issue, and a potential fixe. I believe we just
> >>>> need to require that alignment of global decl's to be the same as
> >>>> their references. I hope that this would allow more optimization of
> >>>> alignment regions. One issue was raised about global complexType's,
> >>>> who's alignment only comes from the references, with no information
> >>>> on the declaration. So that also causes issues with this approach.
> >>>>
> >>>> I think implementing one or both of these options as tunables might
> >>>> help improve the alignment issue and would be reasonable to get in
> 3.3.0.
> >>>>
> >>>>
> >>>> On 2/23/22 11:08 AM, Mike Beckerle wrote:
> >>>>> So, we seem to be seeing a lot of regressions in various DFDL
> >>>>> schemas
> >>>> like
> >>>>> most recently NITF, previously PNG.
> >>>>>
> >>>>> What if users run into this in their own DFDL schemas?
> >>>>>
> >>>>> These are showing unparser deadlocks due to cyclic relationships.
> >>>>> At
> >>>> one
> >>>>> time we discussed adding a "big hammer" property or tunable that
> >>>>> simply turns off alignment, as a workaround for all these sorts of
> >>>>> alignment issues. I am wondering if we will need that so that
> >>>>> users can work
> >>>> around
> >>>>> these alignment issues in their schemas.
> >>>>>
> >>>>> Changing these schemas for 3.3.0 compatibility is highly
> >>>>> undesirable
> >>>> (as
> >>>>> was done for PNG), even if the changes are backward compatible.
> >>>>>
> >>>>> (Though if the schemas are actually incorrect in some way that
> >>>>> we're
> >>>> now
> >>>>> detecting more effectively, that is the right fix.)
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Tue, Feb 22, 2022 at 11:38 AM Interrante, John A (GE Research,
> >>>>> US) < John.Interrante@ge.com> wrote:
> >>>>>
> >>>>>> +1
> >>>>>>
> >>>>>> I personally have no blocker or urgent issues that must be fixed
> >>>> before
> >>>>>> the next release (only some things I will need to start working
> >>>>>> on in
> >>>> the C
> >>>>>> backend, "Runtime 2," to handle some more complicated schemas).
> >>>>>>
> >>>>>> How will the roadmap for upcoming releases (
> >>>>>>
> >>>> https://cwiki.apache.org/confluence/display/DAFFODIL/Roadmap+for+Upc
> >>>> oming+Releases
> >>>> )
> >>>>>> change as a result of 3.3.0 being released asap?
> >>>>>>
> >>>>>> John
> >>>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: Mike Beckerle <mb...@apache.org>
> >>>>>> Sent: Tuesday, February 22, 2022 10:44 AM
> >>>>>> To: dev@daffodil.apache.org
> >>>>>> Subject: EXT: [DISCUSS] need to release Daffodil 3.3.0
> >>>>>>
> >>>>>> WARNING: This email originated from outside of GE. Please
> >>>>>> validate the sender's email address before clicking on links or
> >>>>>> attachments as
> >>>> they may
> >>>>>> not be safe.
> >>>>>>
> >>>>>> A number of people are asking for 3.3.0, with its many bug fixes,
> >>>>>> to
> >>>> be
> >>>>>> released asap.
> >>>>>>
> >>>>>> Are there any remaining issues that must be fixed before this
> release?
> >>>>>>
> >>>>>> Otherwise I'd like to suggest we release 3.3.0.
> >>>>>>
> >>>>>
> >>>>
> >>>>
>
>
Re: [DISCUSS] need to release Daffodil 3.3.0
Posted by Steve Lawrence <sl...@apache.org>.
Agreed. +1 to start the release processes once these are resolved.
On 3/10/22 6:05 PM, Interrante, John A (GE Research, US) wrote:
> Fixing these 3 tickets seems sufficient to me.
>
> -----Original Message-----
> From: Mike Beckerle <mb...@apache.org>
> Sent: Wednesday, March 9, 2022 6:00 PM
> To: dev@daffodil.apache.org
> Subject: EXT: Re: [DISCUSS] need to release Daffodil 3.3.0
>
> Once the revert/fix for regressions is merged, I think we're down to just these 3 tickets as really critical for Daffodil 3.3.0 release:
>
> DAFFODIL-2652 <https://issues.apache.org/jira/browse/DAFFODIL-2652> - Ability to disable all alignment
>
> Given the number of outstanding bugs associated with unparser deadlock and alignment I think this feature is an important hedge allowing progress to be made by schema authors even if they run into these unparser/alignment related issues (like DAFFODIL-2662 or DAFFODIL-2666 which are hard to fix, and I think we don't want to hold back the release for those fixes because they will take a while.)
>
> DAFFODIL-2650 <https://issues.apache.org/jira/browse/DAFFODIL-2650> - using config file with cli parse or save parser causes backtrace
>
> Major usability issue when dealing with DFDL schemas that issue many warnings.
>
> DAFFODIL-2267 <https://issues.apache.org/jira/browse/DAFFODIL-2267> - Warnings emitted on pre-compiled parsers
>
> Major usability issue when dealing with DFDL schemas that issue many warnings in deployed Daffodil-based applications. Clutters the log with too many things users have to know can be ignored.
>
> I will say these latter 2 bugs have been a huge pain in the neck to me of late in debugging efforts associated with some DFDL schema work. They just so clutter the output that you really can't see what is going on sometimes.
>
> Thoughts? Is fixing these enough for us to do a release of 3.3.0 ?
>
> On Wed, Mar 9, 2022 at 2:06 PM Mike Beckerle <mb...@apache.org> wrote:
>
>> I just opened a PR which reverts a change which fixed a bug
>> (DAFFODIL-2626), but caused a number of regressions detected only by
>> other DFDL schemas such as NITF. (DAFFODIL-2666 and DAFFODIL-2662 are
>> regressions it caused.)
>>
>> The original bug is preferable to these regressions.
>>
>> This will get us closer to a releasable 3.3.0.
>>
>>
>>
>>
>>
>> On Wed, Mar 2, 2022 at 2:12 PM Mike Beckerle <mb...@apache.org> wrote:
>>
>>> I've marked all the alignment/cyclic-deadlock regressions as blockers
>>> for
>>> 3.3.0 along with the "hammer" to
>>> just turn off alignment.
>>>
>>> The fixing suggested in the thread here may be the fix, or the "hammer"
>>> fix, but the regressions on unparsing have to be addressed in 3.3.0,
>>> i.e., asap, before we can release it.
>>>
>>> I think other things we "almost" got working, like prefixed length
>>> fixes (of various bugs) could wait for a later release.
>>>
>>> There are numerous user projects I know about that are depending on
>>> 3.3.0 coming out quite soon now, without regressions.
>>>
>>>
>>> On Wed, Feb 23, 2022 at 11:19 AM Steve Lawrence
>>> <sl...@apache.org>
>>> wrote:
>>>
>>>> I assume this is caused by alignment regions not getting optimized
>>>> out with the recent changes to the alignment algorithm. It's now
>>>> more correct, but it's more pessimistic.
>>>>
>>>> A hammer to just disable alignment might be a reasonable solution,
>>>> but I'd be concerned there are alignment regions that are needed,
>>>> it's not usually obvious, especially in complex schemas.
>>>>
>>>> I think the main change that causes regions to fail to optimize out
>>>> is that we can't optimize out alignment related to global
>>>> declarations because we don't know the alignment of the references.
>>>>
>>>> I added comments in
>>>> https://issues.apache.org/jira/browse/DAFFODIL-2626
>>>> that discuss this issue, and a potential fixe. I believe we just
>>>> need to require that alignment of global decl's to be the same as
>>>> their references. I hope that this would allow more optimization of
>>>> alignment regions. One issue was raised about global complexType's,
>>>> who's alignment only comes from the references, with no information
>>>> on the declaration. So that also causes issues with this approach.
>>>>
>>>> I think implementing one or both of these options as tunables might
>>>> help improve the alignment issue and would be reasonable to get in 3.3.0.
>>>>
>>>>
>>>> On 2/23/22 11:08 AM, Mike Beckerle wrote:
>>>>> So, we seem to be seeing a lot of regressions in various DFDL
>>>>> schemas
>>>> like
>>>>> most recently NITF, previously PNG.
>>>>>
>>>>> What if users run into this in their own DFDL schemas?
>>>>>
>>>>> These are showing unparser deadlocks due to cyclic relationships.
>>>>> At
>>>> one
>>>>> time we discussed adding a "big hammer" property or tunable that
>>>>> simply turns off alignment, as a workaround for all these sorts of
>>>>> alignment issues. I am wondering if we will need that so that
>>>>> users can work
>>>> around
>>>>> these alignment issues in their schemas.
>>>>>
>>>>> Changing these schemas for 3.3.0 compatibility is highly
>>>>> undesirable
>>>> (as
>>>>> was done for PNG), even if the changes are backward compatible.
>>>>>
>>>>> (Though if the schemas are actually incorrect in some way that
>>>>> we're
>>>> now
>>>>> detecting more effectively, that is the right fix.)
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Feb 22, 2022 at 11:38 AM Interrante, John A (GE Research,
>>>>> US) < John.Interrante@ge.com> wrote:
>>>>>
>>>>>> +1
>>>>>>
>>>>>> I personally have no blocker or urgent issues that must be fixed
>>>> before
>>>>>> the next release (only some things I will need to start working
>>>>>> on in
>>>> the C
>>>>>> backend, "Runtime 2," to handle some more complicated schemas).
>>>>>>
>>>>>> How will the roadmap for upcoming releases (
>>>>>>
>>>> https://cwiki.apache.org/confluence/display/DAFFODIL/Roadmap+for+Upc
>>>> oming+Releases
>>>> )
>>>>>> change as a result of 3.3.0 being released asap?
>>>>>>
>>>>>> John
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Mike Beckerle <mb...@apache.org>
>>>>>> Sent: Tuesday, February 22, 2022 10:44 AM
>>>>>> To: dev@daffodil.apache.org
>>>>>> Subject: EXT: [DISCUSS] need to release Daffodil 3.3.0
>>>>>>
>>>>>> WARNING: This email originated from outside of GE. Please
>>>>>> validate the sender's email address before clicking on links or
>>>>>> attachments as
>>>> they may
>>>>>> not be safe.
>>>>>>
>>>>>> A number of people are asking for 3.3.0, with its many bug fixes,
>>>>>> to
>>>> be
>>>>>> released asap.
>>>>>>
>>>>>> Are there any remaining issues that must be fixed before this release?
>>>>>>
>>>>>> Otherwise I'd like to suggest we release 3.3.0.
>>>>>>
>>>>>
>>>>
>>>>
[DISCUSS] need to release Daffodil 3.3.0
Posted by "Interrante, John A (GE Research, US)" <Jo...@ge.com>.
Fixing these 3 tickets seems sufficient to me.
-----Original Message-----
From: Mike Beckerle <mb...@apache.org>
Sent: Wednesday, March 9, 2022 6:00 PM
To: dev@daffodil.apache.org
Subject: EXT: Re: [DISCUSS] need to release Daffodil 3.3.0
Once the revert/fix for regressions is merged, I think we're down to just these 3 tickets as really critical for Daffodil 3.3.0 release:
DAFFODIL-2652 <https://issues.apache.org/jira/browse/DAFFODIL-2652> - Ability to disable all alignment
Given the number of outstanding bugs associated with unparser deadlock and alignment I think this feature is an important hedge allowing progress to be made by schema authors even if they run into these unparser/alignment related issues (like DAFFODIL-2662 or DAFFODIL-2666 which are hard to fix, and I think we don't want to hold back the release for those fixes because they will take a while.)
DAFFODIL-2650 <https://issues.apache.org/jira/browse/DAFFODIL-2650> - using config file with cli parse or save parser causes backtrace
Major usability issue when dealing with DFDL schemas that issue many warnings.
DAFFODIL-2267 <https://issues.apache.org/jira/browse/DAFFODIL-2267> - Warnings emitted on pre-compiled parsers
Major usability issue when dealing with DFDL schemas that issue many warnings in deployed Daffodil-based applications. Clutters the log with too many things users have to know can be ignored.
I will say these latter 2 bugs have been a huge pain in the neck to me of late in debugging efforts associated with some DFDL schema work. They just so clutter the output that you really can't see what is going on sometimes.
Thoughts? Is fixing these enough for us to do a release of 3.3.0 ?
On Wed, Mar 9, 2022 at 2:06 PM Mike Beckerle <mb...@apache.org> wrote:
> I just opened a PR which reverts a change which fixed a bug
> (DAFFODIL-2626), but caused a number of regressions detected only by
> other DFDL schemas such as NITF. (DAFFODIL-2666 and DAFFODIL-2662 are
> regressions it caused.)
>
> The original bug is preferable to these regressions.
>
> This will get us closer to a releasable 3.3.0.
>
>
>
>
>
> On Wed, Mar 2, 2022 at 2:12 PM Mike Beckerle <mb...@apache.org> wrote:
>
>> I've marked all the alignment/cyclic-deadlock regressions as blockers
>> for
>> 3.3.0 along with the "hammer" to
>> just turn off alignment.
>>
>> The fixing suggested in the thread here may be the fix, or the "hammer"
>> fix, but the regressions on unparsing have to be addressed in 3.3.0,
>> i.e., asap, before we can release it.
>>
>> I think other things we "almost" got working, like prefixed length
>> fixes (of various bugs) could wait for a later release.
>>
>> There are numerous user projects I know about that are depending on
>> 3.3.0 coming out quite soon now, without regressions.
>>
>>
>> On Wed, Feb 23, 2022 at 11:19 AM Steve Lawrence
>> <sl...@apache.org>
>> wrote:
>>
>>> I assume this is caused by alignment regions not getting optimized
>>> out with the recent changes to the alignment algorithm. It's now
>>> more correct, but it's more pessimistic.
>>>
>>> A hammer to just disable alignment might be a reasonable solution,
>>> but I'd be concerned there are alignment regions that are needed,
>>> it's not usually obvious, especially in complex schemas.
>>>
>>> I think the main change that causes regions to fail to optimize out
>>> is that we can't optimize out alignment related to global
>>> declarations because we don't know the alignment of the references.
>>>
>>> I added comments in
>>> https://issues.apache.org/jira/browse/DAFFODIL-2626
>>> that discuss this issue, and a potential fixe. I believe we just
>>> need to require that alignment of global decl's to be the same as
>>> their references. I hope that this would allow more optimization of
>>> alignment regions. One issue was raised about global complexType's,
>>> who's alignment only comes from the references, with no information
>>> on the declaration. So that also causes issues with this approach.
>>>
>>> I think implementing one or both of these options as tunables might
>>> help improve the alignment issue and would be reasonable to get in 3.3.0.
>>>
>>>
>>> On 2/23/22 11:08 AM, Mike Beckerle wrote:
>>> > So, we seem to be seeing a lot of regressions in various DFDL
>>> > schemas
>>> like
>>> > most recently NITF, previously PNG.
>>> >
>>> > What if users run into this in their own DFDL schemas?
>>> >
>>> > These are showing unparser deadlocks due to cyclic relationships.
>>> > At
>>> one
>>> > time we discussed adding a "big hammer" property or tunable that
>>> > simply turns off alignment, as a workaround for all these sorts of
>>> > alignment issues. I am wondering if we will need that so that
>>> > users can work
>>> around
>>> > these alignment issues in their schemas.
>>> >
>>> > Changing these schemas for 3.3.0 compatibility is highly
>>> > undesirable
>>> (as
>>> > was done for PNG), even if the changes are backward compatible.
>>> >
>>> > (Though if the schemas are actually incorrect in some way that
>>> > we're
>>> now
>>> > detecting more effectively, that is the right fix.)
>>> >
>>> >
>>> >
>>> > On Tue, Feb 22, 2022 at 11:38 AM Interrante, John A (GE Research,
>>> > US) < John.Interrante@ge.com> wrote:
>>> >
>>> >> +1
>>> >>
>>> >> I personally have no blocker or urgent issues that must be fixed
>>> before
>>> >> the next release (only some things I will need to start working
>>> >> on in
>>> the C
>>> >> backend, "Runtime 2," to handle some more complicated schemas).
>>> >>
>>> >> How will the roadmap for upcoming releases (
>>> >>
>>> https://cwiki.apache.org/confluence/display/DAFFODIL/Roadmap+for+Upc
>>> oming+Releases
>>> )
>>> >> change as a result of 3.3.0 being released asap?
>>> >>
>>> >> John
>>> >>
>>> >> -----Original Message-----
>>> >> From: Mike Beckerle <mb...@apache.org>
>>> >> Sent: Tuesday, February 22, 2022 10:44 AM
>>> >> To: dev@daffodil.apache.org
>>> >> Subject: EXT: [DISCUSS] need to release Daffodil 3.3.0
>>> >>
>>> >> WARNING: This email originated from outside of GE. Please
>>> >> validate the sender's email address before clicking on links or
>>> >> attachments as
>>> they may
>>> >> not be safe.
>>> >>
>>> >> A number of people are asking for 3.3.0, with its many bug fixes,
>>> >> to
>>> be
>>> >> released asap.
>>> >>
>>> >> Are there any remaining issues that must be fixed before this release?
>>> >>
>>> >> Otherwise I'd like to suggest we release 3.3.0.
>>> >>
>>> >
>>>
>>>
Re: [DISCUSS] need to release Daffodil 3.3.0
Posted by Mike Beckerle <mb...@apache.org>.
Once the revert/fix for regressions is merged, I think we're down to just
these 3 tickets as really critical for Daffodil 3.3.0 release:
DAFFODIL-2652 <https://issues.apache.org/jira/browse/DAFFODIL-2652> -
Ability to disable all alignment
Given the number of outstanding bugs associated with unparser deadlock and
alignment I think this feature is an important hedge allowing progress to
be made by schema authors even if they run into these unparser/alignment
related issues (like DAFFODIL-2662 or DAFFODIL-2666 which are hard to fix,
and I think we don't want to hold back the release for those fixes because
they will take a while.)
DAFFODIL-2650 <https://issues.apache.org/jira/browse/DAFFODIL-2650> - using
config file with cli parse or save parser causes backtrace
Major usability issue when dealing with DFDL schemas that issue many
warnings.
DAFFODIL-2267 <https://issues.apache.org/jira/browse/DAFFODIL-2267> - Warnings
emitted on pre-compiled parsers
Major usability issue when dealing with DFDL schemas that issue many
warnings in deployed Daffodil-based applications. Clutters the log with too
many things users have to know can be ignored.
I will say these latter 2 bugs have been a huge pain in the neck to me of
late in debugging efforts associated with some DFDL schema work. They just
so clutter the output that you really can't see what is going on sometimes.
Thoughts? Is fixing these enough for us to do a release of 3.3.0 ?
On Wed, Mar 9, 2022 at 2:06 PM Mike Beckerle <mb...@apache.org> wrote:
> I just opened a PR which reverts a change which fixed a bug
> (DAFFODIL-2626), but caused a number of regressions detected only by other
> DFDL schemas such as NITF. (DAFFODIL-2666 and DAFFODIL-2662 are regressions
> it caused.)
>
> The original bug is preferable to these regressions.
>
> This will get us closer to a releasable 3.3.0.
>
>
>
>
>
> On Wed, Mar 2, 2022 at 2:12 PM Mike Beckerle <mb...@apache.org> wrote:
>
>> I've marked all the alignment/cyclic-deadlock regressions as blockers for
>> 3.3.0 along with the "hammer" to
>> just turn off alignment.
>>
>> The fixing suggested in the thread here may be the fix, or the "hammer"
>> fix, but the regressions on unparsing have to be addressed in 3.3.0, i.e.,
>> asap, before we can release it.
>>
>> I think other things we "almost" got working, like prefixed length fixes
>> (of various bugs) could wait for a later release.
>>
>> There are numerous user projects I know about that are depending on 3.3.0
>> coming out quite soon now, without regressions.
>>
>>
>> On Wed, Feb 23, 2022 at 11:19 AM Steve Lawrence <sl...@apache.org>
>> wrote:
>>
>>> I assume this is caused by alignment regions not getting optimized out
>>> with the recent changes to the alignment algorithm. It's now more
>>> correct, but it's more pessimistic.
>>>
>>> A hammer to just disable alignment might be a reasonable solution, but
>>> I'd be concerned there are alignment regions that are needed, it's not
>>> usually obvious, especially in complex schemas.
>>>
>>> I think the main change that causes regions to fail to optimize out is
>>> that we can't optimize out alignment related to global declarations
>>> because we don't know the alignment of the references.
>>>
>>> I added comments in https://issues.apache.org/jira/browse/DAFFODIL-2626
>>> that discuss this issue, and a potential fixe. I believe we just need to
>>> require that alignment of global decl's to be the same as their
>>> references. I hope that this would allow more optimization of alignment
>>> regions. One issue was raised about global complexType's, who's
>>> alignment only comes from the references, with no information on the
>>> declaration. So that also causes issues with this approach.
>>>
>>> I think implementing one or both of these options as tunables might help
>>> improve the alignment issue and would be reasonable to get in 3.3.0.
>>>
>>>
>>> On 2/23/22 11:08 AM, Mike Beckerle wrote:
>>> > So, we seem to be seeing a lot of regressions in various DFDL schemas
>>> like
>>> > most recently NITF, previously PNG.
>>> >
>>> > What if users run into this in their own DFDL schemas?
>>> >
>>> > These are showing unparser deadlocks due to cyclic relationships. At
>>> one
>>> > time we discussed adding a "big hammer" property or tunable that simply
>>> > turns off alignment, as a workaround for all these sorts of alignment
>>> > issues. I am wondering if we will need that so that users can work
>>> around
>>> > these alignment issues in their schemas.
>>> >
>>> > Changing these schemas for 3.3.0 compatibility is highly undesirable
>>> (as
>>> > was done for PNG), even if the changes are backward compatible.
>>> >
>>> > (Though if the schemas are actually incorrect in some way that we're
>>> now
>>> > detecting more effectively, that is the right fix.)
>>> >
>>> >
>>> >
>>> > On Tue, Feb 22, 2022 at 11:38 AM Interrante, John A (GE Research, US) <
>>> > John.Interrante@ge.com> wrote:
>>> >
>>> >> +1
>>> >>
>>> >> I personally have no blocker or urgent issues that must be fixed
>>> before
>>> >> the next release (only some things I will need to start working on in
>>> the C
>>> >> backend, "Runtime 2," to handle some more complicated schemas).
>>> >>
>>> >> How will the roadmap for upcoming releases (
>>> >>
>>> https://cwiki.apache.org/confluence/display/DAFFODIL/Roadmap+for+Upcoming+Releases
>>> )
>>> >> change as a result of 3.3.0 being released asap?
>>> >>
>>> >> John
>>> >>
>>> >> -----Original Message-----
>>> >> From: Mike Beckerle <mb...@apache.org>
>>> >> Sent: Tuesday, February 22, 2022 10:44 AM
>>> >> To: dev@daffodil.apache.org
>>> >> Subject: EXT: [DISCUSS] need to release Daffodil 3.3.0
>>> >>
>>> >> WARNING: This email originated from outside of GE. Please validate the
>>> >> sender's email address before clicking on links or attachments as
>>> they may
>>> >> not be safe.
>>> >>
>>> >> A number of people are asking for 3.3.0, with its many bug fixes, to
>>> be
>>> >> released asap.
>>> >>
>>> >> Are there any remaining issues that must be fixed before this release?
>>> >>
>>> >> Otherwise I'd like to suggest we release 3.3.0.
>>> >>
>>> >
>>>
>>>
Re: [DISCUSS] need to release Daffodil 3.3.0
Posted by Mike Beckerle <mb...@apache.org>.
I just opened a PR which reverts a change which fixed a bug
(DAFFODIL-2626), but caused a number of regressions detected only by other
DFDL schemas such as NITF. (DAFFODIL-2666 and DAFFODIL-2662 are regressions
it caused.)
The original bug is preferable to these regressions.
This will get us closer to a releasable 3.3.0.
On Wed, Mar 2, 2022 at 2:12 PM Mike Beckerle <mb...@apache.org> wrote:
> I've marked all the alignment/cyclic-deadlock regressions as blockers for
> 3.3.0 along with the "hammer" to
> just turn off alignment.
>
> The fixing suggested in the thread here may be the fix, or the "hammer"
> fix, but the regressions on unparsing have to be addressed in 3.3.0, i.e.,
> asap, before we can release it.
>
> I think other things we "almost" got working, like prefixed length fixes
> (of various bugs) could wait for a later release.
>
> There are numerous user projects I know about that are depending on 3.3.0
> coming out quite soon now, without regressions.
>
>
> On Wed, Feb 23, 2022 at 11:19 AM Steve Lawrence <sl...@apache.org>
> wrote:
>
>> I assume this is caused by alignment regions not getting optimized out
>> with the recent changes to the alignment algorithm. It's now more
>> correct, but it's more pessimistic.
>>
>> A hammer to just disable alignment might be a reasonable solution, but
>> I'd be concerned there are alignment regions that are needed, it's not
>> usually obvious, especially in complex schemas.
>>
>> I think the main change that causes regions to fail to optimize out is
>> that we can't optimize out alignment related to global declarations
>> because we don't know the alignment of the references.
>>
>> I added comments in https://issues.apache.org/jira/browse/DAFFODIL-2626
>> that discuss this issue, and a potential fixe. I believe we just need to
>> require that alignment of global decl's to be the same as their
>> references. I hope that this would allow more optimization of alignment
>> regions. One issue was raised about global complexType's, who's
>> alignment only comes from the references, with no information on the
>> declaration. So that also causes issues with this approach.
>>
>> I think implementing one or both of these options as tunables might help
>> improve the alignment issue and would be reasonable to get in 3.3.0.
>>
>>
>> On 2/23/22 11:08 AM, Mike Beckerle wrote:
>> > So, we seem to be seeing a lot of regressions in various DFDL schemas
>> like
>> > most recently NITF, previously PNG.
>> >
>> > What if users run into this in their own DFDL schemas?
>> >
>> > These are showing unparser deadlocks due to cyclic relationships. At one
>> > time we discussed adding a "big hammer" property or tunable that simply
>> > turns off alignment, as a workaround for all these sorts of alignment
>> > issues. I am wondering if we will need that so that users can work
>> around
>> > these alignment issues in their schemas.
>> >
>> > Changing these schemas for 3.3.0 compatibility is highly undesirable (as
>> > was done for PNG), even if the changes are backward compatible.
>> >
>> > (Though if the schemas are actually incorrect in some way that we're now
>> > detecting more effectively, that is the right fix.)
>> >
>> >
>> >
>> > On Tue, Feb 22, 2022 at 11:38 AM Interrante, John A (GE Research, US) <
>> > John.Interrante@ge.com> wrote:
>> >
>> >> +1
>> >>
>> >> I personally have no blocker or urgent issues that must be fixed before
>> >> the next release (only some things I will need to start working on in
>> the C
>> >> backend, "Runtime 2," to handle some more complicated schemas).
>> >>
>> >> How will the roadmap for upcoming releases (
>> >>
>> https://cwiki.apache.org/confluence/display/DAFFODIL/Roadmap+for+Upcoming+Releases
>> )
>> >> change as a result of 3.3.0 being released asap?
>> >>
>> >> John
>> >>
>> >> -----Original Message-----
>> >> From: Mike Beckerle <mb...@apache.org>
>> >> Sent: Tuesday, February 22, 2022 10:44 AM
>> >> To: dev@daffodil.apache.org
>> >> Subject: EXT: [DISCUSS] need to release Daffodil 3.3.0
>> >>
>> >> WARNING: This email originated from outside of GE. Please validate the
>> >> sender's email address before clicking on links or attachments as they
>> may
>> >> not be safe.
>> >>
>> >> A number of people are asking for 3.3.0, with its many bug fixes, to be
>> >> released asap.
>> >>
>> >> Are there any remaining issues that must be fixed before this release?
>> >>
>> >> Otherwise I'd like to suggest we release 3.3.0.
>> >>
>> >
>>
>>