You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@daffodil.apache.org by "nlewis05 (via GitHub)" <gi...@apache.org> on 2023/08/15 17:34:45 UTC
[GitHub] [daffodil-vscode] nlewis05 opened a new issue, #811: Support for adding stacktracing for TDML files
nlewis05 opened a new issue, #811:
URL: https://github.com/apache/daffodil-vscode/issues/811
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for adding stacktracing for TDML files [daffodil-vscode]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on issue #811:
URL: https://github.com/apache/daffodil-vscode/issues/811#issuecomment-1785738896
To me, it feels like a separate artifact makes more sense. Including the actual output in a TDML file, whether it be a stack trace or an incorrect infoset, just adds more complication to an already complicated format. It's usually pretty straightforward to run a TDML file to reproduce the unexpected issue.
It also adds complication, for example, you would need to run your TDML test, capture the output, and then modify that same TDML file. There's a chance that modification might accidentally change the test (e.g. copy/paste to the wrong spot) and make it not reproducible.
If you really want it in the TDML file, maybe it could be added as a comment or as an annotation or something, but I'm not sure I would want it as something the TDML runner understands and uses in any way. Like, even if the TDML runner saw the same stack trace, it still wouldn't want to consider it as a test pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for adding stacktracing for TDML files [daffodil-vscode]
Posted by "michael-hoke (via GitHub)" <gi...@apache.org>.
michael-hoke commented on issue #811:
URL: https://github.com/apache/daffodil-vscode/issues/811#issuecomment-1785845211
> A failed compile/parse/unparse should create a diagnostic (e.g. Schema Definition Error, Parse Error, Unparse Error) which can be expected instead of an infoset or unparse data. But if you get a stack trace instead of one of those diagnostics, it usually means there is a bug in Daffodil, and there is no way in TDML to expect a stack trace because we never consider bugs expected behavior. Similarly, we don't have a way to expect an incorrect infoset, because that also is also considered a bug.
Okay, thanks. I think I see where I was misunderstanding.
The main driver for this feature is to provide a way for a user to verify that they are seeing the same issue as someone else that reported an issue. If the diagnostic(s) will provide a way to do that, we'll look to use those instead. If that's that case, it should be possible for us to get the diagnostic(s) from daffodil and serialize it into the TDML test case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for adding stacktracing for TDML files [daffodil-vscode]
Posted by "michael-hoke (via GitHub)" <gi...@apache.org>.
michael-hoke commented on issue #811:
URL: https://github.com/apache/daffodil-vscode/issues/811#issuecomment-1785704290
Our main use case for TDML is posting test cases (presumably ones that fail to parse) on the user list. It would make sense to be able to attach a stack trace to a test case so that someone can verify their stack trace against the "expected" one that is being reported.
I'm not actually sure where this idea originated from, but we've been floating it around for quite some time as a nice to have feature for our users list scenario. Maybe I was assuming that a stack trace could be considered an output for a test case? eg. If I wanted to capture a daffodil parse that I expect to fail and compare the expected/actual stack traces.
We are planning to look into an automated way of producing a zip archive containing all artifacts required to run all test cases in a TDML file, so maybe it will make more sense to include the stack trace as a separate artifacts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for adding stacktracing for TDML files [daffodil-vscode]
Posted by "michael-hoke (via GitHub)" <gi...@apache.org>.
michael-hoke commented on issue #811:
URL: https://github.com/apache/daffodil-vscode/issues/811#issuecomment-1785773313
We are planning on "hiding" the editing of a TDML file through the extension. We will be exposing commands to users to create/edit TDML files, so there should not be a need to manually edit and risk a copy/paste error in the vast majority of cases. There will be the ability to do so when the need arises, but someone will have to go out of their way to do it.
Do you have a barebones stub of what an annotation might look like? I'm thinking we should prefer a separate artifact to a comment, but I'm not sure what an annotation would look like here.
> Like, even if the TDML runner saw the same stack trace, it still wouldn't want to consider it as a test pass.
Is there currently no way to do a negative test case with the TDMLRunner? Is it only able to compare the result of the parse (pass/fail)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for adding stacktracing for TDML files [daffodil-vscode]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on issue #811:
URL: https://github.com/apache/daffodil-vscode/issues/811#issuecomment-1785685566
> Ideally, we would want the stacktrace of a failed daffodil parse to end up in the TDML file
What do you see the use case of having a stack trace in a TDML file? Normally a TDML file defines the inputs (e.g. schema, parse data, unparse infoset) and expected outputs (parsed infoset or unparsed data). A stack trace is not normally thought of as an expected output, so doesn't feel like it belongs in a TDML file to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for adding stacktracing for TDML files [daffodil-vscode]
Posted by "michael-hoke (via GitHub)" <gi...@apache.org>.
michael-hoke commented on issue #811:
URL: https://github.com/apache/daffodil-vscode/issues/811#issuecomment-1785671327
Ideally, we would want the stacktrace of a failed daffodil parse to end up in the TDML file. There doesn't seem to be a clear place to put the stacktrace in the test case - does it make sense to include it as a separate text file? Is this defined in another schema that we would have to JAXBify?
This may not require information to be sent from server to client as previously thought as we do the JAXB work on the server side, but it would require us to capture the stacktrace and pass it to the JAXB code - currently, the JAXB code is currently only run before a parse, not after.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for adding stacktracing for TDML files [daffodil-vscode]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on issue #811:
URL: https://github.com/apache/daffodil-vscode/issues/811#issuecomment-1785823832
> Is there currently no way to do a negative test case with the TDMLRunner?
Yes, you can do negative tests by expecting strings that should appear in diagnostics (see the `<tdml:errors>` tag). A failed compile/parse/unparse should create a diagnostic (e.g. Schema Definition Error, Parse Error, Unparse Error) which can be expected instead of an infoset or unparse data. But if you get a stack trace instead of one of those diagnostics, it usually means there is a bug in Daffodil, and there is no way in TDML to expect a stack trace because we never consider bugs expected behavior. Similarly, we don't have a way to expect an incorrect infoset, because that also is also considered a bug. TDML files have traditionally been used to describe and test what *should* happen, and not really describe what *actually* happened.
> Do you have a barebones stub of what an annotation might look like?
It's usually something like:
```xml
<xs:annotation>
<xs:appinfo source="someURI">
Your content goes here
</xs:appinfo>
</xs:annotation>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org