[jira] [Commented] (DAFFODIL-2815) Implement Schema Coverage Test Tool in Daffodil

["Steve Lawrence (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DAFFODIL-2815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17728416#comment-17728416 ] 

Steve Lawrence commented on DAFFODIL-2815:
------------------------------------------

Note that my thought about using InfosetInputter/Outputters is probably not correct. That will only ever output information about non-hidden elements. So things like choices, sequences, and hidden elements would never show coverage.

So I think the first suggestion about plugging into parse/unparse calls probably makes more sense, since those do see all TRDs.

That said, it is still a bit tricky because often times a single TRD is associated with multiple parsers/unparsers using combinators. The if the first parse() of a combinator is successful but the second isn't, we may show coverage even though the TRD was not completely successful. Maybe this is just  limitation and we just accept that it's better than nothing.

I did a little digging about using Java Agents and I think it would actually be pretty straighforward and has the added benefit that we dont' have to change Daffodil get coverage, so we could get coverage using older versions of Daffodil. It's just a matter of adding in the right byte code at the right places (probaby the parse()/unparse() functions) to set bits.

Codecov.io lists a bunch of different coverage formats, we probably should try to use one of those so we don't have to reinvent the wheel.

https://docs.codecov.com/docs/supported-report-formats

However, codecov doesn't support XML as a language:

https://docs.codecov.com/docs/supported-languages

I also found https://cakupan-maven.sourceforge.net/, which does do coverage for XSLT files. It's old and unmaintained, but might be a useful historical resource for something similar, at least for the coverage report format and visiaulization, if existing tools won't work.



> Implement Schema Coverage Test Tool in Daffodil
> -----------------------------------------------
>
>                 Key: DAFFODIL-2815
>                 URL: https://issues.apache.org/jira/browse/DAFFODIL-2815
>             Project: Daffodil
>          Issue Type: Improvement
>            Reporter: Olabusayo Kilo
>            Priority: Major
>
> It would be nice if there was a way for daffodil to report how much of a schema is represented by TDML tests.
>  
> Possible Solutions:
>  # for each TRD structure we'd keep a parse and unparse bit in a hash table. At the end of a parse/unparse cycle we could look at all those bits and see what is not touched.
>  # I wonder if there are existing code coverage libraries or file formats we could use? I'm not sure how specific they are to actual programming languages or if it's just like "line X in file Y was hit" kind of thing--information that we already have in the InfosetInputter/Outputters and TRDs. It would be nice if we don't need to reinvent the wheel, especially if we could take advantage of tools that can make visual representations and code coverage analysis.
> I also wonder if maybe there's an approach similar to java debugging and how I _think_ code coverage works in Java, where we just enable a "java agent"? This could modify byte code (e.g. of an InfosetInputter/Outputter) to call functions at the appropriate time to create the code coverage reports without ever having to modify Daffodil itself--sort of a pluggable code coverage thing.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)