Re: bulletproofing exec output

[Andrew Black <ab...@roguewave.com>]

Greetings Martin

I suppose I should elaborate on the third option I put forward, in a
more general manner.  One of the long-term plans for the exec utility,
as I understand it, was to allow for alternate output modes, such as an
'interactive' mode, which would include such sugar as colored output and
run timeout countdown and the verbose output mode.

What I picture is a structure where an arbitrary set of output mode
files can be dropped into the bin directory and compiled into the exec
utility.  It ideally would be possible for the makefiles to
automatically determine which files are output mode files, and each
output mode module would add itself to an internal list of mode modules
as part of the runtime initialization.  A certain number of modules
would ship with stdcxx (including the default and verbose modes, along
with the 'interactive' mode after it's written), and end users would be
able to write other modules of their own devising, if they wished.

With this structure, we would be able to write a module of our own,
designed for the nightly testing infrastructure, add it to the testing
source tree overlay, and specify that the exec utility should use the
new module, rather than the default output.

--Andrew Black

Martin Sebor wrote:
> Andrew Black wrote:
> [snipped descriptive summary]
> ...
>> Looking at the output (as it stands now), I believe the parsers could be
>> more robust if they ignored all lines that started with whitespace or
>> where the first word was in all caps.  Alternately, they could start
>> processing after the first line in all caps, and stop processing before
>> the second line in all caps.  I suppose a third option would be to alter
>> the exec utility so it could produce the output desired, rather than
>> doing a post-processing step.  Thoughts?
> 
> IMO, since exec already has all the data the post-processing stage
> works so hard to compute it might as well write it out in a format
> that's easy to parse by the script you're talking about. I.e., the
> summary would be it. The other script that I was referring to is
> the cross build view generation script that actually needs the
> individual examples, locales, and tests, and doesn't care (yet)
> about the summary.
> 
> So if take this approach we need to "standardize" both the output
> format for the individual programs as well as the summary. The
> summary format currently looks like this:
> 
[snipped sample output and initial problem]

Re: bulletproofing exec output

[Martin Sebor <se...@roguewave.com>]

Andrew Black wrote:
> Greetings Martin
> 
> I suppose I should elaborate on the third option I put forward, in a
> more general manner.  One of the long-term plans for the exec utility,
> as I understand it, was to allow for alternate output modes, such as an
> 'interactive' mode, which would include such sugar as colored output and
> run timeout countdown and the verbose output mode.

Yes. I just implemented the (a) verbose mode.

> 
> What I picture is a structure where an arbitrary set of output mode
> files can be dropped into the bin directory and compiled into the exec
> utility.  It ideally would be possible for the makefiles to
> automatically determine which files are output mode files, and each
> output mode module would add itself to an internal list of mode modules
> as part of the runtime initialization.  A certain number of modules
> would ship with stdcxx (including the default and verbose modes, along
> with the 'interactive' mode after it's written), and end users would be
> able to write other modules of their own devising, if they wished.
> 
> With this structure, we would be able to write a module of our own,
> designed for the nightly testing infrastructure, add it to the testing
> source tree overlay, and specify that the exec utility should use the
> new module, rather than the default output.

That still doesn't help with the coupling problem. We need to come
up with a format that makes scripts that parse it more resilient
to changes we make to the format in the future. The ideal solution
is for the script to dictate the exact format. A less than ideal
one but possibly good enough is to make the format robust enough
so that problems like the one we just ran into are unlikely to
come up. FWIW, I don't think that relying on capitalization is
the best approach. It's conceivable that someone will end up
"uncapitalizing" the strings, or that someone adds a test in all
caps, etc. But if we were to add some special header and footer
tags (as a simpler variation on the command line option specifying
the exact format of the output) it seems less likely that someone
would remove those, especially if they were controlled via a command
line option.

Martin

> 
> --Andrew Black
> 
> Martin Sebor wrote:
>> Andrew Black wrote:
>> [snipped descriptive summary]
>> ...
>>> Looking at the output (as it stands now), I believe the parsers could be
>>> more robust if they ignored all lines that started with whitespace or
>>> where the first word was in all caps.  Alternately, they could start
>>> processing after the first line in all caps, and stop processing before
>>> the second line in all caps.  I suppose a third option would be to alter
>>> the exec utility so it could produce the output desired, rather than
>>> doing a post-processing step.  Thoughts?
>> IMO, since exec already has all the data the post-processing stage
>> works so hard to compute it might as well write it out in a format
>> that's easy to parse by the script you're talking about. I.e., the
>> summary would be it. The other script that I was referring to is
>> the cross build view generation script that actually needs the
>> individual examples, locales, and tests, and doesn't care (yet)
>> about the summary.
>>
>> So if take this approach we need to "standardize" both the output
>> format for the individual programs as well as the summary. The
>> summary format currently looks like this:
>>
> [snipped sample output and initial problem]