what counts as a failure?

[Martin Sebor <se...@roguewave.com>]

 From the reported results for our examples it seems
that we're counting as having failed those examples
that have no reference file (i.e., NREF). Those
should not be considered as failures.

Also, I see a number of examples whose status is
OUTPUT which I assume means their output didn't
match what was expected. I think the label for
this type of failure used to be DIFF and OUTPUT
was actually used for those that had no reference
file to compare the output with (i.e., what's now
being reported as NREF). I don't remember if we
deliberately made the decision to rename these
labels or if it was incidental. If the latter,
I think we should go back to the original codes.

Andrew?

Thanks
Martin

Re: what counts as a failure?

[Andrew Black <ab...@roguewave.com>]

Greetings Martin.

Marking NREF builds as good is simple enough (should be a one line 
change to the glue scripts).  Similarly, altering the output code 
translations should also be simple (Changing two strings in a lookup 
array in display.cpp).  The name change may have occurred during the 
effort to disambiguate what the different codes were.

I'm about ready to head out for the weekend, but I'll make these changes 
on Monday.

--Andrew Black

Martin Sebor wrote:
>  From the reported results for our examples it seems
> that we're counting as having failed those examples
> that have no reference file (i.e., NREF). Those
> should not be considered as failures.
> 
> Also, I see a number of examples whose status is
> OUTPUT which I assume means their output didn't
> match what was expected. I think the label for
> this type of failure used to be DIFF and OUTPUT
> was actually used for those that had no reference
> file to compare the output with (i.e., what's now
> being reported as NREF). I don't remember if we
> deliberately made the decision to rename these
> labels or if it was incidental. If the latter,
> I think we should go back to the original codes.
> 
> Andrew?
> 
> Thanks
> Martin