Re: design of regression tests

[Martin Sebor <se...@roguewave.com>]

Travis Vitek wrote:
>  
[...]
> I'm onboard with reducing the scope of this test to just verify
> MB_LEN_MAX and creating a new test for verifying all of the required
> limits.

Okay, let's do that then.

> 
> That said, I'm actually hoping to get feedback an the guts of the test
> itself. It feels a bit fragile to me because it compiles a separate
> executable to emulate getconf [for the necessary constants only], and to
> do this I had to hardcode the compiler names and flags into the test.
> I'm just hoping that this isn't something that will cause a bunch of
> trouble in the future. If it looks fragile to you guys, then maybe it
> would be best to just pass for windows builds and invoke the system
> getconf as you suggested earlier on other platforms.

I agree that it looks fragile, although I admit I am intrigued
by the idea of programmatically invoking the compiler. I have
been contemplating an API that would let us do this in general
(i.e., build programs or even libraries, either using stdcxx
or the native C++ Standard Library, or even C programs) but so
far it's just an idea.

Despite the fragility, I do see the appeal of this approach:
unlike computing the constants directly in the test that I was
at first inclined to suggest, it guarantees to yield the exact
same values as the C library. (I assume that's why you chose
it?) I suggest we keep the test in Jira and revisit it, maybe
along withe the whole compiler API idea, when we have more
time after the release.

Some observations about the formatting code in the test:

First, there's a macro for the "template <>" syntax to deal with
outdated compiler: _RWSTD_SPECIALIZED_FUNCTION. We have been
talking about dropping support for these kinds of workarounds at
some point in the (near) future but we haven't actually done it
yet. So until we do, we should continue to use these macros (if
nothing else, it'll help us all appreciate the extent of these
workarounds and decide which one we want to get rid of and which
one we might want to keep).

Second, this kind of type-based formatting is unnecessary. It would
be sufficient to convert each constant to long and use %li or %lu
to format them all. The formatting code is also slightly buggy
because it doesn't account for integer promotion and sign extension.
At least one of the conversion specifications (%hc) is also not
defined.

Martin

Re: design of regression tests

[Martin Sebor <se...@roguewave.com>]

Travis Vitek wrote:
>  
[...]
>> The formatting code is also slightly buggy
>> because it doesn't account for integer promotion and sign extension.
>>
> 
> I'm not sure what you mean by this. Can you give an example of how this
> would be a problem?

This was a thinko on my part. We're not concerned about the formatting
of arbitrary character values (e.g., '\x80' and unexpectedly getting
back -128 instead of 128 when char is a signed type) but just the two
values, CHAR_MAX and CHAR_MIN, and we do expect to see the negative
value when CHAR_MIN is negative. Sorry about that!

Martin

RE: design of regression tests

[Travis Vitek <tv...@quovadx.com>]

 
Martin Sebor wrote:
>
>I agree that it looks fragile, although I admit I am intrigued
>by the idea of programmatically invoking the compiler. I have
>been contemplating an API that would let us do this in general
>(i.e., build programs or even libraries, either using stdcxx
>or the native C++ Standard Library, or even C programs) but so
>far it's just an idea.
>
>Despite the fragility, I do see the appeal of this approach:
>unlike computing the constants directly in the test that I was
>at first inclined to suggest, it guarantees to yield the exact
>same values as the C library. (I assume that's why you chose
>it?) I suggest we keep the test in Jira and revisit it, maybe
>along withe the whole compiler API idea, when we have more
>time after the release.
>
>Some observations about the formatting code in the test:
>
>First, there's a macro for the "template <>" syntax to deal with
>outdated compiler: _RWSTD_SPECIALIZED_FUNCTION. We have been
>talking about dropping support for these kinds of workarounds at
>some point in the (near) future but we haven't actually done it
>yet. So until we do, we should continue to use these macros (if
>nothing else, it'll help us all appreciate the extent of these
>workarounds and decide which one we want to get rid of and which
>one we might want to keep).
>

Thanks, that may be useful for future reference.

>
>Second, this kind of type-based formatting is unnecessary. It would
>be sufficient to convert each constant to long and use %li or %lu
>to format them all. 
>

Yup, I realized that and wrote a new version of the test Friday night.
That simplifies things quite a bit.

>
>The formatting code is also slightly buggy
>because it doesn't account for integer promotion and sign extension.
>

I'm not sure what you mean by this. Can you give an example of how this
would be a problem?