failed examples reported as passing with MSVC or vice versa

[Martin Sebor <se...@roguewave.com>]

While going through our MSVC 8.0 jumbled up log I came across
what looks like incorrect behavior of either the exec utility
or the Windows infrastructure. What seems to be happening is
that while exec reports some examples as passing the rest of
the infrastructure actually says they failed. Below are lines
from the log relevant to the accumulate.exe example.

Andrew and/or Farid, could you guys look into it to see which
part of the infrastructure is broken?

Thanks
Martin

422>Performing Custom Build Step
422>running accumulate.exe
422>NAME                      STATUS WARN ASSERTS FAILED PERCNT    USER 
     SYS    REAL
422>accumulate.exe                 0    0 
           0.330
422>example output differs from the expected
422>example failed! Exit code 0
422>

Re: failed examples reported as passing with MSVC or vice versa

[Andrew Black <ab...@roguewave.com>]

Martin Sebor wrote:
> Andrew Black wrote:
> 
> [...]
>> My reading of this message is that the runall.wsf script failed to 
>> complete a file system command.  Unfortunately, I am having 
>> difficulties determining where the failure occurred, as the line 
>> number is meaningless within the context of a text editor.  I tend to 
>> suspect that this translates to line 118 of the runall.wsf script.  A 
>> potential contributing cause is the permissions of the build directory 
>> used by the nightly testing system.
> 
> Permissions on Windows? That's something new! Could you look
> into this theory some more when you have a spare moment? We
> need to get those builds working at some point in the near
> future.

Greetings Martin.

I performed a manual (15d) build on our shared Windows 2003 AMD64 box, 
and I was able to replicate the behavior using the glue script used in 
the nightly build process.  However, this behavior wasn't replicated 
when I built the .examples project through the devenv GUI.  I need to 
try building the solution using the build_msvc-8.0-x64.bat script, but I 
am currently unable to due to fallout from some internal infrastructure 
changes.

--Andrew Black

> 
> Thanks!
> Martin

Re: failed examples reported as passing with MSVC or vice versa

[Martin Sebor <se...@roguewave.com>]

Andrew Black wrote:

[...]
> My reading of this message is that the runall.wsf script failed to 
> complete a file system command.  Unfortunately, I am having difficulties 
> determining where the failure occurred, as the line number is 
> meaningless within the context of a text editor.  I tend to suspect that 
> this translates to line 118 of the runall.wsf script.  A potential 
> contributing cause is the permissions of the build directory used by the 
> nightly testing system.

Permissions on Windows? That's something new! Could you look
into this theory some more when you have a spare moment? We
need to get those builds working at some point in the near
future.

Thanks!
Martin

> 
> --Andrew Black
> 
> Martin Sebor wrote:
> 
>> While going through our MSVC 8.0 jumbled up log I came across
>> what looks like incorrect behavior of either the exec utility
>> or the Windows infrastructure. What seems to be happening is
>> that while exec reports some examples as passing the rest of
>> the infrastructure actually says they failed. Below are lines
>> from the log relevant to the accumulate.exe example.
>>
>> Andrew and/or Farid, could you guys look into it to see which
>> part of the infrastructure is broken?
>>
>> Thanks
>> Martin
>>
>> 422>Performing Custom Build Step
>> 422>running accumulate.exe
>> 422>NAME                      STATUS WARN ASSERTS FAILED PERCNT    
>> USER     SYS    REAL
>> 422>accumulate.exe                 0    0           0.330
>> 422>example output differs from the expected
>> 422>example failed! Exit code 0
>> 422>

Re: failed examples reported as passing with MSVC or vice versa

[Andrew Black <ab...@roguewave.com>]

Greetings Martin.

My first impression is that the problem with the windows infrastructure 
rather than the exec utility.  It should be noted that the build in 
question (likely) is a 64 bit build, necessitating use of one of the 
shared 64 bit build machines.  I suspect the significant portion of the 
log is the few lines following those quoted.  Extracted from a candidate 
build, the lines in question are

422>d:\bman5\builds\33262615\source-buildspace\etc\config\windows\runall.wsf(59, 
6) Microsoft JScript runtime error: Permission denied
422>Project : error PRJ0019: A tool returned an error code from 
"Performing Custom Build Step"
422>Build log was saved at 
"file://D:\bman5\builds\33262615\source-buildspace\build\msvc-8.0-x64\15s\examples\BuildLog.htm"
422>.stdlib_examples - 1 error(s), 0 warning(s)

My reading of this message is that the runall.wsf script failed to 
complete a file system command.  Unfortunately, I am having difficulties 
determining where the failure occurred, as the line number is 
meaningless within the context of a text editor.  I tend to suspect that 
this translates to line 118 of the runall.wsf script.  A potential 
contributing cause is the permissions of the build directory used by the 
nightly testing system.

--Andrew Black

Martin Sebor wrote:
> While going through our MSVC 8.0 jumbled up log I came across
> what looks like incorrect behavior of either the exec utility
> or the Windows infrastructure. What seems to be happening is
> that while exec reports some examples as passing the rest of
> the infrastructure actually says they failed. Below are lines
> from the log relevant to the accumulate.exe example.
> 
> Andrew and/or Farid, could you guys look into it to see which
> part of the infrastructure is broken?
> 
> Thanks
> Martin
> 
> 422>Performing Custom Build Step
> 422>running accumulate.exe
> 422>NAME                      STATUS WARN ASSERTS FAILED PERCNT    USER 
>     SYS    REAL
> 422>accumulate.exe                 0    0           0.330
> 422>example output differs from the expected
> 422>example failed! Exit code 0
> 422>