Re: stdcxx-13 - Windows configration

[Martin Sebor <se...@roguewave.com>]

Alex Ostapenko wrote:
 > Hello, Martin!
 > You wrote to <Al...@moscow.vdiweb.com> on Mon, 22 Aug 2005 17:50:36
 > -0600:
 >
 > ??>> We need to build on following platforms/configurations:
 > ??>> - Windows 2000/XP/2003 32-bit with MSVC 6, 7.0, 7.1 and future 8.0
 > ??>> compiler - Windows 2000/XP/2003 32-bit with Intel ICC compiler -
 > ??>> Windows 2003 64-bit with Windows 2003 SDK compiler - Windows 2003
 > ??>> 64-bit with Intel ICC compiler
 > ??>>
 > ??>> What about VC 8 for x64 and IA64 platforms?  Should they be 
supported?
 >
 > MS> Eventually. I expect people (i.e., the stdcxx community) to want to
 > MS> add other platforms as well (e.g., Como, Digital Mars, Watcom, etc.).
 >
 > Well, in this case the best option, as I have already said, would be
 > using something like Boost.Build. Otherwise people will need to invent
 > bicycles.

Relying on third party tools is not an option at this stage.

 > Although configuration script that I have written shall be easilty
 > exensible for multiple compilers it is quite probable that some of these
 > compilers may require additional changes.

That's great!

 >
 > MS> Even if we don't need to immediately accommodate any of these 
platforms
 > MS> now we should try to be as flexible as possible in order to make 
adding
 > MS> new ones or changing existing ones straightforward. This also means
 > MS> that there should be an easy way of adding, removing, or changing an
 > MS> existing compiler or linker option across all "projects" in the
 > MS> "solution" (I hope I'm using the correct Visual Studio terminology
 > MS> here).
 >
 > I am not sure that Visual Studio terminology is applicable here. Visual
 > Studio does not allow adding, removing or changing compilers without
 > writing special Visual Studio extensions.

That's where the other script to generate the VS Solution will come
in handy :)

 >
 > MS> As an implementation note, I've seen two drop-down boxes in the IDE:
 > MS> Configuration and Platform. It seems that Platform was designed to
 > MS> be used to select a different hardware target (e.g., Win32 or Win64)
 >
 > That is right. In VS 8 there are 3 platforms - Win32, Win64 x64 and
 > Win64 Itanium.
 >
 > MS> while Configuration to choose what we refer to as build type (e.g.,
 > MS> Debug vs Release). If so, it would make sense to extend Platform to
 > MS> include the compiler in addition to the hardware platform.
 >
 > What do you mean by "extend Platform" here? As I have already said
 > Visual Studio does not allow simple extension of its
 > Platform/Configuration model.

There's no way to add a fourth Platform and beyond? There is a way
to add another compiler, isn't there? (I sure hope so, otherwise
we might have quite a bit more work to do than I thought). By
extending Platform I meant adding Intel C++/Win32, Intel C++/Win64,
and Intel C++/Win64 x64 to what's already there for MSVC. If there
is no way to do that, we might just have to generate a different
solution for Intel C++.

 >
 > ??>> Could you explain why MS projects files and GUI is needed?
 >
 > MS> I haven't used Windows as my primary development environment for
 > MS> several years now but my assumption is that Windows programmers like
 > MS> to work within the Visual Studio framework (I did when I was one)
 > MS> and generally tend to avoid the command line (and on Windows they
 > MS> do so for a good reason! ;-)
 >
 > I guess things are not so simple. It is convinient to develop your own
 > projects in Visual Studio. But it is not so suitable for buiding 3rd
 > party libraries (which is almost always one time operation), nightly
 > builds, etc.

I used VS 5.0 to build libraries and executables that depended on
them and I don't recall any serious issues. I admit I never actually
tried to get automation working (or nightly builds), but I'm hopeful
that it won't be a showstopper. As long as dependcies among projects
work it should be possible to set up a system that will do what we
want (configure, build everything, and run it all to produce a
test report).

 > Many teams prefer command line build for this purposes.

So do we. Unfortunately (or perhaps thankfully), the existing Rogue
Wave build infrastructure (RCB) is not part of stdcxx so we cannot
rely on it. The goal of this exercise is to get both the convenience
of the VS Solutions and Project as well as the flexibility of the
command line, ideally in one neat package.

[...]
 > MS> By implementing the Visual Studio projects/solution, I would like
 > MS> to provide a convenient development environment for *contributors*
 > MS> to the stdcxx project. I'm less concerned with the *users* of the
 > MS> library since they only rarely build it, but if we do it the "right
 > MS> way" they might still benefit by being offered an example of a robust
 > MS> development framework on which they could model the development
 > MS> environment for their own applications.
 >
 > Well, that is the reason. However developers of the library could run
 > configuration script only once and then they could use a solution that
 > does not include configuration steps (Is that what you have talked about
 > in the previou e-mail?).

My idea is that developers would generate the Solution with the
configuration script as one of its Projects. They would configure
(and reconfigure after changes to config tests) the library within
the Solution simply by right clicking on the Project and selecting
the Build menu option (at least I think that's how it's done).

 >
[...]
 > ??>> We should be able to invoke the MSVC projects from command line 
or GUI
 > ??>> and that will start the configuration script, run the tests and
 > ??>> generate header, build libraries, locales, examples and tests (tests
 > ??>> to come in future version).
 > ??>>
 > ??>> Probably it will mean that we will have to have different 
projects for
 > ??>> each MSVC version. It will be hard to maintain.
 >
 > MS> Maintenance cost is important. It would be good to minimize it as
 > MS> much as possible. Writing a script to generate the projects and
 > MS> solutions for the different versions might be one way of dealing
 > MS> with the problem. The script input would be a set of compiler and
 > MS> linker options along the lines of the various .config files (such
 > MS> as gcc.config), the top of the stdcxx source tree (analogous to the
 > MS> TOPDIR make variable), and the target version of Visual Studio. The
 > MS> output would be a Visual Studio solution populated with a set of
 > MS> projects, one for the library, and one for every other stdcxx
 > MS> "component" such as each example, test, utility program, and locale,
 > MS> complete with the required Configuration options (build types) and
 > MS> Platform choices (compilers and hardware targets).
 >
 > It will be much harder that simply create a set of solutons from Visual
 > Studio. I think it would be better to create a solution and then
 > describe corresponding compiler, linker and librarian options in the
 > configuration script.

But then we'd have to maintain the compiler options in two different
places: one in each Project within the Solution and the other in the
configuration script. The risk of the two diverging after some time
is just too great (we have quite a bit of experience with this
problem and its conseuqences are usually really nasty).

 >
 > MS> If we take this approach as I suspect we might need to, the script
 > MS> will become *the* configuration and build infrastructure on Windows
 > MS> and the generated solution and projects will be just one of its
 > MS> (transient) products, just like the generated binaries. This approach
 > MS> will then obviate the need to store the project solution and all of
 > MS> its projects in SVN, or to distribute them to users.
 >
 > We could create the same solutions by hands from withing Visual Studio
 > and then store them to SVN.

We could, but with all the problems I mentioned.

 >
[...]
 >
 > MS> I think Andrew might have forwarded you his first tack at such
 > MS> a script written in Perl. You might want to start with it as a
 >
 > No, I have not seen it. Regardless, I have not studied Perl, so it would
 > not be very helpful.

The language doesn't matter. It's the concept that does. Let me
forward it to you privately so you can get an idea of how it
works and what the output is. I think Andrew implemented some
interesting things that might help you better understand what
I'm looking for.

 >
 > MS> reference and rewrite it in the scripting language appropriate
 > MS> for the project. I haven't looked at his script yet, or at its
 > MS> output, but some of the requirements on the final result is to
 > MS> be able to invoke it from with options or arguments telling
 > MS> Visual Studio to compile, link, and run the components of the
 > MS> project and to to collect the output of each stage in a single
 > MS> log file (for testing).
 >
 > Why not just to use ready solution to invoke build?

For all the reasons I mentioned above. I'm not completely ruling
this out but I would very, very much prefer not to have to deal
with the problems I'm concerned this approach would bring.

Martin

Re: stdcxx-13 - Windows configration

[Martin Sebor <se...@roguewave.com>]

Alex Ostapenko wrote:
[...]
> ??>> I am not sure that Visual Studio terminology is applicable here.
> ??>> Visual Studio does not allow adding, removing or changing compilers
> ??>> without writing special Visual Studio extensions.
> 
> MS> That's where the other script to generate the VS Solution will come
> MS> in handy :)
> 
> I am afraid that generation solutions for VS will not be an easy task 
> itself. All VS version have different project files structures (may be 
> except VS 7 and VS 7.1 those have similar project file format). Also, 
> adding support for a new VS version will require significant cahnges in 
> a script.

Yes, that's understood and considered an acceptable cost given the
relatively low frequency of incompatible VS releases.


> I think it would be better to create required project files 
> "by hands" from withing an IDE and use them as they are. This will also 
> minimize efforts on creating project files for a newer version of Visual 
> Studio.

As I said, the two big problems I see with rolling the VS Solutions
by hand is the duplication of compiler and linker options between
the config script and the Solution, and the tedium of manually having
to add each new Project (i.e., ever new example, unit test, or locale)
to the Solution.

FWIW, I initially considered the manual approach as well but wound up
rejecting it due to these two problems. If you can find a way to solve
them I might reconsider my position.

> Own VS converter could be used for that purposes.

That's an interesting idea that we should look into. Initially, let's
focus on VS 7.1. Once we have a working prototype we can investigate
the possible approaches for other versions.

> 
> ??>> What do you mean by "extend Platform" here? As I have already said
> ??>> Visual Studio does not allow simple extension of its
> ??>> Platform/Configuration model.
> 
> MS> There's no way to add a fourth Platform and beyond? There is a way
> MS> to add another compiler, isn't there? (I sure hope so, otherwise
> MS> we might have quite a bit more work to do than I thought). By
> 
> At least no simple way. AFAIK, adding own compiler requires writing a VS 
> add-on. Also, I have not seen any addons adding anything to 
> Platform/Configuration options.
> 
> MS> extending Platform I meant adding Intel C++/Win32, Intel C++/Win64,
> MS> and Intel C++/Win64 x64 to what's already there for MSVC. If there
> 
>  From reviews on the Intel site I have got an impression that only Intel 
> C++/Win32 could be integrated into VS. x64 compiler seems to be command 
> line only. Also, I have not seen anything about ICC for Itanium. Only 
> for x64/EMT64. Also, I do not know how ICC 32 integrates into VS. May be 
> it just has the same name (cl) and simply should be placed before MS cl 
> in the path.

I know they have a way to convert VS Solutions or Projects to use
their compiler but I don't know the details. I'll leave it for you
to investigate :) (But again, let's get a prototype working for VS
7.1 first).

> 
> MS> is no way to do that, we might just have to generate a different
> MS> solution for Intel C++.
> 
> That is probably what we will have to do.

Please look into the single Solution approach first. If it can't
be done, let's try this alternative.

> 
> ??>> Many teams prefer command line build for this purposes.
> 
> MS> So do we. Unfortunately (or perhaps thankfully), the existing Rogue
> MS> Wave build infrastructure (RCB) is not part of stdcxx so we cannot
> MS> rely on it. The goal of this exercise is to get both the convenience
> MS> of the VS Solutions and Project as well as the flexibility of the
> MS> command line, ideally in one neat package.
> 
> Having VS solutions we will have command line build for free. But other 
> compilers (like PSDK ones) will require writing special build scripts 
> for them.

Yes, that is a tradeoff, but I think it's an acceptable one. The vast
majority of developers will be happy using the VS Solutions and Projects
since they have the whole IDE. Developers whose development environment
is the bare SDK with no IDE are, IMO, an insignificant minority (remote
builds on an SDK-only target should be possible from the IDE on an IDE
equipped host). That said, we will eventually have to come up with a
workaround for automated testing, but let's worry about it after we're
done with the IDE-based prototype.

> 
> MS> [...]
> MS>>> By implementing the Visual Studio projects/solution, I would like
> MS>>> to provide a convenient development environment for *contributors*
> MS>>> to the stdcxx project. I'm less concerned with the *users* of the
> MS>>> library since they only rarely build it, but if we do it the "right
> MS>>> way" they might still benefit by being offered an example of a robust
> MS>>> development framework on which they could model the development
> MS>>> environment for their own applications.
> ??>>
> ??>> Well, that is the reason. However developers of the library could run
> ??>> configuration script only once and then they could use a solution that
> ??>> does not include configuration steps (Is that what you have talked
> ??>> about in the previou e-mail?).
> 
> MS> My idea is that developers would generate the Solution with the
> MS> configuration script as one of its Projects. They would configure
> MS> (and reconfigure after changes to config tests) the library within
> MS> the Solution simply by right clicking on the Project and selecting
> MS> the Build menu option (at least I think that's how it's done).
> 
> That will not work. If you are going to generate solution you will have 
> to close Visual Studio, generate solution from a command line and then 
> open it in a Visual Studio.

VS automation can be used to reload a Solution into it or add/remove
Projects from it (I have just seen an demo of this). But that wasn't
necessarily what I was talking about. The idea I was trying to
describe was a script, let's call it generate.bat, to generate the
VS Solution populated with a project to configure the library (call
it configure.bat), another to build the library (call it stdcxx.lib),
and one for each example program, unit test, utility program, and
locale. Generate.bat would be invoked first, with VS invoked next
on its result. An extension of this idea (now validated by the demo
I've just seen) is to have generate.bat be the only project of the
initially otherwise empty VS Solution. Invoking that Project would
populate the Solution with all the other Projects, all within the
same VS session.

> 
> ??>> It will be much harder that simply create a set of solutons from
> ??>> Visual Studio. I think it would be better to create a solution and
> ??>> then describe corresponding compiler, linker and librarian options in
> ??>> the configuration script.
> 
> MS> But then we'd have to maintain the compiler options in two different
> MS> places: one in each Project within the Solution and the other in the
> MS> configuration script. The risk of the two diverging after some time
> MS> is just too great (we have quite a bit of experience with this
> MS> problem and its conseuqences are usually really nasty).
> 
> That is right. But I do not think that risk is really high there.

In our experience it is unacceptably high. Having a single database
of compiler options is a must.

> Number 
> of options that could really cause discrepances between tests options 
> and options for library is really small (just a few optimization options 
> and a few runtime options). And I see no problems to make a contract on 
> changing them in both places simultaneously. Also, in general usage both 
> config script and solutions will be immutable, so no discrepances should 
> occur.

The Solutions will change on a regular basis. Every time a unit test
is added, the Solution will need to be updated. If we had just one
Solution, updating it manually wouldn't be a big deal, but with more
than one Solution likely, and with non-Solution approaches necessary
due to the absence of an IDE in the SDK, adding a test would require
a change in a number of places. It would then be quite easy for a
developer used to the IDE to forget to update one or all of the other
places (I've been there and done that :)

While I agree that compiler options change relatively infrequently,
the price of forgetting to change all the places where they are
duplicated is typically very high (think days of effort to find
out that a test is misbehaving in a mysterious way because the
library is misconfigured).

> Another point is that nothing could prevent user from changing options 
> in generated solution. And consequences could be the same as in the 
> previous case.

But only for the one generated session. The change would be reverted
by regenerating the Solution during automated testing. This might be
elegantly dealt with by a VS automation script that would detect
a change to the compiler or linker options made to the Solution and
offer to make the corresponding change to the central database of
options.

Martin

Re: stdcxx-13 - Windows configration

[Alex Ostapenko <al...@moscow.vdiweb.com>]

Hello, Martin!
You wrote to <st...@incubator.apache.org> on Wed, 24 Aug 2005 07:24:57 
+0400:

 ??>>>> What about VC 8 for x64 and IA64 platforms?  Should they be
 MS> supported?
 ??>>
 MS>>> Eventually. I expect people (i.e., the stdcxx community) to want to
 MS>>> add other platforms as well (e.g., Como, Digital Mars, Watcom, etc.).
 ??>>
 ??>> Well, in this case the best option, as I have already said, would be
 ??>> using something like Boost.Build. Otherwise people will need to invent
 ??>> bicycles.

 MS> Relying on third party tools is not an option at this stage.

OK.

 MS>>> Even if we don't need to immediately accommodate any of these
 MS> platforms
 MS>>> now we should try to be as flexible as possible in order to make
 MS> adding
 MS>>> new ones or changing existing ones straightforward. This also means
 MS>>> that there should be an easy way of adding, removing, or changing an
 MS>>> existing compiler or linker option across all "projects" in the
 MS>>> "solution" (I hope I'm using the correct Visual Studio terminology
 MS>>> here).
 ??>>
 ??>> I am not sure that Visual Studio terminology is applicable here.
 ??>> Visual Studio does not allow adding, removing or changing compilers
 ??>> without writing special Visual Studio extensions.

 MS> That's where the other script to generate the VS Solution will come
 MS> in handy :)

I am afraid that generation solutions for VS will not be an easy task 
itself. All VS version have different project files structures (may be 
except VS 7 and VS 7.1 those have similar project file format). Also, adding 
support for a new VS version will require significant cahnges in a script. I 
think it would be better to create required project files "by hands" from 
withing an IDE and use them as they are. This will also minimize efforts on 
creating project files for a newer version of Visual Studio. Own VS 
converter could be used for that purposes.

 ??>> What do you mean by "extend Platform" here? As I have already said
 ??>> Visual Studio does not allow simple extension of its
 ??>> Platform/Configuration model.

 MS> There's no way to add a fourth Platform and beyond? There is a way
 MS> to add another compiler, isn't there? (I sure hope so, otherwise
 MS> we might have quite a bit more work to do than I thought). By

At least no simple way. AFAIK, adding own compiler requires writing a VS 
add-on. Also, I have not seen any addons adding anything to 
Platform/Configuration options.

 MS> extending Platform I meant adding Intel C++/Win32, Intel C++/Win64,
 MS> and Intel C++/Win64 x64 to what's already there for MSVC. If there

>From reviews on the Intel site I have got an impression that only Intel 
C++/Win32 could be integrated into VS. x64 compiler seems to be command line 
only. Also, I have not seen anything about ICC for Itanium. Only for 
x64/EMT64. Also, I do not know how ICC 32 integrates into VS. May be it just 
has the same name (cl) and simply should be placed before MS cl in the path.

 MS> is no way to do that, we might just have to generate a different
 MS> solution for Intel C++.

That is probably what we will have to do.

 ??>> Many teams prefer command line build for this purposes.

 MS> So do we. Unfortunately (or perhaps thankfully), the existing Rogue
 MS> Wave build infrastructure (RCB) is not part of stdcxx so we cannot
 MS> rely on it. The goal of this exercise is to get both the convenience
 MS> of the VS Solutions and Project as well as the flexibility of the
 MS> command line, ideally in one neat package.

Having VS solutions we will have command line build for free. But other 
compilers (like PSDK ones) will require writing special build scripts for 
them.

 MS> [...]
 MS>>> By implementing the Visual Studio projects/solution, I would like
 MS>>> to provide a convenient development environment for *contributors*
 MS>>> to the stdcxx project. I'm less concerned with the *users* of the
 MS>>> library since they only rarely build it, but if we do it the "right
 MS>>> way" they might still benefit by being offered an example of a robust
 MS>>> development framework on which they could model the development
 MS>>> environment for their own applications.
 ??>>
 ??>> Well, that is the reason. However developers of the library could run
 ??>> configuration script only once and then they could use a solution that
 ??>> does not include configuration steps (Is that what you have talked
 ??>> about in the previou e-mail?).

 MS> My idea is that developers would generate the Solution with the
 MS> configuration script as one of its Projects. They would configure
 MS> (and reconfigure after changes to config tests) the library within
 MS> the Solution simply by right clicking on the Project and selecting
 MS> the Build menu option (at least I think that's how it's done).

That will not work. If you are going to generate solution you will have to 
close Visual Studio, generate solution from a command line and then open it 
in a Visual Studio.

 ??>> It will be much harder that simply create a set of solutons from
 ??>> Visual Studio. I think it would be better to create a solution and
 ??>> then describe corresponding compiler, linker and librarian options in
 ??>> the configuration script.

 MS> But then we'd have to maintain the compiler options in two different
 MS> places: one in each Project within the Solution and the other in the
 MS> configuration script. The risk of the two diverging after some time
 MS> is just too great (we have quite a bit of experience with this
 MS> problem and its conseuqences are usually really nasty).

That is right. But I do not think that risk is really high there. Number of 
options that could really cause discrepances between tests options and 
options for library is really small (just a few optimization options and a 
few runtime options). And I see no problems to make a contract on changing 
them in both places simultaneously. Also, in general usage both config 
script and solutions will be immutable, so no discrepances should occur.
Another point is that nothing could prevent user from changing options in 
generated solution. And consequences could be the same as in the previous 
case.

 MS> The language doesn't matter. It's the concept that does. Let me
 MS> forward it to you privately so you can get an idea of how it
 MS> works and what the output is. I think Andrew implemented some
 MS> interesting things that might help you better understand what
 MS> I'm looking for.

OK.

With best wishes,
Alex Ostapenko.