Re: [testing][cruise-control] how testing infrastructure should be improved

[Vladimir Ivanov <iv...@gmail.com>]

Issue HARMONY-3115 was created to track these changes.

 Thanks, Vladimir

On 1/31/07, Vladimir Ivanov <iv...@gmail.com> wrote:
> On 1/30/07, Geir Magnusson Jr. <ge...@pobox.com> wrote:
> >
> > On Jan 30, 2007, at 12:17 PM, Vladimir Ivanov wrote:
> >
> > > On 1/30/07, Geir Magnusson Jr. <ge...@pobox.com> wrote:
> > >>
> > >>
> > >> I think so.  I think we need better names that are descriptive rather
> > >> than relative, so we can add more w/o resorting to things like "ultra
> > >> medium short long" or whatever...
> > >>
> > >> so maybe :
> > >>
> > >> "build-profile" : just build classlib, drlvm and tools
> > >> "unit_profile" : runs the drlvm tests and classlib tests, can be used
> > >> after "build-profile"
> > >> "interative_profile" : can run after either the above, does iterative
> > >> testing
> > >> "functional1_profile" : ...
> > >> etc
> > >>
> > >> ?
> > >>
> > >> I don't know CC well enough - can I have these pre-set things, and
> > >> tell CC to do "build-profile" only, or do build-profile, and then if
> > >> that passes, do the unit-profile?
> > >
> > >
> > > Seems, that I also don't know CC as well as I want :(  As I
> > > understand it,
> > > CC operates with projects (in CC terms). Each project may depend on
> > > other
> > > project status. For example, in the current buildtest configuration
> > > drlvm
> > > build depends on classlib build status, run of drlvm tests depends
> > > on drlvm
> > > build status and run of classlib tests depends on classlib and
> > > drlvm builds
> > > status. So this dependency defined on the project level. In the
> > > case of
> > > different independent profiles seems them it will depends on
> > > classlib and
> > > drlvm builds only.
> >
> > One solution is to have a two step process - have modular "project-
> > lets"  for which there's a deployment step to produce a CC project file.
> >
> > So on my machine, I specify what I want, "generate", and then run...
> >
> Yes, the CC start process should stay without changes: setup step + run step.
>
>  thanks, Vladimir
>
> > geir
> >
> > >
> > >
> > >> geir
> > >>
> > >> >
> > >> > thanks, Vladimir
> > >> >
> > >> >
> > >> >
> > >> >> >
> > >> >> >>
> > >> >> >> >
> > >> >> >> > It requires some 'standard' interface for all integrated
> > >> >> scripts. I
> > >> >> >> > like
> > >> >> >> > classlib interface so how about:
> > >> >> >> >  - call of "ant setup;ant" will run all available scripts;
> > >> >> >> >  - call of "ant -Dmodule=hit setup;ant" will run current
> > >> >> version of
> > >> >> >> > CC –
> > >> >> >> > Harmony integration tests;
> > >> >> >> >  - call of "ant -Dmodule=eut setup;ant" will run Eclipse
> > >> >> unit test
> > >> >> >> > etc.
> > >> >> >> >
> > >> >> >>
> > >> >> >> > Note, in this case each module should implement proper
> > >> 'setup'
> > >> >> >> > target and
> > >> >> >> > has configuration for CC. The root-script will iterate
> > >> over all
> > >> >> >> > modules to
> > >> >> >> > call their 'setup' and this setup should include whole test
> > >> >> setup
> > >> >> >> > (downloading software, adding modules cc-configuration to
> > >> >> working
> > >> >> >> > configuration etc).
> > >> >> >> >
> > >> >> >> > Is it OK?
> > >> >> >>
> > >> >> >> I dunno - this sounds like disjoint and separate CC runs,
> > >> >> rather than
> > >> >> >> a CC run with multiple projects.
> > >> >> >>
> > >> >> >> For example, I'd like to have a set of "modules", which
> > >> would be
> > >> >> >> incomplete cc config files, that somehow get glommed into a
> > >> >> bigger cc
> > >> >> >> file - maybe the config.xml would have some kind of include
> > >> >> >> mechanism.
> > >> >> >>
> > >> >> >> Suppose that we have :
> > >> >> >>
> > >> >> >> trunk/cc/
> > >> >> >>     config.xml
> > >> >> >>     modules/
> > >> >> >>        default_module.xml
> > >> >> >>        hut_module.xml
> > >> >> >>        eut_module.xml
> > >> >> >>        iterative_module.xml
> > >> >> >>        dacapo_module.xml
> > >> >> >>        specjbb_module.xml
> > >> >> >>        short_module.xml
> > >> >> >>        medium_module.xml
> > >> >> >>        long_modules.xml
> > >> >> >>
> > >> >> >> so then I could do :
> > >> >> >>
> > >> >> >>   $ ant
> > >> >> >>
> > >> >> >> to get what we have now - runs the default module - or
> > >> >> >>
> > >> >> >>   $ ant -Dmodules=hut,eut,dacapo
> > >> >> >>
> > >> >> >> to run those...
> > >> >> >>
> > >> >> >> Something like "medium_module.xml" would look like  (the
> > >> >> following is
> > >> >> >> 'psuedo-code') :
> > >> >> >>
> > >> >> >>     <includeconfig name="default_module.xml"/>
> > >> >> >>     <includeconfig name="dacapo_module.xml"/>
> > >> >> >>
> > >> >> >>
> > >> >> >> so that you can nest this as you want.
> > >> >> >>
> > >> >> >> >
> > >> >> >> > If nobody objects I'll start restructuring of buildtest
> > >> >> module and
> > >> >> >> > will try
> > >> >> >> > to integrate one from extensions.
> > >> >> >>
> > >> >> >> Please describe how you want to do it.
> > >> >> >
> > >> >> >
> > >> >> > I think about following option: in the root file we have
> > >> predefined
> > >> >> > string.
> > >> >> > Something like 'modules=hut'. In the 'long' mode CC will iterate
> > >> >> > over all
> > >> >> > entries. The medium cycle depend on users wishes and defined
> > >> >> > through the
> > >> >> > command line like: 'ant -Dmodules=hut,iterative'. Each module
> > >> has
> > >> >> > predefined
> > >> >> > target (for example, 'setup'). In this target script should
> > >> >> > download all
> > >> >> > software, apply all patches, add module configuration to the
> > >> >> > current CC
> > >> >> > configuration (just copy content of the module configuration
> > >> file
> > >> >> > to the end
> > >> >> > of current configuration) etc. After 'ant <...> setup' we
> > >> will have
> > >> >> > ready-to-run system with user-defined configuration.
> > >> >> >
> > >> >> >
> > >> >> >> >
> > >> >> >> >
> > >> >> >> > Thanks, Vladimir
> > >> >> >> >
> > >> >> >> >
> > >> >> >> >
> > >> >> >> > PS I think the resulting structure should be easy to extend
> > >> >> and may
> > >> >> >> > looks
> > >> >> >> > like this:
> > >> >> >> >
> > >> >> >> > buildtest
> > >> >> >> >
> > >> >> >> > |--config  (default CC configuration to build classlib and
> > >> >> DRLVM)
> > >> >> >> >
> > >> >> >> > |--hit (CC configuration to run Harmony classlib&DRLVM tests)
> > >> >> >> >
> > >> >> >> > |--eclipse
> > >> >> >> >
> > >> >> >> >     |-- eut (setup and CC configuration to run eclipse non-
> > >> >> >> interactive
> > >> >> >> > tests)
> > >> >> >> >
> > >> >> >> >     |-- eclipse3.1.1
> > >> >> >> >
> > >> >> >> >         |-- some scenario
> > >> >> >> >
> > >> >> >> > |-- build.xml (common setup + call of module's 'setup')
> > >> >> >>
> > >> >> >>
> > >> >> >> Interesting.  I think one issue is that it seems like heavy
> > >> >> lifting
> > >> >> >> to add a new module - each module becomes a "peer".   What
> > >> do you
> > >> >> >> think of the other approach above?
> > >> >> >>
> > >> >> >> Either way, we don't want hit, eclipse, etc as peers.  If
> > >> >> anything,
> > >> >> >> they should go in a modules/ directory...
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> > For each module we have at least 2 files: cc-config and build
> > >> file.
> > >> >> > But in
> > >> >> > some cases we will have some additional files (patches etc). For
> > >> >> > example, script for testing of JEdit application (issue 3012)
> > >> has
> > >> >> > about 15
> > >> >> > files. For me is better to store all staff in one place
> > >> instead of
> > >> >> > having
> > >> >> > parallel structure.
> > >> >> >
> > >> >> > thanks, Vladimir
> > >> >> >
> > >> >> > geir
> > >> >> >>
> > >> >> >>
> > >> >>
> > >> >>
> > >>
> > >>
> >
> >
>