Re: request for enhancement: compile, package and artifacts support for C++

[Ittay Dror <it...@gmail.com>]

Hi,

I want to continue this discussion, since I'm not at a point where I have
C++ working for me for 52 projects, and I want to make it more up to buildr
standards. 




Assaf Arkin wrote:
> 
> On Mon, Jul 28, 2008 at 2:42 AM, Ittay Dror <it...@gmail.com> wrote:
>> Hi,
>>
>> I'm working on adding C++ support to buildr. I already have a prototype
>> that
>> builds libraries and executables in Linux. I'd like to share some of the
>> difficulties I had and request changes to buildr to accommodate C++ more
>> easily. (Right now, I've created parallel route to that of building
>> Java-like code)
>>
>> compile
>> ========
>> overview
>> --------------------
>> the compile method in project returns a CompileTask that is generic and
>> uses
>> a Compiler instance to do the actual compilation. In C++, compilation is
>> also dependency based (.o => .cpp, sometimes precompiling headers). Also,
>> the same code can produce several results (static and shared libraries,
>> oj
>> files with debug, profiling, preprocessor defines turned on and off). [1]
>>
>> there is the 'build' task, which is used as a stub to attach dependencies
>> to.
>>
>> suggestion
>> ---------------------
>> * there should be an array of compile tasks (as in packages)
>> * #compile should delegate the call to a factory method which returns a
>> task
>> (again, as in packages)
> 
> Yes.  And I know a few people just waiting for the change to compile
> multiple things in the same project, so here's another reason for
> adding this feature.
> 
> But I have to warn you, it's not as simple as it looks, I took a stab
> at it before and deciding to downscale support to one compiler per
> project.  It's worth doing because a lot of languages would benefit
> from it, but that's also what makes it tricky.  I think it would be
> easier to get C support working without it first, and separately work
> on this feature and then improve C support using it.
> 
> 
I'm adding C++ support for a client project, introducing buildr in the
process. Since it replaces an existing build system, I must provide the same
functionality.

I was able quite easily to have several C++ artifacts from a single build.
However, I could not use the 'compile' API since it was not suitable for
C++, both for lacking methods (defining external includes for example) and 
because it can't produce separate artifacts (which is a must as stated
above). another issue is that I need to consider what platform the build is
running on which is not supported by how compile task considers what
compilers to use.

What I did was:
* add a 'make' method that accepts a list of classifiers.
* for each classifier,  it finds a factory method that returns a task
creating the appropriate artifact which it calls if the task does not
already exist (similar to how packages are managed)
* the return value is a delegator object that forwards all method
invocations to the tasks.

So the project can now have:
  make(:shared, :static).with(....) 
which will create two tasks and configure both with the same dependencies.
The buildfile can then also have
  make(:shared).with(...)
which will add a dependency to the shared task only.

Module dependency (either depending on another project or a third party
artifact) is done by writing my own 'artifacts' method which does the right
thing for C++.

What I'm mostly not happy with is that I had to write parallel
implementations to 'compile' and 'artifacts'. For the first, it is because
C++ requires different API, for the latter it is because dependency is done
differently. 

Ittay


Assaf Arkin wrote:
> 
> 
> 
>> * generic pre-requisites (like 'resources') should either be tacked on
>> 'build' (relying on order of prerequisites), or the compile task can be
>> defined to be a composite (that is, from the outside it is a single task,
>> but it can use other tasks to accomplish its job).
> 
> compile already is: resources is a prerequisite for compile, some
> other tasks (e.g. byte code enhancing) are tacked on to compile by
> enhancing it.
> 
> 
>> package & artifacts
>> =========
>> overview
>> ---------------
>> buildr has a cool concept that all dependencies (in 'compile.with') are
>> converted to tasks that are then simple rake dependencies. However, the
>> conversion is not generic enough. to compile C++ code against a
>> dependency
>> one needs 2 paths: a folder containing headers and another containing
>> libraries. To put this in a repository, these need to be packaged into
>> one
>> file. To use after pulling from the repository, one needs to unpack. So a
>> task representing a repository artifact is in fact an unzip task, that
>> depends on the 'Artifact' task to pull the package from a remote
>> repository.
> 
> Let's take Java for example, let's say we have a task that depends on
> the contents of another WAR.  Specifically the classes (in
> WEB-INF/classes) and libraries (WEB-INF/lib).  A generic unzipping
> artifact won't help much, you'll get the root path which is useless.
> You need the classes path for one, and each file in the lib (pointing
> to the directory itself does nothing interesting).  It won't work with
> EAR either, when you unzip those, you end up with a WAR which you need
> to unzip again.
> 
> But this hypothetical task that uses WAR could be smarter.  It
> understands the semantics of the packages it uses, and all these
> packages follow a common convention, so it only needs to unpack the
> portions of the WAR it cares about, it knows how to construct the
> relevant paths, one to class and one to every JAR inside the lib
> directory.
> 
> I think the same analogy applies to C packages.  If by convention you
> always use include and lib, you can unpack only the portion of the
> package you need, find the relevant paths and use them appropriately.
> 
> 
>> furthermore, when building against another project, there is no need to
>> pack
>> and unpack in the repository. one can simply use the artifacts produced
>> in
>> the 'build' phase of the other project.
> 
> Yes.  Right now it points to the package, which gets invoked and so
> packs everything, whether you need the packing or not.  You don't,
> however, have to unpack it, if you know the packaging type you can be
> smarter and go directly to the source.
> 
>>
>> finally, in C++ in many cases you rely on a system library.
>>
>> in all cases the resulting dependency is two-fold: on a include dir paths
>> and on a library paths. note that these do not necessarily reside under a
>> shared folder. for example, a dependency on another project may depend on
>> two include folders: one just a folder in the source tree, the other of
>> generated files in the target directory
>>
>> suggestion
>> -------------------
>> While usage of Buildr.artifacts is only as a utility method, so one can
>> easily write his own implementation and use that, I think it will be nice
>> to
>> be able to get some reuse.
>>
>> * when given a project, use it as is (not 'spec.packages'), or allow it
>> to
>> return its artifacts ('spec.artifacts').
> 
> Yes.  Except we're missing that whole dependency later (that's
> something 1.4 will add).  Ideally the project would have dependency
> lists it can populates (at least compile and runtime), and other
> projects can get these dependency lists and pick what they want.  So
> the compile dependency list would be the place to put headers and
> libraries, without having to package them.  We don't have that right
> now.
> 
> 
>> * if a symbol, recursively call on the spec from the namespace
>> * if a struct, recursively call
>> * otherwise, classify the artifact and call a factory method to create
>> it.
>> classification can be by packaging (e.g. jar). but actually, i don't have
>> a
>> very good idea here. note that for c++, there need to be a way of
>> defining
>> an artifact to look in the system for include files and libraries  (maybe
>> something like 'openssl:system'? - version and group ids are
>> meaningless).
>>  * the factory method can create different artifacts. for c++ there would
>> be
>> RepositoryArtifact (downloads and unpacks), ProjectArtifact (short
>> circuit
>> to the project's target and source directories) and SystemArtifact.
>>
>> I think that the use of artifact namespaces can help here as it allows to
>> create a more verbose syntax for declaring artifacts, while still
>> allowing
>> the user to create shorter names for them. (as an example in C++ it will
>> allow me to add to the artifact the list of flags to use when
>> compiling/linking with it, assuming they're not inherent to the artifact,
>> e.g. turn debug on). The factory method receives the artifact definition
>> (which can actually be defined by each plugin) and decides what to do
>> with
>> it.
> 
> 1.4 will have a better dependency mechanism, and one thing I looked at
> is associating meta-data with each dependency.  So perhaps that would
> address things like compiling/linking flags.
> 
>> I hope all this makes sense, and I'm looking forward to comments. I
>> intend
>> to share the code once I'm finished.
> 
> Unfortunately, the last time I wrote C code was over tens years ago,
> so my rustiness is showing.  I'm sure I missed some points because of
> that.
> 
> Assaf
> 
> 
>>
>> Thank you,
>> Ittay
>>
>>
>> Notes:
>> [1] I don't consider linking a library as packaging. First, the obj files
>> are not used by themselves as in other languages. Second, packaging is
>> required to manage dependencies, because in order for project P to be
>> built
>> against dependency D, D needs to contain both headers and libraries -
>> this
>> is the package.
>>
>> --
>> --
>> Ittay Dror <it...@gmail.com>
>>
>>
>>
> 
> 

-- 
View this message in context: http://www.nabble.com/request-for-enhancement%3A-compile%2C-package-and-artifacts-support-for-C%2B%2B-tp18687046p19250068.html
Sent from the Buildr - Dev mailing list archive at Nabble.com.