[jira] [Commented] (BIGTOP-1906) Optimize and enahnce build/CI process to reuse artifacts

["Nate DAmico (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BIGTOP-1906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14593869#comment-14593869 ] 

Nate DAmico commented on BIGTOP-1906:
-------------------------------------

in regards to comments about single component breaking the whole process, think both for speed and management of the high number of components it would be optimal to look at grouping and prioritizing the full build into some "phases".., something like:

-bigtop core/base (whatever group votes "core/base" to be, mostly targeting pieces that most/all of other components need or work on)
    -hdfs
    -zookeeper
    -yarn
    -etc

-bigtop "hadoop tools"
    -pig
    -etc

-bigtop streaming
    -etc
    -etc

Dont know how granular we need to get.., but definitely to some level where the most important components can be building/releasing at a regular clip, not being blocked by things such as pig/sqoop/mahout/etc if they have some issues during changes/upgrade phases

> Optimize and enahnce build/CI process to reuse artifacts
> --------------------------------------------------------
>
>                 Key: BIGTOP-1906
>                 URL: https://issues.apache.org/jira/browse/BIGTOP-1906
>             Project: Bigtop
>          Issue Type: Sub-task
>          Components: build
>    Affects Versions: 1.0.0
>         Environment: all (host as well as any container based building)
>            Reporter: Nate DAmico
>              Labels: build, containers, maven
>             Fix For: 1.1.0
>
>
> This was generated from latest "future bigtop" thread on mailing list, capturing core details here from [~oflebbe]
> cc: [~evans_ye]
> ==============================================
> However we have a substantial problem hidden deep in th CI „2.0“ approach using containers
> You may know that we place artifacts (i.e. jars) we built with bigtop into the local maven cache ~/.m2. (look for mvn install in do-component-build). The idea is that later maven builds will pick these artifacts and use them rather downloading them from maven central.
> Placing artifacts into ~/.m2 will not have any effect if we use CI containers the way we do now: The maven cache ~/.m2 is lost when the container ends.
> [This triggered misfeature in JIRA https://issues.apache.org/jira/browse/BIGTOP-1893 , BTW:  gradle rpm/apt behaved differently from a container build with artifacts from maven central.]
> Option 1)  Remove mvn install from all do-component-builds
> Results:
> + We compile projects the way the upstream-developer does.
> - local fixes and configurations will not propagated
> Questions:
> If we do not try to reuse our build-artifacts within compile we have to ask ourself "why do we compile projects at all?“.
> We can build a great test wether someone else has touched / manipulated the maven central cache if we compare artifacts, but is this the really the point of compiling ourselves?
> Option 2) Use mvn install and reuse artifacts even in containers.
> Consequences:
> - Containers are not stateless any more
> - We have to add depencies to CI jobs so they run in order
> - single components may break the whole compile process.
> - Compile does not scale any more
> My Opinion:
> The way we do now "mvn install“ ,  simply tainting the maven cache seems not a really controlled way to propagate artifacts to me.
> Option 3) Use 1) but reuse artifacts in packages by placing symlinks and dependencies between them.
> - Packages will break with subtile problems if we do symlink artifacts  from different releases.
> ----
> Neither Option 1, Option 2 nor Option 3 seems a clever way to fix the problem. Would like to hear comments regarding this issue:
> In my humble opinion we should follow Option 2 with all the grave consequences. But maybe reworking mvn install by placing the artifacts with a bigtop specific name / groupid into the maven cache and upload them to maven central .
> Olaf



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)