[jira] [Commented] (BIGTOP-1682) Build BPS Spark against BigTop Artifacts

["RJ Nowling (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BIGTOP-1682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14326474#comment-14326474 ] 

RJ Nowling commented on BIGTOP-1682:
------------------------------------

Some discussion on IRC about the issue:

{noformat}
<cos1> see, the situation with relying on Bigtop Spark binaries and Spark published maven artifacts is exactly what we are trying to avoid, right?
<rnowling> cos1, I agree.  I see two challenges:
<cos1> it'd be more consistent if Spark artifacts are the ones that we use in the package?
<rnowling> 1. how do we guarantee compatibility with the versions in BigTop?
<rnowling> 2. how do we make it easy for users who want to use BPS outside of BigTop installs?
<rnowling> I think using BigTop artifacts instead of vanilla artifacts helps with #1
<rnowling> I see a few different options: 1. RPMs install local maven/ivy repos   2. Update BPS gradle to depend on BigTop gradle 3. BigTop publishes its JARs as a Maven repo somewhere
<rnowling> I like #2 best, especially if we users can set a flag for building against BigTop or maven artifacts
<rnowling> (which would solve the easy of use issuers for other deployment environments)
<rnowling> fourth option: I know that sbt allows for a directory of "user managed dependencies" (basically, you copy the JARs to that dir).  maybe groovy lets you do something similar?
<rnowling> so then we can just build BigTop JARs, put them in a directory, and in the build system, we can just point to that?
<rnowling> or we ignore the whole issue and keep doing what we're doing :)
<cos1> hmm...
<cos1> lemme think for a sec
<cos1> I think in #4 you meant gradle not groovy, right?
<cos1> Maven has such an option, I believe. So I'd imagine Gradle would too....
<rnowling> cos1, yeah, sorry, yes, gradle
<cos1> I like 2 as well
<cos1> I think it is less intrusive and quite transparent. The only tie would be a need to clone the source repo, but one has to do it anyway ;)
<rnowling> cos1, I like #2 as well but I would want to make it user selectable between vanilla artifacts (via maven central) and BigTop components.  in my particular case, I'm deploying BPS on Spark-only cluster as smoke tests
<rnowling> so I need to be able to build BPS spark using vanilla spark artifacts
{noformat}

> Build BPS Spark against BigTop Artifacts
> ----------------------------------------
>
>                 Key: BIGTOP-1682
>                 URL: https://issues.apache.org/jira/browse/BIGTOP-1682
>             Project: Bigtop
>          Issue Type: Improvement
>          Components: blueprints
>    Affects Versions: 0.9.0
>            Reporter: RJ Nowling
>
> As a demo application in BigTop, BPS Spark should be deployable to BigTop.  As a result, compatibility with the Spark artifact published by BigTop should be a priority.
> BPS Spark currently builds against upstream, vanilla artifacts published by upstream projects through Maven.  This could lead to a situation where BPS Spark is incompatible with the BigTop artifacts or deployments.
> Whatever solution we decide on should consider the following use cases:
> 1. Building & deploying BPS Spark against specific BigTop versions
> 2. Building & deploying BPS Spark against non-BigTop (e.g., upstream) versions
> 3. Ease of building / testing for developers



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)