How to handle transitive dependencies?

["Leidle, Rob" <le...@amazon.com>]

Hello all, I am running into a general problem with trying to use bigtop/maven to resolve dependencies. The common pattern seems to be, define your dependencies as properties, for example if your dependency is hadoop define a variable named “hadoop.version” and put a default version string into that property. In the Bigtop world, we override this value with a –Dhadoop.version=${HADOOP_VERSION} in the do-component-build script.

The problem is the scenario sort of listed here: http://stackoverflow.com/questions/12212440/overriding-maven-properties-on-command-line-maven-2-2-1. Namely, say I bring in a new version of Hadoop (let’s call it 2.6.0-rob) and my Bigtop has a version of Mahout which depends on a previous custom version of hadoop, let’s call it 2.4.0-rob. Also this bigtop has a project named “leidle-project" that depends on Mahout. I can build Mahout just fine with the –Dhadoop.version=2.6.0-rob, however, when I build my “leidle-project” which depends on Mahout it will try to bring in this 2.4.0-rob version of Hadoop as a transitive dependency of Mahout and the build will fail if maven cannot find it. In addition, I would like to produce a complete Maven repository output as a result of my Bigtop build and I would like all the projects in this repository to depend on the same Hadoop version.

An analogous situation in Bigtop is Hive which brings in HBase and Hadoop, which I believe has undergone several changes: https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hive/do-component-build

Any thoughts on this scenario? I could fix it by doing a search and replace in the pom file prior to the build but this seems clunky and problematic.