sbt can grab dependency DFDL schema from git transparently

["Beckerle, Mike" <mb...@owlcyberdefense.com>]

I have a love/hate relationship with SBT, but today I love it, because of this feature I discovered today
which is very cool, and super useful because it allows DFDL schemas to be split into reusable modules easily.

SBT can automatically grab a github/git source release (or tag), download it, build it in a temp location (recursively using sbt), and use it in the build and test of dependent DFDL schema project. All transparently.

I have tried this with the PCAP DFDL Schema, which, due to recent factoring out of the ethernetIP DFDL schema, now depends on that separate ethernetIP DFDL schema.

The build.sbt for PCAP now looks like this:

name := "dfdl-pcap"

organization := "com.tresys"

version := "1.0.0"

scalaVersion := "2.12.14"

libraryDependencies ++= Seq(
  //
  // We want to get rid of this, because it requires a publishLocal
  // or binary server to have this dependency schema present.
  // instead we want to just pull the dependent schema from github
  //
  // "com.owlcyberdefense" % "dfdl-ethernetip" % "1.0.0",
  //
  "org.apache.daffodil" %% "daffodil-tdml-processor" % "3.1.0" % "test",
  "com.novocode" % "junit-interface" % "0.11" % "test",
  "junit" % "junit" % "4.12" % "test",
)

testOptions += Tests.Argument(TestFrameworks.JUnit, "-v")

crossPaths := false

//
// To express the dependency explicitly we have to create
// a project and use dependsOn
//
lazy val main = (project in file(".")).dependsOn(ethernetIP)

//
// This is a dependency on ethernetIP at github
// with a specific tag 1.0.0
//
lazy val ethernetIP =
  RootProject(uri("git://github.com/DFDLSchemas/ethernetIP.git#1.0.0"))

The magic is the last line. SBT knows how to deal with a git repository and tag as a dependency.

Now when I push any change to PCAP, the TravisCI run now works, because when TravisCI runs sbt it will fetch the dependent schema and put it on classpath.

I have verified that this will work with git locations other than github. Ex: this location works fine for a DI2E.net git schema:

lazy val main = (project in file(".")).dependsOn(geoUtil)

lazy val geoUtil =
  RootProject(uri("ssh://git@bitbucket.di2e.net:7999/dfdl/geoutil.git#1.1"))

This should make the continuous integration for DFDL Schemas work nicely.

In general, this avoids the need for a binary server for DFDL schema jars to provide sharing, or for users to fetch all the DFDL schemas on the dependency chain (transitively), do an sbt publishLocal on them, etc.

This also makes it possible to download a DFDL schema as a zip, unzip to a directory, and immediately 'sbt test' without having to worry about what other schemas it depends on, as they will be pulled automatically.

You can try this for PCAP. Try these steps:

  1.  Web Browse https://github.com/DFDLSchemas/PCAP
  2.  Click the green "Code" button and choose download zip.
  3.  Unzip into a directory.
  4.  cd to that directory
  5.  sbt test

This runs the tests, transparently pulling the ethernetIP DFDL schema from github for you.