Running native code in a heterogeneous DUCC cluster

[Burn Lewis <bu...@gmail.com>]

Since native code requires entries in java.library.path and LD_LIBRARY_PATH
that reference platform-specific libraries, it is currently impossible to
scale out a job to machines with different architectures, i.e. big- &
little-endian.  We are considering adding support for a special variable
that will be replaced at run-time with the value of the Java system
property "os.arch".  In this way the location of the native code could be
in architecture-specific directories which would be specified in the job
specification using this new variable.

The variable would be replaced in the parts of the job specification that
cannot be modified once the JVM has started, e.g. environment, JVM options,
and classpath.

The name could be DUCC_OS_ARCH but the syntax has to differ from the
${name} syntax used in the DUCC CLI options since its evaluation must be
delayed until the scaled-out process is launched.  Some choices are:

$[DUCC_OS_ARCH]      [DUCC_OS_ARCH]   #DUCC_OS_ARCH#    <DUCC_OS_ARCH>