Integrating a Native ARchive (NAR) into Apache Spark

[Erik O'Shaughnessy <er...@oracle.com>]

Hello, 

I’ve cobbled together a JNI interface and packaged it with the nar-maven-plugin, but I’m frankly
out of my depth with Java/Scala/Maven and could use some pointers about how to make my new JNI
object available for import in spark-shell.

My NAR project builds a simple Foo object that wraps some native libfoo calls. The unit tests
for the NAR indicate that my C APIs are being called successfully. I installed it into my local .m2
repository cache and was able to include it as a dependency in a simple test project which builds
and calls my shared library.  So far so good.

Wrong.

I’ve tried adding Foo as a dependency in spark/pom.xml and adding nar-maven-plugin to the build
section of the top level POM:

<dependencies>
 <dependency>
  <groupId>com.oracle.sae</groupId>
  <artifactId>Foo</artifactId>
  <version>1.0-SNAPSHOT</version>
  <type>nar</type>
 </dependency>
</dependencies>

<build>
 <plugins>
  <plugin>
    <groupId>com.github.maven-nar</groupId>
    <artifactId>nar-maven-plugin</artifactId>
    <version>3.4.1-SNAPSHOT-SAE</version>
    <extensions>true</extensions>
    <executions>
      <execution>
        <goals>
          <goal>nar-unpack</goal>
        </goals>
      </execution>
    </executions>
  <plugin>
  </plugin>
 </plugins>
</build>

I had to do some hacking on nar-maven-plugin to teach it about my platform/os/linker, which
is the reason for the goofy nar-maven-plugin version number. 

A spark build succeeds, but:

scala> import com.oracle.sae.Foo
<console>:23: error: object Foo is not a member of package com.oracle.sae


I tried adding my Foo maven project as a module in the top-level POM and moving my project
into the Apache Spark tree.  It built everything including my Foo project, however the Foo
object wasn’t available in spark-shell like I had hoped.

I’m a totally neophyte when it comes to the Maven ecosystem, however I’ve come a long way in
the last two weeks. I know it’s a weird request, but I would be most appreciative if someone
could provide some gentle hints about how to make my Foo object available within spark-shell.

Best regards,
Erik


Erik O'Shaughnessy - Principal Software Engineer
Strategic Application Engineering
Austin, TX 512-401-1070
erik.oshaughnessy@oracle.com


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org