[jira] [Created] (SPARK-6029) Spark excludes "fastutil" dependencies of "clearspring" quantiles

["Jim Kleckner (JIRA)" <ji...@apache.org>]

Jim Kleckner created SPARK-6029:
-----------------------------------

             Summary: Spark excludes "fastutil" dependencies of "clearspring" quantiles
                 Key: SPARK-6029
                 URL: https://issues.apache.org/jira/browse/SPARK-6029
             Project: Spark
          Issue Type: Bug
          Components: Build
    Affects Versions: 1.2.1
            Reporter: Jim Kleckner


Spark includes the clearspring analytics package but intentionally excludes the dependencies of the fastutil package.

Spark includes parquet-column which includes fastutil and relocates it under parquet/ but creates a shaded jar file which is incomplete because it shades out some of the fastutil classes, notably Long2LongOpenHashMap, which is present in the fastutil jar file that parquet-column is referencing.

We are using more of the clearspring classes (e.g. QDigest) and those do depend on missing fastutil classes like Long2LongOpenHashMap.

Even though I add them to our assembly jar file, the class loader finds the spark assembly and we get runtime class loader errors when we try to use it.


The [documentaion|http://spark.apache.org/docs/1.2.0/configuration.html#runtime-environment] and possibly related issue [SPARK-939|https://issues.apache.org/jira/browse/SPARK-939] suggest arguments that I tried with spark-submit:
{code}
--conf spark.driver.userClassPathFirst=true \
--conf spark.executor.userClassPathFirst=true
{code}
but we still get the class not found error.

Could this be a bug with {{userClassPathFirst=true}}?  i.e. should it work?

In any case, would it be reasonable to not exclude the "fastutil" dependencies?

See email discussion [here|http://apache-spark-user-list.1001560.n3.nabble.com/Fwd-Spark-excludes-quot-fastutil-quot-dependencies-we-need-tt21812.html]




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org