[jira] [Comment Edited] (TOREE-345) Spark 2.0.0 Fails when trying to use spark-avro

["Hollin Wilkins (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TOREE-345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15540697#comment-15540697 ] 

Hollin Wilkins edited comment on TOREE-345 at 10/2/16 5:30 PM:
---------------------------------------------------------------

Found out why the error happens, slf4j as a dependency to spark-avro causes it for some reason. When including spark-avro as a spark package via __TOREE_SPARK_OPTS__ in the kernel.json file, the error goes away because Spark Packages will automatically evict the slf4j dependency.

Output from including Spark-Avro via Spark Packages
{code}
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
	confs: [default]
	found com.databricks#spark-avro_2.11;3.0.1 in list
	found org.slf4j#slf4j-api;1.7.5 in list
	found org.apache.avro#avro;1.7.6 in list
	found org.codehaus.jackson#jackson-core-asl;1.9.13 in list
	found org.codehaus.jackson#jackson-mapper-asl;1.9.13 in list
	found com.thoughtworks.paranamer#paranamer;2.3 in list
	found org.xerial.snappy#snappy-java;1.0.5 in list
	found org.apache.commons#commons-compress;1.4.1 in list
	found org.tukaani#xz;1.0 in list
:: resolution report :: resolve 341ms :: artifacts dl 5ms
	:: modules in use:
	com.databricks#spark-avro_2.11;3.0.1 from list in [default]
	com.thoughtworks.paranamer#paranamer;2.3 from list in [default]
	org.apache.avro#avro;1.7.6 from list in [default]
	org.apache.commons#commons-compress;1.4.1 from list in [default]
	org.codehaus.jackson#jackson-core-asl;1.9.13 from list in [default]
	org.codehaus.jackson#jackson-mapper-asl;1.9.13 from list in [default]
	org.slf4j#slf4j-api;1.7.5 from list in [default]
	org.tukaani#xz;1.0 from list in [default]
	org.xerial.snappy#snappy-java;1.0.5 from list in [default]
	:: evicted modules:
	org.slf4j#slf4j-api;1.6.4 by [org.slf4j#slf4j-api;1.7.5] in [default]
{code}


was (Author: hollinwilkins):
Found out why the error happens, slf4j as a dependency to spark-avro causes it for some reason. When including spark-avro as a spark package via Spark options in the kernel.json file, the error goes away because Spark Packages will automatically evict the dlf4j dependency.

Output from including Spark-Avro via Spark Packages
{code}
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
	confs: [default]
	found com.databricks#spark-avro_2.11;3.0.1 in list
	found org.slf4j#slf4j-api;1.7.5 in list
	found org.apache.avro#avro;1.7.6 in list
	found org.codehaus.jackson#jackson-core-asl;1.9.13 in list
	found org.codehaus.jackson#jackson-mapper-asl;1.9.13 in list
	found com.thoughtworks.paranamer#paranamer;2.3 in list
	found org.xerial.snappy#snappy-java;1.0.5 in list
	found org.apache.commons#commons-compress;1.4.1 in list
	found org.tukaani#xz;1.0 in list
:: resolution report :: resolve 341ms :: artifacts dl 5ms
	:: modules in use:
	com.databricks#spark-avro_2.11;3.0.1 from list in [default]
	com.thoughtworks.paranamer#paranamer;2.3 from list in [default]
	org.apache.avro#avro;1.7.6 from list in [default]
	org.apache.commons#commons-compress;1.4.1 from list in [default]
	org.codehaus.jackson#jackson-core-asl;1.9.13 from list in [default]
	org.codehaus.jackson#jackson-mapper-asl;1.9.13 from list in [default]
	org.slf4j#slf4j-api;1.7.5 from list in [default]
	org.tukaani#xz;1.0 from list in [default]
	org.xerial.snappy#snappy-java;1.0.5 from list in [default]
	:: evicted modules:
	org.slf4j#slf4j-api;1.6.4 by [org.slf4j#slf4j-api;1.7.5] in [default]
{code}

> Spark 2.0.0 Fails when trying to use spark-avro
> -----------------------------------------------
>
>                 Key: TOREE-345
>                 URL: https://issues.apache.org/jira/browse/TOREE-345
>             Project: TOREE
>          Issue Type: Bug
>         Environment: Spark 2.0.0, Scala 2.11, Hadoop 2.7, Toree Git SHA: 7c1bfb6df7130477c558e69bbb518b0af364e06a
>            Reporter: Hollin Wilkins
>
> When trying to use the spark-avro project to load Avro files from Jupyter, we get errors.
> First:
> {code}
> %AddDeps com.databricks spark-avro_2.11 3.0.1 --transitive --trace
> {code}
> Then try to load an Avro file and show it:
> {code}
> spark.sqlContext.read.format("com.databricks.spark.avro").load("/tmp/test.avro").show()
> {code}
> And we get an error. I will attach the trace as a file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)