[jira] [Created] (SPARK-27205) spark-shell with packages option fails to load transitive dependencies even ivy successfully pulls jars

["Jungtaek Lim (JIRA)" <ji...@apache.org>]

Jungtaek Lim created SPARK-27205:
------------------------------------

             Summary: spark-shell with packages option fails to load transitive dependencies even ivy successfully pulls jars
                 Key: SPARK-27205
                 URL: https://issues.apache.org/jira/browse/SPARK-27205
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 3.0.0
            Reporter: Jungtaek Lim


I found this bug while testing my patch regarding Spark SQL Kafka module - I tend to open spark-shell and link kafka module via `–packages`.

When we run
{code:java}
./bin/spark-shell --packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.4.0{code}
we should be able to import "org.apache.kafka" in spark-shell, but it doesn't work for current master branch.

There's not enough evidence as well as I have no idea what's happening here even with `–verbose` option, so I had to spend couple of hours dealing with git bisect.

Turned out the commit introducing the bug was [81dd21fda99da48ed76adb739a07d1dabf1ffb51|https://github.com/apache/spark/commit/81dd21fda99da48ed76adb739a07d1dabf1ffb51].

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org