You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jungtaek Lim (JIRA)" <ji...@apache.org> on 2019/03/19 14:35:00 UTC
[jira] [Updated] (SPARK-27205) spark-shell with packages option
fails to load transitive dependencies even ivy successfully pulls jars
[ https://issues.apache.org/jira/browse/SPARK-27205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jungtaek Lim updated SPARK-27205:
---------------------------------
Description:
I found this bug while testing my patch regarding Spark SQL Kafka module - I tend to open spark-shell and link kafka module via `–packages`.
When we run
{code:java}
./bin/spark-shell --packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.4.0{code}
we should be able to import "org.apache.kafka" in spark-shell, but it doesn't work for current master branch.
There's not enough evidence as well as I have no idea what's happening here even with `–verbose` option, so I had to spend couple of hours dealing with git bisect.
Turned out the commit introducing the bug was SPARK-26977 ([81dd21fda99da48ed76adb739a07d1dabf1ffb51|https://github.com/apache/spark/commit/81dd21fda99da48ed76adb739a07d1dabf1ffb51]).
was:
I found this bug while testing my patch regarding Spark SQL Kafka module - I tend to open spark-shell and link kafka module via `–packages`.
When we run
{code:java}
./bin/spark-shell --packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.4.0{code}
we should be able to import "org.apache.kafka" in spark-shell, but it doesn't work for current master branch.
There's not enough evidence as well as I have no idea what's happening here even with `–verbose` option, so I had to spend couple of hours dealing with git bisect.
Turned out the commit introducing the bug was [81dd21fda99da48ed76adb739a07d1dabf1ffb51|https://github.com/apache/spark/commit/81dd21fda99da48ed76adb739a07d1dabf1ffb51].
> spark-shell with packages option fails to load transitive dependencies even ivy successfully pulls jars
> -------------------------------------------------------------------------------------------------------
>
> Key: SPARK-27205
> URL: https://issues.apache.org/jira/browse/SPARK-27205
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: Jungtaek Lim
> Priority: Major
>
> I found this bug while testing my patch regarding Spark SQL Kafka module - I tend to open spark-shell and link kafka module via `–packages`.
> When we run
> {code:java}
> ./bin/spark-shell --packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.4.0{code}
> we should be able to import "org.apache.kafka" in spark-shell, but it doesn't work for current master branch.
> There's not enough evidence as well as I have no idea what's happening here even with `–verbose` option, so I had to spend couple of hours dealing with git bisect.
> Turned out the commit introducing the bug was SPARK-26977 ([81dd21fda99da48ed76adb739a07d1dabf1ffb51|https://github.com/apache/spark/commit/81dd21fda99da48ed76adb739a07d1dabf1ffb51]).
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org