You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Ryan Williams (JIRA)" <ji...@apache.org> on 2015/01/07 01:12:34 UTC
[jira] [Updated] (SPARK-5115) Intellij fails to find hadoop classes
in Spark "yarn" modules
[ https://issues.apache.org/jira/browse/SPARK-5115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ryan Williams updated SPARK-5115:
---------------------------------
Description:
Intellij's parsing of Spark's POMs works like a charm for the most part, however it fails to resolve the hadoop and yarn dependencies in the Spark {{yarn}} and {{network/yarn}} modules.
Imports and later references to imported classes show up as errors, e.g.
!http://f.cl.ly/items/0g3w3s0t45402z30011l/Screen%20Shot%202015-01-06%20at%206.42.52%20PM.png!
Opening the module settings, we see that IntelliJ is looking for version {{1.0.4}} of [each yarn JAR that the Spark YARN module depends on|https://github.com/apache/spark/blob/bb38ebb1abd26b57525d7d29703fd449e40cd6de/yarn/pom.xml#L41-L56], and failing to find them:
!http://f.cl.ly/items/2d320l2h2o2N1m0t2X3b/yarn.png!
This, in turn, is due to the parent POM [defaulting {{hadoop.version}} to {{1.0.4}}|https://github.com/apache/spark/blob/bb38ebb1abd26b57525d7d29703fd449e40cd6de/pom.xml#L122].
AFAIK, having the default-hadoop-version be {{1.0.4}} is not that important and may just be an accident of history; people typically select a Maven profile when building Spark that matches the version of Hadoop that they intend to run with.
This suggests one possible fix: bump the default Hadoop version to >= 2. I've tried this locally and it resolves Intellij's difficulties with the "yarn" and "network/yarn" modules; [PR #3917|https://github.com/apache/spark/pull/3917] does this.
Another fix would be to declare a {{hadoop.version}} property in {{yarn/pom.xml}} and add that to the YARN dependencies in that file; [PR #3918|https://github.com/apache/spark/pull/3918] does that.
It is more obvious to me in the former case that the existing rules that govern what {{hadoop.version}} the YARN dependencies inherit will still apply.
For the latter, or potentially other ways to configure IntelliJ / Spark's POMs to address this issue, someone with more Maven/IntelliJ fu may need to chime in.
was:
Intellij's parsing of Spark's POMs works like a charm for the most part, however it fails to resolve the hadoop and yarn dependencies in the Spark {{yarn}} and {{network/yarn}} modules.
Imports and later references to imported classes show up as errors, e.g.
!http://f.cl.ly/items/0g3w3s0t45402z30011l/Screen%20Shot%202015-01-06%20at%206.42.52%20PM.png!
Opening the module settings, we see that IntelliJ is looking for version {{1.0.4}} of [each yarn JAR that the Spark YARN module depends on|https://github.com/apache/spark/blob/bb38ebb1abd26b57525d7d29703fd449e40cd6de/yarn/pom.xml#L41-L56], and failing to find them:
!http://f.cl.ly/items/2d320l2h2o2N1m0t2X3b/yarn.png!
This, in turn, is due to the parent POM [defaulting {{hadoop.version}} to {{1.0.4}}|https://github.com/apache/spark/blob/bb38ebb1abd26b57525d7d29703fd449e40cd6de/pom.xml#L122].
AFAIK, having the default-hadoop-version be {{1.0.4}} is not that important and may just be an accident of history; people typically select a Maven profile when building Spark that matches the version of Hadoop that they intend to run with.
This suggests one possible fix: bump the default Hadoop version to >= 2. I've tried this locally and it resolves Intellij's difficulties with the "yarn" and "network/yarn" modules; [PR #3917|https://github.com/apache/spark/pull/3917] does this.
Another fix would be to declare a {{hadoop.version}} property in {{yarn/pom.xml}} and add that to the YARN dependencies in that file; [PR #3918|https://github.com/apache/spark/pull/3918] does that.
It is more obvious to me in the former case that the existing rules that govern what {{hadoop.version}} the YARN dependencies inherit will still apply.
For the latter, or potentially other ways to configure IntelliJ / Spark's POMs, someone with more Maven/IntelliJ fu may need to chime in.
> Intellij fails to find hadoop classes in Spark "yarn" modules
> -------------------------------------------------------------
>
> Key: SPARK-5115
> URL: https://issues.apache.org/jira/browse/SPARK-5115
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Affects Versions: 1.2.0
> Reporter: Ryan Williams
>
> Intellij's parsing of Spark's POMs works like a charm for the most part, however it fails to resolve the hadoop and yarn dependencies in the Spark {{yarn}} and {{network/yarn}} modules.
> Imports and later references to imported classes show up as errors, e.g.
> !http://f.cl.ly/items/0g3w3s0t45402z30011l/Screen%20Shot%202015-01-06%20at%206.42.52%20PM.png!
> Opening the module settings, we see that IntelliJ is looking for version {{1.0.4}} of [each yarn JAR that the Spark YARN module depends on|https://github.com/apache/spark/blob/bb38ebb1abd26b57525d7d29703fd449e40cd6de/yarn/pom.xml#L41-L56], and failing to find them:
> !http://f.cl.ly/items/2d320l2h2o2N1m0t2X3b/yarn.png!
> This, in turn, is due to the parent POM [defaulting {{hadoop.version}} to {{1.0.4}}|https://github.com/apache/spark/blob/bb38ebb1abd26b57525d7d29703fd449e40cd6de/pom.xml#L122].
> AFAIK, having the default-hadoop-version be {{1.0.4}} is not that important and may just be an accident of history; people typically select a Maven profile when building Spark that matches the version of Hadoop that they intend to run with.
> This suggests one possible fix: bump the default Hadoop version to >= 2. I've tried this locally and it resolves Intellij's difficulties with the "yarn" and "network/yarn" modules; [PR #3917|https://github.com/apache/spark/pull/3917] does this.
> Another fix would be to declare a {{hadoop.version}} property in {{yarn/pom.xml}} and add that to the YARN dependencies in that file; [PR #3918|https://github.com/apache/spark/pull/3918] does that.
> It is more obvious to me in the former case that the existing rules that govern what {{hadoop.version}} the YARN dependencies inherit will still apply.
> For the latter, or potentially other ways to configure IntelliJ / Spark's POMs to address this issue, someone with more Maven/IntelliJ fu may need to chime in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org