You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Yi Tian <ti...@gmail.com> on 2014/08/31 04:38:46 UTC
[SPARK-3324] make yarn module as a unified maven jar project
Hi everyone!
I found the YARN module has nonstandard path structure like:
${SPARK_HOME}
|--yarn
|--alpha (contains yarn api support for 0.23 and 2.0.x)
|--stable (contains yarn api support for 2.2 and later)
| |--pom.xml (spark-yarn)
|--common (Common codes not depending on specific version of Hadoop)
|--pom.xml (yarn-parent)
When we use maven to compile yarn module, maven will import 'alpha' or 'stable' module according to profile setting.
And the submodule like 'stable' use the build propertie defined in yarn/pom.xml to import common codes to sourcePath.
It will cause IntelliJ can't directly recognize sources in common directory as sourcePath.
I thought we should change the yarn module to a unified maven jar project,
and add specify different version of yarn api via maven profile setting.
I created a JIRA ticket: https://issues.apache.org/jira/browse/SPARK-3324
Any advice will be appreciated .
Re: [SPARK-3324] make yarn module as a unified maven jar project
Posted by Sean Owen <so...@cloudera.com>.
Yes, alpha and stable need to stay in two separate modules. I think
this is a little less standard than simply having three modules:
common, stable, alpha.
On Sun, Aug 31, 2014 at 1:32 PM, Yi Tian <ti...@gmail.com> wrote:
> Hi Sean
>
> Before compile-time, maven could dynamically add either stable or alpha source to the yarn/ project.
>
> So there are no incompatible at the compile-time.
>
> Here are an example:
>
> yarn/pom.xml
>
> <plugin>
> <groupId>org.codehaus.mojo</groupId>
> <artifactId>build-helper-maven-plugin</artifactId>
> <executions>
> <execution>
> <id>add-scala-sources</id>
> <phase>generate-sources</phase>
> <goals>
> <goal>add-source</goal>
> </goals>
> <configuration>
> <sources>
> <source>common/src/main/scala</source>
> <source>${yarn.api}/src/main/scala</source>
> </sources>
> </configuration>
> </execution>
> </executions>
> </plugin>
>
>
> On Aug 31, 2014, at 16:19, Sean Owen <so...@cloudera.com> wrote:
>
>> This isn't possible since the two versions of YARN are mutually
>> incompatible at compile-time. However see my comments about how this
>> could be restructured to be a little more standard, and so that
>> IntelliJ would parse it out of the box.
>>
>> Still I imagine it is not worth it if YARN alpha will go away at some
>> point and IntelliJ can easily be told where the extra src/ is.
>>
>> On Sun, Aug 31, 2014 at 3:38 AM, Yi Tian <ti...@gmail.com> wrote:
>>> Hi everyone!
>>>
>>> I found the YARN module has nonstandard path structure like:
>>>
>>> ${SPARK_HOME}
>>> |--yarn
>>> |--alpha (contains yarn api support for 0.23 and 2.0.x)
>>> |--stable (contains yarn api support for 2.2 and later)
>>> | |--pom.xml (spark-yarn)
>>> |--common (Common codes not depending on specific version of Hadoop)
>>> |--pom.xml (yarn-parent)
>>>
>>> When we use maven to compile yarn module, maven will import 'alpha' or 'stable' module according to profile setting.
>>> And the submodule like 'stable' use the build propertie defined in yarn/pom.xml to import common codes to sourcePath.
>>> It will cause IntelliJ can't directly recognize sources in common directory as sourcePath.
>>>
>>> I thought we should change the yarn module to a unified maven jar project,
>>> and add specify different version of yarn api via maven profile setting.
>>>
>>> I created a JIRA ticket: https://issues.apache.org/jira/browse/SPARK-3324
>>>
>>> Any advice will be appreciated .
>>>
>>>
>>>
>>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: [SPARK-3324] make yarn module as a unified maven jar project
Posted by Yi Tian <ti...@gmail.com>.
Hi Sean
Before compile-time, maven could dynamically add either stable or alpha source to the yarn/ project.
So there are no incompatible at the compile-time.
Here are an example:
yarn/pom.xml
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<executions>
<execution>
<id>add-scala-sources</id>
<phase>generate-sources</phase>
<goals>
<goal>add-source</goal>
</goals>
<configuration>
<sources>
<source>common/src/main/scala</source>
<source>${yarn.api}/src/main/scala</source>
</sources>
</configuration>
</execution>
</executions>
</plugin>
On Aug 31, 2014, at 16:19, Sean Owen <so...@cloudera.com> wrote:
> This isn't possible since the two versions of YARN are mutually
> incompatible at compile-time. However see my comments about how this
> could be restructured to be a little more standard, and so that
> IntelliJ would parse it out of the box.
>
> Still I imagine it is not worth it if YARN alpha will go away at some
> point and IntelliJ can easily be told where the extra src/ is.
>
> On Sun, Aug 31, 2014 at 3:38 AM, Yi Tian <ti...@gmail.com> wrote:
>> Hi everyone!
>>
>> I found the YARN module has nonstandard path structure like:
>>
>> ${SPARK_HOME}
>> |--yarn
>> |--alpha (contains yarn api support for 0.23 and 2.0.x)
>> |--stable (contains yarn api support for 2.2 and later)
>> | |--pom.xml (spark-yarn)
>> |--common (Common codes not depending on specific version of Hadoop)
>> |--pom.xml (yarn-parent)
>>
>> When we use maven to compile yarn module, maven will import 'alpha' or 'stable' module according to profile setting.
>> And the submodule like 'stable' use the build propertie defined in yarn/pom.xml to import common codes to sourcePath.
>> It will cause IntelliJ can't directly recognize sources in common directory as sourcePath.
>>
>> I thought we should change the yarn module to a unified maven jar project,
>> and add specify different version of yarn api via maven profile setting.
>>
>> I created a JIRA ticket: https://issues.apache.org/jira/browse/SPARK-3324
>>
>> Any advice will be appreciated .
>>
>>
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: [SPARK-3324] make yarn module as a unified maven jar project
Posted by Sean Owen <so...@cloudera.com>.
This isn't possible since the two versions of YARN are mutually
incompatible at compile-time. However see my comments about how this
could be restructured to be a little more standard, and so that
IntelliJ would parse it out of the box.
Still I imagine it is not worth it if YARN alpha will go away at some
point and IntelliJ can easily be told where the extra src/ is.
On Sun, Aug 31, 2014 at 3:38 AM, Yi Tian <ti...@gmail.com> wrote:
> Hi everyone!
>
> I found the YARN module has nonstandard path structure like:
>
> ${SPARK_HOME}
> |--yarn
> |--alpha (contains yarn api support for 0.23 and 2.0.x)
> |--stable (contains yarn api support for 2.2 and later)
> | |--pom.xml (spark-yarn)
> |--common (Common codes not depending on specific version of Hadoop)
> |--pom.xml (yarn-parent)
>
> When we use maven to compile yarn module, maven will import 'alpha' or 'stable' module according to profile setting.
> And the submodule like 'stable' use the build propertie defined in yarn/pom.xml to import common codes to sourcePath.
> It will cause IntelliJ can't directly recognize sources in common directory as sourcePath.
>
> I thought we should change the yarn module to a unified maven jar project,
> and add specify different version of yarn api via maven profile setting.
>
> I created a JIRA ticket: https://issues.apache.org/jira/browse/SPARK-3324
>
> Any advice will be appreciated .
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org