You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by pwendell <gi...@git.apache.org> on 2014/04/23 09:46:04 UTC

[GitHub] spark pull request: SPARK-1119 and other build improvements

GitHub user pwendell opened a pull request:

    https://github.com/apache/spark/pull/502

    SPARK-1119 and other build improvements

    1. Makes assembly and examples jar naming consistent in maven/sbt.
    2. Updates make-distribution.sh to use Maven and fixes some bugs.
    3. Updates the create-release script to call make-distribution script.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/pwendell/spark make-distribution

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/502.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #502
    
----
commit 1a97f0ddafb28a065650f339d3971936e527533f
Author: Patrick Wendell <pw...@gmail.com>
Date:   2014-04-23T01:26:39Z

    SPARK-1119 and other build improvements
    
    1. Makes assembly and examples jar naming consistent in maven/sbt.
    2. Updates make-distribution.sh to use Maven and fixes some bugs.
    3. Updates the create-release script to call make-distribution script.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1119 and other build improvements

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/502


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1119 and other build improvements

Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:

    https://github.com/apache/spark/pull/502#issuecomment-41189201
  
    Okay merged.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1119 and other build improvements

Posted by berngp <gi...@git.apache.org>.
Github user berngp commented on the pull request:

    https://github.com/apache/spark/pull/502#issuecomment-41238386
  
    Yes it will, not every one uses the same version of HDFS vs YARN.
    
    On Wednesday, April 23, 2014, Li Guoqiang <no...@github.com> wrote:
    
    > @berngp <https://github.com/berngp> @pwendell<https://github.com/pwendell>,
    > Whether we can delete the yarn.version, only using hadoop.version. This
    > will cause any problems?
    >
    > —
    > Reply to this email directly or view it on GitHub<https://github.com/apache/spark/pull/502#issuecomment-41237973>
    > .
    >


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1119 and other build improvements

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/502#issuecomment-41133489
  
     Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1119 and other build improvements

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/502#issuecomment-41136742
  
    All automated tests passed.
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14365/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1119 and other build improvements

Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on the pull request:

    https://github.com/apache/spark/pull/502#issuecomment-41239453
  
    @berngp 
    Most of the people uses the same version of HDFS vs YARN.
    We can be so
    ```xml
       <hadoop.version>1.0.4</hadoop.version>
       <yarn.version>${hadoop.version}</yarn.version>
    ```
    ```xml
     <profile>
          <id>yarn-alpha</id>
          <properties>
            <hadoop.major.version>2</hadoop.major.version>   
            <hadoop.version>0.23.7</hadoop.version>
          </properties>
          <modules>
            <module>yarn</module>
          </modules>
          <dependencyManagement>
            <dependencies>
              <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-yarn-api</artifactId>
                <version>${yarn.version}</version>
              </dependency>
              <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-yarn-common</artifactId>
                <version>${yarn.version}</version>
              </dependency>
              <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-yarn-client</artifactId>
                <version>${yarn.version}</version>
              </dependency>
            </dependencies>
          </dependencyManagement>
        </profile>
    ```
    Most of the people use `mvn -Pyarn -Dhadoop.version=2.3.0 -DskipTests clean package`.
    Others use`mvn -Pyarn -Dhadoop.version=2.3.0 -DskipTests -Dyarn.version=0.23.9 clean package`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1119 and other build improvements

Posted by berngp <gi...@git.apache.org>.
Github user berngp commented on the pull request:

    https://github.com/apache/spark/pull/502#issuecomment-41241443
  
    Makes sense to me.
    
    On Wednesday, April 23, 2014, Li Guoqiang <no...@github.com> wrote:
    
    > @berngp <https://github.com/berngp>
    > Most of the people uses the same version of HDFS vs YARN.
    > We can be so
    >
    >    <hadoop.version>1.0.4</hadoop.version>
    >    <yarn.version>${hadoop.version}</yarn.version>
    >
    >  <profile>
    >       <id>yarn-alpha</id>
    >       <properties>
    >         <hadoop.major.version>2</hadoop.major.version>
    >         <hadoop.version>0.23.7</hadoop.version>
    >       </properties>
    >       <modules>
    >         <module>yarn</module>
    >       </modules>
    >       <dependencyManagement>
    >         <dependencies>
    >           <dependency>
    >             <groupId>org.apache.hadoop</groupId>
    >             <artifactId>hadoop-yarn-api</artifactId>
    >             <version>${yarn.version}</version>
    >           </dependency>
    >           <dependency>
    >             <groupId>org.apache.hadoop</groupId>
    >             <artifactId>hadoop-yarn-common</artifactId>
    >             <version>${yarn.version}</version>
    >           </dependency>
    >           <dependency>
    >             <groupId>org.apache.hadoop</groupId>
    >             <artifactId>hadoop-yarn-client</artifactId>
    >             <version>${yarn.version}</version>
    >           </dependency>
    >         </dependencies>
    >       </dependencyManagement>
    >     </profile>
    >
    > Most of the people use mvn -Pyarn -Dhadoop.version=2.3.0 -DskipTests
    > clean package.
    > Others usemvn -Pyarn -Dhadoop.version=2.3.0 -DskipTests
    > -Dyarn.version=0.23.9 clean package.
    >
    > —
    > Reply to this email directly or view it on GitHub<https://github.com/apache/spark/pull/502#issuecomment-41239453>
    > .
    >


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1119 and other build improvements

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/502#issuecomment-41136738
  
    Merged build finished. All automated tests passed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1119 and other build improvements

Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:

    https://github.com/apache/spark/pull/502#issuecomment-41189076
  
    @witgo we've been building releases with maven for several versions of spark due to better stability. We can look at #480 as well... 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1119 and other build improvements

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/502#issuecomment-41133498
  
    Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1119 and other build improvements

Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on the pull request:

    https://github.com/apache/spark/pull/502#issuecomment-41144046
  
    Why change using Maven build?  [The PR 480](https://github.com/apache/spark/pull/480)  has some relevant changes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1119 and other build improvements

Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on the pull request:

    https://github.com/apache/spark/pull/502#issuecomment-41237973
  
    @berngp @pwendell ,
    Whether we can delete the `yarn.version`, only using `hadoop.version`. This will cause any problems?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---