You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by mgummelt <gi...@git.apache.org> on 2016/08/14 18:23:20 UTC

[GitHub] spark pull request #14637: [WIP] [SPARK-16967] move mesos to module

GitHub user mgummelt opened a pull request:

    https://github.com/apache/spark/pull/14637

    [WIP] [SPARK-16967] move mesos to module

    ## What changes were proposed in this pull request?
    
    Move Mesos code into a mvn module
    
    ## How was this patch tested?
    
    unit tests
    manually submitting a client mode and cluster mode job
    spark/mesos integration test suite
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/mesosphere/spark mesos-module

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/14637.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #14637
    
----
commit ef9c4815a98777d5172c491a4243cbd5839bfd2e
Author: Michael Gummelt <mg...@mesosphere.io>
Date:   2016-08-14T17:42:06Z

    move mesos to module

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64312/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64317/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    **[Test build #64430 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64430/consoleFull)** for PR 14637 at commit [`09f3197`](https://github.com/apache/spark/commit/09f3197e7cac9a45315bf5bdaed57c97bcd0e46d).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #14637: [WIP] [SPARK-16967] move mesos to module

Posted by mgummelt <gi...@git.apache.org>.

Github user mgummelt commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14637#discussion_r74706594
  
    --- Diff: mesos/pom.xml ---
    @@ -0,0 +1,167 @@
    +<?xml version="1.0" encoding="UTF-8"?>
    +<!--
    +  ~ Licensed to the Apache Software Foundation (ASF) under one or more
    +  ~ contributor license agreements.  See the NOTICE file distributed with
    +  ~ this work for additional information regarding copyright ownership.
    +  ~ The ASF licenses this file to You under the Apache License, Version 2.0
    +  ~ (the "License"); you may not use this file except in compliance with
    +  ~ the License.  You may obtain a copy of the License at
    +  ~
    +  ~    http://www.apache.org/licenses/LICENSE-2.0
    +  ~
    +  ~ Unless required by applicable law or agreed to in writing, software
    +  ~ distributed under the License is distributed on an "AS IS" BASIS,
    +  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +  ~ See the License for the specific language governing permissions and
    +  ~ limitations under the License.
    +  -->
    +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    +  <modelVersion>4.0.0</modelVersion>
    +  <parent>
    +    <groupId>org.apache.spark</groupId>
    +    <artifactId>spark-parent_2.11</artifactId>
    +    <version>2.1.0-SNAPSHOT</version>
    +    <relativePath>../pom.xml</relativePath>
    +  </parent>
    +
    +  <artifactId>spark-mesos_2.11</artifactId>
    +  <packaging>jar</packaging>
    +  <name>Spark Project Mesos</name>
    +  <properties>
    +    <sbt.project.name>mesos</sbt.project.name>
    +    <mesos.version>1.0.0</mesos.version>
    +    <mesos.classifier>shaded-protobuf</mesos.classifier>
    +  </properties>
    +
    +  <dependencies>
    +    <dependency>
    +      <groupId>org.apache.spark</groupId>
    +      <artifactId>spark-core_${scala.binary.version}</artifactId>
    +      <version>${project.version}</version>
    +    </dependency>
    +    <dependency>
    +      <groupId>org.apache.spark</groupId>
    +      <artifactId>spark-core_${scala.binary.version}</artifactId>
    +      <version>${project.version}</version>
    +      <type>test-jar</type>
    +      <scope>test</scope>
    +    </dependency>
    +    <dependency>
    +      <groupId>org.apache.mesos</groupId>
    +      <artifactId>mesos</artifactId>
    +      <version>${mesos.version}</version>
    +      <classifier>${mesos.classifier}</classifier>
    +      <exclusions>
    +        <exclusion>
    +          <groupId>com.google.protobuf</groupId>
    +          <artifactId>protobuf-java</artifactId>
    +        </exclusion>
    +      </exclusions>
    +    </dependency>
    +
    +
    +    <!-- Explicit listing of transitive deps that are shaded. Otherwise, odd compiler crashes.
    --- End diff --
    
    This is a WIP.  Please wait to comment.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [WIP] [SPARK-16967] move mesos to module

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #14637: [WIP] [SPARK-16967] move mesos to module

Posted by mgummelt <gi...@git.apache.org>.

Github user mgummelt commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14637#discussion_r75563505
  
    --- Diff: mesos/pom.xml ---
    @@ -0,0 +1,167 @@
    +<?xml version="1.0" encoding="UTF-8"?>
    +<!--
    +  ~ Licensed to the Apache Software Foundation (ASF) under one or more
    +  ~ contributor license agreements.  See the NOTICE file distributed with
    +  ~ this work for additional information regarding copyright ownership.
    +  ~ The ASF licenses this file to You under the Apache License, Version 2.0
    +  ~ (the "License"); you may not use this file except in compliance with
    +  ~ the License.  You may obtain a copy of the License at
    +  ~
    +  ~    http://www.apache.org/licenses/LICENSE-2.0
    +  ~
    +  ~ Unless required by applicable law or agreed to in writing, software
    +  ~ distributed under the License is distributed on an "AS IS" BASIS,
    +  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +  ~ See the License for the specific language governing permissions and
    +  ~ limitations under the License.
    +  -->
    +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    --- End diff --
    
    I don't think line length rules apply to XML.  See the other poms for examples.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [WIP] [SPARK-16967] move mesos to module

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    **[Test build #64109 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64109/consoleFull)** for PR 14637 at commit [`95e473b`](https://github.com/apache/spark/commit/95e473ba046298d5669dfe63264fc139d600b787).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [WIP] [SPARK-16967] move mesos to module

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63753/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [WIP] [SPARK-16967] move mesos to module

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63765/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    **[Test build #64303 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64303/consoleFull)** for PR 14637 at commit [`05614b4`](https://github.com/apache/spark/commit/05614b4f8f40c08cb30993a6aa236c6e4cc57276).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #14637: [WIP] [SPARK-16967] move mesos to module

Posted by jaceklaskowski <gi...@git.apache.org>.

Github user jaceklaskowski commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14637#discussion_r74706494
  
    --- Diff: mesos/pom.xml ---
    @@ -0,0 +1,167 @@
    +<?xml version="1.0" encoding="UTF-8"?>
    +<!--
    +  ~ Licensed to the Apache Software Foundation (ASF) under one or more
    --- End diff --
    
    I think the project uses `*` (stars) for continuation lines.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [WIP] [SPARK-16967] move mesos to module

Posted by mgummelt <gi...@git.apache.org>.

Github user mgummelt commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    @srowen Ready for review.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    **[Test build #64312 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64312/consoleFull)** for PR 14637 at commit [`cdc5753`](https://github.com/apache/spark/commit/cdc5753d9813f3358625bec1c674f54e0d69835e).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #14637: [SPARK-16967] move mesos to module

Posted by mgummelt <gi...@git.apache.org>.

Github user mgummelt commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14637#discussion_r75926093
  
    --- Diff: mesos/pom.xml ---
    @@ -0,0 +1,109 @@
    +<?xml version="1.0" encoding="UTF-8"?>
    +<!--
    +  ~ Licensed to the Apache Software Foundation (ASF) under one or more
    +  ~ contributor license agreements.  See the NOTICE file distributed with
    +  ~ this work for additional information regarding copyright ownership.
    +  ~ The ASF licenses this file to You under the Apache License, Version 2.0
    +  ~ (the "License"); you may not use this file except in compliance with
    +  ~ the License.  You may obtain a copy of the License at
    +  ~
    +  ~    http://www.apache.org/licenses/LICENSE-2.0
    +  ~
    +  ~ Unless required by applicable law or agreed to in writing, software
    +  ~ distributed under the License is distributed on an "AS IS" BASIS,
    +  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +  ~ See the License for the specific language governing permissions and
    +  ~ limitations under the License.
    +  -->
    +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    +  <modelVersion>4.0.0</modelVersion>
    +  <parent>
    +    <groupId>org.apache.spark</groupId>
    +    <artifactId>spark-parent_2.11</artifactId>
    +    <version>2.1.0-SNAPSHOT</version>
    +    <relativePath>../pom.xml</relativePath>
    +  </parent>
    +
    +  <artifactId>spark-mesos_2.11</artifactId>
    +  <packaging>jar</packaging>
    +  <name>Spark Project Mesos</name>
    +  <properties>
    +    <sbt.project.name>mesos</sbt.project.name>
    +    <mesos.version>1.0.0</mesos.version>
    +    <mesos.classifier>shaded-protobuf</mesos.classifier>
    +  </properties>
    +
    +  <dependencies>
    +    <dependency>
    +      <groupId>org.apache.spark</groupId>
    +      <artifactId>spark-core_${scala.binary.version}</artifactId>
    +      <version>${project.version}</version>
    +    </dependency>
    +
    +    <dependency>
    +      <groupId>org.apache.spark</groupId>
    +      <artifactId>spark-core_${scala.binary.version}</artifactId>
    +      <version>${project.version}</version>
    +      <type>test-jar</type>
    +      <scope>test</scope>
    +    </dependency>
    +
    +    <dependency>
    +      <groupId>org.apache.mesos</groupId>
    +      <artifactId>mesos</artifactId>
    +      <version>${mesos.version}</version>
    +      <classifier>${mesos.classifier}</classifier>
    +      <exclusions>
    +        <exclusion>
    +          <groupId>com.google.protobuf</groupId>
    +          <artifactId>protobuf-java</artifactId>
    +        </exclusion>
    +      </exclusions>
    +    </dependency>
    +
    +    <dependency>
    --- End diff --
    
    I just now removed mockito, because it was already a dependency of the parent project.
    
    I'm pretty sure all of the shaded deps need to be listed.  I don't know why they're shaded in the first place, but since they are, they must be listed here to avoid classloading errors at runtime.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [WIP] [SPARK-16967] move mesos to module

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [WIP] [SPARK-16967] move mesos to module

Posted by srowen <gi...@git.apache.org>.

Github user srowen commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    Good call, I think it's dev/sparktests/modules.py that needs to be modified. it can clone YARN's config.
    
    I assume that the release binaries will have this profile activated as with YARN.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    **[Test build #64301 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64301/consoleFull)** for PR 14637 at commit [`e94f5e7`](https://github.com/apache/spark/commit/e94f5e7c9492ec84b1e2b786f4a2f749eb3250c4).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    **[Test build #64312 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64312/consoleFull)** for PR 14637 at commit [`cdc5753`](https://github.com/apache/spark/commit/cdc5753d9813f3358625bec1c674f54e0d69835e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [WIP] [SPARK-16967] move mesos to module

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    **[Test build #64430 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64430/consoleFull)** for PR 14637 at commit [`09f3197`](https://github.com/apache/spark/commit/09f3197e7cac9a45315bf5bdaed57c97bcd0e46d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64430/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    **[Test build #64302 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64302/consoleFull)** for PR 14637 at commit [`fe7d05e`](https://github.com/apache/spark/commit/fe7d05e94a7cc55c57ba14b041ad3ce07139878e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #14637: [SPARK-16967] move mesos to module

Posted by vanzin <gi...@git.apache.org>.

Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14637#discussion_r75930641
  
    --- Diff: mesos/pom.xml ---
    @@ -0,0 +1,109 @@
    +<?xml version="1.0" encoding="UTF-8"?>
    +<!--
    +  ~ Licensed to the Apache Software Foundation (ASF) under one or more
    +  ~ contributor license agreements.  See the NOTICE file distributed with
    +  ~ this work for additional information regarding copyright ownership.
    +  ~ The ASF licenses this file to You under the Apache License, Version 2.0
    +  ~ (the "License"); you may not use this file except in compliance with
    +  ~ the License.  You may obtain a copy of the License at
    +  ~
    +  ~    http://www.apache.org/licenses/LICENSE-2.0
    +  ~
    +  ~ Unless required by applicable law or agreed to in writing, software
    +  ~ distributed under the License is distributed on an "AS IS" BASIS,
    +  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +  ~ See the License for the specific language governing permissions and
    +  ~ limitations under the License.
    +  -->
    +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    +  <modelVersion>4.0.0</modelVersion>
    +  <parent>
    +    <groupId>org.apache.spark</groupId>
    +    <artifactId>spark-parent_2.11</artifactId>
    +    <version>2.1.0-SNAPSHOT</version>
    +    <relativePath>../pom.xml</relativePath>
    +  </parent>
    +
    +  <artifactId>spark-mesos_2.11</artifactId>
    +  <packaging>jar</packaging>
    +  <name>Spark Project Mesos</name>
    +  <properties>
    +    <sbt.project.name>mesos</sbt.project.name>
    +    <mesos.version>1.0.0</mesos.version>
    +    <mesos.classifier>shaded-protobuf</mesos.classifier>
    +  </properties>
    +
    +  <dependencies>
    +    <dependency>
    +      <groupId>org.apache.spark</groupId>
    +      <artifactId>spark-core_${scala.binary.version}</artifactId>
    +      <version>${project.version}</version>
    +    </dependency>
    +
    +    <dependency>
    +      <groupId>org.apache.spark</groupId>
    +      <artifactId>spark-core_${scala.binary.version}</artifactId>
    +      <version>${project.version}</version>
    +      <type>test-jar</type>
    +      <scope>test</scope>
    +    </dependency>
    +
    +    <dependency>
    +      <groupId>org.apache.mesos</groupId>
    +      <artifactId>mesos</artifactId>
    +      <version>${mesos.version}</version>
    +      <classifier>${mesos.classifier}</classifier>
    +      <exclusions>
    +        <exclusion>
    +          <groupId>com.google.protobuf</groupId>
    +          <artifactId>protobuf-java</artifactId>
    +        </exclusion>
    +      </exclusions>
    +    </dependency>
    +
    +    <dependency>
    --- End diff --
    
    They're shaded because people want to use different versions of guava and jetty.
    
    Spark doesn't need to promote them to compile scope because the relocated guava / jetty classes are packaged with Spark itself.
    
    I see what they problem you're running into is now. It has nothing to do with compile scope, you just need to explicit list the dependencies because the mesos code references them directly. So you actually just need to fix the comment since you're not promoting anything to compile scope by explicitly adding the dependencies (they still inherit the scope from the parent pom, which is "provided", and is actually what you want here).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [WIP] [SPARK-16967] move mesos to module

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    **[Test build #63765 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63765/consoleFull)** for PR 14637 at commit [`4e276ab`](https://github.com/apache/spark/commit/4e276ab81848669ec5c7da6d51cf63a46b1bac87).
     * This patch **fails MiMa tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #14637: [WIP] [SPARK-16967] move mesos to module

Posted by jaceklaskowski <gi...@git.apache.org>.

Github user jaceklaskowski commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14637#discussion_r74706502
  
    --- Diff: mesos/pom.xml ---
    @@ -0,0 +1,167 @@
    +<?xml version="1.0" encoding="UTF-8"?>
    +<!--
    +  ~ Licensed to the Apache Software Foundation (ASF) under one or more
    +  ~ contributor license agreements.  See the NOTICE file distributed with
    +  ~ this work for additional information regarding copyright ownership.
    +  ~ The ASF licenses this file to You under the Apache License, Version 2.0
    +  ~ (the "License"); you may not use this file except in compliance with
    +  ~ the License.  You may obtain a copy of the License at
    +  ~
    +  ~    http://www.apache.org/licenses/LICENSE-2.0
    +  ~
    +  ~ Unless required by applicable law or agreed to in writing, software
    +  ~ distributed under the License is distributed on an "AS IS" BASIS,
    +  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +  ~ See the License for the specific language governing permissions and
    +  ~ limitations under the License.
    +  -->
    +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    --- End diff --
    
    Isn't the line too long? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #14637: [SPARK-16967] move mesos to module

Posted by mgummelt <gi...@git.apache.org>.

Github user mgummelt commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14637#discussion_r75929868
  
    --- Diff: mesos/pom.xml ---
    @@ -0,0 +1,109 @@
    +<?xml version="1.0" encoding="UTF-8"?>
    +<!--
    +  ~ Licensed to the Apache Software Foundation (ASF) under one or more
    +  ~ contributor license agreements.  See the NOTICE file distributed with
    +  ~ this work for additional information regarding copyright ownership.
    +  ~ The ASF licenses this file to You under the Apache License, Version 2.0
    +  ~ (the "License"); you may not use this file except in compliance with
    +  ~ the License.  You may obtain a copy of the License at
    +  ~
    +  ~    http://www.apache.org/licenses/LICENSE-2.0
    +  ~
    +  ~ Unless required by applicable law or agreed to in writing, software
    +  ~ distributed under the License is distributed on an "AS IS" BASIS,
    +  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +  ~ See the License for the specific language governing permissions and
    +  ~ limitations under the License.
    +  -->
    +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    +  <modelVersion>4.0.0</modelVersion>
    +  <parent>
    +    <groupId>org.apache.spark</groupId>
    +    <artifactId>spark-parent_2.11</artifactId>
    +    <version>2.1.0-SNAPSHOT</version>
    +    <relativePath>../pom.xml</relativePath>
    +  </parent>
    +
    +  <artifactId>spark-mesos_2.11</artifactId>
    +  <packaging>jar</packaging>
    +  <name>Spark Project Mesos</name>
    +  <properties>
    +    <sbt.project.name>mesos</sbt.project.name>
    +    <mesos.version>1.0.0</mesos.version>
    +    <mesos.classifier>shaded-protobuf</mesos.classifier>
    +  </properties>
    +
    +  <dependencies>
    +    <dependency>
    +      <groupId>org.apache.spark</groupId>
    +      <artifactId>spark-core_${scala.binary.version}</artifactId>
    +      <version>${project.version}</version>
    +    </dependency>
    +
    +    <dependency>
    +      <groupId>org.apache.spark</groupId>
    +      <artifactId>spark-core_${scala.binary.version}</artifactId>
    +      <version>${project.version}</version>
    +      <type>test-jar</type>
    +      <scope>test</scope>
    +    </dependency>
    +
    +    <dependency>
    +      <groupId>org.apache.mesos</groupId>
    +      <artifactId>mesos</artifactId>
    +      <version>${mesos.version}</version>
    +      <classifier>${mesos.classifier}</classifier>
    +      <exclusions>
    +        <exclusion>
    +          <groupId>com.google.protobuf</groupId>
    +          <artifactId>protobuf-java</artifactId>
    +        </exclusion>
    +      </exclusions>
    +    </dependency>
    +
    +    <dependency>
    --- End diff --
    
    YARN does it: https://github.com/apache/spark/blob/master/yarn/pom.xml#L79
    
    which is what I was using as a starting point.  I initially did try to remove them, but got a ClassNotFoundException in WebUI.class when trying to import org.eclipse.jetty.<something>
    
    The parent project seems to rename org.eclipse.jetty to org.spark_project.jetty https://github.com/apache/spark/blob/master/pom.xml#L2273
    
    It's documented that the shaded deps should be put into compile scope in the module: https://github.com/apache/spark/blob/master/pom.xml#L315
    
    Though I still don't understand a) why they're shaded in the first place, and b) how those classes are loaded when spark is built without a module that promotes those dependencies to compile scope.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    **[Test build #64317 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64317/consoleFull)** for PR 14637 at commit [`cdc5753`](https://github.com/apache/spark/commit/cdc5753d9813f3358625bec1c674f54e0d69835e).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [WIP] [SPARK-16967] move mesos to module

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    **[Test build #64109 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64109/consoleFull)** for PR 14637 at commit [`95e473b`](https://github.com/apache/spark/commit/95e473ba046298d5669dfe63264fc139d600b787).
     * This patch **fails MiMa tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    **[Test build #64299 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64299/consoleFull)** for PR 14637 at commit [`3e10acc`](https://github.com/apache/spark/commit/3e10acc733119639bbb4d51c72c5688ddf92bcc6).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    **[Test build #64370 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64370/consoleFull)** for PR 14637 at commit [`cdc5753`](https://github.com/apache/spark/commit/cdc5753d9813f3358625bec1c674f54e0d69835e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [WIP] [SPARK-16967] move mesos to module

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    **[Test build #63754 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63754/consoleFull)** for PR 14637 at commit [`465929e`](https://github.com/apache/spark/commit/465929e41a89941dbf5060b68d1c15958df4af91).
     * This patch **fails MiMa tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #14637: [SPARK-16967] move mesos to module

Posted by mgummelt <gi...@git.apache.org>.

Github user mgummelt commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14637#discussion_r75953420
  
    --- Diff: mesos/pom.xml ---
    @@ -0,0 +1,109 @@
    +<?xml version="1.0" encoding="UTF-8"?>
    +<!--
    +  ~ Licensed to the Apache Software Foundation (ASF) under one or more
    +  ~ contributor license agreements.  See the NOTICE file distributed with
    +  ~ this work for additional information regarding copyright ownership.
    +  ~ The ASF licenses this file to You under the Apache License, Version 2.0
    +  ~ (the "License"); you may not use this file except in compliance with
    +  ~ the License.  You may obtain a copy of the License at
    +  ~
    +  ~    http://www.apache.org/licenses/LICENSE-2.0
    +  ~
    +  ~ Unless required by applicable law or agreed to in writing, software
    +  ~ distributed under the License is distributed on an "AS IS" BASIS,
    +  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +  ~ See the License for the specific language governing permissions and
    +  ~ limitations under the License.
    +  -->
    +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    +  <modelVersion>4.0.0</modelVersion>
    +  <parent>
    +    <groupId>org.apache.spark</groupId>
    +    <artifactId>spark-parent_2.11</artifactId>
    +    <version>2.1.0-SNAPSHOT</version>
    +    <relativePath>../pom.xml</relativePath>
    +  </parent>
    +
    +  <artifactId>spark-mesos_2.11</artifactId>
    +  <packaging>jar</packaging>
    +  <name>Spark Project Mesos</name>
    +  <properties>
    +    <sbt.project.name>mesos</sbt.project.name>
    +    <mesos.version>1.0.0</mesos.version>
    +    <mesos.classifier>shaded-protobuf</mesos.classifier>
    +  </properties>
    +
    +  <dependencies>
    +    <dependency>
    +      <groupId>org.apache.spark</groupId>
    +      <artifactId>spark-core_${scala.binary.version}</artifactId>
    +      <version>${project.version}</version>
    +    </dependency>
    +
    +    <dependency>
    +      <groupId>org.apache.spark</groupId>
    +      <artifactId>spark-core_${scala.binary.version}</artifactId>
    +      <version>${project.version}</version>
    +      <type>test-jar</type>
    +      <scope>test</scope>
    +    </dependency>
    +
    +    <dependency>
    +      <groupId>org.apache.mesos</groupId>
    +      <artifactId>mesos</artifactId>
    +      <version>${mesos.version}</version>
    +      <classifier>${mesos.classifier}</classifier>
    +      <exclusions>
    +        <exclusion>
    +          <groupId>com.google.protobuf</groupId>
    +          <artifactId>protobuf-java</artifactId>
    +        </exclusion>
    +      </exclusions>
    +    </dependency>
    +
    +    <dependency>
    --- End diff --
    
    Ah, I didn't realize the intransitivity of provided dependencies applied to inheritance as well.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #14637: [SPARK-16967] move mesos to module

Posted by vanzin <gi...@git.apache.org>.

Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14637#discussion_r76291436
  
    --- Diff: dev/create-release/release-build.sh ---
    @@ -186,12 +186,13 @@ if [[ "$1" == "package" ]]; then
     
       # We increment the Zinc port each time to avoid OOM's and other craziness if multiple builds
       # share the same Zinc server.
    -  make_binary_release "hadoop2.3" "-Psparkr -Phadoop-2.3 -Phive -Phive-thriftserver -Pyarn" "3033" &
    -  make_binary_release "hadoop2.4" "-Psparkr -Phadoop-2.4 -Phive -Phive-thriftserver -Pyarn" "3034" &
    -  make_binary_release "hadoop2.6" "-Psparkr -Phadoop-2.6 -Phive -Phive-thriftserver -Pyarn" "3035" &
    -  make_binary_release "hadoop2.7" "-Psparkr -Phadoop-2.7 -Phive -Phive-thriftserver -Pyarn" "3036" &
    -  make_binary_release "hadoop2.4-without-hive" "-Psparkr -Phadoop-2.4 -Pyarn" "3037" &
    -  make_binary_release "without-hadoop" "-Psparkr -Phadoop-provided -Pyarn" "3038" &
    +  FLAGS="-Psparkr -Phadoop-2.3 -Phive -Phive-thriftserver -Pyarn -Pmesos"
    +  make_binary_release "hadoop2.3" "$FLAGS" "3033" &
    +  make_binary_release "hadoop2.4" "$FLAGS" "3034" &
    --- End diff --
    
    This is wrong now; "FLAGS" enables "-Phadoop-2.3" when here it shuold be "-Phadoop-2.4" (and matching versions in the lines below).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by mgummelt <gi...@git.apache.org>.

Github user mgummelt commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    @srowen Ready for review


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    **[Test build #64303 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64303/consoleFull)** for PR 14637 at commit [`05614b4`](https://github.com/apache/spark/commit/05614b4f8f40c08cb30993a6aa236c6e4cc57276).
     * This patch **fails from timeout after a configured wait of \`250m\`**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #14637: [WIP] [SPARK-16967] move mesos to module

Posted by jaceklaskowski <gi...@git.apache.org>.

Github user jaceklaskowski commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14637#discussion_r74706552
  
    --- Diff: mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterManager.scala ---
    @@ -0,0 +1,61 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.scheduler.cluster
    +
    +import org.apache.spark.{SparkContext, SparkException}
    +import org.apache.spark.scheduler.{ExternalClusterManager, SchedulerBackend, TaskScheduler, TaskSchedulerImpl}
    +import org.apache.spark.scheduler.cluster.mesos.{MesosCoarseGrainedSchedulerBackend, MesosFineGrainedSchedulerBackend}
    +
    +/**
    + * Cluster Manager for creation of Yarn scheduler and backend
    + */
    +private[spark] class MesosClusterManager extends ExternalClusterManager {
    +  private val MESOS_REGEX = """mesos://(.*)""".r
    +
    +  override def canCreate(masterURL: String): Boolean = {
    +    masterURL.startsWith("mesos")
    +  }
    +
    +  override def createTaskScheduler(sc: SparkContext, masterURL: String): TaskScheduler = {
    +    new TaskSchedulerImpl(sc)
    +  }
    +
    +  override def createSchedulerBackend(sc: SparkContext,
    +                                      masterURL: String,
    --- End diff --
    
    I think the convention is to inline 4 spaces only not to align to the first parameter.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by mgummelt <gi...@git.apache.org>.

Github user mgummelt commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by srowen <gi...@git.apache.org>.

Github user srowen commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    Nice one, LGTM. I'll leave it open for final comments until tomorrow.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [WIP] [SPARK-16967] move mesos to module

Posted by vanzin <gi...@git.apache.org>.

Github user vanzin commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    Also, with the current changes, doesn't it mean the PR builder is not building the mesos code? Shouldn't you modify `dev/run-tests.py` (I think that's the right place)?
    
    The modification to the dependency files also raises another question: how will this be made available to users? Will there be a new package "spark-with-mesos" that needs to be packaged separately? Will users add it with "--packages" instead? There might be changes needed to `dev/create-release/release-build.sh` depending on the answer.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    **[Test build #64299 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64299/consoleFull)** for PR 14637 at commit [`3e10acc`](https://github.com/apache/spark/commit/3e10acc733119639bbb4d51c72c5688ddf92bcc6).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [WIP] [SPARK-16967] move mesos to module

Posted by srowen <gi...@git.apache.org>.

Github user srowen commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    Directionally this is looking spot on, thank you. Ping when you want a review.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64303/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #14637: [WIP] [SPARK-16967] move mesos to module

Posted by jaceklaskowski <gi...@git.apache.org>.

Github user jaceklaskowski commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14637#discussion_r75574292
  
    --- Diff: project/SparkBuild.scala ---
    @@ -56,9 +56,9 @@ object BuildCommons {
         "tags", "sketch"
       ).map(ProjectRef(buildLocation, _)) ++ sqlProjects ++ streamingProjects
     
    -  val optionallyEnabledProjects@Seq(yarn, java8Tests, sparkGangliaLgpl,
    +  val optionallyEnabledProjects@Seq(mesos, yarn, java8Tests, sparkGangliaLgpl,
    --- End diff --
    
    This is a pattern matching on assignment feature in Scala and the line creates one `optionallyEnabledProjects` value for the entire `Seq` of projects with respective projects being their own values `mesos` etc.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #14637: [SPARK-16967] move mesos to module

Posted by mgummelt <gi...@git.apache.org>.

Github user mgummelt commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14637#discussion_r76292137
  
    --- Diff: dev/create-release/release-build.sh ---
    @@ -186,12 +186,13 @@ if [[ "$1" == "package" ]]; then
     
       # We increment the Zinc port each time to avoid OOM's and other craziness if multiple builds
       # share the same Zinc server.
    -  make_binary_release "hadoop2.3" "-Psparkr -Phadoop-2.3 -Phive -Phive-thriftserver -Pyarn" "3033" &
    -  make_binary_release "hadoop2.4" "-Psparkr -Phadoop-2.4 -Phive -Phive-thriftserver -Pyarn" "3034" &
    -  make_binary_release "hadoop2.6" "-Psparkr -Phadoop-2.6 -Phive -Phive-thriftserver -Pyarn" "3035" &
    -  make_binary_release "hadoop2.7" "-Psparkr -Phadoop-2.7 -Phive -Phive-thriftserver -Pyarn" "3036" &
    -  make_binary_release "hadoop2.4-without-hive" "-Psparkr -Phadoop-2.4 -Pyarn" "3037" &
    -  make_binary_release "without-hadoop" "-Psparkr -Phadoop-provided -Pyarn" "3038" &
    +  FLAGS="-Psparkr -Phadoop-2.3 -Phive -Phive-thriftserver -Pyarn -Pmesos"
    +  make_binary_release "hadoop2.3" "$FLAGS" "3033" &
    +  make_binary_release "hadoop2.4" "$FLAGS" "3034" &
    --- End diff --
    
    ah, yea.  fixing...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [WIP] [SPARK-16967] move mesos to module

Posted by vanzin <gi...@git.apache.org>.

Github user vanzin commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    If this is ready for review could you remove "wip" from the pr title?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by vanzin <gi...@git.apache.org>.

Github user vanzin commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    Merging to master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64302/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by mgummelt <gi...@git.apache.org>.

Github user mgummelt commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64301/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64435/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64299/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #14637: [SPARK-16967] move mesos to module

Posted by vanzin <gi...@git.apache.org>.

Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14637#discussion_r75939624
  
    --- Diff: mesos/pom.xml ---
    @@ -0,0 +1,109 @@
    +<?xml version="1.0" encoding="UTF-8"?>
    +<!--
    +  ~ Licensed to the Apache Software Foundation (ASF) under one or more
    +  ~ contributor license agreements.  See the NOTICE file distributed with
    +  ~ this work for additional information regarding copyright ownership.
    +  ~ The ASF licenses this file to You under the Apache License, Version 2.0
    +  ~ (the "License"); you may not use this file except in compliance with
    +  ~ the License.  You may obtain a copy of the License at
    +  ~
    +  ~    http://www.apache.org/licenses/LICENSE-2.0
    +  ~
    +  ~ Unless required by applicable law or agreed to in writing, software
    +  ~ distributed under the License is distributed on an "AS IS" BASIS,
    +  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +  ~ See the License for the specific language governing permissions and
    +  ~ limitations under the License.
    +  -->
    +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    +  <modelVersion>4.0.0</modelVersion>
    +  <parent>
    +    <groupId>org.apache.spark</groupId>
    +    <artifactId>spark-parent_2.11</artifactId>
    +    <version>2.1.0-SNAPSHOT</version>
    +    <relativePath>../pom.xml</relativePath>
    +  </parent>
    +
    +  <artifactId>spark-mesos_2.11</artifactId>
    +  <packaging>jar</packaging>
    +  <name>Spark Project Mesos</name>
    +  <properties>
    +    <sbt.project.name>mesos</sbt.project.name>
    +    <mesos.version>1.0.0</mesos.version>
    +    <mesos.classifier>shaded-protobuf</mesos.classifier>
    +  </properties>
    +
    +  <dependencies>
    +    <dependency>
    +      <groupId>org.apache.spark</groupId>
    +      <artifactId>spark-core_${scala.binary.version}</artifactId>
    +      <version>${project.version}</version>
    +    </dependency>
    +
    +    <dependency>
    +      <groupId>org.apache.spark</groupId>
    +      <artifactId>spark-core_${scala.binary.version}</artifactId>
    +      <version>${project.version}</version>
    +      <type>test-jar</type>
    +      <scope>test</scope>
    +    </dependency>
    +
    +    <dependency>
    +      <groupId>org.apache.mesos</groupId>
    +      <artifactId>mesos</artifactId>
    +      <version>${mesos.version}</version>
    +      <classifier>${mesos.classifier}</classifier>
    +      <exclusions>
    +        <exclusion>
    +          <groupId>com.google.protobuf</groupId>
    +          <artifactId>protobuf-java</artifactId>
    +        </exclusion>
    +      </exclusions>
    +    </dependency>
    +
    +    <dependency>
    --- End diff --
    
    (Note to self: YARN - and this new mesos module - should only need to declare the Guava dependency explicitly, since they don't use Jetty directly.)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #14637: [WIP] [SPARK-16967] move mesos to module

Posted by jaceklaskowski <gi...@git.apache.org>.

Github user jaceklaskowski commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14637#discussion_r74706522
  
    --- Diff: mesos/pom.xml ---
    @@ -0,0 +1,167 @@
    +<?xml version="1.0" encoding="UTF-8"?>
    +<!--
    +  ~ Licensed to the Apache Software Foundation (ASF) under one or more
    +  ~ contributor license agreements.  See the NOTICE file distributed with
    +  ~ this work for additional information regarding copyright ownership.
    +  ~ The ASF licenses this file to You under the Apache License, Version 2.0
    +  ~ (the "License"); you may not use this file except in compliance with
    +  ~ the License.  You may obtain a copy of the License at
    +  ~
    +  ~    http://www.apache.org/licenses/LICENSE-2.0
    +  ~
    +  ~ Unless required by applicable law or agreed to in writing, software
    +  ~ distributed under the License is distributed on an "AS IS" BASIS,
    +  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +  ~ See the License for the specific language governing permissions and
    +  ~ limitations under the License.
    +  -->
    +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    +  <modelVersion>4.0.0</modelVersion>
    +  <parent>
    +    <groupId>org.apache.spark</groupId>
    +    <artifactId>spark-parent_2.11</artifactId>
    +    <version>2.1.0-SNAPSHOT</version>
    +    <relativePath>../pom.xml</relativePath>
    +  </parent>
    +
    +  <artifactId>spark-mesos_2.11</artifactId>
    +  <packaging>jar</packaging>
    +  <name>Spark Project Mesos</name>
    +  <properties>
    +    <sbt.project.name>mesos</sbt.project.name>
    +    <mesos.version>1.0.0</mesos.version>
    +    <mesos.classifier>shaded-protobuf</mesos.classifier>
    +  </properties>
    +
    +  <dependencies>
    +    <dependency>
    +      <groupId>org.apache.spark</groupId>
    +      <artifactId>spark-core_${scala.binary.version}</artifactId>
    +      <version>${project.version}</version>
    +    </dependency>
    +    <dependency>
    +      <groupId>org.apache.spark</groupId>
    +      <artifactId>spark-core_${scala.binary.version}</artifactId>
    +      <version>${project.version}</version>
    +      <type>test-jar</type>
    +      <scope>test</scope>
    +    </dependency>
    +    <dependency>
    +      <groupId>org.apache.mesos</groupId>
    +      <artifactId>mesos</artifactId>
    +      <version>${mesos.version}</version>
    +      <classifier>${mesos.classifier}</classifier>
    +      <exclusions>
    +        <exclusion>
    +          <groupId>com.google.protobuf</groupId>
    +          <artifactId>protobuf-java</artifactId>
    +        </exclusion>
    +      </exclusions>
    +    </dependency>
    +
    +
    +    <!-- Explicit listing of transitive deps that are shaded. Otherwise, odd compiler crashes.
    +    <dependency>
    +      <groupId>com.google.guava</groupId>
    +      <artifactId>guava</artifactId>
    +    </dependency>
    +    <dependency>
    +      <groupId>org.eclipse.jetty</groupId>
    +      <artifactId>jetty-server</artifactId>
    +    </dependency>
    +    <dependency>
    +      <groupId>org.eclipse.jetty</groupId>
    +      <artifactId>jetty-plus</artifactId>
    +    </dependency>
    +    <dependency>
    +      <groupId>org.eclipse.jetty</groupId>
    +      <artifactId>jetty-util</artifactId>
    +    </dependency>
    +    <dependency>
    +      <groupId>org.eclipse.jetty</groupId>
    +      <artifactId>jetty-http</artifactId>
    +    </dependency>
    +    <dependency>
    +      <groupId>org.eclipse.jetty</groupId>
    +      <artifactId>jetty-servlet</artifactId>
    +    </dependency>
    +    <dependency>
    +      <groupId>org.eclipse.jetty</groupId>
    +      <artifactId>jetty-servlets</artifactId>
    +    </dependency>
    +    End of shaded deps. -->
    +
    +    <dependency>
    +      <groupId>org.mockito</groupId>
    +      <artifactId>mockito-core</artifactId>
    +      <scope>test</scope>
    +    </dependency>
    +    <!--
    --- End diff --
    
    And that one too?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    **[Test build #64301 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64301/consoleFull)** for PR 14637 at commit [`e94f5e7`](https://github.com/apache/spark/commit/e94f5e7c9492ec84b1e2b786f4a2f749eb3250c4).
     * This patch **fails executing the `dev/run-tests` script**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64370/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [WIP] [SPARK-16967] move mesos to module

Posted by mgummelt <gi...@git.apache.org>.

Github user mgummelt commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    @srowen OK, added the profile to:
    - .travis.yml
    - release-build.sh
    - lint-java
    - mima
    - scalastyle
    
    What about dev/test-dependencies.sh.
    
    Also, I added mesos to PUBLISH_PROFILES in release-build.sh.  Should I also add it to the make_binary_release profiles?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by vanzin <gi...@git.apache.org>.

Github user vanzin commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    LGTM now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #14637: [SPARK-16967] move mesos to module

Posted by vanzin <gi...@git.apache.org>.

Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14637#discussion_r75927255
  
    --- Diff: mesos/pom.xml ---
    @@ -0,0 +1,109 @@
    +<?xml version="1.0" encoding="UTF-8"?>
    +<!--
    +  ~ Licensed to the Apache Software Foundation (ASF) under one or more
    +  ~ contributor license agreements.  See the NOTICE file distributed with
    +  ~ this work for additional information regarding copyright ownership.
    +  ~ The ASF licenses this file to You under the Apache License, Version 2.0
    +  ~ (the "License"); you may not use this file except in compliance with
    +  ~ the License.  You may obtain a copy of the License at
    +  ~
    +  ~    http://www.apache.org/licenses/LICENSE-2.0
    +  ~
    +  ~ Unless required by applicable law or agreed to in writing, software
    +  ~ distributed under the License is distributed on an "AS IS" BASIS,
    +  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +  ~ See the License for the specific language governing permissions and
    +  ~ limitations under the License.
    +  -->
    +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    +  <modelVersion>4.0.0</modelVersion>
    +  <parent>
    +    <groupId>org.apache.spark</groupId>
    +    <artifactId>spark-parent_2.11</artifactId>
    +    <version>2.1.0-SNAPSHOT</version>
    +    <relativePath>../pom.xml</relativePath>
    +  </parent>
    +
    +  <artifactId>spark-mesos_2.11</artifactId>
    +  <packaging>jar</packaging>
    +  <name>Spark Project Mesos</name>
    +  <properties>
    +    <sbt.project.name>mesos</sbt.project.name>
    +    <mesos.version>1.0.0</mesos.version>
    +    <mesos.classifier>shaded-protobuf</mesos.classifier>
    +  </properties>
    +
    +  <dependencies>
    +    <dependency>
    +      <groupId>org.apache.spark</groupId>
    +      <artifactId>spark-core_${scala.binary.version}</artifactId>
    +      <version>${project.version}</version>
    +    </dependency>
    +
    +    <dependency>
    +      <groupId>org.apache.spark</groupId>
    +      <artifactId>spark-core_${scala.binary.version}</artifactId>
    +      <version>${project.version}</version>
    +      <type>test-jar</type>
    +      <scope>test</scope>
    +    </dependency>
    +
    +    <dependency>
    +      <groupId>org.apache.mesos</groupId>
    +      <artifactId>mesos</artifactId>
    +      <version>${mesos.version}</version>
    +      <classifier>${mesos.classifier}</classifier>
    +      <exclusions>
    +        <exclusion>
    +          <groupId>com.google.protobuf</groupId>
    +          <artifactId>protobuf-java</artifactId>
    +        </exclusion>
    +      </exclusions>
    +    </dependency>
    +
    +    <dependency>
    --- End diff --
    
    You shouldn't need to touch the shaded dependencies (see how no other module does that). If you're running into problems, you're probably missing something else, so post the error so we can take a look.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [WIP] [SPARK-16967] move mesos to module

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    **[Test build #63754 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63754/consoleFull)** for PR 14637 at commit [`465929e`](https://github.com/apache/spark/commit/465929e41a89941dbf5060b68d1c15958df4af91).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by mgummelt <gi...@git.apache.org>.

Github user mgummelt commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    **[Test build #64435 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64435/consoleFull)** for PR 14637 at commit [`09f3197`](https://github.com/apache/spark/commit/09f3197e7cac9a45315bf5bdaed57c97bcd0e46d).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #14637: [WIP] [SPARK-16967] move mesos to module

Posted by jaceklaskowski <gi...@git.apache.org>.

Github user jaceklaskowski commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14637#discussion_r74706517
  
    --- Diff: mesos/pom.xml ---
    @@ -0,0 +1,167 @@
    +<?xml version="1.0" encoding="UTF-8"?>
    +<!--
    +  ~ Licensed to the Apache Software Foundation (ASF) under one or more
    +  ~ contributor license agreements.  See the NOTICE file distributed with
    +  ~ this work for additional information regarding copyright ownership.
    +  ~ The ASF licenses this file to You under the Apache License, Version 2.0
    +  ~ (the "License"); you may not use this file except in compliance with
    +  ~ the License.  You may obtain a copy of the License at
    +  ~
    +  ~    http://www.apache.org/licenses/LICENSE-2.0
    +  ~
    +  ~ Unless required by applicable law or agreed to in writing, software
    +  ~ distributed under the License is distributed on an "AS IS" BASIS,
    +  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +  ~ See the License for the specific language governing permissions and
    +  ~ limitations under the License.
    +  -->
    +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    +  <modelVersion>4.0.0</modelVersion>
    +  <parent>
    +    <groupId>org.apache.spark</groupId>
    +    <artifactId>spark-parent_2.11</artifactId>
    +    <version>2.1.0-SNAPSHOT</version>
    +    <relativePath>../pom.xml</relativePath>
    +  </parent>
    +
    +  <artifactId>spark-mesos_2.11</artifactId>
    +  <packaging>jar</packaging>
    +  <name>Spark Project Mesos</name>
    +  <properties>
    +    <sbt.project.name>mesos</sbt.project.name>
    +    <mesos.version>1.0.0</mesos.version>
    +    <mesos.classifier>shaded-protobuf</mesos.classifier>
    +  </properties>
    +
    +  <dependencies>
    +    <dependency>
    +      <groupId>org.apache.spark</groupId>
    +      <artifactId>spark-core_${scala.binary.version}</artifactId>
    +      <version>${project.version}</version>
    +    </dependency>
    +    <dependency>
    +      <groupId>org.apache.spark</groupId>
    +      <artifactId>spark-core_${scala.binary.version}</artifactId>
    +      <version>${project.version}</version>
    +      <type>test-jar</type>
    +      <scope>test</scope>
    +    </dependency>
    +    <dependency>
    +      <groupId>org.apache.mesos</groupId>
    +      <artifactId>mesos</artifactId>
    +      <version>${mesos.version}</version>
    +      <classifier>${mesos.classifier}</classifier>
    +      <exclusions>
    +        <exclusion>
    +          <groupId>com.google.protobuf</groupId>
    +          <artifactId>protobuf-java</artifactId>
    +        </exclusion>
    +      </exclusions>
    +    </dependency>
    +
    +
    +    <!-- Explicit listing of transitive deps that are shaded. Otherwise, odd compiler crashes.
    --- End diff --
    
    Why do we need that commented-out section?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    **[Test build #64317 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64317/consoleFull)** for PR 14637 at commit [`cdc5753`](https://github.com/apache/spark/commit/cdc5753d9813f3358625bec1c674f54e0d69835e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by srowen <gi...@git.apache.org>.

Github user srowen commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    The release notes are for end users, and this doesn't impact end users. Developers are expected, more or less, to follow commits and dev@ to keep up with changes like this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    **[Test build #64302 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64302/consoleFull)** for PR 14637 at commit [`fe7d05e`](https://github.com/apache/spark/commit/fe7d05e94a7cc55c57ba14b041ad3ce07139878e).
     * This patch **fails build dependency tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [WIP] [SPARK-16967] move mesos to module

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64109/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #14637: [SPARK-16967] move mesos to module

Posted by asfgit <gi...@git.apache.org>.

Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/14637


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #14637: [WIP] [SPARK-16967] move mesos to module

Posted by mgummelt <gi...@git.apache.org>.

Github user mgummelt commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14637#discussion_r75547904
  
    --- Diff: mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterManager.scala ---
    @@ -0,0 +1,61 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.scheduler.cluster
    +
    +import org.apache.spark.{SparkContext, SparkException}
    +import org.apache.spark.scheduler.{ExternalClusterManager, SchedulerBackend, TaskScheduler, TaskSchedulerImpl}
    +import org.apache.spark.scheduler.cluster.mesos.{MesosCoarseGrainedSchedulerBackend, MesosFineGrainedSchedulerBackend}
    +
    +/**
    + * Cluster Manager for creation of Yarn scheduler and backend
    + */
    +private[spark] class MesosClusterManager extends ExternalClusterManager {
    +  private val MESOS_REGEX = """mesos://(.*)""".r
    +
    +  override def canCreate(masterURL: String): Boolean = {
    +    masterURL.startsWith("mesos")
    +  }
    +
    +  override def createTaskScheduler(sc: SparkContext, masterURL: String): TaskScheduler = {
    +    new TaskSchedulerImpl(sc)
    +  }
    +
    +  override def createSchedulerBackend(sc: SparkContext,
    +                                      masterURL: String,
    +                                      scheduler: TaskScheduler): SchedulerBackend = {
    +    val mesosUrl = MESOS_REGEX.findFirstMatchIn(masterURL).get.group(1)
    +    val coarse = sc.conf.getBoolean("spark.mesos.coarse", defaultValue = true)
    --- End diff --
    
    Mesos supports coarse-grained and fine-grained, but fine-grained is deprecated 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #14637: [WIP] [SPARK-16967] move mesos to module

Posted by mgummelt <gi...@git.apache.org>.

Github user mgummelt commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14637#discussion_r75563699
  
    --- Diff: project/SparkBuild.scala ---
    @@ -56,9 +56,9 @@ object BuildCommons {
         "tags", "sketch"
       ).map(ProjectRef(buildLocation, _)) ++ sqlProjects ++ streamingProjects
     
    -  val optionallyEnabledProjects@Seq(yarn, java8Tests, sparkGangliaLgpl,
    +  val optionallyEnabledProjects@Seq(mesos, yarn, java8Tests, sparkGangliaLgpl,
    --- End diff --
    
    FYI I'm just pattern matching here to get `-Pmesos` to work with `sbt`.  I have no idea what this does.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [WIP] [SPARK-16967] move mesos to module

Posted by srowen <gi...@git.apache.org>.

Github user srowen commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    You can ignore the MiMa error and suppress it, yes.
    This is generally looking great, thanks. @vanzin would be a good reviewer too.
    
    I think we'll have to change the Jenkins job config later to set `-Pmesos` -- will see about it.
    There are some other miscellaneous places that may need to add the new profile, `dev/`:
    
    - `lint-java`, `scalastyle`
    - `mima` add to the profiles it uses for MiMa
    - `create-release/release-build.sh` I assume the default tarballs will still build in Mesos
    
    Although it's kind of dead config, you might add the profile to `.travis.yml` too.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    **[Test build #64435 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64435/consoleFull)** for PR 14637 at commit [`09f3197`](https://github.com/apache/spark/commit/09f3197e7cac9a45315bf5bdaed57c97bcd0e46d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [WIP] [SPARK-16967] move mesos to module

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    **[Test build #63753 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63753/consoleFull)** for PR 14637 at commit [`57a191b`](https://github.com/apache/spark/commit/57a191bc6b046d68ed1ed9dde8b5c52710fdf214).
     * This patch **fails MiMa tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by drcrallen <gi...@git.apache.org>.

Github user drcrallen commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    FYI, we have a build process that packages spark core, now that mesos is is in its own artifact, this broke our build and deploy process, and its not called out in release notes


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [WIP] [SPARK-16967] move mesos to module

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    **[Test build #64370 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64370/consoleFull)** for PR 14637 at commit [`cdc5753`](https://github.com/apache/spark/commit/cdc5753d9813f3358625bec1c674f54e0d69835e).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #14637: [WIP] [SPARK-16967] move mesos to module

Posted by srowen <gi...@git.apache.org>.

Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14637#discussion_r75576449
  
    --- Diff: mesos/pom.xml ---
    @@ -0,0 +1,109 @@
    +<?xml version="1.0" encoding="UTF-8"?>
    +<!--
    +  ~ Licensed to the Apache Software Foundation (ASF) under one or more
    +  ~ contributor license agreements.  See the NOTICE file distributed with
    +  ~ this work for additional information regarding copyright ownership.
    +  ~ The ASF licenses this file to You under the Apache License, Version 2.0
    +  ~ (the "License"); you may not use this file except in compliance with
    +  ~ the License.  You may obtain a copy of the License at
    +  ~
    +  ~    http://www.apache.org/licenses/LICENSE-2.0
    +  ~
    +  ~ Unless required by applicable law or agreed to in writing, software
    +  ~ distributed under the License is distributed on an "AS IS" BASIS,
    +  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +  ~ See the License for the specific language governing permissions and
    +  ~ limitations under the License.
    +  -->
    +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    +  <modelVersion>4.0.0</modelVersion>
    +  <parent>
    +    <groupId>org.apache.spark</groupId>
    +    <artifactId>spark-parent_2.11</artifactId>
    +    <version>2.1.0-SNAPSHOT</version>
    +    <relativePath>../pom.xml</relativePath>
    +  </parent>
    +
    +  <artifactId>spark-mesos_2.11</artifactId>
    +  <packaging>jar</packaging>
    +  <name>Spark Project Mesos</name>
    +  <properties>
    +    <sbt.project.name>mesos</sbt.project.name>
    +    <mesos.version>1.0.0</mesos.version>
    +    <mesos.classifier>shaded-protobuf</mesos.classifier>
    +  </properties>
    +
    +  <dependencies>
    +    <dependency>
    +      <groupId>org.apache.spark</groupId>
    +      <artifactId>spark-core_${scala.binary.version}</artifactId>
    +      <version>${project.version}</version>
    +    </dependency>
    +
    +    <dependency>
    +      <groupId>org.apache.spark</groupId>
    +      <artifactId>spark-core_${scala.binary.version}</artifactId>
    +      <version>${project.version}</version>
    +      <type>test-jar</type>
    +      <scope>test</scope>
    +    </dependency>
    +
    +    <dependency>
    +      <groupId>org.apache.mesos</groupId>
    +      <artifactId>mesos</artifactId>
    +      <version>${mesos.version}</version>
    +      <classifier>${mesos.classifier}</classifier>
    +      <exclusions>
    +        <exclusion>
    +          <groupId>com.google.protobuf</groupId>
    +          <artifactId>protobuf-java</artifactId>
    +        </exclusion>
    +      </exclusions>
    +    </dependency>
    +
    +    <dependency>
    --- End diff --
    
    I don't know that you need these dependencies -- literally don't know either way by looking. These are definitely needed? I was wondering whether jetty was relevant.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [WIP] [SPARK-16967] move mesos to module

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    **[Test build #63765 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63765/consoleFull)** for PR 14637 at commit [`4e276ab`](https://github.com/apache/spark/commit/4e276ab81848669ec5c7da6d51cf63a46b1bac87).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #14637: [WIP] [SPARK-16967] move mesos to module

Posted by jaceklaskowski <gi...@git.apache.org>.

Github user jaceklaskowski commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14637#discussion_r74706568
  
    --- Diff: mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterManager.scala ---
    @@ -0,0 +1,61 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.scheduler.cluster
    +
    +import org.apache.spark.{SparkContext, SparkException}
    +import org.apache.spark.scheduler.{ExternalClusterManager, SchedulerBackend, TaskScheduler, TaskSchedulerImpl}
    +import org.apache.spark.scheduler.cluster.mesos.{MesosCoarseGrainedSchedulerBackend, MesosFineGrainedSchedulerBackend}
    +
    +/**
    + * Cluster Manager for creation of Yarn scheduler and backend
    + */
    +private[spark] class MesosClusterManager extends ExternalClusterManager {
    +  private val MESOS_REGEX = """mesos://(.*)""".r
    +
    +  override def canCreate(masterURL: String): Boolean = {
    +    masterURL.startsWith("mesos")
    +  }
    +
    +  override def createTaskScheduler(sc: SparkContext, masterURL: String): TaskScheduler = {
    +    new TaskSchedulerImpl(sc)
    +  }
    +
    +  override def createSchedulerBackend(sc: SparkContext,
    +                                      masterURL: String,
    +                                      scheduler: TaskScheduler): SchedulerBackend = {
    +    val mesosUrl = MESOS_REGEX.findFirstMatchIn(masterURL).get.group(1)
    +    val coarse = sc.conf.getBoolean("spark.mesos.coarse", defaultValue = true)
    --- End diff --
    
    Oh, I thought Spark supports coarse-grain scheduling only (even for Mesos that had fine-grained one too). Will it stay or is this a temporary thing?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [SPARK-16967] move mesos to module

Posted by srowen <gi...@git.apache.org>.

Github user srowen commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    Yes to all three of those.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [WIP] [SPARK-16967] move mesos to module

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    **[Test build #63753 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63753/consoleFull)** for PR 14637 at commit [`57a191b`](https://github.com/apache/spark/commit/57a191bc6b046d68ed1ed9dde8b5c52710fdf214).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [WIP] [SPARK-16967] move mesos to module

Posted by mgummelt <gi...@git.apache.org>.

Github user mgummelt commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    mima seems to be upset about my removal of MESOS_REGEX from `SparkMasterRegex`, but I don't understand why, as it's a private class.  Should I add an entry to MimaExcludes?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #14637: [WIP] [SPARK-16967] move mesos to module

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14637
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63754/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org