You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by ifilonenko <gi...@git.apache.org> on 2018/06/18 16:18:57 UTC

[GitHub] spark pull request #21584: [SPARK-24433][K8S][WIP] Initial R Bindings for Sp...

GitHub user ifilonenko opened a pull request:

    https://github.com/apache/spark/pull/21584

    [SPARK-24433][K8S][WIP] Initial R Bindings for SparkR on K8s

    ## What changes were proposed in this pull request?
    
    Introducing R Bindings for Spark R on K8s
    
    - [ ] Running SparkR Job
    
    ## How was this patch tested?
    
    This patch was tested with 
    
    - [x] Unit Tests
    - [ ] Integration Tests


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ifilonenko/spark spark-r

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/21584.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #21584
    
----
commit 45aaefcd6b148c3e83f03d07575a9bb199835267
Author: Ilan Filonenko <if...@...>
Date:   2018-06-18T05:08:06Z

    initial R bindings

commit 884a4e71f25dd8276cd198ad8c3deb91e61634d3
Author: Ilan Filonenko <if...@...>
Date:   2018-06-18T05:11:03Z

    fix Dockerfile

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR on K8s

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21584
  
    **[Test build #93663 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93663/testReport)** for PR 21584 at commit [`1f0cba5`](https://github.com/apache/spark/commit/1f0cba59b650e4458e9472933068928d52a54777).
     * This patch **fails to build**.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `trait BarrierTaskContext extends TaskContext `
      * `class BarrierTaskInfo(val address: String)`
      * `class RDDBarrier[T: ClassTag](rdd: RDD[T]) `
      * `case class WorkerOffer(`
      * `trait AnalysisHelper extends QueryPlan[LogicalPlan] `


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21584: [SPARK-24433][K8S][WIP] Initial R Bindings for SparkR on...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21584
  
    **[Test build #92034 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92034/testReport)** for PR 21584 at commit [`884a4e7`](https://github.com/apache/spark/commit/884a4e71f25dd8276cd198ad8c3deb91e61634d3).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR ...

Posted by ifilonenko <gi...@git.apache.org>.

Github user ifilonenko commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21584#discussion_r204713851
  
    --- Diff: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/bindings/R/Dockerfile ---
    @@ -0,0 +1,29 @@
    +#
    +# Licensed to the Apache Software Foundation (ASF) under one or more
    +# contributor license agreements.  See the NOTICE file distributed with
    +# this work for additional information regarding copyright ownership.
    +# The ASF licenses this file to You under the Apache License, Version 2.0
    +# (the "License"); you may not use this file except in compliance with
    +# the License.  You may obtain a copy of the License at
    +#
    +#    http://www.apache.org/licenses/LICENSE-2.0
    +#
    +# Unless required by applicable law or agreed to in writing, software
    +# distributed under the License is distributed on an "AS IS" BASIS,
    +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +# See the License for the specific language governing permissions and
    +# limitations under the License.
    +#
    +
    +ARG base_img
    +FROM $base_img
    +WORKDIR /
    +RUN mkdir ${SPARK_HOME}/R
    +COPY R ${SPARK_HOME}/R
    +
    +RUN apk add --no-cache R R-dev
    +
    --- End diff --
    
    This is only for Python packaging. R does not have `/root/.cache` when created by alpine. 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR ...

Posted by felixcheung <gi...@git.apache.org>.

Github user felixcheung commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21584#discussion_r204229663
  
    --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesDriverBuilder.scala ---
    @@ -71,8 +75,9 @@ private[spark] class KubernetesDriverBuilder(
             case JavaMainAppResource(_) =>
               provideJavaStep(kubernetesConf)
             case PythonMainAppResource(_) =>
    -          providePythonStep(kubernetesConf)}
    -      .getOrElse(provideJavaStep(kubernetesConf))
    +          providePythonStep(kubernetesConf)
    +        case RMainAppResource(_) =>
    +          provideRStep(kubernetesConf)}.getOrElse(provideJavaStep(kubernetesConf))
    --- End diff --
    
    I'd move `}.getOrElse(provideJavaStep(kubernetesConf))` to the next line for clarity?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR on K8s

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21584
  
    **[Test build #94862 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94862/testReport)** for PR 21584 at commit [`6584029`](https://github.com/apache/spark/commit/658402919c080ae4d878d355a4b3a14a4d4d0aad).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR ...

Posted by ifilonenko <gi...@git.apache.org>.

Github user ifilonenko commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21584#discussion_r207605696
  
    --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala ---
    @@ -205,7 +218,7 @@ private[spark] object Config extends Logging {
           .createWithDefault(0.1)
     
       val PYSPARK_MAJOR_PYTHON_VERSION =
    -    ConfigBuilder("spark.kubernetes.pyspark.pythonversion")
    +    ConfigBuilder("spark.kubernetes.pyspark.pythonVersion")
    --- End diff --
    
    hmm good call.... I might do that then


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR on K8s

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21584
  
    **[Test build #93294 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93294/testReport)** for PR 21584 at commit [`d809f67`](https://github.com/apache/spark/commit/d809f67357afb955c5f398a9ada6523ed43c7f99).
     * This patch **fails MiMa tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR on K8s

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21584
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93294/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR on K8s

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21584
  
    **[Test build #93547 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93547/testReport)** for PR 21584 at commit [`d036673`](https://github.com/apache/spark/commit/d0366732a9ebef710cb78e52ee42a869002e9040).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR on K8s

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21584
  
    Kubernetes integration test status failure
    URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/1313/



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR on K8s

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21584
  
    Kubernetes integration test starting
    URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/1138/



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR on K8s

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21584
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1399/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21584: [SPARK-24433][K8S][WIP] Initial R Bindings for SparkR on...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21584
  
    **[Test build #92635 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92635/testReport)** for PR 21584 at commit [`a2f2825`](https://github.com/apache/spark/commit/a2f282555b8a30ee2a50448cb7045819a6f3d483).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR on K8s

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21584
  
    **[Test build #94862 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94862/testReport)** for PR 21584 at commit [`6584029`](https://github.com/apache/spark/commit/658402919c080ae4d878d355a4b3a14a4d4d0aad).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR on K8s

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21584
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2256/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR on K8s

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21584
  
    Kubernetes integration test starting
    URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/1201/



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR on K8s

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21584
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR ...

Posted by ifilonenko <gi...@git.apache.org>.

Github user ifilonenko commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21584#discussion_r207382976
  
    --- Diff: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/bindings/R/Dockerfile ---
    @@ -0,0 +1,29 @@
    +#
    +# Licensed to the Apache Software Foundation (ASF) under one or more
    +# contributor license agreements.  See the NOTICE file distributed with
    +# this work for additional information regarding copyright ownership.
    +# The ASF licenses this file to You under the Apache License, Version 2.0
    +# (the "License"); you may not use this file except in compliance with
    +# the License.  You may obtain a copy of the License at
    +#
    +#    http://www.apache.org/licenses/LICENSE-2.0
    +#
    +# Unless required by applicable law or agreed to in writing, software
    +# distributed under the License is distributed on an "AS IS" BASIS,
    +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +# See the License for the specific language governing permissions and
    +# limitations under the License.
    +#
    +
    +ARG base_img
    +FROM $base_img
    +WORKDIR /
    +RUN mkdir ${SPARK_HOME}/R
    +COPY R ${SPARK_HOME}/R
    +
    +RUN apk add --no-cache R R-dev
    --- End diff --
    
    Agreed. Minor versions are something we *might* want to support in the future. Let's look to include this in a follow-up PR if there is enough desire in custom version arguments to be passed as build_args to the `bin/docker-image-tool.sh` 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR on K8s

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21584
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR ...

Posted by ifilonenko <gi...@git.apache.org>.

Github user ifilonenko commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21584#discussion_r205928623
  
    --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala ---
    @@ -205,7 +218,7 @@ private[spark] object Config extends Logging {
           .createWithDefault(0.1)
     
       val PYSPARK_MAJOR_PYTHON_VERSION =
    -    ConfigBuilder("spark.kubernetes.pyspark.pythonversion")
    +    ConfigBuilder("spark.kubernetes.pyspark.pythonVersion")
    --- End diff --
    
    idiomatic for configs to have camelCase. 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR on K8s

Posted by ifilonenko <gi...@git.apache.org>.

Github user ifilonenko commented on the issue:

    https://github.com/apache/spark/pull/21584
  
    @felixcheung the PRB isn’t currently configured with capability to build —r distributions. Waiting on @shaneknapp, which is why it is unable to build the image. With an R supported environment, it passes all tests. 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR ...

Posted by felixcheung <gi...@git.apache.org>.

Github user felixcheung commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21584#discussion_r204229514
  
    --- Diff: bin/docker-image-tool.sh ---
    @@ -79,11 +80,16 @@ function build {
       docker build $NOCACHEARG "${BINDING_BUILD_ARGS[@]}" \
         -t $(image_ref spark-py) \
         -f "$PYDOCKERFILE" .
    +
    +    docker build "${BINDING_BUILD_ARGS[@]}" \
    --- End diff --
    
    nit: this doesn't align with the pattern above


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org