[jira] [Comment Edited] (SPARK-27365) Spark Jenkins supports testing GPU-aware scheduling features

["Calvin Park (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-27365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16820549#comment-16820549 ] 

Calvin Park edited comment on SPARK-27365 at 4/17/19 10:34 PM:
---------------------------------------------------------------

Simple Jenkinsfile
{code:java}
pipeline {
    agent {
        dockerfile {
            label 'docker-gpu'
            args '--runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=all'
        }
    }
    stages {
        stage('smi') {
            steps {
                sh 'nvidia-smi'
            }
        }
    }
}{code}
with Dockerfile
{code:java}
FROM ubuntu:xenial

RUN apt-get update && \
    DEBIAN_FRONTEND=noninteractive apt-get install -y \
    curl \
    flake8 \
    git-core \
    openjdk-8-jdk \
    python2.7 \
    python-pip \
    wget

RUN DEBIAN_FRONTEND=noninteractive pip install \
    requests \
    numpy

# Build script looks for javac in jre dir
ENV JAVA_HOME "/usr/lib/jvm/java-8-openjdk-amd64"

# http://spark.apache.org/docs/latest/building-spark.html#setting-up-mavens-memory-usage
# We have a pretty beefy server
ENV MAVEN_OPTS "-Xmx20g -XX:ReservedCodeCacheSize=2g"
{code}


was (Author: calvinatnvidia):
Simple Jenkinsfile
{code:java}
pipeline {
    agent {
        dockerfile {
            label 'docker-gpu'
            args '--runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=all'
        }
    }
    stages {
        stage('smi') {
            steps {
                sh 'nvidia-smi'
            }
        }
    }
}{code}
with Dockerfile
FROM ubuntu:xenialRUN apt-get update && \    DEBIAN_FRONTEND=noninteractive apt-get install -y \    curl \    flake8 \    git-core \    openjdk-8-jdk \    python2.7 \    python-pip \    wgetRUN DEBIAN_FRONTEND=noninteractive pip install \    requests \    numpy# Build script looks for javac in jre dirENV JAVA_HOME "/usr/lib/jvm/java-8-openjdk-amd64"# http://spark.apache.org/docs/latest/building-spark.html#setting-up-mavens-memory-usage# We have a pretty beefy serverENV MAVEN_OPTS "-Xmx20g -XX:ReservedCodeCacheSize=2g"

> Spark Jenkins supports testing GPU-aware scheduling features
> ------------------------------------------------------------
>
>                 Key: SPARK-27365
>                 URL: https://issues.apache.org/jira/browse/SPARK-27365
>             Project: Spark
>          Issue Type: Story
>          Components: jenkins
>    Affects Versions: 3.0.0
>            Reporter: Xiangrui Meng
>            Priority: Major
>
> Upgrade Spark Jenkins to install GPU cards and run GPU integration tests triggered by "GPU" in PRs.
> cc: [~afeng] [~shaneknapp]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org