You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Calvin Park (JIRA)" <ji...@apache.org> on 2019/04/17 22:34:00 UTC
[jira] [Commented] (SPARK-27365) Spark Jenkins supports testing
GPU-aware scheduling features
[ https://issues.apache.org/jira/browse/SPARK-27365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16820549#comment-16820549 ]
Calvin Park commented on SPARK-27365:
-------------------------------------
Simple Jenkinsfile
{code:java}
pipeline {
agent {
dockerfile {
label 'docker-gpu'
args '--runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=all'
}
}
stages {
stage('smi') {
steps {
sh 'nvidia-smi'
}
}
}
}{code}
with Dockerfile
FROM ubuntu:xenialRUN apt-get update && \ DEBIAN_FRONTEND=noninteractive apt-get install -y \ curl \ flake8 \ git-core \ openjdk-8-jdk \ python2.7 \ python-pip \ wgetRUN DEBIAN_FRONTEND=noninteractive pip install \ requests \ numpy# Build script looks for javac in jre dirENV JAVA_HOME "/usr/lib/jvm/java-8-openjdk-amd64"# http://spark.apache.org/docs/latest/building-spark.html#setting-up-mavens-memory-usage# We have a pretty beefy serverENV MAVEN_OPTS "-Xmx20g -XX:ReservedCodeCacheSize=2g"
> Spark Jenkins supports testing GPU-aware scheduling features
> ------------------------------------------------------------
>
> Key: SPARK-27365
> URL: https://issues.apache.org/jira/browse/SPARK-27365
> Project: Spark
> Issue Type: Story
> Components: jenkins
> Affects Versions: 3.0.0
> Reporter: Xiangrui Meng
> Priority: Major
>
> Upgrade Spark Jenkins to install GPU cards and run GPU integration tests triggered by "GPU" in PRs.
> cc: [~afeng] [~shaneknapp]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org