You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Dimitri (JIRA)" <ji...@apache.org> on 2017/04/28 20:37:04 UTC
[jira] [Created] (MESOS-7436) Running on Mesos the Jenkins and Jest
together crash silently Jenkins with code 137.
Dimitri created MESOS-7436:
------------------------------
Summary: Running on Mesos the Jenkins and Jest together crash silently Jenkins with code 137.
Key: MESOS-7436
URL: https://issues.apache.org/jira/browse/MESOS-7436
Project: Mesos
Issue Type: Bug
Affects Versions: 0.28.1
Environment: CPU
Intel(R) Xeon(R) CPU D-1540 @ 2.00GHz
Cores : 16
Cache : 12288KB
RAM
2x 16384MB
Disks
2 x 2000 GB
Motherboard
X10SDV-TLN4F
Kernel version
3.14.32-xxxx-grs-ipv6-64
node version 7.9.0
Reporter: Dimitri
I apologize for the tutorial, and would like to thanks in advance the person who will achieve it.
I am confident that test will crash in your environment, there is no reason to.
In case you just wan't to see the crash , I have recorded it with my smartphone, it's not a professional footage but you can see it :
https://drive.google.com/file/d/0B6drRJ3_BeQSMVh2RWE4cnc2NUU/view
Prerequesite
==========
In order to be exactly with the same configuration, you will need Host 32H offer at OVH.
```
CPU
Intel(R) Xeon(R) CPU D-1540 @ 2.00GHz
Cores : 16
Cache : 12288KB
RAM
2x 16384MB
Disks
2 x 2000 GB
Motherboard
X10SDV-TLN4F
Kernel version
3.14.32-xxxx-grs-ipv6-64
```
1. Install Debian Jessie. (can be skipped but potentially you won't get the error)
1. Install Mesos 0.28.1/Marathon 1.1.1. (can be skipped but potentially you won't get the error)
1. Install docker-engine 1.13.1. (can be skipped but potentially you won't get the error)
During my test, I use the following configuration with Debian Jessie 8.0:
Jenkins installation
===============
1. Download and run jenkins official image. From a separate bash: (you must have port 8080 available)
docker run -p 8080:8080 jenkins
2. You will see in your terminal password. Copy it.
Jenkins initial setup is required. An admin user has been created and a password generated.
Please use the following password to proceed to installation:
39ae474fe5544c00881167cb8764cb14
This may also be found at: /var/jenkins_home/secrets/initialAdminPassword
Or this will print the content of ` /var/jenkins_home/secrets/initialAdminPassword`:
docker exec -ti $(docker ps | grep jenkins | head -1 | awk -F ' ' '{print $1}') cat /var/jenkins_home/secrets/initialAdminPassword
3. Now open chrome `http://localhost:8080/` and past the code
![image](https://cloud.githubusercontent.com/assets/1866564/25540287/c0d3afbe-2c74-11e7-9935-6312c15d4e0d.png)
4. Click on **Select plugin to install**
![image](https://cloud.githubusercontent.com/assets/1866564/25540884/d6042cc2-2c76-11e7-9f3d-6408f8b34101.png)
5. Go to manage jenkins > manage plugin and complete the installation by checking for installation the following plugins :
- NodeJS
![image](https://cloud.githubusercontent.com/assets/1866564/25540958/251f60ec-2c77-11e7-9b5e-86cc45ff1903.png)
- Git plugin
![image](https://cloud.githubusercontent.com/assets/1866564/25541013/50b5c05c-2c77-11e7-8e51-f5ef33b1b354.png)
Validate the change
6. Wait for the download and installation of the jenkins plugins :
![image](https://cloud.githubusercontent.com/assets/1866564/25541083/87a26f16-2c77-11e7-863a-8aeb08469dae.png)
![image](https://cloud.githubusercontent.com/assets/1866564/25541512/1de8a6e2-2c79-11e7-802d-02617cb6c137.png)
7. Configure the administrator account as follow
![image](https://cloud.githubusercontent.com/assets/1866564/25541916/b0c6e8c4-2c7a-11e7-9eb9-6cc3cb76995b.png)
8. You will arrive on the jenkins dashboard
![image](https://cloud.githubusercontent.com/assets/1866564/25540668/26b551ba-2c76-11e7-83bb-71b367420694.png)
9. Click on "Manage Jenkins"
![image](https://cloud.githubusercontent.com/assets/1866564/25542110/62740aa2-2c7b-11e7-80af-4147116c7718.png)
10. Now click on "Global Tool Configuration"
![image](https://cloud.githubusercontent.com/assets/1866564/25542155/936a4b8a-2c7b-11e7-81d1-dc3255e0c71f.png)
11. Configure a NodeJS version 7.9.0
![image](https://cloud.githubusercontent.com/assets/1866564/25542270/0601af94-2c7c-11e7-9ab5-350c6933749e.png)
12. Click on "New Item"
![image](https://cloud.githubusercontent.com/assets/1866564/25541986/ed761074-2c7a-11e7-8b5d-e02931459dc5.png)
13. Name it and choose "Freestyle project"
![image](https://cloud.githubusercontent.com/assets/1866564/25542035/1e0b45ba-2c7b-11e7-9123-b7da523c5af5.png)
14. Under "Source Code Management", select the radio "git" and add the following repository url : `https://github.com/kopax/jest-crashing-jenkins`
![image](https://cloud.githubusercontent.com/assets/1866564/25542771/18dd687c-2c7e-11e7-9b9b-4def2b5f125f.png)
15. Under "Build Environment" check "provide Node & npm bin/ folder to path" and select the version "7.9.0" of NodeJS.
![image](https://cloud.githubusercontent.com/assets/1866564/25542877/8b64a716-2c7e-11e7-9280-fdbaae751145.png)
16. Under "Build", click on "Execute shell"
![image](https://cloud.githubusercontent.com/assets/1866564/25542956/e11b74fa-2c7e-11e7-8f53-414f8cd5beeb.png)
17. Enter "npm install" and "npm test", then click "Save"
![image](https://cloud.githubusercontent.com/assets/1866564/25543010/10b30c28-2c7f-11e7-8ade-a652cc19a7c4.png)
18. Now click on Build now
![image](https://cloud.githubusercontent.com/assets/1866564/25543059/399264cc-2c7f-11e7-817b-d54d6057404c.png)
19. There will be a new job added. You can click on it and then click on "Console Output"
![image](https://cloud.githubusercontent.com/assets/1866564/25543410/a054974c-2c80-11e7-8f71-68fde4aa0d70.png)
20. While running the build, you can use `htop` to view your memory usage (`sudo apt-get update && sudo apt-get install htop -y`)
![image](https://cloud.githubusercontent.com/assets/1866564/25543602/6f2b4516-2c81-11e7-9aad-88d62fdc583c.png)
21. Also you should note the `CONTAINER_ID` so you will see when it crash
docker ps | grep jenkins
abf10c67f072 jenkins "/bin/tini -- /usr/lo" About an hour ago Up About an hour 0.0.0.0:8080->8080/tcp, 50000/tcp reverent_saha
I wasn't able to reproduce the crash it except in my production environment.
This is how my test fail in production :
![image](https://cloud.githubusercontent.com/assets/1866564/25545305/5c1ba414-2c88-11e7-8005-ad2f4c33e749.png)
Notice the Jobs id 48, because when Jenkins will come back, it will miss that ID and jenkins will do like nothing happened before.
_Tips 1: If you wish to have a more verbose log level when doing logs of your container you can do the following withing the containers:_
docker exec -it $(docker ps | grep jenkins | head -1 | awk -F ' ' '{print $1}') cat > /var/jenkins_home/log.properties <<EOF
handlers=java.util.logging.ConsoleHandler
jenkins.level=FINEST
java.util.logging.ConsoleHandler.level=FINEST
EOF
And then restart your container with :
docker stop $(docker ps | grep jenkins | head -1 | awk -F ' ' '{print $1}')
docker start $(docker ps | grep jenkins | head -1 | awk -F ' ' '{print $1}') --env JAVA_OPTS="-Djava.util.logging.config.file=/var/jenkins_home/log.properties"
_Tips 2: If you want to upgrade jenkins to the latest version (see the warning), you can activate a hidden button "Upgrade jenkins automatically button". For that, you need to give the good permissions to `/usr/share/jenkins` within the container, this will make the button appear in the "Jenkins Configuration" page and you will then be able to upgrade Jenkins_
docker exec -it --user root $(docker ps | grep jenkins | head -1 | awk -F ' ' '{print $1}') chown jenkins:jenkins -R /usr/share/jenkins
Note that I am running my Jenkins container on Mesos/Marathon and I only have the issue here. As attachement this is the marathon deployement json I use for deploying my jenkins container:
```json
{
"id": "developers/jenkins",
"cmd": null,
"mem": 4096,
"cpus": 2,
"instances": 1,
"container": {
"type": "DOCKER",
"docker": {
"image": "jenkins:latest",
"forcePullImage": true,
"network": "BRIDGE",
"portMappings": [
{
"containerPort": 8080,
"hostPort": 0,
"protocol": "tcp"
}
]
},
"volumes": [
{
"containerPath": "/var/jenkins_home",
"hostPath": "/srv/developers/jenkins/var/jenkins_home",
"mode": "RW"
},
{
"containerPath": "/var/run/docker.sock",
"hostPath": "/var/run/docker.sock",
"mode": "RW"
},
{
"containerPath": "/usr/bin/docker",
"hostPath": "/usr/bin/docker",
"mode": "RW"
},
{
"containerPath": "/lib/x86_64-linux-gnu/libapparmor.so.1",
"hostPath": "/usr/lib/x86_64-linux-gnu/libapparmor.so.1",
"mode": "RW"
}
]
},
"uris": [
"file:///etc/docker.tar.gz"
],
"healthChecks": [
{
"portIndex": 0,
"protocol": "TCP",
"gracePeriodSeconds": 30,
"intervalSeconds": 10,
"timeoutSeconds": 30,
"maxConsecutiveFailures": 3
}
],
"upgradeStrategy": {
"minimumHealthCapacity": 1,
"maximumOverCapacity": 0.1
}
}
```
This is the error I have on mesos :
```
processname:marathon groupname:marathon pid:19 channel:stdout
[2017-04-28 20:21:48,630] INFO Received status update for task kopaxgroup_developers_jenkins-2.dc2b0461-2c4d-11e7-9190-92077ff7c44b: TASK_FAILED (Docker container run error: Container exited on error: exited with status 137) (mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$a504fd7e:Thread-1442)
```
I have tried to find why, but the only place where I can reproduce this error is on mesos. Any Idea ?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)