You are viewing a plain text version of this content. The canonical link for it is here.
Posted to infrastructure-issues@apache.org by "stack (JIRA)" <ji...@apache.org> on 2011/06/28 21:22:17 UTC
[jira] [Created] (INFRA-3734) Building hbase we run into 'too many
files'; can we up the ulimit > 1024?
Building hbase we run into 'too many files'; can we up the ulimit > 1024?
-------------------------------------------------------------------------
Key: INFRA-3734
URL: https://issues.apache.org/jira/browse/INFRA-3734
Project: Infrastructure
Issue Type: Task
Security Level: public (Regular issues)
Components: Jenkins
Reporter: stack
Buliding on hudson we'll run into 'too many files' issues when running some of our more intense tests (Yes, hbase is an fd hog): see https://builds.apache.org/view/G-L/view/HBase/job/hbase-0.90/215/artifact/trunk/target/surefire-reports/org.apache.hadoop.hbase.client.TestFromClientSide.txt for an example. I tried upping fds for the the build process by having jenkins execute some shell before the maven build cuts in but this failed -- its not allowed apparently.
Any chance of having the jenkins user's fds upped? By 10x?
Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (INFRA-3734) Building hbase we run into 'too
many files'; can we up the ulimit > 1024?
Posted by "Tony Stevenson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13058841#comment-13058841 ]
Tony Stevenson commented on INFRA-3734:
---------------------------------------
I have fixed the ulimit issue, via ssh to vesta and hemera
- Pre Fix -
bash-3.2$ ssh pctony@vesta.apache.org ulimit -n
1024
- Post Fix -
bash-3.2$ ssh pctony@vesta.apache.org ulimit -n
40000
- Pre Fix -
bash-3.2$ ssh pctony@hemera.apache.org ulimit -n
1024
- Post Fix -
bash-3.2$ ssh pctony@hemera.apache.org ulimit -n
40000
> Building hbase we run into 'too many files'; can we up the ulimit > 1024?
> -------------------------------------------------------------------------
>
> Key: INFRA-3734
> URL: https://issues.apache.org/jira/browse/INFRA-3734
> Project: Infrastructure
> Issue Type: Task
> Security Level: public(Regular issues)
> Components: Jenkins
> Reporter: stack
> Assignee: Tony Stevenson
>
> Buliding on hudson we'll run into 'too many files' issues when running some of our more intense tests (Yes, hbase is an fd hog): see https://builds.apache.org/view/G-L/view/HBase/job/hbase-0.90/215/artifact/trunk/target/surefire-reports/org.apache.hadoop.hbase.client.TestFromClientSide.txt for an example. I tried upping fds for the the build process by having jenkins execute some shell before the maven build cuts in but this failed -- its not allowed apparently.
> Any chance of having the jenkins user's fds upped? By 10x?
> Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (INFRA-3734) Building hbase we run into 'too
many files'; can we up the ulimit > 1024?
Posted by "Gavin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13058233#comment-13058233 ]
Gavin commented on INFRA-3734:
------------------------------
I've compared configs on all 3, did some go-ogling around and tried stuff to no avail.
Fresh eyes needed to look at the issue.
> Building hbase we run into 'too many files'; can we up the ulimit > 1024?
> -------------------------------------------------------------------------
>
> Key: INFRA-3734
> URL: https://issues.apache.org/jira/browse/INFRA-3734
> Project: Infrastructure
> Issue Type: Task
> Security Level: public(Regular issues)
> Components: Jenkins
> Reporter: stack
>
> Buliding on hudson we'll run into 'too many files' issues when running some of our more intense tests (Yes, hbase is an fd hog): see https://builds.apache.org/view/G-L/view/HBase/job/hbase-0.90/215/artifact/trunk/target/surefire-reports/org.apache.hadoop.hbase.client.TestFromClientSide.txt for an example. I tried upping fds for the the build process by having jenkins execute some shell before the maven build cuts in but this failed -- its not allowed apparently.
> Any chance of having the jenkins user's fds upped? By 10x?
> Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (INFRA-3734) Building hbase we run into 'too
many files'; can we up the ulimit > 1024?
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060647#comment-13060647 ]
stack commented on INFRA-3734:
------------------------------
I tried to add 'hemera.apache.org' in place of 'ubuntu2' as I had in our job config and I get this:
..pending -- all nodes of label 'hemera.apache.org' are offline
I left it up here https://builds.apache.org/job/HBase-TRUNK/ in case you want to check.
In the config screen -- https://builds.apache.org/job/HBase-TRUNK/configure -- it also says no such label whether I use hemera or hemera.apache.org. Does same if I specify minerva (I'm reading here http://wiki.apache.org/general/Hudson).
Sorry for all the trouble caused.
> Building hbase we run into 'too many files'; can we up the ulimit > 1024?
> -------------------------------------------------------------------------
>
> Key: INFRA-3734
> URL: https://issues.apache.org/jira/browse/INFRA-3734
> Project: Infrastructure
> Issue Type: Task
> Security Level: public(Regular issues)
> Components: Jenkins
> Reporter: stack
> Assignee: Tony Stevenson
>
> Buliding on hudson we'll run into 'too many files' issues when running some of our more intense tests (Yes, hbase is an fd hog): see https://builds.apache.org/view/G-L/view/HBase/job/hbase-0.90/215/artifact/trunk/target/surefire-reports/org.apache.hadoop.hbase.client.TestFromClientSide.txt for an example. I tried upping fds for the the build process by having jenkins execute some shell before the maven build cuts in but this failed -- its not allowed apparently.
> Any chance of having the jenkins user's fds upped? By 10x?
> Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (INFRA-3734) Building hbase we run into 'too
many files'; can we up the ulimit > 1024?
Posted by "Niklas Gustavsson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13059706#comment-13059706 ]
Niklas Gustavsson commented on INFRA-3734:
------------------------------------------
Fixed on minerva and hemera by adding "ulimit -n 40000" to /etc/init.d/ssh. But it won't bite on vesta, no idea why.
$ sudo -u hudson ssh hemera.apache.org ulimit -n
40000
$ sudo -u hudson ssh minerva.apache.org ulimit -n
40000
$ sudo -u hudson ssh vesta.apache.org ulimit -n
1024
> Building hbase we run into 'too many files'; can we up the ulimit > 1024?
> -------------------------------------------------------------------------
>
> Key: INFRA-3734
> URL: https://issues.apache.org/jira/browse/INFRA-3734
> Project: Infrastructure
> Issue Type: Task
> Security Level: public(Regular issues)
> Components: Jenkins
> Reporter: stack
> Assignee: Tony Stevenson
>
> Buliding on hudson we'll run into 'too many files' issues when running some of our more intense tests (Yes, hbase is an fd hog): see https://builds.apache.org/view/G-L/view/HBase/job/hbase-0.90/215/artifact/trunk/target/surefire-reports/org.apache.hadoop.hbase.client.TestFromClientSide.txt for an example. I tried upping fds for the the build process by having jenkins execute some shell before the maven build cuts in but this failed -- its not allowed apparently.
> Any chance of having the jenkins user's fds upped? By 10x?
> Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (INFRA-3734) Building hbase we run into 'too
many files'; can we up the ulimit > 1024?
Posted by "Gavin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13056802#comment-13056802 ]
Gavin commented on INFRA-3734:
------------------------------
That build was performed on Ubuntu 2 which is Vesta which already has a soft limit of 40000 and a hard limit of 60000 . I would have thought that plenty myself.
> Building hbase we run into 'too many files'; can we up the ulimit > 1024?
> -------------------------------------------------------------------------
>
> Key: INFRA-3734
> URL: https://issues.apache.org/jira/browse/INFRA-3734
> Project: Infrastructure
> Issue Type: Task
> Security Level: public(Regular issues)
> Components: Jenkins
> Reporter: stack
>
> Buliding on hudson we'll run into 'too many files' issues when running some of our more intense tests (Yes, hbase is an fd hog): see https://builds.apache.org/view/G-L/view/HBase/job/hbase-0.90/215/artifact/trunk/target/surefire-reports/org.apache.hadoop.hbase.client.TestFromClientSide.txt for an example. I tried upping fds for the the build process by having jenkins execute some shell before the maven build cuts in but this failed -- its not allowed apparently.
> Any chance of having the jenkins user's fds upped? By 10x?
> Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (INFRA-3734) Building hbase we run into 'too many
files'; can we up the ulimit > 1024?
Posted by "Tony Stevenson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tony Stevenson reassigned INFRA-3734:
-------------------------------------
Assignee: Tony Stevenson
> Building hbase we run into 'too many files'; can we up the ulimit > 1024?
> -------------------------------------------------------------------------
>
> Key: INFRA-3734
> URL: https://issues.apache.org/jira/browse/INFRA-3734
> Project: Infrastructure
> Issue Type: Task
> Security Level: public(Regular issues)
> Components: Jenkins
> Reporter: stack
> Assignee: Tony Stevenson
>
> Buliding on hudson we'll run into 'too many files' issues when running some of our more intense tests (Yes, hbase is an fd hog): see https://builds.apache.org/view/G-L/view/HBase/job/hbase-0.90/215/artifact/trunk/target/surefire-reports/org.apache.hadoop.hbase.client.TestFromClientSide.txt for an example. I tried upping fds for the the build process by having jenkins execute some shell before the maven build cuts in but this failed -- its not allowed apparently.
> Any chance of having the jenkins user's fds upped? By 10x?
> Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (INFRA-3734) Building hbase we run into 'too many
files'; can we up the ulimit > 1024?
Posted by "Tony Stevenson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tony Stevenson closed INFRA-3734.
---------------------------------
Resolution: Fixed
Closing as the ulimit is now correctly set at 40k for any user on all 3 hosts.
> Building hbase we run into 'too many files'; can we up the ulimit > 1024?
> -------------------------------------------------------------------------
>
> Key: INFRA-3734
> URL: https://issues.apache.org/jira/browse/INFRA-3734
> Project: Infrastructure
> Issue Type: Task
> Security Level: public(Regular issues)
> Components: Jenkins
> Reporter: stack
> Assignee: Tony Stevenson
>
> Buliding on hudson we'll run into 'too many files' issues when running some of our more intense tests (Yes, hbase is an fd hog): see https://builds.apache.org/view/G-L/view/HBase/job/hbase-0.90/215/artifact/trunk/target/surefire-reports/org.apache.hadoop.hbase.client.TestFromClientSide.txt for an example. I tried upping fds for the the build process by having jenkins execute some shell before the maven build cuts in but this failed -- its not allowed apparently.
> Any chance of having the jenkins user's fds upped? By 10x?
> Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (INFRA-3734) Building hbase we run into 'too many
files'; can we up the ulimit > 1024?
Posted by "Tony Stevenson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tony Stevenson closed INFRA-3734.
---------------------------------
Resolution: Fixed
> Building hbase we run into 'too many files'; can we up the ulimit > 1024?
> -------------------------------------------------------------------------
>
> Key: INFRA-3734
> URL: https://issues.apache.org/jira/browse/INFRA-3734
> Project: Infrastructure
> Issue Type: Task
> Security Level: public(Regular issues)
> Components: Jenkins
> Reporter: stack
> Assignee: Tony Stevenson
>
> Buliding on hudson we'll run into 'too many files' issues when running some of our more intense tests (Yes, hbase is an fd hog): see https://builds.apache.org/view/G-L/view/HBase/job/hbase-0.90/215/artifact/trunk/target/surefire-reports/org.apache.hadoop.hbase.client.TestFromClientSide.txt for an example. I tried upping fds for the the build process by having jenkins execute some shell before the maven build cuts in but this failed -- its not allowed apparently.
> Any chance of having the jenkins user's fds upped? By 10x?
> Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (INFRA-3734) Building hbase we run into 'too
many files'; can we up the ulimit > 1024?
Posted by "Tony Stevenson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060917#comment-13060917 ]
Tony Stevenson commented on INFRA-3734:
---------------------------------------
I have fixed Vesta now.
bash-3.2$ ssh pctony@vesta.apache.org "ulimit -n"
40000
It seems that ssh was restarted with 'service ssh {restart, stop, start}' This has two issues,
1- it doesnt use the init script, and thus ignores the setting. If you tried restarting sshd with the init script, it would have no effect as sshd was started using the service tool.
2- sshd was started with a -D flag, basically dont daemonize. So I had to kill sshd, and start it with the init script.
> Building hbase we run into 'too many files'; can we up the ulimit > 1024?
> -------------------------------------------------------------------------
>
> Key: INFRA-3734
> URL: https://issues.apache.org/jira/browse/INFRA-3734
> Project: Infrastructure
> Issue Type: Task
> Security Level: public(Regular issues)
> Components: Jenkins
> Reporter: stack
> Assignee: Tony Stevenson
>
> Buliding on hudson we'll run into 'too many files' issues when running some of our more intense tests (Yes, hbase is an fd hog): see https://builds.apache.org/view/G-L/view/HBase/job/hbase-0.90/215/artifact/trunk/target/surefire-reports/org.apache.hadoop.hbase.client.TestFromClientSide.txt for an example. I tried upping fds for the the build process by having jenkins execute some shell before the maven build cuts in but this failed -- its not allowed apparently.
> Any chance of having the jenkins user's fds upped? By 10x?
> Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (INFRA-3734) Building hbase we run into 'too
many files'; can we up the ulimit > 1024?
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13056926#comment-13056926 ]
stack commented on INFRA-3734:
------------------------------
Sounds like you fellas can see the prob.
If you want evidence that user who runs our build only has 1024 fds, see end of this console output where we print out ulimit and then what happens when I try to up the fds: https://builds.apache.org/job/HBase-TRUNK/1994/console (repeated below).
{code}
...
[HBase-TRUNK] $ /bin/bash -xe /tmp/hudson323118671237062776.sh
+ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 63937
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 63937
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
+ ulimit -n 10240
/tmp/hudson323118671237062776.sh: line 3: ulimit: open files: cannot modify limit: Operation not permitted
{code}
> Building hbase we run into 'too many files'; can we up the ulimit > 1024?
> -------------------------------------------------------------------------
>
> Key: INFRA-3734
> URL: https://issues.apache.org/jira/browse/INFRA-3734
> Project: Infrastructure
> Issue Type: Task
> Security Level: public(Regular issues)
> Components: Jenkins
> Reporter: stack
>
> Buliding on hudson we'll run into 'too many files' issues when running some of our more intense tests (Yes, hbase is an fd hog): see https://builds.apache.org/view/G-L/view/HBase/job/hbase-0.90/215/artifact/trunk/target/surefire-reports/org.apache.hadoop.hbase.client.TestFromClientSide.txt for an example. I tried upping fds for the the build process by having jenkins execute some shell before the maven build cuts in but this failed -- its not allowed apparently.
> Any chance of having the jenkins user's fds upped? By 10x?
> Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (INFRA-3734) Building hbase we run into 'too
many files'; can we up the ulimit > 1024?
Posted by "Tony Stevenson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13058842#comment-13058842 ]
Tony Stevenson commented on INFRA-3734:
---------------------------------------
As per svn commit history, the fix was to enable Pam support in the sshd config
> Building hbase we run into 'too many files'; can we up the ulimit > 1024?
> -------------------------------------------------------------------------
>
> Key: INFRA-3734
> URL: https://issues.apache.org/jira/browse/INFRA-3734
> Project: Infrastructure
> Issue Type: Task
> Security Level: public(Regular issues)
> Components: Jenkins
> Reporter: stack
> Assignee: Tony Stevenson
>
> Buliding on hudson we'll run into 'too many files' issues when running some of our more intense tests (Yes, hbase is an fd hog): see https://builds.apache.org/view/G-L/view/HBase/job/hbase-0.90/215/artifact/trunk/target/surefire-reports/org.apache.hadoop.hbase.client.TestFromClientSide.txt for an example. I tried upping fds for the the build process by having jenkins execute some shell before the maven build cuts in but this failed -- its not allowed apparently.
> Any chance of having the jenkins user's fds upped? By 10x?
> Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (INFRA-3734) Building hbase we run into 'too many
files'; can we up the ulimit > 1024?
Posted by "Gavin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gavin reopened INFRA-3734:
--------------------------
Sorry, I reverted those. It completely screwed up my shell on minerva.
Note minvera HAD been working fine all this time WITHOUT pam=yes
Note vesta HAD been working fine WITHOUT pam=yes up until it got rebuilt.
Now I reverted, minerva does not work either. What else got changed?
> Building hbase we run into 'too many files'; can we up the ulimit > 1024?
> -------------------------------------------------------------------------
>
> Key: INFRA-3734
> URL: https://issues.apache.org/jira/browse/INFRA-3734
> Project: Infrastructure
> Issue Type: Task
> Security Level: public(Regular issues)
> Components: Jenkins
> Reporter: stack
> Assignee: Tony Stevenson
>
> Buliding on hudson we'll run into 'too many files' issues when running some of our more intense tests (Yes, hbase is an fd hog): see https://builds.apache.org/view/G-L/view/HBase/job/hbase-0.90/215/artifact/trunk/target/surefire-reports/org.apache.hadoop.hbase.client.TestFromClientSide.txt for an example. I tried upping fds for the the build process by having jenkins execute some shell before the maven build cuts in but this failed -- its not allowed apparently.
> Any chance of having the jenkins user's fds upped? By 10x?
> Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (INFRA-3734) Building hbase we run into 'too
many files'; can we up the ulimit > 1024?
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060925#comment-13060925 ]
stack commented on INFRA-3734:
------------------------------
Thanks all.
> Building hbase we run into 'too many files'; can we up the ulimit > 1024?
> -------------------------------------------------------------------------
>
> Key: INFRA-3734
> URL: https://issues.apache.org/jira/browse/INFRA-3734
> Project: Infrastructure
> Issue Type: Task
> Security Level: public(Regular issues)
> Components: Jenkins
> Reporter: stack
> Assignee: Tony Stevenson
>
> Buliding on hudson we'll run into 'too many files' issues when running some of our more intense tests (Yes, hbase is an fd hog): see https://builds.apache.org/view/G-L/view/HBase/job/hbase-0.90/215/artifact/trunk/target/surefire-reports/org.apache.hadoop.hbase.client.TestFromClientSide.txt for an example. I tried upping fds for the the build process by having jenkins execute some shell before the maven build cuts in but this failed -- its not allowed apparently.
> Any chance of having the jenkins user's fds upped? By 10x?
> Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (INFRA-3734) Building hbase we run into 'too
many files'; can we up the ulimit > 1024?
Posted by "Niklas Gustavsson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13058251#comment-13058251 ]
Niklas Gustavsson commented on INFRA-3734:
------------------------------------------
I did the same, but I'll give it another try.
> Building hbase we run into 'too many files'; can we up the ulimit > 1024?
> -------------------------------------------------------------------------
>
> Key: INFRA-3734
> URL: https://issues.apache.org/jira/browse/INFRA-3734
> Project: Infrastructure
> Issue Type: Task
> Security Level: public(Regular issues)
> Components: Jenkins
> Reporter: stack
>
> Buliding on hudson we'll run into 'too many files' issues when running some of our more intense tests (Yes, hbase is an fd hog): see https://builds.apache.org/view/G-L/view/HBase/job/hbase-0.90/215/artifact/trunk/target/surefire-reports/org.apache.hadoop.hbase.client.TestFromClientSide.txt for an example. I tried upping fds for the the build process by having jenkins execute some shell before the maven build cuts in but this failed -- its not allowed apparently.
> Any chance of having the jenkins user's fds upped? By 10x?
> Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (INFRA-3734) Building hbase we run into 'too
many files'; can we up the ulimit > 1024?
Posted by "Niklas Gustavsson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060639#comment-13060639 ]
Niklas Gustavsson commented on INFRA-3734:
------------------------------------------
That would help you for now. And I would recommend hemera. I will continue to track down the problems on vesta, after that's done, you should move back to ubuntu in general.
> Building hbase we run into 'too many files'; can we up the ulimit > 1024?
> -------------------------------------------------------------------------
>
> Key: INFRA-3734
> URL: https://issues.apache.org/jira/browse/INFRA-3734
> Project: Infrastructure
> Issue Type: Task
> Security Level: public(Regular issues)
> Components: Jenkins
> Reporter: stack
> Assignee: Tony Stevenson
>
> Buliding on hudson we'll run into 'too many files' issues when running some of our more intense tests (Yes, hbase is an fd hog): see https://builds.apache.org/view/G-L/view/HBase/job/hbase-0.90/215/artifact/trunk/target/surefire-reports/org.apache.hadoop.hbase.client.TestFromClientSide.txt for an example. I tried upping fds for the the build process by having jenkins execute some shell before the maven build cuts in but this failed -- its not allowed apparently.
> Any chance of having the jenkins user's fds upped? By 10x?
> Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (INFRA-3734) Building hbase we run into 'too
many files'; can we up the ulimit > 1024?
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13058843#comment-13058843 ]
stack commented on INFRA-3734:
------------------------------
You the man Tony. Thank you.
> Building hbase we run into 'too many files'; can we up the ulimit > 1024?
> -------------------------------------------------------------------------
>
> Key: INFRA-3734
> URL: https://issues.apache.org/jira/browse/INFRA-3734
> Project: Infrastructure
> Issue Type: Task
> Security Level: public(Regular issues)
> Components: Jenkins
> Reporter: stack
> Assignee: Tony Stevenson
>
> Buliding on hudson we'll run into 'too many files' issues when running some of our more intense tests (Yes, hbase is an fd hog): see https://builds.apache.org/view/G-L/view/HBase/job/hbase-0.90/215/artifact/trunk/target/surefire-reports/org.apache.hadoop.hbase.client.TestFromClientSide.txt for an example. I tried upping fds for the the build process by having jenkins execute some shell before the maven build cuts in but this failed -- its not allowed apparently.
> Any chance of having the jenkins user's fds upped? By 10x?
> Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (INFRA-3734) Building hbase we run into 'too
many files'; can we up the ulimit > 1024?
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060635#comment-13060635 ]
stack commented on INFRA-3734:
------------------------------
Should we just move our build to run on hemera or minerva?
> Building hbase we run into 'too many files'; can we up the ulimit > 1024?
> -------------------------------------------------------------------------
>
> Key: INFRA-3734
> URL: https://issues.apache.org/jira/browse/INFRA-3734
> Project: Infrastructure
> Issue Type: Task
> Security Level: public(Regular issues)
> Components: Jenkins
> Reporter: stack
> Assignee: Tony Stevenson
>
> Buliding on hudson we'll run into 'too many files' issues when running some of our more intense tests (Yes, hbase is an fd hog): see https://builds.apache.org/view/G-L/view/HBase/job/hbase-0.90/215/artifact/trunk/target/surefire-reports/org.apache.hadoop.hbase.client.TestFromClientSide.txt for an example. I tried upping fds for the the build process by having jenkins execute some shell before the maven build cuts in but this failed -- its not allowed apparently.
> Any chance of having the jenkins user's fds upped? By 10x?
> Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (INFRA-3734) Building hbase we run into 'too
many files'; can we up the ulimit > 1024?
Posted by "Niklas Gustavsson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060659#comment-13060659 ]
Niklas Gustavsson commented on INFRA-3734:
------------------------------------------
hemera is called "ubuntu3" in Jenkins.
> Building hbase we run into 'too many files'; can we up the ulimit > 1024?
> -------------------------------------------------------------------------
>
> Key: INFRA-3734
> URL: https://issues.apache.org/jira/browse/INFRA-3734
> Project: Infrastructure
> Issue Type: Task
> Security Level: public(Regular issues)
> Components: Jenkins
> Reporter: stack
> Assignee: Tony Stevenson
>
> Buliding on hudson we'll run into 'too many files' issues when running some of our more intense tests (Yes, hbase is an fd hog): see https://builds.apache.org/view/G-L/view/HBase/job/hbase-0.90/215/artifact/trunk/target/surefire-reports/org.apache.hadoop.hbase.client.TestFromClientSide.txt for an example. I tried upping fds for the the build process by having jenkins execute some shell before the maven build cuts in but this failed -- its not allowed apparently.
> Any chance of having the jenkins user's fds upped? By 10x?
> Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (INFRA-3734) Building hbase we run into 'too
many files'; can we up the ulimit > 1024?
Posted by "Niklas Gustavsson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13056803#comment-13056803 ]
Niklas Gustavsson commented on INFRA-3734:
------------------------------------------
vesta has the same problem as we found previously on minerva, where jenkins only get 1024 when connecting over ssh:
ngn@aegis:~$ sudo -u hudson ssh vesta.apache.org ulimit -n
1024
ngn@aegis:~$ sudo -u hudson ssh minerva.apache.org ulimit -n
40000
ngn@aegis:~$ sudo -u hudson ssh hemera.apache.org ulimit -n
1024
But I fail to recall how we fixed it on minerva.
> Building hbase we run into 'too many files'; can we up the ulimit > 1024?
> -------------------------------------------------------------------------
>
> Key: INFRA-3734
> URL: https://issues.apache.org/jira/browse/INFRA-3734
> Project: Infrastructure
> Issue Type: Task
> Security Level: public(Regular issues)
> Components: Jenkins
> Reporter: stack
>
> Buliding on hudson we'll run into 'too many files' issues when running some of our more intense tests (Yes, hbase is an fd hog): see https://builds.apache.org/view/G-L/view/HBase/job/hbase-0.90/215/artifact/trunk/target/surefire-reports/org.apache.hadoop.hbase.client.TestFromClientSide.txt for an example. I tried upping fds for the the build process by having jenkins execute some shell before the maven build cuts in but this failed -- its not allowed apparently.
> Any chance of having the jenkins user's fds upped? By 10x?
> Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira