You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@apache.org by Patrick Hunt <ph...@apache.org> on 2014/09/03 19:20:28 UTC

Re: ulimit changed with Apache Jenkins upgrade?

I've opened a BUILDS jira for this:
https://issues.apache.org/jira/browse/BUILDS-17

On Sat, Aug 30, 2014 at 12:16 PM, Patrick Hunt <ph...@apache.org> wrote:
> Giri could you or one of the other folks look into this? Our tests
> have been broken for some time.
>
> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2303/
>
> The same test is still failing with the same error (too many open
> files) and afaict the ulimit is still just set to 4k. I'm printing the
> ulimit info at the start of the job, here it is:
>
> core file size          (blocks, -c) 0
> data seg size           (kbytes, -d) unlimited
> scheduling priority             (-e) 0
> file size               (blocks, -f) unlimited
> pending signals                 (-i) 386177
> max locked memory       (kbytes, -l) 64
> max memory size         (kbytes, -m) unlimited
> open files                      (-n) 4096
> pipe size            (512 bytes, -p) 8
> POSIX message queues     (bytes, -q) 819200
> real-time priority              (-r) 0
> stack size              (kbytes, -s) 8192
> cpu time               (seconds, -t) unlimited
> max user processes              (-u) 386177
> virtual memory          (kbytes, -v) unlimited
> file locks                      (-x) unlimited
>
>
> Patrick
>
> On Wed, Jul 23, 2014 at 11:21 AM, Patrick Hunt <ph...@apache.org> wrote:
>> Does someone in builds@ have the ability to address this? (restart the
>> jenkins slaves so that the new ulimit limits will take effect)
>>
>> Thanks!
>>
>> Patrick
>>
>> On Wed, Jul 23, 2014 at 9:33 AM, Patrick Hunt <ph...@apache.org> wrote:
>>> Giri do you mean me? I don't have access to that afaict.
>>>
>>> Patrick
>>>
>>> On Wed, Jul 23, 2014 at 12:50 AM, Giridharan Kesavan
>>> <gk...@hortonworks.com> wrote:
>>>> jenkins slaves might need a restart.
>>>>
>>>> Could you pls re-launch the slaves from the jenkins UI configuration page?
>>>>
>>>> -giri
>>>>
>>>>
>>>> On Tue, Jul 22, 2014 at 2:25 PM, Patrick Hunt <ph...@apache.org> wrote:
>>>>>
>>>>> Thanks Giri! Unfortunately though it seems to not have taken effect, I
>>>>> just kicked off a precommit build and I see
>>>>>
>>>>> core file size          (blocks, -c) 0
>>>>> data seg size           (kbytes, -d) unlimited
>>>>> scheduling priority             (-e) 0
>>>>> file size               (blocks, -f) unlimited
>>>>> pending signals                 (-i) 386178
>>>>> max locked memory       (kbytes, -l) 64
>>>>> max memory size         (kbytes, -m) unlimited
>>>>> open files                      (-n) 4096
>>>>> pipe size            (512 bytes, -p) 8
>>>>> POSIX message queues     (bytes, -q) 819200
>>>>> real-time priority              (-r) 0
>>>>> stack size              (kbytes, -s) 8192
>>>>> cpu time               (seconds, -t) unlimited
>>>>> max user processes              (-u) 386178
>>>>> virtual memory          (kbytes, -v) unlimited
>>>>> file locks                      (-x) unlimited
>>>>>
>>>>> more details here:
>>>>>
>>>>> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2215/console
>>>>>
>>>>> Patrick
>>>>>
>>>>> On Tue, Jul 22, 2014 at 1:29 PM, Giridharan Kesavan
>>>>> <gk...@hortonworks.com> wrote:
>>>>> >
>>>>> >
>>>>> > jenkins@asf901:~$ ulimit -a
>>>>> > core file size          (blocks, -c) 0
>>>>> > data seg size           (kbytes, -d) unlimited
>>>>> > scheduling priority             (-e) 0
>>>>> > file size               (blocks, -f) unlimited
>>>>> > pending signals                 (-i) 386178
>>>>> > max locked memory       (kbytes, -l) 64
>>>>> > max memory size         (kbytes, -m) unlimited
>>>>> > open files                      (-n) 60000
>>>>> > pipe size            (512 bytes, -p) 8
>>>>> > POSIX message queues     (bytes, -q) 819200
>>>>> > real-time priority              (-r) 0
>>>>> > stack size              (kbytes, -s) 8192
>>>>> > cpu time               (seconds, -t) unlimited
>>>>> > max user processes              (-u) 10240
>>>>> > virtual memory          (kbytes, -v) unlimited
>>>>> > file locks                      (-x) unlimited
>>>>> >
>>>>> > bumped up the open files and max user processes on all the slaves.
>>>>> >
>>>>> >
>>>>> >
>>>>> > -giri
>>>>> >
>>>>> >
>>>>> > On Tue, Jul 22, 2014 at 12:04 PM, Patrick Hunt <ph...@apache.org> wrote:
>>>>> >>
>>>>> >> Giri any chance can you take a look at the ulimit issue on the H#
>>>>> >> machines? All the ZK precommit builds are failing as a result.
>>>>> >>
>>>>> >> I updated the precommit build last night to output "ulimit -a" and it
>>>>> >> says the current limit is 4096, can we bump that up or set the default
>>>>> >> to unlimited?
>>>>> >>
>>>>> >> Thanks!
>>>>> >>
>>>>> >> Patrick
>>>>> >>
>>>>> >> On Sun, Jul 20, 2014 at 10:32 PM, Rakesh R <ra...@huawei.com> wrote:
>>>>> >> > +1
>>>>> >> >
>>>>> >> >
>>>>> >> > Adding one more point. I could see the following error too in the
>>>>> >> > pre-commit build.
>>>>> >> >
>>>>> >> >      [exec]     [junit] Exception in thread "CommitProcWorkThread-16"
>>>>> >> > java.lang.NoClassDefFoundError:
>>>>> >> > org/apache/zookeeper/server/ConnectionBean
>>>>> >> >      [exec]     [junit]         at
>>>>> >> >
>>>>> >> > org.apache.zookeeper.server.ServerCnxnFactory.registerConnection(ServerCnxnFactory.java:159)
>>>>> >> >      [exec]     [junit]         at
>>>>> >> >
>>>>> >> > org.apache.zookeeper.server.ZooKeeperServer.finishSessionInit(ZooKeeperServer.java:594)
>>>>> >> >      [exec]     [junit]         at
>>>>> >> >
>>>>> >> > org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:198)
>>>>> >> >      [exec]     [junit]         at
>>>>> >> >
>>>>> >> > org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:295)
>>>>> >> >      [exec]     [junit]         at
>>>>> >> >
>>>>> >> > org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:161)
>>>>> >> >      [exec]     [junit]         at
>>>>> >> >
>>>>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>>>> >> >      [exec]     [junit]         at
>>>>> >> >
>>>>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>>>> >> >      [exec]     [junit]         at
>>>>> >> > java.lang.Thread.run(Thread.java:662)
>>>>> >> >
>>>>> >> > -Rakesh
>>>>> >> >
>>>>> >> > -----Original Message-----
>>>>> >> > From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]
>>>>> >> > Sent: 21 July 2014 02:19
>>>>> >> > To: dev@zookeeper.apache.org
>>>>> >> > Cc: Andrew Bayer; builds@apache.org; Giridharan Kesavan
>>>>> >> > Subject: Re: ulimit changed with Apache Jenkins upgrade?
>>>>> >> >
>>>>> >> > +1
>>>>> >> >
>>>>> >> > On 18 Jul 2014, at 18:59, Patrick Hunt <ph...@apache.org> wrote:
>>>>> >> >
>>>>> >> >> Hi builds folks, is this a system wide issue or something we should
>>>>> >> >> address ourselves? Thanks!
>>>>> >> >>
>>>>> >> >> Patrick
>>>>> >> >>
>>>>> >> >> ---------- Forwarded message ----------
>>>>> >> >> From: Patrick Hunt <ph...@apache.org>
>>>>> >> >> Date: Fri, Jul 18, 2014 at 10:38 AM
>>>>> >> >> Subject: ulimit changed with Apache Jenkins upgrade?
>>>>> >> >> To: Giridharan Kesavan <gk...@hortonworks.com>
>>>>> >> >> Cc: DevZooKeeper <de...@zookeeper.apache.org>, Andrew Bayer
>>>>> >> >> <an...@cloudera.com>
>>>>> >> >>
>>>>> >> >>
>>>>> >> >> Hi Giri, can you check that the new hosts (H#) have the ulimit set
>>>>> >> >> to
>>>>> >> >> what it was set to on the original hadoop# hosts? I'm seeing new
>>>>> >> >> test
>>>>> >> >> failures with
>>>>> >> >>
>>>>> >> >>     [exec]     [junit] java.io.FileNotFoundException:
>>>>> >> >>
>>>>> >> >> /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/
>>>>> >> >>
>>>>> >> >> build/test/tmp/test7610638215300246179.junit.dir/version-2/log.1000000
>>>>> >> >> 01
>>>>> >> >> (Too many open files)
>>>>> >> >>
>>>>> >> >> which we've not seen before. I believe this means that we running
>>>>> >> >> out
>>>>> >> >> of file descriptors?
>>>>> >> >>
>>>>> >> >> Can you verify and address if possible?
>>>>> >> >>
>>>>> >> >> Thanks,
>>>>> >> >>
>>>>> >> >> Patrick
>>>>> >> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > CONFIDENTIALITY NOTICE
>>>>> > NOTICE: This message is intended for the use of the individual or entity
>>>>> > to
>>>>> > which it is addressed and may contain information that is confidential,
>>>>> > privileged and exempt from disclosure under applicable law. If the
>>>>> > reader of
>>>>> > this message is not the intended recipient, you are hereby notified that
>>>>> > any
>>>>> > printing, copying, dissemination, distribution, disclosure or forwarding
>>>>> > of
>>>>> > this communication is strictly prohibited. If you have received this
>>>>> > communication in error, please contact the sender immediately and delete
>>>>> > it
>>>>> > from your system. Thank You.
>>>>
>>>>
>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or entity to
>>>> which it is addressed and may contain information that is confidential,
>>>> privileged and exempt from disclosure under applicable law. If the reader of
>>>> this message is not the intended recipient, you are hereby notified that any
>>>> printing, copying, dissemination, distribution, disclosure or forwarding of
>>>> this communication is strictly prohibited. If you have received this
>>>> communication in error, please contact the sender immediately and delete it
>>>> from your system. Thank You.

Re: ulimit changed with Apache Jenkins upgrade?

Posted by Bong Chen Lock <bo...@gmail.com>.
On 4 Sep 2014 00:21, "Patrick Hunt" <ph...@apache.org> wrote:

> I've opened a BUILDS jira for this:
> https://issues.apache.org/jira/browse/BUILDS-17
>
> On Sat, Aug 30, 2014 at 12:16 PM, Patrick Hunt <ph...@apache.org> wrote:
> > Giri could you or one of the other folks look into this? Our tests
> > have been broken for some time.
> >
> >
> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2303/
> >
> > The same test is still failing with the same error (too many open
> > files) and afaict the ulimit is still just set to 4k. I'm printing the
> > ulimit info at the start of the job, here it is:
> >
> > core file size          (blocks, -c) 0
> > data seg size           (kbytes, -d) unlimited
> > scheduling priority             (-e) 0
> > file size               (blocks, -f) unlimited
> > pending signals                 (-i) 386177
> > max locked memory       (kbytes, -l) 64
> > max memory size         (kbytes, -m) unlimited
> > open files                      (-n) 4096
> > pipe size            (512 bytes, -p) 8
> > POSIX message queues     (bytes, -q) 819200
> > real-time priority              (-r) 0
> > stack size              (kbytes, -s) 8192
> > cpu time               (seconds, -t) unlimited
> > max user processes              (-u) 386177
> > virtual memory          (kbytes, -v) unlimited
> > file locks                      (-x) unlimited
> >
> >
> > Patrick
> >
> > On Wed, Jul 23, 2014 at 11:21 AM, Patrick Hunt <ph...@apache.org> wrote:
> >> Does someone in builds@ have the ability to address this? (restart the
> >> jenkins slaves so that the new ulimit limits will take effect)
> >>
> >> Thanks!
> >>
> >> Patrick
> >>
> >> On Wed, Jul 23, 2014 at 9:33 AM, Patrick Hunt <ph...@apache.org> wrote:
> >>> Giri do you mean me? I don't have access to that afaict.
> >>>
> >>> Patrick
> >>>
> >>> On Wed, Jul 23, 2014 at 12:50 AM, Giridharan Kesavan
> >>> <gk...@hortonworks.com> wrote:
> >>>> jenkins slaves might need a restart.
> >>>>
> >>>> Could you pls re-launch the slaves from the jenkins UI configuration
> page?
> >>>>
> >>>> -giri
> >>>>
> >>>>
> >>>> On Tue, Jul 22, 2014 at 2:25 PM, Patrick Hunt <ph...@apache.org>
> wrote:
> >>>>>
> >>>>> Thanks Giri! Unfortunately though it seems to not have taken effect,
> I
> >>>>> just kicked off a precommit build and I see
> >>>>>
> >>>>> core file size          (blocks, -c) 0
> >>>>> data seg size           (kbytes, -d) unlimited
> >>>>> scheduling priority             (-e) 0
> >>>>> file size               (blocks, -f) unlimited
> >>>>> pending signals                 (-i) 386178
> >>>>> max locked memory       (kbytes, -l) 64
> >>>>> max memory size         (kbytes, -m) unlimited
> >>>>> open files                      (-n) 4096
> >>>>> pipe size            (512 bytes, -p) 8
> >>>>> POSIX message queues     (bytes, -q) 819200
> >>>>> real-time priority              (-r) 0
> >>>>> stack size              (kbytes, -s) 8192
> >>>>> cpu time               (seconds, -t) unlimited
> >>>>> max user processes              (-u) 386178
> >>>>> virtual memory          (kbytes, -v) unlimited
> >>>>> file locks                      (-x) unlimited
> >>>>>
> >>>>> more details here:
> >>>>>
> >>>>>
> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2215/console
> >>>>>
> >>>>> Patrick
> >>>>>
> >>>>> On Tue, Jul 22, 2014 at 1:29 PM, Giridharan Kesavan
> >>>>> <gk...@hortonworks.com> wrote:
> >>>>> >
> >>>>> >
> >>>>> > jenkins@asf901:~$ ulimit -a
> >>>>> > core file size          (blocks, -c) 0
> >>>>> > data seg size           (kbytes, -d) unlimited
> >>>>> > scheduling priority             (-e) 0
> >>>>> > file size               (blocks, -f) unlimited
> >>>>> > pending signals                 (-i) 386178
> >>>>> > max locked memory       (kbytes, -l) 64
> >>>>> > max memory size         (kbytes, -m) unlimited
> >>>>> > open files                      (-n) 60000
> >>>>> > pipe size            (512 bytes, -p) 8
> >>>>> > POSIX message queues     (bytes, -q) 819200
> >>>>> > real-time priority              (-r) 0
> >>>>> > stack size              (kbytes, -s) 8192
> >>>>> > cpu time               (seconds, -t) unlimited
> >>>>> > max user processes              (-u) 10240
> >>>>> > virtual memory          (kbytes, -v) unlimited
> >>>>> > file locks                      (-x) unlimited
> >>>>> >
> >>>>> > bumped up the open files and max user processes on all the slaves.
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > -giri
> >>>>> >
> >>>>> >
> >>>>> > On Tue, Jul 22, 2014 at 12:04 PM, Patrick Hunt <ph...@apache.org>
> wrote:
> >>>>> >>
> >>>>> >> Giri any chance can you take a look at the ulimit issue on the H#
> >>>>> >> machines? All the ZK precommit builds are failing as a result.
> >>>>> >>
> >>>>> >> I updated the precommit build last night to output "ulimit -a"
> and it
> >>>>> >> says the current limit is 4096, can we bump that up or set the
> default
> >>>>> >> to unlimited?
> >>>>> >>
> >>>>> >> Thanks!
> >>>>> >>
> >>>>> >> Patrick
> >>>>> >>
> >>>>> >> On Sun, Jul 20, 2014 at 10:32 PM, Rakesh R <ra...@huawei.com>
> wrote:
> >>>>> >> > +1
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > Adding one more point. I could see the following error too in
> the
> >>>>> >> > pre-commit build.
> >>>>> >> >
> >>>>> >> >      [exec]     [junit] Exception in thread
> "CommitProcWorkThread-16"
> >>>>> >> > java.lang.NoClassDefFoundError:
> >>>>> >> > org/apache/zookeeper/server/ConnectionBean
> >>>>> >> >      [exec]     [junit]         at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.ServerCnxnFactory.registerConnection(ServerCnxnFactory.java:159)
> >>>>> >> >      [exec]     [junit]         at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.ZooKeeperServer.finishSessionInit(ZooKeeperServer.java:594)
> >>>>> >> >      [exec]     [junit]         at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:198)
> >>>>> >> >      [exec]     [junit]         at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:295)
> >>>>> >> >      [exec]     [junit]         at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:161)
> >>>>> >> >      [exec]     [junit]         at
> >>>>> >> >
> >>>>> >> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> >>>>> >> >      [exec]     [junit]         at
> >>>>> >> >
> >>>>> >> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> >>>>> >> >      [exec]     [junit]         at
> >>>>> >> > java.lang.Thread.run(Thread.java:662)
> >>>>> >> >
> >>>>> >> > -Rakesh
> >>>>> >> >
> >>>>> >> > -----Original Message-----
> >>>>> >> > From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]
> >>>>> >> > Sent: 21 July 2014 02:19
> >>>>> >> > To: dev@zookeeper.apache.org
> >>>>> >> > Cc: Andrew Bayer; builds@apache.org; Giridharan Kesavan
> >>>>> >> > Subject: Re: ulimit changed with Apache Jenkins upgrade?
> >>>>> >> >
> >>>>> >> > +1
> >>>>> >> >
> >>>>> >> > On 18 Jul 2014, at 18:59, Patrick Hunt <ph...@apache.org>
> wrote:
> >>>>> >> >
> >>>>> >> >> Hi builds folks, is this a system wide issue or something we
> should
> >>>>> >> >> address ourselves? Thanks!
> >>>>> >> >>
> >>>>> >> >> Patrick
> >>>>> >> >>
> >>>>> >> >> ---------- Forwarded message ----------
> >>>>> >> >> From: Patrick Hunt <ph...@apache.org>
> >>>>> >> >> Date: Fri, Jul 18, 2014 at 10:38 AM
> >>>>> >> >> Subject: ulimit changed with Apache Jenkins upgrade?
> >>>>> >> >> To: Giridharan Kesavan <gk...@hortonworks.com>
> >>>>> >> >> Cc: DevZooKeeper <de...@zookeeper.apache.org>, Andrew Bayer
> >>>>> >> >> <an...@cloudera.com>
> >>>>> >> >>
> >>>>> >> >>
> >>>>> >> >> Hi Giri, can you check that the new hosts (H#) have the ulimit
> set
> >>>>> >> >> to
> >>>>> >> >> what it was set to on the original hadoop# hosts? I'm seeing
> new
> >>>>> >> >> test
> >>>>> >> >> failures with
> >>>>> >> >>
> >>>>> >> >>     [exec]     [junit] java.io.FileNotFoundException:
> >>>>> >> >>
> >>>>> >> >>
> /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/
> >>>>> >> >>
> >>>>> >> >>
> build/test/tmp/test7610638215300246179.junit.dir/version-2/log.1000000
> >>>>> >> >> 01
> >>>>> >> >> (Too many open files)
> >>>>> >> >>
> >>>>> >> >> which we've not seen before. I believe this means that we
> running
> >>>>> >> >> out
> >>>>> >> >> of file descriptors?
> >>>>> >> >>
> >>>>> >> >> Can you verify and address if possible?
> >>>>> >> >>
> >>>>> >> >> Thanks,
> >>>>> >> >>
> >>>>> >> >> Patrick
> >>>>> >> >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > CONFIDENTIALITY NOTICE
> >>>>> > NOTICE: This message is intended for the use of the individual or
> entity
> >>>>> > to
> >>>>> > which it is addressed and may contain information that is
> confidential,
> >>>>> > privileged and exempt from disclosure under applicable law. If the
> >>>>> > reader of
> >>>>> > this message is not the intended recipient, you are hereby
> notified that
> >>>>> > any
> >>>>> > printing, copying, dissemination, distribution, disclosure or
> forwarding
> >>>>> > of
> >>>>> > this communication is strictly prohibited. If you have received
> this
> >>>>> > communication in error, please contact the sender immediately and
> delete
> >>>>> > it
> >>>>> > from your system. Thank You.
> >>>>
> >>>>
> >>>>
> >>>> CONFIDENTIALITY NOTICE
> >>>> NOTICE: This message is intended for the use of the individual or
> entity to
> >>>> which it is addressed and may contain information that is
> confidential,
> >>>> privileged and exempt from disclosure under applicable law. If the
> reader of
> >>>> this message is not the intended recipient, you are hereby notified
> that any
> >>>> printing, copying, dissemination, distribution, disclosure or
> forwarding of
> >>>> this communication is strictly prohibited. If you have received this
> >>>> communication in error, please contact the sender immediately and
> delete it
> >>>> from your system. Thank You.
>

Re: ulimit changed with Apache Jenkins upgrade?

Posted by Bong Chen Lock <bo...@gmail.com>.
On 4 Sep 2014 00:24, "Bong Chen Lock" <bo...@gmail.com> wrote:

> On 4 Sep 2014 00:21, "Patrick Hunt" <ph...@apache.org> wrote:
>
>> I've opened a BUILDS jira for this:
>> https://issues.apache.org/jira/browse/BUILDS-17
>>
>> On Sat, Aug 30, 2014 at 12:16 PM, Patrick Hunt <ph...@apache.org> wrote:
>> > Giri could you or one of the other folks look into this? Our tests
>> > have been broken for some time.
>> >
>> >
>> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2303/
>> >
>> > The same test is still failing with the same error (too many open
>> > files) and afaict the ulimit is still just set to 4k. I'm printing the
>> > ulimit info at the start of the job, here it is:
>> >
>> > core file size          (blocks, -c) 0
>> > data seg size           (kbytes, -d) unlimited
>> > scheduling priority             (-e) 0
>> > file size               (blocks, -f) unlimited
>> > pending signals                 (-i) 386177
>> > max locked memory       (kbytes, -l) 64
>> > max memory size         (kbytes, -m) unlimited
>> > open files                      (-n) 4096
>> > pipe size            (512 bytes, -p) 8
>> > POSIX message queues     (bytes, -q) 819200
>> > real-time priority              (-r) 0
>> > stack size              (kbytes, -s) 8192
>> > cpu time               (seconds, -t) unlimited
>> > max user processes              (-u) 386177
>> > virtual memory          (kbytes, -v) unlimited
>> > file locks                      (-x) unlimited
>> >
>> >
>> > Patrick
>> >
>> > On Wed, Jul 23, 2014 at 11:21 AM, Patrick Hunt <ph...@apache.org>
>> wrote:
>> >> Does someone in builds@ have the ability to address this? (restart the
>> >> jenkins slaves so that the new ulimit limits will take effect)
>> >>
>> >> Thanks!
>> >>
>> >> Patrick
>> >>
>> >> On Wed, Jul 23, 2014 at 9:33 AM, Patrick Hunt <ph...@apache.org>
>> wrote:
>> >>> Giri do you mean me? I don't have access to that afaict.
>> >>>
>> >>> Patrick
>> >>>
>> >>> On Wed, Jul 23, 2014 at 12:50 AM, Giridharan Kesavan
>> >>> <gk...@hortonworks.com> wrote:
>> >>>> jenkins slaves might need a restart.
>> >>>>
>> >>>> Could you pls re-launch the slaves from the jenkins UI configuration
>> page?
>> >>>>
>> >>>> -giri
>> >>>>
>> >>>>
>> >>>> On Tue, Jul 22, 2014 at 2:25 PM, Patrick Hunt <ph...@apache.org>
>> wrote:
>> >>>>>
>> >>>>> Thanks Giri! Unfortunately though it seems to not have taken
>> effect, I
>> >>>>> just kicked off a precommit build and I see
>> >>>>>
>> >>>>> core file size          (blocks, -c) 0
>> >>>>> data seg size           (kbytes, -d) unlimited
>> >>>>> scheduling priority             (-e) 0
>> >>>>> file size               (blocks, -f) unlimited
>> >>>>> pending signals                 (-i) 386178
>> >>>>> max locked memory       (kbytes, -l) 64
>> >>>>> max memory size         (kbytes, -m) unlimited
>> >>>>> open files                      (-n) 4096
>> >>>>> pipe size            (512 bytes, -p) 8
>> >>>>> POSIX message queues     (bytes, -q) 819200
>> >>>>> real-time priority              (-r) 0
>> >>>>> stack size              (kbytes, -s) 8192
>> >>>>> cpu time               (seconds, -t) unlimited
>> >>>>> max user processes              (-u) 386178
>> >>>>> virtual memory          (kbytes, -v) unlimited
>> >>>>> file locks                      (-x) unlimited
>> >>>>>
>> >>>>> more details here:
>> >>>>>
>> >>>>>
>> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2215/console
>> >>>>>
>> >>>>> Patrick
>> >>>>>
>> >>>>> On Tue, Jul 22, 2014 at 1:29 PM, Giridharan Kesavan
>> >>>>> <gk...@hortonworks.com> wrote:
>> >>>>> >
>> >>>>> >
>> >>>>> > jenkins@asf901:~$ ulimit -a
>> >>>>> > core file size          (blocks, -c) 0
>> >>>>> > data seg size           (kbytes, -d) unlimited
>> >>>>> > scheduling priority             (-e) 0
>> >>>>> > file size               (blocks, -f) unlimited
>> >>>>> > pending signals                 (-i) 386178
>> >>>>> > max locked memory       (kbytes, -l) 64
>> >>>>> > max memory size         (kbytes, -m) unlimited
>> >>>>> > open files                      (-n) 60000
>> >>>>> > pipe size            (512 bytes, -p) 8
>> >>>>> > POSIX message queues     (bytes, -q) 819200
>> >>>>> > real-time priority              (-r) 0
>> >>>>> > stack size              (kbytes, -s) 8192
>> >>>>> > cpu time               (seconds, -t) unlimited
>> >>>>> > max user processes              (-u) 10240
>> >>>>> > virtual memory          (kbytes, -v) unlimited
>> >>>>> > file locks                      (-x) unlimited
>> >>>>> >
>> >>>>> > bumped up the open files and max user processes on all the slaves.
>> >>>>> >
>> >>>>> >
>> >>>>> >
>> >>>>> > -giri
>> >>>>> >
>> >>>>> >
>> >>>>> > On Tue, Jul 22, 2014 at 12:04 PM, Patrick Hunt <ph...@apache.org>
>> wrote:
>> >>>>> >>
>> >>>>> >> Giri any chance can you take a look at the ulimit issue on the H#
>> >>>>> >> machines? All the ZK precommit builds are failing as a result.
>> >>>>> >>
>> >>>>> >> I updated the precommit build last night to output "ulimit -a"
>> and it
>> >>>>> >> says the current limit is 4096, can we bump that up or set the
>> default
>> >>>>> >> to unlimited?
>> >>>>> >>
>> >>>>> >> Thanks!
>> >>>>> >>
>> >>>>> >> Patrick
>> >>>>> >>
>> >>>>> >> On Sun, Jul 20, 2014 at 10:32 PM, Rakesh R <ra...@huawei.com>
>> wrote:
>> >>>>> >> > +1
>> >>>>> >> >
>> >>>>> >> >
>> >>>>> >> > Adding one more point. I could see the following error too in
>> the
>> >>>>> >> > pre-commit build.
>> >>>>> >> >
>> >>>>> >> >      [exec]     [junit] Exception in thread
>> "CommitProcWorkThread-16"
>> >>>>> >> > java.lang.NoClassDefFoundError:
>> >>>>> >> > org/apache/zookeeper/server/ConnectionBean
>> >>>>> >> >      [exec]     [junit]         at
>> >>>>> >> >
>> >>>>> >> >
>> org.apache.zookeeper.server.ServerCnxnFactory.registerConnection(ServerCnxnFactory.java:159)
>> >>>>> >> >      [exec]     [junit]         at
>> >>>>> >> >
>> >>>>> >> >
>> org.apache.zookeeper.server.ZooKeeperServer.finishSessionInit(ZooKeeperServer.java:594)
>> >>>>> >> >      [exec]     [junit]         at
>> >>>>> >> >
>> >>>>> >> >
>> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:198)
>> >>>>> >> >      [exec]     [junit]         at
>> >>>>> >> >
>> >>>>> >> >
>> org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:295)
>> >>>>> >> >      [exec]     [junit]         at
>> >>>>> >> >
>> >>>>> >> >
>> org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:161)
>> >>>>> >> >      [exec]     [junit]         at
>> >>>>> >> >
>> >>>>> >> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>> >>>>> >> >      [exec]     [junit]         at
>> >>>>> >> >
>> >>>>> >> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>> >>>>> >> >      [exec]     [junit]         at
>> >>>>> >> > java.lang.Thread.run(Thread.java:662)
>> >>>>> >> >
>> >>>>> >> > -Rakesh
>> >>>>> >> >
>> >>>>> >> > -----Original Message-----
>> >>>>> >> > From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]
>> >>>>> >> > Sent: 21 July 2014 02:19
>> >>>>> >> > To: dev@zookeeper.apache.org
>> >>>>> >> > Cc: Andrew Bayer; builds@apache.org; Giridharan Kesavan
>> >>>>> >> > Subject: Re: ulimit changed with Apache Jenkins upgrade?
>> >>>>> >> >
>> >>>>> >> > +1
>> >>>>> >> >
>> >>>>> >> > On 18 Jul 2014, at 18:59, Patrick Hunt <ph...@apache.org>
>> wrote:
>> >>>>> >> >
>> >>>>> >> >> Hi builds folks, is this a system wide issue or something we
>> should
>> >>>>> >> >> address ourselves? Thanks!
>> >>>>> >> >>
>> >>>>> >> >> Patrick
>> >>>>> >> >>
>> >>>>> >> >> ---------- Forwarded message ----------
>> >>>>> >> >> From: Patrick Hunt <ph...@apache.org>
>> >>>>> >> >> Date: Fri, Jul 18, 2014 at 10:38 AM
>> >>>>> >> >> Subject: ulimit changed with Apache Jenkins upgrade?
>> >>>>> >> >> To: Giridharan Kesavan <gk...@hortonworks.com>
>> >>>>> >> >> Cc: DevZooKeeper <de...@zookeeper.apache.org>, Andrew Bayer
>> >>>>> >> >> <an...@cloudera.com>
>> >>>>> >> >>
>> >>>>> >> >>
>> >>>>> >> >> Hi Giri, can you check that the new hosts (H#) have the
>> ulimit set
>> >>>>> >> >> to
>> >>>>> >> >> what it was set to on the original hadoop# hosts? I'm seeing
>> new
>> >>>>> >> >> test
>> >>>>> >> >> failures with
>> >>>>> >> >>
>> >>>>> >> >>     [exec]     [junit] java.io.FileNotFoundException:
>> >>>>> >> >>
>> >>>>> >> >>
>> /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/
>> >>>>> >> >>
>> >>>>> >> >>
>> build/test/tmp/test7610638215300246179.junit.dir/version-2/log.1000000
>> >>>>> >> >> 01
>> >>>>> >> >> (Too many open files)
>> >>>>> >> >>
>> >>>>> >> >> which we've not seen before. I believe this means that we
>> running
>> >>>>> >> >> out
>> >>>>> >> >> of file descriptors?
>> >>>>> >> >>
>> >>>>> >> >> Can you verify and address if possible?
>> >>>>> >> >>
>> >>>>> >> >> Thanks,
>> >>>>> >> >>
>> >>>>> >> >> Patrick
>> >>>>> >> >
>> >>>>> >
>> >>>>> >
>> >>>>> >
>> >>>>> > CONFIDENTIALITY NOTICE
>> >>>>> > NOTICE: This message is intended for the use of the individual or
>> entity
>> >>>>> > to
>> >>>>> > which it is addressed and may contain information that is
>> confidential,
>> >>>>> > privileged and exempt from disclosure under applicable law. If the
>> >>>>> > reader of
>> >>>>> > this message is not the intended recipient, you are hereby
>> notified that
>> >>>>> > any
>> >>>>> > printing, copying, dissemination, distribution, disclosure or
>> forwarding
>> >>>>> > of
>> >>>>> > this communication is strictly prohibited. If you have received
>> this
>> >>>>> > communication in error, please contact the sender immediately and
>> delete
>> >>>>> > it
>> >>>>> > from your system. Thank You.
>> >>>>
>> >>>>
>> >>>>
>> >>>> CONFIDENTIALITY NOTICE
>> >>>> NOTICE: This message is intended for the use of the individual or
>> entity to
>> >>>> which it is addressed and may contain information that is
>> confidential,
>> >>>> privileged and exempt from disclosure under applicable law. If the
>> reader of
>> >>>> this message is not the intended recipient, you are hereby notified
>> that any
>> >>>> printing, copying, dissemination, distribution, disclosure or
>> forwarding of
>> >>>> this communication is strictly prohibited. If you have received this
>> >>>> communication in error, please contact the sender immediately and
>> delete it
>> >>>> from your system. Thank You.
>>
>

Re: ulimit changed with Apache Jenkins upgrade?

Posted by Bong Chen Lock <bo...@gmail.com>.
On 4 Sep 2014 00:21, "Patrick Hunt" <ph...@apache.org> wrote:

> I've opened a BUILDS jira for this:
> https://issues.apache.org/jira/browse/BUILDS-17
>
> On Sat, Aug 30, 2014 at 12:16 PM, Patrick Hunt <ph...@apache.org> wrote:
> > Giri could you or one of the other folks look into this? Our tests
> > have been broken for some time.
> >
> >
> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2303/
> >
> > The same test is still failing with the same error (too many open
> > files) and afaict the ulimit is still just set to 4k. I'm printing the
> > ulimit info at the start of the job, here it is:
> >
> > core file size          (blocks, -c) 0
> > data seg size           (kbytes, -d) unlimited
> > scheduling priority             (-e) 0
> > file size               (blocks, -f) unlimited
> > pending signals                 (-i) 386177
> > max locked memory       (kbytes, -l) 64
> > max memory size         (kbytes, -m) unlimited
> > open files                      (-n) 4096
> > pipe size            (512 bytes, -p) 8
> > POSIX message queues     (bytes, -q) 819200
> > real-time priority              (-r) 0
> > stack size              (kbytes, -s) 8192
> > cpu time               (seconds, -t) unlimited
> > max user processes              (-u) 386177
> > virtual memory          (kbytes, -v) unlimited
> > file locks                      (-x) unlimited
> >
> >
> > Patrick
> >
> > On Wed, Jul 23, 2014 at 11:21 AM, Patrick Hunt <ph...@apache.org> wrote:
> >> Does someone in builds@ have the ability to address this? (restart the
> >> jenkins slaves so that the new ulimit limits will take effect)
> >>
> >> Thanks!
> >>
> >> Patrick
> >>
> >> On Wed, Jul 23, 2014 at 9:33 AM, Patrick Hunt <ph...@apache.org> wrote:
> >>> Giri do you mean me? I don't have access to that afaict.
> >>>
> >>> Patrick
> >>>
> >>> On Wed, Jul 23, 2014 at 12:50 AM, Giridharan Kesavan
> >>> <gk...@hortonworks.com> wrote:
> >>>> jenkins slaves might need a restart.
> >>>>
> >>>> Could you pls re-launch the slaves from the jenkins UI configuration
> page?
> >>>>
> >>>> -giri
> >>>>
> >>>>
> >>>> On Tue, Jul 22, 2014 at 2:25 PM, Patrick Hunt <ph...@apache.org>
> wrote:
> >>>>>
> >>>>> Thanks Giri! Unfortunately though it seems to not have taken effect,
> I
> >>>>> just kicked off a precommit build and I see
> >>>>>
> >>>>> core file size          (blocks, -c) 0
> >>>>> data seg size           (kbytes, -d) unlimited
> >>>>> scheduling priority             (-e) 0
> >>>>> file size               (blocks, -f) unlimited
> >>>>> pending signals                 (-i) 386178
> >>>>> max locked memory       (kbytes, -l) 64
> >>>>> max memory size         (kbytes, -m) unlimited
> >>>>> open files                      (-n) 4096
> >>>>> pipe size            (512 bytes, -p) 8
> >>>>> POSIX message queues     (bytes, -q) 819200
> >>>>> real-time priority              (-r) 0
> >>>>> stack size              (kbytes, -s) 8192
> >>>>> cpu time               (seconds, -t) unlimited
> >>>>> max user processes              (-u) 386178
> >>>>> virtual memory          (kbytes, -v) unlimited
> >>>>> file locks                      (-x) unlimited
> >>>>>
> >>>>> more details here:
> >>>>>
> >>>>>
> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2215/console
> >>>>>
> >>>>> Patrick
> >>>>>
> >>>>> On Tue, Jul 22, 2014 at 1:29 PM, Giridharan Kesavan
> >>>>> <gk...@hortonworks.com> wrote:
> >>>>> >
> >>>>> >
> >>>>> > jenkins@asf901:~$ ulimit -a
> >>>>> > core file size          (blocks, -c) 0
> >>>>> > data seg size           (kbytes, -d) unlimited
> >>>>> > scheduling priority             (-e) 0
> >>>>> > file size               (blocks, -f) unlimited
> >>>>> > pending signals                 (-i) 386178
> >>>>> > max locked memory       (kbytes, -l) 64
> >>>>> > max memory size         (kbytes, -m) unlimited
> >>>>> > open files                      (-n) 60000
> >>>>> > pipe size            (512 bytes, -p) 8
> >>>>> > POSIX message queues     (bytes, -q) 819200
> >>>>> > real-time priority              (-r) 0
> >>>>> > stack size              (kbytes, -s) 8192
> >>>>> > cpu time               (seconds, -t) unlimited
> >>>>> > max user processes              (-u) 10240
> >>>>> > virtual memory          (kbytes, -v) unlimited
> >>>>> > file locks                      (-x) unlimited
> >>>>> >
> >>>>> > bumped up the open files and max user processes on all the slaves.
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > -giri
> >>>>> >
> >>>>> >
> >>>>> > On Tue, Jul 22, 2014 at 12:04 PM, Patrick Hunt <ph...@apache.org>
> wrote:
> >>>>> >>
> >>>>> >> Giri any chance can you take a look at the ulimit issue on the H#
> >>>>> >> machines? All the ZK precommit builds are failing as a result.
> >>>>> >>
> >>>>> >> I updated the precommit build last night to output "ulimit -a"
> and it
> >>>>> >> says the current limit is 4096, can we bump that up or set the
> default
> >>>>> >> to unlimited?
> >>>>> >>
> >>>>> >> Thanks!
> >>>>> >>
> >>>>> >> Patrick
> >>>>> >>
> >>>>> >> On Sun, Jul 20, 2014 at 10:32 PM, Rakesh R <ra...@huawei.com>
> wrote:
> >>>>> >> > +1
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > Adding one more point. I could see the following error too in
> the
> >>>>> >> > pre-commit build.
> >>>>> >> >
> >>>>> >> >      [exec]     [junit] Exception in thread
> "CommitProcWorkThread-16"
> >>>>> >> > java.lang.NoClassDefFoundError:
> >>>>> >> > org/apache/zookeeper/server/ConnectionBean
> >>>>> >> >      [exec]     [junit]         at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.ServerCnxnFactory.registerConnection(ServerCnxnFactory.java:159)
> >>>>> >> >      [exec]     [junit]         at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.ZooKeeperServer.finishSessionInit(ZooKeeperServer.java:594)
> >>>>> >> >      [exec]     [junit]         at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:198)
> >>>>> >> >      [exec]     [junit]         at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:295)
> >>>>> >> >      [exec]     [junit]         at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:161)
> >>>>> >> >      [exec]     [junit]         at
> >>>>> >> >
> >>>>> >> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> >>>>> >> >      [exec]     [junit]         at
> >>>>> >> >
> >>>>> >> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> >>>>> >> >      [exec]     [junit]         at
> >>>>> >> > java.lang.Thread.run(Thread.java:662)
> >>>>> >> >
> >>>>> >> > -Rakesh
> >>>>> >> >
> >>>>> >> > -----Original Message-----
> >>>>> >> > From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]
> >>>>> >> > Sent: 21 July 2014 02:19
> >>>>> >> > To: dev@zookeeper.apache.org
> >>>>> >> > Cc: Andrew Bayer; builds@apache.org; Giridharan Kesavan
> >>>>> >> > Subject: Re: ulimit changed with Apache Jenkins upgrade?
> >>>>> >> >
> >>>>> >> > +1
> >>>>> >> >
> >>>>> >> > On 18 Jul 2014, at 18:59, Patrick Hunt <ph...@apache.org>
> wrote:
> >>>>> >> >
> >>>>> >> >> Hi builds folks, is this a system wide issue or something we
> should
> >>>>> >> >> address ourselves? Thanks!
> >>>>> >> >>
> >>>>> >> >> Patrick
> >>>>> >> >>
> >>>>> >> >> ---------- Forwarded message ----------
> >>>>> >> >> From: Patrick Hunt <ph...@apache.org>
> >>>>> >> >> Date: Fri, Jul 18, 2014 at 10:38 AM
> >>>>> >> >> Subject: ulimit changed with Apache Jenkins upgrade?
> >>>>> >> >> To: Giridharan Kesavan <gk...@hortonworks.com>
> >>>>> >> >> Cc: DevZooKeeper <de...@zookeeper.apache.org>, Andrew Bayer
> >>>>> >> >> <an...@cloudera.com>
> >>>>> >> >>
> >>>>> >> >>
> >>>>> >> >> Hi Giri, can you check that the new hosts (H#) have the ulimit
> set
> >>>>> >> >> to
> >>>>> >> >> what it was set to on the original hadoop# hosts? I'm seeing
> new
> >>>>> >> >> test
> >>>>> >> >> failures with
> >>>>> >> >>
> >>>>> >> >>     [exec]     [junit] java.io.FileNotFoundException:
> >>>>> >> >>
> >>>>> >> >>
> /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/
> >>>>> >> >>
> >>>>> >> >>
> build/test/tmp/test7610638215300246179.junit.dir/version-2/log.1000000
> >>>>> >> >> 01
> >>>>> >> >> (Too many open files)
> >>>>> >> >>
> >>>>> >> >> which we've not seen before. I believe this means that we
> running
> >>>>> >> >> out
> >>>>> >> >> of file descriptors?
> >>>>> >> >>
> >>>>> >> >> Can you verify and address if possible?
> >>>>> >> >>
> >>>>> >> >> Thanks,
> >>>>> >> >>
> >>>>> >> >> Patrick
> >>>>> >> >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > CONFIDENTIALITY NOTICE
> >>>>> > NOTICE: This message is intended for the use of the individual or
> entity
> >>>>> > to
> >>>>> > which it is addressed and may contain information that is
> confidential,
> >>>>> > privileged and exempt from disclosure under applicable law. If the
> >>>>> > reader of
> >>>>> > this message is not the intended recipient, you are hereby
> notified that
> >>>>> > any
> >>>>> > printing, copying, dissemination, distribution, disclosure or
> forwarding
> >>>>> > of
> >>>>> > this communication is strictly prohibited. If you have received
> this
> >>>>> > communication in error, please contact the sender immediately and
> delete
> >>>>> > it
> >>>>> > from your system. Thank You.
> >>>>
> >>>>
> >>>>
> >>>> CONFIDENTIALITY NOTICE
> >>>> NOTICE: This message is intended for the use of the individual or
> entity to
> >>>> which it is addressed and may contain information that is
> confidential,
> >>>> privileged and exempt from disclosure under applicable law. If the
> reader of
> >>>> this message is not the intended recipient, you are hereby notified
> that any
> >>>> printing, copying, dissemination, distribution, disclosure or
> forwarding of
> >>>> this communication is strictly prohibited. If you have received this
> >>>> communication in error, please contact the sender immediately and
> delete it
> >>>> from your system. Thank You.
>