You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by Patrick Hunt <ph...@apache.org> on 2014/08/30 21:16:05 UTC
Re: ulimit changed with Apache Jenkins upgrade?
Giri could you or one of the other folks look into this? Our tests
have been broken for some time.
https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2303/
The same test is still failing with the same error (too many open
files) and afaict the ulimit is still just set to 4k. I'm printing the
ulimit info at the start of the job, here it is:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 386177
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 4096
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 386177
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Patrick
On Wed, Jul 23, 2014 at 11:21 AM, Patrick Hunt <ph...@apache.org> wrote:
> Does someone in builds@ have the ability to address this? (restart the
> jenkins slaves so that the new ulimit limits will take effect)
>
> Thanks!
>
> Patrick
>
> On Wed, Jul 23, 2014 at 9:33 AM, Patrick Hunt <ph...@apache.org> wrote:
>> Giri do you mean me? I don't have access to that afaict.
>>
>> Patrick
>>
>> On Wed, Jul 23, 2014 at 12:50 AM, Giridharan Kesavan
>> <gk...@hortonworks.com> wrote:
>>> jenkins slaves might need a restart.
>>>
>>> Could you pls re-launch the slaves from the jenkins UI configuration page?
>>>
>>> -giri
>>>
>>>
>>> On Tue, Jul 22, 2014 at 2:25 PM, Patrick Hunt <ph...@apache.org> wrote:
>>>>
>>>> Thanks Giri! Unfortunately though it seems to not have taken effect, I
>>>> just kicked off a precommit build and I see
>>>>
>>>> core file size (blocks, -c) 0
>>>> data seg size (kbytes, -d) unlimited
>>>> scheduling priority (-e) 0
>>>> file size (blocks, -f) unlimited
>>>> pending signals (-i) 386178
>>>> max locked memory (kbytes, -l) 64
>>>> max memory size (kbytes, -m) unlimited
>>>> open files (-n) 4096
>>>> pipe size (512 bytes, -p) 8
>>>> POSIX message queues (bytes, -q) 819200
>>>> real-time priority (-r) 0
>>>> stack size (kbytes, -s) 8192
>>>> cpu time (seconds, -t) unlimited
>>>> max user processes (-u) 386178
>>>> virtual memory (kbytes, -v) unlimited
>>>> file locks (-x) unlimited
>>>>
>>>> more details here:
>>>>
>>>> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2215/console
>>>>
>>>> Patrick
>>>>
>>>> On Tue, Jul 22, 2014 at 1:29 PM, Giridharan Kesavan
>>>> <gk...@hortonworks.com> wrote:
>>>> >
>>>> >
>>>> > jenkins@asf901:~$ ulimit -a
>>>> > core file size (blocks, -c) 0
>>>> > data seg size (kbytes, -d) unlimited
>>>> > scheduling priority (-e) 0
>>>> > file size (blocks, -f) unlimited
>>>> > pending signals (-i) 386178
>>>> > max locked memory (kbytes, -l) 64
>>>> > max memory size (kbytes, -m) unlimited
>>>> > open files (-n) 60000
>>>> > pipe size (512 bytes, -p) 8
>>>> > POSIX message queues (bytes, -q) 819200
>>>> > real-time priority (-r) 0
>>>> > stack size (kbytes, -s) 8192
>>>> > cpu time (seconds, -t) unlimited
>>>> > max user processes (-u) 10240
>>>> > virtual memory (kbytes, -v) unlimited
>>>> > file locks (-x) unlimited
>>>> >
>>>> > bumped up the open files and max user processes on all the slaves.
>>>> >
>>>> >
>>>> >
>>>> > -giri
>>>> >
>>>> >
>>>> > On Tue, Jul 22, 2014 at 12:04 PM, Patrick Hunt <ph...@apache.org> wrote:
>>>> >>
>>>> >> Giri any chance can you take a look at the ulimit issue on the H#
>>>> >> machines? All the ZK precommit builds are failing as a result.
>>>> >>
>>>> >> I updated the precommit build last night to output "ulimit -a" and it
>>>> >> says the current limit is 4096, can we bump that up or set the default
>>>> >> to unlimited?
>>>> >>
>>>> >> Thanks!
>>>> >>
>>>> >> Patrick
>>>> >>
>>>> >> On Sun, Jul 20, 2014 at 10:32 PM, Rakesh R <ra...@huawei.com> wrote:
>>>> >> > +1
>>>> >> >
>>>> >> >
>>>> >> > Adding one more point. I could see the following error too in the
>>>> >> > pre-commit build.
>>>> >> >
>>>> >> > [exec] [junit] Exception in thread "CommitProcWorkThread-16"
>>>> >> > java.lang.NoClassDefFoundError:
>>>> >> > org/apache/zookeeper/server/ConnectionBean
>>>> >> > [exec] [junit] at
>>>> >> >
>>>> >> > org.apache.zookeeper.server.ServerCnxnFactory.registerConnection(ServerCnxnFactory.java:159)
>>>> >> > [exec] [junit] at
>>>> >> >
>>>> >> > org.apache.zookeeper.server.ZooKeeperServer.finishSessionInit(ZooKeeperServer.java:594)
>>>> >> > [exec] [junit] at
>>>> >> >
>>>> >> > org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:198)
>>>> >> > [exec] [junit] at
>>>> >> >
>>>> >> > org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:295)
>>>> >> > [exec] [junit] at
>>>> >> >
>>>> >> > org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:161)
>>>> >> > [exec] [junit] at
>>>> >> >
>>>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>>> >> > [exec] [junit] at
>>>> >> >
>>>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>>> >> > [exec] [junit] at
>>>> >> > java.lang.Thread.run(Thread.java:662)
>>>> >> >
>>>> >> > -Rakesh
>>>> >> >
>>>> >> > -----Original Message-----
>>>> >> > From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]
>>>> >> > Sent: 21 July 2014 02:19
>>>> >> > To: dev@zookeeper.apache.org
>>>> >> > Cc: Andrew Bayer; builds@apache.org; Giridharan Kesavan
>>>> >> > Subject: Re: ulimit changed with Apache Jenkins upgrade?
>>>> >> >
>>>> >> > +1
>>>> >> >
>>>> >> > On 18 Jul 2014, at 18:59, Patrick Hunt <ph...@apache.org> wrote:
>>>> >> >
>>>> >> >> Hi builds folks, is this a system wide issue or something we should
>>>> >> >> address ourselves? Thanks!
>>>> >> >>
>>>> >> >> Patrick
>>>> >> >>
>>>> >> >> ---------- Forwarded message ----------
>>>> >> >> From: Patrick Hunt <ph...@apache.org>
>>>> >> >> Date: Fri, Jul 18, 2014 at 10:38 AM
>>>> >> >> Subject: ulimit changed with Apache Jenkins upgrade?
>>>> >> >> To: Giridharan Kesavan <gk...@hortonworks.com>
>>>> >> >> Cc: DevZooKeeper <de...@zookeeper.apache.org>, Andrew Bayer
>>>> >> >> <an...@cloudera.com>
>>>> >> >>
>>>> >> >>
>>>> >> >> Hi Giri, can you check that the new hosts (H#) have the ulimit set
>>>> >> >> to
>>>> >> >> what it was set to on the original hadoop# hosts? I'm seeing new
>>>> >> >> test
>>>> >> >> failures with
>>>> >> >>
>>>> >> >> [exec] [junit] java.io.FileNotFoundException:
>>>> >> >>
>>>> >> >> /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/
>>>> >> >>
>>>> >> >> build/test/tmp/test7610638215300246179.junit.dir/version-2/log.1000000
>>>> >> >> 01
>>>> >> >> (Too many open files)
>>>> >> >>
>>>> >> >> which we've not seen before. I believe this means that we running
>>>> >> >> out
>>>> >> >> of file descriptors?
>>>> >> >>
>>>> >> >> Can you verify and address if possible?
>>>> >> >>
>>>> >> >> Thanks,
>>>> >> >>
>>>> >> >> Patrick
>>>> >> >
>>>> >
>>>> >
>>>> >
>>>> > CONFIDENTIALITY NOTICE
>>>> > NOTICE: This message is intended for the use of the individual or entity
>>>> > to
>>>> > which it is addressed and may contain information that is confidential,
>>>> > privileged and exempt from disclosure under applicable law. If the
>>>> > reader of
>>>> > this message is not the intended recipient, you are hereby notified that
>>>> > any
>>>> > printing, copying, dissemination, distribution, disclosure or forwarding
>>>> > of
>>>> > this communication is strictly prohibited. If you have received this
>>>> > communication in error, please contact the sender immediately and delete
>>>> > it
>>>> > from your system. Thank You.
>>>
>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader of
>>> this message is not the intended recipient, you are hereby notified that any
>>> printing, copying, dissemination, distribution, disclosure or forwarding of
>>> this communication is strictly prohibited. If you have received this
>>> communication in error, please contact the sender immediately and delete it
>>> from your system. Thank You.
Re: ulimit changed with Apache Jenkins upgrade?
Posted by Bong Chen Lock <bo...@gmail.com>.
On 4 Sep 2014 00:21, "Patrick Hunt" <ph...@apache.org> wrote:
> I've opened a BUILDS jira for this:
> https://issues.apache.org/jira/browse/BUILDS-17
>
> On Sat, Aug 30, 2014 at 12:16 PM, Patrick Hunt <ph...@apache.org> wrote:
> > Giri could you or one of the other folks look into this? Our tests
> > have been broken for some time.
> >
> >
> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2303/
> >
> > The same test is still failing with the same error (too many open
> > files) and afaict the ulimit is still just set to 4k. I'm printing the
> > ulimit info at the start of the job, here it is:
> >
> > core file size (blocks, -c) 0
> > data seg size (kbytes, -d) unlimited
> > scheduling priority (-e) 0
> > file size (blocks, -f) unlimited
> > pending signals (-i) 386177
> > max locked memory (kbytes, -l) 64
> > max memory size (kbytes, -m) unlimited
> > open files (-n) 4096
> > pipe size (512 bytes, -p) 8
> > POSIX message queues (bytes, -q) 819200
> > real-time priority (-r) 0
> > stack size (kbytes, -s) 8192
> > cpu time (seconds, -t) unlimited
> > max user processes (-u) 386177
> > virtual memory (kbytes, -v) unlimited
> > file locks (-x) unlimited
> >
> >
> > Patrick
> >
> > On Wed, Jul 23, 2014 at 11:21 AM, Patrick Hunt <ph...@apache.org> wrote:
> >> Does someone in builds@ have the ability to address this? (restart the
> >> jenkins slaves so that the new ulimit limits will take effect)
> >>
> >> Thanks!
> >>
> >> Patrick
> >>
> >> On Wed, Jul 23, 2014 at 9:33 AM, Patrick Hunt <ph...@apache.org> wrote:
> >>> Giri do you mean me? I don't have access to that afaict.
> >>>
> >>> Patrick
> >>>
> >>> On Wed, Jul 23, 2014 at 12:50 AM, Giridharan Kesavan
> >>> <gk...@hortonworks.com> wrote:
> >>>> jenkins slaves might need a restart.
> >>>>
> >>>> Could you pls re-launch the slaves from the jenkins UI configuration
> page?
> >>>>
> >>>> -giri
> >>>>
> >>>>
> >>>> On Tue, Jul 22, 2014 at 2:25 PM, Patrick Hunt <ph...@apache.org>
> wrote:
> >>>>>
> >>>>> Thanks Giri! Unfortunately though it seems to not have taken effect,
> I
> >>>>> just kicked off a precommit build and I see
> >>>>>
> >>>>> core file size (blocks, -c) 0
> >>>>> data seg size (kbytes, -d) unlimited
> >>>>> scheduling priority (-e) 0
> >>>>> file size (blocks, -f) unlimited
> >>>>> pending signals (-i) 386178
> >>>>> max locked memory (kbytes, -l) 64
> >>>>> max memory size (kbytes, -m) unlimited
> >>>>> open files (-n) 4096
> >>>>> pipe size (512 bytes, -p) 8
> >>>>> POSIX message queues (bytes, -q) 819200
> >>>>> real-time priority (-r) 0
> >>>>> stack size (kbytes, -s) 8192
> >>>>> cpu time (seconds, -t) unlimited
> >>>>> max user processes (-u) 386178
> >>>>> virtual memory (kbytes, -v) unlimited
> >>>>> file locks (-x) unlimited
> >>>>>
> >>>>> more details here:
> >>>>>
> >>>>>
> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2215/console
> >>>>>
> >>>>> Patrick
> >>>>>
> >>>>> On Tue, Jul 22, 2014 at 1:29 PM, Giridharan Kesavan
> >>>>> <gk...@hortonworks.com> wrote:
> >>>>> >
> >>>>> >
> >>>>> > jenkins@asf901:~$ ulimit -a
> >>>>> > core file size (blocks, -c) 0
> >>>>> > data seg size (kbytes, -d) unlimited
> >>>>> > scheduling priority (-e) 0
> >>>>> > file size (blocks, -f) unlimited
> >>>>> > pending signals (-i) 386178
> >>>>> > max locked memory (kbytes, -l) 64
> >>>>> > max memory size (kbytes, -m) unlimited
> >>>>> > open files (-n) 60000
> >>>>> > pipe size (512 bytes, -p) 8
> >>>>> > POSIX message queues (bytes, -q) 819200
> >>>>> > real-time priority (-r) 0
> >>>>> > stack size (kbytes, -s) 8192
> >>>>> > cpu time (seconds, -t) unlimited
> >>>>> > max user processes (-u) 10240
> >>>>> > virtual memory (kbytes, -v) unlimited
> >>>>> > file locks (-x) unlimited
> >>>>> >
> >>>>> > bumped up the open files and max user processes on all the slaves.
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > -giri
> >>>>> >
> >>>>> >
> >>>>> > On Tue, Jul 22, 2014 at 12:04 PM, Patrick Hunt <ph...@apache.org>
> wrote:
> >>>>> >>
> >>>>> >> Giri any chance can you take a look at the ulimit issue on the H#
> >>>>> >> machines? All the ZK precommit builds are failing as a result.
> >>>>> >>
> >>>>> >> I updated the precommit build last night to output "ulimit -a"
> and it
> >>>>> >> says the current limit is 4096, can we bump that up or set the
> default
> >>>>> >> to unlimited?
> >>>>> >>
> >>>>> >> Thanks!
> >>>>> >>
> >>>>> >> Patrick
> >>>>> >>
> >>>>> >> On Sun, Jul 20, 2014 at 10:32 PM, Rakesh R <ra...@huawei.com>
> wrote:
> >>>>> >> > +1
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > Adding one more point. I could see the following error too in
> the
> >>>>> >> > pre-commit build.
> >>>>> >> >
> >>>>> >> > [exec] [junit] Exception in thread
> "CommitProcWorkThread-16"
> >>>>> >> > java.lang.NoClassDefFoundError:
> >>>>> >> > org/apache/zookeeper/server/ConnectionBean
> >>>>> >> > [exec] [junit] at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.ServerCnxnFactory.registerConnection(ServerCnxnFactory.java:159)
> >>>>> >> > [exec] [junit] at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.ZooKeeperServer.finishSessionInit(ZooKeeperServer.java:594)
> >>>>> >> > [exec] [junit] at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:198)
> >>>>> >> > [exec] [junit] at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:295)
> >>>>> >> > [exec] [junit] at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:161)
> >>>>> >> > [exec] [junit] at
> >>>>> >> >
> >>>>> >> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> >>>>> >> > [exec] [junit] at
> >>>>> >> >
> >>>>> >> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> >>>>> >> > [exec] [junit] at
> >>>>> >> > java.lang.Thread.run(Thread.java:662)
> >>>>> >> >
> >>>>> >> > -Rakesh
> >>>>> >> >
> >>>>> >> > -----Original Message-----
> >>>>> >> > From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]
> >>>>> >> > Sent: 21 July 2014 02:19
> >>>>> >> > To: dev@zookeeper.apache.org
> >>>>> >> > Cc: Andrew Bayer; builds@apache.org; Giridharan Kesavan
> >>>>> >> > Subject: Re: ulimit changed with Apache Jenkins upgrade?
> >>>>> >> >
> >>>>> >> > +1
> >>>>> >> >
> >>>>> >> > On 18 Jul 2014, at 18:59, Patrick Hunt <ph...@apache.org>
> wrote:
> >>>>> >> >
> >>>>> >> >> Hi builds folks, is this a system wide issue or something we
> should
> >>>>> >> >> address ourselves? Thanks!
> >>>>> >> >>
> >>>>> >> >> Patrick
> >>>>> >> >>
> >>>>> >> >> ---------- Forwarded message ----------
> >>>>> >> >> From: Patrick Hunt <ph...@apache.org>
> >>>>> >> >> Date: Fri, Jul 18, 2014 at 10:38 AM
> >>>>> >> >> Subject: ulimit changed with Apache Jenkins upgrade?
> >>>>> >> >> To: Giridharan Kesavan <gk...@hortonworks.com>
> >>>>> >> >> Cc: DevZooKeeper <de...@zookeeper.apache.org>, Andrew Bayer
> >>>>> >> >> <an...@cloudera.com>
> >>>>> >> >>
> >>>>> >> >>
> >>>>> >> >> Hi Giri, can you check that the new hosts (H#) have the ulimit
> set
> >>>>> >> >> to
> >>>>> >> >> what it was set to on the original hadoop# hosts? I'm seeing
> new
> >>>>> >> >> test
> >>>>> >> >> failures with
> >>>>> >> >>
> >>>>> >> >> [exec] [junit] java.io.FileNotFoundException:
> >>>>> >> >>
> >>>>> >> >>
> /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/
> >>>>> >> >>
> >>>>> >> >>
> build/test/tmp/test7610638215300246179.junit.dir/version-2/log.1000000
> >>>>> >> >> 01
> >>>>> >> >> (Too many open files)
> >>>>> >> >>
> >>>>> >> >> which we've not seen before. I believe this means that we
> running
> >>>>> >> >> out
> >>>>> >> >> of file descriptors?
> >>>>> >> >>
> >>>>> >> >> Can you verify and address if possible?
> >>>>> >> >>
> >>>>> >> >> Thanks,
> >>>>> >> >>
> >>>>> >> >> Patrick
> >>>>> >> >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > CONFIDENTIALITY NOTICE
> >>>>> > NOTICE: This message is intended for the use of the individual or
> entity
> >>>>> > to
> >>>>> > which it is addressed and may contain information that is
> confidential,
> >>>>> > privileged and exempt from disclosure under applicable law. If the
> >>>>> > reader of
> >>>>> > this message is not the intended recipient, you are hereby
> notified that
> >>>>> > any
> >>>>> > printing, copying, dissemination, distribution, disclosure or
> forwarding
> >>>>> > of
> >>>>> > this communication is strictly prohibited. If you have received
> this
> >>>>> > communication in error, please contact the sender immediately and
> delete
> >>>>> > it
> >>>>> > from your system. Thank You.
> >>>>
> >>>>
> >>>>
> >>>> CONFIDENTIALITY NOTICE
> >>>> NOTICE: This message is intended for the use of the individual or
> entity to
> >>>> which it is addressed and may contain information that is
> confidential,
> >>>> privileged and exempt from disclosure under applicable law. If the
> reader of
> >>>> this message is not the intended recipient, you are hereby notified
> that any
> >>>> printing, copying, dissemination, distribution, disclosure or
> forwarding of
> >>>> this communication is strictly prohibited. If you have received this
> >>>> communication in error, please contact the sender immediately and
> delete it
> >>>> from your system. Thank You.
>
Re: ulimit changed with Apache Jenkins upgrade?
Posted by Bong Chen Lock <bo...@gmail.com>.
On 4 Sep 2014 00:24, "Bong Chen Lock" <bo...@gmail.com> wrote:
> On 4 Sep 2014 00:21, "Patrick Hunt" <ph...@apache.org> wrote:
>
>> I've opened a BUILDS jira for this:
>> https://issues.apache.org/jira/browse/BUILDS-17
>>
>> On Sat, Aug 30, 2014 at 12:16 PM, Patrick Hunt <ph...@apache.org> wrote:
>> > Giri could you or one of the other folks look into this? Our tests
>> > have been broken for some time.
>> >
>> >
>> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2303/
>> >
>> > The same test is still failing with the same error (too many open
>> > files) and afaict the ulimit is still just set to 4k. I'm printing the
>> > ulimit info at the start of the job, here it is:
>> >
>> > core file size (blocks, -c) 0
>> > data seg size (kbytes, -d) unlimited
>> > scheduling priority (-e) 0
>> > file size (blocks, -f) unlimited
>> > pending signals (-i) 386177
>> > max locked memory (kbytes, -l) 64
>> > max memory size (kbytes, -m) unlimited
>> > open files (-n) 4096
>> > pipe size (512 bytes, -p) 8
>> > POSIX message queues (bytes, -q) 819200
>> > real-time priority (-r) 0
>> > stack size (kbytes, -s) 8192
>> > cpu time (seconds, -t) unlimited
>> > max user processes (-u) 386177
>> > virtual memory (kbytes, -v) unlimited
>> > file locks (-x) unlimited
>> >
>> >
>> > Patrick
>> >
>> > On Wed, Jul 23, 2014 at 11:21 AM, Patrick Hunt <ph...@apache.org>
>> wrote:
>> >> Does someone in builds@ have the ability to address this? (restart the
>> >> jenkins slaves so that the new ulimit limits will take effect)
>> >>
>> >> Thanks!
>> >>
>> >> Patrick
>> >>
>> >> On Wed, Jul 23, 2014 at 9:33 AM, Patrick Hunt <ph...@apache.org>
>> wrote:
>> >>> Giri do you mean me? I don't have access to that afaict.
>> >>>
>> >>> Patrick
>> >>>
>> >>> On Wed, Jul 23, 2014 at 12:50 AM, Giridharan Kesavan
>> >>> <gk...@hortonworks.com> wrote:
>> >>>> jenkins slaves might need a restart.
>> >>>>
>> >>>> Could you pls re-launch the slaves from the jenkins UI configuration
>> page?
>> >>>>
>> >>>> -giri
>> >>>>
>> >>>>
>> >>>> On Tue, Jul 22, 2014 at 2:25 PM, Patrick Hunt <ph...@apache.org>
>> wrote:
>> >>>>>
>> >>>>> Thanks Giri! Unfortunately though it seems to not have taken
>> effect, I
>> >>>>> just kicked off a precommit build and I see
>> >>>>>
>> >>>>> core file size (blocks, -c) 0
>> >>>>> data seg size (kbytes, -d) unlimited
>> >>>>> scheduling priority (-e) 0
>> >>>>> file size (blocks, -f) unlimited
>> >>>>> pending signals (-i) 386178
>> >>>>> max locked memory (kbytes, -l) 64
>> >>>>> max memory size (kbytes, -m) unlimited
>> >>>>> open files (-n) 4096
>> >>>>> pipe size (512 bytes, -p) 8
>> >>>>> POSIX message queues (bytes, -q) 819200
>> >>>>> real-time priority (-r) 0
>> >>>>> stack size (kbytes, -s) 8192
>> >>>>> cpu time (seconds, -t) unlimited
>> >>>>> max user processes (-u) 386178
>> >>>>> virtual memory (kbytes, -v) unlimited
>> >>>>> file locks (-x) unlimited
>> >>>>>
>> >>>>> more details here:
>> >>>>>
>> >>>>>
>> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2215/console
>> >>>>>
>> >>>>> Patrick
>> >>>>>
>> >>>>> On Tue, Jul 22, 2014 at 1:29 PM, Giridharan Kesavan
>> >>>>> <gk...@hortonworks.com> wrote:
>> >>>>> >
>> >>>>> >
>> >>>>> > jenkins@asf901:~$ ulimit -a
>> >>>>> > core file size (blocks, -c) 0
>> >>>>> > data seg size (kbytes, -d) unlimited
>> >>>>> > scheduling priority (-e) 0
>> >>>>> > file size (blocks, -f) unlimited
>> >>>>> > pending signals (-i) 386178
>> >>>>> > max locked memory (kbytes, -l) 64
>> >>>>> > max memory size (kbytes, -m) unlimited
>> >>>>> > open files (-n) 60000
>> >>>>> > pipe size (512 bytes, -p) 8
>> >>>>> > POSIX message queues (bytes, -q) 819200
>> >>>>> > real-time priority (-r) 0
>> >>>>> > stack size (kbytes, -s) 8192
>> >>>>> > cpu time (seconds, -t) unlimited
>> >>>>> > max user processes (-u) 10240
>> >>>>> > virtual memory (kbytes, -v) unlimited
>> >>>>> > file locks (-x) unlimited
>> >>>>> >
>> >>>>> > bumped up the open files and max user processes on all the slaves.
>> >>>>> >
>> >>>>> >
>> >>>>> >
>> >>>>> > -giri
>> >>>>> >
>> >>>>> >
>> >>>>> > On Tue, Jul 22, 2014 at 12:04 PM, Patrick Hunt <ph...@apache.org>
>> wrote:
>> >>>>> >>
>> >>>>> >> Giri any chance can you take a look at the ulimit issue on the H#
>> >>>>> >> machines? All the ZK precommit builds are failing as a result.
>> >>>>> >>
>> >>>>> >> I updated the precommit build last night to output "ulimit -a"
>> and it
>> >>>>> >> says the current limit is 4096, can we bump that up or set the
>> default
>> >>>>> >> to unlimited?
>> >>>>> >>
>> >>>>> >> Thanks!
>> >>>>> >>
>> >>>>> >> Patrick
>> >>>>> >>
>> >>>>> >> On Sun, Jul 20, 2014 at 10:32 PM, Rakesh R <ra...@huawei.com>
>> wrote:
>> >>>>> >> > +1
>> >>>>> >> >
>> >>>>> >> >
>> >>>>> >> > Adding one more point. I could see the following error too in
>> the
>> >>>>> >> > pre-commit build.
>> >>>>> >> >
>> >>>>> >> > [exec] [junit] Exception in thread
>> "CommitProcWorkThread-16"
>> >>>>> >> > java.lang.NoClassDefFoundError:
>> >>>>> >> > org/apache/zookeeper/server/ConnectionBean
>> >>>>> >> > [exec] [junit] at
>> >>>>> >> >
>> >>>>> >> >
>> org.apache.zookeeper.server.ServerCnxnFactory.registerConnection(ServerCnxnFactory.java:159)
>> >>>>> >> > [exec] [junit] at
>> >>>>> >> >
>> >>>>> >> >
>> org.apache.zookeeper.server.ZooKeeperServer.finishSessionInit(ZooKeeperServer.java:594)
>> >>>>> >> > [exec] [junit] at
>> >>>>> >> >
>> >>>>> >> >
>> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:198)
>> >>>>> >> > [exec] [junit] at
>> >>>>> >> >
>> >>>>> >> >
>> org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:295)
>> >>>>> >> > [exec] [junit] at
>> >>>>> >> >
>> >>>>> >> >
>> org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:161)
>> >>>>> >> > [exec] [junit] at
>> >>>>> >> >
>> >>>>> >> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>> >>>>> >> > [exec] [junit] at
>> >>>>> >> >
>> >>>>> >> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>> >>>>> >> > [exec] [junit] at
>> >>>>> >> > java.lang.Thread.run(Thread.java:662)
>> >>>>> >> >
>> >>>>> >> > -Rakesh
>> >>>>> >> >
>> >>>>> >> > -----Original Message-----
>> >>>>> >> > From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]
>> >>>>> >> > Sent: 21 July 2014 02:19
>> >>>>> >> > To: dev@zookeeper.apache.org
>> >>>>> >> > Cc: Andrew Bayer; builds@apache.org; Giridharan Kesavan
>> >>>>> >> > Subject: Re: ulimit changed with Apache Jenkins upgrade?
>> >>>>> >> >
>> >>>>> >> > +1
>> >>>>> >> >
>> >>>>> >> > On 18 Jul 2014, at 18:59, Patrick Hunt <ph...@apache.org>
>> wrote:
>> >>>>> >> >
>> >>>>> >> >> Hi builds folks, is this a system wide issue or something we
>> should
>> >>>>> >> >> address ourselves? Thanks!
>> >>>>> >> >>
>> >>>>> >> >> Patrick
>> >>>>> >> >>
>> >>>>> >> >> ---------- Forwarded message ----------
>> >>>>> >> >> From: Patrick Hunt <ph...@apache.org>
>> >>>>> >> >> Date: Fri, Jul 18, 2014 at 10:38 AM
>> >>>>> >> >> Subject: ulimit changed with Apache Jenkins upgrade?
>> >>>>> >> >> To: Giridharan Kesavan <gk...@hortonworks.com>
>> >>>>> >> >> Cc: DevZooKeeper <de...@zookeeper.apache.org>, Andrew Bayer
>> >>>>> >> >> <an...@cloudera.com>
>> >>>>> >> >>
>> >>>>> >> >>
>> >>>>> >> >> Hi Giri, can you check that the new hosts (H#) have the
>> ulimit set
>> >>>>> >> >> to
>> >>>>> >> >> what it was set to on the original hadoop# hosts? I'm seeing
>> new
>> >>>>> >> >> test
>> >>>>> >> >> failures with
>> >>>>> >> >>
>> >>>>> >> >> [exec] [junit] java.io.FileNotFoundException:
>> >>>>> >> >>
>> >>>>> >> >>
>> /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/
>> >>>>> >> >>
>> >>>>> >> >>
>> build/test/tmp/test7610638215300246179.junit.dir/version-2/log.1000000
>> >>>>> >> >> 01
>> >>>>> >> >> (Too many open files)
>> >>>>> >> >>
>> >>>>> >> >> which we've not seen before. I believe this means that we
>> running
>> >>>>> >> >> out
>> >>>>> >> >> of file descriptors?
>> >>>>> >> >>
>> >>>>> >> >> Can you verify and address if possible?
>> >>>>> >> >>
>> >>>>> >> >> Thanks,
>> >>>>> >> >>
>> >>>>> >> >> Patrick
>> >>>>> >> >
>> >>>>> >
>> >>>>> >
>> >>>>> >
>> >>>>> > CONFIDENTIALITY NOTICE
>> >>>>> > NOTICE: This message is intended for the use of the individual or
>> entity
>> >>>>> > to
>> >>>>> > which it is addressed and may contain information that is
>> confidential,
>> >>>>> > privileged and exempt from disclosure under applicable law. If the
>> >>>>> > reader of
>> >>>>> > this message is not the intended recipient, you are hereby
>> notified that
>> >>>>> > any
>> >>>>> > printing, copying, dissemination, distribution, disclosure or
>> forwarding
>> >>>>> > of
>> >>>>> > this communication is strictly prohibited. If you have received
>> this
>> >>>>> > communication in error, please contact the sender immediately and
>> delete
>> >>>>> > it
>> >>>>> > from your system. Thank You.
>> >>>>
>> >>>>
>> >>>>
>> >>>> CONFIDENTIALITY NOTICE
>> >>>> NOTICE: This message is intended for the use of the individual or
>> entity to
>> >>>> which it is addressed and may contain information that is
>> confidential,
>> >>>> privileged and exempt from disclosure under applicable law. If the
>> reader of
>> >>>> this message is not the intended recipient, you are hereby notified
>> that any
>> >>>> printing, copying, dissemination, distribution, disclosure or
>> forwarding of
>> >>>> this communication is strictly prohibited. If you have received this
>> >>>> communication in error, please contact the sender immediately and
>> delete it
>> >>>> from your system. Thank You.
>>
>
Re: ulimit changed with Apache Jenkins upgrade?
Posted by Bong Chen Lock <bo...@gmail.com>.
On 4 Sep 2014 00:21, "Patrick Hunt" <ph...@apache.org> wrote:
> I've opened a BUILDS jira for this:
> https://issues.apache.org/jira/browse/BUILDS-17
>
> On Sat, Aug 30, 2014 at 12:16 PM, Patrick Hunt <ph...@apache.org> wrote:
> > Giri could you or one of the other folks look into this? Our tests
> > have been broken for some time.
> >
> >
> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2303/
> >
> > The same test is still failing with the same error (too many open
> > files) and afaict the ulimit is still just set to 4k. I'm printing the
> > ulimit info at the start of the job, here it is:
> >
> > core file size (blocks, -c) 0
> > data seg size (kbytes, -d) unlimited
> > scheduling priority (-e) 0
> > file size (blocks, -f) unlimited
> > pending signals (-i) 386177
> > max locked memory (kbytes, -l) 64
> > max memory size (kbytes, -m) unlimited
> > open files (-n) 4096
> > pipe size (512 bytes, -p) 8
> > POSIX message queues (bytes, -q) 819200
> > real-time priority (-r) 0
> > stack size (kbytes, -s) 8192
> > cpu time (seconds, -t) unlimited
> > max user processes (-u) 386177
> > virtual memory (kbytes, -v) unlimited
> > file locks (-x) unlimited
> >
> >
> > Patrick
> >
> > On Wed, Jul 23, 2014 at 11:21 AM, Patrick Hunt <ph...@apache.org> wrote:
> >> Does someone in builds@ have the ability to address this? (restart the
> >> jenkins slaves so that the new ulimit limits will take effect)
> >>
> >> Thanks!
> >>
> >> Patrick
> >>
> >> On Wed, Jul 23, 2014 at 9:33 AM, Patrick Hunt <ph...@apache.org> wrote:
> >>> Giri do you mean me? I don't have access to that afaict.
> >>>
> >>> Patrick
> >>>
> >>> On Wed, Jul 23, 2014 at 12:50 AM, Giridharan Kesavan
> >>> <gk...@hortonworks.com> wrote:
> >>>> jenkins slaves might need a restart.
> >>>>
> >>>> Could you pls re-launch the slaves from the jenkins UI configuration
> page?
> >>>>
> >>>> -giri
> >>>>
> >>>>
> >>>> On Tue, Jul 22, 2014 at 2:25 PM, Patrick Hunt <ph...@apache.org>
> wrote:
> >>>>>
> >>>>> Thanks Giri! Unfortunately though it seems to not have taken effect,
> I
> >>>>> just kicked off a precommit build and I see
> >>>>>
> >>>>> core file size (blocks, -c) 0
> >>>>> data seg size (kbytes, -d) unlimited
> >>>>> scheduling priority (-e) 0
> >>>>> file size (blocks, -f) unlimited
> >>>>> pending signals (-i) 386178
> >>>>> max locked memory (kbytes, -l) 64
> >>>>> max memory size (kbytes, -m) unlimited
> >>>>> open files (-n) 4096
> >>>>> pipe size (512 bytes, -p) 8
> >>>>> POSIX message queues (bytes, -q) 819200
> >>>>> real-time priority (-r) 0
> >>>>> stack size (kbytes, -s) 8192
> >>>>> cpu time (seconds, -t) unlimited
> >>>>> max user processes (-u) 386178
> >>>>> virtual memory (kbytes, -v) unlimited
> >>>>> file locks (-x) unlimited
> >>>>>
> >>>>> more details here:
> >>>>>
> >>>>>
> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2215/console
> >>>>>
> >>>>> Patrick
> >>>>>
> >>>>> On Tue, Jul 22, 2014 at 1:29 PM, Giridharan Kesavan
> >>>>> <gk...@hortonworks.com> wrote:
> >>>>> >
> >>>>> >
> >>>>> > jenkins@asf901:~$ ulimit -a
> >>>>> > core file size (blocks, -c) 0
> >>>>> > data seg size (kbytes, -d) unlimited
> >>>>> > scheduling priority (-e) 0
> >>>>> > file size (blocks, -f) unlimited
> >>>>> > pending signals (-i) 386178
> >>>>> > max locked memory (kbytes, -l) 64
> >>>>> > max memory size (kbytes, -m) unlimited
> >>>>> > open files (-n) 60000
> >>>>> > pipe size (512 bytes, -p) 8
> >>>>> > POSIX message queues (bytes, -q) 819200
> >>>>> > real-time priority (-r) 0
> >>>>> > stack size (kbytes, -s) 8192
> >>>>> > cpu time (seconds, -t) unlimited
> >>>>> > max user processes (-u) 10240
> >>>>> > virtual memory (kbytes, -v) unlimited
> >>>>> > file locks (-x) unlimited
> >>>>> >
> >>>>> > bumped up the open files and max user processes on all the slaves.
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > -giri
> >>>>> >
> >>>>> >
> >>>>> > On Tue, Jul 22, 2014 at 12:04 PM, Patrick Hunt <ph...@apache.org>
> wrote:
> >>>>> >>
> >>>>> >> Giri any chance can you take a look at the ulimit issue on the H#
> >>>>> >> machines? All the ZK precommit builds are failing as a result.
> >>>>> >>
> >>>>> >> I updated the precommit build last night to output "ulimit -a"
> and it
> >>>>> >> says the current limit is 4096, can we bump that up or set the
> default
> >>>>> >> to unlimited?
> >>>>> >>
> >>>>> >> Thanks!
> >>>>> >>
> >>>>> >> Patrick
> >>>>> >>
> >>>>> >> On Sun, Jul 20, 2014 at 10:32 PM, Rakesh R <ra...@huawei.com>
> wrote:
> >>>>> >> > +1
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > Adding one more point. I could see the following error too in
> the
> >>>>> >> > pre-commit build.
> >>>>> >> >
> >>>>> >> > [exec] [junit] Exception in thread
> "CommitProcWorkThread-16"
> >>>>> >> > java.lang.NoClassDefFoundError:
> >>>>> >> > org/apache/zookeeper/server/ConnectionBean
> >>>>> >> > [exec] [junit] at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.ServerCnxnFactory.registerConnection(ServerCnxnFactory.java:159)
> >>>>> >> > [exec] [junit] at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.ZooKeeperServer.finishSessionInit(ZooKeeperServer.java:594)
> >>>>> >> > [exec] [junit] at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:198)
> >>>>> >> > [exec] [junit] at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:295)
> >>>>> >> > [exec] [junit] at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:161)
> >>>>> >> > [exec] [junit] at
> >>>>> >> >
> >>>>> >> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> >>>>> >> > [exec] [junit] at
> >>>>> >> >
> >>>>> >> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> >>>>> >> > [exec] [junit] at
> >>>>> >> > java.lang.Thread.run(Thread.java:662)
> >>>>> >> >
> >>>>> >> > -Rakesh
> >>>>> >> >
> >>>>> >> > -----Original Message-----
> >>>>> >> > From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]
> >>>>> >> > Sent: 21 July 2014 02:19
> >>>>> >> > To: dev@zookeeper.apache.org
> >>>>> >> > Cc: Andrew Bayer; builds@apache.org; Giridharan Kesavan
> >>>>> >> > Subject: Re: ulimit changed with Apache Jenkins upgrade?
> >>>>> >> >
> >>>>> >> > +1
> >>>>> >> >
> >>>>> >> > On 18 Jul 2014, at 18:59, Patrick Hunt <ph...@apache.org>
> wrote:
> >>>>> >> >
> >>>>> >> >> Hi builds folks, is this a system wide issue or something we
> should
> >>>>> >> >> address ourselves? Thanks!
> >>>>> >> >>
> >>>>> >> >> Patrick
> >>>>> >> >>
> >>>>> >> >> ---------- Forwarded message ----------
> >>>>> >> >> From: Patrick Hunt <ph...@apache.org>
> >>>>> >> >> Date: Fri, Jul 18, 2014 at 10:38 AM
> >>>>> >> >> Subject: ulimit changed with Apache Jenkins upgrade?
> >>>>> >> >> To: Giridharan Kesavan <gk...@hortonworks.com>
> >>>>> >> >> Cc: DevZooKeeper <de...@zookeeper.apache.org>, Andrew Bayer
> >>>>> >> >> <an...@cloudera.com>
> >>>>> >> >>
> >>>>> >> >>
> >>>>> >> >> Hi Giri, can you check that the new hosts (H#) have the ulimit
> set
> >>>>> >> >> to
> >>>>> >> >> what it was set to on the original hadoop# hosts? I'm seeing
> new
> >>>>> >> >> test
> >>>>> >> >> failures with
> >>>>> >> >>
> >>>>> >> >> [exec] [junit] java.io.FileNotFoundException:
> >>>>> >> >>
> >>>>> >> >>
> /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/
> >>>>> >> >>
> >>>>> >> >>
> build/test/tmp/test7610638215300246179.junit.dir/version-2/log.1000000
> >>>>> >> >> 01
> >>>>> >> >> (Too many open files)
> >>>>> >> >>
> >>>>> >> >> which we've not seen before. I believe this means that we
> running
> >>>>> >> >> out
> >>>>> >> >> of file descriptors?
> >>>>> >> >>
> >>>>> >> >> Can you verify and address if possible?
> >>>>> >> >>
> >>>>> >> >> Thanks,
> >>>>> >> >>
> >>>>> >> >> Patrick
> >>>>> >> >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > CONFIDENTIALITY NOTICE
> >>>>> > NOTICE: This message is intended for the use of the individual or
> entity
> >>>>> > to
> >>>>> > which it is addressed and may contain information that is
> confidential,
> >>>>> > privileged and exempt from disclosure under applicable law. If the
> >>>>> > reader of
> >>>>> > this message is not the intended recipient, you are hereby
> notified that
> >>>>> > any
> >>>>> > printing, copying, dissemination, distribution, disclosure or
> forwarding
> >>>>> > of
> >>>>> > this communication is strictly prohibited. If you have received
> this
> >>>>> > communication in error, please contact the sender immediately and
> delete
> >>>>> > it
> >>>>> > from your system. Thank You.
> >>>>
> >>>>
> >>>>
> >>>> CONFIDENTIALITY NOTICE
> >>>> NOTICE: This message is intended for the use of the individual or
> entity to
> >>>> which it is addressed and may contain information that is
> confidential,
> >>>> privileged and exempt from disclosure under applicable law. If the
> reader of
> >>>> this message is not the intended recipient, you are hereby notified
> that any
> >>>> printing, copying, dissemination, distribution, disclosure or
> forwarding of
> >>>> this communication is strictly prohibited. If you have received this
> >>>> communication in error, please contact the sender immediately and
> delete it
> >>>> from your system. Thank You.
>
Re: ulimit changed with Apache Jenkins upgrade?
Posted by Patrick Hunt <ph...@apache.org>.
I've opened a BUILDS jira for this:
https://issues.apache.org/jira/browse/BUILDS-17
On Sat, Aug 30, 2014 at 12:16 PM, Patrick Hunt <ph...@apache.org> wrote:
> Giri could you or one of the other folks look into this? Our tests
> have been broken for some time.
>
> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2303/
>
> The same test is still failing with the same error (too many open
> files) and afaict the ulimit is still just set to 4k. I'm printing the
> ulimit info at the start of the job, here it is:
>
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) unlimited
> pending signals (-i) 386177
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 4096
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 386177
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
>
>
> Patrick
>
> On Wed, Jul 23, 2014 at 11:21 AM, Patrick Hunt <ph...@apache.org> wrote:
>> Does someone in builds@ have the ability to address this? (restart the
>> jenkins slaves so that the new ulimit limits will take effect)
>>
>> Thanks!
>>
>> Patrick
>>
>> On Wed, Jul 23, 2014 at 9:33 AM, Patrick Hunt <ph...@apache.org> wrote:
>>> Giri do you mean me? I don't have access to that afaict.
>>>
>>> Patrick
>>>
>>> On Wed, Jul 23, 2014 at 12:50 AM, Giridharan Kesavan
>>> <gk...@hortonworks.com> wrote:
>>>> jenkins slaves might need a restart.
>>>>
>>>> Could you pls re-launch the slaves from the jenkins UI configuration page?
>>>>
>>>> -giri
>>>>
>>>>
>>>> On Tue, Jul 22, 2014 at 2:25 PM, Patrick Hunt <ph...@apache.org> wrote:
>>>>>
>>>>> Thanks Giri! Unfortunately though it seems to not have taken effect, I
>>>>> just kicked off a precommit build and I see
>>>>>
>>>>> core file size (blocks, -c) 0
>>>>> data seg size (kbytes, -d) unlimited
>>>>> scheduling priority (-e) 0
>>>>> file size (blocks, -f) unlimited
>>>>> pending signals (-i) 386178
>>>>> max locked memory (kbytes, -l) 64
>>>>> max memory size (kbytes, -m) unlimited
>>>>> open files (-n) 4096
>>>>> pipe size (512 bytes, -p) 8
>>>>> POSIX message queues (bytes, -q) 819200
>>>>> real-time priority (-r) 0
>>>>> stack size (kbytes, -s) 8192
>>>>> cpu time (seconds, -t) unlimited
>>>>> max user processes (-u) 386178
>>>>> virtual memory (kbytes, -v) unlimited
>>>>> file locks (-x) unlimited
>>>>>
>>>>> more details here:
>>>>>
>>>>> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2215/console
>>>>>
>>>>> Patrick
>>>>>
>>>>> On Tue, Jul 22, 2014 at 1:29 PM, Giridharan Kesavan
>>>>> <gk...@hortonworks.com> wrote:
>>>>> >
>>>>> >
>>>>> > jenkins@asf901:~$ ulimit -a
>>>>> > core file size (blocks, -c) 0
>>>>> > data seg size (kbytes, -d) unlimited
>>>>> > scheduling priority (-e) 0
>>>>> > file size (blocks, -f) unlimited
>>>>> > pending signals (-i) 386178
>>>>> > max locked memory (kbytes, -l) 64
>>>>> > max memory size (kbytes, -m) unlimited
>>>>> > open files (-n) 60000
>>>>> > pipe size (512 bytes, -p) 8
>>>>> > POSIX message queues (bytes, -q) 819200
>>>>> > real-time priority (-r) 0
>>>>> > stack size (kbytes, -s) 8192
>>>>> > cpu time (seconds, -t) unlimited
>>>>> > max user processes (-u) 10240
>>>>> > virtual memory (kbytes, -v) unlimited
>>>>> > file locks (-x) unlimited
>>>>> >
>>>>> > bumped up the open files and max user processes on all the slaves.
>>>>> >
>>>>> >
>>>>> >
>>>>> > -giri
>>>>> >
>>>>> >
>>>>> > On Tue, Jul 22, 2014 at 12:04 PM, Patrick Hunt <ph...@apache.org> wrote:
>>>>> >>
>>>>> >> Giri any chance can you take a look at the ulimit issue on the H#
>>>>> >> machines? All the ZK precommit builds are failing as a result.
>>>>> >>
>>>>> >> I updated the precommit build last night to output "ulimit -a" and it
>>>>> >> says the current limit is 4096, can we bump that up or set the default
>>>>> >> to unlimited?
>>>>> >>
>>>>> >> Thanks!
>>>>> >>
>>>>> >> Patrick
>>>>> >>
>>>>> >> On Sun, Jul 20, 2014 at 10:32 PM, Rakesh R <ra...@huawei.com> wrote:
>>>>> >> > +1
>>>>> >> >
>>>>> >> >
>>>>> >> > Adding one more point. I could see the following error too in the
>>>>> >> > pre-commit build.
>>>>> >> >
>>>>> >> > [exec] [junit] Exception in thread "CommitProcWorkThread-16"
>>>>> >> > java.lang.NoClassDefFoundError:
>>>>> >> > org/apache/zookeeper/server/ConnectionBean
>>>>> >> > [exec] [junit] at
>>>>> >> >
>>>>> >> > org.apache.zookeeper.server.ServerCnxnFactory.registerConnection(ServerCnxnFactory.java:159)
>>>>> >> > [exec] [junit] at
>>>>> >> >
>>>>> >> > org.apache.zookeeper.server.ZooKeeperServer.finishSessionInit(ZooKeeperServer.java:594)
>>>>> >> > [exec] [junit] at
>>>>> >> >
>>>>> >> > org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:198)
>>>>> >> > [exec] [junit] at
>>>>> >> >
>>>>> >> > org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:295)
>>>>> >> > [exec] [junit] at
>>>>> >> >
>>>>> >> > org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:161)
>>>>> >> > [exec] [junit] at
>>>>> >> >
>>>>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>>>> >> > [exec] [junit] at
>>>>> >> >
>>>>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>>>> >> > [exec] [junit] at
>>>>> >> > java.lang.Thread.run(Thread.java:662)
>>>>> >> >
>>>>> >> > -Rakesh
>>>>> >> >
>>>>> >> > -----Original Message-----
>>>>> >> > From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]
>>>>> >> > Sent: 21 July 2014 02:19
>>>>> >> > To: dev@zookeeper.apache.org
>>>>> >> > Cc: Andrew Bayer; builds@apache.org; Giridharan Kesavan
>>>>> >> > Subject: Re: ulimit changed with Apache Jenkins upgrade?
>>>>> >> >
>>>>> >> > +1
>>>>> >> >
>>>>> >> > On 18 Jul 2014, at 18:59, Patrick Hunt <ph...@apache.org> wrote:
>>>>> >> >
>>>>> >> >> Hi builds folks, is this a system wide issue or something we should
>>>>> >> >> address ourselves? Thanks!
>>>>> >> >>
>>>>> >> >> Patrick
>>>>> >> >>
>>>>> >> >> ---------- Forwarded message ----------
>>>>> >> >> From: Patrick Hunt <ph...@apache.org>
>>>>> >> >> Date: Fri, Jul 18, 2014 at 10:38 AM
>>>>> >> >> Subject: ulimit changed with Apache Jenkins upgrade?
>>>>> >> >> To: Giridharan Kesavan <gk...@hortonworks.com>
>>>>> >> >> Cc: DevZooKeeper <de...@zookeeper.apache.org>, Andrew Bayer
>>>>> >> >> <an...@cloudera.com>
>>>>> >> >>
>>>>> >> >>
>>>>> >> >> Hi Giri, can you check that the new hosts (H#) have the ulimit set
>>>>> >> >> to
>>>>> >> >> what it was set to on the original hadoop# hosts? I'm seeing new
>>>>> >> >> test
>>>>> >> >> failures with
>>>>> >> >>
>>>>> >> >> [exec] [junit] java.io.FileNotFoundException:
>>>>> >> >>
>>>>> >> >> /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/
>>>>> >> >>
>>>>> >> >> build/test/tmp/test7610638215300246179.junit.dir/version-2/log.1000000
>>>>> >> >> 01
>>>>> >> >> (Too many open files)
>>>>> >> >>
>>>>> >> >> which we've not seen before. I believe this means that we running
>>>>> >> >> out
>>>>> >> >> of file descriptors?
>>>>> >> >>
>>>>> >> >> Can you verify and address if possible?
>>>>> >> >>
>>>>> >> >> Thanks,
>>>>> >> >>
>>>>> >> >> Patrick
>>>>> >> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > CONFIDENTIALITY NOTICE
>>>>> > NOTICE: This message is intended for the use of the individual or entity
>>>>> > to
>>>>> > which it is addressed and may contain information that is confidential,
>>>>> > privileged and exempt from disclosure under applicable law. If the
>>>>> > reader of
>>>>> > this message is not the intended recipient, you are hereby notified that
>>>>> > any
>>>>> > printing, copying, dissemination, distribution, disclosure or forwarding
>>>>> > of
>>>>> > this communication is strictly prohibited. If you have received this
>>>>> > communication in error, please contact the sender immediately and delete
>>>>> > it
>>>>> > from your system. Thank You.
>>>>
>>>>
>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or entity to
>>>> which it is addressed and may contain information that is confidential,
>>>> privileged and exempt from disclosure under applicable law. If the reader of
>>>> this message is not the intended recipient, you are hereby notified that any
>>>> printing, copying, dissemination, distribution, disclosure or forwarding of
>>>> this communication is strictly prohibited. If you have received this
>>>> communication in error, please contact the sender immediately and delete it
>>>> from your system. Thank You.
Re: ulimit changed with Apache Jenkins upgrade?
Posted by Patrick Hunt <ph...@apache.org>.
I've opened a BUILDS jira for this:
https://issues.apache.org/jira/browse/BUILDS-17
On Sat, Aug 30, 2014 at 12:16 PM, Patrick Hunt <ph...@apache.org> wrote:
> Giri could you or one of the other folks look into this? Our tests
> have been broken for some time.
>
> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2303/
>
> The same test is still failing with the same error (too many open
> files) and afaict the ulimit is still just set to 4k. I'm printing the
> ulimit info at the start of the job, here it is:
>
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) unlimited
> pending signals (-i) 386177
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 4096
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 386177
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
>
>
> Patrick
>
> On Wed, Jul 23, 2014 at 11:21 AM, Patrick Hunt <ph...@apache.org> wrote:
>> Does someone in builds@ have the ability to address this? (restart the
>> jenkins slaves so that the new ulimit limits will take effect)
>>
>> Thanks!
>>
>> Patrick
>>
>> On Wed, Jul 23, 2014 at 9:33 AM, Patrick Hunt <ph...@apache.org> wrote:
>>> Giri do you mean me? I don't have access to that afaict.
>>>
>>> Patrick
>>>
>>> On Wed, Jul 23, 2014 at 12:50 AM, Giridharan Kesavan
>>> <gk...@hortonworks.com> wrote:
>>>> jenkins slaves might need a restart.
>>>>
>>>> Could you pls re-launch the slaves from the jenkins UI configuration page?
>>>>
>>>> -giri
>>>>
>>>>
>>>> On Tue, Jul 22, 2014 at 2:25 PM, Patrick Hunt <ph...@apache.org> wrote:
>>>>>
>>>>> Thanks Giri! Unfortunately though it seems to not have taken effect, I
>>>>> just kicked off a precommit build and I see
>>>>>
>>>>> core file size (blocks, -c) 0
>>>>> data seg size (kbytes, -d) unlimited
>>>>> scheduling priority (-e) 0
>>>>> file size (blocks, -f) unlimited
>>>>> pending signals (-i) 386178
>>>>> max locked memory (kbytes, -l) 64
>>>>> max memory size (kbytes, -m) unlimited
>>>>> open files (-n) 4096
>>>>> pipe size (512 bytes, -p) 8
>>>>> POSIX message queues (bytes, -q) 819200
>>>>> real-time priority (-r) 0
>>>>> stack size (kbytes, -s) 8192
>>>>> cpu time (seconds, -t) unlimited
>>>>> max user processes (-u) 386178
>>>>> virtual memory (kbytes, -v) unlimited
>>>>> file locks (-x) unlimited
>>>>>
>>>>> more details here:
>>>>>
>>>>> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2215/console
>>>>>
>>>>> Patrick
>>>>>
>>>>> On Tue, Jul 22, 2014 at 1:29 PM, Giridharan Kesavan
>>>>> <gk...@hortonworks.com> wrote:
>>>>> >
>>>>> >
>>>>> > jenkins@asf901:~$ ulimit -a
>>>>> > core file size (blocks, -c) 0
>>>>> > data seg size (kbytes, -d) unlimited
>>>>> > scheduling priority (-e) 0
>>>>> > file size (blocks, -f) unlimited
>>>>> > pending signals (-i) 386178
>>>>> > max locked memory (kbytes, -l) 64
>>>>> > max memory size (kbytes, -m) unlimited
>>>>> > open files (-n) 60000
>>>>> > pipe size (512 bytes, -p) 8
>>>>> > POSIX message queues (bytes, -q) 819200
>>>>> > real-time priority (-r) 0
>>>>> > stack size (kbytes, -s) 8192
>>>>> > cpu time (seconds, -t) unlimited
>>>>> > max user processes (-u) 10240
>>>>> > virtual memory (kbytes, -v) unlimited
>>>>> > file locks (-x) unlimited
>>>>> >
>>>>> > bumped up the open files and max user processes on all the slaves.
>>>>> >
>>>>> >
>>>>> >
>>>>> > -giri
>>>>> >
>>>>> >
>>>>> > On Tue, Jul 22, 2014 at 12:04 PM, Patrick Hunt <ph...@apache.org> wrote:
>>>>> >>
>>>>> >> Giri any chance can you take a look at the ulimit issue on the H#
>>>>> >> machines? All the ZK precommit builds are failing as a result.
>>>>> >>
>>>>> >> I updated the precommit build last night to output "ulimit -a" and it
>>>>> >> says the current limit is 4096, can we bump that up or set the default
>>>>> >> to unlimited?
>>>>> >>
>>>>> >> Thanks!
>>>>> >>
>>>>> >> Patrick
>>>>> >>
>>>>> >> On Sun, Jul 20, 2014 at 10:32 PM, Rakesh R <ra...@huawei.com> wrote:
>>>>> >> > +1
>>>>> >> >
>>>>> >> >
>>>>> >> > Adding one more point. I could see the following error too in the
>>>>> >> > pre-commit build.
>>>>> >> >
>>>>> >> > [exec] [junit] Exception in thread "CommitProcWorkThread-16"
>>>>> >> > java.lang.NoClassDefFoundError:
>>>>> >> > org/apache/zookeeper/server/ConnectionBean
>>>>> >> > [exec] [junit] at
>>>>> >> >
>>>>> >> > org.apache.zookeeper.server.ServerCnxnFactory.registerConnection(ServerCnxnFactory.java:159)
>>>>> >> > [exec] [junit] at
>>>>> >> >
>>>>> >> > org.apache.zookeeper.server.ZooKeeperServer.finishSessionInit(ZooKeeperServer.java:594)
>>>>> >> > [exec] [junit] at
>>>>> >> >
>>>>> >> > org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:198)
>>>>> >> > [exec] [junit] at
>>>>> >> >
>>>>> >> > org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:295)
>>>>> >> > [exec] [junit] at
>>>>> >> >
>>>>> >> > org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:161)
>>>>> >> > [exec] [junit] at
>>>>> >> >
>>>>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>>>> >> > [exec] [junit] at
>>>>> >> >
>>>>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>>>> >> > [exec] [junit] at
>>>>> >> > java.lang.Thread.run(Thread.java:662)
>>>>> >> >
>>>>> >> > -Rakesh
>>>>> >> >
>>>>> >> > -----Original Message-----
>>>>> >> > From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]
>>>>> >> > Sent: 21 July 2014 02:19
>>>>> >> > To: dev@zookeeper.apache.org
>>>>> >> > Cc: Andrew Bayer; builds@apache.org; Giridharan Kesavan
>>>>> >> > Subject: Re: ulimit changed with Apache Jenkins upgrade?
>>>>> >> >
>>>>> >> > +1
>>>>> >> >
>>>>> >> > On 18 Jul 2014, at 18:59, Patrick Hunt <ph...@apache.org> wrote:
>>>>> >> >
>>>>> >> >> Hi builds folks, is this a system wide issue or something we should
>>>>> >> >> address ourselves? Thanks!
>>>>> >> >>
>>>>> >> >> Patrick
>>>>> >> >>
>>>>> >> >> ---------- Forwarded message ----------
>>>>> >> >> From: Patrick Hunt <ph...@apache.org>
>>>>> >> >> Date: Fri, Jul 18, 2014 at 10:38 AM
>>>>> >> >> Subject: ulimit changed with Apache Jenkins upgrade?
>>>>> >> >> To: Giridharan Kesavan <gk...@hortonworks.com>
>>>>> >> >> Cc: DevZooKeeper <de...@zookeeper.apache.org>, Andrew Bayer
>>>>> >> >> <an...@cloudera.com>
>>>>> >> >>
>>>>> >> >>
>>>>> >> >> Hi Giri, can you check that the new hosts (H#) have the ulimit set
>>>>> >> >> to
>>>>> >> >> what it was set to on the original hadoop# hosts? I'm seeing new
>>>>> >> >> test
>>>>> >> >> failures with
>>>>> >> >>
>>>>> >> >> [exec] [junit] java.io.FileNotFoundException:
>>>>> >> >>
>>>>> >> >> /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/
>>>>> >> >>
>>>>> >> >> build/test/tmp/test7610638215300246179.junit.dir/version-2/log.1000000
>>>>> >> >> 01
>>>>> >> >> (Too many open files)
>>>>> >> >>
>>>>> >> >> which we've not seen before. I believe this means that we running
>>>>> >> >> out
>>>>> >> >> of file descriptors?
>>>>> >> >>
>>>>> >> >> Can you verify and address if possible?
>>>>> >> >>
>>>>> >> >> Thanks,
>>>>> >> >>
>>>>> >> >> Patrick
>>>>> >> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > CONFIDENTIALITY NOTICE
>>>>> > NOTICE: This message is intended for the use of the individual or entity
>>>>> > to
>>>>> > which it is addressed and may contain information that is confidential,
>>>>> > privileged and exempt from disclosure under applicable law. If the
>>>>> > reader of
>>>>> > this message is not the intended recipient, you are hereby notified that
>>>>> > any
>>>>> > printing, copying, dissemination, distribution, disclosure or forwarding
>>>>> > of
>>>>> > this communication is strictly prohibited. If you have received this
>>>>> > communication in error, please contact the sender immediately and delete
>>>>> > it
>>>>> > from your system. Thank You.
>>>>
>>>>
>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or entity to
>>>> which it is addressed and may contain information that is confidential,
>>>> privileged and exempt from disclosure under applicable law. If the reader of
>>>> this message is not the intended recipient, you are hereby notified that any
>>>> printing, copying, dissemination, distribution, disclosure or forwarding of
>>>> this communication is strictly prohibited. If you have received this
>>>> communication in error, please contact the sender immediately and delete it
>>>> from your system. Thank You.