You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by Patrick Hunt <ph...@apache.org> on 2014/07/18 19:38:41 UTC
ulimit changed with Apache Jenkins upgrade?
Hi Giri, can you check that the new hosts (H#) have the ulimit set to
what it was set to on the original hadoop# hosts? I'm seeing new test
failures with
[exec] [junit] java.io.FileNotFoundException:
/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/tmp/test7610638215300246179.junit.dir/version-2/log.100000001
(Too many open files)
which we've not seen before. I believe this means that we running out
of file descriptors?
Can you verify and address if possible?
Thanks,
Patrick
Re: ulimit changed with Apache Jenkins upgrade?
Posted by Bong Chen Lock <bo...@gmail.com>.
On 4 Sep 2014 00:21, "Patrick Hunt" <ph...@apache.org> wrote:
> I've opened a BUILDS jira for this:
> https://issues.apache.org/jira/browse/BUILDS-17
>
> On Sat, Aug 30, 2014 at 12:16 PM, Patrick Hunt <ph...@apache.org> wrote:
> > Giri could you or one of the other folks look into this? Our tests
> > have been broken for some time.
> >
> >
> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2303/
> >
> > The same test is still failing with the same error (too many open
> > files) and afaict the ulimit is still just set to 4k. I'm printing the
> > ulimit info at the start of the job, here it is:
> >
> > core file size (blocks, -c) 0
> > data seg size (kbytes, -d) unlimited
> > scheduling priority (-e) 0
> > file size (blocks, -f) unlimited
> > pending signals (-i) 386177
> > max locked memory (kbytes, -l) 64
> > max memory size (kbytes, -m) unlimited
> > open files (-n) 4096
> > pipe size (512 bytes, -p) 8
> > POSIX message queues (bytes, -q) 819200
> > real-time priority (-r) 0
> > stack size (kbytes, -s) 8192
> > cpu time (seconds, -t) unlimited
> > max user processes (-u) 386177
> > virtual memory (kbytes, -v) unlimited
> > file locks (-x) unlimited
> >
> >
> > Patrick
> >
> > On Wed, Jul 23, 2014 at 11:21 AM, Patrick Hunt <ph...@apache.org> wrote:
> >> Does someone in builds@ have the ability to address this? (restart the
> >> jenkins slaves so that the new ulimit limits will take effect)
> >>
> >> Thanks!
> >>
> >> Patrick
> >>
> >> On Wed, Jul 23, 2014 at 9:33 AM, Patrick Hunt <ph...@apache.org> wrote:
> >>> Giri do you mean me? I don't have access to that afaict.
> >>>
> >>> Patrick
> >>>
> >>> On Wed, Jul 23, 2014 at 12:50 AM, Giridharan Kesavan
> >>> <gk...@hortonworks.com> wrote:
> >>>> jenkins slaves might need a restart.
> >>>>
> >>>> Could you pls re-launch the slaves from the jenkins UI configuration
> page?
> >>>>
> >>>> -giri
> >>>>
> >>>>
> >>>> On Tue, Jul 22, 2014 at 2:25 PM, Patrick Hunt <ph...@apache.org>
> wrote:
> >>>>>
> >>>>> Thanks Giri! Unfortunately though it seems to not have taken effect,
> I
> >>>>> just kicked off a precommit build and I see
> >>>>>
> >>>>> core file size (blocks, -c) 0
> >>>>> data seg size (kbytes, -d) unlimited
> >>>>> scheduling priority (-e) 0
> >>>>> file size (blocks, -f) unlimited
> >>>>> pending signals (-i) 386178
> >>>>> max locked memory (kbytes, -l) 64
> >>>>> max memory size (kbytes, -m) unlimited
> >>>>> open files (-n) 4096
> >>>>> pipe size (512 bytes, -p) 8
> >>>>> POSIX message queues (bytes, -q) 819200
> >>>>> real-time priority (-r) 0
> >>>>> stack size (kbytes, -s) 8192
> >>>>> cpu time (seconds, -t) unlimited
> >>>>> max user processes (-u) 386178
> >>>>> virtual memory (kbytes, -v) unlimited
> >>>>> file locks (-x) unlimited
> >>>>>
> >>>>> more details here:
> >>>>>
> >>>>>
> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2215/console
> >>>>>
> >>>>> Patrick
> >>>>>
> >>>>> On Tue, Jul 22, 2014 at 1:29 PM, Giridharan Kesavan
> >>>>> <gk...@hortonworks.com> wrote:
> >>>>> >
> >>>>> >
> >>>>> > jenkins@asf901:~$ ulimit -a
> >>>>> > core file size (blocks, -c) 0
> >>>>> > data seg size (kbytes, -d) unlimited
> >>>>> > scheduling priority (-e) 0
> >>>>> > file size (blocks, -f) unlimited
> >>>>> > pending signals (-i) 386178
> >>>>> > max locked memory (kbytes, -l) 64
> >>>>> > max memory size (kbytes, -m) unlimited
> >>>>> > open files (-n) 60000
> >>>>> > pipe size (512 bytes, -p) 8
> >>>>> > POSIX message queues (bytes, -q) 819200
> >>>>> > real-time priority (-r) 0
> >>>>> > stack size (kbytes, -s) 8192
> >>>>> > cpu time (seconds, -t) unlimited
> >>>>> > max user processes (-u) 10240
> >>>>> > virtual memory (kbytes, -v) unlimited
> >>>>> > file locks (-x) unlimited
> >>>>> >
> >>>>> > bumped up the open files and max user processes on all the slaves.
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > -giri
> >>>>> >
> >>>>> >
> >>>>> > On Tue, Jul 22, 2014 at 12:04 PM, Patrick Hunt <ph...@apache.org>
> wrote:
> >>>>> >>
> >>>>> >> Giri any chance can you take a look at the ulimit issue on the H#
> >>>>> >> machines? All the ZK precommit builds are failing as a result.
> >>>>> >>
> >>>>> >> I updated the precommit build last night to output "ulimit -a"
> and it
> >>>>> >> says the current limit is 4096, can we bump that up or set the
> default
> >>>>> >> to unlimited?
> >>>>> >>
> >>>>> >> Thanks!
> >>>>> >>
> >>>>> >> Patrick
> >>>>> >>
> >>>>> >> On Sun, Jul 20, 2014 at 10:32 PM, Rakesh R <ra...@huawei.com>
> wrote:
> >>>>> >> > +1
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > Adding one more point. I could see the following error too in
> the
> >>>>> >> > pre-commit build.
> >>>>> >> >
> >>>>> >> > [exec] [junit] Exception in thread
> "CommitProcWorkThread-16"
> >>>>> >> > java.lang.NoClassDefFoundError:
> >>>>> >> > org/apache/zookeeper/server/ConnectionBean
> >>>>> >> > [exec] [junit] at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.ServerCnxnFactory.registerConnection(ServerCnxnFactory.java:159)
> >>>>> >> > [exec] [junit] at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.ZooKeeperServer.finishSessionInit(ZooKeeperServer.java:594)
> >>>>> >> > [exec] [junit] at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:198)
> >>>>> >> > [exec] [junit] at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:295)
> >>>>> >> > [exec] [junit] at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:161)
> >>>>> >> > [exec] [junit] at
> >>>>> >> >
> >>>>> >> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> >>>>> >> > [exec] [junit] at
> >>>>> >> >
> >>>>> >> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> >>>>> >> > [exec] [junit] at
> >>>>> >> > java.lang.Thread.run(Thread.java:662)
> >>>>> >> >
> >>>>> >> > -Rakesh
> >>>>> >> >
> >>>>> >> > -----Original Message-----
> >>>>> >> > From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]
> >>>>> >> > Sent: 21 July 2014 02:19
> >>>>> >> > To: dev@zookeeper.apache.org
> >>>>> >> > Cc: Andrew Bayer; builds@apache.org; Giridharan Kesavan
> >>>>> >> > Subject: Re: ulimit changed with Apache Jenkins upgrade?
> >>>>> >> >
> >>>>> >> > +1
> >>>>> >> >
> >>>>> >> > On 18 Jul 2014, at 18:59, Patrick Hunt <ph...@apache.org>
> wrote:
> >>>>> >> >
> >>>>> >> >> Hi builds folks, is this a system wide issue or something we
> should
> >>>>> >> >> address ourselves? Thanks!
> >>>>> >> >>
> >>>>> >> >> Patrick
> >>>>> >> >>
> >>>>> >> >> ---------- Forwarded message ----------
> >>>>> >> >> From: Patrick Hunt <ph...@apache.org>
> >>>>> >> >> Date: Fri, Jul 18, 2014 at 10:38 AM
> >>>>> >> >> Subject: ulimit changed with Apache Jenkins upgrade?
> >>>>> >> >> To: Giridharan Kesavan <gk...@hortonworks.com>
> >>>>> >> >> Cc: DevZooKeeper <de...@zookeeper.apache.org>, Andrew Bayer
> >>>>> >> >> <an...@cloudera.com>
> >>>>> >> >>
> >>>>> >> >>
> >>>>> >> >> Hi Giri, can you check that the new hosts (H#) have the ulimit
> set
> >>>>> >> >> to
> >>>>> >> >> what it was set to on the original hadoop# hosts? I'm seeing
> new
> >>>>> >> >> test
> >>>>> >> >> failures with
> >>>>> >> >>
> >>>>> >> >> [exec] [junit] java.io.FileNotFoundException:
> >>>>> >> >>
> >>>>> >> >>
> /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/
> >>>>> >> >>
> >>>>> >> >>
> build/test/tmp/test7610638215300246179.junit.dir/version-2/log.1000000
> >>>>> >> >> 01
> >>>>> >> >> (Too many open files)
> >>>>> >> >>
> >>>>> >> >> which we've not seen before. I believe this means that we
> running
> >>>>> >> >> out
> >>>>> >> >> of file descriptors?
> >>>>> >> >>
> >>>>> >> >> Can you verify and address if possible?
> >>>>> >> >>
> >>>>> >> >> Thanks,
> >>>>> >> >>
> >>>>> >> >> Patrick
> >>>>> >> >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > CONFIDENTIALITY NOTICE
> >>>>> > NOTICE: This message is intended for the use of the individual or
> entity
> >>>>> > to
> >>>>> > which it is addressed and may contain information that is
> confidential,
> >>>>> > privileged and exempt from disclosure under applicable law. If the
> >>>>> > reader of
> >>>>> > this message is not the intended recipient, you are hereby
> notified that
> >>>>> > any
> >>>>> > printing, copying, dissemination, distribution, disclosure or
> forwarding
> >>>>> > of
> >>>>> > this communication is strictly prohibited. If you have received
> this
> >>>>> > communication in error, please contact the sender immediately and
> delete
> >>>>> > it
> >>>>> > from your system. Thank You.
> >>>>
> >>>>
> >>>>
> >>>> CONFIDENTIALITY NOTICE
> >>>> NOTICE: This message is intended for the use of the individual or
> entity to
> >>>> which it is addressed and may contain information that is
> confidential,
> >>>> privileged and exempt from disclosure under applicable law. If the
> reader of
> >>>> this message is not the intended recipient, you are hereby notified
> that any
> >>>> printing, copying, dissemination, distribution, disclosure or
> forwarding of
> >>>> this communication is strictly prohibited. If you have received this
> >>>> communication in error, please contact the sender immediately and
> delete it
> >>>> from your system. Thank You.
>
Re: ulimit changed with Apache Jenkins upgrade?
Posted by Bong Chen Lock <bo...@gmail.com>.
On 4 Sep 2014 00:24, "Bong Chen Lock" <bo...@gmail.com> wrote:
> On 4 Sep 2014 00:21, "Patrick Hunt" <ph...@apache.org> wrote:
>
>> I've opened a BUILDS jira for this:
>> https://issues.apache.org/jira/browse/BUILDS-17
>>
>> On Sat, Aug 30, 2014 at 12:16 PM, Patrick Hunt <ph...@apache.org> wrote:
>> > Giri could you or one of the other folks look into this? Our tests
>> > have been broken for some time.
>> >
>> >
>> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2303/
>> >
>> > The same test is still failing with the same error (too many open
>> > files) and afaict the ulimit is still just set to 4k. I'm printing the
>> > ulimit info at the start of the job, here it is:
>> >
>> > core file size (blocks, -c) 0
>> > data seg size (kbytes, -d) unlimited
>> > scheduling priority (-e) 0
>> > file size (blocks, -f) unlimited
>> > pending signals (-i) 386177
>> > max locked memory (kbytes, -l) 64
>> > max memory size (kbytes, -m) unlimited
>> > open files (-n) 4096
>> > pipe size (512 bytes, -p) 8
>> > POSIX message queues (bytes, -q) 819200
>> > real-time priority (-r) 0
>> > stack size (kbytes, -s) 8192
>> > cpu time (seconds, -t) unlimited
>> > max user processes (-u) 386177
>> > virtual memory (kbytes, -v) unlimited
>> > file locks (-x) unlimited
>> >
>> >
>> > Patrick
>> >
>> > On Wed, Jul 23, 2014 at 11:21 AM, Patrick Hunt <ph...@apache.org>
>> wrote:
>> >> Does someone in builds@ have the ability to address this? (restart the
>> >> jenkins slaves so that the new ulimit limits will take effect)
>> >>
>> >> Thanks!
>> >>
>> >> Patrick
>> >>
>> >> On Wed, Jul 23, 2014 at 9:33 AM, Patrick Hunt <ph...@apache.org>
>> wrote:
>> >>> Giri do you mean me? I don't have access to that afaict.
>> >>>
>> >>> Patrick
>> >>>
>> >>> On Wed, Jul 23, 2014 at 12:50 AM, Giridharan Kesavan
>> >>> <gk...@hortonworks.com> wrote:
>> >>>> jenkins slaves might need a restart.
>> >>>>
>> >>>> Could you pls re-launch the slaves from the jenkins UI configuration
>> page?
>> >>>>
>> >>>> -giri
>> >>>>
>> >>>>
>> >>>> On Tue, Jul 22, 2014 at 2:25 PM, Patrick Hunt <ph...@apache.org>
>> wrote:
>> >>>>>
>> >>>>> Thanks Giri! Unfortunately though it seems to not have taken
>> effect, I
>> >>>>> just kicked off a precommit build and I see
>> >>>>>
>> >>>>> core file size (blocks, -c) 0
>> >>>>> data seg size (kbytes, -d) unlimited
>> >>>>> scheduling priority (-e) 0
>> >>>>> file size (blocks, -f) unlimited
>> >>>>> pending signals (-i) 386178
>> >>>>> max locked memory (kbytes, -l) 64
>> >>>>> max memory size (kbytes, -m) unlimited
>> >>>>> open files (-n) 4096
>> >>>>> pipe size (512 bytes, -p) 8
>> >>>>> POSIX message queues (bytes, -q) 819200
>> >>>>> real-time priority (-r) 0
>> >>>>> stack size (kbytes, -s) 8192
>> >>>>> cpu time (seconds, -t) unlimited
>> >>>>> max user processes (-u) 386178
>> >>>>> virtual memory (kbytes, -v) unlimited
>> >>>>> file locks (-x) unlimited
>> >>>>>
>> >>>>> more details here:
>> >>>>>
>> >>>>>
>> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2215/console
>> >>>>>
>> >>>>> Patrick
>> >>>>>
>> >>>>> On Tue, Jul 22, 2014 at 1:29 PM, Giridharan Kesavan
>> >>>>> <gk...@hortonworks.com> wrote:
>> >>>>> >
>> >>>>> >
>> >>>>> > jenkins@asf901:~$ ulimit -a
>> >>>>> > core file size (blocks, -c) 0
>> >>>>> > data seg size (kbytes, -d) unlimited
>> >>>>> > scheduling priority (-e) 0
>> >>>>> > file size (blocks, -f) unlimited
>> >>>>> > pending signals (-i) 386178
>> >>>>> > max locked memory (kbytes, -l) 64
>> >>>>> > max memory size (kbytes, -m) unlimited
>> >>>>> > open files (-n) 60000
>> >>>>> > pipe size (512 bytes, -p) 8
>> >>>>> > POSIX message queues (bytes, -q) 819200
>> >>>>> > real-time priority (-r) 0
>> >>>>> > stack size (kbytes, -s) 8192
>> >>>>> > cpu time (seconds, -t) unlimited
>> >>>>> > max user processes (-u) 10240
>> >>>>> > virtual memory (kbytes, -v) unlimited
>> >>>>> > file locks (-x) unlimited
>> >>>>> >
>> >>>>> > bumped up the open files and max user processes on all the slaves.
>> >>>>> >
>> >>>>> >
>> >>>>> >
>> >>>>> > -giri
>> >>>>> >
>> >>>>> >
>> >>>>> > On Tue, Jul 22, 2014 at 12:04 PM, Patrick Hunt <ph...@apache.org>
>> wrote:
>> >>>>> >>
>> >>>>> >> Giri any chance can you take a look at the ulimit issue on the H#
>> >>>>> >> machines? All the ZK precommit builds are failing as a result.
>> >>>>> >>
>> >>>>> >> I updated the precommit build last night to output "ulimit -a"
>> and it
>> >>>>> >> says the current limit is 4096, can we bump that up or set the
>> default
>> >>>>> >> to unlimited?
>> >>>>> >>
>> >>>>> >> Thanks!
>> >>>>> >>
>> >>>>> >> Patrick
>> >>>>> >>
>> >>>>> >> On Sun, Jul 20, 2014 at 10:32 PM, Rakesh R <ra...@huawei.com>
>> wrote:
>> >>>>> >> > +1
>> >>>>> >> >
>> >>>>> >> >
>> >>>>> >> > Adding one more point. I could see the following error too in
>> the
>> >>>>> >> > pre-commit build.
>> >>>>> >> >
>> >>>>> >> > [exec] [junit] Exception in thread
>> "CommitProcWorkThread-16"
>> >>>>> >> > java.lang.NoClassDefFoundError:
>> >>>>> >> > org/apache/zookeeper/server/ConnectionBean
>> >>>>> >> > [exec] [junit] at
>> >>>>> >> >
>> >>>>> >> >
>> org.apache.zookeeper.server.ServerCnxnFactory.registerConnection(ServerCnxnFactory.java:159)
>> >>>>> >> > [exec] [junit] at
>> >>>>> >> >
>> >>>>> >> >
>> org.apache.zookeeper.server.ZooKeeperServer.finishSessionInit(ZooKeeperServer.java:594)
>> >>>>> >> > [exec] [junit] at
>> >>>>> >> >
>> >>>>> >> >
>> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:198)
>> >>>>> >> > [exec] [junit] at
>> >>>>> >> >
>> >>>>> >> >
>> org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:295)
>> >>>>> >> > [exec] [junit] at
>> >>>>> >> >
>> >>>>> >> >
>> org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:161)
>> >>>>> >> > [exec] [junit] at
>> >>>>> >> >
>> >>>>> >> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>> >>>>> >> > [exec] [junit] at
>> >>>>> >> >
>> >>>>> >> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>> >>>>> >> > [exec] [junit] at
>> >>>>> >> > java.lang.Thread.run(Thread.java:662)
>> >>>>> >> >
>> >>>>> >> > -Rakesh
>> >>>>> >> >
>> >>>>> >> > -----Original Message-----
>> >>>>> >> > From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]
>> >>>>> >> > Sent: 21 July 2014 02:19
>> >>>>> >> > To: dev@zookeeper.apache.org
>> >>>>> >> > Cc: Andrew Bayer; builds@apache.org; Giridharan Kesavan
>> >>>>> >> > Subject: Re: ulimit changed with Apache Jenkins upgrade?
>> >>>>> >> >
>> >>>>> >> > +1
>> >>>>> >> >
>> >>>>> >> > On 18 Jul 2014, at 18:59, Patrick Hunt <ph...@apache.org>
>> wrote:
>> >>>>> >> >
>> >>>>> >> >> Hi builds folks, is this a system wide issue or something we
>> should
>> >>>>> >> >> address ourselves? Thanks!
>> >>>>> >> >>
>> >>>>> >> >> Patrick
>> >>>>> >> >>
>> >>>>> >> >> ---------- Forwarded message ----------
>> >>>>> >> >> From: Patrick Hunt <ph...@apache.org>
>> >>>>> >> >> Date: Fri, Jul 18, 2014 at 10:38 AM
>> >>>>> >> >> Subject: ulimit changed with Apache Jenkins upgrade?
>> >>>>> >> >> To: Giridharan Kesavan <gk...@hortonworks.com>
>> >>>>> >> >> Cc: DevZooKeeper <de...@zookeeper.apache.org>, Andrew Bayer
>> >>>>> >> >> <an...@cloudera.com>
>> >>>>> >> >>
>> >>>>> >> >>
>> >>>>> >> >> Hi Giri, can you check that the new hosts (H#) have the
>> ulimit set
>> >>>>> >> >> to
>> >>>>> >> >> what it was set to on the original hadoop# hosts? I'm seeing
>> new
>> >>>>> >> >> test
>> >>>>> >> >> failures with
>> >>>>> >> >>
>> >>>>> >> >> [exec] [junit] java.io.FileNotFoundException:
>> >>>>> >> >>
>> >>>>> >> >>
>> /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/
>> >>>>> >> >>
>> >>>>> >> >>
>> build/test/tmp/test7610638215300246179.junit.dir/version-2/log.1000000
>> >>>>> >> >> 01
>> >>>>> >> >> (Too many open files)
>> >>>>> >> >>
>> >>>>> >> >> which we've not seen before. I believe this means that we
>> running
>> >>>>> >> >> out
>> >>>>> >> >> of file descriptors?
>> >>>>> >> >>
>> >>>>> >> >> Can you verify and address if possible?
>> >>>>> >> >>
>> >>>>> >> >> Thanks,
>> >>>>> >> >>
>> >>>>> >> >> Patrick
>> >>>>> >> >
>> >>>>> >
>> >>>>> >
>> >>>>> >
>> >>>>> > CONFIDENTIALITY NOTICE
>> >>>>> > NOTICE: This message is intended for the use of the individual or
>> entity
>> >>>>> > to
>> >>>>> > which it is addressed and may contain information that is
>> confidential,
>> >>>>> > privileged and exempt from disclosure under applicable law. If the
>> >>>>> > reader of
>> >>>>> > this message is not the intended recipient, you are hereby
>> notified that
>> >>>>> > any
>> >>>>> > printing, copying, dissemination, distribution, disclosure or
>> forwarding
>> >>>>> > of
>> >>>>> > this communication is strictly prohibited. If you have received
>> this
>> >>>>> > communication in error, please contact the sender immediately and
>> delete
>> >>>>> > it
>> >>>>> > from your system. Thank You.
>> >>>>
>> >>>>
>> >>>>
>> >>>> CONFIDENTIALITY NOTICE
>> >>>> NOTICE: This message is intended for the use of the individual or
>> entity to
>> >>>> which it is addressed and may contain information that is
>> confidential,
>> >>>> privileged and exempt from disclosure under applicable law. If the
>> reader of
>> >>>> this message is not the intended recipient, you are hereby notified
>> that any
>> >>>> printing, copying, dissemination, distribution, disclosure or
>> forwarding of
>> >>>> this communication is strictly prohibited. If you have received this
>> >>>> communication in error, please contact the sender immediately and
>> delete it
>> >>>> from your system. Thank You.
>>
>
Re: ulimit changed with Apache Jenkins upgrade?
Posted by Bong Chen Lock <bo...@gmail.com>.
On 4 Sep 2014 00:21, "Patrick Hunt" <ph...@apache.org> wrote:
> I've opened a BUILDS jira for this:
> https://issues.apache.org/jira/browse/BUILDS-17
>
> On Sat, Aug 30, 2014 at 12:16 PM, Patrick Hunt <ph...@apache.org> wrote:
> > Giri could you or one of the other folks look into this? Our tests
> > have been broken for some time.
> >
> >
> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2303/
> >
> > The same test is still failing with the same error (too many open
> > files) and afaict the ulimit is still just set to 4k. I'm printing the
> > ulimit info at the start of the job, here it is:
> >
> > core file size (blocks, -c) 0
> > data seg size (kbytes, -d) unlimited
> > scheduling priority (-e) 0
> > file size (blocks, -f) unlimited
> > pending signals (-i) 386177
> > max locked memory (kbytes, -l) 64
> > max memory size (kbytes, -m) unlimited
> > open files (-n) 4096
> > pipe size (512 bytes, -p) 8
> > POSIX message queues (bytes, -q) 819200
> > real-time priority (-r) 0
> > stack size (kbytes, -s) 8192
> > cpu time (seconds, -t) unlimited
> > max user processes (-u) 386177
> > virtual memory (kbytes, -v) unlimited
> > file locks (-x) unlimited
> >
> >
> > Patrick
> >
> > On Wed, Jul 23, 2014 at 11:21 AM, Patrick Hunt <ph...@apache.org> wrote:
> >> Does someone in builds@ have the ability to address this? (restart the
> >> jenkins slaves so that the new ulimit limits will take effect)
> >>
> >> Thanks!
> >>
> >> Patrick
> >>
> >> On Wed, Jul 23, 2014 at 9:33 AM, Patrick Hunt <ph...@apache.org> wrote:
> >>> Giri do you mean me? I don't have access to that afaict.
> >>>
> >>> Patrick
> >>>
> >>> On Wed, Jul 23, 2014 at 12:50 AM, Giridharan Kesavan
> >>> <gk...@hortonworks.com> wrote:
> >>>> jenkins slaves might need a restart.
> >>>>
> >>>> Could you pls re-launch the slaves from the jenkins UI configuration
> page?
> >>>>
> >>>> -giri
> >>>>
> >>>>
> >>>> On Tue, Jul 22, 2014 at 2:25 PM, Patrick Hunt <ph...@apache.org>
> wrote:
> >>>>>
> >>>>> Thanks Giri! Unfortunately though it seems to not have taken effect,
> I
> >>>>> just kicked off a precommit build and I see
> >>>>>
> >>>>> core file size (blocks, -c) 0
> >>>>> data seg size (kbytes, -d) unlimited
> >>>>> scheduling priority (-e) 0
> >>>>> file size (blocks, -f) unlimited
> >>>>> pending signals (-i) 386178
> >>>>> max locked memory (kbytes, -l) 64
> >>>>> max memory size (kbytes, -m) unlimited
> >>>>> open files (-n) 4096
> >>>>> pipe size (512 bytes, -p) 8
> >>>>> POSIX message queues (bytes, -q) 819200
> >>>>> real-time priority (-r) 0
> >>>>> stack size (kbytes, -s) 8192
> >>>>> cpu time (seconds, -t) unlimited
> >>>>> max user processes (-u) 386178
> >>>>> virtual memory (kbytes, -v) unlimited
> >>>>> file locks (-x) unlimited
> >>>>>
> >>>>> more details here:
> >>>>>
> >>>>>
> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2215/console
> >>>>>
> >>>>> Patrick
> >>>>>
> >>>>> On Tue, Jul 22, 2014 at 1:29 PM, Giridharan Kesavan
> >>>>> <gk...@hortonworks.com> wrote:
> >>>>> >
> >>>>> >
> >>>>> > jenkins@asf901:~$ ulimit -a
> >>>>> > core file size (blocks, -c) 0
> >>>>> > data seg size (kbytes, -d) unlimited
> >>>>> > scheduling priority (-e) 0
> >>>>> > file size (blocks, -f) unlimited
> >>>>> > pending signals (-i) 386178
> >>>>> > max locked memory (kbytes, -l) 64
> >>>>> > max memory size (kbytes, -m) unlimited
> >>>>> > open files (-n) 60000
> >>>>> > pipe size (512 bytes, -p) 8
> >>>>> > POSIX message queues (bytes, -q) 819200
> >>>>> > real-time priority (-r) 0
> >>>>> > stack size (kbytes, -s) 8192
> >>>>> > cpu time (seconds, -t) unlimited
> >>>>> > max user processes (-u) 10240
> >>>>> > virtual memory (kbytes, -v) unlimited
> >>>>> > file locks (-x) unlimited
> >>>>> >
> >>>>> > bumped up the open files and max user processes on all the slaves.
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > -giri
> >>>>> >
> >>>>> >
> >>>>> > On Tue, Jul 22, 2014 at 12:04 PM, Patrick Hunt <ph...@apache.org>
> wrote:
> >>>>> >>
> >>>>> >> Giri any chance can you take a look at the ulimit issue on the H#
> >>>>> >> machines? All the ZK precommit builds are failing as a result.
> >>>>> >>
> >>>>> >> I updated the precommit build last night to output "ulimit -a"
> and it
> >>>>> >> says the current limit is 4096, can we bump that up or set the
> default
> >>>>> >> to unlimited?
> >>>>> >>
> >>>>> >> Thanks!
> >>>>> >>
> >>>>> >> Patrick
> >>>>> >>
> >>>>> >> On Sun, Jul 20, 2014 at 10:32 PM, Rakesh R <ra...@huawei.com>
> wrote:
> >>>>> >> > +1
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > Adding one more point. I could see the following error too in
> the
> >>>>> >> > pre-commit build.
> >>>>> >> >
> >>>>> >> > [exec] [junit] Exception in thread
> "CommitProcWorkThread-16"
> >>>>> >> > java.lang.NoClassDefFoundError:
> >>>>> >> > org/apache/zookeeper/server/ConnectionBean
> >>>>> >> > [exec] [junit] at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.ServerCnxnFactory.registerConnection(ServerCnxnFactory.java:159)
> >>>>> >> > [exec] [junit] at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.ZooKeeperServer.finishSessionInit(ZooKeeperServer.java:594)
> >>>>> >> > [exec] [junit] at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:198)
> >>>>> >> > [exec] [junit] at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:295)
> >>>>> >> > [exec] [junit] at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:161)
> >>>>> >> > [exec] [junit] at
> >>>>> >> >
> >>>>> >> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> >>>>> >> > [exec] [junit] at
> >>>>> >> >
> >>>>> >> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> >>>>> >> > [exec] [junit] at
> >>>>> >> > java.lang.Thread.run(Thread.java:662)
> >>>>> >> >
> >>>>> >> > -Rakesh
> >>>>> >> >
> >>>>> >> > -----Original Message-----
> >>>>> >> > From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]
> >>>>> >> > Sent: 21 July 2014 02:19
> >>>>> >> > To: dev@zookeeper.apache.org
> >>>>> >> > Cc: Andrew Bayer; builds@apache.org; Giridharan Kesavan
> >>>>> >> > Subject: Re: ulimit changed with Apache Jenkins upgrade?
> >>>>> >> >
> >>>>> >> > +1
> >>>>> >> >
> >>>>> >> > On 18 Jul 2014, at 18:59, Patrick Hunt <ph...@apache.org>
> wrote:
> >>>>> >> >
> >>>>> >> >> Hi builds folks, is this a system wide issue or something we
> should
> >>>>> >> >> address ourselves? Thanks!
> >>>>> >> >>
> >>>>> >> >> Patrick
> >>>>> >> >>
> >>>>> >> >> ---------- Forwarded message ----------
> >>>>> >> >> From: Patrick Hunt <ph...@apache.org>
> >>>>> >> >> Date: Fri, Jul 18, 2014 at 10:38 AM
> >>>>> >> >> Subject: ulimit changed with Apache Jenkins upgrade?
> >>>>> >> >> To: Giridharan Kesavan <gk...@hortonworks.com>
> >>>>> >> >> Cc: DevZooKeeper <de...@zookeeper.apache.org>, Andrew Bayer
> >>>>> >> >> <an...@cloudera.com>
> >>>>> >> >>
> >>>>> >> >>
> >>>>> >> >> Hi Giri, can you check that the new hosts (H#) have the ulimit
> set
> >>>>> >> >> to
> >>>>> >> >> what it was set to on the original hadoop# hosts? I'm seeing
> new
> >>>>> >> >> test
> >>>>> >> >> failures with
> >>>>> >> >>
> >>>>> >> >> [exec] [junit] java.io.FileNotFoundException:
> >>>>> >> >>
> >>>>> >> >>
> /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/
> >>>>> >> >>
> >>>>> >> >>
> build/test/tmp/test7610638215300246179.junit.dir/version-2/log.1000000
> >>>>> >> >> 01
> >>>>> >> >> (Too many open files)
> >>>>> >> >>
> >>>>> >> >> which we've not seen before. I believe this means that we
> running
> >>>>> >> >> out
> >>>>> >> >> of file descriptors?
> >>>>> >> >>
> >>>>> >> >> Can you verify and address if possible?
> >>>>> >> >>
> >>>>> >> >> Thanks,
> >>>>> >> >>
> >>>>> >> >> Patrick
> >>>>> >> >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > CONFIDENTIALITY NOTICE
> >>>>> > NOTICE: This message is intended for the use of the individual or
> entity
> >>>>> > to
> >>>>> > which it is addressed and may contain information that is
> confidential,
> >>>>> > privileged and exempt from disclosure under applicable law. If the
> >>>>> > reader of
> >>>>> > this message is not the intended recipient, you are hereby
> notified that
> >>>>> > any
> >>>>> > printing, copying, dissemination, distribution, disclosure or
> forwarding
> >>>>> > of
> >>>>> > this communication is strictly prohibited. If you have received
> this
> >>>>> > communication in error, please contact the sender immediately and
> delete
> >>>>> > it
> >>>>> > from your system. Thank You.
> >>>>
> >>>>
> >>>>
> >>>> CONFIDENTIALITY NOTICE
> >>>> NOTICE: This message is intended for the use of the individual or
> entity to
> >>>> which it is addressed and may contain information that is
> confidential,
> >>>> privileged and exempt from disclosure under applicable law. If the
> reader of
> >>>> this message is not the intended recipient, you are hereby notified
> that any
> >>>> printing, copying, dissemination, distribution, disclosure or
> forwarding of
> >>>> this communication is strictly prohibited. If you have received this
> >>>> communication in error, please contact the sender immediately and
> delete it
> >>>> from your system. Thank You.
>
Re: ulimit changed with Apache Jenkins upgrade?
Posted by Patrick Hunt <ph...@apache.org>.
I've opened a BUILDS jira for this:
https://issues.apache.org/jira/browse/BUILDS-17
On Sat, Aug 30, 2014 at 12:16 PM, Patrick Hunt <ph...@apache.org> wrote:
> Giri could you or one of the other folks look into this? Our tests
> have been broken for some time.
>
> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2303/
>
> The same test is still failing with the same error (too many open
> files) and afaict the ulimit is still just set to 4k. I'm printing the
> ulimit info at the start of the job, here it is:
>
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) unlimited
> pending signals (-i) 386177
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 4096
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 386177
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
>
>
> Patrick
>
> On Wed, Jul 23, 2014 at 11:21 AM, Patrick Hunt <ph...@apache.org> wrote:
>> Does someone in builds@ have the ability to address this? (restart the
>> jenkins slaves so that the new ulimit limits will take effect)
>>
>> Thanks!
>>
>> Patrick
>>
>> On Wed, Jul 23, 2014 at 9:33 AM, Patrick Hunt <ph...@apache.org> wrote:
>>> Giri do you mean me? I don't have access to that afaict.
>>>
>>> Patrick
>>>
>>> On Wed, Jul 23, 2014 at 12:50 AM, Giridharan Kesavan
>>> <gk...@hortonworks.com> wrote:
>>>> jenkins slaves might need a restart.
>>>>
>>>> Could you pls re-launch the slaves from the jenkins UI configuration page?
>>>>
>>>> -giri
>>>>
>>>>
>>>> On Tue, Jul 22, 2014 at 2:25 PM, Patrick Hunt <ph...@apache.org> wrote:
>>>>>
>>>>> Thanks Giri! Unfortunately though it seems to not have taken effect, I
>>>>> just kicked off a precommit build and I see
>>>>>
>>>>> core file size (blocks, -c) 0
>>>>> data seg size (kbytes, -d) unlimited
>>>>> scheduling priority (-e) 0
>>>>> file size (blocks, -f) unlimited
>>>>> pending signals (-i) 386178
>>>>> max locked memory (kbytes, -l) 64
>>>>> max memory size (kbytes, -m) unlimited
>>>>> open files (-n) 4096
>>>>> pipe size (512 bytes, -p) 8
>>>>> POSIX message queues (bytes, -q) 819200
>>>>> real-time priority (-r) 0
>>>>> stack size (kbytes, -s) 8192
>>>>> cpu time (seconds, -t) unlimited
>>>>> max user processes (-u) 386178
>>>>> virtual memory (kbytes, -v) unlimited
>>>>> file locks (-x) unlimited
>>>>>
>>>>> more details here:
>>>>>
>>>>> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2215/console
>>>>>
>>>>> Patrick
>>>>>
>>>>> On Tue, Jul 22, 2014 at 1:29 PM, Giridharan Kesavan
>>>>> <gk...@hortonworks.com> wrote:
>>>>> >
>>>>> >
>>>>> > jenkins@asf901:~$ ulimit -a
>>>>> > core file size (blocks, -c) 0
>>>>> > data seg size (kbytes, -d) unlimited
>>>>> > scheduling priority (-e) 0
>>>>> > file size (blocks, -f) unlimited
>>>>> > pending signals (-i) 386178
>>>>> > max locked memory (kbytes, -l) 64
>>>>> > max memory size (kbytes, -m) unlimited
>>>>> > open files (-n) 60000
>>>>> > pipe size (512 bytes, -p) 8
>>>>> > POSIX message queues (bytes, -q) 819200
>>>>> > real-time priority (-r) 0
>>>>> > stack size (kbytes, -s) 8192
>>>>> > cpu time (seconds, -t) unlimited
>>>>> > max user processes (-u) 10240
>>>>> > virtual memory (kbytes, -v) unlimited
>>>>> > file locks (-x) unlimited
>>>>> >
>>>>> > bumped up the open files and max user processes on all the slaves.
>>>>> >
>>>>> >
>>>>> >
>>>>> > -giri
>>>>> >
>>>>> >
>>>>> > On Tue, Jul 22, 2014 at 12:04 PM, Patrick Hunt <ph...@apache.org> wrote:
>>>>> >>
>>>>> >> Giri any chance can you take a look at the ulimit issue on the H#
>>>>> >> machines? All the ZK precommit builds are failing as a result.
>>>>> >>
>>>>> >> I updated the precommit build last night to output "ulimit -a" and it
>>>>> >> says the current limit is 4096, can we bump that up or set the default
>>>>> >> to unlimited?
>>>>> >>
>>>>> >> Thanks!
>>>>> >>
>>>>> >> Patrick
>>>>> >>
>>>>> >> On Sun, Jul 20, 2014 at 10:32 PM, Rakesh R <ra...@huawei.com> wrote:
>>>>> >> > +1
>>>>> >> >
>>>>> >> >
>>>>> >> > Adding one more point. I could see the following error too in the
>>>>> >> > pre-commit build.
>>>>> >> >
>>>>> >> > [exec] [junit] Exception in thread "CommitProcWorkThread-16"
>>>>> >> > java.lang.NoClassDefFoundError:
>>>>> >> > org/apache/zookeeper/server/ConnectionBean
>>>>> >> > [exec] [junit] at
>>>>> >> >
>>>>> >> > org.apache.zookeeper.server.ServerCnxnFactory.registerConnection(ServerCnxnFactory.java:159)
>>>>> >> > [exec] [junit] at
>>>>> >> >
>>>>> >> > org.apache.zookeeper.server.ZooKeeperServer.finishSessionInit(ZooKeeperServer.java:594)
>>>>> >> > [exec] [junit] at
>>>>> >> >
>>>>> >> > org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:198)
>>>>> >> > [exec] [junit] at
>>>>> >> >
>>>>> >> > org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:295)
>>>>> >> > [exec] [junit] at
>>>>> >> >
>>>>> >> > org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:161)
>>>>> >> > [exec] [junit] at
>>>>> >> >
>>>>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>>>> >> > [exec] [junit] at
>>>>> >> >
>>>>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>>>> >> > [exec] [junit] at
>>>>> >> > java.lang.Thread.run(Thread.java:662)
>>>>> >> >
>>>>> >> > -Rakesh
>>>>> >> >
>>>>> >> > -----Original Message-----
>>>>> >> > From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]
>>>>> >> > Sent: 21 July 2014 02:19
>>>>> >> > To: dev@zookeeper.apache.org
>>>>> >> > Cc: Andrew Bayer; builds@apache.org; Giridharan Kesavan
>>>>> >> > Subject: Re: ulimit changed with Apache Jenkins upgrade?
>>>>> >> >
>>>>> >> > +1
>>>>> >> >
>>>>> >> > On 18 Jul 2014, at 18:59, Patrick Hunt <ph...@apache.org> wrote:
>>>>> >> >
>>>>> >> >> Hi builds folks, is this a system wide issue or something we should
>>>>> >> >> address ourselves? Thanks!
>>>>> >> >>
>>>>> >> >> Patrick
>>>>> >> >>
>>>>> >> >> ---------- Forwarded message ----------
>>>>> >> >> From: Patrick Hunt <ph...@apache.org>
>>>>> >> >> Date: Fri, Jul 18, 2014 at 10:38 AM
>>>>> >> >> Subject: ulimit changed with Apache Jenkins upgrade?
>>>>> >> >> To: Giridharan Kesavan <gk...@hortonworks.com>
>>>>> >> >> Cc: DevZooKeeper <de...@zookeeper.apache.org>, Andrew Bayer
>>>>> >> >> <an...@cloudera.com>
>>>>> >> >>
>>>>> >> >>
>>>>> >> >> Hi Giri, can you check that the new hosts (H#) have the ulimit set
>>>>> >> >> to
>>>>> >> >> what it was set to on the original hadoop# hosts? I'm seeing new
>>>>> >> >> test
>>>>> >> >> failures with
>>>>> >> >>
>>>>> >> >> [exec] [junit] java.io.FileNotFoundException:
>>>>> >> >>
>>>>> >> >> /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/
>>>>> >> >>
>>>>> >> >> build/test/tmp/test7610638215300246179.junit.dir/version-2/log.1000000
>>>>> >> >> 01
>>>>> >> >> (Too many open files)
>>>>> >> >>
>>>>> >> >> which we've not seen before. I believe this means that we running
>>>>> >> >> out
>>>>> >> >> of file descriptors?
>>>>> >> >>
>>>>> >> >> Can you verify and address if possible?
>>>>> >> >>
>>>>> >> >> Thanks,
>>>>> >> >>
>>>>> >> >> Patrick
>>>>> >> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > CONFIDENTIALITY NOTICE
>>>>> > NOTICE: This message is intended for the use of the individual or entity
>>>>> > to
>>>>> > which it is addressed and may contain information that is confidential,
>>>>> > privileged and exempt from disclosure under applicable law. If the
>>>>> > reader of
>>>>> > this message is not the intended recipient, you are hereby notified that
>>>>> > any
>>>>> > printing, copying, dissemination, distribution, disclosure or forwarding
>>>>> > of
>>>>> > this communication is strictly prohibited. If you have received this
>>>>> > communication in error, please contact the sender immediately and delete
>>>>> > it
>>>>> > from your system. Thank You.
>>>>
>>>>
>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or entity to
>>>> which it is addressed and may contain information that is confidential,
>>>> privileged and exempt from disclosure under applicable law. If the reader of
>>>> this message is not the intended recipient, you are hereby notified that any
>>>> printing, copying, dissemination, distribution, disclosure or forwarding of
>>>> this communication is strictly prohibited. If you have received this
>>>> communication in error, please contact the sender immediately and delete it
>>>> from your system. Thank You.
Re: ulimit changed with Apache Jenkins upgrade?
Posted by Patrick Hunt <ph...@apache.org>.
I've opened a BUILDS jira for this:
https://issues.apache.org/jira/browse/BUILDS-17
On Sat, Aug 30, 2014 at 12:16 PM, Patrick Hunt <ph...@apache.org> wrote:
> Giri could you or one of the other folks look into this? Our tests
> have been broken for some time.
>
> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2303/
>
> The same test is still failing with the same error (too many open
> files) and afaict the ulimit is still just set to 4k. I'm printing the
> ulimit info at the start of the job, here it is:
>
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) unlimited
> pending signals (-i) 386177
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 4096
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 386177
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
>
>
> Patrick
>
> On Wed, Jul 23, 2014 at 11:21 AM, Patrick Hunt <ph...@apache.org> wrote:
>> Does someone in builds@ have the ability to address this? (restart the
>> jenkins slaves so that the new ulimit limits will take effect)
>>
>> Thanks!
>>
>> Patrick
>>
>> On Wed, Jul 23, 2014 at 9:33 AM, Patrick Hunt <ph...@apache.org> wrote:
>>> Giri do you mean me? I don't have access to that afaict.
>>>
>>> Patrick
>>>
>>> On Wed, Jul 23, 2014 at 12:50 AM, Giridharan Kesavan
>>> <gk...@hortonworks.com> wrote:
>>>> jenkins slaves might need a restart.
>>>>
>>>> Could you pls re-launch the slaves from the jenkins UI configuration page?
>>>>
>>>> -giri
>>>>
>>>>
>>>> On Tue, Jul 22, 2014 at 2:25 PM, Patrick Hunt <ph...@apache.org> wrote:
>>>>>
>>>>> Thanks Giri! Unfortunately though it seems to not have taken effect, I
>>>>> just kicked off a precommit build and I see
>>>>>
>>>>> core file size (blocks, -c) 0
>>>>> data seg size (kbytes, -d) unlimited
>>>>> scheduling priority (-e) 0
>>>>> file size (blocks, -f) unlimited
>>>>> pending signals (-i) 386178
>>>>> max locked memory (kbytes, -l) 64
>>>>> max memory size (kbytes, -m) unlimited
>>>>> open files (-n) 4096
>>>>> pipe size (512 bytes, -p) 8
>>>>> POSIX message queues (bytes, -q) 819200
>>>>> real-time priority (-r) 0
>>>>> stack size (kbytes, -s) 8192
>>>>> cpu time (seconds, -t) unlimited
>>>>> max user processes (-u) 386178
>>>>> virtual memory (kbytes, -v) unlimited
>>>>> file locks (-x) unlimited
>>>>>
>>>>> more details here:
>>>>>
>>>>> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2215/console
>>>>>
>>>>> Patrick
>>>>>
>>>>> On Tue, Jul 22, 2014 at 1:29 PM, Giridharan Kesavan
>>>>> <gk...@hortonworks.com> wrote:
>>>>> >
>>>>> >
>>>>> > jenkins@asf901:~$ ulimit -a
>>>>> > core file size (blocks, -c) 0
>>>>> > data seg size (kbytes, -d) unlimited
>>>>> > scheduling priority (-e) 0
>>>>> > file size (blocks, -f) unlimited
>>>>> > pending signals (-i) 386178
>>>>> > max locked memory (kbytes, -l) 64
>>>>> > max memory size (kbytes, -m) unlimited
>>>>> > open files (-n) 60000
>>>>> > pipe size (512 bytes, -p) 8
>>>>> > POSIX message queues (bytes, -q) 819200
>>>>> > real-time priority (-r) 0
>>>>> > stack size (kbytes, -s) 8192
>>>>> > cpu time (seconds, -t) unlimited
>>>>> > max user processes (-u) 10240
>>>>> > virtual memory (kbytes, -v) unlimited
>>>>> > file locks (-x) unlimited
>>>>> >
>>>>> > bumped up the open files and max user processes on all the slaves.
>>>>> >
>>>>> >
>>>>> >
>>>>> > -giri
>>>>> >
>>>>> >
>>>>> > On Tue, Jul 22, 2014 at 12:04 PM, Patrick Hunt <ph...@apache.org> wrote:
>>>>> >>
>>>>> >> Giri any chance can you take a look at the ulimit issue on the H#
>>>>> >> machines? All the ZK precommit builds are failing as a result.
>>>>> >>
>>>>> >> I updated the precommit build last night to output "ulimit -a" and it
>>>>> >> says the current limit is 4096, can we bump that up or set the default
>>>>> >> to unlimited?
>>>>> >>
>>>>> >> Thanks!
>>>>> >>
>>>>> >> Patrick
>>>>> >>
>>>>> >> On Sun, Jul 20, 2014 at 10:32 PM, Rakesh R <ra...@huawei.com> wrote:
>>>>> >> > +1
>>>>> >> >
>>>>> >> >
>>>>> >> > Adding one more point. I could see the following error too in the
>>>>> >> > pre-commit build.
>>>>> >> >
>>>>> >> > [exec] [junit] Exception in thread "CommitProcWorkThread-16"
>>>>> >> > java.lang.NoClassDefFoundError:
>>>>> >> > org/apache/zookeeper/server/ConnectionBean
>>>>> >> > [exec] [junit] at
>>>>> >> >
>>>>> >> > org.apache.zookeeper.server.ServerCnxnFactory.registerConnection(ServerCnxnFactory.java:159)
>>>>> >> > [exec] [junit] at
>>>>> >> >
>>>>> >> > org.apache.zookeeper.server.ZooKeeperServer.finishSessionInit(ZooKeeperServer.java:594)
>>>>> >> > [exec] [junit] at
>>>>> >> >
>>>>> >> > org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:198)
>>>>> >> > [exec] [junit] at
>>>>> >> >
>>>>> >> > org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:295)
>>>>> >> > [exec] [junit] at
>>>>> >> >
>>>>> >> > org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:161)
>>>>> >> > [exec] [junit] at
>>>>> >> >
>>>>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>>>> >> > [exec] [junit] at
>>>>> >> >
>>>>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>>>> >> > [exec] [junit] at
>>>>> >> > java.lang.Thread.run(Thread.java:662)
>>>>> >> >
>>>>> >> > -Rakesh
>>>>> >> >
>>>>> >> > -----Original Message-----
>>>>> >> > From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]
>>>>> >> > Sent: 21 July 2014 02:19
>>>>> >> > To: dev@zookeeper.apache.org
>>>>> >> > Cc: Andrew Bayer; builds@apache.org; Giridharan Kesavan
>>>>> >> > Subject: Re: ulimit changed with Apache Jenkins upgrade?
>>>>> >> >
>>>>> >> > +1
>>>>> >> >
>>>>> >> > On 18 Jul 2014, at 18:59, Patrick Hunt <ph...@apache.org> wrote:
>>>>> >> >
>>>>> >> >> Hi builds folks, is this a system wide issue or something we should
>>>>> >> >> address ourselves? Thanks!
>>>>> >> >>
>>>>> >> >> Patrick
>>>>> >> >>
>>>>> >> >> ---------- Forwarded message ----------
>>>>> >> >> From: Patrick Hunt <ph...@apache.org>
>>>>> >> >> Date: Fri, Jul 18, 2014 at 10:38 AM
>>>>> >> >> Subject: ulimit changed with Apache Jenkins upgrade?
>>>>> >> >> To: Giridharan Kesavan <gk...@hortonworks.com>
>>>>> >> >> Cc: DevZooKeeper <de...@zookeeper.apache.org>, Andrew Bayer
>>>>> >> >> <an...@cloudera.com>
>>>>> >> >>
>>>>> >> >>
>>>>> >> >> Hi Giri, can you check that the new hosts (H#) have the ulimit set
>>>>> >> >> to
>>>>> >> >> what it was set to on the original hadoop# hosts? I'm seeing new
>>>>> >> >> test
>>>>> >> >> failures with
>>>>> >> >>
>>>>> >> >> [exec] [junit] java.io.FileNotFoundException:
>>>>> >> >>
>>>>> >> >> /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/
>>>>> >> >>
>>>>> >> >> build/test/tmp/test7610638215300246179.junit.dir/version-2/log.1000000
>>>>> >> >> 01
>>>>> >> >> (Too many open files)
>>>>> >> >>
>>>>> >> >> which we've not seen before. I believe this means that we running
>>>>> >> >> out
>>>>> >> >> of file descriptors?
>>>>> >> >>
>>>>> >> >> Can you verify and address if possible?
>>>>> >> >>
>>>>> >> >> Thanks,
>>>>> >> >>
>>>>> >> >> Patrick
>>>>> >> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > CONFIDENTIALITY NOTICE
>>>>> > NOTICE: This message is intended for the use of the individual or entity
>>>>> > to
>>>>> > which it is addressed and may contain information that is confidential,
>>>>> > privileged and exempt from disclosure under applicable law. If the
>>>>> > reader of
>>>>> > this message is not the intended recipient, you are hereby notified that
>>>>> > any
>>>>> > printing, copying, dissemination, distribution, disclosure or forwarding
>>>>> > of
>>>>> > this communication is strictly prohibited. If you have received this
>>>>> > communication in error, please contact the sender immediately and delete
>>>>> > it
>>>>> > from your system. Thank You.
>>>>
>>>>
>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or entity to
>>>> which it is addressed and may contain information that is confidential,
>>>> privileged and exempt from disclosure under applicable law. If the reader of
>>>> this message is not the intended recipient, you are hereby notified that any
>>>> printing, copying, dissemination, distribution, disclosure or forwarding of
>>>> this communication is strictly prohibited. If you have received this
>>>> communication in error, please contact the sender immediately and delete it
>>>> from your system. Thank You.
Re: ulimit changed with Apache Jenkins upgrade?
Posted by Patrick Hunt <ph...@apache.org>.
Giri could you or one of the other folks look into this? Our tests
have been broken for some time.
https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2303/
The same test is still failing with the same error (too many open
files) and afaict the ulimit is still just set to 4k. I'm printing the
ulimit info at the start of the job, here it is:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 386177
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 4096
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 386177
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Patrick
On Wed, Jul 23, 2014 at 11:21 AM, Patrick Hunt <ph...@apache.org> wrote:
> Does someone in builds@ have the ability to address this? (restart the
> jenkins slaves so that the new ulimit limits will take effect)
>
> Thanks!
>
> Patrick
>
> On Wed, Jul 23, 2014 at 9:33 AM, Patrick Hunt <ph...@apache.org> wrote:
>> Giri do you mean me? I don't have access to that afaict.
>>
>> Patrick
>>
>> On Wed, Jul 23, 2014 at 12:50 AM, Giridharan Kesavan
>> <gk...@hortonworks.com> wrote:
>>> jenkins slaves might need a restart.
>>>
>>> Could you pls re-launch the slaves from the jenkins UI configuration page?
>>>
>>> -giri
>>>
>>>
>>> On Tue, Jul 22, 2014 at 2:25 PM, Patrick Hunt <ph...@apache.org> wrote:
>>>>
>>>> Thanks Giri! Unfortunately though it seems to not have taken effect, I
>>>> just kicked off a precommit build and I see
>>>>
>>>> core file size (blocks, -c) 0
>>>> data seg size (kbytes, -d) unlimited
>>>> scheduling priority (-e) 0
>>>> file size (blocks, -f) unlimited
>>>> pending signals (-i) 386178
>>>> max locked memory (kbytes, -l) 64
>>>> max memory size (kbytes, -m) unlimited
>>>> open files (-n) 4096
>>>> pipe size (512 bytes, -p) 8
>>>> POSIX message queues (bytes, -q) 819200
>>>> real-time priority (-r) 0
>>>> stack size (kbytes, -s) 8192
>>>> cpu time (seconds, -t) unlimited
>>>> max user processes (-u) 386178
>>>> virtual memory (kbytes, -v) unlimited
>>>> file locks (-x) unlimited
>>>>
>>>> more details here:
>>>>
>>>> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2215/console
>>>>
>>>> Patrick
>>>>
>>>> On Tue, Jul 22, 2014 at 1:29 PM, Giridharan Kesavan
>>>> <gk...@hortonworks.com> wrote:
>>>> >
>>>> >
>>>> > jenkins@asf901:~$ ulimit -a
>>>> > core file size (blocks, -c) 0
>>>> > data seg size (kbytes, -d) unlimited
>>>> > scheduling priority (-e) 0
>>>> > file size (blocks, -f) unlimited
>>>> > pending signals (-i) 386178
>>>> > max locked memory (kbytes, -l) 64
>>>> > max memory size (kbytes, -m) unlimited
>>>> > open files (-n) 60000
>>>> > pipe size (512 bytes, -p) 8
>>>> > POSIX message queues (bytes, -q) 819200
>>>> > real-time priority (-r) 0
>>>> > stack size (kbytes, -s) 8192
>>>> > cpu time (seconds, -t) unlimited
>>>> > max user processes (-u) 10240
>>>> > virtual memory (kbytes, -v) unlimited
>>>> > file locks (-x) unlimited
>>>> >
>>>> > bumped up the open files and max user processes on all the slaves.
>>>> >
>>>> >
>>>> >
>>>> > -giri
>>>> >
>>>> >
>>>> > On Tue, Jul 22, 2014 at 12:04 PM, Patrick Hunt <ph...@apache.org> wrote:
>>>> >>
>>>> >> Giri any chance can you take a look at the ulimit issue on the H#
>>>> >> machines? All the ZK precommit builds are failing as a result.
>>>> >>
>>>> >> I updated the precommit build last night to output "ulimit -a" and it
>>>> >> says the current limit is 4096, can we bump that up or set the default
>>>> >> to unlimited?
>>>> >>
>>>> >> Thanks!
>>>> >>
>>>> >> Patrick
>>>> >>
>>>> >> On Sun, Jul 20, 2014 at 10:32 PM, Rakesh R <ra...@huawei.com> wrote:
>>>> >> > +1
>>>> >> >
>>>> >> >
>>>> >> > Adding one more point. I could see the following error too in the
>>>> >> > pre-commit build.
>>>> >> >
>>>> >> > [exec] [junit] Exception in thread "CommitProcWorkThread-16"
>>>> >> > java.lang.NoClassDefFoundError:
>>>> >> > org/apache/zookeeper/server/ConnectionBean
>>>> >> > [exec] [junit] at
>>>> >> >
>>>> >> > org.apache.zookeeper.server.ServerCnxnFactory.registerConnection(ServerCnxnFactory.java:159)
>>>> >> > [exec] [junit] at
>>>> >> >
>>>> >> > org.apache.zookeeper.server.ZooKeeperServer.finishSessionInit(ZooKeeperServer.java:594)
>>>> >> > [exec] [junit] at
>>>> >> >
>>>> >> > org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:198)
>>>> >> > [exec] [junit] at
>>>> >> >
>>>> >> > org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:295)
>>>> >> > [exec] [junit] at
>>>> >> >
>>>> >> > org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:161)
>>>> >> > [exec] [junit] at
>>>> >> >
>>>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>>> >> > [exec] [junit] at
>>>> >> >
>>>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>>> >> > [exec] [junit] at
>>>> >> > java.lang.Thread.run(Thread.java:662)
>>>> >> >
>>>> >> > -Rakesh
>>>> >> >
>>>> >> > -----Original Message-----
>>>> >> > From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]
>>>> >> > Sent: 21 July 2014 02:19
>>>> >> > To: dev@zookeeper.apache.org
>>>> >> > Cc: Andrew Bayer; builds@apache.org; Giridharan Kesavan
>>>> >> > Subject: Re: ulimit changed with Apache Jenkins upgrade?
>>>> >> >
>>>> >> > +1
>>>> >> >
>>>> >> > On 18 Jul 2014, at 18:59, Patrick Hunt <ph...@apache.org> wrote:
>>>> >> >
>>>> >> >> Hi builds folks, is this a system wide issue or something we should
>>>> >> >> address ourselves? Thanks!
>>>> >> >>
>>>> >> >> Patrick
>>>> >> >>
>>>> >> >> ---------- Forwarded message ----------
>>>> >> >> From: Patrick Hunt <ph...@apache.org>
>>>> >> >> Date: Fri, Jul 18, 2014 at 10:38 AM
>>>> >> >> Subject: ulimit changed with Apache Jenkins upgrade?
>>>> >> >> To: Giridharan Kesavan <gk...@hortonworks.com>
>>>> >> >> Cc: DevZooKeeper <de...@zookeeper.apache.org>, Andrew Bayer
>>>> >> >> <an...@cloudera.com>
>>>> >> >>
>>>> >> >>
>>>> >> >> Hi Giri, can you check that the new hosts (H#) have the ulimit set
>>>> >> >> to
>>>> >> >> what it was set to on the original hadoop# hosts? I'm seeing new
>>>> >> >> test
>>>> >> >> failures with
>>>> >> >>
>>>> >> >> [exec] [junit] java.io.FileNotFoundException:
>>>> >> >>
>>>> >> >> /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/
>>>> >> >>
>>>> >> >> build/test/tmp/test7610638215300246179.junit.dir/version-2/log.1000000
>>>> >> >> 01
>>>> >> >> (Too many open files)
>>>> >> >>
>>>> >> >> which we've not seen before. I believe this means that we running
>>>> >> >> out
>>>> >> >> of file descriptors?
>>>> >> >>
>>>> >> >> Can you verify and address if possible?
>>>> >> >>
>>>> >> >> Thanks,
>>>> >> >>
>>>> >> >> Patrick
>>>> >> >
>>>> >
>>>> >
>>>> >
>>>> > CONFIDENTIALITY NOTICE
>>>> > NOTICE: This message is intended for the use of the individual or entity
>>>> > to
>>>> > which it is addressed and may contain information that is confidential,
>>>> > privileged and exempt from disclosure under applicable law. If the
>>>> > reader of
>>>> > this message is not the intended recipient, you are hereby notified that
>>>> > any
>>>> > printing, copying, dissemination, distribution, disclosure or forwarding
>>>> > of
>>>> > this communication is strictly prohibited. If you have received this
>>>> > communication in error, please contact the sender immediately and delete
>>>> > it
>>>> > from your system. Thank You.
>>>
>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader of
>>> this message is not the intended recipient, you are hereby notified that any
>>> printing, copying, dissemination, distribution, disclosure or forwarding of
>>> this communication is strictly prohibited. If you have received this
>>> communication in error, please contact the sender immediately and delete it
>>> from your system. Thank You.
Re: ulimit changed with Apache Jenkins upgrade?
Posted by Gavin McDonald <ga...@16degrees.com.au>.
This Bong Chen Lock who replys to every mail without saying anything is just about to get banned.
Gav…
On 03/09/2014, at 6:24 PM, Bong Chen Lock <bo...@gmail.com> wrote:
> On 24 Jul 2014 01:21, "Patrick Hunt" <ph...@apache.org> wrote:
>
>> Does someone in builds@ have the ability to address this? (restart the
>> jenkins slaves so that the new ulimit limits will take effect)
>>
>> Thanks!
>>
>> Patrick
>>
>> On Wed, Jul 23, 2014 at 9:33 AM, Patrick Hunt <ph...@apache.org> wrote:
>>> Giri do you mean me? I don't have access to that afaict.
>>>
>>> Patrick
>>>
>>> On Wed, Jul 23, 2014 at 12:50 AM, Giridharan Kesavan
>>> <gk...@hortonworks.com> wrote:
>>>> jenkins slaves might need a restart.
>>>>
>>>> Could you pls re-launch the slaves from the jenkins UI configuration
>> page?
>>>>
>>>> -giri
>>>>
>>>>
>>>> On Tue, Jul 22, 2014 at 2:25 PM, Patrick Hunt <ph...@apache.org> wrote:
>>>>>
>>>>> Thanks Giri! Unfortunately though it seems to not have taken effect, I
>>>>> just kicked off a precommit build and I see
>>>>>
>>>>> core file size (blocks, -c) 0
>>>>> data seg size (kbytes, -d) unlimited
>>>>> scheduling priority (-e) 0
>>>>> file size (blocks, -f) unlimited
>>>>> pending signals (-i) 386178
>>>>> max locked memory (kbytes, -l) 64
>>>>> max memory size (kbytes, -m) unlimited
>>>>> open files (-n) 4096
>>>>> pipe size (512 bytes, -p) 8
>>>>> POSIX message queues (bytes, -q) 819200
>>>>> real-time priority (-r) 0
>>>>> stack size (kbytes, -s) 8192
>>>>> cpu time (seconds, -t) unlimited
>>>>> max user processes (-u) 386178
>>>>> virtual memory (kbytes, -v) unlimited
>>>>> file locks (-x) unlimited
>>>>>
>>>>> more details here:
>>>>>
>>>>>
>> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2215/console
>>>>>
>>>>> Patrick
>>>>>
>>>>> On Tue, Jul 22, 2014 at 1:29 PM, Giridharan Kesavan
>>>>> <gk...@hortonworks.com> wrote:
>>>>>>
>>>>>>
>>>>>> jenkins@asf901:~$ ulimit -a
>>>>>> core file size (blocks, -c) 0
>>>>>> data seg size (kbytes, -d) unlimited
>>>>>> scheduling priority (-e) 0
>>>>>> file size (blocks, -f) unlimited
>>>>>> pending signals (-i) 386178
>>>>>> max locked memory (kbytes, -l) 64
>>>>>> max memory size (kbytes, -m) unlimited
>>>>>> open files (-n) 60000
>>>>>> pipe size (512 bytes, -p) 8
>>>>>> POSIX message queues (bytes, -q) 819200
>>>>>> real-time priority (-r) 0
>>>>>> stack size (kbytes, -s) 8192
>>>>>> cpu time (seconds, -t) unlimited
>>>>>> max user processes (-u) 10240
>>>>>> virtual memory (kbytes, -v) unlimited
>>>>>> file locks (-x) unlimited
>>>>>>
>>>>>> bumped up the open files and max user processes on all the slaves.
>>>>>>
>>>>>>
>>>>>>
>>>>>> -giri
>>>>>>
>>>>>>
>>>>>> On Tue, Jul 22, 2014 at 12:04 PM, Patrick Hunt <ph...@apache.org>
>> wrote:
>>>>>>>
>>>>>>> Giri any chance can you take a look at the ulimit issue on the H#
>>>>>>> machines? All the ZK precommit builds are failing as a result.
>>>>>>>
>>>>>>> I updated the precommit build last night to output "ulimit -a" and
>> it
>>>>>>> says the current limit is 4096, can we bump that up or set the
>> default
>>>>>>> to unlimited?
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> Patrick
>>>>>>>
>>>>>>> On Sun, Jul 20, 2014 at 10:32 PM, Rakesh R <ra...@huawei.com>
>> wrote:
>>>>>>>> +1
>>>>>>>>
>>>>>>>>
>>>>>>>> Adding one more point. I could see the following error too in the
>>>>>>>> pre-commit build.
>>>>>>>>
>>>>>>>> [exec] [junit] Exception in thread
>> "CommitProcWorkThread-16"
>>>>>>>> java.lang.NoClassDefFoundError:
>>>>>>>> org/apache/zookeeper/server/ConnectionBean
>>>>>>>> [exec] [junit] at
>>>>>>>>
>>>>>>>>
>> org.apache.zookeeper.server.ServerCnxnFactory.registerConnection(ServerCnxnFactory.java:159)
>>>>>>>> [exec] [junit] at
>>>>>>>>
>>>>>>>>
>> org.apache.zookeeper.server.ZooKeeperServer.finishSessionInit(ZooKeeperServer.java:594)
>>>>>>>> [exec] [junit] at
>>>>>>>>
>>>>>>>>
>> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:198)
>>>>>>>> [exec] [junit] at
>>>>>>>>
>>>>>>>>
>> org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:295)
>>>>>>>> [exec] [junit] at
>>>>>>>>
>>>>>>>>
>> org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:161)
>>>>>>>> [exec] [junit] at
>>>>>>>>
>>>>>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>>>>>>> [exec] [junit] at
>>>>>>>>
>>>>>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>>>>>>> [exec] [junit] at
>>>>>>>> java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> -Rakesh
>>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]
>>>>>>>> Sent: 21 July 2014 02:19
>>>>>>>> To: dev@zookeeper.apache.org
>>>>>>>> Cc: Andrew Bayer; builds@apache.org; Giridharan Kesavan
>>>>>>>> Subject: Re: ulimit changed with Apache Jenkins upgrade?
>>>>>>>>
>>>>>>>> +1
>>>>>>>>
>>>>>>>> On 18 Jul 2014, at 18:59, Patrick Hunt <ph...@apache.org> wrote:
>>>>>>>>
>>>>>>>>> Hi builds folks, is this a system wide issue or something we
>> should
>>>>>>>>> address ourselves? Thanks!
>>>>>>>>>
>>>>>>>>> Patrick
>>>>>>>>>
>>>>>>>>> ---------- Forwarded message ----------
>>>>>>>>> From: Patrick Hunt <ph...@apache.org>
>>>>>>>>> Date: Fri, Jul 18, 2014 at 10:38 AM
>>>>>>>>> Subject: ulimit changed with Apache Jenkins upgrade?
>>>>>>>>> To: Giridharan Kesavan <gk...@hortonworks.com>
>>>>>>>>> Cc: DevZooKeeper <de...@zookeeper.apache.org>, Andrew Bayer
>>>>>>>>> <an...@cloudera.com>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi Giri, can you check that the new hosts (H#) have the ulimit
>> set
>>>>>>>>> to
>>>>>>>>> what it was set to on the original hadoop# hosts? I'm seeing new
>>>>>>>>> test
>>>>>>>>> failures with
>>>>>>>>>
>>>>>>>>> [exec] [junit] java.io.FileNotFoundException:
>>>>>>>>>
>>>>>>>>>
>> /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/
>>>>>>>>>
>>>>>>>>>
>> build/test/tmp/test7610638215300246179.junit.dir/version-2/log.1000000
>>>>>>>>> 01
>>>>>>>>> (Too many open files)
>>>>>>>>>
>>>>>>>>> which we've not seen before. I believe this means that we running
>>>>>>>>> out
>>>>>>>>> of file descriptors?
>>>>>>>>>
>>>>>>>>> Can you verify and address if possible?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Patrick
>>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> CONFIDENTIALITY NOTICE
>>>>>> NOTICE: This message is intended for the use of the individual or
>> entity
>>>>>> to
>>>>>> which it is addressed and may contain information that is
>> confidential,
>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>>> reader of
>>>>>> this message is not the intended recipient, you are hereby notified
>> that
>>>>>> any
>>>>>> printing, copying, dissemination, distribution, disclosure or
>> forwarding
>>>>>> of
>>>>>> this communication is strictly prohibited. If you have received this
>>>>>> communication in error, please contact the sender immediately and
>> delete
>>>>>> it
>>>>>> from your system. Thank You.
>>>>
>>>>
>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>> entity to
>>>> which it is addressed and may contain information that is confidential,
>>>> privileged and exempt from disclosure under applicable law. If the
>> reader of
>>>> this message is not the intended recipient, you are hereby notified
>> that any
>>>> printing, copying, dissemination, distribution, disclosure or
>> forwarding of
>>>> this communication is strictly prohibited. If you have received this
>>>> communication in error, please contact the sender immediately and
>> delete it
>>>> from your system. Thank You.
>>
Re: ulimit changed with Apache Jenkins upgrade?
Posted by Bong Chen Lock <bo...@gmail.com>.
On 24 Jul 2014 01:21, "Patrick Hunt" <ph...@apache.org> wrote:
> Does someone in builds@ have the ability to address this? (restart the
> jenkins slaves so that the new ulimit limits will take effect)
>
> Thanks!
>
> Patrick
>
> On Wed, Jul 23, 2014 at 9:33 AM, Patrick Hunt <ph...@apache.org> wrote:
> > Giri do you mean me? I don't have access to that afaict.
> >
> > Patrick
> >
> > On Wed, Jul 23, 2014 at 12:50 AM, Giridharan Kesavan
> > <gk...@hortonworks.com> wrote:
> >> jenkins slaves might need a restart.
> >>
> >> Could you pls re-launch the slaves from the jenkins UI configuration
> page?
> >>
> >> -giri
> >>
> >>
> >> On Tue, Jul 22, 2014 at 2:25 PM, Patrick Hunt <ph...@apache.org> wrote:
> >>>
> >>> Thanks Giri! Unfortunately though it seems to not have taken effect, I
> >>> just kicked off a precommit build and I see
> >>>
> >>> core file size (blocks, -c) 0
> >>> data seg size (kbytes, -d) unlimited
> >>> scheduling priority (-e) 0
> >>> file size (blocks, -f) unlimited
> >>> pending signals (-i) 386178
> >>> max locked memory (kbytes, -l) 64
> >>> max memory size (kbytes, -m) unlimited
> >>> open files (-n) 4096
> >>> pipe size (512 bytes, -p) 8
> >>> POSIX message queues (bytes, -q) 819200
> >>> real-time priority (-r) 0
> >>> stack size (kbytes, -s) 8192
> >>> cpu time (seconds, -t) unlimited
> >>> max user processes (-u) 386178
> >>> virtual memory (kbytes, -v) unlimited
> >>> file locks (-x) unlimited
> >>>
> >>> more details here:
> >>>
> >>>
> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2215/console
> >>>
> >>> Patrick
> >>>
> >>> On Tue, Jul 22, 2014 at 1:29 PM, Giridharan Kesavan
> >>> <gk...@hortonworks.com> wrote:
> >>> >
> >>> >
> >>> > jenkins@asf901:~$ ulimit -a
> >>> > core file size (blocks, -c) 0
> >>> > data seg size (kbytes, -d) unlimited
> >>> > scheduling priority (-e) 0
> >>> > file size (blocks, -f) unlimited
> >>> > pending signals (-i) 386178
> >>> > max locked memory (kbytes, -l) 64
> >>> > max memory size (kbytes, -m) unlimited
> >>> > open files (-n) 60000
> >>> > pipe size (512 bytes, -p) 8
> >>> > POSIX message queues (bytes, -q) 819200
> >>> > real-time priority (-r) 0
> >>> > stack size (kbytes, -s) 8192
> >>> > cpu time (seconds, -t) unlimited
> >>> > max user processes (-u) 10240
> >>> > virtual memory (kbytes, -v) unlimited
> >>> > file locks (-x) unlimited
> >>> >
> >>> > bumped up the open files and max user processes on all the slaves.
> >>> >
> >>> >
> >>> >
> >>> > -giri
> >>> >
> >>> >
> >>> > On Tue, Jul 22, 2014 at 12:04 PM, Patrick Hunt <ph...@apache.org>
> wrote:
> >>> >>
> >>> >> Giri any chance can you take a look at the ulimit issue on the H#
> >>> >> machines? All the ZK precommit builds are failing as a result.
> >>> >>
> >>> >> I updated the precommit build last night to output "ulimit -a" and
> it
> >>> >> says the current limit is 4096, can we bump that up or set the
> default
> >>> >> to unlimited?
> >>> >>
> >>> >> Thanks!
> >>> >>
> >>> >> Patrick
> >>> >>
> >>> >> On Sun, Jul 20, 2014 at 10:32 PM, Rakesh R <ra...@huawei.com>
> wrote:
> >>> >> > +1
> >>> >> >
> >>> >> >
> >>> >> > Adding one more point. I could see the following error too in the
> >>> >> > pre-commit build.
> >>> >> >
> >>> >> > [exec] [junit] Exception in thread
> "CommitProcWorkThread-16"
> >>> >> > java.lang.NoClassDefFoundError:
> >>> >> > org/apache/zookeeper/server/ConnectionBean
> >>> >> > [exec] [junit] at
> >>> >> >
> >>> >> >
> org.apache.zookeeper.server.ServerCnxnFactory.registerConnection(ServerCnxnFactory.java:159)
> >>> >> > [exec] [junit] at
> >>> >> >
> >>> >> >
> org.apache.zookeeper.server.ZooKeeperServer.finishSessionInit(ZooKeeperServer.java:594)
> >>> >> > [exec] [junit] at
> >>> >> >
> >>> >> >
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:198)
> >>> >> > [exec] [junit] at
> >>> >> >
> >>> >> >
> org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:295)
> >>> >> > [exec] [junit] at
> >>> >> >
> >>> >> >
> org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:161)
> >>> >> > [exec] [junit] at
> >>> >> >
> >>> >> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> >>> >> > [exec] [junit] at
> >>> >> >
> >>> >> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> >>> >> > [exec] [junit] at
> >>> >> > java.lang.Thread.run(Thread.java:662)
> >>> >> >
> >>> >> > -Rakesh
> >>> >> >
> >>> >> > -----Original Message-----
> >>> >> > From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]
> >>> >> > Sent: 21 July 2014 02:19
> >>> >> > To: dev@zookeeper.apache.org
> >>> >> > Cc: Andrew Bayer; builds@apache.org; Giridharan Kesavan
> >>> >> > Subject: Re: ulimit changed with Apache Jenkins upgrade?
> >>> >> >
> >>> >> > +1
> >>> >> >
> >>> >> > On 18 Jul 2014, at 18:59, Patrick Hunt <ph...@apache.org> wrote:
> >>> >> >
> >>> >> >> Hi builds folks, is this a system wide issue or something we
> should
> >>> >> >> address ourselves? Thanks!
> >>> >> >>
> >>> >> >> Patrick
> >>> >> >>
> >>> >> >> ---------- Forwarded message ----------
> >>> >> >> From: Patrick Hunt <ph...@apache.org>
> >>> >> >> Date: Fri, Jul 18, 2014 at 10:38 AM
> >>> >> >> Subject: ulimit changed with Apache Jenkins upgrade?
> >>> >> >> To: Giridharan Kesavan <gk...@hortonworks.com>
> >>> >> >> Cc: DevZooKeeper <de...@zookeeper.apache.org>, Andrew Bayer
> >>> >> >> <an...@cloudera.com>
> >>> >> >>
> >>> >> >>
> >>> >> >> Hi Giri, can you check that the new hosts (H#) have the ulimit
> set
> >>> >> >> to
> >>> >> >> what it was set to on the original hadoop# hosts? I'm seeing new
> >>> >> >> test
> >>> >> >> failures with
> >>> >> >>
> >>> >> >> [exec] [junit] java.io.FileNotFoundException:
> >>> >> >>
> >>> >> >>
> /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/
> >>> >> >>
> >>> >> >>
> build/test/tmp/test7610638215300246179.junit.dir/version-2/log.1000000
> >>> >> >> 01
> >>> >> >> (Too many open files)
> >>> >> >>
> >>> >> >> which we've not seen before. I believe this means that we running
> >>> >> >> out
> >>> >> >> of file descriptors?
> >>> >> >>
> >>> >> >> Can you verify and address if possible?
> >>> >> >>
> >>> >> >> Thanks,
> >>> >> >>
> >>> >> >> Patrick
> >>> >> >
> >>> >
> >>> >
> >>> >
> >>> > CONFIDENTIALITY NOTICE
> >>> > NOTICE: This message is intended for the use of the individual or
> entity
> >>> > to
> >>> > which it is addressed and may contain information that is
> confidential,
> >>> > privileged and exempt from disclosure under applicable law. If the
> >>> > reader of
> >>> > this message is not the intended recipient, you are hereby notified
> that
> >>> > any
> >>> > printing, copying, dissemination, distribution, disclosure or
> forwarding
> >>> > of
> >>> > this communication is strictly prohibited. If you have received this
> >>> > communication in error, please contact the sender immediately and
> delete
> >>> > it
> >>> > from your system. Thank You.
> >>
> >>
> >>
> >> CONFIDENTIALITY NOTICE
> >> NOTICE: This message is intended for the use of the individual or
> entity to
> >> which it is addressed and may contain information that is confidential,
> >> privileged and exempt from disclosure under applicable law. If the
> reader of
> >> this message is not the intended recipient, you are hereby notified
> that any
> >> printing, copying, dissemination, distribution, disclosure or
> forwarding of
> >> this communication is strictly prohibited. If you have received this
> >> communication in error, please contact the sender immediately and
> delete it
> >> from your system. Thank You.
>
Re: ulimit changed with Apache Jenkins upgrade?
Posted by Patrick Hunt <ph...@apache.org>.
Giri could you or one of the other folks look into this? Our tests
have been broken for some time.
https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2303/
The same test is still failing with the same error (too many open
files) and afaict the ulimit is still just set to 4k. I'm printing the
ulimit info at the start of the job, here it is:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 386177
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 4096
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 386177
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Patrick
On Wed, Jul 23, 2014 at 11:21 AM, Patrick Hunt <ph...@apache.org> wrote:
> Does someone in builds@ have the ability to address this? (restart the
> jenkins slaves so that the new ulimit limits will take effect)
>
> Thanks!
>
> Patrick
>
> On Wed, Jul 23, 2014 at 9:33 AM, Patrick Hunt <ph...@apache.org> wrote:
>> Giri do you mean me? I don't have access to that afaict.
>>
>> Patrick
>>
>> On Wed, Jul 23, 2014 at 12:50 AM, Giridharan Kesavan
>> <gk...@hortonworks.com> wrote:
>>> jenkins slaves might need a restart.
>>>
>>> Could you pls re-launch the slaves from the jenkins UI configuration page?
>>>
>>> -giri
>>>
>>>
>>> On Tue, Jul 22, 2014 at 2:25 PM, Patrick Hunt <ph...@apache.org> wrote:
>>>>
>>>> Thanks Giri! Unfortunately though it seems to not have taken effect, I
>>>> just kicked off a precommit build and I see
>>>>
>>>> core file size (blocks, -c) 0
>>>> data seg size (kbytes, -d) unlimited
>>>> scheduling priority (-e) 0
>>>> file size (blocks, -f) unlimited
>>>> pending signals (-i) 386178
>>>> max locked memory (kbytes, -l) 64
>>>> max memory size (kbytes, -m) unlimited
>>>> open files (-n) 4096
>>>> pipe size (512 bytes, -p) 8
>>>> POSIX message queues (bytes, -q) 819200
>>>> real-time priority (-r) 0
>>>> stack size (kbytes, -s) 8192
>>>> cpu time (seconds, -t) unlimited
>>>> max user processes (-u) 386178
>>>> virtual memory (kbytes, -v) unlimited
>>>> file locks (-x) unlimited
>>>>
>>>> more details here:
>>>>
>>>> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2215/console
>>>>
>>>> Patrick
>>>>
>>>> On Tue, Jul 22, 2014 at 1:29 PM, Giridharan Kesavan
>>>> <gk...@hortonworks.com> wrote:
>>>> >
>>>> >
>>>> > jenkins@asf901:~$ ulimit -a
>>>> > core file size (blocks, -c) 0
>>>> > data seg size (kbytes, -d) unlimited
>>>> > scheduling priority (-e) 0
>>>> > file size (blocks, -f) unlimited
>>>> > pending signals (-i) 386178
>>>> > max locked memory (kbytes, -l) 64
>>>> > max memory size (kbytes, -m) unlimited
>>>> > open files (-n) 60000
>>>> > pipe size (512 bytes, -p) 8
>>>> > POSIX message queues (bytes, -q) 819200
>>>> > real-time priority (-r) 0
>>>> > stack size (kbytes, -s) 8192
>>>> > cpu time (seconds, -t) unlimited
>>>> > max user processes (-u) 10240
>>>> > virtual memory (kbytes, -v) unlimited
>>>> > file locks (-x) unlimited
>>>> >
>>>> > bumped up the open files and max user processes on all the slaves.
>>>> >
>>>> >
>>>> >
>>>> > -giri
>>>> >
>>>> >
>>>> > On Tue, Jul 22, 2014 at 12:04 PM, Patrick Hunt <ph...@apache.org> wrote:
>>>> >>
>>>> >> Giri any chance can you take a look at the ulimit issue on the H#
>>>> >> machines? All the ZK precommit builds are failing as a result.
>>>> >>
>>>> >> I updated the precommit build last night to output "ulimit -a" and it
>>>> >> says the current limit is 4096, can we bump that up or set the default
>>>> >> to unlimited?
>>>> >>
>>>> >> Thanks!
>>>> >>
>>>> >> Patrick
>>>> >>
>>>> >> On Sun, Jul 20, 2014 at 10:32 PM, Rakesh R <ra...@huawei.com> wrote:
>>>> >> > +1
>>>> >> >
>>>> >> >
>>>> >> > Adding one more point. I could see the following error too in the
>>>> >> > pre-commit build.
>>>> >> >
>>>> >> > [exec] [junit] Exception in thread "CommitProcWorkThread-16"
>>>> >> > java.lang.NoClassDefFoundError:
>>>> >> > org/apache/zookeeper/server/ConnectionBean
>>>> >> > [exec] [junit] at
>>>> >> >
>>>> >> > org.apache.zookeeper.server.ServerCnxnFactory.registerConnection(ServerCnxnFactory.java:159)
>>>> >> > [exec] [junit] at
>>>> >> >
>>>> >> > org.apache.zookeeper.server.ZooKeeperServer.finishSessionInit(ZooKeeperServer.java:594)
>>>> >> > [exec] [junit] at
>>>> >> >
>>>> >> > org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:198)
>>>> >> > [exec] [junit] at
>>>> >> >
>>>> >> > org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:295)
>>>> >> > [exec] [junit] at
>>>> >> >
>>>> >> > org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:161)
>>>> >> > [exec] [junit] at
>>>> >> >
>>>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>>> >> > [exec] [junit] at
>>>> >> >
>>>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>>> >> > [exec] [junit] at
>>>> >> > java.lang.Thread.run(Thread.java:662)
>>>> >> >
>>>> >> > -Rakesh
>>>> >> >
>>>> >> > -----Original Message-----
>>>> >> > From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]
>>>> >> > Sent: 21 July 2014 02:19
>>>> >> > To: dev@zookeeper.apache.org
>>>> >> > Cc: Andrew Bayer; builds@apache.org; Giridharan Kesavan
>>>> >> > Subject: Re: ulimit changed with Apache Jenkins upgrade?
>>>> >> >
>>>> >> > +1
>>>> >> >
>>>> >> > On 18 Jul 2014, at 18:59, Patrick Hunt <ph...@apache.org> wrote:
>>>> >> >
>>>> >> >> Hi builds folks, is this a system wide issue or something we should
>>>> >> >> address ourselves? Thanks!
>>>> >> >>
>>>> >> >> Patrick
>>>> >> >>
>>>> >> >> ---------- Forwarded message ----------
>>>> >> >> From: Patrick Hunt <ph...@apache.org>
>>>> >> >> Date: Fri, Jul 18, 2014 at 10:38 AM
>>>> >> >> Subject: ulimit changed with Apache Jenkins upgrade?
>>>> >> >> To: Giridharan Kesavan <gk...@hortonworks.com>
>>>> >> >> Cc: DevZooKeeper <de...@zookeeper.apache.org>, Andrew Bayer
>>>> >> >> <an...@cloudera.com>
>>>> >> >>
>>>> >> >>
>>>> >> >> Hi Giri, can you check that the new hosts (H#) have the ulimit set
>>>> >> >> to
>>>> >> >> what it was set to on the original hadoop# hosts? I'm seeing new
>>>> >> >> test
>>>> >> >> failures with
>>>> >> >>
>>>> >> >> [exec] [junit] java.io.FileNotFoundException:
>>>> >> >>
>>>> >> >> /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/
>>>> >> >>
>>>> >> >> build/test/tmp/test7610638215300246179.junit.dir/version-2/log.1000000
>>>> >> >> 01
>>>> >> >> (Too many open files)
>>>> >> >>
>>>> >> >> which we've not seen before. I believe this means that we running
>>>> >> >> out
>>>> >> >> of file descriptors?
>>>> >> >>
>>>> >> >> Can you verify and address if possible?
>>>> >> >>
>>>> >> >> Thanks,
>>>> >> >>
>>>> >> >> Patrick
>>>> >> >
>>>> >
>>>> >
>>>> >
>>>> > CONFIDENTIALITY NOTICE
>>>> > NOTICE: This message is intended for the use of the individual or entity
>>>> > to
>>>> > which it is addressed and may contain information that is confidential,
>>>> > privileged and exempt from disclosure under applicable law. If the
>>>> > reader of
>>>> > this message is not the intended recipient, you are hereby notified that
>>>> > any
>>>> > printing, copying, dissemination, distribution, disclosure or forwarding
>>>> > of
>>>> > this communication is strictly prohibited. If you have received this
>>>> > communication in error, please contact the sender immediately and delete
>>>> > it
>>>> > from your system. Thank You.
>>>
>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader of
>>> this message is not the intended recipient, you are hereby notified that any
>>> printing, copying, dissemination, distribution, disclosure or forwarding of
>>> this communication is strictly prohibited. If you have received this
>>> communication in error, please contact the sender immediately and delete it
>>> from your system. Thank You.
Re: ulimit changed with Apache Jenkins upgrade?
Posted by Patrick Hunt <ph...@apache.org>.
Does someone in builds@ have the ability to address this? (restart the
jenkins slaves so that the new ulimit limits will take effect)
Thanks!
Patrick
On Wed, Jul 23, 2014 at 9:33 AM, Patrick Hunt <ph...@apache.org> wrote:
> Giri do you mean me? I don't have access to that afaict.
>
> Patrick
>
> On Wed, Jul 23, 2014 at 12:50 AM, Giridharan Kesavan
> <gk...@hortonworks.com> wrote:
>> jenkins slaves might need a restart.
>>
>> Could you pls re-launch the slaves from the jenkins UI configuration page?
>>
>> -giri
>>
>>
>> On Tue, Jul 22, 2014 at 2:25 PM, Patrick Hunt <ph...@apache.org> wrote:
>>>
>>> Thanks Giri! Unfortunately though it seems to not have taken effect, I
>>> just kicked off a precommit build and I see
>>>
>>> core file size (blocks, -c) 0
>>> data seg size (kbytes, -d) unlimited
>>> scheduling priority (-e) 0
>>> file size (blocks, -f) unlimited
>>> pending signals (-i) 386178
>>> max locked memory (kbytes, -l) 64
>>> max memory size (kbytes, -m) unlimited
>>> open files (-n) 4096
>>> pipe size (512 bytes, -p) 8
>>> POSIX message queues (bytes, -q) 819200
>>> real-time priority (-r) 0
>>> stack size (kbytes, -s) 8192
>>> cpu time (seconds, -t) unlimited
>>> max user processes (-u) 386178
>>> virtual memory (kbytes, -v) unlimited
>>> file locks (-x) unlimited
>>>
>>> more details here:
>>>
>>> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2215/console
>>>
>>> Patrick
>>>
>>> On Tue, Jul 22, 2014 at 1:29 PM, Giridharan Kesavan
>>> <gk...@hortonworks.com> wrote:
>>> >
>>> >
>>> > jenkins@asf901:~$ ulimit -a
>>> > core file size (blocks, -c) 0
>>> > data seg size (kbytes, -d) unlimited
>>> > scheduling priority (-e) 0
>>> > file size (blocks, -f) unlimited
>>> > pending signals (-i) 386178
>>> > max locked memory (kbytes, -l) 64
>>> > max memory size (kbytes, -m) unlimited
>>> > open files (-n) 60000
>>> > pipe size (512 bytes, -p) 8
>>> > POSIX message queues (bytes, -q) 819200
>>> > real-time priority (-r) 0
>>> > stack size (kbytes, -s) 8192
>>> > cpu time (seconds, -t) unlimited
>>> > max user processes (-u) 10240
>>> > virtual memory (kbytes, -v) unlimited
>>> > file locks (-x) unlimited
>>> >
>>> > bumped up the open files and max user processes on all the slaves.
>>> >
>>> >
>>> >
>>> > -giri
>>> >
>>> >
>>> > On Tue, Jul 22, 2014 at 12:04 PM, Patrick Hunt <ph...@apache.org> wrote:
>>> >>
>>> >> Giri any chance can you take a look at the ulimit issue on the H#
>>> >> machines? All the ZK precommit builds are failing as a result.
>>> >>
>>> >> I updated the precommit build last night to output "ulimit -a" and it
>>> >> says the current limit is 4096, can we bump that up or set the default
>>> >> to unlimited?
>>> >>
>>> >> Thanks!
>>> >>
>>> >> Patrick
>>> >>
>>> >> On Sun, Jul 20, 2014 at 10:32 PM, Rakesh R <ra...@huawei.com> wrote:
>>> >> > +1
>>> >> >
>>> >> >
>>> >> > Adding one more point. I could see the following error too in the
>>> >> > pre-commit build.
>>> >> >
>>> >> > [exec] [junit] Exception in thread "CommitProcWorkThread-16"
>>> >> > java.lang.NoClassDefFoundError:
>>> >> > org/apache/zookeeper/server/ConnectionBean
>>> >> > [exec] [junit] at
>>> >> >
>>> >> > org.apache.zookeeper.server.ServerCnxnFactory.registerConnection(ServerCnxnFactory.java:159)
>>> >> > [exec] [junit] at
>>> >> >
>>> >> > org.apache.zookeeper.server.ZooKeeperServer.finishSessionInit(ZooKeeperServer.java:594)
>>> >> > [exec] [junit] at
>>> >> >
>>> >> > org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:198)
>>> >> > [exec] [junit] at
>>> >> >
>>> >> > org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:295)
>>> >> > [exec] [junit] at
>>> >> >
>>> >> > org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:161)
>>> >> > [exec] [junit] at
>>> >> >
>>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>> >> > [exec] [junit] at
>>> >> >
>>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>> >> > [exec] [junit] at
>>> >> > java.lang.Thread.run(Thread.java:662)
>>> >> >
>>> >> > -Rakesh
>>> >> >
>>> >> > -----Original Message-----
>>> >> > From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]
>>> >> > Sent: 21 July 2014 02:19
>>> >> > To: dev@zookeeper.apache.org
>>> >> > Cc: Andrew Bayer; builds@apache.org; Giridharan Kesavan
>>> >> > Subject: Re: ulimit changed with Apache Jenkins upgrade?
>>> >> >
>>> >> > +1
>>> >> >
>>> >> > On 18 Jul 2014, at 18:59, Patrick Hunt <ph...@apache.org> wrote:
>>> >> >
>>> >> >> Hi builds folks, is this a system wide issue or something we should
>>> >> >> address ourselves? Thanks!
>>> >> >>
>>> >> >> Patrick
>>> >> >>
>>> >> >> ---------- Forwarded message ----------
>>> >> >> From: Patrick Hunt <ph...@apache.org>
>>> >> >> Date: Fri, Jul 18, 2014 at 10:38 AM
>>> >> >> Subject: ulimit changed with Apache Jenkins upgrade?
>>> >> >> To: Giridharan Kesavan <gk...@hortonworks.com>
>>> >> >> Cc: DevZooKeeper <de...@zookeeper.apache.org>, Andrew Bayer
>>> >> >> <an...@cloudera.com>
>>> >> >>
>>> >> >>
>>> >> >> Hi Giri, can you check that the new hosts (H#) have the ulimit set
>>> >> >> to
>>> >> >> what it was set to on the original hadoop# hosts? I'm seeing new
>>> >> >> test
>>> >> >> failures with
>>> >> >>
>>> >> >> [exec] [junit] java.io.FileNotFoundException:
>>> >> >>
>>> >> >> /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/
>>> >> >>
>>> >> >> build/test/tmp/test7610638215300246179.junit.dir/version-2/log.1000000
>>> >> >> 01
>>> >> >> (Too many open files)
>>> >> >>
>>> >> >> which we've not seen before. I believe this means that we running
>>> >> >> out
>>> >> >> of file descriptors?
>>> >> >>
>>> >> >> Can you verify and address if possible?
>>> >> >>
>>> >> >> Thanks,
>>> >> >>
>>> >> >> Patrick
>>> >> >
>>> >
>>> >
>>> >
>>> > CONFIDENTIALITY NOTICE
>>> > NOTICE: This message is intended for the use of the individual or entity
>>> > to
>>> > which it is addressed and may contain information that is confidential,
>>> > privileged and exempt from disclosure under applicable law. If the
>>> > reader of
>>> > this message is not the intended recipient, you are hereby notified that
>>> > any
>>> > printing, copying, dissemination, distribution, disclosure or forwarding
>>> > of
>>> > this communication is strictly prohibited. If you have received this
>>> > communication in error, please contact the sender immediately and delete
>>> > it
>>> > from your system. Thank You.
>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader of
>> this message is not the intended recipient, you are hereby notified that any
>> printing, copying, dissemination, distribution, disclosure or forwarding of
>> this communication is strictly prohibited. If you have received this
>> communication in error, please contact the sender immediately and delete it
>> from your system. Thank You.
Re: ulimit changed with Apache Jenkins upgrade?
Posted by Patrick Hunt <ph...@apache.org>.
Does someone in builds@ have the ability to address this? (restart the
jenkins slaves so that the new ulimit limits will take effect)
Thanks!
Patrick
On Wed, Jul 23, 2014 at 9:33 AM, Patrick Hunt <ph...@apache.org> wrote:
> Giri do you mean me? I don't have access to that afaict.
>
> Patrick
>
> On Wed, Jul 23, 2014 at 12:50 AM, Giridharan Kesavan
> <gk...@hortonworks.com> wrote:
>> jenkins slaves might need a restart.
>>
>> Could you pls re-launch the slaves from the jenkins UI configuration page?
>>
>> -giri
>>
>>
>> On Tue, Jul 22, 2014 at 2:25 PM, Patrick Hunt <ph...@apache.org> wrote:
>>>
>>> Thanks Giri! Unfortunately though it seems to not have taken effect, I
>>> just kicked off a precommit build and I see
>>>
>>> core file size (blocks, -c) 0
>>> data seg size (kbytes, -d) unlimited
>>> scheduling priority (-e) 0
>>> file size (blocks, -f) unlimited
>>> pending signals (-i) 386178
>>> max locked memory (kbytes, -l) 64
>>> max memory size (kbytes, -m) unlimited
>>> open files (-n) 4096
>>> pipe size (512 bytes, -p) 8
>>> POSIX message queues (bytes, -q) 819200
>>> real-time priority (-r) 0
>>> stack size (kbytes, -s) 8192
>>> cpu time (seconds, -t) unlimited
>>> max user processes (-u) 386178
>>> virtual memory (kbytes, -v) unlimited
>>> file locks (-x) unlimited
>>>
>>> more details here:
>>>
>>> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2215/console
>>>
>>> Patrick
>>>
>>> On Tue, Jul 22, 2014 at 1:29 PM, Giridharan Kesavan
>>> <gk...@hortonworks.com> wrote:
>>> >
>>> >
>>> > jenkins@asf901:~$ ulimit -a
>>> > core file size (blocks, -c) 0
>>> > data seg size (kbytes, -d) unlimited
>>> > scheduling priority (-e) 0
>>> > file size (blocks, -f) unlimited
>>> > pending signals (-i) 386178
>>> > max locked memory (kbytes, -l) 64
>>> > max memory size (kbytes, -m) unlimited
>>> > open files (-n) 60000
>>> > pipe size (512 bytes, -p) 8
>>> > POSIX message queues (bytes, -q) 819200
>>> > real-time priority (-r) 0
>>> > stack size (kbytes, -s) 8192
>>> > cpu time (seconds, -t) unlimited
>>> > max user processes (-u) 10240
>>> > virtual memory (kbytes, -v) unlimited
>>> > file locks (-x) unlimited
>>> >
>>> > bumped up the open files and max user processes on all the slaves.
>>> >
>>> >
>>> >
>>> > -giri
>>> >
>>> >
>>> > On Tue, Jul 22, 2014 at 12:04 PM, Patrick Hunt <ph...@apache.org> wrote:
>>> >>
>>> >> Giri any chance can you take a look at the ulimit issue on the H#
>>> >> machines? All the ZK precommit builds are failing as a result.
>>> >>
>>> >> I updated the precommit build last night to output "ulimit -a" and it
>>> >> says the current limit is 4096, can we bump that up or set the default
>>> >> to unlimited?
>>> >>
>>> >> Thanks!
>>> >>
>>> >> Patrick
>>> >>
>>> >> On Sun, Jul 20, 2014 at 10:32 PM, Rakesh R <ra...@huawei.com> wrote:
>>> >> > +1
>>> >> >
>>> >> >
>>> >> > Adding one more point. I could see the following error too in the
>>> >> > pre-commit build.
>>> >> >
>>> >> > [exec] [junit] Exception in thread "CommitProcWorkThread-16"
>>> >> > java.lang.NoClassDefFoundError:
>>> >> > org/apache/zookeeper/server/ConnectionBean
>>> >> > [exec] [junit] at
>>> >> >
>>> >> > org.apache.zookeeper.server.ServerCnxnFactory.registerConnection(ServerCnxnFactory.java:159)
>>> >> > [exec] [junit] at
>>> >> >
>>> >> > org.apache.zookeeper.server.ZooKeeperServer.finishSessionInit(ZooKeeperServer.java:594)
>>> >> > [exec] [junit] at
>>> >> >
>>> >> > org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:198)
>>> >> > [exec] [junit] at
>>> >> >
>>> >> > org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:295)
>>> >> > [exec] [junit] at
>>> >> >
>>> >> > org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:161)
>>> >> > [exec] [junit] at
>>> >> >
>>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>> >> > [exec] [junit] at
>>> >> >
>>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>> >> > [exec] [junit] at
>>> >> > java.lang.Thread.run(Thread.java:662)
>>> >> >
>>> >> > -Rakesh
>>> >> >
>>> >> > -----Original Message-----
>>> >> > From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]
>>> >> > Sent: 21 July 2014 02:19
>>> >> > To: dev@zookeeper.apache.org
>>> >> > Cc: Andrew Bayer; builds@apache.org; Giridharan Kesavan
>>> >> > Subject: Re: ulimit changed with Apache Jenkins upgrade?
>>> >> >
>>> >> > +1
>>> >> >
>>> >> > On 18 Jul 2014, at 18:59, Patrick Hunt <ph...@apache.org> wrote:
>>> >> >
>>> >> >> Hi builds folks, is this a system wide issue or something we should
>>> >> >> address ourselves? Thanks!
>>> >> >>
>>> >> >> Patrick
>>> >> >>
>>> >> >> ---------- Forwarded message ----------
>>> >> >> From: Patrick Hunt <ph...@apache.org>
>>> >> >> Date: Fri, Jul 18, 2014 at 10:38 AM
>>> >> >> Subject: ulimit changed with Apache Jenkins upgrade?
>>> >> >> To: Giridharan Kesavan <gk...@hortonworks.com>
>>> >> >> Cc: DevZooKeeper <de...@zookeeper.apache.org>, Andrew Bayer
>>> >> >> <an...@cloudera.com>
>>> >> >>
>>> >> >>
>>> >> >> Hi Giri, can you check that the new hosts (H#) have the ulimit set
>>> >> >> to
>>> >> >> what it was set to on the original hadoop# hosts? I'm seeing new
>>> >> >> test
>>> >> >> failures with
>>> >> >>
>>> >> >> [exec] [junit] java.io.FileNotFoundException:
>>> >> >>
>>> >> >> /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/
>>> >> >>
>>> >> >> build/test/tmp/test7610638215300246179.junit.dir/version-2/log.1000000
>>> >> >> 01
>>> >> >> (Too many open files)
>>> >> >>
>>> >> >> which we've not seen before. I believe this means that we running
>>> >> >> out
>>> >> >> of file descriptors?
>>> >> >>
>>> >> >> Can you verify and address if possible?
>>> >> >>
>>> >> >> Thanks,
>>> >> >>
>>> >> >> Patrick
>>> >> >
>>> >
>>> >
>>> >
>>> > CONFIDENTIALITY NOTICE
>>> > NOTICE: This message is intended for the use of the individual or entity
>>> > to
>>> > which it is addressed and may contain information that is confidential,
>>> > privileged and exempt from disclosure under applicable law. If the
>>> > reader of
>>> > this message is not the intended recipient, you are hereby notified that
>>> > any
>>> > printing, copying, dissemination, distribution, disclosure or forwarding
>>> > of
>>> > this communication is strictly prohibited. If you have received this
>>> > communication in error, please contact the sender immediately and delete
>>> > it
>>> > from your system. Thank You.
>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader of
>> this message is not the intended recipient, you are hereby notified that any
>> printing, copying, dissemination, distribution, disclosure or forwarding of
>> this communication is strictly prohibited. If you have received this
>> communication in error, please contact the sender immediately and delete it
>> from your system. Thank You.
Re: ulimit changed with Apache Jenkins upgrade?
Posted by Patrick Hunt <ph...@apache.org>.
Giri do you mean me? I don't have access to that afaict.
Patrick
On Wed, Jul 23, 2014 at 12:50 AM, Giridharan Kesavan
<gk...@hortonworks.com> wrote:
> jenkins slaves might need a restart.
>
> Could you pls re-launch the slaves from the jenkins UI configuration page?
>
> -giri
>
>
> On Tue, Jul 22, 2014 at 2:25 PM, Patrick Hunt <ph...@apache.org> wrote:
>>
>> Thanks Giri! Unfortunately though it seems to not have taken effect, I
>> just kicked off a precommit build and I see
>>
>> core file size (blocks, -c) 0
>> data seg size (kbytes, -d) unlimited
>> scheduling priority (-e) 0
>> file size (blocks, -f) unlimited
>> pending signals (-i) 386178
>> max locked memory (kbytes, -l) 64
>> max memory size (kbytes, -m) unlimited
>> open files (-n) 4096
>> pipe size (512 bytes, -p) 8
>> POSIX message queues (bytes, -q) 819200
>> real-time priority (-r) 0
>> stack size (kbytes, -s) 8192
>> cpu time (seconds, -t) unlimited
>> max user processes (-u) 386178
>> virtual memory (kbytes, -v) unlimited
>> file locks (-x) unlimited
>>
>> more details here:
>>
>> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2215/console
>>
>> Patrick
>>
>> On Tue, Jul 22, 2014 at 1:29 PM, Giridharan Kesavan
>> <gk...@hortonworks.com> wrote:
>> >
>> >
>> > jenkins@asf901:~$ ulimit -a
>> > core file size (blocks, -c) 0
>> > data seg size (kbytes, -d) unlimited
>> > scheduling priority (-e) 0
>> > file size (blocks, -f) unlimited
>> > pending signals (-i) 386178
>> > max locked memory (kbytes, -l) 64
>> > max memory size (kbytes, -m) unlimited
>> > open files (-n) 60000
>> > pipe size (512 bytes, -p) 8
>> > POSIX message queues (bytes, -q) 819200
>> > real-time priority (-r) 0
>> > stack size (kbytes, -s) 8192
>> > cpu time (seconds, -t) unlimited
>> > max user processes (-u) 10240
>> > virtual memory (kbytes, -v) unlimited
>> > file locks (-x) unlimited
>> >
>> > bumped up the open files and max user processes on all the slaves.
>> >
>> >
>> >
>> > -giri
>> >
>> >
>> > On Tue, Jul 22, 2014 at 12:04 PM, Patrick Hunt <ph...@apache.org> wrote:
>> >>
>> >> Giri any chance can you take a look at the ulimit issue on the H#
>> >> machines? All the ZK precommit builds are failing as a result.
>> >>
>> >> I updated the precommit build last night to output "ulimit -a" and it
>> >> says the current limit is 4096, can we bump that up or set the default
>> >> to unlimited?
>> >>
>> >> Thanks!
>> >>
>> >> Patrick
>> >>
>> >> On Sun, Jul 20, 2014 at 10:32 PM, Rakesh R <ra...@huawei.com> wrote:
>> >> > +1
>> >> >
>> >> >
>> >> > Adding one more point. I could see the following error too in the
>> >> > pre-commit build.
>> >> >
>> >> > [exec] [junit] Exception in thread "CommitProcWorkThread-16"
>> >> > java.lang.NoClassDefFoundError:
>> >> > org/apache/zookeeper/server/ConnectionBean
>> >> > [exec] [junit] at
>> >> >
>> >> > org.apache.zookeeper.server.ServerCnxnFactory.registerConnection(ServerCnxnFactory.java:159)
>> >> > [exec] [junit] at
>> >> >
>> >> > org.apache.zookeeper.server.ZooKeeperServer.finishSessionInit(ZooKeeperServer.java:594)
>> >> > [exec] [junit] at
>> >> >
>> >> > org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:198)
>> >> > [exec] [junit] at
>> >> >
>> >> > org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:295)
>> >> > [exec] [junit] at
>> >> >
>> >> > org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:161)
>> >> > [exec] [junit] at
>> >> >
>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>> >> > [exec] [junit] at
>> >> >
>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>> >> > [exec] [junit] at
>> >> > java.lang.Thread.run(Thread.java:662)
>> >> >
>> >> > -Rakesh
>> >> >
>> >> > -----Original Message-----
>> >> > From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]
>> >> > Sent: 21 July 2014 02:19
>> >> > To: dev@zookeeper.apache.org
>> >> > Cc: Andrew Bayer; builds@apache.org; Giridharan Kesavan
>> >> > Subject: Re: ulimit changed with Apache Jenkins upgrade?
>> >> >
>> >> > +1
>> >> >
>> >> > On 18 Jul 2014, at 18:59, Patrick Hunt <ph...@apache.org> wrote:
>> >> >
>> >> >> Hi builds folks, is this a system wide issue or something we should
>> >> >> address ourselves? Thanks!
>> >> >>
>> >> >> Patrick
>> >> >>
>> >> >> ---------- Forwarded message ----------
>> >> >> From: Patrick Hunt <ph...@apache.org>
>> >> >> Date: Fri, Jul 18, 2014 at 10:38 AM
>> >> >> Subject: ulimit changed with Apache Jenkins upgrade?
>> >> >> To: Giridharan Kesavan <gk...@hortonworks.com>
>> >> >> Cc: DevZooKeeper <de...@zookeeper.apache.org>, Andrew Bayer
>> >> >> <an...@cloudera.com>
>> >> >>
>> >> >>
>> >> >> Hi Giri, can you check that the new hosts (H#) have the ulimit set
>> >> >> to
>> >> >> what it was set to on the original hadoop# hosts? I'm seeing new
>> >> >> test
>> >> >> failures with
>> >> >>
>> >> >> [exec] [junit] java.io.FileNotFoundException:
>> >> >>
>> >> >> /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/
>> >> >>
>> >> >> build/test/tmp/test7610638215300246179.junit.dir/version-2/log.1000000
>> >> >> 01
>> >> >> (Too many open files)
>> >> >>
>> >> >> which we've not seen before. I believe this means that we running
>> >> >> out
>> >> >> of file descriptors?
>> >> >>
>> >> >> Can you verify and address if possible?
>> >> >>
>> >> >> Thanks,
>> >> >>
>> >> >> Patrick
>> >> >
>> >
>> >
>> >
>> > CONFIDENTIALITY NOTICE
>> > NOTICE: This message is intended for the use of the individual or entity
>> > to
>> > which it is addressed and may contain information that is confidential,
>> > privileged and exempt from disclosure under applicable law. If the
>> > reader of
>> > this message is not the intended recipient, you are hereby notified that
>> > any
>> > printing, copying, dissemination, distribution, disclosure or forwarding
>> > of
>> > this communication is strictly prohibited. If you have received this
>> > communication in error, please contact the sender immediately and delete
>> > it
>> > from your system. Thank You.
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader of
> this message is not the intended recipient, you are hereby notified that any
> printing, copying, dissemination, distribution, disclosure or forwarding of
> this communication is strictly prohibited. If you have received this
> communication in error, please contact the sender immediately and delete it
> from your system. Thank You.
Re: ulimit changed with Apache Jenkins upgrade?
Posted by Giridharan Kesavan <gk...@hortonworks.com>.
jenkins slaves might need a restart.
Could you pls re-launch the slaves from the jenkins UI configuration page?
-giri
On Tue, Jul 22, 2014 at 2:25 PM, Patrick Hunt <ph...@apache.org> wrote:
> Thanks Giri! Unfortunately though it seems to not have taken effect, I
> just kicked off a precommit build and I see
>
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) unlimited
> pending signals (-i) 386178
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 4096
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 386178
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
>
> more details here:
>
> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2215/console
>
> Patrick
>
> On Tue, Jul 22, 2014 at 1:29 PM, Giridharan Kesavan
> <gk...@hortonworks.com> wrote:
> >
> >
> > jenkins@asf901:~$ ulimit -a
> > core file size (blocks, -c) 0
> > data seg size (kbytes, -d) unlimited
> > scheduling priority (-e) 0
> > file size (blocks, -f) unlimited
> > pending signals (-i) 386178
> > max locked memory (kbytes, -l) 64
> > max memory size (kbytes, -m) unlimited
> > open files (-n) 60000
> > pipe size (512 bytes, -p) 8
> > POSIX message queues (bytes, -q) 819200
> > real-time priority (-r) 0
> > stack size (kbytes, -s) 8192
> > cpu time (seconds, -t) unlimited
> > max user processes (-u) 10240
> > virtual memory (kbytes, -v) unlimited
> > file locks (-x) unlimited
> >
> > bumped up the open files and max user processes on all the slaves.
> >
> >
> >
> > -giri
> >
> >
> > On Tue, Jul 22, 2014 at 12:04 PM, Patrick Hunt <ph...@apache.org> wrote:
> >>
> >> Giri any chance can you take a look at the ulimit issue on the H#
> >> machines? All the ZK precommit builds are failing as a result.
> >>
> >> I updated the precommit build last night to output "ulimit -a" and it
> >> says the current limit is 4096, can we bump that up or set the default
> >> to unlimited?
> >>
> >> Thanks!
> >>
> >> Patrick
> >>
> >> On Sun, Jul 20, 2014 at 10:32 PM, Rakesh R <ra...@huawei.com> wrote:
> >> > +1
> >> >
> >> >
> >> > Adding one more point. I could see the following error too in the
> >> > pre-commit build.
> >> >
> >> > [exec] [junit] Exception in thread "CommitProcWorkThread-16"
> >> > java.lang.NoClassDefFoundError:
> org/apache/zookeeper/server/ConnectionBean
> >> > [exec] [junit] at
> >> >
> org.apache.zookeeper.server.ServerCnxnFactory.registerConnection(ServerCnxnFactory.java:159)
> >> > [exec] [junit] at
> >> >
> org.apache.zookeeper.server.ZooKeeperServer.finishSessionInit(ZooKeeperServer.java:594)
> >> > [exec] [junit] at
> >> >
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:198)
> >> > [exec] [junit] at
> >> >
> org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:295)
> >> > [exec] [junit] at
> >> >
> org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:161)
> >> > [exec] [junit] at
> >> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> >> > [exec] [junit] at
> >> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> >> > [exec] [junit] at
> java.lang.Thread.run(Thread.java:662)
> >> >
> >> > -Rakesh
> >> >
> >> > -----Original Message-----
> >> > From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]
> >> > Sent: 21 July 2014 02:19
> >> > To: dev@zookeeper.apache.org
> >> > Cc: Andrew Bayer; builds@apache.org; Giridharan Kesavan
> >> > Subject: Re: ulimit changed with Apache Jenkins upgrade?
> >> >
> >> > +1
> >> >
> >> > On 18 Jul 2014, at 18:59, Patrick Hunt <ph...@apache.org> wrote:
> >> >
> >> >> Hi builds folks, is this a system wide issue or something we should
> >> >> address ourselves? Thanks!
> >> >>
> >> >> Patrick
> >> >>
> >> >> ---------- Forwarded message ----------
> >> >> From: Patrick Hunt <ph...@apache.org>
> >> >> Date: Fri, Jul 18, 2014 at 10:38 AM
> >> >> Subject: ulimit changed with Apache Jenkins upgrade?
> >> >> To: Giridharan Kesavan <gk...@hortonworks.com>
> >> >> Cc: DevZooKeeper <de...@zookeeper.apache.org>, Andrew Bayer
> >> >> <an...@cloudera.com>
> >> >>
> >> >>
> >> >> Hi Giri, can you check that the new hosts (H#) have the ulimit set to
> >> >> what it was set to on the original hadoop# hosts? I'm seeing new test
> >> >> failures with
> >> >>
> >> >> [exec] [junit] java.io.FileNotFoundException:
> >> >>
> /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/
> >> >>
> build/test/tmp/test7610638215300246179.junit.dir/version-2/log.1000000
> >> >> 01
> >> >> (Too many open files)
> >> >>
> >> >> which we've not seen before. I believe this means that we running out
> >> >> of file descriptors?
> >> >>
> >> >> Can you verify and address if possible?
> >> >>
> >> >> Thanks,
> >> >>
> >> >> Patrick
> >> >
> >
> >
> >
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the
> reader of
> > this message is not the intended recipient, you are hereby notified that
> any
> > printing, copying, dissemination, distribution, disclosure or forwarding
> of
> > this communication is strictly prohibited. If you have received this
> > communication in error, please contact the sender immediately and delete
> it
> > from your system. Thank You.
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: ulimit changed with Apache Jenkins upgrade?
Posted by Patrick Hunt <ph...@apache.org>.
Thanks Giri! Unfortunately though it seems to not have taken effect, I
just kicked off a precommit build and I see
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 386178
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 4096
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 386178
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
more details here:
https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2215/console
Patrick
On Tue, Jul 22, 2014 at 1:29 PM, Giridharan Kesavan
<gk...@hortonworks.com> wrote:
>
>
> jenkins@asf901:~$ ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) unlimited
> pending signals (-i) 386178
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 60000
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 10240
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
>
> bumped up the open files and max user processes on all the slaves.
>
>
>
> -giri
>
>
> On Tue, Jul 22, 2014 at 12:04 PM, Patrick Hunt <ph...@apache.org> wrote:
>>
>> Giri any chance can you take a look at the ulimit issue on the H#
>> machines? All the ZK precommit builds are failing as a result.
>>
>> I updated the precommit build last night to output "ulimit -a" and it
>> says the current limit is 4096, can we bump that up or set the default
>> to unlimited?
>>
>> Thanks!
>>
>> Patrick
>>
>> On Sun, Jul 20, 2014 at 10:32 PM, Rakesh R <ra...@huawei.com> wrote:
>> > +1
>> >
>> >
>> > Adding one more point. I could see the following error too in the
>> > pre-commit build.
>> >
>> > [exec] [junit] Exception in thread "CommitProcWorkThread-16"
>> > java.lang.NoClassDefFoundError: org/apache/zookeeper/server/ConnectionBean
>> > [exec] [junit] at
>> > org.apache.zookeeper.server.ServerCnxnFactory.registerConnection(ServerCnxnFactory.java:159)
>> > [exec] [junit] at
>> > org.apache.zookeeper.server.ZooKeeperServer.finishSessionInit(ZooKeeperServer.java:594)
>> > [exec] [junit] at
>> > org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:198)
>> > [exec] [junit] at
>> > org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:295)
>> > [exec] [junit] at
>> > org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:161)
>> > [exec] [junit] at
>> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>> > [exec] [junit] at
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>> > [exec] [junit] at java.lang.Thread.run(Thread.java:662)
>> >
>> > -Rakesh
>> >
>> > -----Original Message-----
>> > From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]
>> > Sent: 21 July 2014 02:19
>> > To: dev@zookeeper.apache.org
>> > Cc: Andrew Bayer; builds@apache.org; Giridharan Kesavan
>> > Subject: Re: ulimit changed with Apache Jenkins upgrade?
>> >
>> > +1
>> >
>> > On 18 Jul 2014, at 18:59, Patrick Hunt <ph...@apache.org> wrote:
>> >
>> >> Hi builds folks, is this a system wide issue or something we should
>> >> address ourselves? Thanks!
>> >>
>> >> Patrick
>> >>
>> >> ---------- Forwarded message ----------
>> >> From: Patrick Hunt <ph...@apache.org>
>> >> Date: Fri, Jul 18, 2014 at 10:38 AM
>> >> Subject: ulimit changed with Apache Jenkins upgrade?
>> >> To: Giridharan Kesavan <gk...@hortonworks.com>
>> >> Cc: DevZooKeeper <de...@zookeeper.apache.org>, Andrew Bayer
>> >> <an...@cloudera.com>
>> >>
>> >>
>> >> Hi Giri, can you check that the new hosts (H#) have the ulimit set to
>> >> what it was set to on the original hadoop# hosts? I'm seeing new test
>> >> failures with
>> >>
>> >> [exec] [junit] java.io.FileNotFoundException:
>> >> /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/
>> >> build/test/tmp/test7610638215300246179.junit.dir/version-2/log.1000000
>> >> 01
>> >> (Too many open files)
>> >>
>> >> which we've not seen before. I believe this means that we running out
>> >> of file descriptors?
>> >>
>> >> Can you verify and address if possible?
>> >>
>> >> Thanks,
>> >>
>> >> Patrick
>> >
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader of
> this message is not the intended recipient, you are hereby notified that any
> printing, copying, dissemination, distribution, disclosure or forwarding of
> this communication is strictly prohibited. If you have received this
> communication in error, please contact the sender immediately and delete it
> from your system. Thank You.
Re: ulimit changed with Apache Jenkins upgrade?
Posted by Giridharan Kesavan <gk...@hortonworks.com>.
jenkins@asf901:~$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 386178
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 60000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 10240
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
bumped up the open files and max user processes on all the slaves.
-giri
On Tue, Jul 22, 2014 at 12:04 PM, Patrick Hunt <ph...@apache.org> wrote:
> Giri any chance can you take a look at the ulimit issue on the H#
> machines? All the ZK precommit builds are failing as a result.
>
> I updated the precommit build last night to output "ulimit -a" and it
> says the current limit is 4096, can we bump that up or set the default
> to unlimited?
>
> Thanks!
>
> Patrick
>
> On Sun, Jul 20, 2014 at 10:32 PM, Rakesh R <ra...@huawei.com> wrote:
> > +1
> >
> >
> > Adding one more point. I could see the following error too in the
> pre-commit build.
> >
> > [exec] [junit] Exception in thread "CommitProcWorkThread-16"
> java.lang.NoClassDefFoundError: org/apache/zookeeper/server/ConnectionBean
> > [exec] [junit] at
> org.apache.zookeeper.server.ServerCnxnFactory.registerConnection(ServerCnxnFactory.java:159)
> > [exec] [junit] at
> org.apache.zookeeper.server.ZooKeeperServer.finishSessionInit(ZooKeeperServer.java:594)
> > [exec] [junit] at
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:198)
> > [exec] [junit] at
> org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:295)
> > [exec] [junit] at
> org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:161)
> > [exec] [junit] at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> > [exec] [junit] at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> > [exec] [junit] at java.lang.Thread.run(Thread.java:662)
> >
> > -Rakesh
> >
> > -----Original Message-----
> > From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]
> > Sent: 21 July 2014 02:19
> > To: dev@zookeeper.apache.org
> > Cc: Andrew Bayer; builds@apache.org; Giridharan Kesavan
> > Subject: Re: ulimit changed with Apache Jenkins upgrade?
> >
> > +1
> >
> > On 18 Jul 2014, at 18:59, Patrick Hunt <ph...@apache.org> wrote:
> >
> >> Hi builds folks, is this a system wide issue or something we should
> >> address ourselves? Thanks!
> >>
> >> Patrick
> >>
> >> ---------- Forwarded message ----------
> >> From: Patrick Hunt <ph...@apache.org>
> >> Date: Fri, Jul 18, 2014 at 10:38 AM
> >> Subject: ulimit changed with Apache Jenkins upgrade?
> >> To: Giridharan Kesavan <gk...@hortonworks.com>
> >> Cc: DevZooKeeper <de...@zookeeper.apache.org>, Andrew Bayer
> >> <an...@cloudera.com>
> >>
> >>
> >> Hi Giri, can you check that the new hosts (H#) have the ulimit set to
> >> what it was set to on the original hadoop# hosts? I'm seeing new test
> >> failures with
> >>
> >> [exec] [junit] java.io.FileNotFoundException:
> >> /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/
> >> build/test/tmp/test7610638215300246179.junit.dir/version-2/log.1000000
> >> 01
> >> (Too many open files)
> >>
> >> which we've not seen before. I believe this means that we running out
> >> of file descriptors?
> >>
> >> Can you verify and address if possible?
> >>
> >> Thanks,
> >>
> >> Patrick
> >
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: ulimit changed with Apache Jenkins upgrade?
Posted by Patrick Hunt <ph...@apache.org>.
Giri any chance can you take a look at the ulimit issue on the H#
machines? All the ZK precommit builds are failing as a result.
I updated the precommit build last night to output "ulimit -a" and it
says the current limit is 4096, can we bump that up or set the default
to unlimited?
Thanks!
Patrick
On Sun, Jul 20, 2014 at 10:32 PM, Rakesh R <ra...@huawei.com> wrote:
> +1
>
>
> Adding one more point. I could see the following error too in the pre-commit build.
>
> [exec] [junit] Exception in thread "CommitProcWorkThread-16" java.lang.NoClassDefFoundError: org/apache/zookeeper/server/ConnectionBean
> [exec] [junit] at org.apache.zookeeper.server.ServerCnxnFactory.registerConnection(ServerCnxnFactory.java:159)
> [exec] [junit] at org.apache.zookeeper.server.ZooKeeperServer.finishSessionInit(ZooKeeperServer.java:594)
> [exec] [junit] at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:198)
> [exec] [junit] at org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:295)
> [exec] [junit] at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:161)
> [exec] [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> [exec] [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> [exec] [junit] at java.lang.Thread.run(Thread.java:662)
>
> -Rakesh
>
> -----Original Message-----
> From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]
> Sent: 21 July 2014 02:19
> To: dev@zookeeper.apache.org
> Cc: Andrew Bayer; builds@apache.org; Giridharan Kesavan
> Subject: Re: ulimit changed with Apache Jenkins upgrade?
>
> +1
>
> On 18 Jul 2014, at 18:59, Patrick Hunt <ph...@apache.org> wrote:
>
>> Hi builds folks, is this a system wide issue or something we should
>> address ourselves? Thanks!
>>
>> Patrick
>>
>> ---------- Forwarded message ----------
>> From: Patrick Hunt <ph...@apache.org>
>> Date: Fri, Jul 18, 2014 at 10:38 AM
>> Subject: ulimit changed with Apache Jenkins upgrade?
>> To: Giridharan Kesavan <gk...@hortonworks.com>
>> Cc: DevZooKeeper <de...@zookeeper.apache.org>, Andrew Bayer
>> <an...@cloudera.com>
>>
>>
>> Hi Giri, can you check that the new hosts (H#) have the ulimit set to
>> what it was set to on the original hadoop# hosts? I'm seeing new test
>> failures with
>>
>> [exec] [junit] java.io.FileNotFoundException:
>> /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/
>> build/test/tmp/test7610638215300246179.junit.dir/version-2/log.1000000
>> 01
>> (Too many open files)
>>
>> which we've not seen before. I believe this means that we running out
>> of file descriptors?
>>
>> Can you verify and address if possible?
>>
>> Thanks,
>>
>> Patrick
>
Re: ulimit changed with Apache Jenkins upgrade?
Posted by Patrick Hunt <ph...@cloudera.com>.
I believe that's because we've run out of files and the class file
can't be loaded as a result.
Patrick
On Sun, Jul 20, 2014 at 10:32 PM, Rakesh R <ra...@huawei.com> wrote:
> +1
>
>
> Adding one more point. I could see the following error too in the pre-commit build.
>
> [exec] [junit] Exception in thread "CommitProcWorkThread-16" java.lang.NoClassDefFoundError: org/apache/zookeeper/server/ConnectionBean
> [exec] [junit] at org.apache.zookeeper.server.ServerCnxnFactory.registerConnection(ServerCnxnFactory.java:159)
> [exec] [junit] at org.apache.zookeeper.server.ZooKeeperServer.finishSessionInit(ZooKeeperServer.java:594)
> [exec] [junit] at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:198)
> [exec] [junit] at org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:295)
> [exec] [junit] at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:161)
> [exec] [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> [exec] [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> [exec] [junit] at java.lang.Thread.run(Thread.java:662)
>
> -Rakesh
>
> -----Original Message-----
> From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]
> Sent: 21 July 2014 02:19
> To: dev@zookeeper.apache.org
> Cc: Andrew Bayer; builds@apache.org; Giridharan Kesavan
> Subject: Re: ulimit changed with Apache Jenkins upgrade?
>
> +1
>
> On 18 Jul 2014, at 18:59, Patrick Hunt <ph...@apache.org> wrote:
>
>> Hi builds folks, is this a system wide issue or something we should
>> address ourselves? Thanks!
>>
>> Patrick
>>
>> ---------- Forwarded message ----------
>> From: Patrick Hunt <ph...@apache.org>
>> Date: Fri, Jul 18, 2014 at 10:38 AM
>> Subject: ulimit changed with Apache Jenkins upgrade?
>> To: Giridharan Kesavan <gk...@hortonworks.com>
>> Cc: DevZooKeeper <de...@zookeeper.apache.org>, Andrew Bayer
>> <an...@cloudera.com>
>>
>>
>> Hi Giri, can you check that the new hosts (H#) have the ulimit set to
>> what it was set to on the original hadoop# hosts? I'm seeing new test
>> failures with
>>
>> [exec] [junit] java.io.FileNotFoundException:
>> /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/
>> build/test/tmp/test7610638215300246179.junit.dir/version-2/log.1000000
>> 01
>> (Too many open files)
>>
>> which we've not seen before. I believe this means that we running out
>> of file descriptors?
>>
>> Can you verify and address if possible?
>>
>> Thanks,
>>
>> Patrick
>
RE: ulimit changed with Apache Jenkins upgrade?
Posted by Rakesh R <ra...@huawei.com>.
+1
Adding one more point. I could see the following error too in the pre-commit build.
[exec] [junit] Exception in thread "CommitProcWorkThread-16" java.lang.NoClassDefFoundError: org/apache/zookeeper/server/ConnectionBean
[exec] [junit] at org.apache.zookeeper.server.ServerCnxnFactory.registerConnection(ServerCnxnFactory.java:159)
[exec] [junit] at org.apache.zookeeper.server.ZooKeeperServer.finishSessionInit(ZooKeeperServer.java:594)
[exec] [junit] at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:198)
[exec] [junit] at org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:295)
[exec] [junit] at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:161)
[exec] [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
[exec] [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
[exec] [junit] at java.lang.Thread.run(Thread.java:662)
-Rakesh
-----Original Message-----
From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]
Sent: 21 July 2014 02:19
To: dev@zookeeper.apache.org
Cc: Andrew Bayer; builds@apache.org; Giridharan Kesavan
Subject: Re: ulimit changed with Apache Jenkins upgrade?
+1
On 18 Jul 2014, at 18:59, Patrick Hunt <ph...@apache.org> wrote:
> Hi builds folks, is this a system wide issue or something we should
> address ourselves? Thanks!
>
> Patrick
>
> ---------- Forwarded message ----------
> From: Patrick Hunt <ph...@apache.org>
> Date: Fri, Jul 18, 2014 at 10:38 AM
> Subject: ulimit changed with Apache Jenkins upgrade?
> To: Giridharan Kesavan <gk...@hortonworks.com>
> Cc: DevZooKeeper <de...@zookeeper.apache.org>, Andrew Bayer
> <an...@cloudera.com>
>
>
> Hi Giri, can you check that the new hosts (H#) have the ulimit set to
> what it was set to on the original hadoop# hosts? I'm seeing new test
> failures with
>
> [exec] [junit] java.io.FileNotFoundException:
> /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/
> build/test/tmp/test7610638215300246179.junit.dir/version-2/log.1000000
> 01
> (Too many open files)
>
> which we've not seen before. I believe this means that we running out
> of file descriptors?
>
> Can you verify and address if possible?
>
> Thanks,
>
> Patrick
RE: ulimit changed with Apache Jenkins upgrade?
Posted by Rakesh R <ra...@huawei.com>.
+1
Adding one more point. I could see the following error too in the pre-commit build.
[exec] [junit] Exception in thread "CommitProcWorkThread-16" java.lang.NoClassDefFoundError: org/apache/zookeeper/server/ConnectionBean
[exec] [junit] at org.apache.zookeeper.server.ServerCnxnFactory.registerConnection(ServerCnxnFactory.java:159)
[exec] [junit] at org.apache.zookeeper.server.ZooKeeperServer.finishSessionInit(ZooKeeperServer.java:594)
[exec] [junit] at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:198)
[exec] [junit] at org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:295)
[exec] [junit] at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:161)
[exec] [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
[exec] [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
[exec] [junit] at java.lang.Thread.run(Thread.java:662)
-Rakesh
-----Original Message-----
From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]
Sent: 21 July 2014 02:19
To: dev@zookeeper.apache.org
Cc: Andrew Bayer; builds@apache.org; Giridharan Kesavan
Subject: Re: ulimit changed with Apache Jenkins upgrade?
+1
On 18 Jul 2014, at 18:59, Patrick Hunt <ph...@apache.org> wrote:
> Hi builds folks, is this a system wide issue or something we should
> address ourselves? Thanks!
>
> Patrick
>
> ---------- Forwarded message ----------
> From: Patrick Hunt <ph...@apache.org>
> Date: Fri, Jul 18, 2014 at 10:38 AM
> Subject: ulimit changed with Apache Jenkins upgrade?
> To: Giridharan Kesavan <gk...@hortonworks.com>
> Cc: DevZooKeeper <de...@zookeeper.apache.org>, Andrew Bayer
> <an...@cloudera.com>
>
>
> Hi Giri, can you check that the new hosts (H#) have the ulimit set to
> what it was set to on the original hadoop# hosts? I'm seeing new test
> failures with
>
> [exec] [junit] java.io.FileNotFoundException:
> /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/
> build/test/tmp/test7610638215300246179.junit.dir/version-2/log.1000000
> 01
> (Too many open files)
>
> which we've not seen before. I believe this means that we running out
> of file descriptors?
>
> Can you verify and address if possible?
>
> Thanks,
>
> Patrick
Re: ulimit changed with Apache Jenkins upgrade?
Posted by Flavio Junqueira <fp...@yahoo.com.INVALID>.
+1
On 18 Jul 2014, at 18:59, Patrick Hunt <ph...@apache.org> wrote:
> Hi builds folks, is this a system wide issue or something we should
> address ourselves? Thanks!
>
> Patrick
>
> ---------- Forwarded message ----------
> From: Patrick Hunt <ph...@apache.org>
> Date: Fri, Jul 18, 2014 at 10:38 AM
> Subject: ulimit changed with Apache Jenkins upgrade?
> To: Giridharan Kesavan <gk...@hortonworks.com>
> Cc: DevZooKeeper <de...@zookeeper.apache.org>, Andrew Bayer <an...@cloudera.com>
>
>
> Hi Giri, can you check that the new hosts (H#) have the ulimit set to
> what it was set to on the original hadoop# hosts? I'm seeing new test
> failures with
>
> [exec] [junit] java.io.FileNotFoundException:
> /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/tmp/test7610638215300246179.junit.dir/version-2/log.100000001
> (Too many open files)
>
> which we've not seen before. I believe this means that we running out
> of file descriptors?
>
> Can you verify and address if possible?
>
> Thanks,
>
> Patrick
Re: ulimit changed with Apache Jenkins upgrade?
Posted by Flavio Junqueira <fp...@yahoo.com.INVALID>.
+1
On 18 Jul 2014, at 18:59, Patrick Hunt <ph...@apache.org> wrote:
> Hi builds folks, is this a system wide issue or something we should
> address ourselves? Thanks!
>
> Patrick
>
> ---------- Forwarded message ----------
> From: Patrick Hunt <ph...@apache.org>
> Date: Fri, Jul 18, 2014 at 10:38 AM
> Subject: ulimit changed with Apache Jenkins upgrade?
> To: Giridharan Kesavan <gk...@hortonworks.com>
> Cc: DevZooKeeper <de...@zookeeper.apache.org>, Andrew Bayer <an...@cloudera.com>
>
>
> Hi Giri, can you check that the new hosts (H#) have the ulimit set to
> what it was set to on the original hadoop# hosts? I'm seeing new test
> failures with
>
> [exec] [junit] java.io.FileNotFoundException:
> /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/tmp/test7610638215300246179.junit.dir/version-2/log.100000001
> (Too many open files)
>
> which we've not seen before. I believe this means that we running out
> of file descriptors?
>
> Can you verify and address if possible?
>
> Thanks,
>
> Patrick
Fwd: ulimit changed with Apache Jenkins upgrade?
Posted by Patrick Hunt <ph...@apache.org>.
Hi builds folks, is this a system wide issue or something we should
address ourselves? Thanks!
Patrick
---------- Forwarded message ----------
From: Patrick Hunt <ph...@apache.org>
Date: Fri, Jul 18, 2014 at 10:38 AM
Subject: ulimit changed with Apache Jenkins upgrade?
To: Giridharan Kesavan <gk...@hortonworks.com>
Cc: DevZooKeeper <de...@zookeeper.apache.org>, Andrew Bayer <an...@cloudera.com>
Hi Giri, can you check that the new hosts (H#) have the ulimit set to
what it was set to on the original hadoop# hosts? I'm seeing new test
failures with
[exec] [junit] java.io.FileNotFoundException:
/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/tmp/test7610638215300246179.junit.dir/version-2/log.100000001
(Too many open files)
which we've not seen before. I believe this means that we running out
of file descriptors?
Can you verify and address if possible?
Thanks,
Patrick
Fwd: ulimit changed with Apache Jenkins upgrade?
Posted by Patrick Hunt <ph...@apache.org>.
Hi builds folks, is this a system wide issue or something we should
address ourselves? Thanks!
Patrick
---------- Forwarded message ----------
From: Patrick Hunt <ph...@apache.org>
Date: Fri, Jul 18, 2014 at 10:38 AM
Subject: ulimit changed with Apache Jenkins upgrade?
To: Giridharan Kesavan <gk...@hortonworks.com>
Cc: DevZooKeeper <de...@zookeeper.apache.org>, Andrew Bayer <an...@cloudera.com>
Hi Giri, can you check that the new hosts (H#) have the ulimit set to
what it was set to on the original hadoop# hosts? I'm seeing new test
failures with
[exec] [junit] java.io.FileNotFoundException:
/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/tmp/test7610638215300246179.junit.dir/version-2/log.100000001
(Too many open files)
which we've not seen before. I believe this means that we running out
of file descriptors?
Can you verify and address if possible?
Thanks,
Patrick