You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Viswanathan J <ja...@gmail.com> on 2013/12/17 18:10:54 UTC
System ulimit for hadoop jobtracker node
Hi,
What value(ulimit) will be fair enough for jobtracker node. If it's too
high will that cause thread block or any issue in jobtracker.
Please help.
Re: System ulimit for hadoop jobtracker node
Posted by Tao Xiao <xi...@gmail.com>.
set the following contents in /etc/security/limits.d/90-nproc.conf
hadoop-user soft nproc 32000
hadoop-user hard nproc 32000
hadoop-user soft nofile 65535
hadoop-user hard nofile 65535
2013/12/18 Viswanathan J <ja...@gmail.com>
> Hi,
>
> I have the ulimit of 999999, should be a problem? If yes what prob will
> arise?
> On Dec 18, 2013 12:05 AM, "rtejac" <rt...@gmail.com> wrote:
>
>> Depends on various factors like number of users, number of tasks like
>> size of MR jobs etc.. 32K-64K is a safe bet. 128K is a bad number, but
>> shouldn't harm.
>>
>> On Dec 17, 2013, at 9:10 AM, Viswanathan J <ja...@gmail.com>
>> wrote:
>>
>> Hi,
>>
>> What value(ulimit) will be fair enough for jobtracker node. If it's too
>> high will that cause thread block or any issue in jobtracker.
>>
>> Please help.
>>
>>
>>
Re: System ulimit for hadoop jobtracker node
Posted by Tao Xiao <xi...@gmail.com>.
set the following contents in /etc/security/limits.d/90-nproc.conf
hadoop-user soft nproc 32000
hadoop-user hard nproc 32000
hadoop-user soft nofile 65535
hadoop-user hard nofile 65535
2013/12/18 Viswanathan J <ja...@gmail.com>
> Hi,
>
> I have the ulimit of 999999, should be a problem? If yes what prob will
> arise?
> On Dec 18, 2013 12:05 AM, "rtejac" <rt...@gmail.com> wrote:
>
>> Depends on various factors like number of users, number of tasks like
>> size of MR jobs etc.. 32K-64K is a safe bet. 128K is a bad number, but
>> shouldn't harm.
>>
>> On Dec 17, 2013, at 9:10 AM, Viswanathan J <ja...@gmail.com>
>> wrote:
>>
>> Hi,
>>
>> What value(ulimit) will be fair enough for jobtracker node. If it's too
>> high will that cause thread block or any issue in jobtracker.
>>
>> Please help.
>>
>>
>>
Re: System ulimit for hadoop jobtracker node
Posted by Tao Xiao <xi...@gmail.com>.
set the following contents in /etc/security/limits.d/90-nproc.conf
hadoop-user soft nproc 32000
hadoop-user hard nproc 32000
hadoop-user soft nofile 65535
hadoop-user hard nofile 65535
2013/12/18 Viswanathan J <ja...@gmail.com>
> Hi,
>
> I have the ulimit of 999999, should be a problem? If yes what prob will
> arise?
> On Dec 18, 2013 12:05 AM, "rtejac" <rt...@gmail.com> wrote:
>
>> Depends on various factors like number of users, number of tasks like
>> size of MR jobs etc.. 32K-64K is a safe bet. 128K is a bad number, but
>> shouldn't harm.
>>
>> On Dec 17, 2013, at 9:10 AM, Viswanathan J <ja...@gmail.com>
>> wrote:
>>
>> Hi,
>>
>> What value(ulimit) will be fair enough for jobtracker node. If it's too
>> high will that cause thread block or any issue in jobtracker.
>>
>> Please help.
>>
>>
>>
Re: System ulimit for hadoop jobtracker node
Posted by Tao Xiao <xi...@gmail.com>.
set the following contents in /etc/security/limits.d/90-nproc.conf
hadoop-user soft nproc 32000
hadoop-user hard nproc 32000
hadoop-user soft nofile 65535
hadoop-user hard nofile 65535
2013/12/18 Viswanathan J <ja...@gmail.com>
> Hi,
>
> I have the ulimit of 999999, should be a problem? If yes what prob will
> arise?
> On Dec 18, 2013 12:05 AM, "rtejac" <rt...@gmail.com> wrote:
>
>> Depends on various factors like number of users, number of tasks like
>> size of MR jobs etc.. 32K-64K is a safe bet. 128K is a bad number, but
>> shouldn't harm.
>>
>> On Dec 17, 2013, at 9:10 AM, Viswanathan J <ja...@gmail.com>
>> wrote:
>>
>> Hi,
>>
>> What value(ulimit) will be fair enough for jobtracker node. If it's too
>> high will that cause thread block or any issue in jobtracker.
>>
>> Please help.
>>
>>
>>
Re: System ulimit for hadoop jobtracker node
Posted by Viswanathan J <ja...@gmail.com>.
Hi,
I have the ulimit of 999999, should be a problem? If yes what prob will
arise?
On Dec 18, 2013 12:05 AM, "rtejac" <rt...@gmail.com> wrote:
> Depends on various factors like number of users, number of tasks like size
> of MR jobs etc.. 32K-64K is a safe bet. 128K is a bad number, but
> shouldn't harm.
>
> On Dec 17, 2013, at 9:10 AM, Viswanathan J <ja...@gmail.com>
> wrote:
>
> Hi,
>
> What value(ulimit) will be fair enough for jobtracker node. If it's too
> high will that cause thread block or any issue in jobtracker.
>
> Please help.
>
>
>
Re: System ulimit for hadoop jobtracker node
Posted by Viswanathan J <ja...@gmail.com>.
Hi,
I have the ulimit of 999999, should be a problem? If yes what prob will
arise?
On Dec 18, 2013 12:05 AM, "rtejac" <rt...@gmail.com> wrote:
> Depends on various factors like number of users, number of tasks like size
> of MR jobs etc.. 32K-64K is a safe bet. 128K is a bad number, but
> shouldn't harm.
>
> On Dec 17, 2013, at 9:10 AM, Viswanathan J <ja...@gmail.com>
> wrote:
>
> Hi,
>
> What value(ulimit) will be fair enough for jobtracker node. If it's too
> high will that cause thread block or any issue in jobtracker.
>
> Please help.
>
>
>
Re: System ulimit for hadoop jobtracker node
Posted by Viswanathan J <ja...@gmail.com>.
Hi,
I have the ulimit of 999999, should be a problem? If yes what prob will
arise?
On Dec 18, 2013 12:05 AM, "rtejac" <rt...@gmail.com> wrote:
> Depends on various factors like number of users, number of tasks like size
> of MR jobs etc.. 32K-64K is a safe bet. 128K is a bad number, but
> shouldn't harm.
>
> On Dec 17, 2013, at 9:10 AM, Viswanathan J <ja...@gmail.com>
> wrote:
>
> Hi,
>
> What value(ulimit) will be fair enough for jobtracker node. If it's too
> high will that cause thread block or any issue in jobtracker.
>
> Please help.
>
>
>
Re: System ulimit for hadoop jobtracker node
Posted by Viswanathan J <ja...@gmail.com>.
Hi,
I have the ulimit of 999999, should be a problem? If yes what prob will
arise?
On Dec 18, 2013 12:05 AM, "rtejac" <rt...@gmail.com> wrote:
> Depends on various factors like number of users, number of tasks like size
> of MR jobs etc.. 32K-64K is a safe bet. 128K is a bad number, but
> shouldn't harm.
>
> On Dec 17, 2013, at 9:10 AM, Viswanathan J <ja...@gmail.com>
> wrote:
>
> Hi,
>
> What value(ulimit) will be fair enough for jobtracker node. If it's too
> high will that cause thread block or any issue in jobtracker.
>
> Please help.
>
>
>
Re: System ulimit for hadoop jobtracker node
Posted by rtejac <rt...@gmail.com>.
Depends on various factors like number of users, number of tasks like size of MR jobs etc.. 32K-64K is a safe bet. 128K is a bad number, but shouldn't harm.
On Dec 17, 2013, at 9:10 AM, Viswanathan J <ja...@gmail.com> wrote:
> Hi,
>
> What value(ulimit) will be fair enough for jobtracker node. If it's too high will that cause thread block or any issue in jobtracker.
>
> Please help.
>
Re: System ulimit for hadoop jobtracker node
Posted by rtejac <rt...@gmail.com>.
Depends on various factors like number of users, number of tasks like size of MR jobs etc.. 32K-64K is a safe bet. 128K is a bad number, but shouldn't harm.
On Dec 17, 2013, at 9:10 AM, Viswanathan J <ja...@gmail.com> wrote:
> Hi,
>
> What value(ulimit) will be fair enough for jobtracker node. If it's too high will that cause thread block or any issue in jobtracker.
>
> Please help.
>
Re: System ulimit for hadoop jobtracker node
Posted by rtejac <rt...@gmail.com>.
Depends on various factors like number of users, number of tasks like size of MR jobs etc.. 32K-64K is a safe bet. 128K is a bad number, but shouldn't harm.
On Dec 17, 2013, at 9:10 AM, Viswanathan J <ja...@gmail.com> wrote:
> Hi,
>
> What value(ulimit) will be fair enough for jobtracker node. If it's too high will that cause thread block or any issue in jobtracker.
>
> Please help.
>
Re: System ulimit for hadoop jobtracker node
Posted by rtejac <rt...@gmail.com>.
Depends on various factors like number of users, number of tasks like size of MR jobs etc.. 32K-64K is a safe bet. 128K is a bad number, but shouldn't harm.
On Dec 17, 2013, at 9:10 AM, Viswanathan J <ja...@gmail.com> wrote:
> Hi,
>
> What value(ulimit) will be fair enough for jobtracker node. If it's too high will that cause thread block or any issue in jobtracker.
>
> Please help.
>