You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Liam Clarke-Hutchinson <lc...@redhat.com> on 2021/05/13 05:02:07 UTC
Re: too many open files
Hi Gu Haihua,
What kinds of Kafka Connect (KC) tasks are running?
Where are the too many open files errors being thrown? Is it on the
brokers? Or the standalone KC?
Also, on the machine affected, can you please run `ulimit -a` under the
relevant user (the user KC is running under, or the user the broker runs
under, depending on where you're seeing the error) and post the output
here?
Cheers,
Liam Clarke-Hutchinson
On Thu, May 13, 2021 at 4:54 PM Gu Haihua <gu...@outlook.com> wrote:
> Hi everyone
>
> We have a kafka connect service, which runs in standalone mode. Recently,
> there are too many tasks running on it, and sometimes too many open files
> error will be thrown. Through the lsof command, it can be seen that there
> are many commit.offsets files open.
>
> $ lsof -p 16862|wc -l
> 36960
>
> $ lsof -p 16862|grep connect.offsets|wc -l
> 36161
>
> Does anyone know the reason?
>
> Regards
> Haihua
>
Re: too many open files
Posted by Liam Clarke-Hutchinson <lc...@redhat.com>.
PS, your emails along these lines are probably more appropriately directed
to users@kafka.apache.org :)
On Thu, May 13, 2021 at 5:02 PM Liam Clarke-Hutchinson <lc...@redhat.com>
wrote:
> Hi Gu Haihua,
>
> What kinds of Kafka Connect (KC) tasks are running?
>
> Where are the too many open files errors being thrown? Is it on the
> brokers? Or the standalone KC?
>
> Also, on the machine affected, can you please run `ulimit -a` under the
> relevant user (the user KC is running under, or the user the broker runs
> under, depending on where you're seeing the error) and post the output
> here?
>
> Cheers,
>
> Liam Clarke-Hutchinson
>
> On Thu, May 13, 2021 at 4:54 PM Gu Haihua <gu...@outlook.com>
> wrote:
>
>> Hi everyone
>>
>> We have a kafka connect service, which runs in standalone mode. Recently,
>> there are too many tasks running on it, and sometimes too many open files
>> error will be thrown. Through the lsof command, it can be seen that there
>> are many commit.offsets files open.
>>
>> $ lsof -p 16862|wc -l
>> 36960
>>
>> $ lsof -p 16862|grep connect.offsets|wc -l
>> 36161
>>
>> Does anyone know the reason?
>>
>> Regards
>> Haihua
>>
>