You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Junda Yang <yo...@gmail.com> on 2019/04/05 18:44:16 UTC
CgroupTaskRunner Failure
Hi there,
Does anyone have experience in running CgroupTaskRunner? I tested it on our
staging cluster but kept getting errors:
Traceback (most recent call last):
File
"/srv/venvs/service/trusty/service_venv_python3.6/src/apache-airflow/airflow/jobs.py",
line 2540, in _execute
self.task_runner.start()
File
"/srv/venvs/service/trusty/service_venv_python3.6/src/apache-airflow/airflow/contrib/task_runner/cgroup_task_runner.py",
line 154, in start
mem_cgroup_node.controller.limit_in_bytes = self._mem_mb_limit * 1024 *
1024
File
"/srv/venvs/service/trusty/service_venv_python3.6/lib/python3.6/site-packages/cgroupspy/interfaces.py",
line 63, in __set__
return instance.set_property(self.filename, value)
File
"/srv/venvs/service/trusty/service_venv_python3.6/lib/python3.6/site-packages/cgroupspy/controllers.py",
line 63, in set_property
return f.write(str(value))
OSError: [Errno 28] No space left on device
Actually it succeeded in creating memory group but failed later creating
cpu group in
https://github.com/apache/airflow/blob/c3939c8e721870d263997e7aeaebc28e678d544b/airflow/contrib/task_runner/cgroup_task_runner.py#L139-#L156.
However, If I switch the order, it could create cpu group and then failed
creating memory group. So it sounds to me that it can only succeed in
creating the first cgroup configuration. I did some researches online but
relevant information is little. Any help is appreciated!
FYI, this is the system version that our Airflow is running on:
NAME="Ubuntu"
VERSION="14.04.6 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.6 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
Thanks,
Junda
Re: CgroupTaskRunner Failure
Posted by yo...@gmail.com.
Yeah. There is enough disk space left on the machine. The latter theory sounds reasonable. Going to verify that. Thanks Ash!
> On Apr 5, 2019, at 11:47, Ash Berlin-Taylor <as...@apache.org> wrote:
>
> Have you checked the obvious:
>
> Do you have disk space left?
>
> If you do it's possible you've hit a limit on the number of cgroups - either cos there are too many running at the same time, or because something isn't cleaning them up.
>
> -ash
>
>> On 5 Apr 2019, at 19:44, Junda Yang <yo...@gmail.com> wrote:
>>
>> Hi there,
>>
>> Does anyone have experience in running CgroupTaskRunner? I tested it on our
>> staging cluster but kept getting errors:
>>
>> Traceback (most recent call last):
>>
>> File
>> "/srv/venvs/service/trusty/service_venv_python3.6/src/apache-airflow/airflow/jobs.py",
>> line 2540, in _execute
>>
>> self.task_runner.start()
>>
>> File
>> "/srv/venvs/service/trusty/service_venv_python3.6/src/apache-airflow/airflow/contrib/task_runner/cgroup_task_runner.py",
>> line 154, in start
>>
>> mem_cgroup_node.controller.limit_in_bytes = self._mem_mb_limit * 1024 *
>> 1024
>>
>> File
>> "/srv/venvs/service/trusty/service_venv_python3.6/lib/python3.6/site-packages/cgroupspy/interfaces.py",
>> line 63, in __set__
>>
>> return instance.set_property(self.filename, value)
>>
>> File
>> "/srv/venvs/service/trusty/service_venv_python3.6/lib/python3.6/site-packages/cgroupspy/controllers.py",
>> line 63, in set_property
>>
>> return f.write(str(value))
>>
>> OSError: [Errno 28] No space left on device
>>
>>
>> Actually it succeeded in creating memory group but failed later creating
>> cpu group in
>> https://github.com/apache/airflow/blob/c3939c8e721870d263997e7aeaebc28e678d544b/airflow/contrib/task_runner/cgroup_task_runner.py#L139-#L156.
>> However, If I switch the order, it could create cpu group and then failed
>> creating memory group. So it sounds to me that it can only succeed in
>> creating the first cgroup configuration. I did some researches online but
>> relevant information is little. Any help is appreciated!
>>
>> FYI, this is the system version that our Airflow is running on:
>>
>> NAME="Ubuntu"
>>
>> VERSION="14.04.6 LTS, Trusty Tahr"
>>
>> ID=ubuntu
>>
>> ID_LIKE=debian
>>
>> PRETTY_NAME="Ubuntu 14.04.6 LTS"
>>
>> VERSION_ID="14.04"
>>
>> HOME_URL="http://www.ubuntu.com/"
>>
>> SUPPORT_URL="http://help.ubuntu.com/"
>>
>> Thanks,
>> Junda
>
Re: CgroupTaskRunner Failure
Posted by Ash Berlin-Taylor <as...@apache.org>.
Have you checked the obvious:
Do you have disk space left?
If you do it's possible you've hit a limit on the number of cgroups - either cos there are too many running at the same time, or because something isn't cleaning them up.
-ash
> On 5 Apr 2019, at 19:44, Junda Yang <yo...@gmail.com> wrote:
>
> Hi there,
>
> Does anyone have experience in running CgroupTaskRunner? I tested it on our
> staging cluster but kept getting errors:
>
> Traceback (most recent call last):
>
> File
> "/srv/venvs/service/trusty/service_venv_python3.6/src/apache-airflow/airflow/jobs.py",
> line 2540, in _execute
>
> self.task_runner.start()
>
> File
> "/srv/venvs/service/trusty/service_venv_python3.6/src/apache-airflow/airflow/contrib/task_runner/cgroup_task_runner.py",
> line 154, in start
>
> mem_cgroup_node.controller.limit_in_bytes = self._mem_mb_limit * 1024 *
> 1024
>
> File
> "/srv/venvs/service/trusty/service_venv_python3.6/lib/python3.6/site-packages/cgroupspy/interfaces.py",
> line 63, in __set__
>
> return instance.set_property(self.filename, value)
>
> File
> "/srv/venvs/service/trusty/service_venv_python3.6/lib/python3.6/site-packages/cgroupspy/controllers.py",
> line 63, in set_property
>
> return f.write(str(value))
>
> OSError: [Errno 28] No space left on device
>
>
> Actually it succeeded in creating memory group but failed later creating
> cpu group in
> https://github.com/apache/airflow/blob/c3939c8e721870d263997e7aeaebc28e678d544b/airflow/contrib/task_runner/cgroup_task_runner.py#L139-#L156.
> However, If I switch the order, it could create cpu group and then failed
> creating memory group. So it sounds to me that it can only succeed in
> creating the first cgroup configuration. I did some researches online but
> relevant information is little. Any help is appreciated!
>
> FYI, this is the system version that our Airflow is running on:
>
> NAME="Ubuntu"
>
> VERSION="14.04.6 LTS, Trusty Tahr"
>
> ID=ubuntu
>
> ID_LIKE=debian
>
> PRETTY_NAME="Ubuntu 14.04.6 LTS"
>
> VERSION_ID="14.04"
>
> HOME_URL="http://www.ubuntu.com/"
>
> SUPPORT_URL="http://help.ubuntu.com/"
>
> Thanks,
> Junda