You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Mehmet Ersoy <me...@gmail.com> on 2020/02/19 09:56:21 UTC
Airflow Celery Executor - Some Task instances turning to failed
without running and exporting log
Hello Friends,
I'm new to Airflow and I'm using Airflow Celery executor with Postgres
backend and Redis Message Queue service. For now, there is 4 worker, 1
Scheduler and 1 Web Server.
I have been preparing parallel Sqoop Jobs in my daily DAGs.
When I scheduled a daily DAG, Often some task instances turning to failed
without running state after started state. Then I can't see the logs of
them. It's blank. And when I run that task manually it's running without
any problem.
I don't really understand if there is an inconsistent situation in my DAG
writing.
I have attached one of my DAGs.
Thank you in advance,
Best Regards.
Mehmet.
Re: Airflow Celery Executor - Some Task instances turning to failed
without running and exporting log
Posted by Mehmet Ersoy <me...@gmail.com>.
Hi Tomasz,
What do you mean when you say "The mentioned DAG is missing"? I was add the
DAG, but gmail may have rejected that .py file.
I have attached again the file as .txt.
Thanks,
Mehmet.
Mehmet Ersoy <me...@gmail.com>, 19 Şub 2020 Çar, 15:20 tarihinde
şunu yazdı:
> I'm using 1.10.6 version of Airflow.
> Yes, this problem occurs in all of my parallel DAGs. And I attached one of
> my DAGs in my first mail. In addition, graph view of my DAG is as follows:
>
> [image: image.png]
>
> Thanks.
>
>
> Tomasz Urbaszek <to...@polidea.com>, 19 Şub 2020 Çar, 15:09
> tarihinde şunu yazdı:
>
>> What version of Airflow do you use?
>>
>> The mentioned DAG is missing but I'm curious about "parallel" jobs you are
>> running :) Does this problem occur with only one DAG?
>>
>> T.
>>
>> On Wed, Feb 19, 2020 at 12:58 PM Mehmet Ersoy <mehmet.ersoy1418@gmail.com
>> >
>> wrote:
>>
>> > Hi Tomasz,
>> > For now, I'm syncing the DAGs manually sending across airflow hosts. So,
>> > for now there is no git repository etc.
>> > In addition, My configs related with parallelism are as follows:
>> >
>> >
>> >
>> > # How many processes CeleryExecutor uses to sync task state.
>> >
>> > # 0 means to use max(1, number of cores - 1) processes.
>> >
>> > *sync_parallelism = 0*
>> >
>> >
>> >
>> >
>> >
>> > # The amount of parallelism as a setting to the executor. This defines
>> >
>> > # the max number of task instances that should run simultaneously
>> >
>> > # on this airflow installation
>> >
>> > *parallelism = 128*
>> >
>> >
>> >
>> >
>> >
>> > # The number of task instances allowed to run concurrently by the
>> scheduler
>> >
>> > *dag_concurrency = 128*
>> >
>> >
>> >
>> >
>> >
>> > # The concurrency that will be used when starting workers with the
>> >
>> > # "airflow worker" command. This defines the number of task instances
>> that
>> >
>> > # a worker will take, so size up your workers based on the resources on
>> >
>> > # your worker box and the nature of your tasks
>> >
>> > *worker_concurrency = 32*
>> >
>> >
>> >
>> >
>> >
>> > # The maximum number of active DAG runs per DAG
>> >
>> > *max_active_runs_per_dag = 32*
>> >
>> >
>> >
>> > Thank you,
>> >
>> > Best regards.
>> >
>> > Tomasz Urbaszek <tu...@apache.org>, 19 Şub 2020 Çar, 13:12
>> tarihinde
>> > şunu yazdı:
>> >
>> > > Can you please tell me more about your environment? Especially how do
>> > > you sync your DAGs / logs from celery workers? I know one setup where
>> > > I've seen I/O error when wiritting to a log...
>> > >
>> > > T.
>> > >
>> > >
>> > > On Wed, Feb 19, 2020 at 10:56 AM Mehmet Ersoy
>> > > <me...@gmail.com> wrote:
>> > > >
>> > > > Hello Friends,
>> > > >
>> > > > I'm new to Airflow and I'm using Airflow Celery executor with
>> Postgres
>> > > backend and Redis Message Queue service. For now, there is 4 worker, 1
>> > > Scheduler and 1 Web Server.
>> > > > I have been preparing parallel Sqoop Jobs in my daily DAGs.
>> > > > When I scheduled a daily DAG, Often some task instances turning to
>> > > failed without running state after started state. Then I can't see the
>> > logs
>> > > of them. It's blank. And when I run that task manually it's running
>> > without
>> > > any problem.
>> > > > I don't really understand if there is an inconsistent situation in
>> my
>> > > DAG writing.
>> > > > I have attached one of my DAGs.
>> > > >
>> > > > Thank you in advance,
>> > > > Best Regards.
>> > > > Mehmet.
>> > >
>> >
>> >
>> > --
>> > Mehmet ERSOY
>> >
>>
>>
>> --
>>
>> Tomasz Urbaszek
>> Polidea <https://www.polidea.com/> | Software Engineer
>>
>> M: +48 505 628 493 <+48505628493>
>> E: tomasz.urbaszek@polidea.com <to...@polidea.com>
>>
>> Unique Tech
>> Check out our projects! <https://www.polidea.com/our-work>
>>
>
>
> --
> Mehmet ERSOY
>
--
Mehmet ERSOY
Re: Airflow Celery Executor - Some Task instances turning to failed
without running and exporting log
Posted by Mehmet Ersoy <me...@gmail.com>.
I'm using 1.10.6 version of Airflow.
Yes, this problem occurs in all of my parallel DAGs. And I attached one of
my DAGs in my first mail. In addition, graph view of my DAG is as follows:
[image: image.png]
Thanks.
Tomasz Urbaszek <to...@polidea.com>, 19 Şub 2020 Çar, 15:09
tarihinde şunu yazdı:
> What version of Airflow do you use?
>
> The mentioned DAG is missing but I'm curious about "parallel" jobs you are
> running :) Does this problem occur with only one DAG?
>
> T.
>
> On Wed, Feb 19, 2020 at 12:58 PM Mehmet Ersoy <me...@gmail.com>
> wrote:
>
> > Hi Tomasz,
> > For now, I'm syncing the DAGs manually sending across airflow hosts. So,
> > for now there is no git repository etc.
> > In addition, My configs related with parallelism are as follows:
> >
> >
> >
> > # How many processes CeleryExecutor uses to sync task state.
> >
> > # 0 means to use max(1, number of cores - 1) processes.
> >
> > *sync_parallelism = 0*
> >
> >
> >
> >
> >
> > # The amount of parallelism as a setting to the executor. This defines
> >
> > # the max number of task instances that should run simultaneously
> >
> > # on this airflow installation
> >
> > *parallelism = 128*
> >
> >
> >
> >
> >
> > # The number of task instances allowed to run concurrently by the
> scheduler
> >
> > *dag_concurrency = 128*
> >
> >
> >
> >
> >
> > # The concurrency that will be used when starting workers with the
> >
> > # "airflow worker" command. This defines the number of task instances
> that
> >
> > # a worker will take, so size up your workers based on the resources on
> >
> > # your worker box and the nature of your tasks
> >
> > *worker_concurrency = 32*
> >
> >
> >
> >
> >
> > # The maximum number of active DAG runs per DAG
> >
> > *max_active_runs_per_dag = 32*
> >
> >
> >
> > Thank you,
> >
> > Best regards.
> >
> > Tomasz Urbaszek <tu...@apache.org>, 19 Şub 2020 Çar, 13:12 tarihinde
> > şunu yazdı:
> >
> > > Can you please tell me more about your environment? Especially how do
> > > you sync your DAGs / logs from celery workers? I know one setup where
> > > I've seen I/O error when wiritting to a log...
> > >
> > > T.
> > >
> > >
> > > On Wed, Feb 19, 2020 at 10:56 AM Mehmet Ersoy
> > > <me...@gmail.com> wrote:
> > > >
> > > > Hello Friends,
> > > >
> > > > I'm new to Airflow and I'm using Airflow Celery executor with
> Postgres
> > > backend and Redis Message Queue service. For now, there is 4 worker, 1
> > > Scheduler and 1 Web Server.
> > > > I have been preparing parallel Sqoop Jobs in my daily DAGs.
> > > > When I scheduled a daily DAG, Often some task instances turning to
> > > failed without running state after started state. Then I can't see the
> > logs
> > > of them. It's blank. And when I run that task manually it's running
> > without
> > > any problem.
> > > > I don't really understand if there is an inconsistent situation in my
> > > DAG writing.
> > > > I have attached one of my DAGs.
> > > >
> > > > Thank you in advance,
> > > > Best Regards.
> > > > Mehmet.
> > >
> >
> >
> > --
> > Mehmet ERSOY
> >
>
>
> --
>
> Tomasz Urbaszek
> Polidea <https://www.polidea.com/> | Software Engineer
>
> M: +48 505 628 493 <+48505628493>
> E: tomasz.urbaszek@polidea.com <to...@polidea.com>
>
> Unique Tech
> Check out our projects! <https://www.polidea.com/our-work>
>
--
Mehmet ERSOY
Re: Airflow Celery Executor - Some Task instances turning to failed
without running and exporting log
Posted by Tomasz Urbaszek <to...@polidea.com>.
What version of Airflow do you use?
The mentioned DAG is missing but I'm curious about "parallel" jobs you are
running :) Does this problem occur with only one DAG?
T.
On Wed, Feb 19, 2020 at 12:58 PM Mehmet Ersoy <me...@gmail.com>
wrote:
> Hi Tomasz,
> For now, I'm syncing the DAGs manually sending across airflow hosts. So,
> for now there is no git repository etc.
> In addition, My configs related with parallelism are as follows:
>
>
>
> # How many processes CeleryExecutor uses to sync task state.
>
> # 0 means to use max(1, number of cores - 1) processes.
>
> *sync_parallelism = 0*
>
>
>
>
>
> # The amount of parallelism as a setting to the executor. This defines
>
> # the max number of task instances that should run simultaneously
>
> # on this airflow installation
>
> *parallelism = 128*
>
>
>
>
>
> # The number of task instances allowed to run concurrently by the scheduler
>
> *dag_concurrency = 128*
>
>
>
>
>
> # The concurrency that will be used when starting workers with the
>
> # "airflow worker" command. This defines the number of task instances that
>
> # a worker will take, so size up your workers based on the resources on
>
> # your worker box and the nature of your tasks
>
> *worker_concurrency = 32*
>
>
>
>
>
> # The maximum number of active DAG runs per DAG
>
> *max_active_runs_per_dag = 32*
>
>
>
> Thank you,
>
> Best regards.
>
> Tomasz Urbaszek <tu...@apache.org>, 19 Şub 2020 Çar, 13:12 tarihinde
> şunu yazdı:
>
> > Can you please tell me more about your environment? Especially how do
> > you sync your DAGs / logs from celery workers? I know one setup where
> > I've seen I/O error when wiritting to a log...
> >
> > T.
> >
> >
> > On Wed, Feb 19, 2020 at 10:56 AM Mehmet Ersoy
> > <me...@gmail.com> wrote:
> > >
> > > Hello Friends,
> > >
> > > I'm new to Airflow and I'm using Airflow Celery executor with Postgres
> > backend and Redis Message Queue service. For now, there is 4 worker, 1
> > Scheduler and 1 Web Server.
> > > I have been preparing parallel Sqoop Jobs in my daily DAGs.
> > > When I scheduled a daily DAG, Often some task instances turning to
> > failed without running state after started state. Then I can't see the
> logs
> > of them. It's blank. And when I run that task manually it's running
> without
> > any problem.
> > > I don't really understand if there is an inconsistent situation in my
> > DAG writing.
> > > I have attached one of my DAGs.
> > >
> > > Thank you in advance,
> > > Best Regards.
> > > Mehmet.
> >
>
>
> --
> Mehmet ERSOY
>
--
Tomasz Urbaszek
Polidea <https://www.polidea.com/> | Software Engineer
M: +48 505 628 493 <+48505628493>
E: tomasz.urbaszek@polidea.com <to...@polidea.com>
Unique Tech
Check out our projects! <https://www.polidea.com/our-work>
Re: Airflow Celery Executor - Some Task instances turning to failed
without running and exporting log
Posted by Mehmet Ersoy <me...@gmail.com>.
Hi Tomasz,
For now, I'm syncing the DAGs manually sending across airflow hosts. So,
for now there is no git repository etc.
In addition, My configs related with parallelism are as follows:
# How many processes CeleryExecutor uses to sync task state.
# 0 means to use max(1, number of cores - 1) processes.
*sync_parallelism = 0*
# The amount of parallelism as a setting to the executor. This defines
# the max number of task instances that should run simultaneously
# on this airflow installation
*parallelism = 128*
# The number of task instances allowed to run concurrently by the scheduler
*dag_concurrency = 128*
# The concurrency that will be used when starting workers with the
# "airflow worker" command. This defines the number of task instances that
# a worker will take, so size up your workers based on the resources on
# your worker box and the nature of your tasks
*worker_concurrency = 32*
# The maximum number of active DAG runs per DAG
*max_active_runs_per_dag = 32*
Thank you,
Best regards.
Tomasz Urbaszek <tu...@apache.org>, 19 Şub 2020 Çar, 13:12 tarihinde
şunu yazdı:
> Can you please tell me more about your environment? Especially how do
> you sync your DAGs / logs from celery workers? I know one setup where
> I've seen I/O error when wiritting to a log...
>
> T.
>
>
> On Wed, Feb 19, 2020 at 10:56 AM Mehmet Ersoy
> <me...@gmail.com> wrote:
> >
> > Hello Friends,
> >
> > I'm new to Airflow and I'm using Airflow Celery executor with Postgres
> backend and Redis Message Queue service. For now, there is 4 worker, 1
> Scheduler and 1 Web Server.
> > I have been preparing parallel Sqoop Jobs in my daily DAGs.
> > When I scheduled a daily DAG, Often some task instances turning to
> failed without running state after started state. Then I can't see the logs
> of them. It's blank. And when I run that task manually it's running without
> any problem.
> > I don't really understand if there is an inconsistent situation in my
> DAG writing.
> > I have attached one of my DAGs.
> >
> > Thank you in advance,
> > Best Regards.
> > Mehmet.
>
--
Mehmet ERSOY
Re: Airflow Celery Executor - Some Task instances turning to failed
without running and exporting log
Posted by Tomasz Urbaszek <tu...@apache.org>.
Can you please tell me more about your environment? Especially how do
you sync your DAGs / logs from celery workers? I know one setup where
I've seen I/O error when wiritting to a log...
T.
On Wed, Feb 19, 2020 at 10:56 AM Mehmet Ersoy
<me...@gmail.com> wrote:
>
> Hello Friends,
>
> I'm new to Airflow and I'm using Airflow Celery executor with Postgres backend and Redis Message Queue service. For now, there is 4 worker, 1 Scheduler and 1 Web Server.
> I have been preparing parallel Sqoop Jobs in my daily DAGs.
> When I scheduled a daily DAG, Often some task instances turning to failed without running state after started state. Then I can't see the logs of them. It's blank. And when I run that task manually it's running without any problem.
> I don't really understand if there is an inconsistent situation in my DAG writing.
> I have attached one of my DAGs.
>
> Thank you in advance,
> Best Regards.
> Mehmet.