Using large numbers of sensors, resource consumption

[Pedro Machado <pe...@205datalab.com>]

I have a few DAGs that use time sensors to wait until data is ready, which
can be several days.

I have one daily DAG where, for each execution date, I have to repull the
data for the next 7 days to capture changes (late arriving revenue data).
This DAG currently starts 7 TimeDeltaSensors for each execution days with
delays that range from 0 to 6 days.

I was wondering what the recommendation is for cases like this where a
large number of sensors is needed.

Are there ways to reduce the footprint of these sensors so that they use
less CPU and memory?

I noticed that in one of the DAGs that Germain Tanguy had in the
presentation he shared today a sensor was set to time out every 30 seconds
but had a large retry count so instead of running constantly, it runs every
15 minutes for 30 seconds and then dies.

Are other people using this pattern? Do you have other suggestions?

Thanks,

Pedro

Re: Using large numbers of sensors, resource consumption

[Alexander Shorin <kx...@gmail.com>]

On Tue, Jul 10, 2018 at 5:05 PM, Pedro Machado <pe...@205datalab.com> wrote:
> Are there ways to reduce the footprint of these sensors so that they use
> less CPU and memory?

Sensors are basically long running I/O polling tasks. In some cases
they are sleep-based tasks. Both lands perfectly on asyncio: instead
of spawning full features OS process which runs a sensor, you need
just a single process which spawns coroutines instead and takes care
about them. This will reduce resource consumption a lot when you run
hundreds sensors. And no need to dance with retries and timeouts,
workers starvation wouldn't be a problem as well.

--
,,,^..^,,,

Re: Using large numbers of sensors, resource consumption

[Ash Berlin-Taylor <as...@firemirror.com>]

We are also using the "high number of retries" pattern rather than sensors (S3KeySensor in our case) for similar reason - we have data that arrives for a week "some point after Thursday midnight" -- but that can take 5 or even 8 days for it to arrive. Yay third parties.

It would be nice to have a different kind of sensor (or a flag to the existing ones) so that rather than sitting in a busy loop on an executor they just go back and re-schedule themselves. We've just not gotten around to writing that (we being where I work).

-ash

> On 10 Jul 2018, at 15:05, Pedro Machado <pe...@205datalab.com> wrote:
> 
> I have a few DAGs that use time sensors to wait until data is ready, which
> can be several days.
> 
> I have one daily DAG where, for each execution date, I have to repull the
> data for the next 7 days to capture changes (late arriving revenue data).
> This DAG currently starts 7 TimeDeltaSensors for each execution days with
> delays that range from 0 to 6 days.
> 
> I was wondering what the recommendation is for cases like this where a
> large number of sensors is needed.
> 
> Are there ways to reduce the footprint of these sensors so that they use
> less CPU and memory?
> 
> I noticed that in one of the DAGs that Germain Tanguy had in the
> presentation he shared today a sensor was set to time out every 30 seconds
> but had a large retry count so instead of running constantly, it runs every
> 15 minutes for 30 seconds and then dies.
> 
> Are other people using this pattern? Do you have other suggestions?
> 
> Thanks,
> 
> Pedro

Re: Using large numbers of sensors, resource consumption

[Ash Berlin-Taylor <as...@firemirror.com>]

We are also using the "high number of retries" pattern rather than sensors (S3KeySensor in our case) for similar reason - we have data that arrives for a week "some point after Thursday midnight" -- but that can take 5 or even 8 days for it to arrive. Yay third parties.

It would be nice to have a different kind of sensor (or a flag to the existing ones) so that rather than sitting in a busy loop on an executor they just go back and re-schedule themselves. We've just not gotten around to writing that (we being where I work).

-ash

> On 10 Jul 2018, at 15:05, Pedro Machado <pe...@205datalab.com> wrote:
> 
> I have a few DAGs that use time sensors to wait until data is ready, which
> can be several days.
> 
> I have one daily DAG where, for each execution date, I have to repull the
> data for the next 7 days to capture changes (late arriving revenue data).
> This DAG currently starts 7 TimeDeltaSensors for each execution days with
> delays that range from 0 to 6 days.
> 
> I was wondering what the recommendation is for cases like this where a
> large number of sensors is needed.
> 
> Are there ways to reduce the footprint of these sensors so that they use
> less CPU and memory?
> 
> I noticed that in one of the DAGs that Germain Tanguy had in the
> presentation he shared today a sensor was set to time out every 30 seconds
> but had a large retry count so instead of running constantly, it runs every
> 15 minutes for 30 seconds and then dies.
> 
> Are other people using this pattern? Do you have other suggestions?
> 
> Thanks,
> 
> Pedro

Re: Using large numbers of sensors, resource consumption

[Maxime Beauchemin <ma...@gmail.com>]

There have been conversations in the past around the idea of adding an
`evaluation_method` argument in BaseSensor that would allow for different
options:
1. the current approach which is taking up a slot and poking periodically
(heavy on slot usage)
2. one approach closer to fail/retry approach, likely introducing a new
state representing that it's waiting for the next sensing event (heavy on
overhead, MQ traffic, ...)
3. one where the scheduler itself runs the "poke" method in line, in many
cases it represents very little overhead for the scheduler to run that
task, and the scheduler is already insulated (DAG is parsed in a sub
process) (heavy on the scheduler machine). I think it's reasonable to do
this even without a distributed scheduler, especially for cheap-to-check
sensors.

Another way to mitigate resources that we used at Airbnb is to have a
dedicated sensor queue with machines that are provisioned more aggressively
(say 16 or 32 slots per CPU core), and route the cheap sensing tasks to
those machines.

Max

On Thu, Jul 12, 2018 at 11:51 AM Pedro Machado <pe...@205datalab.com> wrote:

> Thanks, Ash, Alexander, and Stefan for your replies.
>
> I am relatively new to airflow and not familiar with the code base. I like
> the idea of having a more efficient sensor.
>
> The async approach makes sense, but I don't know how well it would fit
> within the existing architecture.
>
> I like that Stefan's "reschedule" approach can fit the current architecture
> and could be implemented sooner. From the user point of view, my only
> feedback is that the UI should not show sensors that are still running as
> failed or up for retry as that would draw attention to things that are
> running as expected. I'll add this comment to the JIRA issue.
>
> Thanks!
>
> Pedro
>
>
> On Tue, Jul 10, 2018 at 9:44 AM Stefan Seelmann <ma...@stefan-seelmann.de>
> wrote:
>
> > I also have that requirement and I'm working on a proposal for
> > rescheduling tasks. My current PoC can be found at [1] which uses
> > up_for_retry state which has some problems. I started to make some
> > changes, I hope can make a first proposal this week.
> >
> > The basic idea is:
> > * A new "reschedule" flag for sensors, if set to True it will raise an
> > AirflowRescheduleException (with the new schedule date) that causes a
> > reschedule
> > * Reschedule requests are recorded in new `task_reschedule` table and
> > visualized in the Gantt view.
> > * A new TI dependency that checks if a task is ready to be re-scheduled
> >
> > Advantages:
> > * This change is backward compatible. Existing sensors behave like
> > before. But it's possible to set the "reschedule" flag.
> > * The timeout and poke_interval are still respected and used to
> > calculate the next schedule time
> > * Custom sensor implementations can even define the next sensible
> > schedule date.
> > * This mechanism can also be used by non-sensor operators
> >
> > Kind Regards,
> > Stefan
> >
> > [1]
> https://github.com/seelmann/incubator-airflow/tree/reschedule-sensor-3
> >
> > On 07/10/2018 04:05 PM, Pedro Machado wrote:
> > > I have a few DAGs that use time sensors to wait until data is ready,
> > which
> > > can be several days.
> > >
> > > I have one daily DAG where, for each execution date, I have to repull
> the
> > > data for the next 7 days to capture changes (late arriving revenue
> data).
> > > This DAG currently starts 7 TimeDeltaSensors for each execution days
> with
> > > delays that range from 0 to 6 days.
> > >
> > > I was wondering what the recommendation is for cases like this where a
> > > large number of sensors is needed.
> > >
> > > Are there ways to reduce the footprint of these sensors so that they
> use
> > > less CPU and memory?
> > >
> > > I noticed that in one of the DAGs that Germain Tanguy had in the
> > > presentation he shared today a sensor was set to time out every 30
> > seconds
> > > but had a large retry count so instead of running constantly, it runs
> > every
> > > 15 minutes for 30 seconds and then dies.
> > >
> > > Are other people using this pattern? Do you have other suggestions?
> > >
> > > Thanks,
> > >
> > > Pedro
> > >
> >
> >
>

Re: Using large numbers of sensors, resource consumption

[Pedro Machado <pe...@205datalab.com>]

Thanks, Ash, Alexander, and Stefan for your replies.

I am relatively new to airflow and not familiar with the code base. I like
the idea of having a more efficient sensor.

The async approach makes sense, but I don't know how well it would fit
within the existing architecture.

I like that Stefan's "reschedule" approach can fit the current architecture
and could be implemented sooner. From the user point of view, my only
feedback is that the UI should not show sensors that are still running as
failed or up for retry as that would draw attention to things that are
running as expected. I'll add this comment to the JIRA issue.

Thanks!

Pedro


On Tue, Jul 10, 2018 at 9:44 AM Stefan Seelmann <ma...@stefan-seelmann.de>
wrote:

> I also have that requirement and I'm working on a proposal for
> rescheduling tasks. My current PoC can be found at [1] which uses
> up_for_retry state which has some problems. I started to make some
> changes, I hope can make a first proposal this week.
>
> The basic idea is:
> * A new "reschedule" flag for sensors, if set to True it will raise an
> AirflowRescheduleException (with the new schedule date) that causes a
> reschedule
> * Reschedule requests are recorded in new `task_reschedule` table and
> visualized in the Gantt view.
> * A new TI dependency that checks if a task is ready to be re-scheduled
>
> Advantages:
> * This change is backward compatible. Existing sensors behave like
> before. But it's possible to set the "reschedule" flag.
> * The timeout and poke_interval are still respected and used to
> calculate the next schedule time
> * Custom sensor implementations can even define the next sensible
> schedule date.
> * This mechanism can also be used by non-sensor operators
>
> Kind Regards,
> Stefan
>
> [1] https://github.com/seelmann/incubator-airflow/tree/reschedule-sensor-3
>
> On 07/10/2018 04:05 PM, Pedro Machado wrote:
> > I have a few DAGs that use time sensors to wait until data is ready,
> which
> > can be several days.
> >
> > I have one daily DAG where, for each execution date, I have to repull the
> > data for the next 7 days to capture changes (late arriving revenue data).
> > This DAG currently starts 7 TimeDeltaSensors for each execution days with
> > delays that range from 0 to 6 days.
> >
> > I was wondering what the recommendation is for cases like this where a
> > large number of sensors is needed.
> >
> > Are there ways to reduce the footprint of these sensors so that they use
> > less CPU and memory?
> >
> > I noticed that in one of the DAGs that Germain Tanguy had in the
> > presentation he shared today a sensor was set to time out every 30
> seconds
> > but had a large retry count so instead of running constantly, it runs
> every
> > 15 minutes for 30 seconds and then dies.
> >
> > Are other people using this pattern? Do you have other suggestions?
> >
> > Thanks,
> >
> > Pedro
> >
>
>

Re: Using large numbers of sensors, resource consumption

[Stefan Seelmann <ma...@stefan-seelmann.de>]

I also have that requirement and I'm working on a proposal for
rescheduling tasks. My current PoC can be found at [1] which uses
up_for_retry state which has some problems. I started to make some
changes, I hope can make a first proposal this week.

The basic idea is:
* A new "reschedule" flag for sensors, if set to True it will raise an
AirflowRescheduleException (with the new schedule date) that causes a
reschedule
* Reschedule requests are recorded in new `task_reschedule` table and
visualized in the Gantt view.
* A new TI dependency that checks if a task is ready to be re-scheduled

Advantages:
* This change is backward compatible. Existing sensors behave like
before. But it's possible to set the "reschedule" flag.
* The timeout and poke_interval are still respected and used to
calculate the next schedule time
* Custom sensor implementations can even define the next sensible
schedule date.
* This mechanism can also be used by non-sensor operators

Kind Regards,
Stefan

[1] https://github.com/seelmann/incubator-airflow/tree/reschedule-sensor-3

On 07/10/2018 04:05 PM, Pedro Machado wrote:
> I have a few DAGs that use time sensors to wait until data is ready, which
> can be several days.
> 
> I have one daily DAG where, for each execution date, I have to repull the
> data for the next 7 days to capture changes (late arriving revenue data).
> This DAG currently starts 7 TimeDeltaSensors for each execution days with
> delays that range from 0 to 6 days.
> 
> I was wondering what the recommendation is for cases like this where a
> large number of sensors is needed.
> 
> Are there ways to reduce the footprint of these sensors so that they use
> less CPU and memory?
> 
> I noticed that in one of the DAGs that Germain Tanguy had in the
> presentation he shared today a sensor was set to time out every 30 seconds
> but had a large retry count so instead of running constantly, it runs every
> 15 minutes for 30 seconds and then dies.
> 
> Are other people using this pattern? Do you have other suggestions?
> 
> Thanks,
> 
> Pedro
>