[Discuss] Airflow sensor optimization

[Yingbo Wang <yb...@gmail.com>]

hi,

I would like to open an AIP for Airflow sensor optimization.


*Motivation*:

Low efficiency in Airflow Sensor Implementation

Sensors are a special kind of operator that will keep running until a
certain criterion is met. Examples include a specific file landing in HDFS
or S3, a partition appearing in Hive, or a specific time of the day.
Sensors are derived from BaseSensorOperator and run a poke method at a
specified poke_interval until it returns True.

The reason that the sensor tasks are inefficient is because in current
design, we sprawn a separate worker process for each partition sensor. This
worker might last a long time, until the target partition is available.  In
the case where there are many sensor tasks that need to run within certain
time limits, we have to allocate a lot of resources to have enough workers
for the sensor tasks.

*Idea:*

We propose two approaches that could address this issues, batch-sensor
and smart-sensor.



Batch-sensor

The basic idea of batch-sensor is to batch process sensor tasks to save
resources. During running, a batch-sensor will take N partition sensor
requests as the input and poke those N partitions periodically. If the
batch-sensor finds that the criteria of some sensor task is met, the
batch-sensor will update the database about this sensor tasks.


To do this, we need to create a sensor basic class called ‘batchable’ and
make all sensors inherit from this basic class. We also need to change the
behavior of schedule regarding a batchable sensor tasks. The schedule will
find as many as possible batchable sensor tasks and run those tasks in a
batch.


Smart-sensor

Smart-sensor is an improvement on top of batch-sensor.

The idea of smart-sensor is that the worker process of smart-sensor will
run like a service. To do this, we need to persist Sensor details in
Airflow DB and the worker process periodically queries task-instance table
to find sensor tasks; poke the metastore and update the task instance table
if it detects that certain partition or file created.

Re: [Discuss] Airflow sensor optimization

[Alex Guziel <al...@airbnb.com.INVALID>]

Sensor-service thing seems to open the door to make sensors a pubsub-type
deal where possible. For example, in Hive, you can keep an in-memory
registry of what partitions to sense for, and tail the audit log to see
when they are populated, instead of polling.

On Wed, Mar 6, 2019 at 1:51 PM Alex Guziel <al...@airbnb.com> wrote:

> Smart sensor seems like a good idea, but I wonder how much performance
> will be improved in practice. And of course, one must think about sharding
> and such.
>
> I'm not sure how helpful rescheduling sensors is, since it will add
> scheduler and DB load seemingly, which is already a bottleneck.
>
> On Wed, Mar 6, 2019 at 12:43 PM Yingbo Wang <yb...@gmail.com> wrote:
>
>> I would still like to get some feedback on the batch sensor/smart sensor
>> idea after viewing the sensor rescheduling PR. Since the reschedule mode
>> does not reduce the number of worker processes for sensor. The batch
>> sensor
>> idea is proposed for this purpose and should work well with reschedule
>> mode.
>>
>> On Wed, Mar 6, 2019 at 11:30 AM Yingbo Wang <yb...@gmail.com> wrote:
>>
>> > Wow, Great work from Seelmann! Thanks Fokko for letting us know it. We
>> are
>> > super happy to have this feature.
>> >
>> > On Wed, Mar 6, 2019 at 11:24 AM Driesprong, Fokko <fokko@driesprong.frl
>> >
>> > wrote:
>> >
>> >> Thanks for bringing this up. I've added a comment on the Wiki:
>> >>
>> >>
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization
>> >>
>> >> Have you looked into the work by Seelmann? Recently he introduced the
>> >> ability to reschedule sensors. When rescheduling, the slot will be
>> given
>> >> back to the scheduler after a poke operation. Therefore the slot won't
>> be
>> >> occupied all the time. The details are in the PR
>> >> https://github.com/apache/airflow/pull/3596
>> >>
>> >> I would propose to make this the default behavior in Airflow 2.0.
>> >>
>> >> Cheers, Fokko
>> >>
>> >> Op wo 6 mrt. 2019 om 15:32 schreef Yingbo Wang <yb...@gmail.com>:
>> >>
>> >> > hi,
>> >> >
>> >> > I would like to open an AIP for Airflow sensor optimization.
>> >> >
>> >> >
>> >> > *Motivation*:
>> >> >
>> >> > Low efficiency in Airflow Sensor Implementation
>> >> >
>> >> > Sensors are a special kind of operator that will keep running until a
>> >> > certain criterion is met. Examples include a specific file landing in
>> >> HDFS
>> >> > or S3, a partition appearing in Hive, or a specific time of the day.
>> >> > Sensors are derived from BaseSensorOperator and run a poke method at
>> a
>> >> > specified poke_interval until it returns True.
>> >> >
>> >> > The reason that the sensor tasks are inefficient is because in
>> current
>> >> > design, we sprawn a separate worker process for each partition
>> sensor.
>> >> This
>> >> > worker might last a long time, until the target partition is
>> >> available.  In
>> >> > the case where there are many sensor tasks that need to run within
>> >> certain
>> >> > time limits, we have to allocate a lot of resources to have enough
>> >> workers
>> >> > for the sensor tasks.
>> >> >
>> >> > *Idea:*
>> >> >
>> >> > We propose two approaches that could address this issues,
>> batch-sensor
>> >> > and smart-sensor.
>> >> >
>> >> >
>> >> >
>> >> > Batch-sensor
>> >> >
>> >> > The basic idea of batch-sensor is to batch process sensor tasks to
>> save
>> >> > resources. During running, a batch-sensor will take N partition
>> sensor
>> >> > requests as the input and poke those N partitions periodically. If
>> the
>> >> > batch-sensor finds that the criteria of some sensor task is met, the
>> >> > batch-sensor will update the database about this sensor tasks.
>> >> >
>> >> >
>> >> > To do this, we need to create a sensor basic class called ‘batchable’
>> >> and
>> >> > make all sensors inherit from this basic class. We also need to
>> change
>> >> the
>> >> > behavior of schedule regarding a batchable sensor tasks. The schedule
>> >> will
>> >> > find as many as possible batchable sensor tasks and run those tasks
>> in a
>> >> > batch.
>> >> >
>> >> >
>> >> > Smart-sensor
>> >> >
>> >> > Smart-sensor is an improvement on top of batch-sensor.
>> >> >
>> >> > The idea of smart-sensor is that the worker process of smart-sensor
>> will
>> >> > run like a service. To do this, we need to persist Sensor details in
>> >> > Airflow DB and the worker process periodically queries task-instance
>> >> table
>> >> > to find sensor tasks; poke the metastore and update the task instance
>> >> table
>> >> > if it detects that certain partition or file created.
>> >> >
>> >>
>> >
>>
>

Re: [Discuss] Airflow sensor optimization

[Yingbo Wang <yb...@gmail.com>]

There are two dimension to evaluate how much resource all sensors take in
Airflow: the number of sensors and the duration of each sensor task take.
Batch/smart sensor idea is proposed for the first one and the rescheduling
is for the second one. For airflow cluster running large number of sensor
tasks, the batch/smart sensor use less than 10% of sensor resource compared
with regular sensor.

On Thu, Mar 7, 2019 at 2:36 AM Ash Berlin-Taylor <as...@apache.org> wrote:

> Rescheduling is of massive use for a DAG where we are waiting for a weekly
> S3 file delivery from a third party supplier with _massive_ variance in the
> delivery time. It'll appear at some point between Thursday AM and Sunday
> evening. Not having an executor slot tied up with the S3KeySensor is great
> for this.
>
> -ash
>
> > On 6 Mar 2019, at 21:51, Alex Guziel <al...@airbnb.com.INVALID>
> wrote:
> >
> > Smart sensor seems like a good idea, but I wonder how much performance
> will
> > be improved in practice. And of course, one must think about sharding and
> > such.
> >
> > I'm not sure how helpful rescheduling sensors is, since it will add
> > scheduler and DB load seemingly, which is already a bottleneck.
> >
> > On Wed, Mar 6, 2019 at 12:43 PM Yingbo Wang <yb...@gmail.com> wrote:
> >
> >> I would still like to get some feedback on the batch sensor/smart sensor
> >> idea after viewing the sensor rescheduling PR. Since the reschedule mode
> >> does not reduce the number of worker processes for sensor. The batch
> sensor
> >> idea is proposed for this purpose and should work well with reschedule
> >> mode.
> >>
> >> On Wed, Mar 6, 2019 at 11:30 AM Yingbo Wang <yb...@gmail.com> wrote:
> >>
> >>> Wow, Great work from Seelmann! Thanks Fokko for letting us know it. We
> >> are
> >>> super happy to have this feature.
> >>>
> >>> On Wed, Mar 6, 2019 at 11:24 AM Driesprong, Fokko <fokko@driesprong.frl
> >
> >>> wrote:
> >>>
> >>>> Thanks for bringing this up. I've added a comment on the Wiki:
> >>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization
> >>>>
> >>>> Have you looked into the work by Seelmann? Recently he introduced the
> >>>> ability to reschedule sensors. When rescheduling, the slot will be
> given
> >>>> back to the scheduler after a poke operation. Therefore the slot won't
> >> be
> >>>> occupied all the time. The details are in the PR
> >>>> https://github.com/apache/airflow/pull/3596
> >>>>
> >>>> I would propose to make this the default behavior in Airflow 2.0.
> >>>>
> >>>> Cheers, Fokko
> >>>>
> >>>> Op wo 6 mrt. 2019 om 15:32 schreef Yingbo Wang <yb...@gmail.com>:
> >>>>
> >>>>> hi,
> >>>>>
> >>>>> I would like to open an AIP for Airflow sensor optimization.
> >>>>>
> >>>>>
> >>>>> *Motivation*:
> >>>>>
> >>>>> Low efficiency in Airflow Sensor Implementation
> >>>>>
> >>>>> Sensors are a special kind of operator that will keep running until a
> >>>>> certain criterion is met. Examples include a specific file landing in
> >>>> HDFS
> >>>>> or S3, a partition appearing in Hive, or a specific time of the day.
> >>>>> Sensors are derived from BaseSensorOperator and run a poke method at
> a
> >>>>> specified poke_interval until it returns True.
> >>>>>
> >>>>> The reason that the sensor tasks are inefficient is because in
> current
> >>>>> design, we sprawn a separate worker process for each partition
> sensor.
> >>>> This
> >>>>> worker might last a long time, until the target partition is
> >>>> available.  In
> >>>>> the case where there are many sensor tasks that need to run within
> >>>> certain
> >>>>> time limits, we have to allocate a lot of resources to have enough
> >>>> workers
> >>>>> for the sensor tasks.
> >>>>>
> >>>>> *Idea:*
> >>>>>
> >>>>> We propose two approaches that could address this issues,
> batch-sensor
> >>>>> and smart-sensor.
> >>>>>
> >>>>>
> >>>>>
> >>>>> Batch-sensor
> >>>>>
> >>>>> The basic idea of batch-sensor is to batch process sensor tasks to
> >> save
> >>>>> resources. During running, a batch-sensor will take N partition
> sensor
> >>>>> requests as the input and poke those N partitions periodically. If
> the
> >>>>> batch-sensor finds that the criteria of some sensor task is met, the
> >>>>> batch-sensor will update the database about this sensor tasks.
> >>>>>
> >>>>>
> >>>>> To do this, we need to create a sensor basic class called ‘batchable’
> >>>> and
> >>>>> make all sensors inherit from this basic class. We also need to
> change
> >>>> the
> >>>>> behavior of schedule regarding a batchable sensor tasks. The schedule
> >>>> will
> >>>>> find as many as possible batchable sensor tasks and run those tasks
> >> in a
> >>>>> batch.
> >>>>>
> >>>>>
> >>>>> Smart-sensor
> >>>>>
> >>>>> Smart-sensor is an improvement on top of batch-sensor.
> >>>>>
> >>>>> The idea of smart-sensor is that the worker process of smart-sensor
> >> will
> >>>>> run like a service. To do this, we need to persist Sensor details in
> >>>>> Airflow DB and the worker process periodically queries task-instance
> >>>> table
> >>>>> to find sensor tasks; poke the metastore and update the task instance
> >>>> table
> >>>>> if it detects that certain partition or file created.
> >>>>>
> >>>>
> >>>
> >>
>
>

Re: [Discuss] Airflow sensor optimization

[Ash Berlin-Taylor <as...@apache.org>]

Rescheduling is of massive use for a DAG where we are waiting for a weekly S3 file delivery from a third party supplier with _massive_ variance in the delivery time. It'll appear at some point between Thursday AM and Sunday evening. Not having an executor slot tied up with the S3KeySensor is great for this.

-ash

> On 6 Mar 2019, at 21:51, Alex Guziel <al...@airbnb.com.INVALID> wrote:
> 
> Smart sensor seems like a good idea, but I wonder how much performance will
> be improved in practice. And of course, one must think about sharding and
> such.
> 
> I'm not sure how helpful rescheduling sensors is, since it will add
> scheduler and DB load seemingly, which is already a bottleneck.
> 
> On Wed, Mar 6, 2019 at 12:43 PM Yingbo Wang <yb...@gmail.com> wrote:
> 
>> I would still like to get some feedback on the batch sensor/smart sensor
>> idea after viewing the sensor rescheduling PR. Since the reschedule mode
>> does not reduce the number of worker processes for sensor. The batch sensor
>> idea is proposed for this purpose and should work well with reschedule
>> mode.
>> 
>> On Wed, Mar 6, 2019 at 11:30 AM Yingbo Wang <yb...@gmail.com> wrote:
>> 
>>> Wow, Great work from Seelmann! Thanks Fokko for letting us know it. We
>> are
>>> super happy to have this feature.
>>> 
>>> On Wed, Mar 6, 2019 at 11:24 AM Driesprong, Fokko <fo...@driesprong.frl>
>>> wrote:
>>> 
>>>> Thanks for bringing this up. I've added a comment on the Wiki:
>>>> 
>>>> 
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization
>>>> 
>>>> Have you looked into the work by Seelmann? Recently he introduced the
>>>> ability to reschedule sensors. When rescheduling, the slot will be given
>>>> back to the scheduler after a poke operation. Therefore the slot won't
>> be
>>>> occupied all the time. The details are in the PR
>>>> https://github.com/apache/airflow/pull/3596
>>>> 
>>>> I would propose to make this the default behavior in Airflow 2.0.
>>>> 
>>>> Cheers, Fokko
>>>> 
>>>> Op wo 6 mrt. 2019 om 15:32 schreef Yingbo Wang <yb...@gmail.com>:
>>>> 
>>>>> hi,
>>>>> 
>>>>> I would like to open an AIP for Airflow sensor optimization.
>>>>> 
>>>>> 
>>>>> *Motivation*:
>>>>> 
>>>>> Low efficiency in Airflow Sensor Implementation
>>>>> 
>>>>> Sensors are a special kind of operator that will keep running until a
>>>>> certain criterion is met. Examples include a specific file landing in
>>>> HDFS
>>>>> or S3, a partition appearing in Hive, or a specific time of the day.
>>>>> Sensors are derived from BaseSensorOperator and run a poke method at a
>>>>> specified poke_interval until it returns True.
>>>>> 
>>>>> The reason that the sensor tasks are inefficient is because in current
>>>>> design, we sprawn a separate worker process for each partition sensor.
>>>> This
>>>>> worker might last a long time, until the target partition is
>>>> available.  In
>>>>> the case where there are many sensor tasks that need to run within
>>>> certain
>>>>> time limits, we have to allocate a lot of resources to have enough
>>>> workers
>>>>> for the sensor tasks.
>>>>> 
>>>>> *Idea:*
>>>>> 
>>>>> We propose two approaches that could address this issues, batch-sensor
>>>>> and smart-sensor.
>>>>> 
>>>>> 
>>>>> 
>>>>> Batch-sensor
>>>>> 
>>>>> The basic idea of batch-sensor is to batch process sensor tasks to
>> save
>>>>> resources. During running, a batch-sensor will take N partition sensor
>>>>> requests as the input and poke those N partitions periodically. If the
>>>>> batch-sensor finds that the criteria of some sensor task is met, the
>>>>> batch-sensor will update the database about this sensor tasks.
>>>>> 
>>>>> 
>>>>> To do this, we need to create a sensor basic class called ‘batchable’
>>>> and
>>>>> make all sensors inherit from this basic class. We also need to change
>>>> the
>>>>> behavior of schedule regarding a batchable sensor tasks. The schedule
>>>> will
>>>>> find as many as possible batchable sensor tasks and run those tasks
>> in a
>>>>> batch.
>>>>> 
>>>>> 
>>>>> Smart-sensor
>>>>> 
>>>>> Smart-sensor is an improvement on top of batch-sensor.
>>>>> 
>>>>> The idea of smart-sensor is that the worker process of smart-sensor
>> will
>>>>> run like a service. To do this, we need to persist Sensor details in
>>>>> Airflow DB and the worker process periodically queries task-instance
>>>> table
>>>>> to find sensor tasks; poke the metastore and update the task instance
>>>> table
>>>>> if it detects that certain partition or file created.
>>>>> 
>>>> 
>>> 
>> 

Re: [Discuss] Airflow sensor optimization

[Alex Guziel <al...@airbnb.com.INVALID>]

Smart sensor seems like a good idea, but I wonder how much performance will
be improved in practice. And of course, one must think about sharding and
such.

I'm not sure how helpful rescheduling sensors is, since it will add
scheduler and DB load seemingly, which is already a bottleneck.

On Wed, Mar 6, 2019 at 12:43 PM Yingbo Wang <yb...@gmail.com> wrote:

> I would still like to get some feedback on the batch sensor/smart sensor
> idea after viewing the sensor rescheduling PR. Since the reschedule mode
> does not reduce the number of worker processes for sensor. The batch sensor
> idea is proposed for this purpose and should work well with reschedule
> mode.
>
> On Wed, Mar 6, 2019 at 11:30 AM Yingbo Wang <yb...@gmail.com> wrote:
>
> > Wow, Great work from Seelmann! Thanks Fokko for letting us know it. We
> are
> > super happy to have this feature.
> >
> > On Wed, Mar 6, 2019 at 11:24 AM Driesprong, Fokko <fo...@driesprong.frl>
> > wrote:
> >
> >> Thanks for bringing this up. I've added a comment on the Wiki:
> >>
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization
> >>
> >> Have you looked into the work by Seelmann? Recently he introduced the
> >> ability to reschedule sensors. When rescheduling, the slot will be given
> >> back to the scheduler after a poke operation. Therefore the slot won't
> be
> >> occupied all the time. The details are in the PR
> >> https://github.com/apache/airflow/pull/3596
> >>
> >> I would propose to make this the default behavior in Airflow 2.0.
> >>
> >> Cheers, Fokko
> >>
> >> Op wo 6 mrt. 2019 om 15:32 schreef Yingbo Wang <yb...@gmail.com>:
> >>
> >> > hi,
> >> >
> >> > I would like to open an AIP for Airflow sensor optimization.
> >> >
> >> >
> >> > *Motivation*:
> >> >
> >> > Low efficiency in Airflow Sensor Implementation
> >> >
> >> > Sensors are a special kind of operator that will keep running until a
> >> > certain criterion is met. Examples include a specific file landing in
> >> HDFS
> >> > or S3, a partition appearing in Hive, or a specific time of the day.
> >> > Sensors are derived from BaseSensorOperator and run a poke method at a
> >> > specified poke_interval until it returns True.
> >> >
> >> > The reason that the sensor tasks are inefficient is because in current
> >> > design, we sprawn a separate worker process for each partition sensor.
> >> This
> >> > worker might last a long time, until the target partition is
> >> available.  In
> >> > the case where there are many sensor tasks that need to run within
> >> certain
> >> > time limits, we have to allocate a lot of resources to have enough
> >> workers
> >> > for the sensor tasks.
> >> >
> >> > *Idea:*
> >> >
> >> > We propose two approaches that could address this issues, batch-sensor
> >> > and smart-sensor.
> >> >
> >> >
> >> >
> >> > Batch-sensor
> >> >
> >> > The basic idea of batch-sensor is to batch process sensor tasks to
> save
> >> > resources. During running, a batch-sensor will take N partition sensor
> >> > requests as the input and poke those N partitions periodically. If the
> >> > batch-sensor finds that the criteria of some sensor task is met, the
> >> > batch-sensor will update the database about this sensor tasks.
> >> >
> >> >
> >> > To do this, we need to create a sensor basic class called ‘batchable’
> >> and
> >> > make all sensors inherit from this basic class. We also need to change
> >> the
> >> > behavior of schedule regarding a batchable sensor tasks. The schedule
> >> will
> >> > find as many as possible batchable sensor tasks and run those tasks
> in a
> >> > batch.
> >> >
> >> >
> >> > Smart-sensor
> >> >
> >> > Smart-sensor is an improvement on top of batch-sensor.
> >> >
> >> > The idea of smart-sensor is that the worker process of smart-sensor
> will
> >> > run like a service. To do this, we need to persist Sensor details in
> >> > Airflow DB and the worker process periodically queries task-instance
> >> table
> >> > to find sensor tasks; poke the metastore and update the task instance
> >> table
> >> > if it detects that certain partition or file created.
> >> >
> >>
> >
>

Re: [Discuss] Airflow sensor optimization

[Yingbo Wang <yb...@gmail.com>]

I would still like to get some feedback on the batch sensor/smart sensor
idea after viewing the sensor rescheduling PR. Since the reschedule mode
does not reduce the number of worker processes for sensor. The batch sensor
idea is proposed for this purpose and should work well with reschedule
mode.

On Wed, Mar 6, 2019 at 11:30 AM Yingbo Wang <yb...@gmail.com> wrote:

> Wow, Great work from Seelmann! Thanks Fokko for letting us know it. We are
> super happy to have this feature.
>
> On Wed, Mar 6, 2019 at 11:24 AM Driesprong, Fokko <fo...@driesprong.frl>
> wrote:
>
>> Thanks for bringing this up. I've added a comment on the Wiki:
>>
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization
>>
>> Have you looked into the work by Seelmann? Recently he introduced the
>> ability to reschedule sensors. When rescheduling, the slot will be given
>> back to the scheduler after a poke operation. Therefore the slot won't be
>> occupied all the time. The details are in the PR
>> https://github.com/apache/airflow/pull/3596
>>
>> I would propose to make this the default behavior in Airflow 2.0.
>>
>> Cheers, Fokko
>>
>> Op wo 6 mrt. 2019 om 15:32 schreef Yingbo Wang <yb...@gmail.com>:
>>
>> > hi,
>> >
>> > I would like to open an AIP for Airflow sensor optimization.
>> >
>> >
>> > *Motivation*:
>> >
>> > Low efficiency in Airflow Sensor Implementation
>> >
>> > Sensors are a special kind of operator that will keep running until a
>> > certain criterion is met. Examples include a specific file landing in
>> HDFS
>> > or S3, a partition appearing in Hive, or a specific time of the day.
>> > Sensors are derived from BaseSensorOperator and run a poke method at a
>> > specified poke_interval until it returns True.
>> >
>> > The reason that the sensor tasks are inefficient is because in current
>> > design, we sprawn a separate worker process for each partition sensor.
>> This
>> > worker might last a long time, until the target partition is
>> available.  In
>> > the case where there are many sensor tasks that need to run within
>> certain
>> > time limits, we have to allocate a lot of resources to have enough
>> workers
>> > for the sensor tasks.
>> >
>> > *Idea:*
>> >
>> > We propose two approaches that could address this issues, batch-sensor
>> > and smart-sensor.
>> >
>> >
>> >
>> > Batch-sensor
>> >
>> > The basic idea of batch-sensor is to batch process sensor tasks to save
>> > resources. During running, a batch-sensor will take N partition sensor
>> > requests as the input and poke those N partitions periodically. If the
>> > batch-sensor finds that the criteria of some sensor task is met, the
>> > batch-sensor will update the database about this sensor tasks.
>> >
>> >
>> > To do this, we need to create a sensor basic class called ‘batchable’
>> and
>> > make all sensors inherit from this basic class. We also need to change
>> the
>> > behavior of schedule regarding a batchable sensor tasks. The schedule
>> will
>> > find as many as possible batchable sensor tasks and run those tasks in a
>> > batch.
>> >
>> >
>> > Smart-sensor
>> >
>> > Smart-sensor is an improvement on top of batch-sensor.
>> >
>> > The idea of smart-sensor is that the worker process of smart-sensor will
>> > run like a service. To do this, we need to persist Sensor details in
>> > Airflow DB and the worker process periodically queries task-instance
>> table
>> > to find sensor tasks; poke the metastore and update the task instance
>> table
>> > if it detects that certain partition or file created.
>> >
>>
>

Re: [Discuss] Airflow sensor optimization

[Yingbo Wang <yb...@gmail.com>]

Wow, Great work from Seelmann! Thanks Fokko for letting us know it. We are
super happy to have this feature.

On Wed, Mar 6, 2019 at 11:24 AM Driesprong, Fokko <fo...@driesprong.frl>
wrote:

> Thanks for bringing this up. I've added a comment on the Wiki:
>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization
>
> Have you looked into the work by Seelmann? Recently he introduced the
> ability to reschedule sensors. When rescheduling, the slot will be given
> back to the scheduler after a poke operation. Therefore the slot won't be
> occupied all the time. The details are in the PR
> https://github.com/apache/airflow/pull/3596
>
> I would propose to make this the default behavior in Airflow 2.0.
>
> Cheers, Fokko
>
> Op wo 6 mrt. 2019 om 15:32 schreef Yingbo Wang <yb...@gmail.com>:
>
> > hi,
> >
> > I would like to open an AIP for Airflow sensor optimization.
> >
> >
> > *Motivation*:
> >
> > Low efficiency in Airflow Sensor Implementation
> >
> > Sensors are a special kind of operator that will keep running until a
> > certain criterion is met. Examples include a specific file landing in
> HDFS
> > or S3, a partition appearing in Hive, or a specific time of the day.
> > Sensors are derived from BaseSensorOperator and run a poke method at a
> > specified poke_interval until it returns True.
> >
> > The reason that the sensor tasks are inefficient is because in current
> > design, we sprawn a separate worker process for each partition sensor.
> This
> > worker might last a long time, until the target partition is available.
> In
> > the case where there are many sensor tasks that need to run within
> certain
> > time limits, we have to allocate a lot of resources to have enough
> workers
> > for the sensor tasks.
> >
> > *Idea:*
> >
> > We propose two approaches that could address this issues, batch-sensor
> > and smart-sensor.
> >
> >
> >
> > Batch-sensor
> >
> > The basic idea of batch-sensor is to batch process sensor tasks to save
> > resources. During running, a batch-sensor will take N partition sensor
> > requests as the input and poke those N partitions periodically. If the
> > batch-sensor finds that the criteria of some sensor task is met, the
> > batch-sensor will update the database about this sensor tasks.
> >
> >
> > To do this, we need to create a sensor basic class called ‘batchable’ and
> > make all sensors inherit from this basic class. We also need to change
> the
> > behavior of schedule regarding a batchable sensor tasks. The schedule
> will
> > find as many as possible batchable sensor tasks and run those tasks in a
> > batch.
> >
> >
> > Smart-sensor
> >
> > Smart-sensor is an improvement on top of batch-sensor.
> >
> > The idea of smart-sensor is that the worker process of smart-sensor will
> > run like a service. To do this, we need to persist Sensor details in
> > Airflow DB and the worker process periodically queries task-instance
> table
> > to find sensor tasks; poke the metastore and update the task instance
> table
> > if it detects that certain partition or file created.
> >
>

Re: [Discuss] Airflow sensor optimization

["Driesprong, Fokko" <fo...@driesprong.frl>]

Thanks for bringing this up. I've added a comment on the Wiki:
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization

Have you looked into the work by Seelmann? Recently he introduced the
ability to reschedule sensors. When rescheduling, the slot will be given
back to the scheduler after a poke operation. Therefore the slot won't be
occupied all the time. The details are in the PR
https://github.com/apache/airflow/pull/3596

I would propose to make this the default behavior in Airflow 2.0.

Cheers, Fokko

Op wo 6 mrt. 2019 om 15:32 schreef Yingbo Wang <yb...@gmail.com>:

> hi,
>
> I would like to open an AIP for Airflow sensor optimization.
>
>
> *Motivation*:
>
> Low efficiency in Airflow Sensor Implementation
>
> Sensors are a special kind of operator that will keep running until a
> certain criterion is met. Examples include a specific file landing in HDFS
> or S3, a partition appearing in Hive, or a specific time of the day.
> Sensors are derived from BaseSensorOperator and run a poke method at a
> specified poke_interval until it returns True.
>
> The reason that the sensor tasks are inefficient is because in current
> design, we sprawn a separate worker process for each partition sensor. This
> worker might last a long time, until the target partition is available.  In
> the case where there are many sensor tasks that need to run within certain
> time limits, we have to allocate a lot of resources to have enough workers
> for the sensor tasks.
>
> *Idea:*
>
> We propose two approaches that could address this issues, batch-sensor
> and smart-sensor.
>
>
>
> Batch-sensor
>
> The basic idea of batch-sensor is to batch process sensor tasks to save
> resources. During running, a batch-sensor will take N partition sensor
> requests as the input and poke those N partitions periodically. If the
> batch-sensor finds that the criteria of some sensor task is met, the
> batch-sensor will update the database about this sensor tasks.
>
>
> To do this, we need to create a sensor basic class called ‘batchable’ and
> make all sensors inherit from this basic class. We also need to change the
> behavior of schedule regarding a batchable sensor tasks. The schedule will
> find as many as possible batchable sensor tasks and run those tasks in a
> batch.
>
>
> Smart-sensor
>
> Smart-sensor is an improvement on top of batch-sensor.
>
> The idea of smart-sensor is that the worker process of smart-sensor will
> run like a service. To do this, we need to persist Sensor details in
> Airflow DB and the worker process periodically queries task-instance table
> to find sensor tasks; poke the metastore and update the task instance table
> if it detects that certain partition or file created.
>