You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Kaxil Naik <ka...@gmail.com> on 2019/07/26 14:36:38 UTC
Airflow DAG Serialisation
Hi all,
We, at Astronomer, are going to spend time working on DAG Serialisation.
There are 2 AIPs that are somewhat related to what we plan to work on:
- AIP-18 Persist all information from DAG file in DB
<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB>
- AIP-19 Making the webserver stateless
<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless>
We plan to use JSON as the Serialisation format and store it as a blob in
metadata DB.
*Goals:*
- Make Webserver Stateless
- Use the same version of the DAG across Webserver & Scheduler
- Keep backward compatibility and have a flag (globally & at DAG level)
to turn this feature on/off
- Enable DAG Versioning (extended Goal)
We will be preparing a proposal (AIP) after some research and some initial
work and open it for the suggestions of the community.
We already had some good brain-storming sessions with Twitter folks (DanD &
Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from Uber) which will
be a good starting point for us.
If anyone in the community is interested in it or has some experience about
the same and want to collaborate please let me know and join
#dag-serialisation channel on Airflow Slack.
Regards,
Kaxil
Re: Airflow DAG Serialisation
Posted by Kevin Yang <yr...@gmail.com>.
s/dagbag import exception/dag import timeout exception/
On Wed, Jul 31, 2019 at 11:17 PM Kevin Yang <yr...@gmail.com> wrote:
> Hi Jonathan, for your problem, aside waiting for AIP-24 for the long term,
> you can try set the dagbag_import_timeout
> <https://github.com/apache/airflow/blob/master/airflow/config_templates/default_airflow.cfg#L162>
> to a smaller value so that those slow DAG file parsing ends faster. Also I
> don't think one DAG parsing can block parsing of other DAG files even we
> parse all of them in a single thread in the webserver. All exceptions are
> captured, including the dagbag import exception, will be captured and
> logged
> <https://github.com/apache/airflow/blob/master/airflow/models/dagbag.py#L197-L203>
> .
>
> Love Dan's ideas and agree with Fokko to start small and expand.
>
> I scan through PR5701 <https://github.com/apache/airflow/pull/5701> and
> it is exactly what I care the most for this AIP--to me others come
> naturally after we have a DAG serialization pattern defined. Good job Zhou
> and pardon me for not having enough bandwidth to review it thoroughly. +1
> for limit the scope of this AIP to item 1 and 3 in your proposed timeline.
>
>
> Cheers,
> Kevin Y
>
> On Wed, Jul 31, 2019 at 5:11 PM Zhou Fang <zh...@google.com.invalid>
> wrote:
>
>> I implemented the first version of DAG serialization part in AIP-24:
>> https://github.com/apache/airflow/pull/5701. Please take a look if you
>> are
>> interested @all. Thanks!
>>
>> It contains almost all fields of DAGs and tasks in the serialization (an
>> example of serialized DAG here:
>>
>> https://github.com/apache/airflow/blob/35e38f19b09646a0f85a2a7866a8d9aacc345252/tests/dags/test_dag_serialization.py#L100
>> ).
>> So basically the webserver can still treat them as before. No webserver UI
>> code change is needed. The benefit is that we can use it for 1.10.*.
>>
>> Of course, it is a short-term fix compared to many long-term proposals.
>>
>> It only contains serialization. I verified its usage in UI end-to-end by
>> using the Async DAG Loader in https://github.com/apache/airflow/pull/5594
>> .
>> I split the DAG serialization out of 5594 since Async DAG Loader is an
>> optional one. (I suddenly recall that if there are N webserver process + 1
>> async DAG loading process, it may solve webserver inconsistency problem??)
>>
>>
>> On Wed, Jul 31, 2019 at 10:33 AM Tao Feng <fe...@gmail.com> wrote:
>>
>> > hey Zhou,
>> >
>> > Great to see this happens and make it backward compatible. I will
>> persist
>> > DAG into DB is definitely needed. And it will make migration easier
>> with a
>> > lightweight approach. At Lyft we sometimes observe nondeterministic
>> > increased scheduling delay once users add some dynamic generated large
>> DAGs
>> > with thousands of tasks.
>> >
>> > I will spend some time to look at your proposal more in more detail.
>> But I
>> > agree that this is the most important pain point that we should address.
>> > And let me know if anything I could help to facilitate this.
>> >
>> >
>> > On Mon, Jul 29, 2019 at 2:13 PM Zhou Fang <zh...@google.com.invalid>
>> > wrote:
>> >
>> > > Thanks everyone for the discussion. The comments are very helpful.
>> > >
>> > > AIP-24 that we proposed here is really a short-term one to minimize
>> the
>> > > change for fast launch and compatibility. I agree with the benefits of
>> > the
>> > > long-term proposals. It would be great if AIP-24 can be a first step
>> (if
>> > we
>> > > can agree with the basic serialization approach). Then we can
>> gradually
>> > > apply long-term fixes.
>> > >
>> > > I summarized a few long-term proposals (from Fokko and Ash) and added
>> a
>> > > 'timeline' in AIP-24 (make things more clear):
>> > >
>> > > *Terms*
>> > >
>> > > - (this) stringified DAG: a patch to current DAG that can be
>> JSONified
>> > > - (long-term) serialized DAG: a new serializable DAG class used by
>> > > webserver/scheduler
>> > >
>> > > *Proposed timeline*
>> > >
>> > > 1. (this) JSON Serialization of DAGs
>> > > 1. will be out with https://github.com/apache/airflow/pull/5594
>> > >
>> > > 2. (this, optional) Asynchronous DAG loading in webserver
>> > > 1. webserver process uses a background process to collect DAGs,
>> > > solve scalability issue before DAG persistence in DB being out
>> > > 2. webserver process itself does not need to restart every 30s
>> to
>> > > collect DAGs
>> > > 3. will be out with https://github.com/apache/airflow/pull/5594
>> > >
>> > > 3. (this) DAG persistence in DB for webserver
>> > > 1. minimal Airflow code change
>> > > 2. an optional feature enabled via configuration
>> > > 3. rolled out with Airflow 1.10.5
>> > >
>> > > 4. (this, optional) Using DAG cached in DB for scheduling
>> > >
>> > > 5. (long-term) Defining serialized DAG for webserver
>> > > 1. this proposal keeps all fields of DAG/Operator, however, some
>> > > fields are not used by webserver or scheduler
>> > > 2. trimming these fields are easy, just providing a list of
>> fields
>> > to
>> > > include or exclude (Sec 2.3): _serialize_object(x, visited_dags)
>> > > =>_serialize_object(x, visited_dags, include=['foo'],
>> > > exclude=['bar'])
>> > > 3. we should carefully check all webserver/scheduler code to
>> make
>> > > sure trimmed fields are not used, e.g., *task.owner* is used in
>> > > webserver
>> > >
>> > > 6. (long-term) Defining serialized DAG for scheduler
>> > > 1. Once we have 'stringified DAG' or 'serialized DAG',
>> > > SimpleDAG/SimpleTaskInstance used by scheduler are not needed
>> > > 2. adding more fields to stringified DAGs to be compatible with
>> > > scheduler
>> > >
>> > > 7. (long-term) Directly reading DAGs from DB in webserver
>> > > 1. let webserver process fetch data from DB, instead of making a
>> > DAG
>> > > bag and refresh it
>> > > 2. it solves the webserver inconsistency issue
>> > >
>> > > 8. (long-term) Event-driven DAG parsing
>> > > 1. Instead of polling DAG files for updating/deleting DAGs,
>> event
>> > > based approaches, *e.g.*, inotify (
>> > > https://pypi.org/project/inotify_simple/) can be used
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > On Mon, Jul 29, 2019 at 3:23 AM Kaxil Naik <ka...@gmail.com>
>> wrote:
>> > >
>> > > > Thanks all for the input and thanks Zhou too for the detailed AIP.
>> > > >
>> > > > The WIP PR can be a good first step to overall optimization.
>> > > >
>> > > > Let's sync-up on the progress you have already made & what we want
>> to
>> > > > target.
>> > > >
>> > > > @Jarek Potiuk <Ja...@polidea.com> & @Fokko - If we manage
>> to
>> > > make
>> > > > it entirely backward-compatible with an enable/disable flag as we
>> > > > mentioned, we can think of including it in 1.10.5 but I am in favor
>> of
>> > > > removing / cleaning stuff like pickles, drop Py 2.0 and cut Airflow
>> 2.0
>> > > and
>> > > > include this change there.
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Mon, Jul 29, 2019 at 1:03 PM Jarek Potiuk <
>> Jarek.Potiuk@polidea.com
>> > >
>> > > > wrote:
>> > > >
>> > > > > Actually I am also doing a lot of v1-10-test merges during the
>> last
>> > few
>> > > > > months (probably several tens of them already). Rarely the
>> conflicts
>> > > are
>> > > > > difficult to solve in fact. We have usually small, localised
>> changes
>> > > and
>> > > > > until we go for full Black file re-formatting, we should be ok
>> (and
>> > the
>> > > > > change from Zhou seems rather small and localised).
>> > > > >
>> > > > > J.
>> > > > >
>> > > > > On Mon, Jul 29, 2019 at 9:25 AM Driesprong, Fokko
>> > <fokko@driesprong.frl
>> > > >
>> > > > > wrote:
>> > > > >
>> > > > > > I would be hesitant to merge it into 1.10.5. When I try to
>> backport
>> > > > > > anything into the 1.x branch, I get a whole bunch on merge
>> > conflicts,
>> > > > > even
>> > > > > > on the trivial tickets. For me, the only one who can really
>> comment
>> > > on
>> > > > > this
>> > > > > > would be Ash, since he's doing the bulk of the conflict
>> resolving.
>> > > > Apart
>> > > > > > from that, I'm really excited to make this happen!
>> > > > > >
>> > > > > > Cheers, Fokko
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > Op zo 28 jul. 2019 om 20:23 schreef Jarek Potiuk <
>> > > > > Jarek.Potiuk@polidea.com
>> > > > > > >:
>> > > > > >
>> > > > > > > Some thought I have after looking at the proposal from Zhou.
>> > > > > > >
>> > > > > > > I think this is one of the most important things feature-wise
>> for
>> > > > > > Airflow.
>> > > > > > > It looks like we have several in-progress attempts to solve
>> the
>> > > > problem
>> > > > > > and
>> > > > > > > I guess we should agree common approach.
>> > > > > > >
>> > > > > > > I like very much the approach of Zhou (AIP-24). It does seem
>> to
>> > > > > minimise
>> > > > > > > the changes needed in Airflow and it means that we with some
>> > > > > > optimisations
>> > > > > > > (caching mentioned by Fokko) - it can solve the major pain
>> points
>> > > > and I
>> > > > > > > think relatively quick and is potentially portable to 1.10.5
>> if
>> > we
>> > > > have
>> > > > > > it.
>> > > > > > >
>> > > > > > > I wonder how much it overlaps/differs from what Kaxil and Ash
>> > ideas
>> > > > > are.
>> > > > > > If
>> > > > > > > I read it correctly - it sounds like this idea will contain
>> some
>> > > more
>> > > > > > > "fundamental" changes. Ones that are likely less
>> > > > backwards-compatible,
>> > > > > > and
>> > > > > > > potentially taking longer time to implement and test. And
>> likely
>> > > > > solving
>> > > > > > > some of the problems better or even solving other problems.
>> Am I
>> > > > right
>> > > > > > with
>> > > > > > > my assumptions?
>> > > > > > >
>> > > > > > > I think more information on this might be helpful so that we
>> all
>> > > know
>> > > > > if
>> > > > > > > those are two different AIPs, or whether they can be joined in
>> > one
>> > > > > > effort,
>> > > > > > > and how they relate to AIP-18/AIP-19 (should those be
>> deprecated
>> > or
>> > > > > > > independently implemented ?). Also - since 2.0.0 release is
>> half
>> > a
>> > > > year
>> > > > > > > ahead we should consider how it impact the roadmap.
>> > > > > > >
>> > > > > > > I can see three approaches here that we as community can
>> follow
>> > > > (maybe
>> > > > > I
>> > > > > > am
>> > > > > > > missing some :) ):
>> > > > > > >
>> > > > > > > 1) focus our work on single "complete" solution that will take
>> > > longer
>> > > > > > time
>> > > > > > > and targets 2.0.0.
>> > > > > > > 2) work on two of them: one quick/fast - potentially portable
>> to
>> > > > > 1.10.5m
>> > > > > > > one longer-term for 2.0.0.
>> > > > > > > 3) decide that the simple solution we have from Zhou (maybe
>> with
>> > > some
>> > > > > > > modifications) is our target solution (for both 1.10.5 if we
>> have
>> > > it
>> > > > > and
>> > > > > > > 2.0.0):
>> > > > > > >
>> > > > > > > J.
>> > > > > > >
>> > > > > > > On Sat, Jul 27, 2019 at 11:43 AM Kevin Yang <
>> yrqls21@gmail.com>
>> > > > wrote:
>> > > > > > >
>> > > > > > > > Nice job Zhou!
>> > > > > > > >
>> > > > > > > > Really excited, exactly what we wanted for the webserver
>> > scaling
>> > > > > issue.
>> > > > > > > > Want to add another big drive for Airbnb to start think
>> about
>> > > this
>> > > > > > > > previously to support the effort: it can not only bring
>> > > consistency
>> > > > > > > between
>> > > > > > > > webservers but also bring consistency between webserver and
>> > > > > > > > scheduler/workers. It may be less of a problem if total DAG
>> > > parsing
>> > > > > > time
>> > > > > > > is
>> > > > > > > > small, but for us the total DAG parsing time is 15+ mins
>> and we
>> > > had
>> > > > > to
>> > > > > > > set
>> > > > > > > > the webserver( gunicorn subprocesses) restart interval to 20
>> > > mins,
>> > > > > > which
>> > > > > > > > leads to a worst case 15+20+15=50 mins delay between
>> scheduler
>> > > > start
>> > > > > to
>> > > > > > > > schedule things and users can see their deployed
>> > DAGs/changes...
>> > > > > > > >
>> > > > > > > > I'm not so sure about the scheduler performance improvement:
>> > > > > currently
>> > > > > > we
>> > > > > > > > already feed the main scheduler process with SimpleDag
>> through
>> > > > > > > > DagFileProcessorManager running in a subprocess--in the
>> future
>> > we
>> > > > > feed
>> > > > > > it
>> > > > > > > > with data from DB, which is likely slower( tho the diff
>> should
>> > > have
>> > > > > > > > negligible impact to the scheduler performance). In fact if
>> > we'd
>> > > > keep
>> > > > > > the
>> > > > > > > > existing behavior, try schedule only fresh parsed DAGs,
>> then we
>> > > may
>> > > > > > need
>> > > > > > > to
>> > > > > > > > deal with some consistency issue--dag processor and the
>> > scheduler
>> > > > > race
>> > > > > > > for
>> > > > > > > > updating the flag indicating if the DAG is newly parsed. No
>> big
>> > > > deal
>> > > > > > > there
>> > > > > > > > but just some thoughts on the top of my head and hopefully
>> can
>> > be
>> > > > > > > helpful.
>> > > > > > > >
>> > > > > > > > And good idea on pre-rendering the template, believe
>> template
>> > > > > rendering
>> > > > > > > was
>> > > > > > > > the biggest concern in the previous discussion. We've also
>> > chose
>> > > > the
>> > > > > > > > pre-rendering+JSON approach in our smart sensor API
>> > > > > > > > <
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization
>> > > > > > > > >
>> > > > > > > > and
>> > > > > > > > seems to be working fine--a supporting case for ur proposal
>> ;)
>> > > > > There's
>> > > > > > a
>> > > > > > > > WIP
>> > > > > > > > PR <https://github.com/apache/airflow/pull/5499> for it
>> just
>> > in
>> > > > case
>> > > > > > you
>> > > > > > > > are interested--maybe we can even share some logics.
>> > > > > > > >
>> > > > > > > > Thumbs-up again for this and please don't heisitate to reach
>> > out
>> > > if
>> > > > > you
>> > > > > > > > want to discuss further with us or need any help from us.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > Cheers,
>> > > > > > > > Kevin Y
>> > > > > > > >
>> > > > > > > > On Sat, Jul 27, 2019 at 12:54 AM Driesprong, Fokko
>> > > > > > <fokko@driesprong.frl
>> > > > > > > >
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Looks great Zhou,
>> > > > > > > > >
>> > > > > > > > > I have one thing that pops in my mind while reading the
>> AIP;
>> > > > should
>> > > > > > > keep
>> > > > > > > > > the caching on the webserver level. As the famous quote
>> goes:
>> > > > > *"There
>> > > > > > > are
>> > > > > > > > > only two hard things in Computer Science: cache
>> invalidation
>> > > and
>> > > > > > naming
>> > > > > > > > > things." -- Phil Karlton*
>> > > > > > > > >
>> > > > > > > > > Right now, the fundamental change that is being proposed
>> in
>> > the
>> > > > AIP
>> > > > > > is
>> > > > > > > > > fetching the DAGs from the database in a serialized
>> format,
>> > and
>> > > > not
>> > > > > > > > parsing
>> > > > > > > > > the Python files all the time. This will give already a
>> great
>> > > > > > > performance
>> > > > > > > > > improvement on the webserver side because it removes a
>> lot of
>> > > the
>> > > > > > > > > processing. However, since we're still fetching the DAGs
>> from
>> > > the
>> > > > > > > > database
>> > > > > > > > > in a regular interval, cache it in the local process, so
>> we
>> > > still
>> > > > > > have
>> > > > > > > > the
>> > > > > > > > > two issues that Airflow is suffering from right now:
>> > > > > > > > >
>> > > > > > > > > 1. No snappy UI because it is still polling the
>> database
>> > in
>> > > a
>> > > > > > > regular
>> > > > > > > > > interval.
>> > > > > > > > > 2. Inconsistency between webservers because they might
>> > poll
>> > > > in a
>> > > > > > > > > different interval, I think we've all seen this:
>> > > > > > > > > https://www.youtube.com/watch?v=sNrBruPS3r4
>> > > > > > > > >
>> > > > > > > > > As I also mentioned in the Slack channel, I strongly feel
>> > that
>> > > we
>> > > > > > > should
>> > > > > > > > be
>> > > > > > > > > able to render most views from the tables in the
>> database, so
>> > > > > without
>> > > > > > > > > touching the blob. For specific views, we could just pull
>> the
>> > > > blob
>> > > > > > from
>> > > > > > > > the
>> > > > > > > > > database. In this case we always have the latest version,
>> and
>> > > we
>> > > > > > tackle
>> > > > > > > > the
>> > > > > > > > > second point above.
>> > > > > > > > >
>> > > > > > > > > To tackle the first one, I also have an idea. We should
>> > change
>> > > > the
>> > > > > > DAG
>> > > > > > > > > parser from a loop to something that uses inotify
>> > > > > > > > > https://pypi.org/project/inotify_simple/. This will
>> change
>> > it
>> > > > from
>> > > > > > > > polling
>> > > > > > > > > to an event-driven design, which is much more performant
>> and
>> > > less
>> > > > > > > > resource
>> > > > > > > > > hungry. But this would be an AIP on its own.
>> > > > > > > > >
>> > > > > > > > > Again, great design and a comprehensive AIP, but I would
>> > > include
>> > > > > the
>> > > > > > > > > caching on the webserver to greatly improve the user
>> > experience
>> > > > in
>> > > > > > the
>> > > > > > > > UI.
>> > > > > > > > > Looking forward to the opinion of others on this.
>> > > > > > > > >
>> > > > > > > > > Cheers, Fokko
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > Op za 27 jul. 2019 om 01:44 schreef Zhou Fang
>> > > > > > > > <zhoufang@google.com.invalid
>> > > > > > > > > >:
>> > > > > > > > >
>> > > > > > > > > > Hi Kaxi,
>> > > > > > > > > >
>> > > > > > > > > > Just sent out the AIP:
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistence+in+DB+using+JSON+for+Airflow+Webserver+and+%28optional%29+Scheduler
>> > > > > > > > > >
>> > > > > > > > > > Thanks!
>> > > > > > > > > > Zhou
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On Fri, Jul 26, 2019 at 1:33 PM Zhou Fang <
>> > > zhoufang@google.com
>> > > > >
>> > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Hi Kaxil,
>> > > > > > > > > > >
>> > > > > > > > > > > We are also working on persisting DAGs into DB using
>> JSON
>> > > for
>> > > > > > > Airflow
>> > > > > > > > > > > webserver in Google Composer. We target at minimizing
>> the
>> > > > > change
>> > > > > > to
>> > > > > > > > the
>> > > > > > > > > > > current Airflow code. Happy to get synced on this!
>> > > > > > > > > > >
>> > > > > > > > > > > Here is our progress:
>> > > > > > > > > > > (1) Serializing DAGs using Pickle to be used in
>> webserver
>> > > > > > > > > > > It has been launched in Composer. I am working on the
>> PR
>> > to
>> > > > > > > upstream
>> > > > > > > > > it:
>> > > > > > > > > > > https://github.com/apache/airflow/pull/5594
>> > > > > > > > > > > Currently it does not support non-Airflow operators
>> and
>> > we
>> > > > are
>> > > > > > > > working
>> > > > > > > > > on
>> > > > > > > > > > > a fix.
>> > > > > > > > > > >
>> > > > > > > > > > > (2) Caching Pickled DAGs in DB to be used by webserver
>> > > > > > > > > > > We have a proof-of-concept implementation, working on
>> an
>> > > AIP
>> > > > > now.
>> > > > > > > > > > >
>> > > > > > > > > > > (3) Using JSON instead of Pickle in (1) and (2)
>> > > > > > > > > > > Decided to use JSON because Pickle is not secure and
>> > human
>> > > > > > > readable.
>> > > > > > > > > The
>> > > > > > > > > > > serialization approach is very similar to (1).
>> > > > > > > > > > >
>> > > > > > > > > > > I will update the RP (
>> > > > > > https://github.com/apache/airflow/pull/5594)
>> > > > > > > > to
>> > > > > > > > > > > replace Pickle by JSON, and send our design of (2) as
>> an
>> > > AIP
>> > > > > next
>> > > > > > > > week.
>> > > > > > > > > > > Glad to check together whether our implementation
>> makes
>> > > sense
>> > > > > and
>> > > > > > > do
>> > > > > > > > > > > improvements on that.
>> > > > > > > > > > >
>> > > > > > > > > > > Thanks!
>> > > > > > > > > > > Zhou
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > On Fri, Jul 26, 2019 at 7:37 AM Kaxil Naik <
>> > > > > kaxilnaik@gmail.com>
>> > > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > >> Hi all,
>> > > > > > > > > > >>
>> > > > > > > > > > >> We, at Astronomer, are going to spend time working on
>> > DAG
>> > > > > > > > > Serialisation.
>> > > > > > > > > > >> There are 2 AIPs that are somewhat related to what we
>> > plan
>> > > > to
>> > > > > > work
>> > > > > > > > on:
>> > > > > > > > > > >>
>> > > > > > > > > > >> - AIP-18 Persist all information from DAG file in
>> DB
>> > > > > > > > > > >> <
>> > > > > > > > > > >>
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB
>> > > > > > > > > > >> >
>> > > > > > > > > > >> - AIP-19 Making the webserver stateless
>> > > > > > > > > > >> <
>> > > > > > > > > > >>
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless
>> > > > > > > > > > >> >
>> > > > > > > > > > >>
>> > > > > > > > > > >> We plan to use JSON as the Serialisation format and
>> > store
>> > > it
>> > > > > as
>> > > > > > a
>> > > > > > > > blob
>> > > > > > > > > > in
>> > > > > > > > > > >> metadata DB.
>> > > > > > > > > > >>
>> > > > > > > > > > >> *Goals:*
>> > > > > > > > > > >>
>> > > > > > > > > > >> - Make Webserver Stateless
>> > > > > > > > > > >> - Use the same version of the DAG across
>> Webserver &
>> > > > > > Scheduler
>> > > > > > > > > > >> - Keep backward compatibility and have a flag
>> > > (globally &
>> > > > > at
>> > > > > > > DAG
>> > > > > > > > > > level)
>> > > > > > > > > > >> to turn this feature on/off
>> > > > > > > > > > >> - Enable DAG Versioning (extended Goal)
>> > > > > > > > > > >>
>> > > > > > > > > > >>
>> > > > > > > > > > >> We will be preparing a proposal (AIP) after some
>> > research
>> > > > and
>> > > > > > some
>> > > > > > > > > > initial
>> > > > > > > > > > >> work and open it for the suggestions of the
>> community.
>> > > > > > > > > > >>
>> > > > > > > > > > >> We already had some good brain-storming sessions with
>> > > > Twitter
>> > > > > > > folks
>> > > > > > > > > > (DanD
>> > > > > > > > > > >> &
>> > > > > > > > > > >> Sumit), folks from GoDataDriven (Fokko & Bas) & Alex
>> > (from
>> > > > > Uber)
>> > > > > > > > which
>> > > > > > > > > > >> will
>> > > > > > > > > > >> be a good starting point for us.
>> > > > > > > > > > >>
>> > > > > > > > > > >> If anyone in the community is interested in it or has
>> > some
>> > > > > > > > experience
>> > > > > > > > > > >> about
>> > > > > > > > > > >> the same and want to collaborate please let me know
>> and
>> > > join
>> > > > > > > > > > >> #dag-serialisation channel on Airflow Slack.
>> > > > > > > > > > >>
>> > > > > > > > > > >> Regards,
>> > > > > > > > > > >> Kaxil
>> > > > > > > > > > >>
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > --
>> > > > > > >
>> > > > > > > Jarek Potiuk
>> > > > > > > Polidea <https://www.polidea.com/> | Principal Software
>> Engineer
>> > > > > > >
>> > > > > > > M: +48 660 796 129 <+48660796129>
>> > > > > > > [image: Polidea] <https://www.polidea.com/>
>> > > > > > >
>> > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > >
>> > > > > Jarek Potiuk
>> > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
>> > > > >
>> > > > > M: +48 660 796 129 <+48660796129>
>> > > > > [image: Polidea] <https://www.polidea.com/>
>> > > > >
>> > > >
>> > > >
>> > > > --
>> > > > *Kaxil Naik*
>> > > > *Big Data Consultant | DevOps Data Engineer*
>> > > > *Certified *Google Cloud Data Engineer | *Certified* Apache Spark &
>> > Neo4j
>> > > > Developer
>> > > > *LinkedIn*: https://www.linkedin.com/in/kaxil
>> > > >
>> > >
>> >
>>
>
Re: Airflow DAG Serialisation
Posted by Kevin Yang <yr...@gmail.com>.
Hi Jonathan, for your problem, aside waiting for AIP-24 for the long term,
you can try set the dagbag_import_timeout
<https://github.com/apache/airflow/blob/master/airflow/config_templates/default_airflow.cfg#L162>
to a smaller value so that those slow DAG file parsing ends faster. Also I
don't think one DAG parsing can block parsing of other DAG files even we
parse all of them in a single thread in the webserver. All exceptions are
captured, including the dagbag import exception, will be captured and logged
<https://github.com/apache/airflow/blob/master/airflow/models/dagbag.py#L197-L203>
.
Love Dan's ideas and agree with Fokko to start small and expand.
I scan through PR5701 <https://github.com/apache/airflow/pull/5701> and it
is exactly what I care the most for this AIP--to me others come naturally
after we have a DAG serialization pattern defined. Good job Zhou and pardon
me for not having enough bandwidth to review it thoroughly. +1 for limit
the scope of this AIP to item 1 and 3 in your proposed timeline.
Cheers,
Kevin Y
On Wed, Jul 31, 2019 at 5:11 PM Zhou Fang <zh...@google.com.invalid>
wrote:
> I implemented the first version of DAG serialization part in AIP-24:
> https://github.com/apache/airflow/pull/5701. Please take a look if you are
> interested @all. Thanks!
>
> It contains almost all fields of DAGs and tasks in the serialization (an
> example of serialized DAG here:
>
> https://github.com/apache/airflow/blob/35e38f19b09646a0f85a2a7866a8d9aacc345252/tests/dags/test_dag_serialization.py#L100
> ).
> So basically the webserver can still treat them as before. No webserver UI
> code change is needed. The benefit is that we can use it for 1.10.*.
>
> Of course, it is a short-term fix compared to many long-term proposals.
>
> It only contains serialization. I verified its usage in UI end-to-end by
> using the Async DAG Loader in https://github.com/apache/airflow/pull/5594.
> I split the DAG serialization out of 5594 since Async DAG Loader is an
> optional one. (I suddenly recall that if there are N webserver process + 1
> async DAG loading process, it may solve webserver inconsistency problem??)
>
>
> On Wed, Jul 31, 2019 at 10:33 AM Tao Feng <fe...@gmail.com> wrote:
>
> > hey Zhou,
> >
> > Great to see this happens and make it backward compatible. I will persist
> > DAG into DB is definitely needed. And it will make migration easier with
> a
> > lightweight approach. At Lyft we sometimes observe nondeterministic
> > increased scheduling delay once users add some dynamic generated large
> DAGs
> > with thousands of tasks.
> >
> > I will spend some time to look at your proposal more in more detail. But
> I
> > agree that this is the most important pain point that we should address.
> > And let me know if anything I could help to facilitate this.
> >
> >
> > On Mon, Jul 29, 2019 at 2:13 PM Zhou Fang <zh...@google.com.invalid>
> > wrote:
> >
> > > Thanks everyone for the discussion. The comments are very helpful.
> > >
> > > AIP-24 that we proposed here is really a short-term one to minimize the
> > > change for fast launch and compatibility. I agree with the benefits of
> > the
> > > long-term proposals. It would be great if AIP-24 can be a first step
> (if
> > we
> > > can agree with the basic serialization approach). Then we can gradually
> > > apply long-term fixes.
> > >
> > > I summarized a few long-term proposals (from Fokko and Ash) and added a
> > > 'timeline' in AIP-24 (make things more clear):
> > >
> > > *Terms*
> > >
> > > - (this) stringified DAG: a patch to current DAG that can be
> JSONified
> > > - (long-term) serialized DAG: a new serializable DAG class used by
> > > webserver/scheduler
> > >
> > > *Proposed timeline*
> > >
> > > 1. (this) JSON Serialization of DAGs
> > > 1. will be out with https://github.com/apache/airflow/pull/5594
> > >
> > > 2. (this, optional) Asynchronous DAG loading in webserver
> > > 1. webserver process uses a background process to collect DAGs,
> > > solve scalability issue before DAG persistence in DB being out
> > > 2. webserver process itself does not need to restart every 30s to
> > > collect DAGs
> > > 3. will be out with https://github.com/apache/airflow/pull/5594
> > >
> > > 3. (this) DAG persistence in DB for webserver
> > > 1. minimal Airflow code change
> > > 2. an optional feature enabled via configuration
> > > 3. rolled out with Airflow 1.10.5
> > >
> > > 4. (this, optional) Using DAG cached in DB for scheduling
> > >
> > > 5. (long-term) Defining serialized DAG for webserver
> > > 1. this proposal keeps all fields of DAG/Operator, however, some
> > > fields are not used by webserver or scheduler
> > > 2. trimming these fields are easy, just providing a list of
> fields
> > to
> > > include or exclude (Sec 2.3): _serialize_object(x, visited_dags)
> > > =>_serialize_object(x, visited_dags, include=['foo'],
> > > exclude=['bar'])
> > > 3. we should carefully check all webserver/scheduler code to make
> > > sure trimmed fields are not used, e.g., *task.owner* is used in
> > > webserver
> > >
> > > 6. (long-term) Defining serialized DAG for scheduler
> > > 1. Once we have 'stringified DAG' or 'serialized DAG',
> > > SimpleDAG/SimpleTaskInstance used by scheduler are not needed
> > > 2. adding more fields to stringified DAGs to be compatible with
> > > scheduler
> > >
> > > 7. (long-term) Directly reading DAGs from DB in webserver
> > > 1. let webserver process fetch data from DB, instead of making a
> > DAG
> > > bag and refresh it
> > > 2. it solves the webserver inconsistency issue
> > >
> > > 8. (long-term) Event-driven DAG parsing
> > > 1. Instead of polling DAG files for updating/deleting DAGs, event
> > > based approaches, *e.g.*, inotify (
> > > https://pypi.org/project/inotify_simple/) can be used
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Jul 29, 2019 at 3:23 AM Kaxil Naik <ka...@gmail.com>
> wrote:
> > >
> > > > Thanks all for the input and thanks Zhou too for the detailed AIP.
> > > >
> > > > The WIP PR can be a good first step to overall optimization.
> > > >
> > > > Let's sync-up on the progress you have already made & what we want to
> > > > target.
> > > >
> > > > @Jarek Potiuk <Ja...@polidea.com> & @Fokko - If we manage to
> > > make
> > > > it entirely backward-compatible with an enable/disable flag as we
> > > > mentioned, we can think of including it in 1.10.5 but I am in favor
> of
> > > > removing / cleaning stuff like pickles, drop Py 2.0 and cut Airflow
> 2.0
> > > and
> > > > include this change there.
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Jul 29, 2019 at 1:03 PM Jarek Potiuk <
> Jarek.Potiuk@polidea.com
> > >
> > > > wrote:
> > > >
> > > > > Actually I am also doing a lot of v1-10-test merges during the last
> > few
> > > > > months (probably several tens of them already). Rarely the
> conflicts
> > > are
> > > > > difficult to solve in fact. We have usually small, localised
> changes
> > > and
> > > > > until we go for full Black file re-formatting, we should be ok (and
> > the
> > > > > change from Zhou seems rather small and localised).
> > > > >
> > > > > J.
> > > > >
> > > > > On Mon, Jul 29, 2019 at 9:25 AM Driesprong, Fokko
> > <fokko@driesprong.frl
> > > >
> > > > > wrote:
> > > > >
> > > > > > I would be hesitant to merge it into 1.10.5. When I try to
> backport
> > > > > > anything into the 1.x branch, I get a whole bunch on merge
> > conflicts,
> > > > > even
> > > > > > on the trivial tickets. For me, the only one who can really
> comment
> > > on
> > > > > this
> > > > > > would be Ash, since he's doing the bulk of the conflict
> resolving.
> > > > Apart
> > > > > > from that, I'm really excited to make this happen!
> > > > > >
> > > > > > Cheers, Fokko
> > > > > >
> > > > > >
> > > > > >
> > > > > > Op zo 28 jul. 2019 om 20:23 schreef Jarek Potiuk <
> > > > > Jarek.Potiuk@polidea.com
> > > > > > >:
> > > > > >
> > > > > > > Some thought I have after looking at the proposal from Zhou.
> > > > > > >
> > > > > > > I think this is one of the most important things feature-wise
> for
> > > > > > Airflow.
> > > > > > > It looks like we have several in-progress attempts to solve the
> > > > problem
> > > > > > and
> > > > > > > I guess we should agree common approach.
> > > > > > >
> > > > > > > I like very much the approach of Zhou (AIP-24). It does seem to
> > > > > minimise
> > > > > > > the changes needed in Airflow and it means that we with some
> > > > > > optimisations
> > > > > > > (caching mentioned by Fokko) - it can solve the major pain
> points
> > > > and I
> > > > > > > think relatively quick and is potentially portable to 1.10.5 if
> > we
> > > > have
> > > > > > it.
> > > > > > >
> > > > > > > I wonder how much it overlaps/differs from what Kaxil and Ash
> > ideas
> > > > > are.
> > > > > > If
> > > > > > > I read it correctly - it sounds like this idea will contain
> some
> > > more
> > > > > > > "fundamental" changes. Ones that are likely less
> > > > backwards-compatible,
> > > > > > and
> > > > > > > potentially taking longer time to implement and test. And
> likely
> > > > > solving
> > > > > > > some of the problems better or even solving other problems. Am
> I
> > > > right
> > > > > > with
> > > > > > > my assumptions?
> > > > > > >
> > > > > > > I think more information on this might be helpful so that we
> all
> > > know
> > > > > if
> > > > > > > those are two different AIPs, or whether they can be joined in
> > one
> > > > > > effort,
> > > > > > > and how they relate to AIP-18/AIP-19 (should those be
> deprecated
> > or
> > > > > > > independently implemented ?). Also - since 2.0.0 release is
> half
> > a
> > > > year
> > > > > > > ahead we should consider how it impact the roadmap.
> > > > > > >
> > > > > > > I can see three approaches here that we as community can follow
> > > > (maybe
> > > > > I
> > > > > > am
> > > > > > > missing some :) ):
> > > > > > >
> > > > > > > 1) focus our work on single "complete" solution that will take
> > > longer
> > > > > > time
> > > > > > > and targets 2.0.0.
> > > > > > > 2) work on two of them: one quick/fast - potentially portable
> to
> > > > > 1.10.5m
> > > > > > > one longer-term for 2.0.0.
> > > > > > > 3) decide that the simple solution we have from Zhou (maybe
> with
> > > some
> > > > > > > modifications) is our target solution (for both 1.10.5 if we
> have
> > > it
> > > > > and
> > > > > > > 2.0.0):
> > > > > > >
> > > > > > > J.
> > > > > > >
> > > > > > > On Sat, Jul 27, 2019 at 11:43 AM Kevin Yang <yrqls21@gmail.com
> >
> > > > wrote:
> > > > > > >
> > > > > > > > Nice job Zhou!
> > > > > > > >
> > > > > > > > Really excited, exactly what we wanted for the webserver
> > scaling
> > > > > issue.
> > > > > > > > Want to add another big drive for Airbnb to start think about
> > > this
> > > > > > > > previously to support the effort: it can not only bring
> > > consistency
> > > > > > > between
> > > > > > > > webservers but also bring consistency between webserver and
> > > > > > > > scheduler/workers. It may be less of a problem if total DAG
> > > parsing
> > > > > > time
> > > > > > > is
> > > > > > > > small, but for us the total DAG parsing time is 15+ mins and
> we
> > > had
> > > > > to
> > > > > > > set
> > > > > > > > the webserver( gunicorn subprocesses) restart interval to 20
> > > mins,
> > > > > > which
> > > > > > > > leads to a worst case 15+20+15=50 mins delay between
> scheduler
> > > > start
> > > > > to
> > > > > > > > schedule things and users can see their deployed
> > DAGs/changes...
> > > > > > > >
> > > > > > > > I'm not so sure about the scheduler performance improvement:
> > > > > currently
> > > > > > we
> > > > > > > > already feed the main scheduler process with SimpleDag
> through
> > > > > > > > DagFileProcessorManager running in a subprocess--in the
> future
> > we
> > > > > feed
> > > > > > it
> > > > > > > > with data from DB, which is likely slower( tho the diff
> should
> > > have
> > > > > > > > negligible impact to the scheduler performance). In fact if
> > we'd
> > > > keep
> > > > > > the
> > > > > > > > existing behavior, try schedule only fresh parsed DAGs, then
> we
> > > may
> > > > > > need
> > > > > > > to
> > > > > > > > deal with some consistency issue--dag processor and the
> > scheduler
> > > > > race
> > > > > > > for
> > > > > > > > updating the flag indicating if the DAG is newly parsed. No
> big
> > > > deal
> > > > > > > there
> > > > > > > > but just some thoughts on the top of my head and hopefully
> can
> > be
> > > > > > > helpful.
> > > > > > > >
> > > > > > > > And good idea on pre-rendering the template, believe template
> > > > > rendering
> > > > > > > was
> > > > > > > > the biggest concern in the previous discussion. We've also
> > chose
> > > > the
> > > > > > > > pre-rendering+JSON approach in our smart sensor API
> > > > > > > > <
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization
> > > > > > > > >
> > > > > > > > and
> > > > > > > > seems to be working fine--a supporting case for ur proposal
> ;)
> > > > > There's
> > > > > > a
> > > > > > > > WIP
> > > > > > > > PR <https://github.com/apache/airflow/pull/5499> for it just
> > in
> > > > case
> > > > > > you
> > > > > > > > are interested--maybe we can even share some logics.
> > > > > > > >
> > > > > > > > Thumbs-up again for this and please don't heisitate to reach
> > out
> > > if
> > > > > you
> > > > > > > > want to discuss further with us or need any help from us.
> > > > > > > >
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Kevin Y
> > > > > > > >
> > > > > > > > On Sat, Jul 27, 2019 at 12:54 AM Driesprong, Fokko
> > > > > > <fokko@driesprong.frl
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Looks great Zhou,
> > > > > > > > >
> > > > > > > > > I have one thing that pops in my mind while reading the
> AIP;
> > > > should
> > > > > > > keep
> > > > > > > > > the caching on the webserver level. As the famous quote
> goes:
> > > > > *"There
> > > > > > > are
> > > > > > > > > only two hard things in Computer Science: cache
> invalidation
> > > and
> > > > > > naming
> > > > > > > > > things." -- Phil Karlton*
> > > > > > > > >
> > > > > > > > > Right now, the fundamental change that is being proposed in
> > the
> > > > AIP
> > > > > > is
> > > > > > > > > fetching the DAGs from the database in a serialized format,
> > and
> > > > not
> > > > > > > > parsing
> > > > > > > > > the Python files all the time. This will give already a
> great
> > > > > > > performance
> > > > > > > > > improvement on the webserver side because it removes a lot
> of
> > > the
> > > > > > > > > processing. However, since we're still fetching the DAGs
> from
> > > the
> > > > > > > > database
> > > > > > > > > in a regular interval, cache it in the local process, so we
> > > still
> > > > > > have
> > > > > > > > the
> > > > > > > > > two issues that Airflow is suffering from right now:
> > > > > > > > >
> > > > > > > > > 1. No snappy UI because it is still polling the database
> > in
> > > a
> > > > > > > regular
> > > > > > > > > interval.
> > > > > > > > > 2. Inconsistency between webservers because they might
> > poll
> > > > in a
> > > > > > > > > different interval, I think we've all seen this:
> > > > > > > > > https://www.youtube.com/watch?v=sNrBruPS3r4
> > > > > > > > >
> > > > > > > > > As I also mentioned in the Slack channel, I strongly feel
> > that
> > > we
> > > > > > > should
> > > > > > > > be
> > > > > > > > > able to render most views from the tables in the database,
> so
> > > > > without
> > > > > > > > > touching the blob. For specific views, we could just pull
> the
> > > > blob
> > > > > > from
> > > > > > > > the
> > > > > > > > > database. In this case we always have the latest version,
> and
> > > we
> > > > > > tackle
> > > > > > > > the
> > > > > > > > > second point above.
> > > > > > > > >
> > > > > > > > > To tackle the first one, I also have an idea. We should
> > change
> > > > the
> > > > > > DAG
> > > > > > > > > parser from a loop to something that uses inotify
> > > > > > > > > https://pypi.org/project/inotify_simple/. This will change
> > it
> > > > from
> > > > > > > > polling
> > > > > > > > > to an event-driven design, which is much more performant
> and
> > > less
> > > > > > > > resource
> > > > > > > > > hungry. But this would be an AIP on its own.
> > > > > > > > >
> > > > > > > > > Again, great design and a comprehensive AIP, but I would
> > > include
> > > > > the
> > > > > > > > > caching on the webserver to greatly improve the user
> > experience
> > > > in
> > > > > > the
> > > > > > > > UI.
> > > > > > > > > Looking forward to the opinion of others on this.
> > > > > > > > >
> > > > > > > > > Cheers, Fokko
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Op za 27 jul. 2019 om 01:44 schreef Zhou Fang
> > > > > > > > <zhoufang@google.com.invalid
> > > > > > > > > >:
> > > > > > > > >
> > > > > > > > > > Hi Kaxi,
> > > > > > > > > >
> > > > > > > > > > Just sent out the AIP:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistence+in+DB+using+JSON+for+Airflow+Webserver+and+%28optional%29+Scheduler
> > > > > > > > > >
> > > > > > > > > > Thanks!
> > > > > > > > > > Zhou
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Jul 26, 2019 at 1:33 PM Zhou Fang <
> > > zhoufang@google.com
> > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Kaxil,
> > > > > > > > > > >
> > > > > > > > > > > We are also working on persisting DAGs into DB using
> JSON
> > > for
> > > > > > > Airflow
> > > > > > > > > > > webserver in Google Composer. We target at minimizing
> the
> > > > > change
> > > > > > to
> > > > > > > > the
> > > > > > > > > > > current Airflow code. Happy to get synced on this!
> > > > > > > > > > >
> > > > > > > > > > > Here is our progress:
> > > > > > > > > > > (1) Serializing DAGs using Pickle to be used in
> webserver
> > > > > > > > > > > It has been launched in Composer. I am working on the
> PR
> > to
> > > > > > > upstream
> > > > > > > > > it:
> > > > > > > > > > > https://github.com/apache/airflow/pull/5594
> > > > > > > > > > > Currently it does not support non-Airflow operators and
> > we
> > > > are
> > > > > > > > working
> > > > > > > > > on
> > > > > > > > > > > a fix.
> > > > > > > > > > >
> > > > > > > > > > > (2) Caching Pickled DAGs in DB to be used by webserver
> > > > > > > > > > > We have a proof-of-concept implementation, working on
> an
> > > AIP
> > > > > now.
> > > > > > > > > > >
> > > > > > > > > > > (3) Using JSON instead of Pickle in (1) and (2)
> > > > > > > > > > > Decided to use JSON because Pickle is not secure and
> > human
> > > > > > > readable.
> > > > > > > > > The
> > > > > > > > > > > serialization approach is very similar to (1).
> > > > > > > > > > >
> > > > > > > > > > > I will update the RP (
> > > > > > https://github.com/apache/airflow/pull/5594)
> > > > > > > > to
> > > > > > > > > > > replace Pickle by JSON, and send our design of (2) as
> an
> > > AIP
> > > > > next
> > > > > > > > week.
> > > > > > > > > > > Glad to check together whether our implementation makes
> > > sense
> > > > > and
> > > > > > > do
> > > > > > > > > > > improvements on that.
> > > > > > > > > > >
> > > > > > > > > > > Thanks!
> > > > > > > > > > > Zhou
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Jul 26, 2019 at 7:37 AM Kaxil Naik <
> > > > > kaxilnaik@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> Hi all,
> > > > > > > > > > >>
> > > > > > > > > > >> We, at Astronomer, are going to spend time working on
> > DAG
> > > > > > > > > Serialisation.
> > > > > > > > > > >> There are 2 AIPs that are somewhat related to what we
> > plan
> > > > to
> > > > > > work
> > > > > > > > on:
> > > > > > > > > > >>
> > > > > > > > > > >> - AIP-18 Persist all information from DAG file in
> DB
> > > > > > > > > > >> <
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB
> > > > > > > > > > >> >
> > > > > > > > > > >> - AIP-19 Making the webserver stateless
> > > > > > > > > > >> <
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > > >> We plan to use JSON as the Serialisation format and
> > store
> > > it
> > > > > as
> > > > > > a
> > > > > > > > blob
> > > > > > > > > > in
> > > > > > > > > > >> metadata DB.
> > > > > > > > > > >>
> > > > > > > > > > >> *Goals:*
> > > > > > > > > > >>
> > > > > > > > > > >> - Make Webserver Stateless
> > > > > > > > > > >> - Use the same version of the DAG across Webserver
> &
> > > > > > Scheduler
> > > > > > > > > > >> - Keep backward compatibility and have a flag
> > > (globally &
> > > > > at
> > > > > > > DAG
> > > > > > > > > > level)
> > > > > > > > > > >> to turn this feature on/off
> > > > > > > > > > >> - Enable DAG Versioning (extended Goal)
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >> We will be preparing a proposal (AIP) after some
> > research
> > > > and
> > > > > > some
> > > > > > > > > > initial
> > > > > > > > > > >> work and open it for the suggestions of the community.
> > > > > > > > > > >>
> > > > > > > > > > >> We already had some good brain-storming sessions with
> > > > Twitter
> > > > > > > folks
> > > > > > > > > > (DanD
> > > > > > > > > > >> &
> > > > > > > > > > >> Sumit), folks from GoDataDriven (Fokko & Bas) & Alex
> > (from
> > > > > Uber)
> > > > > > > > which
> > > > > > > > > > >> will
> > > > > > > > > > >> be a good starting point for us.
> > > > > > > > > > >>
> > > > > > > > > > >> If anyone in the community is interested in it or has
> > some
> > > > > > > > experience
> > > > > > > > > > >> about
> > > > > > > > > > >> the same and want to collaborate please let me know
> and
> > > join
> > > > > > > > > > >> #dag-serialisation channel on Airflow Slack.
> > > > > > > > > > >>
> > > > > > > > > > >> Regards,
> > > > > > > > > > >> Kaxil
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > Jarek Potiuk
> > > > > > > Polidea <https://www.polidea.com/> | Principal Software
> Engineer
> > > > > > >
> > > > > > > M: +48 660 796 129 <+48660796129>
> > > > > > > [image: Polidea] <https://www.polidea.com/>
> > > > > > >
> > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Jarek Potiuk
> > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > > >
> > > > > M: +48 660 796 129 <+48660796129>
> > > > > [image: Polidea] <https://www.polidea.com/>
> > > > >
> > > >
> > > >
> > > > --
> > > > *Kaxil Naik*
> > > > *Big Data Consultant | DevOps Data Engineer*
> > > > *Certified *Google Cloud Data Engineer | *Certified* Apache Spark &
> > Neo4j
> > > > Developer
> > > > *LinkedIn*: https://www.linkedin.com/in/kaxil
> > > >
> > >
> >
>
Re: Airflow DAG Serialisation
Posted by Zhou Fang <zh...@google.com.INVALID>.
I implemented the first version of DAG serialization part in AIP-24:
https://github.com/apache/airflow/pull/5701. Please take a look if you are
interested @all. Thanks!
It contains almost all fields of DAGs and tasks in the serialization (an
example of serialized DAG here:
https://github.com/apache/airflow/blob/35e38f19b09646a0f85a2a7866a8d9aacc345252/tests/dags/test_dag_serialization.py#L100).
So basically the webserver can still treat them as before. No webserver UI
code change is needed. The benefit is that we can use it for 1.10.*.
Of course, it is a short-term fix compared to many long-term proposals.
It only contains serialization. I verified its usage in UI end-to-end by
using the Async DAG Loader in https://github.com/apache/airflow/pull/5594.
I split the DAG serialization out of 5594 since Async DAG Loader is an
optional one. (I suddenly recall that if there are N webserver process + 1
async DAG loading process, it may solve webserver inconsistency problem??)
On Wed, Jul 31, 2019 at 10:33 AM Tao Feng <fe...@gmail.com> wrote:
> hey Zhou,
>
> Great to see this happens and make it backward compatible. I will persist
> DAG into DB is definitely needed. And it will make migration easier with a
> lightweight approach. At Lyft we sometimes observe nondeterministic
> increased scheduling delay once users add some dynamic generated large DAGs
> with thousands of tasks.
>
> I will spend some time to look at your proposal more in more detail. But I
> agree that this is the most important pain point that we should address.
> And let me know if anything I could help to facilitate this.
>
>
> On Mon, Jul 29, 2019 at 2:13 PM Zhou Fang <zh...@google.com.invalid>
> wrote:
>
> > Thanks everyone for the discussion. The comments are very helpful.
> >
> > AIP-24 that we proposed here is really a short-term one to minimize the
> > change for fast launch and compatibility. I agree with the benefits of
> the
> > long-term proposals. It would be great if AIP-24 can be a first step (if
> we
> > can agree with the basic serialization approach). Then we can gradually
> > apply long-term fixes.
> >
> > I summarized a few long-term proposals (from Fokko and Ash) and added a
> > 'timeline' in AIP-24 (make things more clear):
> >
> > *Terms*
> >
> > - (this) stringified DAG: a patch to current DAG that can be JSONified
> > - (long-term) serialized DAG: a new serializable DAG class used by
> > webserver/scheduler
> >
> > *Proposed timeline*
> >
> > 1. (this) JSON Serialization of DAGs
> > 1. will be out with https://github.com/apache/airflow/pull/5594
> >
> > 2. (this, optional) Asynchronous DAG loading in webserver
> > 1. webserver process uses a background process to collect DAGs,
> > solve scalability issue before DAG persistence in DB being out
> > 2. webserver process itself does not need to restart every 30s to
> > collect DAGs
> > 3. will be out with https://github.com/apache/airflow/pull/5594
> >
> > 3. (this) DAG persistence in DB for webserver
> > 1. minimal Airflow code change
> > 2. an optional feature enabled via configuration
> > 3. rolled out with Airflow 1.10.5
> >
> > 4. (this, optional) Using DAG cached in DB for scheduling
> >
> > 5. (long-term) Defining serialized DAG for webserver
> > 1. this proposal keeps all fields of DAG/Operator, however, some
> > fields are not used by webserver or scheduler
> > 2. trimming these fields are easy, just providing a list of fields
> to
> > include or exclude (Sec 2.3): _serialize_object(x, visited_dags)
> > =>_serialize_object(x, visited_dags, include=['foo'],
> > exclude=['bar'])
> > 3. we should carefully check all webserver/scheduler code to make
> > sure trimmed fields are not used, e.g., *task.owner* is used in
> > webserver
> >
> > 6. (long-term) Defining serialized DAG for scheduler
> > 1. Once we have 'stringified DAG' or 'serialized DAG',
> > SimpleDAG/SimpleTaskInstance used by scheduler are not needed
> > 2. adding more fields to stringified DAGs to be compatible with
> > scheduler
> >
> > 7. (long-term) Directly reading DAGs from DB in webserver
> > 1. let webserver process fetch data from DB, instead of making a
> DAG
> > bag and refresh it
> > 2. it solves the webserver inconsistency issue
> >
> > 8. (long-term) Event-driven DAG parsing
> > 1. Instead of polling DAG files for updating/deleting DAGs, event
> > based approaches, *e.g.*, inotify (
> > https://pypi.org/project/inotify_simple/) can be used
> >
> >
> >
> >
> >
> > On Mon, Jul 29, 2019 at 3:23 AM Kaxil Naik <ka...@gmail.com> wrote:
> >
> > > Thanks all for the input and thanks Zhou too for the detailed AIP.
> > >
> > > The WIP PR can be a good first step to overall optimization.
> > >
> > > Let's sync-up on the progress you have already made & what we want to
> > > target.
> > >
> > > @Jarek Potiuk <Ja...@polidea.com> & @Fokko - If we manage to
> > make
> > > it entirely backward-compatible with an enable/disable flag as we
> > > mentioned, we can think of including it in 1.10.5 but I am in favor of
> > > removing / cleaning stuff like pickles, drop Py 2.0 and cut Airflow 2.0
> > and
> > > include this change there.
> > >
> > >
> > >
> > >
> > > On Mon, Jul 29, 2019 at 1:03 PM Jarek Potiuk <Jarek.Potiuk@polidea.com
> >
> > > wrote:
> > >
> > > > Actually I am also doing a lot of v1-10-test merges during the last
> few
> > > > months (probably several tens of them already). Rarely the conflicts
> > are
> > > > difficult to solve in fact. We have usually small, localised changes
> > and
> > > > until we go for full Black file re-formatting, we should be ok (and
> the
> > > > change from Zhou seems rather small and localised).
> > > >
> > > > J.
> > > >
> > > > On Mon, Jul 29, 2019 at 9:25 AM Driesprong, Fokko
> <fokko@driesprong.frl
> > >
> > > > wrote:
> > > >
> > > > > I would be hesitant to merge it into 1.10.5. When I try to backport
> > > > > anything into the 1.x branch, I get a whole bunch on merge
> conflicts,
> > > > even
> > > > > on the trivial tickets. For me, the only one who can really comment
> > on
> > > > this
> > > > > would be Ash, since he's doing the bulk of the conflict resolving.
> > > Apart
> > > > > from that, I'm really excited to make this happen!
> > > > >
> > > > > Cheers, Fokko
> > > > >
> > > > >
> > > > >
> > > > > Op zo 28 jul. 2019 om 20:23 schreef Jarek Potiuk <
> > > > Jarek.Potiuk@polidea.com
> > > > > >:
> > > > >
> > > > > > Some thought I have after looking at the proposal from Zhou.
> > > > > >
> > > > > > I think this is one of the most important things feature-wise for
> > > > > Airflow.
> > > > > > It looks like we have several in-progress attempts to solve the
> > > problem
> > > > > and
> > > > > > I guess we should agree common approach.
> > > > > >
> > > > > > I like very much the approach of Zhou (AIP-24). It does seem to
> > > > minimise
> > > > > > the changes needed in Airflow and it means that we with some
> > > > > optimisations
> > > > > > (caching mentioned by Fokko) - it can solve the major pain points
> > > and I
> > > > > > think relatively quick and is potentially portable to 1.10.5 if
> we
> > > have
> > > > > it.
> > > > > >
> > > > > > I wonder how much it overlaps/differs from what Kaxil and Ash
> ideas
> > > > are.
> > > > > If
> > > > > > I read it correctly - it sounds like this idea will contain some
> > more
> > > > > > "fundamental" changes. Ones that are likely less
> > > backwards-compatible,
> > > > > and
> > > > > > potentially taking longer time to implement and test. And likely
> > > > solving
> > > > > > some of the problems better or even solving other problems. Am I
> > > right
> > > > > with
> > > > > > my assumptions?
> > > > > >
> > > > > > I think more information on this might be helpful so that we all
> > know
> > > > if
> > > > > > those are two different AIPs, or whether they can be joined in
> one
> > > > > effort,
> > > > > > and how they relate to AIP-18/AIP-19 (should those be deprecated
> or
> > > > > > independently implemented ?). Also - since 2.0.0 release is half
> a
> > > year
> > > > > > ahead we should consider how it impact the roadmap.
> > > > > >
> > > > > > I can see three approaches here that we as community can follow
> > > (maybe
> > > > I
> > > > > am
> > > > > > missing some :) ):
> > > > > >
> > > > > > 1) focus our work on single "complete" solution that will take
> > longer
> > > > > time
> > > > > > and targets 2.0.0.
> > > > > > 2) work on two of them: one quick/fast - potentially portable to
> > > > 1.10.5m
> > > > > > one longer-term for 2.0.0.
> > > > > > 3) decide that the simple solution we have from Zhou (maybe with
> > some
> > > > > > modifications) is our target solution (for both 1.10.5 if we have
> > it
> > > > and
> > > > > > 2.0.0):
> > > > > >
> > > > > > J.
> > > > > >
> > > > > > On Sat, Jul 27, 2019 at 11:43 AM Kevin Yang <yr...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > Nice job Zhou!
> > > > > > >
> > > > > > > Really excited, exactly what we wanted for the webserver
> scaling
> > > > issue.
> > > > > > > Want to add another big drive for Airbnb to start think about
> > this
> > > > > > > previously to support the effort: it can not only bring
> > consistency
> > > > > > between
> > > > > > > webservers but also bring consistency between webserver and
> > > > > > > scheduler/workers. It may be less of a problem if total DAG
> > parsing
> > > > > time
> > > > > > is
> > > > > > > small, but for us the total DAG parsing time is 15+ mins and we
> > had
> > > > to
> > > > > > set
> > > > > > > the webserver( gunicorn subprocesses) restart interval to 20
> > mins,
> > > > > which
> > > > > > > leads to a worst case 15+20+15=50 mins delay between scheduler
> > > start
> > > > to
> > > > > > > schedule things and users can see their deployed
> DAGs/changes...
> > > > > > >
> > > > > > > I'm not so sure about the scheduler performance improvement:
> > > > currently
> > > > > we
> > > > > > > already feed the main scheduler process with SimpleDag through
> > > > > > > DagFileProcessorManager running in a subprocess--in the future
> we
> > > > feed
> > > > > it
> > > > > > > with data from DB, which is likely slower( tho the diff should
> > have
> > > > > > > negligible impact to the scheduler performance). In fact if
> we'd
> > > keep
> > > > > the
> > > > > > > existing behavior, try schedule only fresh parsed DAGs, then we
> > may
> > > > > need
> > > > > > to
> > > > > > > deal with some consistency issue--dag processor and the
> scheduler
> > > > race
> > > > > > for
> > > > > > > updating the flag indicating if the DAG is newly parsed. No big
> > > deal
> > > > > > there
> > > > > > > but just some thoughts on the top of my head and hopefully can
> be
> > > > > > helpful.
> > > > > > >
> > > > > > > And good idea on pre-rendering the template, believe template
> > > > rendering
> > > > > > was
> > > > > > > the biggest concern in the previous discussion. We've also
> chose
> > > the
> > > > > > > pre-rendering+JSON approach in our smart sensor API
> > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization
> > > > > > > >
> > > > > > > and
> > > > > > > seems to be working fine--a supporting case for ur proposal ;)
> > > > There's
> > > > > a
> > > > > > > WIP
> > > > > > > PR <https://github.com/apache/airflow/pull/5499> for it just
> in
> > > case
> > > > > you
> > > > > > > are interested--maybe we can even share some logics.
> > > > > > >
> > > > > > > Thumbs-up again for this and please don't heisitate to reach
> out
> > if
> > > > you
> > > > > > > want to discuss further with us or need any help from us.
> > > > > > >
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Kevin Y
> > > > > > >
> > > > > > > On Sat, Jul 27, 2019 at 12:54 AM Driesprong, Fokko
> > > > > <fokko@driesprong.frl
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Looks great Zhou,
> > > > > > > >
> > > > > > > > I have one thing that pops in my mind while reading the AIP;
> > > should
> > > > > > keep
> > > > > > > > the caching on the webserver level. As the famous quote goes:
> > > > *"There
> > > > > > are
> > > > > > > > only two hard things in Computer Science: cache invalidation
> > and
> > > > > naming
> > > > > > > > things." -- Phil Karlton*
> > > > > > > >
> > > > > > > > Right now, the fundamental change that is being proposed in
> the
> > > AIP
> > > > > is
> > > > > > > > fetching the DAGs from the database in a serialized format,
> and
> > > not
> > > > > > > parsing
> > > > > > > > the Python files all the time. This will give already a great
> > > > > > performance
> > > > > > > > improvement on the webserver side because it removes a lot of
> > the
> > > > > > > > processing. However, since we're still fetching the DAGs from
> > the
> > > > > > > database
> > > > > > > > in a regular interval, cache it in the local process, so we
> > still
> > > > > have
> > > > > > > the
> > > > > > > > two issues that Airflow is suffering from right now:
> > > > > > > >
> > > > > > > > 1. No snappy UI because it is still polling the database
> in
> > a
> > > > > > regular
> > > > > > > > interval.
> > > > > > > > 2. Inconsistency between webservers because they might
> poll
> > > in a
> > > > > > > > different interval, I think we've all seen this:
> > > > > > > > https://www.youtube.com/watch?v=sNrBruPS3r4
> > > > > > > >
> > > > > > > > As I also mentioned in the Slack channel, I strongly feel
> that
> > we
> > > > > > should
> > > > > > > be
> > > > > > > > able to render most views from the tables in the database, so
> > > > without
> > > > > > > > touching the blob. For specific views, we could just pull the
> > > blob
> > > > > from
> > > > > > > the
> > > > > > > > database. In this case we always have the latest version, and
> > we
> > > > > tackle
> > > > > > > the
> > > > > > > > second point above.
> > > > > > > >
> > > > > > > > To tackle the first one, I also have an idea. We should
> change
> > > the
> > > > > DAG
> > > > > > > > parser from a loop to something that uses inotify
> > > > > > > > https://pypi.org/project/inotify_simple/. This will change
> it
> > > from
> > > > > > > polling
> > > > > > > > to an event-driven design, which is much more performant and
> > less
> > > > > > > resource
> > > > > > > > hungry. But this would be an AIP on its own.
> > > > > > > >
> > > > > > > > Again, great design and a comprehensive AIP, but I would
> > include
> > > > the
> > > > > > > > caching on the webserver to greatly improve the user
> experience
> > > in
> > > > > the
> > > > > > > UI.
> > > > > > > > Looking forward to the opinion of others on this.
> > > > > > > >
> > > > > > > > Cheers, Fokko
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Op za 27 jul. 2019 om 01:44 schreef Zhou Fang
> > > > > > > <zhoufang@google.com.invalid
> > > > > > > > >:
> > > > > > > >
> > > > > > > > > Hi Kaxi,
> > > > > > > > >
> > > > > > > > > Just sent out the AIP:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistence+in+DB+using+JSON+for+Airflow+Webserver+and+%28optional%29+Scheduler
> > > > > > > > >
> > > > > > > > > Thanks!
> > > > > > > > > Zhou
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Jul 26, 2019 at 1:33 PM Zhou Fang <
> > zhoufang@google.com
> > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Kaxil,
> > > > > > > > > >
> > > > > > > > > > We are also working on persisting DAGs into DB using JSON
> > for
> > > > > > Airflow
> > > > > > > > > > webserver in Google Composer. We target at minimizing the
> > > > change
> > > > > to
> > > > > > > the
> > > > > > > > > > current Airflow code. Happy to get synced on this!
> > > > > > > > > >
> > > > > > > > > > Here is our progress:
> > > > > > > > > > (1) Serializing DAGs using Pickle to be used in webserver
> > > > > > > > > > It has been launched in Composer. I am working on the PR
> to
> > > > > > upstream
> > > > > > > > it:
> > > > > > > > > > https://github.com/apache/airflow/pull/5594
> > > > > > > > > > Currently it does not support non-Airflow operators and
> we
> > > are
> > > > > > > working
> > > > > > > > on
> > > > > > > > > > a fix.
> > > > > > > > > >
> > > > > > > > > > (2) Caching Pickled DAGs in DB to be used by webserver
> > > > > > > > > > We have a proof-of-concept implementation, working on an
> > AIP
> > > > now.
> > > > > > > > > >
> > > > > > > > > > (3) Using JSON instead of Pickle in (1) and (2)
> > > > > > > > > > Decided to use JSON because Pickle is not secure and
> human
> > > > > > readable.
> > > > > > > > The
> > > > > > > > > > serialization approach is very similar to (1).
> > > > > > > > > >
> > > > > > > > > > I will update the RP (
> > > > > https://github.com/apache/airflow/pull/5594)
> > > > > > > to
> > > > > > > > > > replace Pickle by JSON, and send our design of (2) as an
> > AIP
> > > > next
> > > > > > > week.
> > > > > > > > > > Glad to check together whether our implementation makes
> > sense
> > > > and
> > > > > > do
> > > > > > > > > > improvements on that.
> > > > > > > > > >
> > > > > > > > > > Thanks!
> > > > > > > > > > Zhou
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Jul 26, 2019 at 7:37 AM Kaxil Naik <
> > > > kaxilnaik@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Hi all,
> > > > > > > > > >>
> > > > > > > > > >> We, at Astronomer, are going to spend time working on
> DAG
> > > > > > > > Serialisation.
> > > > > > > > > >> There are 2 AIPs that are somewhat related to what we
> plan
> > > to
> > > > > work
> > > > > > > on:
> > > > > > > > > >>
> > > > > > > > > >> - AIP-18 Persist all information from DAG file in DB
> > > > > > > > > >> <
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB
> > > > > > > > > >> >
> > > > > > > > > >> - AIP-19 Making the webserver stateless
> > > > > > > > > >> <
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > > >> We plan to use JSON as the Serialisation format and
> store
> > it
> > > > as
> > > > > a
> > > > > > > blob
> > > > > > > > > in
> > > > > > > > > >> metadata DB.
> > > > > > > > > >>
> > > > > > > > > >> *Goals:*
> > > > > > > > > >>
> > > > > > > > > >> - Make Webserver Stateless
> > > > > > > > > >> - Use the same version of the DAG across Webserver &
> > > > > Scheduler
> > > > > > > > > >> - Keep backward compatibility and have a flag
> > (globally &
> > > > at
> > > > > > DAG
> > > > > > > > > level)
> > > > > > > > > >> to turn this feature on/off
> > > > > > > > > >> - Enable DAG Versioning (extended Goal)
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> We will be preparing a proposal (AIP) after some
> research
> > > and
> > > > > some
> > > > > > > > > initial
> > > > > > > > > >> work and open it for the suggestions of the community.
> > > > > > > > > >>
> > > > > > > > > >> We already had some good brain-storming sessions with
> > > Twitter
> > > > > > folks
> > > > > > > > > (DanD
> > > > > > > > > >> &
> > > > > > > > > >> Sumit), folks from GoDataDriven (Fokko & Bas) & Alex
> (from
> > > > Uber)
> > > > > > > which
> > > > > > > > > >> will
> > > > > > > > > >> be a good starting point for us.
> > > > > > > > > >>
> > > > > > > > > >> If anyone in the community is interested in it or has
> some
> > > > > > > experience
> > > > > > > > > >> about
> > > > > > > > > >> the same and want to collaborate please let me know and
> > join
> > > > > > > > > >> #dag-serialisation channel on Airflow Slack.
> > > > > > > > > >>
> > > > > > > > > >> Regards,
> > > > > > > > > >> Kaxil
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Jarek Potiuk
> > > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > > > >
> > > > > > M: +48 660 796 129 <+48660796129>
> > > > > > [image: Polidea] <https://www.polidea.com/>
> > > > > >
> > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Jarek Potiuk
> > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > >
> > > > M: +48 660 796 129 <+48660796129>
> > > > [image: Polidea] <https://www.polidea.com/>
> > > >
> > >
> > >
> > > --
> > > *Kaxil Naik*
> > > *Big Data Consultant | DevOps Data Engineer*
> > > *Certified *Google Cloud Data Engineer | *Certified* Apache Spark &
> Neo4j
> > > Developer
> > > *LinkedIn*: https://www.linkedin.com/in/kaxil
> > >
> >
>
Re: Airflow DAG Serialisation
Posted by Tao Feng <fe...@gmail.com>.
hey Zhou,
Great to see this happens and make it backward compatible. I will persist
DAG into DB is definitely needed. And it will make migration easier with a
lightweight approach. At Lyft we sometimes observe nondeterministic
increased scheduling delay once users add some dynamic generated large DAGs
with thousands of tasks.
I will spend some time to look at your proposal more in more detail. But I
agree that this is the most important pain point that we should address.
And let me know if anything I could help to facilitate this.
On Mon, Jul 29, 2019 at 2:13 PM Zhou Fang <zh...@google.com.invalid>
wrote:
> Thanks everyone for the discussion. The comments are very helpful.
>
> AIP-24 that we proposed here is really a short-term one to minimize the
> change for fast launch and compatibility. I agree with the benefits of the
> long-term proposals. It would be great if AIP-24 can be a first step (if we
> can agree with the basic serialization approach). Then we can gradually
> apply long-term fixes.
>
> I summarized a few long-term proposals (from Fokko and Ash) and added a
> 'timeline' in AIP-24 (make things more clear):
>
> *Terms*
>
> - (this) stringified DAG: a patch to current DAG that can be JSONified
> - (long-term) serialized DAG: a new serializable DAG class used by
> webserver/scheduler
>
> *Proposed timeline*
>
> 1. (this) JSON Serialization of DAGs
> 1. will be out with https://github.com/apache/airflow/pull/5594
>
> 2. (this, optional) Asynchronous DAG loading in webserver
> 1. webserver process uses a background process to collect DAGs,
> solve scalability issue before DAG persistence in DB being out
> 2. webserver process itself does not need to restart every 30s to
> collect DAGs
> 3. will be out with https://github.com/apache/airflow/pull/5594
>
> 3. (this) DAG persistence in DB for webserver
> 1. minimal Airflow code change
> 2. an optional feature enabled via configuration
> 3. rolled out with Airflow 1.10.5
>
> 4. (this, optional) Using DAG cached in DB for scheduling
>
> 5. (long-term) Defining serialized DAG for webserver
> 1. this proposal keeps all fields of DAG/Operator, however, some
> fields are not used by webserver or scheduler
> 2. trimming these fields are easy, just providing a list of fields to
> include or exclude (Sec 2.3): _serialize_object(x, visited_dags)
> =>_serialize_object(x, visited_dags, include=['foo'],
> exclude=['bar'])
> 3. we should carefully check all webserver/scheduler code to make
> sure trimmed fields are not used, e.g., *task.owner* is used in
> webserver
>
> 6. (long-term) Defining serialized DAG for scheduler
> 1. Once we have 'stringified DAG' or 'serialized DAG',
> SimpleDAG/SimpleTaskInstance used by scheduler are not needed
> 2. adding more fields to stringified DAGs to be compatible with
> scheduler
>
> 7. (long-term) Directly reading DAGs from DB in webserver
> 1. let webserver process fetch data from DB, instead of making a DAG
> bag and refresh it
> 2. it solves the webserver inconsistency issue
>
> 8. (long-term) Event-driven DAG parsing
> 1. Instead of polling DAG files for updating/deleting DAGs, event
> based approaches, *e.g.*, inotify (
> https://pypi.org/project/inotify_simple/) can be used
>
>
>
>
>
> On Mon, Jul 29, 2019 at 3:23 AM Kaxil Naik <ka...@gmail.com> wrote:
>
> > Thanks all for the input and thanks Zhou too for the detailed AIP.
> >
> > The WIP PR can be a good first step to overall optimization.
> >
> > Let's sync-up on the progress you have already made & what we want to
> > target.
> >
> > @Jarek Potiuk <Ja...@polidea.com> & @Fokko - If we manage to
> make
> > it entirely backward-compatible with an enable/disable flag as we
> > mentioned, we can think of including it in 1.10.5 but I am in favor of
> > removing / cleaning stuff like pickles, drop Py 2.0 and cut Airflow 2.0
> and
> > include this change there.
> >
> >
> >
> >
> > On Mon, Jul 29, 2019 at 1:03 PM Jarek Potiuk <Ja...@polidea.com>
> > wrote:
> >
> > > Actually I am also doing a lot of v1-10-test merges during the last few
> > > months (probably several tens of them already). Rarely the conflicts
> are
> > > difficult to solve in fact. We have usually small, localised changes
> and
> > > until we go for full Black file re-formatting, we should be ok (and the
> > > change from Zhou seems rather small and localised).
> > >
> > > J.
> > >
> > > On Mon, Jul 29, 2019 at 9:25 AM Driesprong, Fokko <fokko@driesprong.frl
> >
> > > wrote:
> > >
> > > > I would be hesitant to merge it into 1.10.5. When I try to backport
> > > > anything into the 1.x branch, I get a whole bunch on merge conflicts,
> > > even
> > > > on the trivial tickets. For me, the only one who can really comment
> on
> > > this
> > > > would be Ash, since he's doing the bulk of the conflict resolving.
> > Apart
> > > > from that, I'm really excited to make this happen!
> > > >
> > > > Cheers, Fokko
> > > >
> > > >
> > > >
> > > > Op zo 28 jul. 2019 om 20:23 schreef Jarek Potiuk <
> > > Jarek.Potiuk@polidea.com
> > > > >:
> > > >
> > > > > Some thought I have after looking at the proposal from Zhou.
> > > > >
> > > > > I think this is one of the most important things feature-wise for
> > > > Airflow.
> > > > > It looks like we have several in-progress attempts to solve the
> > problem
> > > > and
> > > > > I guess we should agree common approach.
> > > > >
> > > > > I like very much the approach of Zhou (AIP-24). It does seem to
> > > minimise
> > > > > the changes needed in Airflow and it means that we with some
> > > > optimisations
> > > > > (caching mentioned by Fokko) - it can solve the major pain points
> > and I
> > > > > think relatively quick and is potentially portable to 1.10.5 if we
> > have
> > > > it.
> > > > >
> > > > > I wonder how much it overlaps/differs from what Kaxil and Ash ideas
> > > are.
> > > > If
> > > > > I read it correctly - it sounds like this idea will contain some
> more
> > > > > "fundamental" changes. Ones that are likely less
> > backwards-compatible,
> > > > and
> > > > > potentially taking longer time to implement and test. And likely
> > > solving
> > > > > some of the problems better or even solving other problems. Am I
> > right
> > > > with
> > > > > my assumptions?
> > > > >
> > > > > I think more information on this might be helpful so that we all
> know
> > > if
> > > > > those are two different AIPs, or whether they can be joined in one
> > > > effort,
> > > > > and how they relate to AIP-18/AIP-19 (should those be deprecated or
> > > > > independently implemented ?). Also - since 2.0.0 release is half a
> > year
> > > > > ahead we should consider how it impact the roadmap.
> > > > >
> > > > > I can see three approaches here that we as community can follow
> > (maybe
> > > I
> > > > am
> > > > > missing some :) ):
> > > > >
> > > > > 1) focus our work on single "complete" solution that will take
> longer
> > > > time
> > > > > and targets 2.0.0.
> > > > > 2) work on two of them: one quick/fast - potentially portable to
> > > 1.10.5m
> > > > > one longer-term for 2.0.0.
> > > > > 3) decide that the simple solution we have from Zhou (maybe with
> some
> > > > > modifications) is our target solution (for both 1.10.5 if we have
> it
> > > and
> > > > > 2.0.0):
> > > > >
> > > > > J.
> > > > >
> > > > > On Sat, Jul 27, 2019 at 11:43 AM Kevin Yang <yr...@gmail.com>
> > wrote:
> > > > >
> > > > > > Nice job Zhou!
> > > > > >
> > > > > > Really excited, exactly what we wanted for the webserver scaling
> > > issue.
> > > > > > Want to add another big drive for Airbnb to start think about
> this
> > > > > > previously to support the effort: it can not only bring
> consistency
> > > > > between
> > > > > > webservers but also bring consistency between webserver and
> > > > > > scheduler/workers. It may be less of a problem if total DAG
> parsing
> > > > time
> > > > > is
> > > > > > small, but for us the total DAG parsing time is 15+ mins and we
> had
> > > to
> > > > > set
> > > > > > the webserver( gunicorn subprocesses) restart interval to 20
> mins,
> > > > which
> > > > > > leads to a worst case 15+20+15=50 mins delay between scheduler
> > start
> > > to
> > > > > > schedule things and users can see their deployed DAGs/changes...
> > > > > >
> > > > > > I'm not so sure about the scheduler performance improvement:
> > > currently
> > > > we
> > > > > > already feed the main scheduler process with SimpleDag through
> > > > > > DagFileProcessorManager running in a subprocess--in the future we
> > > feed
> > > > it
> > > > > > with data from DB, which is likely slower( tho the diff should
> have
> > > > > > negligible impact to the scheduler performance). In fact if we'd
> > keep
> > > > the
> > > > > > existing behavior, try schedule only fresh parsed DAGs, then we
> may
> > > > need
> > > > > to
> > > > > > deal with some consistency issue--dag processor and the scheduler
> > > race
> > > > > for
> > > > > > updating the flag indicating if the DAG is newly parsed. No big
> > deal
> > > > > there
> > > > > > but just some thoughts on the top of my head and hopefully can be
> > > > > helpful.
> > > > > >
> > > > > > And good idea on pre-rendering the template, believe template
> > > rendering
> > > > > was
> > > > > > the biggest concern in the previous discussion. We've also chose
> > the
> > > > > > pre-rendering+JSON approach in our smart sensor API
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization
> > > > > > >
> > > > > > and
> > > > > > seems to be working fine--a supporting case for ur proposal ;)
> > > There's
> > > > a
> > > > > > WIP
> > > > > > PR <https://github.com/apache/airflow/pull/5499> for it just in
> > case
> > > > you
> > > > > > are interested--maybe we can even share some logics.
> > > > > >
> > > > > > Thumbs-up again for this and please don't heisitate to reach out
> if
> > > you
> > > > > > want to discuss further with us or need any help from us.
> > > > > >
> > > > > >
> > > > > > Cheers,
> > > > > > Kevin Y
> > > > > >
> > > > > > On Sat, Jul 27, 2019 at 12:54 AM Driesprong, Fokko
> > > > <fokko@driesprong.frl
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Looks great Zhou,
> > > > > > >
> > > > > > > I have one thing that pops in my mind while reading the AIP;
> > should
> > > > > keep
> > > > > > > the caching on the webserver level. As the famous quote goes:
> > > *"There
> > > > > are
> > > > > > > only two hard things in Computer Science: cache invalidation
> and
> > > > naming
> > > > > > > things." -- Phil Karlton*
> > > > > > >
> > > > > > > Right now, the fundamental change that is being proposed in the
> > AIP
> > > > is
> > > > > > > fetching the DAGs from the database in a serialized format, and
> > not
> > > > > > parsing
> > > > > > > the Python files all the time. This will give already a great
> > > > > performance
> > > > > > > improvement on the webserver side because it removes a lot of
> the
> > > > > > > processing. However, since we're still fetching the DAGs from
> the
> > > > > > database
> > > > > > > in a regular interval, cache it in the local process, so we
> still
> > > > have
> > > > > > the
> > > > > > > two issues that Airflow is suffering from right now:
> > > > > > >
> > > > > > > 1. No snappy UI because it is still polling the database in
> a
> > > > > regular
> > > > > > > interval.
> > > > > > > 2. Inconsistency between webservers because they might poll
> > in a
> > > > > > > different interval, I think we've all seen this:
> > > > > > > https://www.youtube.com/watch?v=sNrBruPS3r4
> > > > > > >
> > > > > > > As I also mentioned in the Slack channel, I strongly feel that
> we
> > > > > should
> > > > > > be
> > > > > > > able to render most views from the tables in the database, so
> > > without
> > > > > > > touching the blob. For specific views, we could just pull the
> > blob
> > > > from
> > > > > > the
> > > > > > > database. In this case we always have the latest version, and
> we
> > > > tackle
> > > > > > the
> > > > > > > second point above.
> > > > > > >
> > > > > > > To tackle the first one, I also have an idea. We should change
> > the
> > > > DAG
> > > > > > > parser from a loop to something that uses inotify
> > > > > > > https://pypi.org/project/inotify_simple/. This will change it
> > from
> > > > > > polling
> > > > > > > to an event-driven design, which is much more performant and
> less
> > > > > > resource
> > > > > > > hungry. But this would be an AIP on its own.
> > > > > > >
> > > > > > > Again, great design and a comprehensive AIP, but I would
> include
> > > the
> > > > > > > caching on the webserver to greatly improve the user experience
> > in
> > > > the
> > > > > > UI.
> > > > > > > Looking forward to the opinion of others on this.
> > > > > > >
> > > > > > > Cheers, Fokko
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Op za 27 jul. 2019 om 01:44 schreef Zhou Fang
> > > > > > <zhoufang@google.com.invalid
> > > > > > > >:
> > > > > > >
> > > > > > > > Hi Kaxi,
> > > > > > > >
> > > > > > > > Just sent out the AIP:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistence+in+DB+using+JSON+for+Airflow+Webserver+and+%28optional%29+Scheduler
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > > Zhou
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Jul 26, 2019 at 1:33 PM Zhou Fang <
> zhoufang@google.com
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Kaxil,
> > > > > > > > >
> > > > > > > > > We are also working on persisting DAGs into DB using JSON
> for
> > > > > Airflow
> > > > > > > > > webserver in Google Composer. We target at minimizing the
> > > change
> > > > to
> > > > > > the
> > > > > > > > > current Airflow code. Happy to get synced on this!
> > > > > > > > >
> > > > > > > > > Here is our progress:
> > > > > > > > > (1) Serializing DAGs using Pickle to be used in webserver
> > > > > > > > > It has been launched in Composer. I am working on the PR to
> > > > > upstream
> > > > > > > it:
> > > > > > > > > https://github.com/apache/airflow/pull/5594
> > > > > > > > > Currently it does not support non-Airflow operators and we
> > are
> > > > > > working
> > > > > > > on
> > > > > > > > > a fix.
> > > > > > > > >
> > > > > > > > > (2) Caching Pickled DAGs in DB to be used by webserver
> > > > > > > > > We have a proof-of-concept implementation, working on an
> AIP
> > > now.
> > > > > > > > >
> > > > > > > > > (3) Using JSON instead of Pickle in (1) and (2)
> > > > > > > > > Decided to use JSON because Pickle is not secure and human
> > > > > readable.
> > > > > > > The
> > > > > > > > > serialization approach is very similar to (1).
> > > > > > > > >
> > > > > > > > > I will update the RP (
> > > > https://github.com/apache/airflow/pull/5594)
> > > > > > to
> > > > > > > > > replace Pickle by JSON, and send our design of (2) as an
> AIP
> > > next
> > > > > > week.
> > > > > > > > > Glad to check together whether our implementation makes
> sense
> > > and
> > > > > do
> > > > > > > > > improvements on that.
> > > > > > > > >
> > > > > > > > > Thanks!
> > > > > > > > > Zhou
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Jul 26, 2019 at 7:37 AM Kaxil Naik <
> > > kaxilnaik@gmail.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Hi all,
> > > > > > > > >>
> > > > > > > > >> We, at Astronomer, are going to spend time working on DAG
> > > > > > > Serialisation.
> > > > > > > > >> There are 2 AIPs that are somewhat related to what we plan
> > to
> > > > work
> > > > > > on:
> > > > > > > > >>
> > > > > > > > >> - AIP-18 Persist all information from DAG file in DB
> > > > > > > > >> <
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB
> > > > > > > > >> >
> > > > > > > > >> - AIP-19 Making the webserver stateless
> > > > > > > > >> <
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >> We plan to use JSON as the Serialisation format and store
> it
> > > as
> > > > a
> > > > > > blob
> > > > > > > > in
> > > > > > > > >> metadata DB.
> > > > > > > > >>
> > > > > > > > >> *Goals:*
> > > > > > > > >>
> > > > > > > > >> - Make Webserver Stateless
> > > > > > > > >> - Use the same version of the DAG across Webserver &
> > > > Scheduler
> > > > > > > > >> - Keep backward compatibility and have a flag
> (globally &
> > > at
> > > > > DAG
> > > > > > > > level)
> > > > > > > > >> to turn this feature on/off
> > > > > > > > >> - Enable DAG Versioning (extended Goal)
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> We will be preparing a proposal (AIP) after some research
> > and
> > > > some
> > > > > > > > initial
> > > > > > > > >> work and open it for the suggestions of the community.
> > > > > > > > >>
> > > > > > > > >> We already had some good brain-storming sessions with
> > Twitter
> > > > > folks
> > > > > > > > (DanD
> > > > > > > > >> &
> > > > > > > > >> Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from
> > > Uber)
> > > > > > which
> > > > > > > > >> will
> > > > > > > > >> be a good starting point for us.
> > > > > > > > >>
> > > > > > > > >> If anyone in the community is interested in it or has some
> > > > > > experience
> > > > > > > > >> about
> > > > > > > > >> the same and want to collaborate please let me know and
> join
> > > > > > > > >> #dag-serialisation channel on Airflow Slack.
> > > > > > > > >>
> > > > > > > > >> Regards,
> > > > > > > > >> Kaxil
> > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Jarek Potiuk
> > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > > >
> > > > > M: +48 660 796 129 <+48660796129>
> > > > > [image: Polidea] <https://www.polidea.com/>
> > > > >
> > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > >
> > > Jarek Potiuk
> > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > >
> > > M: +48 660 796 129 <+48660796129>
> > > [image: Polidea] <https://www.polidea.com/>
> > >
> >
> >
> > --
> > *Kaxil Naik*
> > *Big Data Consultant | DevOps Data Engineer*
> > *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
> > Developer
> > *LinkedIn*: https://www.linkedin.com/in/kaxil
> >
>
Re: Airflow DAG Serialisation
Posted by Zhou Fang <zh...@google.com.INVALID>.
Thanks everyone for the discussion. The comments are very helpful.
AIP-24 that we proposed here is really a short-term one to minimize the
change for fast launch and compatibility. I agree with the benefits of the
long-term proposals. It would be great if AIP-24 can be a first step (if we
can agree with the basic serialization approach). Then we can gradually
apply long-term fixes.
I summarized a few long-term proposals (from Fokko and Ash) and added a
'timeline' in AIP-24 (make things more clear):
*Terms*
- (this) stringified DAG: a patch to current DAG that can be JSONified
- (long-term) serialized DAG: a new serializable DAG class used by
webserver/scheduler
*Proposed timeline*
1. (this) JSON Serialization of DAGs
1. will be out with https://github.com/apache/airflow/pull/5594
2. (this, optional) Asynchronous DAG loading in webserver
1. webserver process uses a background process to collect DAGs,
solve scalability issue before DAG persistence in DB being out
2. webserver process itself does not need to restart every 30s to
collect DAGs
3. will be out with https://github.com/apache/airflow/pull/5594
3. (this) DAG persistence in DB for webserver
1. minimal Airflow code change
2. an optional feature enabled via configuration
3. rolled out with Airflow 1.10.5
4. (this, optional) Using DAG cached in DB for scheduling
5. (long-term) Defining serialized DAG for webserver
1. this proposal keeps all fields of DAG/Operator, however, some
fields are not used by webserver or scheduler
2. trimming these fields are easy, just providing a list of fields to
include or exclude (Sec 2.3): _serialize_object(x, visited_dags)
=>_serialize_object(x, visited_dags, include=['foo'], exclude=['bar'])
3. we should carefully check all webserver/scheduler code to make
sure trimmed fields are not used, e.g., *task.owner* is used in
webserver
6. (long-term) Defining serialized DAG for scheduler
1. Once we have 'stringified DAG' or 'serialized DAG',
SimpleDAG/SimpleTaskInstance used by scheduler are not needed
2. adding more fields to stringified DAGs to be compatible with
scheduler
7. (long-term) Directly reading DAGs from DB in webserver
1. let webserver process fetch data from DB, instead of making a DAG
bag and refresh it
2. it solves the webserver inconsistency issue
8. (long-term) Event-driven DAG parsing
1. Instead of polling DAG files for updating/deleting DAGs, event
based approaches, *e.g.*, inotify (
https://pypi.org/project/inotify_simple/) can be used
On Mon, Jul 29, 2019 at 3:23 AM Kaxil Naik <ka...@gmail.com> wrote:
> Thanks all for the input and thanks Zhou too for the detailed AIP.
>
> The WIP PR can be a good first step to overall optimization.
>
> Let's sync-up on the progress you have already made & what we want to
> target.
>
> @Jarek Potiuk <Ja...@polidea.com> & @Fokko - If we manage to make
> it entirely backward-compatible with an enable/disable flag as we
> mentioned, we can think of including it in 1.10.5 but I am in favor of
> removing / cleaning stuff like pickles, drop Py 2.0 and cut Airflow 2.0 and
> include this change there.
>
>
>
>
> On Mon, Jul 29, 2019 at 1:03 PM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
> > Actually I am also doing a lot of v1-10-test merges during the last few
> > months (probably several tens of them already). Rarely the conflicts are
> > difficult to solve in fact. We have usually small, localised changes and
> > until we go for full Black file re-formatting, we should be ok (and the
> > change from Zhou seems rather small and localised).
> >
> > J.
> >
> > On Mon, Jul 29, 2019 at 9:25 AM Driesprong, Fokko <fo...@driesprong.frl>
> > wrote:
> >
> > > I would be hesitant to merge it into 1.10.5. When I try to backport
> > > anything into the 1.x branch, I get a whole bunch on merge conflicts,
> > even
> > > on the trivial tickets. For me, the only one who can really comment on
> > this
> > > would be Ash, since he's doing the bulk of the conflict resolving.
> Apart
> > > from that, I'm really excited to make this happen!
> > >
> > > Cheers, Fokko
> > >
> > >
> > >
> > > Op zo 28 jul. 2019 om 20:23 schreef Jarek Potiuk <
> > Jarek.Potiuk@polidea.com
> > > >:
> > >
> > > > Some thought I have after looking at the proposal from Zhou.
> > > >
> > > > I think this is one of the most important things feature-wise for
> > > Airflow.
> > > > It looks like we have several in-progress attempts to solve the
> problem
> > > and
> > > > I guess we should agree common approach.
> > > >
> > > > I like very much the approach of Zhou (AIP-24). It does seem to
> > minimise
> > > > the changes needed in Airflow and it means that we with some
> > > optimisations
> > > > (caching mentioned by Fokko) - it can solve the major pain points
> and I
> > > > think relatively quick and is potentially portable to 1.10.5 if we
> have
> > > it.
> > > >
> > > > I wonder how much it overlaps/differs from what Kaxil and Ash ideas
> > are.
> > > If
> > > > I read it correctly - it sounds like this idea will contain some more
> > > > "fundamental" changes. Ones that are likely less
> backwards-compatible,
> > > and
> > > > potentially taking longer time to implement and test. And likely
> > solving
> > > > some of the problems better or even solving other problems. Am I
> right
> > > with
> > > > my assumptions?
> > > >
> > > > I think more information on this might be helpful so that we all know
> > if
> > > > those are two different AIPs, or whether they can be joined in one
> > > effort,
> > > > and how they relate to AIP-18/AIP-19 (should those be deprecated or
> > > > independently implemented ?). Also - since 2.0.0 release is half a
> year
> > > > ahead we should consider how it impact the roadmap.
> > > >
> > > > I can see three approaches here that we as community can follow
> (maybe
> > I
> > > am
> > > > missing some :) ):
> > > >
> > > > 1) focus our work on single "complete" solution that will take longer
> > > time
> > > > and targets 2.0.0.
> > > > 2) work on two of them: one quick/fast - potentially portable to
> > 1.10.5m
> > > > one longer-term for 2.0.0.
> > > > 3) decide that the simple solution we have from Zhou (maybe with some
> > > > modifications) is our target solution (for both 1.10.5 if we have it
> > and
> > > > 2.0.0):
> > > >
> > > > J.
> > > >
> > > > On Sat, Jul 27, 2019 at 11:43 AM Kevin Yang <yr...@gmail.com>
> wrote:
> > > >
> > > > > Nice job Zhou!
> > > > >
> > > > > Really excited, exactly what we wanted for the webserver scaling
> > issue.
> > > > > Want to add another big drive for Airbnb to start think about this
> > > > > previously to support the effort: it can not only bring consistency
> > > > between
> > > > > webservers but also bring consistency between webserver and
> > > > > scheduler/workers. It may be less of a problem if total DAG parsing
> > > time
> > > > is
> > > > > small, but for us the total DAG parsing time is 15+ mins and we had
> > to
> > > > set
> > > > > the webserver( gunicorn subprocesses) restart interval to 20 mins,
> > > which
> > > > > leads to a worst case 15+20+15=50 mins delay between scheduler
> start
> > to
> > > > > schedule things and users can see their deployed DAGs/changes...
> > > > >
> > > > > I'm not so sure about the scheduler performance improvement:
> > currently
> > > we
> > > > > already feed the main scheduler process with SimpleDag through
> > > > > DagFileProcessorManager running in a subprocess--in the future we
> > feed
> > > it
> > > > > with data from DB, which is likely slower( tho the diff should have
> > > > > negligible impact to the scheduler performance). In fact if we'd
> keep
> > > the
> > > > > existing behavior, try schedule only fresh parsed DAGs, then we may
> > > need
> > > > to
> > > > > deal with some consistency issue--dag processor and the scheduler
> > race
> > > > for
> > > > > updating the flag indicating if the DAG is newly parsed. No big
> deal
> > > > there
> > > > > but just some thoughts on the top of my head and hopefully can be
> > > > helpful.
> > > > >
> > > > > And good idea on pre-rendering the template, believe template
> > rendering
> > > > was
> > > > > the biggest concern in the previous discussion. We've also chose
> the
> > > > > pre-rendering+JSON approach in our smart sensor API
> > > > > <
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization
> > > > > >
> > > > > and
> > > > > seems to be working fine--a supporting case for ur proposal ;)
> > There's
> > > a
> > > > > WIP
> > > > > PR <https://github.com/apache/airflow/pull/5499> for it just in
> case
> > > you
> > > > > are interested--maybe we can even share some logics.
> > > > >
> > > > > Thumbs-up again for this and please don't heisitate to reach out if
> > you
> > > > > want to discuss further with us or need any help from us.
> > > > >
> > > > >
> > > > > Cheers,
> > > > > Kevin Y
> > > > >
> > > > > On Sat, Jul 27, 2019 at 12:54 AM Driesprong, Fokko
> > > <fokko@driesprong.frl
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Looks great Zhou,
> > > > > >
> > > > > > I have one thing that pops in my mind while reading the AIP;
> should
> > > > keep
> > > > > > the caching on the webserver level. As the famous quote goes:
> > *"There
> > > > are
> > > > > > only two hard things in Computer Science: cache invalidation and
> > > naming
> > > > > > things." -- Phil Karlton*
> > > > > >
> > > > > > Right now, the fundamental change that is being proposed in the
> AIP
> > > is
> > > > > > fetching the DAGs from the database in a serialized format, and
> not
> > > > > parsing
> > > > > > the Python files all the time. This will give already a great
> > > > performance
> > > > > > improvement on the webserver side because it removes a lot of the
> > > > > > processing. However, since we're still fetching the DAGs from the
> > > > > database
> > > > > > in a regular interval, cache it in the local process, so we still
> > > have
> > > > > the
> > > > > > two issues that Airflow is suffering from right now:
> > > > > >
> > > > > > 1. No snappy UI because it is still polling the database in a
> > > > regular
> > > > > > interval.
> > > > > > 2. Inconsistency between webservers because they might poll
> in a
> > > > > > different interval, I think we've all seen this:
> > > > > > https://www.youtube.com/watch?v=sNrBruPS3r4
> > > > > >
> > > > > > As I also mentioned in the Slack channel, I strongly feel that we
> > > > should
> > > > > be
> > > > > > able to render most views from the tables in the database, so
> > without
> > > > > > touching the blob. For specific views, we could just pull the
> blob
> > > from
> > > > > the
> > > > > > database. In this case we always have the latest version, and we
> > > tackle
> > > > > the
> > > > > > second point above.
> > > > > >
> > > > > > To tackle the first one, I also have an idea. We should change
> the
> > > DAG
> > > > > > parser from a loop to something that uses inotify
> > > > > > https://pypi.org/project/inotify_simple/. This will change it
> from
> > > > > polling
> > > > > > to an event-driven design, which is much more performant and less
> > > > > resource
> > > > > > hungry. But this would be an AIP on its own.
> > > > > >
> > > > > > Again, great design and a comprehensive AIP, but I would include
> > the
> > > > > > caching on the webserver to greatly improve the user experience
> in
> > > the
> > > > > UI.
> > > > > > Looking forward to the opinion of others on this.
> > > > > >
> > > > > > Cheers, Fokko
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Op za 27 jul. 2019 om 01:44 schreef Zhou Fang
> > > > > <zhoufang@google.com.invalid
> > > > > > >:
> > > > > >
> > > > > > > Hi Kaxi,
> > > > > > >
> > > > > > > Just sent out the AIP:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistence+in+DB+using+JSON+for+Airflow+Webserver+and+%28optional%29+Scheduler
> > > > > > >
> > > > > > > Thanks!
> > > > > > > Zhou
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Jul 26, 2019 at 1:33 PM Zhou Fang <zhoufang@google.com
> >
> > > > wrote:
> > > > > > >
> > > > > > > > Hi Kaxil,
> > > > > > > >
> > > > > > > > We are also working on persisting DAGs into DB using JSON for
> > > > Airflow
> > > > > > > > webserver in Google Composer. We target at minimizing the
> > change
> > > to
> > > > > the
> > > > > > > > current Airflow code. Happy to get synced on this!
> > > > > > > >
> > > > > > > > Here is our progress:
> > > > > > > > (1) Serializing DAGs using Pickle to be used in webserver
> > > > > > > > It has been launched in Composer. I am working on the PR to
> > > > upstream
> > > > > > it:
> > > > > > > > https://github.com/apache/airflow/pull/5594
> > > > > > > > Currently it does not support non-Airflow operators and we
> are
> > > > > working
> > > > > > on
> > > > > > > > a fix.
> > > > > > > >
> > > > > > > > (2) Caching Pickled DAGs in DB to be used by webserver
> > > > > > > > We have a proof-of-concept implementation, working on an AIP
> > now.
> > > > > > > >
> > > > > > > > (3) Using JSON instead of Pickle in (1) and (2)
> > > > > > > > Decided to use JSON because Pickle is not secure and human
> > > > readable.
> > > > > > The
> > > > > > > > serialization approach is very similar to (1).
> > > > > > > >
> > > > > > > > I will update the RP (
> > > https://github.com/apache/airflow/pull/5594)
> > > > > to
> > > > > > > > replace Pickle by JSON, and send our design of (2) as an AIP
> > next
> > > > > week.
> > > > > > > > Glad to check together whether our implementation makes sense
> > and
> > > > do
> > > > > > > > improvements on that.
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > > Zhou
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Jul 26, 2019 at 7:37 AM Kaxil Naik <
> > kaxilnaik@gmail.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > >> Hi all,
> > > > > > > >>
> > > > > > > >> We, at Astronomer, are going to spend time working on DAG
> > > > > > Serialisation.
> > > > > > > >> There are 2 AIPs that are somewhat related to what we plan
> to
> > > work
> > > > > on:
> > > > > > > >>
> > > > > > > >> - AIP-18 Persist all information from DAG file in DB
> > > > > > > >> <
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB
> > > > > > > >> >
> > > > > > > >> - AIP-19 Making the webserver stateless
> > > > > > > >> <
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >> We plan to use JSON as the Serialisation format and store it
> > as
> > > a
> > > > > blob
> > > > > > > in
> > > > > > > >> metadata DB.
> > > > > > > >>
> > > > > > > >> *Goals:*
> > > > > > > >>
> > > > > > > >> - Make Webserver Stateless
> > > > > > > >> - Use the same version of the DAG across Webserver &
> > > Scheduler
> > > > > > > >> - Keep backward compatibility and have a flag (globally &
> > at
> > > > DAG
> > > > > > > level)
> > > > > > > >> to turn this feature on/off
> > > > > > > >> - Enable DAG Versioning (extended Goal)
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> We will be preparing a proposal (AIP) after some research
> and
> > > some
> > > > > > > initial
> > > > > > > >> work and open it for the suggestions of the community.
> > > > > > > >>
> > > > > > > >> We already had some good brain-storming sessions with
> Twitter
> > > > folks
> > > > > > > (DanD
> > > > > > > >> &
> > > > > > > >> Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from
> > Uber)
> > > > > which
> > > > > > > >> will
> > > > > > > >> be a good starting point for us.
> > > > > > > >>
> > > > > > > >> If anyone in the community is interested in it or has some
> > > > > experience
> > > > > > > >> about
> > > > > > > >> the same and want to collaborate please let me know and join
> > > > > > > >> #dag-serialisation channel on Airflow Slack.
> > > > > > > >>
> > > > > > > >> Regards,
> > > > > > > >> Kaxil
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Jarek Potiuk
> > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > >
> > > > M: +48 660 796 129 <+48660796129>
> > > > [image: Polidea] <https://www.polidea.com/>
> > > >
> > >
> > > >
> > >
> >
> >
> > --
> >
> > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: +48 660 796 129 <+48660796129>
> > [image: Polidea] <https://www.polidea.com/>
> >
>
>
> --
> *Kaxil Naik*
> *Big Data Consultant | DevOps Data Engineer*
> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
> Developer
> *LinkedIn*: https://www.linkedin.com/in/kaxil
>
Re: Airflow DAG Serialisation
Posted by Kaxil Naik <ka...@gmail.com>.
Thanks all for the input and thanks Zhou too for the detailed AIP.
The WIP PR can be a good first step to overall optimization.
Let's sync-up on the progress you have already made & what we want to
target.
@Jarek Potiuk <Ja...@polidea.com> & @Fokko - If we manage to make
it entirely backward-compatible with an enable/disable flag as we
mentioned, we can think of including it in 1.10.5 but I am in favor of
removing / cleaning stuff like pickles, drop Py 2.0 and cut Airflow 2.0 and
include this change there.
On Mon, Jul 29, 2019 at 1:03 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> Actually I am also doing a lot of v1-10-test merges during the last few
> months (probably several tens of them already). Rarely the conflicts are
> difficult to solve in fact. We have usually small, localised changes and
> until we go for full Black file re-formatting, we should be ok (and the
> change from Zhou seems rather small and localised).
>
> J.
>
> On Mon, Jul 29, 2019 at 9:25 AM Driesprong, Fokko <fo...@driesprong.frl>
> wrote:
>
> > I would be hesitant to merge it into 1.10.5. When I try to backport
> > anything into the 1.x branch, I get a whole bunch on merge conflicts,
> even
> > on the trivial tickets. For me, the only one who can really comment on
> this
> > would be Ash, since he's doing the bulk of the conflict resolving. Apart
> > from that, I'm really excited to make this happen!
> >
> > Cheers, Fokko
> >
> >
> >
> > Op zo 28 jul. 2019 om 20:23 schreef Jarek Potiuk <
> Jarek.Potiuk@polidea.com
> > >:
> >
> > > Some thought I have after looking at the proposal from Zhou.
> > >
> > > I think this is one of the most important things feature-wise for
> > Airflow.
> > > It looks like we have several in-progress attempts to solve the problem
> > and
> > > I guess we should agree common approach.
> > >
> > > I like very much the approach of Zhou (AIP-24). It does seem to
> minimise
> > > the changes needed in Airflow and it means that we with some
> > optimisations
> > > (caching mentioned by Fokko) - it can solve the major pain points and I
> > > think relatively quick and is potentially portable to 1.10.5 if we have
> > it.
> > >
> > > I wonder how much it overlaps/differs from what Kaxil and Ash ideas
> are.
> > If
> > > I read it correctly - it sounds like this idea will contain some more
> > > "fundamental" changes. Ones that are likely less backwards-compatible,
> > and
> > > potentially taking longer time to implement and test. And likely
> solving
> > > some of the problems better or even solving other problems. Am I right
> > with
> > > my assumptions?
> > >
> > > I think more information on this might be helpful so that we all know
> if
> > > those are two different AIPs, or whether they can be joined in one
> > effort,
> > > and how they relate to AIP-18/AIP-19 (should those be deprecated or
> > > independently implemented ?). Also - since 2.0.0 release is half a year
> > > ahead we should consider how it impact the roadmap.
> > >
> > > I can see three approaches here that we as community can follow (maybe
> I
> > am
> > > missing some :) ):
> > >
> > > 1) focus our work on single "complete" solution that will take longer
> > time
> > > and targets 2.0.0.
> > > 2) work on two of them: one quick/fast - potentially portable to
> 1.10.5m
> > > one longer-term for 2.0.0.
> > > 3) decide that the simple solution we have from Zhou (maybe with some
> > > modifications) is our target solution (for both 1.10.5 if we have it
> and
> > > 2.0.0):
> > >
> > > J.
> > >
> > > On Sat, Jul 27, 2019 at 11:43 AM Kevin Yang <yr...@gmail.com> wrote:
> > >
> > > > Nice job Zhou!
> > > >
> > > > Really excited, exactly what we wanted for the webserver scaling
> issue.
> > > > Want to add another big drive for Airbnb to start think about this
> > > > previously to support the effort: it can not only bring consistency
> > > between
> > > > webservers but also bring consistency between webserver and
> > > > scheduler/workers. It may be less of a problem if total DAG parsing
> > time
> > > is
> > > > small, but for us the total DAG parsing time is 15+ mins and we had
> to
> > > set
> > > > the webserver( gunicorn subprocesses) restart interval to 20 mins,
> > which
> > > > leads to a worst case 15+20+15=50 mins delay between scheduler start
> to
> > > > schedule things and users can see their deployed DAGs/changes...
> > > >
> > > > I'm not so sure about the scheduler performance improvement:
> currently
> > we
> > > > already feed the main scheduler process with SimpleDag through
> > > > DagFileProcessorManager running in a subprocess--in the future we
> feed
> > it
> > > > with data from DB, which is likely slower( tho the diff should have
> > > > negligible impact to the scheduler performance). In fact if we'd keep
> > the
> > > > existing behavior, try schedule only fresh parsed DAGs, then we may
> > need
> > > to
> > > > deal with some consistency issue--dag processor and the scheduler
> race
> > > for
> > > > updating the flag indicating if the DAG is newly parsed. No big deal
> > > there
> > > > but just some thoughts on the top of my head and hopefully can be
> > > helpful.
> > > >
> > > > And good idea on pre-rendering the template, believe template
> rendering
> > > was
> > > > the biggest concern in the previous discussion. We've also chose the
> > > > pre-rendering+JSON approach in our smart sensor API
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization
> > > > >
> > > > and
> > > > seems to be working fine--a supporting case for ur proposal ;)
> There's
> > a
> > > > WIP
> > > > PR <https://github.com/apache/airflow/pull/5499> for it just in case
> > you
> > > > are interested--maybe we can even share some logics.
> > > >
> > > > Thumbs-up again for this and please don't heisitate to reach out if
> you
> > > > want to discuss further with us or need any help from us.
> > > >
> > > >
> > > > Cheers,
> > > > Kevin Y
> > > >
> > > > On Sat, Jul 27, 2019 at 12:54 AM Driesprong, Fokko
> > <fokko@driesprong.frl
> > > >
> > > > wrote:
> > > >
> > > > > Looks great Zhou,
> > > > >
> > > > > I have one thing that pops in my mind while reading the AIP; should
> > > keep
> > > > > the caching on the webserver level. As the famous quote goes:
> *"There
> > > are
> > > > > only two hard things in Computer Science: cache invalidation and
> > naming
> > > > > things." -- Phil Karlton*
> > > > >
> > > > > Right now, the fundamental change that is being proposed in the AIP
> > is
> > > > > fetching the DAGs from the database in a serialized format, and not
> > > > parsing
> > > > > the Python files all the time. This will give already a great
> > > performance
> > > > > improvement on the webserver side because it removes a lot of the
> > > > > processing. However, since we're still fetching the DAGs from the
> > > > database
> > > > > in a regular interval, cache it in the local process, so we still
> > have
> > > > the
> > > > > two issues that Airflow is suffering from right now:
> > > > >
> > > > > 1. No snappy UI because it is still polling the database in a
> > > regular
> > > > > interval.
> > > > > 2. Inconsistency between webservers because they might poll in a
> > > > > different interval, I think we've all seen this:
> > > > > https://www.youtube.com/watch?v=sNrBruPS3r4
> > > > >
> > > > > As I also mentioned in the Slack channel, I strongly feel that we
> > > should
> > > > be
> > > > > able to render most views from the tables in the database, so
> without
> > > > > touching the blob. For specific views, we could just pull the blob
> > from
> > > > the
> > > > > database. In this case we always have the latest version, and we
> > tackle
> > > > the
> > > > > second point above.
> > > > >
> > > > > To tackle the first one, I also have an idea. We should change the
> > DAG
> > > > > parser from a loop to something that uses inotify
> > > > > https://pypi.org/project/inotify_simple/. This will change it from
> > > > polling
> > > > > to an event-driven design, which is much more performant and less
> > > > resource
> > > > > hungry. But this would be an AIP on its own.
> > > > >
> > > > > Again, great design and a comprehensive AIP, but I would include
> the
> > > > > caching on the webserver to greatly improve the user experience in
> > the
> > > > UI.
> > > > > Looking forward to the opinion of others on this.
> > > > >
> > > > > Cheers, Fokko
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Op za 27 jul. 2019 om 01:44 schreef Zhou Fang
> > > > <zhoufang@google.com.invalid
> > > > > >:
> > > > >
> > > > > > Hi Kaxi,
> > > > > >
> > > > > > Just sent out the AIP:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistence+in+DB+using+JSON+for+Airflow+Webserver+and+%28optional%29+Scheduler
> > > > > >
> > > > > > Thanks!
> > > > > > Zhou
> > > > > >
> > > > > >
> > > > > > On Fri, Jul 26, 2019 at 1:33 PM Zhou Fang <zh...@google.com>
> > > wrote:
> > > > > >
> > > > > > > Hi Kaxil,
> > > > > > >
> > > > > > > We are also working on persisting DAGs into DB using JSON for
> > > Airflow
> > > > > > > webserver in Google Composer. We target at minimizing the
> change
> > to
> > > > the
> > > > > > > current Airflow code. Happy to get synced on this!
> > > > > > >
> > > > > > > Here is our progress:
> > > > > > > (1) Serializing DAGs using Pickle to be used in webserver
> > > > > > > It has been launched in Composer. I am working on the PR to
> > > upstream
> > > > > it:
> > > > > > > https://github.com/apache/airflow/pull/5594
> > > > > > > Currently it does not support non-Airflow operators and we are
> > > > working
> > > > > on
> > > > > > > a fix.
> > > > > > >
> > > > > > > (2) Caching Pickled DAGs in DB to be used by webserver
> > > > > > > We have a proof-of-concept implementation, working on an AIP
> now.
> > > > > > >
> > > > > > > (3) Using JSON instead of Pickle in (1) and (2)
> > > > > > > Decided to use JSON because Pickle is not secure and human
> > > readable.
> > > > > The
> > > > > > > serialization approach is very similar to (1).
> > > > > > >
> > > > > > > I will update the RP (
> > https://github.com/apache/airflow/pull/5594)
> > > > to
> > > > > > > replace Pickle by JSON, and send our design of (2) as an AIP
> next
> > > > week.
> > > > > > > Glad to check together whether our implementation makes sense
> and
> > > do
> > > > > > > improvements on that.
> > > > > > >
> > > > > > > Thanks!
> > > > > > > Zhou
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Jul 26, 2019 at 7:37 AM Kaxil Naik <
> kaxilnaik@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > >> Hi all,
> > > > > > >>
> > > > > > >> We, at Astronomer, are going to spend time working on DAG
> > > > > Serialisation.
> > > > > > >> There are 2 AIPs that are somewhat related to what we plan to
> > work
> > > > on:
> > > > > > >>
> > > > > > >> - AIP-18 Persist all information from DAG file in DB
> > > > > > >> <
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB
> > > > > > >> >
> > > > > > >> - AIP-19 Making the webserver stateless
> > > > > > >> <
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless
> > > > > > >> >
> > > > > > >>
> > > > > > >> We plan to use JSON as the Serialisation format and store it
> as
> > a
> > > > blob
> > > > > > in
> > > > > > >> metadata DB.
> > > > > > >>
> > > > > > >> *Goals:*
> > > > > > >>
> > > > > > >> - Make Webserver Stateless
> > > > > > >> - Use the same version of the DAG across Webserver &
> > Scheduler
> > > > > > >> - Keep backward compatibility and have a flag (globally &
> at
> > > DAG
> > > > > > level)
> > > > > > >> to turn this feature on/off
> > > > > > >> - Enable DAG Versioning (extended Goal)
> > > > > > >>
> > > > > > >>
> > > > > > >> We will be preparing a proposal (AIP) after some research and
> > some
> > > > > > initial
> > > > > > >> work and open it for the suggestions of the community.
> > > > > > >>
> > > > > > >> We already had some good brain-storming sessions with Twitter
> > > folks
> > > > > > (DanD
> > > > > > >> &
> > > > > > >> Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from
> Uber)
> > > > which
> > > > > > >> will
> > > > > > >> be a good starting point for us.
> > > > > > >>
> > > > > > >> If anyone in the community is interested in it or has some
> > > > experience
> > > > > > >> about
> > > > > > >> the same and want to collaborate please let me know and join
> > > > > > >> #dag-serialisation channel on Airflow Slack.
> > > > > > >>
> > > > > > >> Regards,
> > > > > > >> Kaxil
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > >
> > > Jarek Potiuk
> > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > >
> > > M: +48 660 796 129 <+48660796129>
> > > [image: Polidea] <https://www.polidea.com/>
> > >
> >
> > >
> >
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
--
*Kaxil Naik*
*Big Data Consultant | DevOps Data Engineer*
*Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
Developer
*LinkedIn*: https://www.linkedin.com/in/kaxil
Re: Airflow DAG Serialisation
Posted by Jarek Potiuk <Ja...@polidea.com>.
Actually I am also doing a lot of v1-10-test merges during the last few
months (probably several tens of them already). Rarely the conflicts are
difficult to solve in fact. We have usually small, localised changes and
until we go for full Black file re-formatting, we should be ok (and the
change from Zhou seems rather small and localised).
J.
On Mon, Jul 29, 2019 at 9:25 AM Driesprong, Fokko <fo...@driesprong.frl>
wrote:
> I would be hesitant to merge it into 1.10.5. When I try to backport
> anything into the 1.x branch, I get a whole bunch on merge conflicts, even
> on the trivial tickets. For me, the only one who can really comment on this
> would be Ash, since he's doing the bulk of the conflict resolving. Apart
> from that, I'm really excited to make this happen!
>
> Cheers, Fokko
>
>
>
> Op zo 28 jul. 2019 om 20:23 schreef Jarek Potiuk <Jarek.Potiuk@polidea.com
> >:
>
> > Some thought I have after looking at the proposal from Zhou.
> >
> > I think this is one of the most important things feature-wise for
> Airflow.
> > It looks like we have several in-progress attempts to solve the problem
> and
> > I guess we should agree common approach.
> >
> > I like very much the approach of Zhou (AIP-24). It does seem to minimise
> > the changes needed in Airflow and it means that we with some
> optimisations
> > (caching mentioned by Fokko) - it can solve the major pain points and I
> > think relatively quick and is potentially portable to 1.10.5 if we have
> it.
> >
> > I wonder how much it overlaps/differs from what Kaxil and Ash ideas are.
> If
> > I read it correctly - it sounds like this idea will contain some more
> > "fundamental" changes. Ones that are likely less backwards-compatible,
> and
> > potentially taking longer time to implement and test. And likely solving
> > some of the problems better or even solving other problems. Am I right
> with
> > my assumptions?
> >
> > I think more information on this might be helpful so that we all know if
> > those are two different AIPs, or whether they can be joined in one
> effort,
> > and how they relate to AIP-18/AIP-19 (should those be deprecated or
> > independently implemented ?). Also - since 2.0.0 release is half a year
> > ahead we should consider how it impact the roadmap.
> >
> > I can see three approaches here that we as community can follow (maybe I
> am
> > missing some :) ):
> >
> > 1) focus our work on single "complete" solution that will take longer
> time
> > and targets 2.0.0.
> > 2) work on two of them: one quick/fast - potentially portable to 1.10.5m
> > one longer-term for 2.0.0.
> > 3) decide that the simple solution we have from Zhou (maybe with some
> > modifications) is our target solution (for both 1.10.5 if we have it and
> > 2.0.0):
> >
> > J.
> >
> > On Sat, Jul 27, 2019 at 11:43 AM Kevin Yang <yr...@gmail.com> wrote:
> >
> > > Nice job Zhou!
> > >
> > > Really excited, exactly what we wanted for the webserver scaling issue.
> > > Want to add another big drive for Airbnb to start think about this
> > > previously to support the effort: it can not only bring consistency
> > between
> > > webservers but also bring consistency between webserver and
> > > scheduler/workers. It may be less of a problem if total DAG parsing
> time
> > is
> > > small, but for us the total DAG parsing time is 15+ mins and we had to
> > set
> > > the webserver( gunicorn subprocesses) restart interval to 20 mins,
> which
> > > leads to a worst case 15+20+15=50 mins delay between scheduler start to
> > > schedule things and users can see their deployed DAGs/changes...
> > >
> > > I'm not so sure about the scheduler performance improvement: currently
> we
> > > already feed the main scheduler process with SimpleDag through
> > > DagFileProcessorManager running in a subprocess--in the future we feed
> it
> > > with data from DB, which is likely slower( tho the diff should have
> > > negligible impact to the scheduler performance). In fact if we'd keep
> the
> > > existing behavior, try schedule only fresh parsed DAGs, then we may
> need
> > to
> > > deal with some consistency issue--dag processor and the scheduler race
> > for
> > > updating the flag indicating if the DAG is newly parsed. No big deal
> > there
> > > but just some thoughts on the top of my head and hopefully can be
> > helpful.
> > >
> > > And good idea on pre-rendering the template, believe template rendering
> > was
> > > the biggest concern in the previous discussion. We've also chose the
> > > pre-rendering+JSON approach in our smart sensor API
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization
> > > >
> > > and
> > > seems to be working fine--a supporting case for ur proposal ;) There's
> a
> > > WIP
> > > PR <https://github.com/apache/airflow/pull/5499> for it just in case
> you
> > > are interested--maybe we can even share some logics.
> > >
> > > Thumbs-up again for this and please don't heisitate to reach out if you
> > > want to discuss further with us or need any help from us.
> > >
> > >
> > > Cheers,
> > > Kevin Y
> > >
> > > On Sat, Jul 27, 2019 at 12:54 AM Driesprong, Fokko
> <fokko@driesprong.frl
> > >
> > > wrote:
> > >
> > > > Looks great Zhou,
> > > >
> > > > I have one thing that pops in my mind while reading the AIP; should
> > keep
> > > > the caching on the webserver level. As the famous quote goes: *"There
> > are
> > > > only two hard things in Computer Science: cache invalidation and
> naming
> > > > things." -- Phil Karlton*
> > > >
> > > > Right now, the fundamental change that is being proposed in the AIP
> is
> > > > fetching the DAGs from the database in a serialized format, and not
> > > parsing
> > > > the Python files all the time. This will give already a great
> > performance
> > > > improvement on the webserver side because it removes a lot of the
> > > > processing. However, since we're still fetching the DAGs from the
> > > database
> > > > in a regular interval, cache it in the local process, so we still
> have
> > > the
> > > > two issues that Airflow is suffering from right now:
> > > >
> > > > 1. No snappy UI because it is still polling the database in a
> > regular
> > > > interval.
> > > > 2. Inconsistency between webservers because they might poll in a
> > > > different interval, I think we've all seen this:
> > > > https://www.youtube.com/watch?v=sNrBruPS3r4
> > > >
> > > > As I also mentioned in the Slack channel, I strongly feel that we
> > should
> > > be
> > > > able to render most views from the tables in the database, so without
> > > > touching the blob. For specific views, we could just pull the blob
> from
> > > the
> > > > database. In this case we always have the latest version, and we
> tackle
> > > the
> > > > second point above.
> > > >
> > > > To tackle the first one, I also have an idea. We should change the
> DAG
> > > > parser from a loop to something that uses inotify
> > > > https://pypi.org/project/inotify_simple/. This will change it from
> > > polling
> > > > to an event-driven design, which is much more performant and less
> > > resource
> > > > hungry. But this would be an AIP on its own.
> > > >
> > > > Again, great design and a comprehensive AIP, but I would include the
> > > > caching on the webserver to greatly improve the user experience in
> the
> > > UI.
> > > > Looking forward to the opinion of others on this.
> > > >
> > > > Cheers, Fokko
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Op za 27 jul. 2019 om 01:44 schreef Zhou Fang
> > > <zhoufang@google.com.invalid
> > > > >:
> > > >
> > > > > Hi Kaxi,
> > > > >
> > > > > Just sent out the AIP:
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistence+in+DB+using+JSON+for+Airflow+Webserver+and+%28optional%29+Scheduler
> > > > >
> > > > > Thanks!
> > > > > Zhou
> > > > >
> > > > >
> > > > > On Fri, Jul 26, 2019 at 1:33 PM Zhou Fang <zh...@google.com>
> > wrote:
> > > > >
> > > > > > Hi Kaxil,
> > > > > >
> > > > > > We are also working on persisting DAGs into DB using JSON for
> > Airflow
> > > > > > webserver in Google Composer. We target at minimizing the change
> to
> > > the
> > > > > > current Airflow code. Happy to get synced on this!
> > > > > >
> > > > > > Here is our progress:
> > > > > > (1) Serializing DAGs using Pickle to be used in webserver
> > > > > > It has been launched in Composer. I am working on the PR to
> > upstream
> > > > it:
> > > > > > https://github.com/apache/airflow/pull/5594
> > > > > > Currently it does not support non-Airflow operators and we are
> > > working
> > > > on
> > > > > > a fix.
> > > > > >
> > > > > > (2) Caching Pickled DAGs in DB to be used by webserver
> > > > > > We have a proof-of-concept implementation, working on an AIP now.
> > > > > >
> > > > > > (3) Using JSON instead of Pickle in (1) and (2)
> > > > > > Decided to use JSON because Pickle is not secure and human
> > readable.
> > > > The
> > > > > > serialization approach is very similar to (1).
> > > > > >
> > > > > > I will update the RP (
> https://github.com/apache/airflow/pull/5594)
> > > to
> > > > > > replace Pickle by JSON, and send our design of (2) as an AIP next
> > > week.
> > > > > > Glad to check together whether our implementation makes sense and
> > do
> > > > > > improvements on that.
> > > > > >
> > > > > > Thanks!
> > > > > > Zhou
> > > > > >
> > > > > >
> > > > > > On Fri, Jul 26, 2019 at 7:37 AM Kaxil Naik <ka...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > >> Hi all,
> > > > > >>
> > > > > >> We, at Astronomer, are going to spend time working on DAG
> > > > Serialisation.
> > > > > >> There are 2 AIPs that are somewhat related to what we plan to
> work
> > > on:
> > > > > >>
> > > > > >> - AIP-18 Persist all information from DAG file in DB
> > > > > >> <
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB
> > > > > >> >
> > > > > >> - AIP-19 Making the webserver stateless
> > > > > >> <
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless
> > > > > >> >
> > > > > >>
> > > > > >> We plan to use JSON as the Serialisation format and store it as
> a
> > > blob
> > > > > in
> > > > > >> metadata DB.
> > > > > >>
> > > > > >> *Goals:*
> > > > > >>
> > > > > >> - Make Webserver Stateless
> > > > > >> - Use the same version of the DAG across Webserver &
> Scheduler
> > > > > >> - Keep backward compatibility and have a flag (globally & at
> > DAG
> > > > > level)
> > > > > >> to turn this feature on/off
> > > > > >> - Enable DAG Versioning (extended Goal)
> > > > > >>
> > > > > >>
> > > > > >> We will be preparing a proposal (AIP) after some research and
> some
> > > > > initial
> > > > > >> work and open it for the suggestions of the community.
> > > > > >>
> > > > > >> We already had some good brain-storming sessions with Twitter
> > folks
> > > > > (DanD
> > > > > >> &
> > > > > >> Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from Uber)
> > > which
> > > > > >> will
> > > > > >> be a good starting point for us.
> > > > > >>
> > > > > >> If anyone in the community is interested in it or has some
> > > experience
> > > > > >> about
> > > > > >> the same and want to collaborate please let me know and join
> > > > > >> #dag-serialisation channel on Airflow Slack.
> > > > > >>
> > > > > >> Regards,
> > > > > >> Kaxil
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> >
> > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: +48 660 796 129 <+48660796129>
> > [image: Polidea] <https://www.polidea.com/>
> >
>
> >
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Airflow DAG Serialisation
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
I would be hesitant to merge it into 1.10.5. When I try to backport
anything into the 1.x branch, I get a whole bunch on merge conflicts, even
on the trivial tickets. For me, the only one who can really comment on this
would be Ash, since he's doing the bulk of the conflict resolving. Apart
from that, I'm really excited to make this happen!
Cheers, Fokko
Op zo 28 jul. 2019 om 20:23 schreef Jarek Potiuk <Ja...@polidea.com>:
> Some thought I have after looking at the proposal from Zhou.
>
> I think this is one of the most important things feature-wise for Airflow.
> It looks like we have several in-progress attempts to solve the problem and
> I guess we should agree common approach.
>
> I like very much the approach of Zhou (AIP-24). It does seem to minimise
> the changes needed in Airflow and it means that we with some optimisations
> (caching mentioned by Fokko) - it can solve the major pain points and I
> think relatively quick and is potentially portable to 1.10.5 if we have it.
>
> I wonder how much it overlaps/differs from what Kaxil and Ash ideas are. If
> I read it correctly - it sounds like this idea will contain some more
> "fundamental" changes. Ones that are likely less backwards-compatible, and
> potentially taking longer time to implement and test. And likely solving
> some of the problems better or even solving other problems. Am I right with
> my assumptions?
>
> I think more information on this might be helpful so that we all know if
> those are two different AIPs, or whether they can be joined in one effort,
> and how they relate to AIP-18/AIP-19 (should those be deprecated or
> independently implemented ?). Also - since 2.0.0 release is half a year
> ahead we should consider how it impact the roadmap.
>
> I can see three approaches here that we as community can follow (maybe I am
> missing some :) ):
>
> 1) focus our work on single "complete" solution that will take longer time
> and targets 2.0.0.
> 2) work on two of them: one quick/fast - potentially portable to 1.10.5m
> one longer-term for 2.0.0.
> 3) decide that the simple solution we have from Zhou (maybe with some
> modifications) is our target solution (for both 1.10.5 if we have it and
> 2.0.0):
>
> J.
>
> On Sat, Jul 27, 2019 at 11:43 AM Kevin Yang <yr...@gmail.com> wrote:
>
> > Nice job Zhou!
> >
> > Really excited, exactly what we wanted for the webserver scaling issue.
> > Want to add another big drive for Airbnb to start think about this
> > previously to support the effort: it can not only bring consistency
> between
> > webservers but also bring consistency between webserver and
> > scheduler/workers. It may be less of a problem if total DAG parsing time
> is
> > small, but for us the total DAG parsing time is 15+ mins and we had to
> set
> > the webserver( gunicorn subprocesses) restart interval to 20 mins, which
> > leads to a worst case 15+20+15=50 mins delay between scheduler start to
> > schedule things and users can see their deployed DAGs/changes...
> >
> > I'm not so sure about the scheduler performance improvement: currently we
> > already feed the main scheduler process with SimpleDag through
> > DagFileProcessorManager running in a subprocess--in the future we feed it
> > with data from DB, which is likely slower( tho the diff should have
> > negligible impact to the scheduler performance). In fact if we'd keep the
> > existing behavior, try schedule only fresh parsed DAGs, then we may need
> to
> > deal with some consistency issue--dag processor and the scheduler race
> for
> > updating the flag indicating if the DAG is newly parsed. No big deal
> there
> > but just some thoughts on the top of my head and hopefully can be
> helpful.
> >
> > And good idea on pre-rendering the template, believe template rendering
> was
> > the biggest concern in the previous discussion. We've also chose the
> > pre-rendering+JSON approach in our smart sensor API
> > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization
> > >
> > and
> > seems to be working fine--a supporting case for ur proposal ;) There's a
> > WIP
> > PR <https://github.com/apache/airflow/pull/5499> for it just in case you
> > are interested--maybe we can even share some logics.
> >
> > Thumbs-up again for this and please don't heisitate to reach out if you
> > want to discuss further with us or need any help from us.
> >
> >
> > Cheers,
> > Kevin Y
> >
> > On Sat, Jul 27, 2019 at 12:54 AM Driesprong, Fokko <fokko@driesprong.frl
> >
> > wrote:
> >
> > > Looks great Zhou,
> > >
> > > I have one thing that pops in my mind while reading the AIP; should
> keep
> > > the caching on the webserver level. As the famous quote goes: *"There
> are
> > > only two hard things in Computer Science: cache invalidation and naming
> > > things." -- Phil Karlton*
> > >
> > > Right now, the fundamental change that is being proposed in the AIP is
> > > fetching the DAGs from the database in a serialized format, and not
> > parsing
> > > the Python files all the time. This will give already a great
> performance
> > > improvement on the webserver side because it removes a lot of the
> > > processing. However, since we're still fetching the DAGs from the
> > database
> > > in a regular interval, cache it in the local process, so we still have
> > the
> > > two issues that Airflow is suffering from right now:
> > >
> > > 1. No snappy UI because it is still polling the database in a
> regular
> > > interval.
> > > 2. Inconsistency between webservers because they might poll in a
> > > different interval, I think we've all seen this:
> > > https://www.youtube.com/watch?v=sNrBruPS3r4
> > >
> > > As I also mentioned in the Slack channel, I strongly feel that we
> should
> > be
> > > able to render most views from the tables in the database, so without
> > > touching the blob. For specific views, we could just pull the blob from
> > the
> > > database. In this case we always have the latest version, and we tackle
> > the
> > > second point above.
> > >
> > > To tackle the first one, I also have an idea. We should change the DAG
> > > parser from a loop to something that uses inotify
> > > https://pypi.org/project/inotify_simple/. This will change it from
> > polling
> > > to an event-driven design, which is much more performant and less
> > resource
> > > hungry. But this would be an AIP on its own.
> > >
> > > Again, great design and a comprehensive AIP, but I would include the
> > > caching on the webserver to greatly improve the user experience in the
> > UI.
> > > Looking forward to the opinion of others on this.
> > >
> > > Cheers, Fokko
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > Op za 27 jul. 2019 om 01:44 schreef Zhou Fang
> > <zhoufang@google.com.invalid
> > > >:
> > >
> > > > Hi Kaxi,
> > > >
> > > > Just sent out the AIP:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistence+in+DB+using+JSON+for+Airflow+Webserver+and+%28optional%29+Scheduler
> > > >
> > > > Thanks!
> > > > Zhou
> > > >
> > > >
> > > > On Fri, Jul 26, 2019 at 1:33 PM Zhou Fang <zh...@google.com>
> wrote:
> > > >
> > > > > Hi Kaxil,
> > > > >
> > > > > We are also working on persisting DAGs into DB using JSON for
> Airflow
> > > > > webserver in Google Composer. We target at minimizing the change to
> > the
> > > > > current Airflow code. Happy to get synced on this!
> > > > >
> > > > > Here is our progress:
> > > > > (1) Serializing DAGs using Pickle to be used in webserver
> > > > > It has been launched in Composer. I am working on the PR to
> upstream
> > > it:
> > > > > https://github.com/apache/airflow/pull/5594
> > > > > Currently it does not support non-Airflow operators and we are
> > working
> > > on
> > > > > a fix.
> > > > >
> > > > > (2) Caching Pickled DAGs in DB to be used by webserver
> > > > > We have a proof-of-concept implementation, working on an AIP now.
> > > > >
> > > > > (3) Using JSON instead of Pickle in (1) and (2)
> > > > > Decided to use JSON because Pickle is not secure and human
> readable.
> > > The
> > > > > serialization approach is very similar to (1).
> > > > >
> > > > > I will update the RP (https://github.com/apache/airflow/pull/5594)
> > to
> > > > > replace Pickle by JSON, and send our design of (2) as an AIP next
> > week.
> > > > > Glad to check together whether our implementation makes sense and
> do
> > > > > improvements on that.
> > > > >
> > > > > Thanks!
> > > > > Zhou
> > > > >
> > > > >
> > > > > On Fri, Jul 26, 2019 at 7:37 AM Kaxil Naik <ka...@gmail.com>
> > > wrote:
> > > > >
> > > > >> Hi all,
> > > > >>
> > > > >> We, at Astronomer, are going to spend time working on DAG
> > > Serialisation.
> > > > >> There are 2 AIPs that are somewhat related to what we plan to work
> > on:
> > > > >>
> > > > >> - AIP-18 Persist all information from DAG file in DB
> > > > >> <
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB
> > > > >> >
> > > > >> - AIP-19 Making the webserver stateless
> > > > >> <
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless
> > > > >> >
> > > > >>
> > > > >> We plan to use JSON as the Serialisation format and store it as a
> > blob
> > > > in
> > > > >> metadata DB.
> > > > >>
> > > > >> *Goals:*
> > > > >>
> > > > >> - Make Webserver Stateless
> > > > >> - Use the same version of the DAG across Webserver & Scheduler
> > > > >> - Keep backward compatibility and have a flag (globally & at
> DAG
> > > > level)
> > > > >> to turn this feature on/off
> > > > >> - Enable DAG Versioning (extended Goal)
> > > > >>
> > > > >>
> > > > >> We will be preparing a proposal (AIP) after some research and some
> > > > initial
> > > > >> work and open it for the suggestions of the community.
> > > > >>
> > > > >> We already had some good brain-storming sessions with Twitter
> folks
> > > > (DanD
> > > > >> &
> > > > >> Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from Uber)
> > which
> > > > >> will
> > > > >> be a good starting point for us.
> > > > >>
> > > > >> If anyone in the community is interested in it or has some
> > experience
> > > > >> about
> > > > >> the same and want to collaborate please let me know and join
> > > > >> #dag-serialisation channel on Airflow Slack.
> > > > >>
> > > > >> Regards,
> > > > >> Kaxil
> > > > >>
> > > > >
> > > >
> > >
> >
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
Re: Airflow DAG Serialisation
Posted by Jarek Potiuk <Ja...@polidea.com>.
Some thought I have after looking at the proposal from Zhou.
I think this is one of the most important things feature-wise for Airflow.
It looks like we have several in-progress attempts to solve the problem and
I guess we should agree common approach.
I like very much the approach of Zhou (AIP-24). It does seem to minimise
the changes needed in Airflow and it means that we with some optimisations
(caching mentioned by Fokko) - it can solve the major pain points and I
think relatively quick and is potentially portable to 1.10.5 if we have it.
I wonder how much it overlaps/differs from what Kaxil and Ash ideas are. If
I read it correctly - it sounds like this idea will contain some more
"fundamental" changes. Ones that are likely less backwards-compatible, and
potentially taking longer time to implement and test. And likely solving
some of the problems better or even solving other problems. Am I right with
my assumptions?
I think more information on this might be helpful so that we all know if
those are two different AIPs, or whether they can be joined in one effort,
and how they relate to AIP-18/AIP-19 (should those be deprecated or
independently implemented ?). Also - since 2.0.0 release is half a year
ahead we should consider how it impact the roadmap.
I can see three approaches here that we as community can follow (maybe I am
missing some :) ):
1) focus our work on single "complete" solution that will take longer time
and targets 2.0.0.
2) work on two of them: one quick/fast - potentially portable to 1.10.5m
one longer-term for 2.0.0.
3) decide that the simple solution we have from Zhou (maybe with some
modifications) is our target solution (for both 1.10.5 if we have it and
2.0.0):
J.
On Sat, Jul 27, 2019 at 11:43 AM Kevin Yang <yr...@gmail.com> wrote:
> Nice job Zhou!
>
> Really excited, exactly what we wanted for the webserver scaling issue.
> Want to add another big drive for Airbnb to start think about this
> previously to support the effort: it can not only bring consistency between
> webservers but also bring consistency between webserver and
> scheduler/workers. It may be less of a problem if total DAG parsing time is
> small, but for us the total DAG parsing time is 15+ mins and we had to set
> the webserver( gunicorn subprocesses) restart interval to 20 mins, which
> leads to a worst case 15+20+15=50 mins delay between scheduler start to
> schedule things and users can see their deployed DAGs/changes...
>
> I'm not so sure about the scheduler performance improvement: currently we
> already feed the main scheduler process with SimpleDag through
> DagFileProcessorManager running in a subprocess--in the future we feed it
> with data from DB, which is likely slower( tho the diff should have
> negligible impact to the scheduler performance). In fact if we'd keep the
> existing behavior, try schedule only fresh parsed DAGs, then we may need to
> deal with some consistency issue--dag processor and the scheduler race for
> updating the flag indicating if the DAG is newly parsed. No big deal there
> but just some thoughts on the top of my head and hopefully can be helpful.
>
> And good idea on pre-rendering the template, believe template rendering was
> the biggest concern in the previous discussion. We've also chose the
> pre-rendering+JSON approach in our smart sensor API
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization
> >
> and
> seems to be working fine--a supporting case for ur proposal ;) There's a
> WIP
> PR <https://github.com/apache/airflow/pull/5499> for it just in case you
> are interested--maybe we can even share some logics.
>
> Thumbs-up again for this and please don't heisitate to reach out if you
> want to discuss further with us or need any help from us.
>
>
> Cheers,
> Kevin Y
>
> On Sat, Jul 27, 2019 at 12:54 AM Driesprong, Fokko <fo...@driesprong.frl>
> wrote:
>
> > Looks great Zhou,
> >
> > I have one thing that pops in my mind while reading the AIP; should keep
> > the caching on the webserver level. As the famous quote goes: *"There are
> > only two hard things in Computer Science: cache invalidation and naming
> > things." -- Phil Karlton*
> >
> > Right now, the fundamental change that is being proposed in the AIP is
> > fetching the DAGs from the database in a serialized format, and not
> parsing
> > the Python files all the time. This will give already a great performance
> > improvement on the webserver side because it removes a lot of the
> > processing. However, since we're still fetching the DAGs from the
> database
> > in a regular interval, cache it in the local process, so we still have
> the
> > two issues that Airflow is suffering from right now:
> >
> > 1. No snappy UI because it is still polling the database in a regular
> > interval.
> > 2. Inconsistency between webservers because they might poll in a
> > different interval, I think we've all seen this:
> > https://www.youtube.com/watch?v=sNrBruPS3r4
> >
> > As I also mentioned in the Slack channel, I strongly feel that we should
> be
> > able to render most views from the tables in the database, so without
> > touching the blob. For specific views, we could just pull the blob from
> the
> > database. In this case we always have the latest version, and we tackle
> the
> > second point above.
> >
> > To tackle the first one, I also have an idea. We should change the DAG
> > parser from a loop to something that uses inotify
> > https://pypi.org/project/inotify_simple/. This will change it from
> polling
> > to an event-driven design, which is much more performant and less
> resource
> > hungry. But this would be an AIP on its own.
> >
> > Again, great design and a comprehensive AIP, but I would include the
> > caching on the webserver to greatly improve the user experience in the
> UI.
> > Looking forward to the opinion of others on this.
> >
> > Cheers, Fokko
> >
> >
> >
> >
> >
> >
> >
> >
> > Op za 27 jul. 2019 om 01:44 schreef Zhou Fang
> <zhoufang@google.com.invalid
> > >:
> >
> > > Hi Kaxi,
> > >
> > > Just sent out the AIP:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistence+in+DB+using+JSON+for+Airflow+Webserver+and+%28optional%29+Scheduler
> > >
> > > Thanks!
> > > Zhou
> > >
> > >
> > > On Fri, Jul 26, 2019 at 1:33 PM Zhou Fang <zh...@google.com> wrote:
> > >
> > > > Hi Kaxil,
> > > >
> > > > We are also working on persisting DAGs into DB using JSON for Airflow
> > > > webserver in Google Composer. We target at minimizing the change to
> the
> > > > current Airflow code. Happy to get synced on this!
> > > >
> > > > Here is our progress:
> > > > (1) Serializing DAGs using Pickle to be used in webserver
> > > > It has been launched in Composer. I am working on the PR to upstream
> > it:
> > > > https://github.com/apache/airflow/pull/5594
> > > > Currently it does not support non-Airflow operators and we are
> working
> > on
> > > > a fix.
> > > >
> > > > (2) Caching Pickled DAGs in DB to be used by webserver
> > > > We have a proof-of-concept implementation, working on an AIP now.
> > > >
> > > > (3) Using JSON instead of Pickle in (1) and (2)
> > > > Decided to use JSON because Pickle is not secure and human readable.
> > The
> > > > serialization approach is very similar to (1).
> > > >
> > > > I will update the RP (https://github.com/apache/airflow/pull/5594)
> to
> > > > replace Pickle by JSON, and send our design of (2) as an AIP next
> week.
> > > > Glad to check together whether our implementation makes sense and do
> > > > improvements on that.
> > > >
> > > > Thanks!
> > > > Zhou
> > > >
> > > >
> > > > On Fri, Jul 26, 2019 at 7:37 AM Kaxil Naik <ka...@gmail.com>
> > wrote:
> > > >
> > > >> Hi all,
> > > >>
> > > >> We, at Astronomer, are going to spend time working on DAG
> > Serialisation.
> > > >> There are 2 AIPs that are somewhat related to what we plan to work
> on:
> > > >>
> > > >> - AIP-18 Persist all information from DAG file in DB
> > > >> <
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB
> > > >> >
> > > >> - AIP-19 Making the webserver stateless
> > > >> <
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless
> > > >> >
> > > >>
> > > >> We plan to use JSON as the Serialisation format and store it as a
> blob
> > > in
> > > >> metadata DB.
> > > >>
> > > >> *Goals:*
> > > >>
> > > >> - Make Webserver Stateless
> > > >> - Use the same version of the DAG across Webserver & Scheduler
> > > >> - Keep backward compatibility and have a flag (globally & at DAG
> > > level)
> > > >> to turn this feature on/off
> > > >> - Enable DAG Versioning (extended Goal)
> > > >>
> > > >>
> > > >> We will be preparing a proposal (AIP) after some research and some
> > > initial
> > > >> work and open it for the suggestions of the community.
> > > >>
> > > >> We already had some good brain-storming sessions with Twitter folks
> > > (DanD
> > > >> &
> > > >> Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from Uber)
> which
> > > >> will
> > > >> be a good starting point for us.
> > > >>
> > > >> If anyone in the community is interested in it or has some
> experience
> > > >> about
> > > >> the same and want to collaborate please let me know and join
> > > >> #dag-serialisation channel on Airflow Slack.
> > > >>
> > > >> Regards,
> > > >> Kaxil
> > > >>
> > > >
> > >
> >
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Airflow DAG Serialisation
Posted by Philippe Gagnon <ph...@gmail.com>.
I am not sure that pushing the responsibility of generating DAGs to clients
outright is the right approach (I think it might raise the barrier to entry
for newcomers), but I like the design that you are proposing. If we were to
completely decouple DAG parsing from the scheduler then DAG processing
could be scaled horizontally.
On Wed, Jul 31, 2019 at 8:13 AM Dan Davydov <dd...@twitter.com.invalid>
wrote:
> An idea for serialization of dynamic DAGs is moving the serialization to
> the actual clients.
> This would require having a python Airflow API that the clients could call
> like dag.publish(). This enables a couple of things:
> 1) Clients can serialize as often as they like, and can even serialize in
> an event-driven approach (e.g. when one of the underlying data sources
> changes) as opposed to polling.
> 2) Very lightweight processing for the scheduler (only need to parse JSON
> rather than DAG python files), especially for DAGs that are harder to
> parse. Eventually we can even make the Scheduler event based once this is
> done (DAG "publishes" retrigger DAGs to be parsed rather than polling on an
> interval).
> 3) Enables security/multi-tenancy; A) the scheduler no long has to run
> arbitrary python code from DAG definitions which could potentially do
> something nefarious, and wouldn't even need to "sudo su". B) there would be
> first-class support for auth at the publish stage.
> 4) Clients can decide what kinds of dependencies they want to use in order
> to parse their DAGs, the scheduler no longer e.g. needs to include the
> mysql python library, and all Airflow schedulers can have the exact same
> set of dependencies.
>
>
> On Wed, Jul 31, 2019 at 9:19 AM Driesprong, Fokko <fo...@driesprong.frl>
> wrote:
>
> > Hi Jon,
> >
> > I would argue that that would be the wrong approach. The processing of
> the
> > DAGs is being moved to one single place, and there they will be parsed in
> > parallel. This is already being done in the scheduler (look for
> max_threads
> > in the code). As soon as we have the DAG serialized properly, the work on
> > the webserver side should be minimal, i.e. deserializing and showing the
> > DAG. The dynamic generation will be done on the scheduler.
> >
> > The issue with the dynamic DAGs is similar. For example, my consultancy
> > company implements a LOT of Airflow dags at customers and dynamic dags
> are
> > still challenging. For example, we had one Python file which would
> generate
> > around 350 DAGs with 12 tasks. Each time one of the executors would run
> one
> > of the tasks, it would inflate the whole DAG and run the specific task.
> > This introduced a lot of overhead. @Bas Harenslak
> > <ba...@gmail.com> solved
> > this by generating 350 DAGs from a Jinja template, which works quite
> well.
> >
> > Keeping AIP-24 small and simple sounds like a very good idea, we've seen
> > scope creeps before on similar AIP's :-)
> >
> > Cheers, Fokko Driesprong
> >
> > Op di 30 jul. 2019 om 15:46 schreef Ash Berlin-Taylor <as...@apache.org>:
> >
> > > Hi Jon,
> > >
> > > As part of this AIP(24) we aren't going to touch the scheduler any more
> > > than absolutely required, but yes, better support of dynamic DAGs is
> > _very
> > > much_ on Kaxil and I's hit list.
> > >
> > > Our rough approach right now is to design the serialisation format well
> > > enough (including versioning it so we can change it over time) such
> that
> > we
> > > can change the scheduler to not be as coupled with the dag parsing
> loop.
> > > But for the sake of small, reviewable PRs we'll do it bit-by-bit.
> > >
> > > -ash
> > >
> > > On 2019/07/30 13:33:53, Jonathan Miles <jo...@cybus.co.uk> wrote:
> > > > Another ask for the long-term list.
> > > >
> > > > From a superficial read of the code, it looks like this asynchronous
> > > > DAG loading approach could also be a stepping stone towards loading
> > DAGs
> > > > in parallel? I've come across a case of someone dynamically
> generating
> > a
> > > > DAG based on an external data source. Problem with that is when the
> > data
> > > > source isn't available or is slow, it can block the loading of other
> > > > DAGs. Loading in parallel could isolate the failing or slow DAGs from
> > > > the good ones.
> > > >
> > > > I suppose even with this patch, randomising the load order of DAGs
> > could
> > > > also provide some basic protection against a small set of failing
> DAGs.
> > > > At least some would get updated.
> > > >
> > > > Do the changes only affect the webserver or also loading in the
> > > scheduler?
> > > >
> > > > Thanks,
> > > >
> > > > Jon
> > > >
> > > > On 29/07/2019 22:18, Zhou Fang wrote:
> > > > > Hi Kevin,
> > > > >
> > > > > The problem that DAG parsing takes a long time can be solved by
> > > > > Asynchronous DAG loading:
> > https://github.com/apache/airflow/pull/5594
> > > > >
> > > > > The idea is the a background process parses DAG files, and sends
> DAGs
> > > to
> > > > > webserver process every [webserver] dagbag_sync_interval = 10s.
> > > > >
> > > > > We have launched it in Composer, so our users can set webserver
> > worker
> > > > > restart interval to 1 hour (or longer). The background DAG parsing
> > > > > processing refresh all DAGs per [webserver] =
> collect_dags_interval =
> > > 30s.
> > > > >
> > > > > If parsing all DAGs take 15min, you can see DAGs being gradually
> > > freshed
> > > > > with this feature.
> > > > >
> > > > > Thanks,
> > > > > Zhou
> > > > >
> > > > >
> > > > > On Sat, Jul 27, 2019 at 2:43 AM Kevin Yang <yr...@gmail.com>
> > wrote:
> > > > >
> > > > >> Nice job Zhou!
> > > > >>
> > > > >> Really excited, exactly what we wanted for the webserver scaling
> > > issue.
> > > > >> Want to add another big drive for Airbnb to start think about this
> > > > >> previously to support the effort: it can not only bring
> consistency
> > > between
> > > > >> webservers but also bring consistency between webserver and
> > > > >> scheduler/workers. It may be less of a problem if total DAG
> parsing
> > > time is
> > > > >> small, but for us the total DAG parsing time is 15+ mins and we
> had
> > > to set
> > > > >> the webserver( gunicorn subprocesses) restart interval to 20 mins,
> > > which
> > > > >> leads to a worst case 15+20+15=50 mins delay between scheduler
> start
> > > to
> > > > >> schedule things and users can see their deployed DAGs/changes...
> > > > >>
> > > > >> I'm not so sure about the scheduler performance improvement:
> > > currently we
> > > > >> already feed the main scheduler process with SimpleDag through
> > > > >> DagFileProcessorManager running in a subprocess--in the future we
> > > feed it
> > > > >> with data from DB, which is likely slower( tho the diff should
> have
> > > > >> negligible impact to the scheduler performance). In fact if we'd
> > keep
> > > the
> > > > >> existing behavior, try schedule only fresh parsed DAGs, then we
> may
> > > need to
> > > > >> deal with some consistency issue--dag processor and the scheduler
> > > race for
> > > > >> updating the flag indicating if the DAG is newly parsed. No big
> deal
> > > there
> > > > >> but just some thoughts on the top of my head and hopefully can be
> > > helpful.
> > > > >>
> > > > >> And good idea on pre-rendering the template, believe template
> > > rendering was
> > > > >> the biggest concern in the previous discussion. We've also chose
> the
> > > > >> pre-rendering+JSON approach in our smart sensor API
> > > > >> <
> > > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization
> > > > >> and
> > > > >> seems to be working fine--a supporting case for ur proposal ;)
> > > There's a
> > > > >> WIP
> > > > >> PR <https://github.com/apache/airflow/pull/5499> for it just in
> > case
> > > you
> > > > >> are interested--maybe we can even share some logics.
> > > > >>
> > > > >> Thumbs-up again for this and please don't heisitate to reach out
> if
> > > you
> > > > >> want to discuss further with us or need any help from us.
> > > > >>
> > > > >>
> > > > >> Cheers,
> > > > >> Kevin Y
> > > > >>
> > > > >> On Sat, Jul 27, 2019 at 12:54 AM Driesprong, Fokko
> > > <fo...@driesprong.frl>
> > > > >> wrote:
> > > > >>
> > > > >>> Looks great Zhou,
> > > > >>>
> > > > >>> I have one thing that pops in my mind while reading the AIP;
> should
> > > keep
> > > > >>> the caching on the webserver level. As the famous quote goes:
> > > *"There are
> > > > >>> only two hard things in Computer Science: cache invalidation and
> > > naming
> > > > >>> things." -- Phil Karlton*
> > > > >>>
> > > > >>> Right now, the fundamental change that is being proposed in the
> AIP
> > > is
> > > > >>> fetching the DAGs from the database in a serialized format, and
> not
> > > > >> parsing
> > > > >>> the Python files all the time. This will give already a great
> > > performance
> > > > >>> improvement on the webserver side because it removes a lot of the
> > > > >>> processing. However, since we're still fetching the DAGs from the
> > > > >> database
> > > > >>> in a regular interval, cache it in the local process, so we still
> > > have
> > > > >> the
> > > > >>> two issues that Airflow is suffering from right now:
> > > > >>>
> > > > >>> 1. No snappy UI because it is still polling the database in a
> > > regular
> > > > >>> interval.
> > > > >>> 2. Inconsistency between webservers because they might poll
> in
> > a
> > > > >>> different interval, I think we've all seen this:
> > > > >>> https://www.youtube.com/watch?v=sNrBruPS3r4
> > > > >>>
> > > > >>> As I also mentioned in the Slack channel, I strongly feel that we
> > > should
> > > > >> be
> > > > >>> able to render most views from the tables in the database, so
> > without
> > > > >>> touching the blob. For specific views, we could just pull the
> blob
> > > from
> > > > >> the
> > > > >>> database. In this case we always have the latest version, and we
> > > tackle
> > > > >> the
> > > > >>> second point above.
> > > > >>>
> > > > >>> To tackle the first one, I also have an idea. We should change
> the
> > > DAG
> > > > >>> parser from a loop to something that uses inotify
> > > > >>> https://pypi.org/project/inotify_simple/. This will change it
> from
> > > > >> polling
> > > > >>> to an event-driven design, which is much more performant and less
> > > > >> resource
> > > > >>> hungry. But this would be an AIP on its own.
> > > > >>>
> > > > >>> Again, great design and a comprehensive AIP, but I would include
> > the
> > > > >>> caching on the webserver to greatly improve the user experience
> in
> > > the
> > > > >> UI.
> > > > >>> Looking forward to the opinion of others on this.
> > > > >>>
> > > > >>> Cheers, Fokko
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> Op za 27 jul. 2019 om 01:44 schreef Zhou Fang
> > > > >> <zhoufang@google.com.invalid
> > > > >>>> :
> > > > >>>> Hi Kaxi,
> > > > >>>>
> > > > >>>> Just sent out the AIP:
> > > > >>>>
> > > > >>>>
> > > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistence+in+DB+using+JSON+for+Airflow+Webserver+and+%28optional%29+Scheduler
> > > > >>>> Thanks!
> > > > >>>> Zhou
> > > > >>>>
> > > > >>>>
> > > > >>>> On Fri, Jul 26, 2019 at 1:33 PM Zhou Fang <zh...@google.com>
> > > wrote:
> > > > >>>>
> > > > >>>>> Hi Kaxil,
> > > > >>>>>
> > > > >>>>> We are also working on persisting DAGs into DB using JSON for
> > > Airflow
> > > > >>>>> webserver in Google Composer. We target at minimizing the
> change
> > to
> > > > >> the
> > > > >>>>> current Airflow code. Happy to get synced on this!
> > > > >>>>>
> > > > >>>>> Here is our progress:
> > > > >>>>> (1) Serializing DAGs using Pickle to be used in webserver
> > > > >>>>> It has been launched in Composer. I am working on the PR to
> > > upstream
> > > > >>> it:
> > > > >>>>> https://github.com/apache/airflow/pull/5594
> > > > >>>>> Currently it does not support non-Airflow operators and we are
> > > > >> working
> > > > >>> on
> > > > >>>>> a fix.
> > > > >>>>>
> > > > >>>>> (2) Caching Pickled DAGs in DB to be used by webserver
> > > > >>>>> We have a proof-of-concept implementation, working on an AIP
> now.
> > > > >>>>>
> > > > >>>>> (3) Using JSON instead of Pickle in (1) and (2)
> > > > >>>>> Decided to use JSON because Pickle is not secure and human
> > > readable.
> > > > >>> The
> > > > >>>>> serialization approach is very similar to (1).
> > > > >>>>>
> > > > >>>>> I will update the RP (
> > https://github.com/apache/airflow/pull/5594)
> > > > >> to
> > > > >>>>> replace Pickle by JSON, and send our design of (2) as an AIP
> next
> > > > >> week.
> > > > >>>>> Glad to check together whether our implementation makes sense
> and
> > > do
> > > > >>>>> improvements on that.
> > > > >>>>>
> > > > >>>>> Thanks!
> > > > >>>>> Zhou
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> On Fri, Jul 26, 2019 at 7:37 AM Kaxil Naik <
> kaxilnaik@gmail.com>
> > > > >>> wrote:
> > > > >>>>>> Hi all,
> > > > >>>>>>
> > > > >>>>>> We, at Astronomer, are going to spend time working on DAG
> > > > >>> Serialisation.
> > > > >>>>>> There are 2 AIPs that are somewhat related to what we plan to
> > work
> > > > >> on:
> > > > >>>>>> - AIP-18 Persist all information from DAG file in DB
> > > > >>>>>> <
> > > > >>>>>>
> > > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB
> > > > >>>>>> - AIP-19 Making the webserver stateless
> > > > >>>>>> <
> > > > >>>>>>
> > > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless
> > > > >>>>>> We plan to use JSON as the Serialisation format and store it
> as
> > a
> > > > >> blob
> > > > >>>> in
> > > > >>>>>> metadata DB.
> > > > >>>>>>
> > > > >>>>>> *Goals:*
> > > > >>>>>>
> > > > >>>>>> - Make Webserver Stateless
> > > > >>>>>> - Use the same version of the DAG across Webserver &
> > Scheduler
> > > > >>>>>> - Keep backward compatibility and have a flag (globally &
> at
> > > DAG
> > > > >>>> level)
> > > > >>>>>> to turn this feature on/off
> > > > >>>>>> - Enable DAG Versioning (extended Goal)
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> We will be preparing a proposal (AIP) after some research and
> > some
> > > > >>>> initial
> > > > >>>>>> work and open it for the suggestions of the community.
> > > > >>>>>>
> > > > >>>>>> We already had some good brain-storming sessions with Twitter
> > > folks
> > > > >>>> (DanD
> > > > >>>>>> &
> > > > >>>>>> Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from
> Uber)
> > > > >> which
> > > > >>>>>> will
> > > > >>>>>> be a good starting point for us.
> > > > >>>>>>
> > > > >>>>>> If anyone in the community is interested in it or has some
> > > > >> experience
> > > > >>>>>> about
> > > > >>>>>> the same and want to collaborate please let me know and join
> > > > >>>>>> #dag-serialisation channel on Airflow Slack.
> > > > >>>>>>
> > > > >>>>>> Regards,
> > > > >>>>>> Kaxil
> > > > >>>>>>
> > > >
> > >
> >
>
Re: Airflow DAG Serialisation
Posted by Dan Davydov <dd...@twitter.com.INVALID>.
An idea for serialization of dynamic DAGs is moving the serialization to
the actual clients.
This would require having a python Airflow API that the clients could call
like dag.publish(). This enables a couple of things:
1) Clients can serialize as often as they like, and can even serialize in
an event-driven approach (e.g. when one of the underlying data sources
changes) as opposed to polling.
2) Very lightweight processing for the scheduler (only need to parse JSON
rather than DAG python files), especially for DAGs that are harder to
parse. Eventually we can even make the Scheduler event based once this is
done (DAG "publishes" retrigger DAGs to be parsed rather than polling on an
interval).
3) Enables security/multi-tenancy; A) the scheduler no long has to run
arbitrary python code from DAG definitions which could potentially do
something nefarious, and wouldn't even need to "sudo su". B) there would be
first-class support for auth at the publish stage.
4) Clients can decide what kinds of dependencies they want to use in order
to parse their DAGs, the scheduler no longer e.g. needs to include the
mysql python library, and all Airflow schedulers can have the exact same
set of dependencies.
On Wed, Jul 31, 2019 at 9:19 AM Driesprong, Fokko <fo...@driesprong.frl>
wrote:
> Hi Jon,
>
> I would argue that that would be the wrong approach. The processing of the
> DAGs is being moved to one single place, and there they will be parsed in
> parallel. This is already being done in the scheduler (look for max_threads
> in the code). As soon as we have the DAG serialized properly, the work on
> the webserver side should be minimal, i.e. deserializing and showing the
> DAG. The dynamic generation will be done on the scheduler.
>
> The issue with the dynamic DAGs is similar. For example, my consultancy
> company implements a LOT of Airflow dags at customers and dynamic dags are
> still challenging. For example, we had one Python file which would generate
> around 350 DAGs with 12 tasks. Each time one of the executors would run one
> of the tasks, it would inflate the whole DAG and run the specific task.
> This introduced a lot of overhead. @Bas Harenslak
> <ba...@gmail.com> solved
> this by generating 350 DAGs from a Jinja template, which works quite well.
>
> Keeping AIP-24 small and simple sounds like a very good idea, we've seen
> scope creeps before on similar AIP's :-)
>
> Cheers, Fokko Driesprong
>
> Op di 30 jul. 2019 om 15:46 schreef Ash Berlin-Taylor <as...@apache.org>:
>
> > Hi Jon,
> >
> > As part of this AIP(24) we aren't going to touch the scheduler any more
> > than absolutely required, but yes, better support of dynamic DAGs is
> _very
> > much_ on Kaxil and I's hit list.
> >
> > Our rough approach right now is to design the serialisation format well
> > enough (including versioning it so we can change it over time) such that
> we
> > can change the scheduler to not be as coupled with the dag parsing loop.
> > But for the sake of small, reviewable PRs we'll do it bit-by-bit.
> >
> > -ash
> >
> > On 2019/07/30 13:33:53, Jonathan Miles <jo...@cybus.co.uk> wrote:
> > > Another ask for the long-term list.
> > >
> > > From a superficial read of the code, it looks like this asynchronous
> > > DAG loading approach could also be a stepping stone towards loading
> DAGs
> > > in parallel? I've come across a case of someone dynamically generating
> a
> > > DAG based on an external data source. Problem with that is when the
> data
> > > source isn't available or is slow, it can block the loading of other
> > > DAGs. Loading in parallel could isolate the failing or slow DAGs from
> > > the good ones.
> > >
> > > I suppose even with this patch, randomising the load order of DAGs
> could
> > > also provide some basic protection against a small set of failing DAGs.
> > > At least some would get updated.
> > >
> > > Do the changes only affect the webserver or also loading in the
> > scheduler?
> > >
> > > Thanks,
> > >
> > > Jon
> > >
> > > On 29/07/2019 22:18, Zhou Fang wrote:
> > > > Hi Kevin,
> > > >
> > > > The problem that DAG parsing takes a long time can be solved by
> > > > Asynchronous DAG loading:
> https://github.com/apache/airflow/pull/5594
> > > >
> > > > The idea is the a background process parses DAG files, and sends DAGs
> > to
> > > > webserver process every [webserver] dagbag_sync_interval = 10s.
> > > >
> > > > We have launched it in Composer, so our users can set webserver
> worker
> > > > restart interval to 1 hour (or longer). The background DAG parsing
> > > > processing refresh all DAGs per [webserver] = collect_dags_interval =
> > 30s.
> > > >
> > > > If parsing all DAGs take 15min, you can see DAGs being gradually
> > freshed
> > > > with this feature.
> > > >
> > > > Thanks,
> > > > Zhou
> > > >
> > > >
> > > > On Sat, Jul 27, 2019 at 2:43 AM Kevin Yang <yr...@gmail.com>
> wrote:
> > > >
> > > >> Nice job Zhou!
> > > >>
> > > >> Really excited, exactly what we wanted for the webserver scaling
> > issue.
> > > >> Want to add another big drive for Airbnb to start think about this
> > > >> previously to support the effort: it can not only bring consistency
> > between
> > > >> webservers but also bring consistency between webserver and
> > > >> scheduler/workers. It may be less of a problem if total DAG parsing
> > time is
> > > >> small, but for us the total DAG parsing time is 15+ mins and we had
> > to set
> > > >> the webserver( gunicorn subprocesses) restart interval to 20 mins,
> > which
> > > >> leads to a worst case 15+20+15=50 mins delay between scheduler start
> > to
> > > >> schedule things and users can see their deployed DAGs/changes...
> > > >>
> > > >> I'm not so sure about the scheduler performance improvement:
> > currently we
> > > >> already feed the main scheduler process with SimpleDag through
> > > >> DagFileProcessorManager running in a subprocess--in the future we
> > feed it
> > > >> with data from DB, which is likely slower( tho the diff should have
> > > >> negligible impact to the scheduler performance). In fact if we'd
> keep
> > the
> > > >> existing behavior, try schedule only fresh parsed DAGs, then we may
> > need to
> > > >> deal with some consistency issue--dag processor and the scheduler
> > race for
> > > >> updating the flag indicating if the DAG is newly parsed. No big deal
> > there
> > > >> but just some thoughts on the top of my head and hopefully can be
> > helpful.
> > > >>
> > > >> And good idea on pre-rendering the template, believe template
> > rendering was
> > > >> the biggest concern in the previous discussion. We've also chose the
> > > >> pre-rendering+JSON approach in our smart sensor API
> > > >> <
> > > >>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization
> > > >> and
> > > >> seems to be working fine--a supporting case for ur proposal ;)
> > There's a
> > > >> WIP
> > > >> PR <https://github.com/apache/airflow/pull/5499> for it just in
> case
> > you
> > > >> are interested--maybe we can even share some logics.
> > > >>
> > > >> Thumbs-up again for this and please don't heisitate to reach out if
> > you
> > > >> want to discuss further with us or need any help from us.
> > > >>
> > > >>
> > > >> Cheers,
> > > >> Kevin Y
> > > >>
> > > >> On Sat, Jul 27, 2019 at 12:54 AM Driesprong, Fokko
> > <fo...@driesprong.frl>
> > > >> wrote:
> > > >>
> > > >>> Looks great Zhou,
> > > >>>
> > > >>> I have one thing that pops in my mind while reading the AIP; should
> > keep
> > > >>> the caching on the webserver level. As the famous quote goes:
> > *"There are
> > > >>> only two hard things in Computer Science: cache invalidation and
> > naming
> > > >>> things." -- Phil Karlton*
> > > >>>
> > > >>> Right now, the fundamental change that is being proposed in the AIP
> > is
> > > >>> fetching the DAGs from the database in a serialized format, and not
> > > >> parsing
> > > >>> the Python files all the time. This will give already a great
> > performance
> > > >>> improvement on the webserver side because it removes a lot of the
> > > >>> processing. However, since we're still fetching the DAGs from the
> > > >> database
> > > >>> in a regular interval, cache it in the local process, so we still
> > have
> > > >> the
> > > >>> two issues that Airflow is suffering from right now:
> > > >>>
> > > >>> 1. No snappy UI because it is still polling the database in a
> > regular
> > > >>> interval.
> > > >>> 2. Inconsistency between webservers because they might poll in
> a
> > > >>> different interval, I think we've all seen this:
> > > >>> https://www.youtube.com/watch?v=sNrBruPS3r4
> > > >>>
> > > >>> As I also mentioned in the Slack channel, I strongly feel that we
> > should
> > > >> be
> > > >>> able to render most views from the tables in the database, so
> without
> > > >>> touching the blob. For specific views, we could just pull the blob
> > from
> > > >> the
> > > >>> database. In this case we always have the latest version, and we
> > tackle
> > > >> the
> > > >>> second point above.
> > > >>>
> > > >>> To tackle the first one, I also have an idea. We should change the
> > DAG
> > > >>> parser from a loop to something that uses inotify
> > > >>> https://pypi.org/project/inotify_simple/. This will change it from
> > > >> polling
> > > >>> to an event-driven design, which is much more performant and less
> > > >> resource
> > > >>> hungry. But this would be an AIP on its own.
> > > >>>
> > > >>> Again, great design and a comprehensive AIP, but I would include
> the
> > > >>> caching on the webserver to greatly improve the user experience in
> > the
> > > >> UI.
> > > >>> Looking forward to the opinion of others on this.
> > > >>>
> > > >>> Cheers, Fokko
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> Op za 27 jul. 2019 om 01:44 schreef Zhou Fang
> > > >> <zhoufang@google.com.invalid
> > > >>>> :
> > > >>>> Hi Kaxi,
> > > >>>>
> > > >>>> Just sent out the AIP:
> > > >>>>
> > > >>>>
> > > >>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistence+in+DB+using+JSON+for+Airflow+Webserver+and+%28optional%29+Scheduler
> > > >>>> Thanks!
> > > >>>> Zhou
> > > >>>>
> > > >>>>
> > > >>>> On Fri, Jul 26, 2019 at 1:33 PM Zhou Fang <zh...@google.com>
> > wrote:
> > > >>>>
> > > >>>>> Hi Kaxil,
> > > >>>>>
> > > >>>>> We are also working on persisting DAGs into DB using JSON for
> > Airflow
> > > >>>>> webserver in Google Composer. We target at minimizing the change
> to
> > > >> the
> > > >>>>> current Airflow code. Happy to get synced on this!
> > > >>>>>
> > > >>>>> Here is our progress:
> > > >>>>> (1) Serializing DAGs using Pickle to be used in webserver
> > > >>>>> It has been launched in Composer. I am working on the PR to
> > upstream
> > > >>> it:
> > > >>>>> https://github.com/apache/airflow/pull/5594
> > > >>>>> Currently it does not support non-Airflow operators and we are
> > > >> working
> > > >>> on
> > > >>>>> a fix.
> > > >>>>>
> > > >>>>> (2) Caching Pickled DAGs in DB to be used by webserver
> > > >>>>> We have a proof-of-concept implementation, working on an AIP now.
> > > >>>>>
> > > >>>>> (3) Using JSON instead of Pickle in (1) and (2)
> > > >>>>> Decided to use JSON because Pickle is not secure and human
> > readable.
> > > >>> The
> > > >>>>> serialization approach is very similar to (1).
> > > >>>>>
> > > >>>>> I will update the RP (
> https://github.com/apache/airflow/pull/5594)
> > > >> to
> > > >>>>> replace Pickle by JSON, and send our design of (2) as an AIP next
> > > >> week.
> > > >>>>> Glad to check together whether our implementation makes sense and
> > do
> > > >>>>> improvements on that.
> > > >>>>>
> > > >>>>> Thanks!
> > > >>>>> Zhou
> > > >>>>>
> > > >>>>>
> > > >>>>> On Fri, Jul 26, 2019 at 7:37 AM Kaxil Naik <ka...@gmail.com>
> > > >>> wrote:
> > > >>>>>> Hi all,
> > > >>>>>>
> > > >>>>>> We, at Astronomer, are going to spend time working on DAG
> > > >>> Serialisation.
> > > >>>>>> There are 2 AIPs that are somewhat related to what we plan to
> work
> > > >> on:
> > > >>>>>> - AIP-18 Persist all information from DAG file in DB
> > > >>>>>> <
> > > >>>>>>
> > > >>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB
> > > >>>>>> - AIP-19 Making the webserver stateless
> > > >>>>>> <
> > > >>>>>>
> > > >>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless
> > > >>>>>> We plan to use JSON as the Serialisation format and store it as
> a
> > > >> blob
> > > >>>> in
> > > >>>>>> metadata DB.
> > > >>>>>>
> > > >>>>>> *Goals:*
> > > >>>>>>
> > > >>>>>> - Make Webserver Stateless
> > > >>>>>> - Use the same version of the DAG across Webserver &
> Scheduler
> > > >>>>>> - Keep backward compatibility and have a flag (globally & at
> > DAG
> > > >>>> level)
> > > >>>>>> to turn this feature on/off
> > > >>>>>> - Enable DAG Versioning (extended Goal)
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> We will be preparing a proposal (AIP) after some research and
> some
> > > >>>> initial
> > > >>>>>> work and open it for the suggestions of the community.
> > > >>>>>>
> > > >>>>>> We already had some good brain-storming sessions with Twitter
> > folks
> > > >>>> (DanD
> > > >>>>>> &
> > > >>>>>> Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from Uber)
> > > >> which
> > > >>>>>> will
> > > >>>>>> be a good starting point for us.
> > > >>>>>>
> > > >>>>>> If anyone in the community is interested in it or has some
> > > >> experience
> > > >>>>>> about
> > > >>>>>> the same and want to collaborate please let me know and join
> > > >>>>>> #dag-serialisation channel on Airflow Slack.
> > > >>>>>>
> > > >>>>>> Regards,
> > > >>>>>> Kaxil
> > > >>>>>>
> > >
> >
>
Re: Airflow DAG Serialisation
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Hi Jon,
I would argue that that would be the wrong approach. The processing of the
DAGs is being moved to one single place, and there they will be parsed in
parallel. This is already being done in the scheduler (look for max_threads
in the code). As soon as we have the DAG serialized properly, the work on
the webserver side should be minimal, i.e. deserializing and showing the
DAG. The dynamic generation will be done on the scheduler.
The issue with the dynamic DAGs is similar. For example, my consultancy
company implements a LOT of Airflow dags at customers and dynamic dags are
still challenging. For example, we had one Python file which would generate
around 350 DAGs with 12 tasks. Each time one of the executors would run one
of the tasks, it would inflate the whole DAG and run the specific task.
This introduced a lot of overhead. @Bas Harenslak
<ba...@gmail.com> solved
this by generating 350 DAGs from a Jinja template, which works quite well.
Keeping AIP-24 small and simple sounds like a very good idea, we've seen
scope creeps before on similar AIP's :-)
Cheers, Fokko Driesprong
Op di 30 jul. 2019 om 15:46 schreef Ash Berlin-Taylor <as...@apache.org>:
> Hi Jon,
>
> As part of this AIP(24) we aren't going to touch the scheduler any more
> than absolutely required, but yes, better support of dynamic DAGs is _very
> much_ on Kaxil and I's hit list.
>
> Our rough approach right now is to design the serialisation format well
> enough (including versioning it so we can change it over time) such that we
> can change the scheduler to not be as coupled with the dag parsing loop.
> But for the sake of small, reviewable PRs we'll do it bit-by-bit.
>
> -ash
>
> On 2019/07/30 13:33:53, Jonathan Miles <jo...@cybus.co.uk> wrote:
> > Another ask for the long-term list.
> >
> > From a superficial read of the code, it looks like this asynchronous
> > DAG loading approach could also be a stepping stone towards loading DAGs
> > in parallel? I've come across a case of someone dynamically generating a
> > DAG based on an external data source. Problem with that is when the data
> > source isn't available or is slow, it can block the loading of other
> > DAGs. Loading in parallel could isolate the failing or slow DAGs from
> > the good ones.
> >
> > I suppose even with this patch, randomising the load order of DAGs could
> > also provide some basic protection against a small set of failing DAGs.
> > At least some would get updated.
> >
> > Do the changes only affect the webserver or also loading in the
> scheduler?
> >
> > Thanks,
> >
> > Jon
> >
> > On 29/07/2019 22:18, Zhou Fang wrote:
> > > Hi Kevin,
> > >
> > > The problem that DAG parsing takes a long time can be solved by
> > > Asynchronous DAG loading: https://github.com/apache/airflow/pull/5594
> > >
> > > The idea is the a background process parses DAG files, and sends DAGs
> to
> > > webserver process every [webserver] dagbag_sync_interval = 10s.
> > >
> > > We have launched it in Composer, so our users can set webserver worker
> > > restart interval to 1 hour (or longer). The background DAG parsing
> > > processing refresh all DAGs per [webserver] = collect_dags_interval =
> 30s.
> > >
> > > If parsing all DAGs take 15min, you can see DAGs being gradually
> freshed
> > > with this feature.
> > >
> > > Thanks,
> > > Zhou
> > >
> > >
> > > On Sat, Jul 27, 2019 at 2:43 AM Kevin Yang <yr...@gmail.com> wrote:
> > >
> > >> Nice job Zhou!
> > >>
> > >> Really excited, exactly what we wanted for the webserver scaling
> issue.
> > >> Want to add another big drive for Airbnb to start think about this
> > >> previously to support the effort: it can not only bring consistency
> between
> > >> webservers but also bring consistency between webserver and
> > >> scheduler/workers. It may be less of a problem if total DAG parsing
> time is
> > >> small, but for us the total DAG parsing time is 15+ mins and we had
> to set
> > >> the webserver( gunicorn subprocesses) restart interval to 20 mins,
> which
> > >> leads to a worst case 15+20+15=50 mins delay between scheduler start
> to
> > >> schedule things and users can see their deployed DAGs/changes...
> > >>
> > >> I'm not so sure about the scheduler performance improvement:
> currently we
> > >> already feed the main scheduler process with SimpleDag through
> > >> DagFileProcessorManager running in a subprocess--in the future we
> feed it
> > >> with data from DB, which is likely slower( tho the diff should have
> > >> negligible impact to the scheduler performance). In fact if we'd keep
> the
> > >> existing behavior, try schedule only fresh parsed DAGs, then we may
> need to
> > >> deal with some consistency issue--dag processor and the scheduler
> race for
> > >> updating the flag indicating if the DAG is newly parsed. No big deal
> there
> > >> but just some thoughts on the top of my head and hopefully can be
> helpful.
> > >>
> > >> And good idea on pre-rendering the template, believe template
> rendering was
> > >> the biggest concern in the previous discussion. We've also chose the
> > >> pre-rendering+JSON approach in our smart sensor API
> > >> <
> > >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization
> > >> and
> > >> seems to be working fine--a supporting case for ur proposal ;)
> There's a
> > >> WIP
> > >> PR <https://github.com/apache/airflow/pull/5499> for it just in case
> you
> > >> are interested--maybe we can even share some logics.
> > >>
> > >> Thumbs-up again for this and please don't heisitate to reach out if
> you
> > >> want to discuss further with us or need any help from us.
> > >>
> > >>
> > >> Cheers,
> > >> Kevin Y
> > >>
> > >> On Sat, Jul 27, 2019 at 12:54 AM Driesprong, Fokko
> <fo...@driesprong.frl>
> > >> wrote:
> > >>
> > >>> Looks great Zhou,
> > >>>
> > >>> I have one thing that pops in my mind while reading the AIP; should
> keep
> > >>> the caching on the webserver level. As the famous quote goes:
> *"There are
> > >>> only two hard things in Computer Science: cache invalidation and
> naming
> > >>> things." -- Phil Karlton*
> > >>>
> > >>> Right now, the fundamental change that is being proposed in the AIP
> is
> > >>> fetching the DAGs from the database in a serialized format, and not
> > >> parsing
> > >>> the Python files all the time. This will give already a great
> performance
> > >>> improvement on the webserver side because it removes a lot of the
> > >>> processing. However, since we're still fetching the DAGs from the
> > >> database
> > >>> in a regular interval, cache it in the local process, so we still
> have
> > >> the
> > >>> two issues that Airflow is suffering from right now:
> > >>>
> > >>> 1. No snappy UI because it is still polling the database in a
> regular
> > >>> interval.
> > >>> 2. Inconsistency between webservers because they might poll in a
> > >>> different interval, I think we've all seen this:
> > >>> https://www.youtube.com/watch?v=sNrBruPS3r4
> > >>>
> > >>> As I also mentioned in the Slack channel, I strongly feel that we
> should
> > >> be
> > >>> able to render most views from the tables in the database, so without
> > >>> touching the blob. For specific views, we could just pull the blob
> from
> > >> the
> > >>> database. In this case we always have the latest version, and we
> tackle
> > >> the
> > >>> second point above.
> > >>>
> > >>> To tackle the first one, I also have an idea. We should change the
> DAG
> > >>> parser from a loop to something that uses inotify
> > >>> https://pypi.org/project/inotify_simple/. This will change it from
> > >> polling
> > >>> to an event-driven design, which is much more performant and less
> > >> resource
> > >>> hungry. But this would be an AIP on its own.
> > >>>
> > >>> Again, great design and a comprehensive AIP, but I would include the
> > >>> caching on the webserver to greatly improve the user experience in
> the
> > >> UI.
> > >>> Looking forward to the opinion of others on this.
> > >>>
> > >>> Cheers, Fokko
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> Op za 27 jul. 2019 om 01:44 schreef Zhou Fang
> > >> <zhoufang@google.com.invalid
> > >>>> :
> > >>>> Hi Kaxi,
> > >>>>
> > >>>> Just sent out the AIP:
> > >>>>
> > >>>>
> > >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistence+in+DB+using+JSON+for+Airflow+Webserver+and+%28optional%29+Scheduler
> > >>>> Thanks!
> > >>>> Zhou
> > >>>>
> > >>>>
> > >>>> On Fri, Jul 26, 2019 at 1:33 PM Zhou Fang <zh...@google.com>
> wrote:
> > >>>>
> > >>>>> Hi Kaxil,
> > >>>>>
> > >>>>> We are also working on persisting DAGs into DB using JSON for
> Airflow
> > >>>>> webserver in Google Composer. We target at minimizing the change to
> > >> the
> > >>>>> current Airflow code. Happy to get synced on this!
> > >>>>>
> > >>>>> Here is our progress:
> > >>>>> (1) Serializing DAGs using Pickle to be used in webserver
> > >>>>> It has been launched in Composer. I am working on the PR to
> upstream
> > >>> it:
> > >>>>> https://github.com/apache/airflow/pull/5594
> > >>>>> Currently it does not support non-Airflow operators and we are
> > >> working
> > >>> on
> > >>>>> a fix.
> > >>>>>
> > >>>>> (2) Caching Pickled DAGs in DB to be used by webserver
> > >>>>> We have a proof-of-concept implementation, working on an AIP now.
> > >>>>>
> > >>>>> (3) Using JSON instead of Pickle in (1) and (2)
> > >>>>> Decided to use JSON because Pickle is not secure and human
> readable.
> > >>> The
> > >>>>> serialization approach is very similar to (1).
> > >>>>>
> > >>>>> I will update the RP (https://github.com/apache/airflow/pull/5594)
> > >> to
> > >>>>> replace Pickle by JSON, and send our design of (2) as an AIP next
> > >> week.
> > >>>>> Glad to check together whether our implementation makes sense and
> do
> > >>>>> improvements on that.
> > >>>>>
> > >>>>> Thanks!
> > >>>>> Zhou
> > >>>>>
> > >>>>>
> > >>>>> On Fri, Jul 26, 2019 at 7:37 AM Kaxil Naik <ka...@gmail.com>
> > >>> wrote:
> > >>>>>> Hi all,
> > >>>>>>
> > >>>>>> We, at Astronomer, are going to spend time working on DAG
> > >>> Serialisation.
> > >>>>>> There are 2 AIPs that are somewhat related to what we plan to work
> > >> on:
> > >>>>>> - AIP-18 Persist all information from DAG file in DB
> > >>>>>> <
> > >>>>>>
> > >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB
> > >>>>>> - AIP-19 Making the webserver stateless
> > >>>>>> <
> > >>>>>>
> > >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless
> > >>>>>> We plan to use JSON as the Serialisation format and store it as a
> > >> blob
> > >>>> in
> > >>>>>> metadata DB.
> > >>>>>>
> > >>>>>> *Goals:*
> > >>>>>>
> > >>>>>> - Make Webserver Stateless
> > >>>>>> - Use the same version of the DAG across Webserver & Scheduler
> > >>>>>> - Keep backward compatibility and have a flag (globally & at
> DAG
> > >>>> level)
> > >>>>>> to turn this feature on/off
> > >>>>>> - Enable DAG Versioning (extended Goal)
> > >>>>>>
> > >>>>>>
> > >>>>>> We will be preparing a proposal (AIP) after some research and some
> > >>>> initial
> > >>>>>> work and open it for the suggestions of the community.
> > >>>>>>
> > >>>>>> We already had some good brain-storming sessions with Twitter
> folks
> > >>>> (DanD
> > >>>>>> &
> > >>>>>> Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from Uber)
> > >> which
> > >>>>>> will
> > >>>>>> be a good starting point for us.
> > >>>>>>
> > >>>>>> If anyone in the community is interested in it or has some
> > >> experience
> > >>>>>> about
> > >>>>>> the same and want to collaborate please let me know and join
> > >>>>>> #dag-serialisation channel on Airflow Slack.
> > >>>>>>
> > >>>>>> Regards,
> > >>>>>> Kaxil
> > >>>>>>
> >
>
Re: Airflow DAG Serialisation
Posted by Ash Berlin-Taylor <as...@apache.org>.
Hi Jon,
As part of this AIP(24) we aren't going to touch the scheduler any more than absolutely required, but yes, better support of dynamic DAGs is _very much_ on Kaxil and I's hit list.
Our rough approach right now is to design the serialisation format well enough (including versioning it so we can change it over time) such that we can change the scheduler to not be as coupled with the dag parsing loop. But for the sake of small, reviewable PRs we'll do it bit-by-bit.
-ash
On 2019/07/30 13:33:53, Jonathan Miles <jo...@cybus.co.uk> wrote:
> Another ask for the long-term list.
>
> From a superficial read of the code, it looks like this asynchronous
> DAG loading approach could also be a stepping stone towards loading DAGs
> in parallel? I've come across a case of someone dynamically generating a
> DAG based on an external data source. Problem with that is when the data
> source isn't available or is slow, it can block the loading of other
> DAGs. Loading in parallel could isolate the failing or slow DAGs from
> the good ones.
>
> I suppose even with this patch, randomising the load order of DAGs could
> also provide some basic protection against a small set of failing DAGs.
> At least some would get updated.
>
> Do the changes only affect the webserver or also loading in the scheduler?
>
> Thanks,
>
> Jon
>
> On 29/07/2019 22:18, Zhou Fang wrote:
> > Hi Kevin,
> >
> > The problem that DAG parsing takes a long time can be solved by
> > Asynchronous DAG loading: https://github.com/apache/airflow/pull/5594
> >
> > The idea is the a background process parses DAG files, and sends DAGs to
> > webserver process every [webserver] dagbag_sync_interval = 10s.
> >
> > We have launched it in Composer, so our users can set webserver worker
> > restart interval to 1 hour (or longer). The background DAG parsing
> > processing refresh all DAGs per [webserver] = collect_dags_interval = 30s.
> >
> > If parsing all DAGs take 15min, you can see DAGs being gradually freshed
> > with this feature.
> >
> > Thanks,
> > Zhou
> >
> >
> > On Sat, Jul 27, 2019 at 2:43 AM Kevin Yang <yr...@gmail.com> wrote:
> >
> >> Nice job Zhou!
> >>
> >> Really excited, exactly what we wanted for the webserver scaling issue.
> >> Want to add another big drive for Airbnb to start think about this
> >> previously to support the effort: it can not only bring consistency between
> >> webservers but also bring consistency between webserver and
> >> scheduler/workers. It may be less of a problem if total DAG parsing time is
> >> small, but for us the total DAG parsing time is 15+ mins and we had to set
> >> the webserver( gunicorn subprocesses) restart interval to 20 mins, which
> >> leads to a worst case 15+20+15=50 mins delay between scheduler start to
> >> schedule things and users can see their deployed DAGs/changes...
> >>
> >> I'm not so sure about the scheduler performance improvement: currently we
> >> already feed the main scheduler process with SimpleDag through
> >> DagFileProcessorManager running in a subprocess--in the future we feed it
> >> with data from DB, which is likely slower( tho the diff should have
> >> negligible impact to the scheduler performance). In fact if we'd keep the
> >> existing behavior, try schedule only fresh parsed DAGs, then we may need to
> >> deal with some consistency issue--dag processor and the scheduler race for
> >> updating the flag indicating if the DAG is newly parsed. No big deal there
> >> but just some thoughts on the top of my head and hopefully can be helpful.
> >>
> >> And good idea on pre-rendering the template, believe template rendering was
> >> the biggest concern in the previous discussion. We've also chose the
> >> pre-rendering+JSON approach in our smart sensor API
> >> <
> >> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization
> >> and
> >> seems to be working fine--a supporting case for ur proposal ;) There's a
> >> WIP
> >> PR <https://github.com/apache/airflow/pull/5499> for it just in case you
> >> are interested--maybe we can even share some logics.
> >>
> >> Thumbs-up again for this and please don't heisitate to reach out if you
> >> want to discuss further with us or need any help from us.
> >>
> >>
> >> Cheers,
> >> Kevin Y
> >>
> >> On Sat, Jul 27, 2019 at 12:54 AM Driesprong, Fokko <fo...@driesprong.frl>
> >> wrote:
> >>
> >>> Looks great Zhou,
> >>>
> >>> I have one thing that pops in my mind while reading the AIP; should keep
> >>> the caching on the webserver level. As the famous quote goes: *"There are
> >>> only two hard things in Computer Science: cache invalidation and naming
> >>> things." -- Phil Karlton*
> >>>
> >>> Right now, the fundamental change that is being proposed in the AIP is
> >>> fetching the DAGs from the database in a serialized format, and not
> >> parsing
> >>> the Python files all the time. This will give already a great performance
> >>> improvement on the webserver side because it removes a lot of the
> >>> processing. However, since we're still fetching the DAGs from the
> >> database
> >>> in a regular interval, cache it in the local process, so we still have
> >> the
> >>> two issues that Airflow is suffering from right now:
> >>>
> >>> 1. No snappy UI because it is still polling the database in a regular
> >>> interval.
> >>> 2. Inconsistency between webservers because they might poll in a
> >>> different interval, I think we've all seen this:
> >>> https://www.youtube.com/watch?v=sNrBruPS3r4
> >>>
> >>> As I also mentioned in the Slack channel, I strongly feel that we should
> >> be
> >>> able to render most views from the tables in the database, so without
> >>> touching the blob. For specific views, we could just pull the blob from
> >> the
> >>> database. In this case we always have the latest version, and we tackle
> >> the
> >>> second point above.
> >>>
> >>> To tackle the first one, I also have an idea. We should change the DAG
> >>> parser from a loop to something that uses inotify
> >>> https://pypi.org/project/inotify_simple/. This will change it from
> >> polling
> >>> to an event-driven design, which is much more performant and less
> >> resource
> >>> hungry. But this would be an AIP on its own.
> >>>
> >>> Again, great design and a comprehensive AIP, but I would include the
> >>> caching on the webserver to greatly improve the user experience in the
> >> UI.
> >>> Looking forward to the opinion of others on this.
> >>>
> >>> Cheers, Fokko
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> Op za 27 jul. 2019 om 01:44 schreef Zhou Fang
> >> <zhoufang@google.com.invalid
> >>>> :
> >>>> Hi Kaxi,
> >>>>
> >>>> Just sent out the AIP:
> >>>>
> >>>>
> >> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistence+in+DB+using+JSON+for+Airflow+Webserver+and+%28optional%29+Scheduler
> >>>> Thanks!
> >>>> Zhou
> >>>>
> >>>>
> >>>> On Fri, Jul 26, 2019 at 1:33 PM Zhou Fang <zh...@google.com> wrote:
> >>>>
> >>>>> Hi Kaxil,
> >>>>>
> >>>>> We are also working on persisting DAGs into DB using JSON for Airflow
> >>>>> webserver in Google Composer. We target at minimizing the change to
> >> the
> >>>>> current Airflow code. Happy to get synced on this!
> >>>>>
> >>>>> Here is our progress:
> >>>>> (1) Serializing DAGs using Pickle to be used in webserver
> >>>>> It has been launched in Composer. I am working on the PR to upstream
> >>> it:
> >>>>> https://github.com/apache/airflow/pull/5594
> >>>>> Currently it does not support non-Airflow operators and we are
> >> working
> >>> on
> >>>>> a fix.
> >>>>>
> >>>>> (2) Caching Pickled DAGs in DB to be used by webserver
> >>>>> We have a proof-of-concept implementation, working on an AIP now.
> >>>>>
> >>>>> (3) Using JSON instead of Pickle in (1) and (2)
> >>>>> Decided to use JSON because Pickle is not secure and human readable.
> >>> The
> >>>>> serialization approach is very similar to (1).
> >>>>>
> >>>>> I will update the RP (https://github.com/apache/airflow/pull/5594)
> >> to
> >>>>> replace Pickle by JSON, and send our design of (2) as an AIP next
> >> week.
> >>>>> Glad to check together whether our implementation makes sense and do
> >>>>> improvements on that.
> >>>>>
> >>>>> Thanks!
> >>>>> Zhou
> >>>>>
> >>>>>
> >>>>> On Fri, Jul 26, 2019 at 7:37 AM Kaxil Naik <ka...@gmail.com>
> >>> wrote:
> >>>>>> Hi all,
> >>>>>>
> >>>>>> We, at Astronomer, are going to spend time working on DAG
> >>> Serialisation.
> >>>>>> There are 2 AIPs that are somewhat related to what we plan to work
> >> on:
> >>>>>> - AIP-18 Persist all information from DAG file in DB
> >>>>>> <
> >>>>>>
> >> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB
> >>>>>> - AIP-19 Making the webserver stateless
> >>>>>> <
> >>>>>>
> >> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless
> >>>>>> We plan to use JSON as the Serialisation format and store it as a
> >> blob
> >>>> in
> >>>>>> metadata DB.
> >>>>>>
> >>>>>> *Goals:*
> >>>>>>
> >>>>>> - Make Webserver Stateless
> >>>>>> - Use the same version of the DAG across Webserver & Scheduler
> >>>>>> - Keep backward compatibility and have a flag (globally & at DAG
> >>>> level)
> >>>>>> to turn this feature on/off
> >>>>>> - Enable DAG Versioning (extended Goal)
> >>>>>>
> >>>>>>
> >>>>>> We will be preparing a proposal (AIP) after some research and some
> >>>> initial
> >>>>>> work and open it for the suggestions of the community.
> >>>>>>
> >>>>>> We already had some good brain-storming sessions with Twitter folks
> >>>> (DanD
> >>>>>> &
> >>>>>> Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from Uber)
> >> which
> >>>>>> will
> >>>>>> be a good starting point for us.
> >>>>>>
> >>>>>> If anyone in the community is interested in it or has some
> >> experience
> >>>>>> about
> >>>>>> the same and want to collaborate please let me know and join
> >>>>>> #dag-serialisation channel on Airflow Slack.
> >>>>>>
> >>>>>> Regards,
> >>>>>> Kaxil
> >>>>>>
>
Re: Airflow DAG Serialisation
Posted by Jonathan Miles <jo...@cybus.co.uk>.
Another ask for the long-term list.
From a superficial read of the code, it looks like this asynchronous
DAG loading approach could also be a stepping stone towards loading DAGs
in parallel? I've come across a case of someone dynamically generating a
DAG based on an external data source. Problem with that is when the data
source isn't available or is slow, it can block the loading of other
DAGs. Loading in parallel could isolate the failing or slow DAGs from
the good ones.
I suppose even with this patch, randomising the load order of DAGs could
also provide some basic protection against a small set of failing DAGs.
At least some would get updated.
Do the changes only affect the webserver or also loading in the scheduler?
Thanks,
Jon
On 29/07/2019 22:18, Zhou Fang wrote:
> Hi Kevin,
>
> The problem that DAG parsing takes a long time can be solved by
> Asynchronous DAG loading: https://github.com/apache/airflow/pull/5594
>
> The idea is the a background process parses DAG files, and sends DAGs to
> webserver process every [webserver] dagbag_sync_interval = 10s.
>
> We have launched it in Composer, so our users can set webserver worker
> restart interval to 1 hour (or longer). The background DAG parsing
> processing refresh all DAGs per [webserver] = collect_dags_interval = 30s.
>
> If parsing all DAGs take 15min, you can see DAGs being gradually freshed
> with this feature.
>
> Thanks,
> Zhou
>
>
> On Sat, Jul 27, 2019 at 2:43 AM Kevin Yang <yr...@gmail.com> wrote:
>
>> Nice job Zhou!
>>
>> Really excited, exactly what we wanted for the webserver scaling issue.
>> Want to add another big drive for Airbnb to start think about this
>> previously to support the effort: it can not only bring consistency between
>> webservers but also bring consistency between webserver and
>> scheduler/workers. It may be less of a problem if total DAG parsing time is
>> small, but for us the total DAG parsing time is 15+ mins and we had to set
>> the webserver( gunicorn subprocesses) restart interval to 20 mins, which
>> leads to a worst case 15+20+15=50 mins delay between scheduler start to
>> schedule things and users can see their deployed DAGs/changes...
>>
>> I'm not so sure about the scheduler performance improvement: currently we
>> already feed the main scheduler process with SimpleDag through
>> DagFileProcessorManager running in a subprocess--in the future we feed it
>> with data from DB, which is likely slower( tho the diff should have
>> negligible impact to the scheduler performance). In fact if we'd keep the
>> existing behavior, try schedule only fresh parsed DAGs, then we may need to
>> deal with some consistency issue--dag processor and the scheduler race for
>> updating the flag indicating if the DAG is newly parsed. No big deal there
>> but just some thoughts on the top of my head and hopefully can be helpful.
>>
>> And good idea on pre-rendering the template, believe template rendering was
>> the biggest concern in the previous discussion. We've also chose the
>> pre-rendering+JSON approach in our smart sensor API
>> <
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization
>> and
>> seems to be working fine--a supporting case for ur proposal ;) There's a
>> WIP
>> PR <https://github.com/apache/airflow/pull/5499> for it just in case you
>> are interested--maybe we can even share some logics.
>>
>> Thumbs-up again for this and please don't heisitate to reach out if you
>> want to discuss further with us or need any help from us.
>>
>>
>> Cheers,
>> Kevin Y
>>
>> On Sat, Jul 27, 2019 at 12:54 AM Driesprong, Fokko <fo...@driesprong.frl>
>> wrote:
>>
>>> Looks great Zhou,
>>>
>>> I have one thing that pops in my mind while reading the AIP; should keep
>>> the caching on the webserver level. As the famous quote goes: *"There are
>>> only two hard things in Computer Science: cache invalidation and naming
>>> things." -- Phil Karlton*
>>>
>>> Right now, the fundamental change that is being proposed in the AIP is
>>> fetching the DAGs from the database in a serialized format, and not
>> parsing
>>> the Python files all the time. This will give already a great performance
>>> improvement on the webserver side because it removes a lot of the
>>> processing. However, since we're still fetching the DAGs from the
>> database
>>> in a regular interval, cache it in the local process, so we still have
>> the
>>> two issues that Airflow is suffering from right now:
>>>
>>> 1. No snappy UI because it is still polling the database in a regular
>>> interval.
>>> 2. Inconsistency between webservers because they might poll in a
>>> different interval, I think we've all seen this:
>>> https://www.youtube.com/watch?v=sNrBruPS3r4
>>>
>>> As I also mentioned in the Slack channel, I strongly feel that we should
>> be
>>> able to render most views from the tables in the database, so without
>>> touching the blob. For specific views, we could just pull the blob from
>> the
>>> database. In this case we always have the latest version, and we tackle
>> the
>>> second point above.
>>>
>>> To tackle the first one, I also have an idea. We should change the DAG
>>> parser from a loop to something that uses inotify
>>> https://pypi.org/project/inotify_simple/. This will change it from
>> polling
>>> to an event-driven design, which is much more performant and less
>> resource
>>> hungry. But this would be an AIP on its own.
>>>
>>> Again, great design and a comprehensive AIP, but I would include the
>>> caching on the webserver to greatly improve the user experience in the
>> UI.
>>> Looking forward to the opinion of others on this.
>>>
>>> Cheers, Fokko
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Op za 27 jul. 2019 om 01:44 schreef Zhou Fang
>> <zhoufang@google.com.invalid
>>>> :
>>>> Hi Kaxi,
>>>>
>>>> Just sent out the AIP:
>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistence+in+DB+using+JSON+for+Airflow+Webserver+and+%28optional%29+Scheduler
>>>> Thanks!
>>>> Zhou
>>>>
>>>>
>>>> On Fri, Jul 26, 2019 at 1:33 PM Zhou Fang <zh...@google.com> wrote:
>>>>
>>>>> Hi Kaxil,
>>>>>
>>>>> We are also working on persisting DAGs into DB using JSON for Airflow
>>>>> webserver in Google Composer. We target at minimizing the change to
>> the
>>>>> current Airflow code. Happy to get synced on this!
>>>>>
>>>>> Here is our progress:
>>>>> (1) Serializing DAGs using Pickle to be used in webserver
>>>>> It has been launched in Composer. I am working on the PR to upstream
>>> it:
>>>>> https://github.com/apache/airflow/pull/5594
>>>>> Currently it does not support non-Airflow operators and we are
>> working
>>> on
>>>>> a fix.
>>>>>
>>>>> (2) Caching Pickled DAGs in DB to be used by webserver
>>>>> We have a proof-of-concept implementation, working on an AIP now.
>>>>>
>>>>> (3) Using JSON instead of Pickle in (1) and (2)
>>>>> Decided to use JSON because Pickle is not secure and human readable.
>>> The
>>>>> serialization approach is very similar to (1).
>>>>>
>>>>> I will update the RP (https://github.com/apache/airflow/pull/5594)
>> to
>>>>> replace Pickle by JSON, and send our design of (2) as an AIP next
>> week.
>>>>> Glad to check together whether our implementation makes sense and do
>>>>> improvements on that.
>>>>>
>>>>> Thanks!
>>>>> Zhou
>>>>>
>>>>>
>>>>> On Fri, Jul 26, 2019 at 7:37 AM Kaxil Naik <ka...@gmail.com>
>>> wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> We, at Astronomer, are going to spend time working on DAG
>>> Serialisation.
>>>>>> There are 2 AIPs that are somewhat related to what we plan to work
>> on:
>>>>>> - AIP-18 Persist all information from DAG file in DB
>>>>>> <
>>>>>>
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB
>>>>>> - AIP-19 Making the webserver stateless
>>>>>> <
>>>>>>
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless
>>>>>> We plan to use JSON as the Serialisation format and store it as a
>> blob
>>>> in
>>>>>> metadata DB.
>>>>>>
>>>>>> *Goals:*
>>>>>>
>>>>>> - Make Webserver Stateless
>>>>>> - Use the same version of the DAG across Webserver & Scheduler
>>>>>> - Keep backward compatibility and have a flag (globally & at DAG
>>>> level)
>>>>>> to turn this feature on/off
>>>>>> - Enable DAG Versioning (extended Goal)
>>>>>>
>>>>>>
>>>>>> We will be preparing a proposal (AIP) after some research and some
>>>> initial
>>>>>> work and open it for the suggestions of the community.
>>>>>>
>>>>>> We already had some good brain-storming sessions with Twitter folks
>>>> (DanD
>>>>>> &
>>>>>> Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from Uber)
>> which
>>>>>> will
>>>>>> be a good starting point for us.
>>>>>>
>>>>>> If anyone in the community is interested in it or has some
>> experience
>>>>>> about
>>>>>> the same and want to collaborate please let me know and join
>>>>>> #dag-serialisation channel on Airflow Slack.
>>>>>>
>>>>>> Regards,
>>>>>> Kaxil
>>>>>>
Re: Airflow DAG Serialisation
Posted by Ash Berlin-Taylor <as...@apache.org>.
The one added complexity in back-porting this to 1.10.x is that we have two webservers (classic and RBAC) so either we only add this feature to the RBAC path for a 1.10.5 (which I am okay with) or someone other than me ports any changes to the classic UI once it's merged to master ;)
-ash
> On 30 Jul 2019, at 09:23, Jarek Potiuk <Ja...@polidea.com> wrote:
>
> I think Zhou's change is pretty much backwards-compatible with 1.10.x -
> it's basically optimisation that people might find really useful until 2.0
> is out. I believe (correct me if I am wrong) it does not require any change
> from the user's perspective. Airflow will continue to behave the same way
> as 1.10.4, - only making it more resilient in case of huge DAG folders. It
> also seems battle-tested already and I think it's a small-ish effort to
> continue reviewing it and add the few changes that are being discussed in
> the PR.
>
> I think, maybe then a viable approach is to ONLY implement it in v1-10-test
> branch and make a 1.10.5 release much faster than by the end of year. If we
> can get the 1.10.5 with only that change (and maybe few other bugk fixes )
> in a month or two from now - that would be beneficial for the users - they
> will get much better performance faster, and they could have more time
> preparing for 2.0.0 backwards-incompatible migration later on. I guess it
> will take additional few months for anyone to migrate to 2.0.0 to convert
> their DAGs and do additional testing.
>
> There is a small risk (and bigger diversion between 1.10.x and 2.0.0) but I
> think the risk is very small here - especially that I am sure Zhou and
> Composer team will find and fix (and back-port) any issues found in
> Composer in case they find them (and this is one of the best testing
> grounds for Airflow ;)) .
>
> I volunteer to continue merging simple changes to 1.10. branch if we go
> that direction (and help with 1.10.5 release when the time comes). I kind
> of got used to it already and it's a bit of a habit to try to cherry-pick
> those changes back to the v1-10-test branch.
>
> J.
>
> On Tue, Jul 30, 2019 at 8:28 AM Kaxil Naik <ka...@gmail.com> wrote:
>
>> Thanks Kevin for the input.
>>
>> I am working on this full-time as well with help from Ash.
>>
>> Persistence to DB is what we want to achieve while making Webserver
>> stateless and hence mitigating the issues you described.
>>
>> My opinion is that we should work towards that goal and aim to include it
>> in Airflow 2.0 . However I am not completely against a short-term solution
>> that will be completely optional, that might increase the work though.
>>
>> The diff between master and the current release branch is huge. So we
>> should aim to cut 2.0 by the end of this year with Dag Serialisation &
>> peristence in DB feature.
>>
>> We can then have minor releases directly from Master and follow SemVer as
>> well.
>>
>> Regards,
>> Kaxil
>>
>>
>> On Tue, Jul 30, 2019, 09:02 Kevin Yang <yr...@gmail.com> wrote:
>>
>>> For sure! I'll try my best to squeeze some time for it during the weekend
>>> and see how I can help facilitate the effort( don't get the hopes up too
>>> much tho, got ~2 million things piled in my TODO list :P ). Will bring
>> this
>>> up to the team and see if my team can help too.
>>>
>>> You guys have showed pretty solid understanding and skills so I believe
>> you
>>> can handle it well w/o me but just in case you need me, don't hesitate to
>>> shoot me a direct mail.
>>>
>>> Cheers,
>>> Kevin Y
>>>
>>> On Mon, Jul 29, 2019 at 8:19 PM Zhou Fang <zh...@google.com> wrote:
>>>
>>>> Hi Kevin, it makes sense. Thanks for the explanation! Hope we can get
>> DB
>>>> persistence move faster.
>>>>
>>>> Zhou
>>>>
>>>> On Mon, Jul 29, 2019, 5:46 PM Kevin Yang <yr...@gmail.com> wrote:
>>>>
>>>>> oops, s/consistent file/consistent file order/
>>>>>
>>>>> On Mon, Jul 29, 2019 at 5:42 PM Kevin Yang <yr...@gmail.com> wrote:
>>>>>
>>>>>> Hi Zhou,
>>>>>>
>>>>>> Totally understood, thank you for that. Streaming logic does cover
>> most
>>>>>> cases, tho we still have the worst cases where os.walk doesn't give
>> us
>>>>>> consistent file and file/dir additions/renames causing a different
>>> result
>>>>>> order of list_py_file_paths( e.g. right after we parsed the first
>> dir,
>>> it
>>>>>> was renamed and will be parsed last in the 2nd DAG loading, or we
>>> merged a
>>>>>> new file right after the file paths are collected). Maybe there's a
>>> way to
>>>>>> guarantee the order of parsing but not sure if it worth the effort
>>> given
>>>>>> that it is less of a problem if the end to end parsing time is small
>>>>>> enough. I understand it may be started as a short term improvement
>> but
>>>>>> since it should not be too much more complicated, we'd rather start
>>> with
>>>>>> unified long term pattern.
>>>>>>
>>>>>> Cheers,
>>>>>> Kevin Y
>>>>>>
>>>>>>
>>>>>> On Mon, Jul 29, 2019 at 3:59 PM Zhou Fang <zh...@google.com>
>> wrote:
>>>>>>
>>>>>>> Hi Kevin,
>>>>>>>
>>>>>>> Yes. DAG persistence in DB is definitely the way to go. I referred
>> to
>>>>>>> the aysnc dag loader because it may alleviate your current problem
>>> (since
>>>>>>> it is code ready).
>>>>>>>
>>>>>>> It actually reduces the time to 15min, because DAGs are refreshed by
>>>>>>> the background process in a streaming way and you don't need to
>>> restart
>>>>>>> webserver per 20min.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Zhou
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Jul 29, 2019 at 3:14 PM Kevin Yang <yr...@gmail.com>
>> wrote:
>>>>>>>
>>>>>>>> Hi Zhou,
>>>>>>>>
>>>>>>>> Thank you for the pointer. This solves the issue gunicorn restart
>>> rate
>>>>>>>> throttles webserver refresh rate but not the long DAG parsing time
>>> issue,
>>>>>>>> right? Worst case scenario we still wait 30 mins for the change to
>>> show up,
>>>>>>>> comparing to the previous 35 mins( I was wrong on the number, it
>>> should be
>>>>>>>> 35 mins instead of 55 mins as the clock starts whenever the
>> webserver
>>>>>>>> restarts). I believe in the previous discussion, we firstly
>> proposed
>>> this
>>>>>>>> local webserver DAG parsing optimization to use the same DAG
>> parsing
>>> logic
>>>>>>>> in scheduler to speed up the parsing. Then the stateless webserver
>>> proposal
>>>>>>>> came up and we were brought in that it is a better idea to persist
>>> DAGs
>>>>>>>> into the DB and read directly from the DB, better DAG def
>>> consistency and
>>>>>>>> webserver cluster consistency. I'm all supportive on the proposed
>>> structure
>>>>>>>> in AIP-24 but -1 on just feed webserver with a single subprocess
>>> parsing
>>>>>>>> the DAGs. I would image there won't be too many additional work to
>>> fetch
>>>>>>>> from DB instead of a subprocess, would there?( haven't look into
>> the
>>>>>>>> serialization format part but assuming they are the same/similar)
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Kevin Y
>>>>>>>>
>>>>>>>> On Mon, Jul 29, 2019 at 2:18 PM Zhou Fang <zh...@google.com>
>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Kevin,
>>>>>>>>>
>>>>>>>>> The problem that DAG parsing takes a long time can be solved by
>>>>>>>>> Asynchronous DAG loading:
>>> https://github.com/apache/airflow/pull/5594
>>>>>>>>>
>>>>>>>>> The idea is the a background process parses DAG files, and sends
>>> DAGs
>>>>>>>>> to webserver process every [webserver] dagbag_sync_interval = 10s.
>>>>>>>>>
>>>>>>>>> We have launched it in Composer, so our users can set
>>>>>>>>> webserver worker restart interval to 1 hour (or longer). The
>>> background DAG
>>>>>>>>> parsing processing refresh all DAGs per [webserver] =
>>> collect_dags_interval
>>>>>>>>> = 30s.
>>>>>>>>>
>>>>>>>>> If parsing all DAGs take 15min, you can see DAGs being gradually
>>>>>>>>> freshed with this feature.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Zhou
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sat, Jul 27, 2019 at 2:43 AM Kevin Yang <yr...@gmail.com>
>>> wrote:
>>>>>>>>>
>>>>>>>>>> Nice job Zhou!
>>>>>>>>>>
>>>>>>>>>> Really excited, exactly what we wanted for the webserver scaling
>>>>>>>>>> issue.
>>>>>>>>>> Want to add another big drive for Airbnb to start think about
>> this
>>>>>>>>>> previously to support the effort: it can not only bring
>> consistency
>>>>>>>>>> between
>>>>>>>>>> webservers but also bring consistency between webserver and
>>>>>>>>>> scheduler/workers. It may be less of a problem if total DAG
>> parsing
>>>>>>>>>> time is
>>>>>>>>>> small, but for us the total DAG parsing time is 15+ mins and we
>> had
>>>>>>>>>> to set
>>>>>>>>>> the webserver( gunicorn subprocesses) restart interval to 20
>> mins,
>>>>>>>>>> which
>>>>>>>>>> leads to a worst case 15+20+15=50 mins delay between scheduler
>>> start
>>>>>>>>>> to
>>>>>>>>>> schedule things and users can see their deployed DAGs/changes...
>>>>>>>>>>
>>>>>>>>>> I'm not so sure about the scheduler performance improvement:
>>>>>>>>>> currently we
>>>>>>>>>> already feed the main scheduler process with SimpleDag through
>>>>>>>>>> DagFileProcessorManager running in a subprocess--in the future we
>>>>>>>>>> feed it
>>>>>>>>>> with data from DB, which is likely slower( tho the diff should
>> have
>>>>>>>>>> negligible impact to the scheduler performance). In fact if we'd
>>>>>>>>>> keep the
>>>>>>>>>> existing behavior, try schedule only fresh parsed DAGs, then we
>> may
>>>>>>>>>> need to
>>>>>>>>>> deal with some consistency issue--dag processor and the scheduler
>>>>>>>>>> race for
>>>>>>>>>> updating the flag indicating if the DAG is newly parsed. No big
>>> deal
>>>>>>>>>> there
>>>>>>>>>> but just some thoughts on the top of my head and hopefully can be
>>>>>>>>>> helpful.
>>>>>>>>>>
>>>>>>>>>> And good idea on pre-rendering the template, believe template
>>>>>>>>>> rendering was
>>>>>>>>>> the biggest concern in the previous discussion. We've also chose
>>> the
>>>>>>>>>> pre-rendering+JSON approach in our smart sensor API
>>>>>>>>>> <
>>>>>>>>>>
>>>
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization
>>>>>>>>>>>
>>>>>>>>>> and
>>>>>>>>>> seems to be working fine--a supporting case for ur proposal ;)
>>>>>>>>>> There's a WIP
>>>>>>>>>> PR <https://github.com/apache/airflow/pull/5499> for it just in
>>>>>>>>>> case you
>>>>>>>>>> are interested--maybe we can even share some logics.
>>>>>>>>>>
>>>>>>>>>> Thumbs-up again for this and please don't heisitate to reach out
>> if
>>>>>>>>>> you
>>>>>>>>>> want to discuss further with us or need any help from us.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> Kevin Y
>>>>>>>>>>
>>>>>>>>>> On Sat, Jul 27, 2019 at 12:54 AM Driesprong, Fokko
>>>>>>>>>> <fo...@driesprong.frl>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Looks great Zhou,
>>>>>>>>>>>
>>>>>>>>>>> I have one thing that pops in my mind while reading the AIP;
>>>>>>>>>> should keep
>>>>>>>>>>> the caching on the webserver level. As the famous quote goes:
>>>>>>>>>> *"There are
>>>>>>>>>>> only two hard things in Computer Science: cache invalidation
>> and
>>>>>>>>>> naming
>>>>>>>>>>> things." -- Phil Karlton*
>>>>>>>>>>>
>>>>>>>>>>> Right now, the fundamental change that is being proposed in the
>>>>>>>>>> AIP is
>>>>>>>>>>> fetching the DAGs from the database in a serialized format, and
>>>>>>>>>> not parsing
>>>>>>>>>>> the Python files all the time. This will give already a great
>>>>>>>>>> performance
>>>>>>>>>>> improvement on the webserver side because it removes a lot of
>> the
>>>>>>>>>>> processing. However, since we're still fetching the DAGs from
>> the
>>>>>>>>>> database
>>>>>>>>>>> in a regular interval, cache it in the local process, so we
>> still
>>>>>>>>>> have the
>>>>>>>>>>> two issues that Airflow is suffering from right now:
>>>>>>>>>>>
>>>>>>>>>>> 1. No snappy UI because it is still polling the database in
>> a
>>>>>>>>>> regular
>>>>>>>>>>> interval.
>>>>>>>>>>> 2. Inconsistency between webservers because they might poll
>>> in a
>>>>>>>>>>> different interval, I think we've all seen this:
>>>>>>>>>>> https://www.youtube.com/watch?v=sNrBruPS3r4
>>>>>>>>>>>
>>>>>>>>>>> As I also mentioned in the Slack channel, I strongly feel that
>> we
>>>>>>>>>> should be
>>>>>>>>>>> able to render most views from the tables in the database, so
>>>>>>>>>> without
>>>>>>>>>>> touching the blob. For specific views, we could just pull the
>>> blob
>>>>>>>>>> from the
>>>>>>>>>>> database. In this case we always have the latest version, and
>> we
>>>>>>>>>> tackle the
>>>>>>>>>>> second point above.
>>>>>>>>>>>
>>>>>>>>>>> To tackle the first one, I also have an idea. We should change
>>> the
>>>>>>>>>> DAG
>>>>>>>>>>> parser from a loop to something that uses inotify
>>>>>>>>>>> https://pypi.org/project/inotify_simple/. This will change it
>>>>>>>>>> from polling
>>>>>>>>>>> to an event-driven design, which is much more performant and
>> less
>>>>>>>>>> resource
>>>>>>>>>>> hungry. But this would be an AIP on its own.
>>>>>>>>>>>
>>>>>>>>>>> Again, great design and a comprehensive AIP, but I would
>> include
>>>>>>>>>> the
>>>>>>>>>>> caching on the webserver to greatly improve the user experience
>>> in
>>>>>>>>>> the UI.
>>>>>>>>>>> Looking forward to the opinion of others on this.
>>>>>>>>>>>
>>>>>>>>>>> Cheers, Fokko
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Op za 27 jul. 2019 om 01:44 schreef Zhou Fang
>>>>>>>>>> <zhoufang@google.com.invalid
>>>>>>>>>>>> :
>>>>>>>>>>>
>>>>>>>>>>>> Hi Kaxi,
>>>>>>>>>>>>
>>>>>>>>>>>> Just sent out the AIP:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistence+in+DB+using+JSON+for+Airflow+Webserver+and+%28optional%29+Scheduler
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks!
>>>>>>>>>>>> Zhou
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Jul 26, 2019 at 1:33 PM Zhou Fang <
>> zhoufang@google.com
>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Kaxil,
>>>>>>>>>>>>>
>>>>>>>>>>>>> We are also working on persisting DAGs into DB using JSON
>> for
>>>>>>>>>> Airflow
>>>>>>>>>>>>> webserver in Google Composer. We target at minimizing the
>>>>>>>>>> change to the
>>>>>>>>>>>>> current Airflow code. Happy to get synced on this!
>>>>>>>>>>>>>
>>>>>>>>>>>>> Here is our progress:
>>>>>>>>>>>>> (1) Serializing DAGs using Pickle to be used in webserver
>>>>>>>>>>>>> It has been launched in Composer. I am working on the PR to
>>>>>>>>>> upstream
>>>>>>>>>>> it:
>>>>>>>>>>>>> https://github.com/apache/airflow/pull/5594
>>>>>>>>>>>>> Currently it does not support non-Airflow operators and we
>>> are
>>>>>>>>>> working
>>>>>>>>>>> on
>>>>>>>>>>>>> a fix.
>>>>>>>>>>>>>
>>>>>>>>>>>>> (2) Caching Pickled DAGs in DB to be used by webserver
>>>>>>>>>>>>> We have a proof-of-concept implementation, working on an
>> AIP
>>>>>>>>>> now.
>>>>>>>>>>>>>
>>>>>>>>>>>>> (3) Using JSON instead of Pickle in (1) and (2)
>>>>>>>>>>>>> Decided to use JSON because Pickle is not secure and human
>>>>>>>>>> readable.
>>>>>>>>>>> The
>>>>>>>>>>>>> serialization approach is very similar to (1).
>>>>>>>>>>>>>
>>>>>>>>>>>>> I will update the RP (
>>>>>>>>>> https://github.com/apache/airflow/pull/5594) to
>>>>>>>>>>>>> replace Pickle by JSON, and send our design of (2) as an
>> AIP
>>>>>>>>>> next week.
>>>>>>>>>>>>> Glad to check together whether our implementation makes
>> sense
>>>>>>>>>> and do
>>>>>>>>>>>>> improvements on that.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>> Zhou
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 7:37 AM Kaxil Naik <
>>>>>>>>>> kaxilnaik@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We, at Astronomer, are going to spend time working on DAG
>>>>>>>>>>> Serialisation.
>>>>>>>>>>>>>> There are 2 AIPs that are somewhat related to what we plan
>>> to
>>>>>>>>>> work on:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - AIP-18 Persist all information from DAG file in DB
>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - AIP-19 Making the webserver stateless
>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We plan to use JSON as the Serialisation format and store
>> it
>>>>>>>>>> as a blob
>>>>>>>>>>>> in
>>>>>>>>>>>>>> metadata DB.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *Goals:*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - Make Webserver Stateless
>>>>>>>>>>>>>> - Use the same version of the DAG across Webserver &
>>>>>>>>>> Scheduler
>>>>>>>>>>>>>> - Keep backward compatibility and have a flag
>> (globally &
>>>>>>>>>> at DAG
>>>>>>>>>>>> level)
>>>>>>>>>>>>>> to turn this feature on/off
>>>>>>>>>>>>>> - Enable DAG Versioning (extended Goal)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We will be preparing a proposal (AIP) after some research
>>> and
>>>>>>>>>> some
>>>>>>>>>>>> initial
>>>>>>>>>>>>>> work and open it for the suggestions of the community.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We already had some good brain-storming sessions with
>>> Twitter
>>>>>>>>>> folks
>>>>>>>>>>>> (DanD
>>>>>>>>>>>>>> &
>>>>>>>>>>>>>> Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from
>>>>>>>>>> Uber) which
>>>>>>>>>>>>>> will
>>>>>>>>>>>>>> be a good starting point for us.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If anyone in the community is interested in it or has some
>>>>>>>>>> experience
>>>>>>>>>>>>>> about
>>>>>>>>>>>>>> the same and want to collaborate please let me know and
>> join
>>>>>>>>>>>>>> #dag-serialisation channel on Airflow Slack.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Kaxil
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>
>>
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
Re: Airflow DAG Serialisation
Posted by Jarek Potiuk <Ja...@polidea.com>.
I think Zhou's change is pretty much backwards-compatible with 1.10.x -
it's basically optimisation that people might find really useful until 2.0
is out. I believe (correct me if I am wrong) it does not require any change
from the user's perspective. Airflow will continue to behave the same way
as 1.10.4, - only making it more resilient in case of huge DAG folders. It
also seems battle-tested already and I think it's a small-ish effort to
continue reviewing it and add the few changes that are being discussed in
the PR.
I think, maybe then a viable approach is to ONLY implement it in v1-10-test
branch and make a 1.10.5 release much faster than by the end of year. If we
can get the 1.10.5 with only that change (and maybe few other bugk fixes )
in a month or two from now - that would be beneficial for the users - they
will get much better performance faster, and they could have more time
preparing for 2.0.0 backwards-incompatible migration later on. I guess it
will take additional few months for anyone to migrate to 2.0.0 to convert
their DAGs and do additional testing.
There is a small risk (and bigger diversion between 1.10.x and 2.0.0) but I
think the risk is very small here - especially that I am sure Zhou and
Composer team will find and fix (and back-port) any issues found in
Composer in case they find them (and this is one of the best testing
grounds for Airflow ;)) .
I volunteer to continue merging simple changes to 1.10. branch if we go
that direction (and help with 1.10.5 release when the time comes). I kind
of got used to it already and it's a bit of a habit to try to cherry-pick
those changes back to the v1-10-test branch.
J.
On Tue, Jul 30, 2019 at 8:28 AM Kaxil Naik <ka...@gmail.com> wrote:
> Thanks Kevin for the input.
>
> I am working on this full-time as well with help from Ash.
>
> Persistence to DB is what we want to achieve while making Webserver
> stateless and hence mitigating the issues you described.
>
> My opinion is that we should work towards that goal and aim to include it
> in Airflow 2.0 . However I am not completely against a short-term solution
> that will be completely optional, that might increase the work though.
>
> The diff between master and the current release branch is huge. So we
> should aim to cut 2.0 by the end of this year with Dag Serialisation &
> peristence in DB feature.
>
> We can then have minor releases directly from Master and follow SemVer as
> well.
>
> Regards,
> Kaxil
>
>
> On Tue, Jul 30, 2019, 09:02 Kevin Yang <yr...@gmail.com> wrote:
>
> > For sure! I'll try my best to squeeze some time for it during the weekend
> > and see how I can help facilitate the effort( don't get the hopes up too
> > much tho, got ~2 million things piled in my TODO list :P ). Will bring
> this
> > up to the team and see if my team can help too.
> >
> > You guys have showed pretty solid understanding and skills so I believe
> you
> > can handle it well w/o me but just in case you need me, don't hesitate to
> > shoot me a direct mail.
> >
> > Cheers,
> > Kevin Y
> >
> > On Mon, Jul 29, 2019 at 8:19 PM Zhou Fang <zh...@google.com> wrote:
> >
> > > Hi Kevin, it makes sense. Thanks for the explanation! Hope we can get
> DB
> > > persistence move faster.
> > >
> > > Zhou
> > >
> > > On Mon, Jul 29, 2019, 5:46 PM Kevin Yang <yr...@gmail.com> wrote:
> > >
> > >> oops, s/consistent file/consistent file order/
> > >>
> > >> On Mon, Jul 29, 2019 at 5:42 PM Kevin Yang <yr...@gmail.com> wrote:
> > >>
> > >>> Hi Zhou,
> > >>>
> > >>> Totally understood, thank you for that. Streaming logic does cover
> most
> > >>> cases, tho we still have the worst cases where os.walk doesn't give
> us
> > >>> consistent file and file/dir additions/renames causing a different
> > result
> > >>> order of list_py_file_paths( e.g. right after we parsed the first
> dir,
> > it
> > >>> was renamed and will be parsed last in the 2nd DAG loading, or we
> > merged a
> > >>> new file right after the file paths are collected). Maybe there's a
> > way to
> > >>> guarantee the order of parsing but not sure if it worth the effort
> > given
> > >>> that it is less of a problem if the end to end parsing time is small
> > >>> enough. I understand it may be started as a short term improvement
> but
> > >>> since it should not be too much more complicated, we'd rather start
> > with
> > >>> unified long term pattern.
> > >>>
> > >>> Cheers,
> > >>> Kevin Y
> > >>>
> > >>>
> > >>> On Mon, Jul 29, 2019 at 3:59 PM Zhou Fang <zh...@google.com>
> wrote:
> > >>>
> > >>>> Hi Kevin,
> > >>>>
> > >>>> Yes. DAG persistence in DB is definitely the way to go. I referred
> to
> > >>>> the aysnc dag loader because it may alleviate your current problem
> > (since
> > >>>> it is code ready).
> > >>>>
> > >>>> It actually reduces the time to 15min, because DAGs are refreshed by
> > >>>> the background process in a streaming way and you don't need to
> > restart
> > >>>> webserver per 20min.
> > >>>>
> > >>>>
> > >>>>
> > >>>> Thanks,
> > >>>> Zhou
> > >>>>
> > >>>>
> > >>>> On Mon, Jul 29, 2019 at 3:14 PM Kevin Yang <yr...@gmail.com>
> wrote:
> > >>>>
> > >>>>> Hi Zhou,
> > >>>>>
> > >>>>> Thank you for the pointer. This solves the issue gunicorn restart
> > rate
> > >>>>> throttles webserver refresh rate but not the long DAG parsing time
> > issue,
> > >>>>> right? Worst case scenario we still wait 30 mins for the change to
> > show up,
> > >>>>> comparing to the previous 35 mins( I was wrong on the number, it
> > should be
> > >>>>> 35 mins instead of 55 mins as the clock starts whenever the
> webserver
> > >>>>> restarts). I believe in the previous discussion, we firstly
> proposed
> > this
> > >>>>> local webserver DAG parsing optimization to use the same DAG
> parsing
> > logic
> > >>>>> in scheduler to speed up the parsing. Then the stateless webserver
> > proposal
> > >>>>> came up and we were brought in that it is a better idea to persist
> > DAGs
> > >>>>> into the DB and read directly from the DB, better DAG def
> > consistency and
> > >>>>> webserver cluster consistency. I'm all supportive on the proposed
> > structure
> > >>>>> in AIP-24 but -1 on just feed webserver with a single subprocess
> > parsing
> > >>>>> the DAGs. I would image there won't be too many additional work to
> > fetch
> > >>>>> from DB instead of a subprocess, would there?( haven't look into
> the
> > >>>>> serialization format part but assuming they are the same/similar)
> > >>>>>
> > >>>>> Cheers,
> > >>>>> Kevin Y
> > >>>>>
> > >>>>> On Mon, Jul 29, 2019 at 2:18 PM Zhou Fang <zh...@google.com>
> > wrote:
> > >>>>>
> > >>>>>> Hi Kevin,
> > >>>>>>
> > >>>>>> The problem that DAG parsing takes a long time can be solved by
> > >>>>>> Asynchronous DAG loading:
> > https://github.com/apache/airflow/pull/5594
> > >>>>>>
> > >>>>>> The idea is the a background process parses DAG files, and sends
> > DAGs
> > >>>>>> to webserver process every [webserver] dagbag_sync_interval = 10s.
> > >>>>>>
> > >>>>>> We have launched it in Composer, so our users can set
> > >>>>>> webserver worker restart interval to 1 hour (or longer). The
> > background DAG
> > >>>>>> parsing processing refresh all DAGs per [webserver] =
> > collect_dags_interval
> > >>>>>> = 30s.
> > >>>>>>
> > >>>>>> If parsing all DAGs take 15min, you can see DAGs being gradually
> > >>>>>> freshed with this feature.
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>> Zhou
> > >>>>>>
> > >>>>>>
> > >>>>>> On Sat, Jul 27, 2019 at 2:43 AM Kevin Yang <yr...@gmail.com>
> > wrote:
> > >>>>>>
> > >>>>>>> Nice job Zhou!
> > >>>>>>>
> > >>>>>>> Really excited, exactly what we wanted for the webserver scaling
> > >>>>>>> issue.
> > >>>>>>> Want to add another big drive for Airbnb to start think about
> this
> > >>>>>>> previously to support the effort: it can not only bring
> consistency
> > >>>>>>> between
> > >>>>>>> webservers but also bring consistency between webserver and
> > >>>>>>> scheduler/workers. It may be less of a problem if total DAG
> parsing
> > >>>>>>> time is
> > >>>>>>> small, but for us the total DAG parsing time is 15+ mins and we
> had
> > >>>>>>> to set
> > >>>>>>> the webserver( gunicorn subprocesses) restart interval to 20
> mins,
> > >>>>>>> which
> > >>>>>>> leads to a worst case 15+20+15=50 mins delay between scheduler
> > start
> > >>>>>>> to
> > >>>>>>> schedule things and users can see their deployed DAGs/changes...
> > >>>>>>>
> > >>>>>>> I'm not so sure about the scheduler performance improvement:
> > >>>>>>> currently we
> > >>>>>>> already feed the main scheduler process with SimpleDag through
> > >>>>>>> DagFileProcessorManager running in a subprocess--in the future we
> > >>>>>>> feed it
> > >>>>>>> with data from DB, which is likely slower( tho the diff should
> have
> > >>>>>>> negligible impact to the scheduler performance). In fact if we'd
> > >>>>>>> keep the
> > >>>>>>> existing behavior, try schedule only fresh parsed DAGs, then we
> may
> > >>>>>>> need to
> > >>>>>>> deal with some consistency issue--dag processor and the scheduler
> > >>>>>>> race for
> > >>>>>>> updating the flag indicating if the DAG is newly parsed. No big
> > deal
> > >>>>>>> there
> > >>>>>>> but just some thoughts on the top of my head and hopefully can be
> > >>>>>>> helpful.
> > >>>>>>>
> > >>>>>>> And good idea on pre-rendering the template, believe template
> > >>>>>>> rendering was
> > >>>>>>> the biggest concern in the previous discussion. We've also chose
> > the
> > >>>>>>> pre-rendering+JSON approach in our smart sensor API
> > >>>>>>> <
> > >>>>>>>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization
> > >>>>>>> >
> > >>>>>>> and
> > >>>>>>> seems to be working fine--a supporting case for ur proposal ;)
> > >>>>>>> There's a WIP
> > >>>>>>> PR <https://github.com/apache/airflow/pull/5499> for it just in
> > >>>>>>> case you
> > >>>>>>> are interested--maybe we can even share some logics.
> > >>>>>>>
> > >>>>>>> Thumbs-up again for this and please don't heisitate to reach out
> if
> > >>>>>>> you
> > >>>>>>> want to discuss further with us or need any help from us.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Cheers,
> > >>>>>>> Kevin Y
> > >>>>>>>
> > >>>>>>> On Sat, Jul 27, 2019 at 12:54 AM Driesprong, Fokko
> > >>>>>>> <fo...@driesprong.frl>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>> > Looks great Zhou,
> > >>>>>>> >
> > >>>>>>> > I have one thing that pops in my mind while reading the AIP;
> > >>>>>>> should keep
> > >>>>>>> > the caching on the webserver level. As the famous quote goes:
> > >>>>>>> *"There are
> > >>>>>>> > only two hard things in Computer Science: cache invalidation
> and
> > >>>>>>> naming
> > >>>>>>> > things." -- Phil Karlton*
> > >>>>>>> >
> > >>>>>>> > Right now, the fundamental change that is being proposed in the
> > >>>>>>> AIP is
> > >>>>>>> > fetching the DAGs from the database in a serialized format, and
> > >>>>>>> not parsing
> > >>>>>>> > the Python files all the time. This will give already a great
> > >>>>>>> performance
> > >>>>>>> > improvement on the webserver side because it removes a lot of
> the
> > >>>>>>> > processing. However, since we're still fetching the DAGs from
> the
> > >>>>>>> database
> > >>>>>>> > in a regular interval, cache it in the local process, so we
> still
> > >>>>>>> have the
> > >>>>>>> > two issues that Airflow is suffering from right now:
> > >>>>>>> >
> > >>>>>>> > 1. No snappy UI because it is still polling the database in
> a
> > >>>>>>> regular
> > >>>>>>> > interval.
> > >>>>>>> > 2. Inconsistency between webservers because they might poll
> > in a
> > >>>>>>> > different interval, I think we've all seen this:
> > >>>>>>> > https://www.youtube.com/watch?v=sNrBruPS3r4
> > >>>>>>> >
> > >>>>>>> > As I also mentioned in the Slack channel, I strongly feel that
> we
> > >>>>>>> should be
> > >>>>>>> > able to render most views from the tables in the database, so
> > >>>>>>> without
> > >>>>>>> > touching the blob. For specific views, we could just pull the
> > blob
> > >>>>>>> from the
> > >>>>>>> > database. In this case we always have the latest version, and
> we
> > >>>>>>> tackle the
> > >>>>>>> > second point above.
> > >>>>>>> >
> > >>>>>>> > To tackle the first one, I also have an idea. We should change
> > the
> > >>>>>>> DAG
> > >>>>>>> > parser from a loop to something that uses inotify
> > >>>>>>> > https://pypi.org/project/inotify_simple/. This will change it
> > >>>>>>> from polling
> > >>>>>>> > to an event-driven design, which is much more performant and
> less
> > >>>>>>> resource
> > >>>>>>> > hungry. But this would be an AIP on its own.
> > >>>>>>> >
> > >>>>>>> > Again, great design and a comprehensive AIP, but I would
> include
> > >>>>>>> the
> > >>>>>>> > caching on the webserver to greatly improve the user experience
> > in
> > >>>>>>> the UI.
> > >>>>>>> > Looking forward to the opinion of others on this.
> > >>>>>>> >
> > >>>>>>> > Cheers, Fokko
> > >>>>>>> >
> > >>>>>>> >
> > >>>>>>> >
> > >>>>>>> >
> > >>>>>>> >
> > >>>>>>> >
> > >>>>>>> >
> > >>>>>>> >
> > >>>>>>> > Op za 27 jul. 2019 om 01:44 schreef Zhou Fang
> > >>>>>>> <zhoufang@google.com.invalid
> > >>>>>>> > >:
> > >>>>>>> >
> > >>>>>>> > > Hi Kaxi,
> > >>>>>>> > >
> > >>>>>>> > > Just sent out the AIP:
> > >>>>>>> > >
> > >>>>>>> > >
> > >>>>>>> >
> > >>>>>>>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistence+in+DB+using+JSON+for+Airflow+Webserver+and+%28optional%29+Scheduler
> > >>>>>>> > >
> > >>>>>>> > > Thanks!
> > >>>>>>> > > Zhou
> > >>>>>>> > >
> > >>>>>>> > >
> > >>>>>>> > > On Fri, Jul 26, 2019 at 1:33 PM Zhou Fang <
> zhoufang@google.com
> > >
> > >>>>>>> wrote:
> > >>>>>>> > >
> > >>>>>>> > > > Hi Kaxil,
> > >>>>>>> > > >
> > >>>>>>> > > > We are also working on persisting DAGs into DB using JSON
> for
> > >>>>>>> Airflow
> > >>>>>>> > > > webserver in Google Composer. We target at minimizing the
> > >>>>>>> change to the
> > >>>>>>> > > > current Airflow code. Happy to get synced on this!
> > >>>>>>> > > >
> > >>>>>>> > > > Here is our progress:
> > >>>>>>> > > > (1) Serializing DAGs using Pickle to be used in webserver
> > >>>>>>> > > > It has been launched in Composer. I am working on the PR to
> > >>>>>>> upstream
> > >>>>>>> > it:
> > >>>>>>> > > > https://github.com/apache/airflow/pull/5594
> > >>>>>>> > > > Currently it does not support non-Airflow operators and we
> > are
> > >>>>>>> working
> > >>>>>>> > on
> > >>>>>>> > > > a fix.
> > >>>>>>> > > >
> > >>>>>>> > > > (2) Caching Pickled DAGs in DB to be used by webserver
> > >>>>>>> > > > We have a proof-of-concept implementation, working on an
> AIP
> > >>>>>>> now.
> > >>>>>>> > > >
> > >>>>>>> > > > (3) Using JSON instead of Pickle in (1) and (2)
> > >>>>>>> > > > Decided to use JSON because Pickle is not secure and human
> > >>>>>>> readable.
> > >>>>>>> > The
> > >>>>>>> > > > serialization approach is very similar to (1).
> > >>>>>>> > > >
> > >>>>>>> > > > I will update the RP (
> > >>>>>>> https://github.com/apache/airflow/pull/5594) to
> > >>>>>>> > > > replace Pickle by JSON, and send our design of (2) as an
> AIP
> > >>>>>>> next week.
> > >>>>>>> > > > Glad to check together whether our implementation makes
> sense
> > >>>>>>> and do
> > >>>>>>> > > > improvements on that.
> > >>>>>>> > > >
> > >>>>>>> > > > Thanks!
> > >>>>>>> > > > Zhou
> > >>>>>>> > > >
> > >>>>>>> > > >
> > >>>>>>> > > > On Fri, Jul 26, 2019 at 7:37 AM Kaxil Naik <
> > >>>>>>> kaxilnaik@gmail.com>
> > >>>>>>> > wrote:
> > >>>>>>> > > >
> > >>>>>>> > > >> Hi all,
> > >>>>>>> > > >>
> > >>>>>>> > > >> We, at Astronomer, are going to spend time working on DAG
> > >>>>>>> > Serialisation.
> > >>>>>>> > > >> There are 2 AIPs that are somewhat related to what we plan
> > to
> > >>>>>>> work on:
> > >>>>>>> > > >>
> > >>>>>>> > > >> - AIP-18 Persist all information from DAG file in DB
> > >>>>>>> > > >> <
> > >>>>>>> > > >>
> > >>>>>>> > >
> > >>>>>>> >
> > >>>>>>>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB
> > >>>>>>> > > >> >
> > >>>>>>> > > >> - AIP-19 Making the webserver stateless
> > >>>>>>> > > >> <
> > >>>>>>> > > >>
> > >>>>>>> > >
> > >>>>>>> >
> > >>>>>>>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless
> > >>>>>>> > > >> >
> > >>>>>>> > > >>
> > >>>>>>> > > >> We plan to use JSON as the Serialisation format and store
> it
> > >>>>>>> as a blob
> > >>>>>>> > > in
> > >>>>>>> > > >> metadata DB.
> > >>>>>>> > > >>
> > >>>>>>> > > >> *Goals:*
> > >>>>>>> > > >>
> > >>>>>>> > > >> - Make Webserver Stateless
> > >>>>>>> > > >> - Use the same version of the DAG across Webserver &
> > >>>>>>> Scheduler
> > >>>>>>> > > >> - Keep backward compatibility and have a flag
> (globally &
> > >>>>>>> at DAG
> > >>>>>>> > > level)
> > >>>>>>> > > >> to turn this feature on/off
> > >>>>>>> > > >> - Enable DAG Versioning (extended Goal)
> > >>>>>>> > > >>
> > >>>>>>> > > >>
> > >>>>>>> > > >> We will be preparing a proposal (AIP) after some research
> > and
> > >>>>>>> some
> > >>>>>>> > > initial
> > >>>>>>> > > >> work and open it for the suggestions of the community.
> > >>>>>>> > > >>
> > >>>>>>> > > >> We already had some good brain-storming sessions with
> > Twitter
> > >>>>>>> folks
> > >>>>>>> > > (DanD
> > >>>>>>> > > >> &
> > >>>>>>> > > >> Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from
> > >>>>>>> Uber) which
> > >>>>>>> > > >> will
> > >>>>>>> > > >> be a good starting point for us.
> > >>>>>>> > > >>
> > >>>>>>> > > >> If anyone in the community is interested in it or has some
> > >>>>>>> experience
> > >>>>>>> > > >> about
> > >>>>>>> > > >> the same and want to collaborate please let me know and
> join
> > >>>>>>> > > >> #dag-serialisation channel on Airflow Slack.
> > >>>>>>> > > >>
> > >>>>>>> > > >> Regards,
> > >>>>>>> > > >> Kaxil
> > >>>>>>> > > >>
> > >>>>>>> > > >
> > >>>>>>> > >
> > >>>>>>> >
> > >>>>>>>
> > >>>>>>
> >
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Airflow DAG Serialisation
Posted by Kaxil Naik <ka...@gmail.com>.
Thanks Kevin for the input.
I am working on this full-time as well with help from Ash.
Persistence to DB is what we want to achieve while making Webserver
stateless and hence mitigating the issues you described.
My opinion is that we should work towards that goal and aim to include it
in Airflow 2.0 . However I am not completely against a short-term solution
that will be completely optional, that might increase the work though.
The diff between master and the current release branch is huge. So we
should aim to cut 2.0 by the end of this year with Dag Serialisation &
peristence in DB feature.
We can then have minor releases directly from Master and follow SemVer as
well.
Regards,
Kaxil
On Tue, Jul 30, 2019, 09:02 Kevin Yang <yr...@gmail.com> wrote:
> For sure! I'll try my best to squeeze some time for it during the weekend
> and see how I can help facilitate the effort( don't get the hopes up too
> much tho, got ~2 million things piled in my TODO list :P ). Will bring this
> up to the team and see if my team can help too.
>
> You guys have showed pretty solid understanding and skills so I believe you
> can handle it well w/o me but just in case you need me, don't hesitate to
> shoot me a direct mail.
>
> Cheers,
> Kevin Y
>
> On Mon, Jul 29, 2019 at 8:19 PM Zhou Fang <zh...@google.com> wrote:
>
> > Hi Kevin, it makes sense. Thanks for the explanation! Hope we can get DB
> > persistence move faster.
> >
> > Zhou
> >
> > On Mon, Jul 29, 2019, 5:46 PM Kevin Yang <yr...@gmail.com> wrote:
> >
> >> oops, s/consistent file/consistent file order/
> >>
> >> On Mon, Jul 29, 2019 at 5:42 PM Kevin Yang <yr...@gmail.com> wrote:
> >>
> >>> Hi Zhou,
> >>>
> >>> Totally understood, thank you for that. Streaming logic does cover most
> >>> cases, tho we still have the worst cases where os.walk doesn't give us
> >>> consistent file and file/dir additions/renames causing a different
> result
> >>> order of list_py_file_paths( e.g. right after we parsed the first dir,
> it
> >>> was renamed and will be parsed last in the 2nd DAG loading, or we
> merged a
> >>> new file right after the file paths are collected). Maybe there's a
> way to
> >>> guarantee the order of parsing but not sure if it worth the effort
> given
> >>> that it is less of a problem if the end to end parsing time is small
> >>> enough. I understand it may be started as a short term improvement but
> >>> since it should not be too much more complicated, we'd rather start
> with
> >>> unified long term pattern.
> >>>
> >>> Cheers,
> >>> Kevin Y
> >>>
> >>>
> >>> On Mon, Jul 29, 2019 at 3:59 PM Zhou Fang <zh...@google.com> wrote:
> >>>
> >>>> Hi Kevin,
> >>>>
> >>>> Yes. DAG persistence in DB is definitely the way to go. I referred to
> >>>> the aysnc dag loader because it may alleviate your current problem
> (since
> >>>> it is code ready).
> >>>>
> >>>> It actually reduces the time to 15min, because DAGs are refreshed by
> >>>> the background process in a streaming way and you don't need to
> restart
> >>>> webserver per 20min.
> >>>>
> >>>>
> >>>>
> >>>> Thanks,
> >>>> Zhou
> >>>>
> >>>>
> >>>> On Mon, Jul 29, 2019 at 3:14 PM Kevin Yang <yr...@gmail.com> wrote:
> >>>>
> >>>>> Hi Zhou,
> >>>>>
> >>>>> Thank you for the pointer. This solves the issue gunicorn restart
> rate
> >>>>> throttles webserver refresh rate but not the long DAG parsing time
> issue,
> >>>>> right? Worst case scenario we still wait 30 mins for the change to
> show up,
> >>>>> comparing to the previous 35 mins( I was wrong on the number, it
> should be
> >>>>> 35 mins instead of 55 mins as the clock starts whenever the webserver
> >>>>> restarts). I believe in the previous discussion, we firstly proposed
> this
> >>>>> local webserver DAG parsing optimization to use the same DAG parsing
> logic
> >>>>> in scheduler to speed up the parsing. Then the stateless webserver
> proposal
> >>>>> came up and we were brought in that it is a better idea to persist
> DAGs
> >>>>> into the DB and read directly from the DB, better DAG def
> consistency and
> >>>>> webserver cluster consistency. I'm all supportive on the proposed
> structure
> >>>>> in AIP-24 but -1 on just feed webserver with a single subprocess
> parsing
> >>>>> the DAGs. I would image there won't be too many additional work to
> fetch
> >>>>> from DB instead of a subprocess, would there?( haven't look into the
> >>>>> serialization format part but assuming they are the same/similar)
> >>>>>
> >>>>> Cheers,
> >>>>> Kevin Y
> >>>>>
> >>>>> On Mon, Jul 29, 2019 at 2:18 PM Zhou Fang <zh...@google.com>
> wrote:
> >>>>>
> >>>>>> Hi Kevin,
> >>>>>>
> >>>>>> The problem that DAG parsing takes a long time can be solved by
> >>>>>> Asynchronous DAG loading:
> https://github.com/apache/airflow/pull/5594
> >>>>>>
> >>>>>> The idea is the a background process parses DAG files, and sends
> DAGs
> >>>>>> to webserver process every [webserver] dagbag_sync_interval = 10s.
> >>>>>>
> >>>>>> We have launched it in Composer, so our users can set
> >>>>>> webserver worker restart interval to 1 hour (or longer). The
> background DAG
> >>>>>> parsing processing refresh all DAGs per [webserver] =
> collect_dags_interval
> >>>>>> = 30s.
> >>>>>>
> >>>>>> If parsing all DAGs take 15min, you can see DAGs being gradually
> >>>>>> freshed with this feature.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Zhou
> >>>>>>
> >>>>>>
> >>>>>> On Sat, Jul 27, 2019 at 2:43 AM Kevin Yang <yr...@gmail.com>
> wrote:
> >>>>>>
> >>>>>>> Nice job Zhou!
> >>>>>>>
> >>>>>>> Really excited, exactly what we wanted for the webserver scaling
> >>>>>>> issue.
> >>>>>>> Want to add another big drive for Airbnb to start think about this
> >>>>>>> previously to support the effort: it can not only bring consistency
> >>>>>>> between
> >>>>>>> webservers but also bring consistency between webserver and
> >>>>>>> scheduler/workers. It may be less of a problem if total DAG parsing
> >>>>>>> time is
> >>>>>>> small, but for us the total DAG parsing time is 15+ mins and we had
> >>>>>>> to set
> >>>>>>> the webserver( gunicorn subprocesses) restart interval to 20 mins,
> >>>>>>> which
> >>>>>>> leads to a worst case 15+20+15=50 mins delay between scheduler
> start
> >>>>>>> to
> >>>>>>> schedule things and users can see their deployed DAGs/changes...
> >>>>>>>
> >>>>>>> I'm not so sure about the scheduler performance improvement:
> >>>>>>> currently we
> >>>>>>> already feed the main scheduler process with SimpleDag through
> >>>>>>> DagFileProcessorManager running in a subprocess--in the future we
> >>>>>>> feed it
> >>>>>>> with data from DB, which is likely slower( tho the diff should have
> >>>>>>> negligible impact to the scheduler performance). In fact if we'd
> >>>>>>> keep the
> >>>>>>> existing behavior, try schedule only fresh parsed DAGs, then we may
> >>>>>>> need to
> >>>>>>> deal with some consistency issue--dag processor and the scheduler
> >>>>>>> race for
> >>>>>>> updating the flag indicating if the DAG is newly parsed. No big
> deal
> >>>>>>> there
> >>>>>>> but just some thoughts on the top of my head and hopefully can be
> >>>>>>> helpful.
> >>>>>>>
> >>>>>>> And good idea on pre-rendering the template, believe template
> >>>>>>> rendering was
> >>>>>>> the biggest concern in the previous discussion. We've also chose
> the
> >>>>>>> pre-rendering+JSON approach in our smart sensor API
> >>>>>>> <
> >>>>>>>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization
> >>>>>>> >
> >>>>>>> and
> >>>>>>> seems to be working fine--a supporting case for ur proposal ;)
> >>>>>>> There's a WIP
> >>>>>>> PR <https://github.com/apache/airflow/pull/5499> for it just in
> >>>>>>> case you
> >>>>>>> are interested--maybe we can even share some logics.
> >>>>>>>
> >>>>>>> Thumbs-up again for this and please don't heisitate to reach out if
> >>>>>>> you
> >>>>>>> want to discuss further with us or need any help from us.
> >>>>>>>
> >>>>>>>
> >>>>>>> Cheers,
> >>>>>>> Kevin Y
> >>>>>>>
> >>>>>>> On Sat, Jul 27, 2019 at 12:54 AM Driesprong, Fokko
> >>>>>>> <fo...@driesprong.frl>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>> > Looks great Zhou,
> >>>>>>> >
> >>>>>>> > I have one thing that pops in my mind while reading the AIP;
> >>>>>>> should keep
> >>>>>>> > the caching on the webserver level. As the famous quote goes:
> >>>>>>> *"There are
> >>>>>>> > only two hard things in Computer Science: cache invalidation and
> >>>>>>> naming
> >>>>>>> > things." -- Phil Karlton*
> >>>>>>> >
> >>>>>>> > Right now, the fundamental change that is being proposed in the
> >>>>>>> AIP is
> >>>>>>> > fetching the DAGs from the database in a serialized format, and
> >>>>>>> not parsing
> >>>>>>> > the Python files all the time. This will give already a great
> >>>>>>> performance
> >>>>>>> > improvement on the webserver side because it removes a lot of the
> >>>>>>> > processing. However, since we're still fetching the DAGs from the
> >>>>>>> database
> >>>>>>> > in a regular interval, cache it in the local process, so we still
> >>>>>>> have the
> >>>>>>> > two issues that Airflow is suffering from right now:
> >>>>>>> >
> >>>>>>> > 1. No snappy UI because it is still polling the database in a
> >>>>>>> regular
> >>>>>>> > interval.
> >>>>>>> > 2. Inconsistency between webservers because they might poll
> in a
> >>>>>>> > different interval, I think we've all seen this:
> >>>>>>> > https://www.youtube.com/watch?v=sNrBruPS3r4
> >>>>>>> >
> >>>>>>> > As I also mentioned in the Slack channel, I strongly feel that we
> >>>>>>> should be
> >>>>>>> > able to render most views from the tables in the database, so
> >>>>>>> without
> >>>>>>> > touching the blob. For specific views, we could just pull the
> blob
> >>>>>>> from the
> >>>>>>> > database. In this case we always have the latest version, and we
> >>>>>>> tackle the
> >>>>>>> > second point above.
> >>>>>>> >
> >>>>>>> > To tackle the first one, I also have an idea. We should change
> the
> >>>>>>> DAG
> >>>>>>> > parser from a loop to something that uses inotify
> >>>>>>> > https://pypi.org/project/inotify_simple/. This will change it
> >>>>>>> from polling
> >>>>>>> > to an event-driven design, which is much more performant and less
> >>>>>>> resource
> >>>>>>> > hungry. But this would be an AIP on its own.
> >>>>>>> >
> >>>>>>> > Again, great design and a comprehensive AIP, but I would include
> >>>>>>> the
> >>>>>>> > caching on the webserver to greatly improve the user experience
> in
> >>>>>>> the UI.
> >>>>>>> > Looking forward to the opinion of others on this.
> >>>>>>> >
> >>>>>>> > Cheers, Fokko
> >>>>>>> >
> >>>>>>> >
> >>>>>>> >
> >>>>>>> >
> >>>>>>> >
> >>>>>>> >
> >>>>>>> >
> >>>>>>> >
> >>>>>>> > Op za 27 jul. 2019 om 01:44 schreef Zhou Fang
> >>>>>>> <zhoufang@google.com.invalid
> >>>>>>> > >:
> >>>>>>> >
> >>>>>>> > > Hi Kaxi,
> >>>>>>> > >
> >>>>>>> > > Just sent out the AIP:
> >>>>>>> > >
> >>>>>>> > >
> >>>>>>> >
> >>>>>>>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistence+in+DB+using+JSON+for+Airflow+Webserver+and+%28optional%29+Scheduler
> >>>>>>> > >
> >>>>>>> > > Thanks!
> >>>>>>> > > Zhou
> >>>>>>> > >
> >>>>>>> > >
> >>>>>>> > > On Fri, Jul 26, 2019 at 1:33 PM Zhou Fang <zhoufang@google.com
> >
> >>>>>>> wrote:
> >>>>>>> > >
> >>>>>>> > > > Hi Kaxil,
> >>>>>>> > > >
> >>>>>>> > > > We are also working on persisting DAGs into DB using JSON for
> >>>>>>> Airflow
> >>>>>>> > > > webserver in Google Composer. We target at minimizing the
> >>>>>>> change to the
> >>>>>>> > > > current Airflow code. Happy to get synced on this!
> >>>>>>> > > >
> >>>>>>> > > > Here is our progress:
> >>>>>>> > > > (1) Serializing DAGs using Pickle to be used in webserver
> >>>>>>> > > > It has been launched in Composer. I am working on the PR to
> >>>>>>> upstream
> >>>>>>> > it:
> >>>>>>> > > > https://github.com/apache/airflow/pull/5594
> >>>>>>> > > > Currently it does not support non-Airflow operators and we
> are
> >>>>>>> working
> >>>>>>> > on
> >>>>>>> > > > a fix.
> >>>>>>> > > >
> >>>>>>> > > > (2) Caching Pickled DAGs in DB to be used by webserver
> >>>>>>> > > > We have a proof-of-concept implementation, working on an AIP
> >>>>>>> now.
> >>>>>>> > > >
> >>>>>>> > > > (3) Using JSON instead of Pickle in (1) and (2)
> >>>>>>> > > > Decided to use JSON because Pickle is not secure and human
> >>>>>>> readable.
> >>>>>>> > The
> >>>>>>> > > > serialization approach is very similar to (1).
> >>>>>>> > > >
> >>>>>>> > > > I will update the RP (
> >>>>>>> https://github.com/apache/airflow/pull/5594) to
> >>>>>>> > > > replace Pickle by JSON, and send our design of (2) as an AIP
> >>>>>>> next week.
> >>>>>>> > > > Glad to check together whether our implementation makes sense
> >>>>>>> and do
> >>>>>>> > > > improvements on that.
> >>>>>>> > > >
> >>>>>>> > > > Thanks!
> >>>>>>> > > > Zhou
> >>>>>>> > > >
> >>>>>>> > > >
> >>>>>>> > > > On Fri, Jul 26, 2019 at 7:37 AM Kaxil Naik <
> >>>>>>> kaxilnaik@gmail.com>
> >>>>>>> > wrote:
> >>>>>>> > > >
> >>>>>>> > > >> Hi all,
> >>>>>>> > > >>
> >>>>>>> > > >> We, at Astronomer, are going to spend time working on DAG
> >>>>>>> > Serialisation.
> >>>>>>> > > >> There are 2 AIPs that are somewhat related to what we plan
> to
> >>>>>>> work on:
> >>>>>>> > > >>
> >>>>>>> > > >> - AIP-18 Persist all information from DAG file in DB
> >>>>>>> > > >> <
> >>>>>>> > > >>
> >>>>>>> > >
> >>>>>>> >
> >>>>>>>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB
> >>>>>>> > > >> >
> >>>>>>> > > >> - AIP-19 Making the webserver stateless
> >>>>>>> > > >> <
> >>>>>>> > > >>
> >>>>>>> > >
> >>>>>>> >
> >>>>>>>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless
> >>>>>>> > > >> >
> >>>>>>> > > >>
> >>>>>>> > > >> We plan to use JSON as the Serialisation format and store it
> >>>>>>> as a blob
> >>>>>>> > > in
> >>>>>>> > > >> metadata DB.
> >>>>>>> > > >>
> >>>>>>> > > >> *Goals:*
> >>>>>>> > > >>
> >>>>>>> > > >> - Make Webserver Stateless
> >>>>>>> > > >> - Use the same version of the DAG across Webserver &
> >>>>>>> Scheduler
> >>>>>>> > > >> - Keep backward compatibility and have a flag (globally &
> >>>>>>> at DAG
> >>>>>>> > > level)
> >>>>>>> > > >> to turn this feature on/off
> >>>>>>> > > >> - Enable DAG Versioning (extended Goal)
> >>>>>>> > > >>
> >>>>>>> > > >>
> >>>>>>> > > >> We will be preparing a proposal (AIP) after some research
> and
> >>>>>>> some
> >>>>>>> > > initial
> >>>>>>> > > >> work and open it for the suggestions of the community.
> >>>>>>> > > >>
> >>>>>>> > > >> We already had some good brain-storming sessions with
> Twitter
> >>>>>>> folks
> >>>>>>> > > (DanD
> >>>>>>> > > >> &
> >>>>>>> > > >> Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from
> >>>>>>> Uber) which
> >>>>>>> > > >> will
> >>>>>>> > > >> be a good starting point for us.
> >>>>>>> > > >>
> >>>>>>> > > >> If anyone in the community is interested in it or has some
> >>>>>>> experience
> >>>>>>> > > >> about
> >>>>>>> > > >> the same and want to collaborate please let me know and join
> >>>>>>> > > >> #dag-serialisation channel on Airflow Slack.
> >>>>>>> > > >>
> >>>>>>> > > >> Regards,
> >>>>>>> > > >> Kaxil
> >>>>>>> > > >>
> >>>>>>> > > >
> >>>>>>> > >
> >>>>>>> >
> >>>>>>>
> >>>>>>
>
Re: Airflow DAG Serialisation
Posted by Kevin Yang <yr...@gmail.com>.
For sure! I'll try my best to squeeze some time for it during the weekend
and see how I can help facilitate the effort( don't get the hopes up too
much tho, got ~2 million things piled in my TODO list :P ). Will bring this
up to the team and see if my team can help too.
You guys have showed pretty solid understanding and skills so I believe you
can handle it well w/o me but just in case you need me, don't hesitate to
shoot me a direct mail.
Cheers,
Kevin Y
On Mon, Jul 29, 2019 at 8:19 PM Zhou Fang <zh...@google.com> wrote:
> Hi Kevin, it makes sense. Thanks for the explanation! Hope we can get DB
> persistence move faster.
>
> Zhou
>
> On Mon, Jul 29, 2019, 5:46 PM Kevin Yang <yr...@gmail.com> wrote:
>
>> oops, s/consistent file/consistent file order/
>>
>> On Mon, Jul 29, 2019 at 5:42 PM Kevin Yang <yr...@gmail.com> wrote:
>>
>>> Hi Zhou,
>>>
>>> Totally understood, thank you for that. Streaming logic does cover most
>>> cases, tho we still have the worst cases where os.walk doesn't give us
>>> consistent file and file/dir additions/renames causing a different result
>>> order of list_py_file_paths( e.g. right after we parsed the first dir, it
>>> was renamed and will be parsed last in the 2nd DAG loading, or we merged a
>>> new file right after the file paths are collected). Maybe there's a way to
>>> guarantee the order of parsing but not sure if it worth the effort given
>>> that it is less of a problem if the end to end parsing time is small
>>> enough. I understand it may be started as a short term improvement but
>>> since it should not be too much more complicated, we'd rather start with
>>> unified long term pattern.
>>>
>>> Cheers,
>>> Kevin Y
>>>
>>>
>>> On Mon, Jul 29, 2019 at 3:59 PM Zhou Fang <zh...@google.com> wrote:
>>>
>>>> Hi Kevin,
>>>>
>>>> Yes. DAG persistence in DB is definitely the way to go. I referred to
>>>> the aysnc dag loader because it may alleviate your current problem (since
>>>> it is code ready).
>>>>
>>>> It actually reduces the time to 15min, because DAGs are refreshed by
>>>> the background process in a streaming way and you don't need to restart
>>>> webserver per 20min.
>>>>
>>>>
>>>>
>>>> Thanks,
>>>> Zhou
>>>>
>>>>
>>>> On Mon, Jul 29, 2019 at 3:14 PM Kevin Yang <yr...@gmail.com> wrote:
>>>>
>>>>> Hi Zhou,
>>>>>
>>>>> Thank you for the pointer. This solves the issue gunicorn restart rate
>>>>> throttles webserver refresh rate but not the long DAG parsing time issue,
>>>>> right? Worst case scenario we still wait 30 mins for the change to show up,
>>>>> comparing to the previous 35 mins( I was wrong on the number, it should be
>>>>> 35 mins instead of 55 mins as the clock starts whenever the webserver
>>>>> restarts). I believe in the previous discussion, we firstly proposed this
>>>>> local webserver DAG parsing optimization to use the same DAG parsing logic
>>>>> in scheduler to speed up the parsing. Then the stateless webserver proposal
>>>>> came up and we were brought in that it is a better idea to persist DAGs
>>>>> into the DB and read directly from the DB, better DAG def consistency and
>>>>> webserver cluster consistency. I'm all supportive on the proposed structure
>>>>> in AIP-24 but -1 on just feed webserver with a single subprocess parsing
>>>>> the DAGs. I would image there won't be too many additional work to fetch
>>>>> from DB instead of a subprocess, would there?( haven't look into the
>>>>> serialization format part but assuming they are the same/similar)
>>>>>
>>>>> Cheers,
>>>>> Kevin Y
>>>>>
>>>>> On Mon, Jul 29, 2019 at 2:18 PM Zhou Fang <zh...@google.com> wrote:
>>>>>
>>>>>> Hi Kevin,
>>>>>>
>>>>>> The problem that DAG parsing takes a long time can be solved by
>>>>>> Asynchronous DAG loading: https://github.com/apache/airflow/pull/5594
>>>>>>
>>>>>> The idea is the a background process parses DAG files, and sends DAGs
>>>>>> to webserver process every [webserver] dagbag_sync_interval = 10s.
>>>>>>
>>>>>> We have launched it in Composer, so our users can set
>>>>>> webserver worker restart interval to 1 hour (or longer). The background DAG
>>>>>> parsing processing refresh all DAGs per [webserver] = collect_dags_interval
>>>>>> = 30s.
>>>>>>
>>>>>> If parsing all DAGs take 15min, you can see DAGs being gradually
>>>>>> freshed with this feature.
>>>>>>
>>>>>> Thanks,
>>>>>> Zhou
>>>>>>
>>>>>>
>>>>>> On Sat, Jul 27, 2019 at 2:43 AM Kevin Yang <yr...@gmail.com> wrote:
>>>>>>
>>>>>>> Nice job Zhou!
>>>>>>>
>>>>>>> Really excited, exactly what we wanted for the webserver scaling
>>>>>>> issue.
>>>>>>> Want to add another big drive for Airbnb to start think about this
>>>>>>> previously to support the effort: it can not only bring consistency
>>>>>>> between
>>>>>>> webservers but also bring consistency between webserver and
>>>>>>> scheduler/workers. It may be less of a problem if total DAG parsing
>>>>>>> time is
>>>>>>> small, but for us the total DAG parsing time is 15+ mins and we had
>>>>>>> to set
>>>>>>> the webserver( gunicorn subprocesses) restart interval to 20 mins,
>>>>>>> which
>>>>>>> leads to a worst case 15+20+15=50 mins delay between scheduler start
>>>>>>> to
>>>>>>> schedule things and users can see their deployed DAGs/changes...
>>>>>>>
>>>>>>> I'm not so sure about the scheduler performance improvement:
>>>>>>> currently we
>>>>>>> already feed the main scheduler process with SimpleDag through
>>>>>>> DagFileProcessorManager running in a subprocess--in the future we
>>>>>>> feed it
>>>>>>> with data from DB, which is likely slower( tho the diff should have
>>>>>>> negligible impact to the scheduler performance). In fact if we'd
>>>>>>> keep the
>>>>>>> existing behavior, try schedule only fresh parsed DAGs, then we may
>>>>>>> need to
>>>>>>> deal with some consistency issue--dag processor and the scheduler
>>>>>>> race for
>>>>>>> updating the flag indicating if the DAG is newly parsed. No big deal
>>>>>>> there
>>>>>>> but just some thoughts on the top of my head and hopefully can be
>>>>>>> helpful.
>>>>>>>
>>>>>>> And good idea on pre-rendering the template, believe template
>>>>>>> rendering was
>>>>>>> the biggest concern in the previous discussion. We've also chose the
>>>>>>> pre-rendering+JSON approach in our smart sensor API
>>>>>>> <
>>>>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization
>>>>>>> >
>>>>>>> and
>>>>>>> seems to be working fine--a supporting case for ur proposal ;)
>>>>>>> There's a WIP
>>>>>>> PR <https://github.com/apache/airflow/pull/5499> for it just in
>>>>>>> case you
>>>>>>> are interested--maybe we can even share some logics.
>>>>>>>
>>>>>>> Thumbs-up again for this and please don't heisitate to reach out if
>>>>>>> you
>>>>>>> want to discuss further with us or need any help from us.
>>>>>>>
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Kevin Y
>>>>>>>
>>>>>>> On Sat, Jul 27, 2019 at 12:54 AM Driesprong, Fokko
>>>>>>> <fo...@driesprong.frl>
>>>>>>> wrote:
>>>>>>>
>>>>>>> > Looks great Zhou,
>>>>>>> >
>>>>>>> > I have one thing that pops in my mind while reading the AIP;
>>>>>>> should keep
>>>>>>> > the caching on the webserver level. As the famous quote goes:
>>>>>>> *"There are
>>>>>>> > only two hard things in Computer Science: cache invalidation and
>>>>>>> naming
>>>>>>> > things." -- Phil Karlton*
>>>>>>> >
>>>>>>> > Right now, the fundamental change that is being proposed in the
>>>>>>> AIP is
>>>>>>> > fetching the DAGs from the database in a serialized format, and
>>>>>>> not parsing
>>>>>>> > the Python files all the time. This will give already a great
>>>>>>> performance
>>>>>>> > improvement on the webserver side because it removes a lot of the
>>>>>>> > processing. However, since we're still fetching the DAGs from the
>>>>>>> database
>>>>>>> > in a regular interval, cache it in the local process, so we still
>>>>>>> have the
>>>>>>> > two issues that Airflow is suffering from right now:
>>>>>>> >
>>>>>>> > 1. No snappy UI because it is still polling the database in a
>>>>>>> regular
>>>>>>> > interval.
>>>>>>> > 2. Inconsistency between webservers because they might poll in a
>>>>>>> > different interval, I think we've all seen this:
>>>>>>> > https://www.youtube.com/watch?v=sNrBruPS3r4
>>>>>>> >
>>>>>>> > As I also mentioned in the Slack channel, I strongly feel that we
>>>>>>> should be
>>>>>>> > able to render most views from the tables in the database, so
>>>>>>> without
>>>>>>> > touching the blob. For specific views, we could just pull the blob
>>>>>>> from the
>>>>>>> > database. In this case we always have the latest version, and we
>>>>>>> tackle the
>>>>>>> > second point above.
>>>>>>> >
>>>>>>> > To tackle the first one, I also have an idea. We should change the
>>>>>>> DAG
>>>>>>> > parser from a loop to something that uses inotify
>>>>>>> > https://pypi.org/project/inotify_simple/. This will change it
>>>>>>> from polling
>>>>>>> > to an event-driven design, which is much more performant and less
>>>>>>> resource
>>>>>>> > hungry. But this would be an AIP on its own.
>>>>>>> >
>>>>>>> > Again, great design and a comprehensive AIP, but I would include
>>>>>>> the
>>>>>>> > caching on the webserver to greatly improve the user experience in
>>>>>>> the UI.
>>>>>>> > Looking forward to the opinion of others on this.
>>>>>>> >
>>>>>>> > Cheers, Fokko
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > Op za 27 jul. 2019 om 01:44 schreef Zhou Fang
>>>>>>> <zhoufang@google.com.invalid
>>>>>>> > >:
>>>>>>> >
>>>>>>> > > Hi Kaxi,
>>>>>>> > >
>>>>>>> > > Just sent out the AIP:
>>>>>>> > >
>>>>>>> > >
>>>>>>> >
>>>>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistence+in+DB+using+JSON+for+Airflow+Webserver+and+%28optional%29+Scheduler
>>>>>>> > >
>>>>>>> > > Thanks!
>>>>>>> > > Zhou
>>>>>>> > >
>>>>>>> > >
>>>>>>> > > On Fri, Jul 26, 2019 at 1:33 PM Zhou Fang <zh...@google.com>
>>>>>>> wrote:
>>>>>>> > >
>>>>>>> > > > Hi Kaxil,
>>>>>>> > > >
>>>>>>> > > > We are also working on persisting DAGs into DB using JSON for
>>>>>>> Airflow
>>>>>>> > > > webserver in Google Composer. We target at minimizing the
>>>>>>> change to the
>>>>>>> > > > current Airflow code. Happy to get synced on this!
>>>>>>> > > >
>>>>>>> > > > Here is our progress:
>>>>>>> > > > (1) Serializing DAGs using Pickle to be used in webserver
>>>>>>> > > > It has been launched in Composer. I am working on the PR to
>>>>>>> upstream
>>>>>>> > it:
>>>>>>> > > > https://github.com/apache/airflow/pull/5594
>>>>>>> > > > Currently it does not support non-Airflow operators and we are
>>>>>>> working
>>>>>>> > on
>>>>>>> > > > a fix.
>>>>>>> > > >
>>>>>>> > > > (2) Caching Pickled DAGs in DB to be used by webserver
>>>>>>> > > > We have a proof-of-concept implementation, working on an AIP
>>>>>>> now.
>>>>>>> > > >
>>>>>>> > > > (3) Using JSON instead of Pickle in (1) and (2)
>>>>>>> > > > Decided to use JSON because Pickle is not secure and human
>>>>>>> readable.
>>>>>>> > The
>>>>>>> > > > serialization approach is very similar to (1).
>>>>>>> > > >
>>>>>>> > > > I will update the RP (
>>>>>>> https://github.com/apache/airflow/pull/5594) to
>>>>>>> > > > replace Pickle by JSON, and send our design of (2) as an AIP
>>>>>>> next week.
>>>>>>> > > > Glad to check together whether our implementation makes sense
>>>>>>> and do
>>>>>>> > > > improvements on that.
>>>>>>> > > >
>>>>>>> > > > Thanks!
>>>>>>> > > > Zhou
>>>>>>> > > >
>>>>>>> > > >
>>>>>>> > > > On Fri, Jul 26, 2019 at 7:37 AM Kaxil Naik <
>>>>>>> kaxilnaik@gmail.com>
>>>>>>> > wrote:
>>>>>>> > > >
>>>>>>> > > >> Hi all,
>>>>>>> > > >>
>>>>>>> > > >> We, at Astronomer, are going to spend time working on DAG
>>>>>>> > Serialisation.
>>>>>>> > > >> There are 2 AIPs that are somewhat related to what we plan to
>>>>>>> work on:
>>>>>>> > > >>
>>>>>>> > > >> - AIP-18 Persist all information from DAG file in DB
>>>>>>> > > >> <
>>>>>>> > > >>
>>>>>>> > >
>>>>>>> >
>>>>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB
>>>>>>> > > >> >
>>>>>>> > > >> - AIP-19 Making the webserver stateless
>>>>>>> > > >> <
>>>>>>> > > >>
>>>>>>> > >
>>>>>>> >
>>>>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless
>>>>>>> > > >> >
>>>>>>> > > >>
>>>>>>> > > >> We plan to use JSON as the Serialisation format and store it
>>>>>>> as a blob
>>>>>>> > > in
>>>>>>> > > >> metadata DB.
>>>>>>> > > >>
>>>>>>> > > >> *Goals:*
>>>>>>> > > >>
>>>>>>> > > >> - Make Webserver Stateless
>>>>>>> > > >> - Use the same version of the DAG across Webserver &
>>>>>>> Scheduler
>>>>>>> > > >> - Keep backward compatibility and have a flag (globally &
>>>>>>> at DAG
>>>>>>> > > level)
>>>>>>> > > >> to turn this feature on/off
>>>>>>> > > >> - Enable DAG Versioning (extended Goal)
>>>>>>> > > >>
>>>>>>> > > >>
>>>>>>> > > >> We will be preparing a proposal (AIP) after some research and
>>>>>>> some
>>>>>>> > > initial
>>>>>>> > > >> work and open it for the suggestions of the community.
>>>>>>> > > >>
>>>>>>> > > >> We already had some good brain-storming sessions with Twitter
>>>>>>> folks
>>>>>>> > > (DanD
>>>>>>> > > >> &
>>>>>>> > > >> Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from
>>>>>>> Uber) which
>>>>>>> > > >> will
>>>>>>> > > >> be a good starting point for us.
>>>>>>> > > >>
>>>>>>> > > >> If anyone in the community is interested in it or has some
>>>>>>> experience
>>>>>>> > > >> about
>>>>>>> > > >> the same and want to collaborate please let me know and join
>>>>>>> > > >> #dag-serialisation channel on Airflow Slack.
>>>>>>> > > >>
>>>>>>> > > >> Regards,
>>>>>>> > > >> Kaxil
>>>>>>> > > >>
>>>>>>> > > >
>>>>>>> > >
>>>>>>> >
>>>>>>>
>>>>>>
Re: Airflow DAG Serialisation
Posted by Zhou Fang <zh...@google.com.INVALID>.
Hi Kevin, it makes sense. Thanks for the explanation! Hope we can get DB
persistence move faster.
Zhou
On Mon, Jul 29, 2019, 5:46 PM Kevin Yang <yr...@gmail.com> wrote:
> oops, s/consistent file/consistent file order/
>
> On Mon, Jul 29, 2019 at 5:42 PM Kevin Yang <yr...@gmail.com> wrote:
>
>> Hi Zhou,
>>
>> Totally understood, thank you for that. Streaming logic does cover most
>> cases, tho we still have the worst cases where os.walk doesn't give us
>> consistent file and file/dir additions/renames causing a different result
>> order of list_py_file_paths( e.g. right after we parsed the first dir, it
>> was renamed and will be parsed last in the 2nd DAG loading, or we merged a
>> new file right after the file paths are collected). Maybe there's a way to
>> guarantee the order of parsing but not sure if it worth the effort given
>> that it is less of a problem if the end to end parsing time is small
>> enough. I understand it may be started as a short term improvement but
>> since it should not be too much more complicated, we'd rather start with
>> unified long term pattern.
>>
>> Cheers,
>> Kevin Y
>>
>>
>> On Mon, Jul 29, 2019 at 3:59 PM Zhou Fang <zh...@google.com> wrote:
>>
>>> Hi Kevin,
>>>
>>> Yes. DAG persistence in DB is definitely the way to go. I referred to
>>> the aysnc dag loader because it may alleviate your current problem (since
>>> it is code ready).
>>>
>>> It actually reduces the time to 15min, because DAGs are refreshed by the
>>> background process in a streaming way and you don't need to restart
>>> webserver per 20min.
>>>
>>>
>>>
>>> Thanks,
>>> Zhou
>>>
>>>
>>> On Mon, Jul 29, 2019 at 3:14 PM Kevin Yang <yr...@gmail.com> wrote:
>>>
>>>> Hi Zhou,
>>>>
>>>> Thank you for the pointer. This solves the issue gunicorn restart rate
>>>> throttles webserver refresh rate but not the long DAG parsing time issue,
>>>> right? Worst case scenario we still wait 30 mins for the change to show up,
>>>> comparing to the previous 35 mins( I was wrong on the number, it should be
>>>> 35 mins instead of 55 mins as the clock starts whenever the webserver
>>>> restarts). I believe in the previous discussion, we firstly proposed this
>>>> local webserver DAG parsing optimization to use the same DAG parsing logic
>>>> in scheduler to speed up the parsing. Then the stateless webserver proposal
>>>> came up and we were brought in that it is a better idea to persist DAGs
>>>> into the DB and read directly from the DB, better DAG def consistency and
>>>> webserver cluster consistency. I'm all supportive on the proposed structure
>>>> in AIP-24 but -1 on just feed webserver with a single subprocess parsing
>>>> the DAGs. I would image there won't be too many additional work to fetch
>>>> from DB instead of a subprocess, would there?( haven't look into the
>>>> serialization format part but assuming they are the same/similar)
>>>>
>>>> Cheers,
>>>> Kevin Y
>>>>
>>>> On Mon, Jul 29, 2019 at 2:18 PM Zhou Fang <zh...@google.com> wrote:
>>>>
>>>>> Hi Kevin,
>>>>>
>>>>> The problem that DAG parsing takes a long time can be solved by
>>>>> Asynchronous DAG loading: https://github.com/apache/airflow/pull/5594
>>>>>
>>>>> The idea is the a background process parses DAG files, and sends DAGs
>>>>> to webserver process every [webserver] dagbag_sync_interval = 10s.
>>>>>
>>>>> We have launched it in Composer, so our users can set webserver worker
>>>>> restart interval to 1 hour (or longer). The background DAG parsing
>>>>> processing refresh all DAGs per [webserver] = collect_dags_interval = 30s
>>>>> .
>>>>>
>>>>> If parsing all DAGs take 15min, you can see DAGs being gradually
>>>>> freshed with this feature.
>>>>>
>>>>> Thanks,
>>>>> Zhou
>>>>>
>>>>>
>>>>> On Sat, Jul 27, 2019 at 2:43 AM Kevin Yang <yr...@gmail.com> wrote:
>>>>>
>>>>>> Nice job Zhou!
>>>>>>
>>>>>> Really excited, exactly what we wanted for the webserver scaling
>>>>>> issue.
>>>>>> Want to add another big drive for Airbnb to start think about this
>>>>>> previously to support the effort: it can not only bring consistency
>>>>>> between
>>>>>> webservers but also bring consistency between webserver and
>>>>>> scheduler/workers. It may be less of a problem if total DAG parsing
>>>>>> time is
>>>>>> small, but for us the total DAG parsing time is 15+ mins and we had
>>>>>> to set
>>>>>> the webserver( gunicorn subprocesses) restart interval to 20 mins,
>>>>>> which
>>>>>> leads to a worst case 15+20+15=50 mins delay between scheduler start
>>>>>> to
>>>>>> schedule things and users can see their deployed DAGs/changes...
>>>>>>
>>>>>> I'm not so sure about the scheduler performance improvement:
>>>>>> currently we
>>>>>> already feed the main scheduler process with SimpleDag through
>>>>>> DagFileProcessorManager running in a subprocess--in the future we
>>>>>> feed it
>>>>>> with data from DB, which is likely slower( tho the diff should have
>>>>>> negligible impact to the scheduler performance). In fact if we'd keep
>>>>>> the
>>>>>> existing behavior, try schedule only fresh parsed DAGs, then we may
>>>>>> need to
>>>>>> deal with some consistency issue--dag processor and the scheduler
>>>>>> race for
>>>>>> updating the flag indicating if the DAG is newly parsed. No big deal
>>>>>> there
>>>>>> but just some thoughts on the top of my head and hopefully can be
>>>>>> helpful.
>>>>>>
>>>>>> And good idea on pre-rendering the template, believe template
>>>>>> rendering was
>>>>>> the biggest concern in the previous discussion. We've also chose the
>>>>>> pre-rendering+JSON approach in our smart sensor API
>>>>>> <
>>>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization
>>>>>> >
>>>>>> and
>>>>>> seems to be working fine--a supporting case for ur proposal ;)
>>>>>> There's a WIP
>>>>>> PR <https://github.com/apache/airflow/pull/5499> for it just in case
>>>>>> you
>>>>>> are interested--maybe we can even share some logics.
>>>>>>
>>>>>> Thumbs-up again for this and please don't heisitate to reach out if
>>>>>> you
>>>>>> want to discuss further with us or need any help from us.
>>>>>>
>>>>>>
>>>>>> Cheers,
>>>>>> Kevin Y
>>>>>>
>>>>>> On Sat, Jul 27, 2019 at 12:54 AM Driesprong, Fokko
>>>>>> <fo...@driesprong.frl>
>>>>>> wrote:
>>>>>>
>>>>>> > Looks great Zhou,
>>>>>> >
>>>>>> > I have one thing that pops in my mind while reading the AIP; should
>>>>>> keep
>>>>>> > the caching on the webserver level. As the famous quote goes:
>>>>>> *"There are
>>>>>> > only two hard things in Computer Science: cache invalidation and
>>>>>> naming
>>>>>> > things." -- Phil Karlton*
>>>>>> >
>>>>>> > Right now, the fundamental change that is being proposed in the AIP
>>>>>> is
>>>>>> > fetching the DAGs from the database in a serialized format, and not
>>>>>> parsing
>>>>>> > the Python files all the time. This will give already a great
>>>>>> performance
>>>>>> > improvement on the webserver side because it removes a lot of the
>>>>>> > processing. However, since we're still fetching the DAGs from the
>>>>>> database
>>>>>> > in a regular interval, cache it in the local process, so we still
>>>>>> have the
>>>>>> > two issues that Airflow is suffering from right now:
>>>>>> >
>>>>>> > 1. No snappy UI because it is still polling the database in a
>>>>>> regular
>>>>>> > interval.
>>>>>> > 2. Inconsistency between webservers because they might poll in a
>>>>>> > different interval, I think we've all seen this:
>>>>>> > https://www.youtube.com/watch?v=sNrBruPS3r4
>>>>>> >
>>>>>> > As I also mentioned in the Slack channel, I strongly feel that we
>>>>>> should be
>>>>>> > able to render most views from the tables in the database, so
>>>>>> without
>>>>>> > touching the blob. For specific views, we could just pull the blob
>>>>>> from the
>>>>>> > database. In this case we always have the latest version, and we
>>>>>> tackle the
>>>>>> > second point above.
>>>>>> >
>>>>>> > To tackle the first one, I also have an idea. We should change the
>>>>>> DAG
>>>>>> > parser from a loop to something that uses inotify
>>>>>> > https://pypi.org/project/inotify_simple/. This will change it from
>>>>>> polling
>>>>>> > to an event-driven design, which is much more performant and less
>>>>>> resource
>>>>>> > hungry. But this would be an AIP on its own.
>>>>>> >
>>>>>> > Again, great design and a comprehensive AIP, but I would include the
>>>>>> > caching on the webserver to greatly improve the user experience in
>>>>>> the UI.
>>>>>> > Looking forward to the opinion of others on this.
>>>>>> >
>>>>>> > Cheers, Fokko
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > Op za 27 jul. 2019 om 01:44 schreef Zhou Fang
>>>>>> <zhoufang@google.com.invalid
>>>>>> > >:
>>>>>> >
>>>>>> > > Hi Kaxi,
>>>>>> > >
>>>>>> > > Just sent out the AIP:
>>>>>> > >
>>>>>> > >
>>>>>> >
>>>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistence+in+DB+using+JSON+for+Airflow+Webserver+and+%28optional%29+Scheduler
>>>>>> > >
>>>>>> > > Thanks!
>>>>>> > > Zhou
>>>>>> > >
>>>>>> > >
>>>>>> > > On Fri, Jul 26, 2019 at 1:33 PM Zhou Fang <zh...@google.com>
>>>>>> wrote:
>>>>>> > >
>>>>>> > > > Hi Kaxil,
>>>>>> > > >
>>>>>> > > > We are also working on persisting DAGs into DB using JSON for
>>>>>> Airflow
>>>>>> > > > webserver in Google Composer. We target at minimizing the
>>>>>> change to the
>>>>>> > > > current Airflow code. Happy to get synced on this!
>>>>>> > > >
>>>>>> > > > Here is our progress:
>>>>>> > > > (1) Serializing DAGs using Pickle to be used in webserver
>>>>>> > > > It has been launched in Composer. I am working on the PR to
>>>>>> upstream
>>>>>> > it:
>>>>>> > > > https://github.com/apache/airflow/pull/5594
>>>>>> > > > Currently it does not support non-Airflow operators and we are
>>>>>> working
>>>>>> > on
>>>>>> > > > a fix.
>>>>>> > > >
>>>>>> > > > (2) Caching Pickled DAGs in DB to be used by webserver
>>>>>> > > > We have a proof-of-concept implementation, working on an AIP
>>>>>> now.
>>>>>> > > >
>>>>>> > > > (3) Using JSON instead of Pickle in (1) and (2)
>>>>>> > > > Decided to use JSON because Pickle is not secure and human
>>>>>> readable.
>>>>>> > The
>>>>>> > > > serialization approach is very similar to (1).
>>>>>> > > >
>>>>>> > > > I will update the RP (
>>>>>> https://github.com/apache/airflow/pull/5594) to
>>>>>> > > > replace Pickle by JSON, and send our design of (2) as an AIP
>>>>>> next week.
>>>>>> > > > Glad to check together whether our implementation makes sense
>>>>>> and do
>>>>>> > > > improvements on that.
>>>>>> > > >
>>>>>> > > > Thanks!
>>>>>> > > > Zhou
>>>>>> > > >
>>>>>> > > >
>>>>>> > > > On Fri, Jul 26, 2019 at 7:37 AM Kaxil Naik <kaxilnaik@gmail.com
>>>>>> >
>>>>>> > wrote:
>>>>>> > > >
>>>>>> > > >> Hi all,
>>>>>> > > >>
>>>>>> > > >> We, at Astronomer, are going to spend time working on DAG
>>>>>> > Serialisation.
>>>>>> > > >> There are 2 AIPs that are somewhat related to what we plan to
>>>>>> work on:
>>>>>> > > >>
>>>>>> > > >> - AIP-18 Persist all information from DAG file in DB
>>>>>> > > >> <
>>>>>> > > >>
>>>>>> > >
>>>>>> >
>>>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB
>>>>>> > > >> >
>>>>>> > > >> - AIP-19 Making the webserver stateless
>>>>>> > > >> <
>>>>>> > > >>
>>>>>> > >
>>>>>> >
>>>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless
>>>>>> > > >> >
>>>>>> > > >>
>>>>>> > > >> We plan to use JSON as the Serialisation format and store it
>>>>>> as a blob
>>>>>> > > in
>>>>>> > > >> metadata DB.
>>>>>> > > >>
>>>>>> > > >> *Goals:*
>>>>>> > > >>
>>>>>> > > >> - Make Webserver Stateless
>>>>>> > > >> - Use the same version of the DAG across Webserver &
>>>>>> Scheduler
>>>>>> > > >> - Keep backward compatibility and have a flag (globally &
>>>>>> at DAG
>>>>>> > > level)
>>>>>> > > >> to turn this feature on/off
>>>>>> > > >> - Enable DAG Versioning (extended Goal)
>>>>>> > > >>
>>>>>> > > >>
>>>>>> > > >> We will be preparing a proposal (AIP) after some research and
>>>>>> some
>>>>>> > > initial
>>>>>> > > >> work and open it for the suggestions of the community.
>>>>>> > > >>
>>>>>> > > >> We already had some good brain-storming sessions with Twitter
>>>>>> folks
>>>>>> > > (DanD
>>>>>> > > >> &
>>>>>> > > >> Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from
>>>>>> Uber) which
>>>>>> > > >> will
>>>>>> > > >> be a good starting point for us.
>>>>>> > > >>
>>>>>> > > >> If anyone in the community is interested in it or has some
>>>>>> experience
>>>>>> > > >> about
>>>>>> > > >> the same and want to collaborate please let me know and join
>>>>>> > > >> #dag-serialisation channel on Airflow Slack.
>>>>>> > > >>
>>>>>> > > >> Regards,
>>>>>> > > >> Kaxil
>>>>>> > > >>
>>>>>> > > >
>>>>>> > >
>>>>>> >
>>>>>>
>>>>>
Re: Airflow DAG Serialisation
Posted by Kevin Yang <yr...@gmail.com>.
oops, s/consistent file/consistent file order/
On Mon, Jul 29, 2019 at 5:42 PM Kevin Yang <yr...@gmail.com> wrote:
> Hi Zhou,
>
> Totally understood, thank you for that. Streaming logic does cover most
> cases, tho we still have the worst cases where os.walk doesn't give us
> consistent file and file/dir additions/renames causing a different result
> order of list_py_file_paths( e.g. right after we parsed the first dir, it
> was renamed and will be parsed last in the 2nd DAG loading, or we merged a
> new file right after the file paths are collected). Maybe there's a way to
> guarantee the order of parsing but not sure if it worth the effort given
> that it is less of a problem if the end to end parsing time is small
> enough. I understand it may be started as a short term improvement but
> since it should not be too much more complicated, we'd rather start with
> unified long term pattern.
>
> Cheers,
> Kevin Y
>
>
> On Mon, Jul 29, 2019 at 3:59 PM Zhou Fang <zh...@google.com> wrote:
>
>> Hi Kevin,
>>
>> Yes. DAG persistence in DB is definitely the way to go. I referred to the
>> aysnc dag loader because it may alleviate your current problem (since it is
>> code ready).
>>
>> It actually reduces the time to 15min, because DAGs are refreshed by the
>> background process in a streaming way and you don't need to restart
>> webserver per 20min.
>>
>>
>>
>> Thanks,
>> Zhou
>>
>>
>> On Mon, Jul 29, 2019 at 3:14 PM Kevin Yang <yr...@gmail.com> wrote:
>>
>>> Hi Zhou,
>>>
>>> Thank you for the pointer. This solves the issue gunicorn restart rate
>>> throttles webserver refresh rate but not the long DAG parsing time issue,
>>> right? Worst case scenario we still wait 30 mins for the change to show up,
>>> comparing to the previous 35 mins( I was wrong on the number, it should be
>>> 35 mins instead of 55 mins as the clock starts whenever the webserver
>>> restarts). I believe in the previous discussion, we firstly proposed this
>>> local webserver DAG parsing optimization to use the same DAG parsing logic
>>> in scheduler to speed up the parsing. Then the stateless webserver proposal
>>> came up and we were brought in that it is a better idea to persist DAGs
>>> into the DB and read directly from the DB, better DAG def consistency and
>>> webserver cluster consistency. I'm all supportive on the proposed structure
>>> in AIP-24 but -1 on just feed webserver with a single subprocess parsing
>>> the DAGs. I would image there won't be too many additional work to fetch
>>> from DB instead of a subprocess, would there?( haven't look into the
>>> serialization format part but assuming they are the same/similar)
>>>
>>> Cheers,
>>> Kevin Y
>>>
>>> On Mon, Jul 29, 2019 at 2:18 PM Zhou Fang <zh...@google.com> wrote:
>>>
>>>> Hi Kevin,
>>>>
>>>> The problem that DAG parsing takes a long time can be solved by
>>>> Asynchronous DAG loading: https://github.com/apache/airflow/pull/5594
>>>>
>>>> The idea is the a background process parses DAG files, and sends DAGs
>>>> to webserver process every [webserver] dagbag_sync_interval = 10s.
>>>>
>>>> We have launched it in Composer, so our users can set webserver worker
>>>> restart interval to 1 hour (or longer). The background DAG parsing
>>>> processing refresh all DAGs per [webserver] = collect_dags_interval = 30s
>>>> .
>>>>
>>>> If parsing all DAGs take 15min, you can see DAGs being gradually
>>>> freshed with this feature.
>>>>
>>>> Thanks,
>>>> Zhou
>>>>
>>>>
>>>> On Sat, Jul 27, 2019 at 2:43 AM Kevin Yang <yr...@gmail.com> wrote:
>>>>
>>>>> Nice job Zhou!
>>>>>
>>>>> Really excited, exactly what we wanted for the webserver scaling issue.
>>>>> Want to add another big drive for Airbnb to start think about this
>>>>> previously to support the effort: it can not only bring consistency
>>>>> between
>>>>> webservers but also bring consistency between webserver and
>>>>> scheduler/workers. It may be less of a problem if total DAG parsing
>>>>> time is
>>>>> small, but for us the total DAG parsing time is 15+ mins and we had to
>>>>> set
>>>>> the webserver( gunicorn subprocesses) restart interval to 20 mins,
>>>>> which
>>>>> leads to a worst case 15+20+15=50 mins delay between scheduler start to
>>>>> schedule things and users can see their deployed DAGs/changes...
>>>>>
>>>>> I'm not so sure about the scheduler performance improvement: currently
>>>>> we
>>>>> already feed the main scheduler process with SimpleDag through
>>>>> DagFileProcessorManager running in a subprocess--in the future we feed
>>>>> it
>>>>> with data from DB, which is likely slower( tho the diff should have
>>>>> negligible impact to the scheduler performance). In fact if we'd keep
>>>>> the
>>>>> existing behavior, try schedule only fresh parsed DAGs, then we may
>>>>> need to
>>>>> deal with some consistency issue--dag processor and the scheduler race
>>>>> for
>>>>> updating the flag indicating if the DAG is newly parsed. No big deal
>>>>> there
>>>>> but just some thoughts on the top of my head and hopefully can be
>>>>> helpful.
>>>>>
>>>>> And good idea on pre-rendering the template, believe template
>>>>> rendering was
>>>>> the biggest concern in the previous discussion. We've also chose the
>>>>> pre-rendering+JSON approach in our smart sensor API
>>>>> <
>>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization
>>>>> >
>>>>> and
>>>>> seems to be working fine--a supporting case for ur proposal ;) There's
>>>>> a WIP
>>>>> PR <https://github.com/apache/airflow/pull/5499> for it just in case
>>>>> you
>>>>> are interested--maybe we can even share some logics.
>>>>>
>>>>> Thumbs-up again for this and please don't heisitate to reach out if you
>>>>> want to discuss further with us or need any help from us.
>>>>>
>>>>>
>>>>> Cheers,
>>>>> Kevin Y
>>>>>
>>>>> On Sat, Jul 27, 2019 at 12:54 AM Driesprong, Fokko
>>>>> <fo...@driesprong.frl>
>>>>> wrote:
>>>>>
>>>>> > Looks great Zhou,
>>>>> >
>>>>> > I have one thing that pops in my mind while reading the AIP; should
>>>>> keep
>>>>> > the caching on the webserver level. As the famous quote goes:
>>>>> *"There are
>>>>> > only two hard things in Computer Science: cache invalidation and
>>>>> naming
>>>>> > things." -- Phil Karlton*
>>>>> >
>>>>> > Right now, the fundamental change that is being proposed in the AIP
>>>>> is
>>>>> > fetching the DAGs from the database in a serialized format, and not
>>>>> parsing
>>>>> > the Python files all the time. This will give already a great
>>>>> performance
>>>>> > improvement on the webserver side because it removes a lot of the
>>>>> > processing. However, since we're still fetching the DAGs from the
>>>>> database
>>>>> > in a regular interval, cache it in the local process, so we still
>>>>> have the
>>>>> > two issues that Airflow is suffering from right now:
>>>>> >
>>>>> > 1. No snappy UI because it is still polling the database in a
>>>>> regular
>>>>> > interval.
>>>>> > 2. Inconsistency between webservers because they might poll in a
>>>>> > different interval, I think we've all seen this:
>>>>> > https://www.youtube.com/watch?v=sNrBruPS3r4
>>>>> >
>>>>> > As I also mentioned in the Slack channel, I strongly feel that we
>>>>> should be
>>>>> > able to render most views from the tables in the database, so without
>>>>> > touching the blob. For specific views, we could just pull the blob
>>>>> from the
>>>>> > database. In this case we always have the latest version, and we
>>>>> tackle the
>>>>> > second point above.
>>>>> >
>>>>> > To tackle the first one, I also have an idea. We should change the
>>>>> DAG
>>>>> > parser from a loop to something that uses inotify
>>>>> > https://pypi.org/project/inotify_simple/. This will change it from
>>>>> polling
>>>>> > to an event-driven design, which is much more performant and less
>>>>> resource
>>>>> > hungry. But this would be an AIP on its own.
>>>>> >
>>>>> > Again, great design and a comprehensive AIP, but I would include the
>>>>> > caching on the webserver to greatly improve the user experience in
>>>>> the UI.
>>>>> > Looking forward to the opinion of others on this.
>>>>> >
>>>>> > Cheers, Fokko
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > Op za 27 jul. 2019 om 01:44 schreef Zhou Fang
>>>>> <zhoufang@google.com.invalid
>>>>> > >:
>>>>> >
>>>>> > > Hi Kaxi,
>>>>> > >
>>>>> > > Just sent out the AIP:
>>>>> > >
>>>>> > >
>>>>> >
>>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistence+in+DB+using+JSON+for+Airflow+Webserver+and+%28optional%29+Scheduler
>>>>> > >
>>>>> > > Thanks!
>>>>> > > Zhou
>>>>> > >
>>>>> > >
>>>>> > > On Fri, Jul 26, 2019 at 1:33 PM Zhou Fang <zh...@google.com>
>>>>> wrote:
>>>>> > >
>>>>> > > > Hi Kaxil,
>>>>> > > >
>>>>> > > > We are also working on persisting DAGs into DB using JSON for
>>>>> Airflow
>>>>> > > > webserver in Google Composer. We target at minimizing the change
>>>>> to the
>>>>> > > > current Airflow code. Happy to get synced on this!
>>>>> > > >
>>>>> > > > Here is our progress:
>>>>> > > > (1) Serializing DAGs using Pickle to be used in webserver
>>>>> > > > It has been launched in Composer. I am working on the PR to
>>>>> upstream
>>>>> > it:
>>>>> > > > https://github.com/apache/airflow/pull/5594
>>>>> > > > Currently it does not support non-Airflow operators and we are
>>>>> working
>>>>> > on
>>>>> > > > a fix.
>>>>> > > >
>>>>> > > > (2) Caching Pickled DAGs in DB to be used by webserver
>>>>> > > > We have a proof-of-concept implementation, working on an AIP now.
>>>>> > > >
>>>>> > > > (3) Using JSON instead of Pickle in (1) and (2)
>>>>> > > > Decided to use JSON because Pickle is not secure and human
>>>>> readable.
>>>>> > The
>>>>> > > > serialization approach is very similar to (1).
>>>>> > > >
>>>>> > > > I will update the RP (
>>>>> https://github.com/apache/airflow/pull/5594) to
>>>>> > > > replace Pickle by JSON, and send our design of (2) as an AIP
>>>>> next week.
>>>>> > > > Glad to check together whether our implementation makes sense
>>>>> and do
>>>>> > > > improvements on that.
>>>>> > > >
>>>>> > > > Thanks!
>>>>> > > > Zhou
>>>>> > > >
>>>>> > > >
>>>>> > > > On Fri, Jul 26, 2019 at 7:37 AM Kaxil Naik <ka...@gmail.com>
>>>>> > wrote:
>>>>> > > >
>>>>> > > >> Hi all,
>>>>> > > >>
>>>>> > > >> We, at Astronomer, are going to spend time working on DAG
>>>>> > Serialisation.
>>>>> > > >> There are 2 AIPs that are somewhat related to what we plan to
>>>>> work on:
>>>>> > > >>
>>>>> > > >> - AIP-18 Persist all information from DAG file in DB
>>>>> > > >> <
>>>>> > > >>
>>>>> > >
>>>>> >
>>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB
>>>>> > > >> >
>>>>> > > >> - AIP-19 Making the webserver stateless
>>>>> > > >> <
>>>>> > > >>
>>>>> > >
>>>>> >
>>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless
>>>>> > > >> >
>>>>> > > >>
>>>>> > > >> We plan to use JSON as the Serialisation format and store it as
>>>>> a blob
>>>>> > > in
>>>>> > > >> metadata DB.
>>>>> > > >>
>>>>> > > >> *Goals:*
>>>>> > > >>
>>>>> > > >> - Make Webserver Stateless
>>>>> > > >> - Use the same version of the DAG across Webserver &
>>>>> Scheduler
>>>>> > > >> - Keep backward compatibility and have a flag (globally & at
>>>>> DAG
>>>>> > > level)
>>>>> > > >> to turn this feature on/off
>>>>> > > >> - Enable DAG Versioning (extended Goal)
>>>>> > > >>
>>>>> > > >>
>>>>> > > >> We will be preparing a proposal (AIP) after some research and
>>>>> some
>>>>> > > initial
>>>>> > > >> work and open it for the suggestions of the community.
>>>>> > > >>
>>>>> > > >> We already had some good brain-storming sessions with Twitter
>>>>> folks
>>>>> > > (DanD
>>>>> > > >> &
>>>>> > > >> Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from
>>>>> Uber) which
>>>>> > > >> will
>>>>> > > >> be a good starting point for us.
>>>>> > > >>
>>>>> > > >> If anyone in the community is interested in it or has some
>>>>> experience
>>>>> > > >> about
>>>>> > > >> the same and want to collaborate please let me know and join
>>>>> > > >> #dag-serialisation channel on Airflow Slack.
>>>>> > > >>
>>>>> > > >> Regards,
>>>>> > > >> Kaxil
>>>>> > > >>
>>>>> > > >
>>>>> > >
>>>>> >
>>>>>
>>>>
Re: Airflow DAG Serialisation
Posted by Kevin Yang <yr...@gmail.com>.
Hi Zhou,
Totally understood, thank you for that. Streaming logic does cover most
cases, tho we still have the worst cases where os.walk doesn't give us
consistent file and file/dir additions/renames causing a different result
order of list_py_file_paths( e.g. right after we parsed the first dir, it
was renamed and will be parsed last in the 2nd DAG loading, or we merged a
new file right after the file paths are collected). Maybe there's a way to
guarantee the order of parsing but not sure if it worth the effort given
that it is less of a problem if the end to end parsing time is small
enough. I understand it may be started as a short term improvement but
since it should not be too much more complicated, we'd rather start with
unified long term pattern.
Cheers,
Kevin Y
On Mon, Jul 29, 2019 at 3:59 PM Zhou Fang <zh...@google.com> wrote:
> Hi Kevin,
>
> Yes. DAG persistence in DB is definitely the way to go. I referred to the
> aysnc dag loader because it may alleviate your current problem (since it is
> code ready).
>
> It actually reduces the time to 15min, because DAGs are refreshed by the
> background process in a streaming way and you don't need to restart
> webserver per 20min.
>
>
>
> Thanks,
> Zhou
>
>
> On Mon, Jul 29, 2019 at 3:14 PM Kevin Yang <yr...@gmail.com> wrote:
>
>> Hi Zhou,
>>
>> Thank you for the pointer. This solves the issue gunicorn restart rate
>> throttles webserver refresh rate but not the long DAG parsing time issue,
>> right? Worst case scenario we still wait 30 mins for the change to show up,
>> comparing to the previous 35 mins( I was wrong on the number, it should be
>> 35 mins instead of 55 mins as the clock starts whenever the webserver
>> restarts). I believe in the previous discussion, we firstly proposed this
>> local webserver DAG parsing optimization to use the same DAG parsing logic
>> in scheduler to speed up the parsing. Then the stateless webserver proposal
>> came up and we were brought in that it is a better idea to persist DAGs
>> into the DB and read directly from the DB, better DAG def consistency and
>> webserver cluster consistency. I'm all supportive on the proposed structure
>> in AIP-24 but -1 on just feed webserver with a single subprocess parsing
>> the DAGs. I would image there won't be too many additional work to fetch
>> from DB instead of a subprocess, would there?( haven't look into the
>> serialization format part but assuming they are the same/similar)
>>
>> Cheers,
>> Kevin Y
>>
>> On Mon, Jul 29, 2019 at 2:18 PM Zhou Fang <zh...@google.com> wrote:
>>
>>> Hi Kevin,
>>>
>>> The problem that DAG parsing takes a long time can be solved by
>>> Asynchronous DAG loading: https://github.com/apache/airflow/pull/5594
>>>
>>> The idea is the a background process parses DAG files, and sends DAGs to
>>> webserver process every [webserver] dagbag_sync_interval = 10s.
>>>
>>> We have launched it in Composer, so our users can set webserver worker
>>> restart interval to 1 hour (or longer). The background DAG parsing
>>> processing refresh all DAGs per [webserver] = collect_dags_interval = 30s
>>> .
>>>
>>> If parsing all DAGs take 15min, you can see DAGs being gradually freshed
>>> with this feature.
>>>
>>> Thanks,
>>> Zhou
>>>
>>>
>>> On Sat, Jul 27, 2019 at 2:43 AM Kevin Yang <yr...@gmail.com> wrote:
>>>
>>>> Nice job Zhou!
>>>>
>>>> Really excited, exactly what we wanted for the webserver scaling issue.
>>>> Want to add another big drive for Airbnb to start think about this
>>>> previously to support the effort: it can not only bring consistency
>>>> between
>>>> webservers but also bring consistency between webserver and
>>>> scheduler/workers. It may be less of a problem if total DAG parsing
>>>> time is
>>>> small, but for us the total DAG parsing time is 15+ mins and we had to
>>>> set
>>>> the webserver( gunicorn subprocesses) restart interval to 20 mins, which
>>>> leads to a worst case 15+20+15=50 mins delay between scheduler start to
>>>> schedule things and users can see their deployed DAGs/changes...
>>>>
>>>> I'm not so sure about the scheduler performance improvement: currently
>>>> we
>>>> already feed the main scheduler process with SimpleDag through
>>>> DagFileProcessorManager running in a subprocess--in the future we feed
>>>> it
>>>> with data from DB, which is likely slower( tho the diff should have
>>>> negligible impact to the scheduler performance). In fact if we'd keep
>>>> the
>>>> existing behavior, try schedule only fresh parsed DAGs, then we may
>>>> need to
>>>> deal with some consistency issue--dag processor and the scheduler race
>>>> for
>>>> updating the flag indicating if the DAG is newly parsed. No big deal
>>>> there
>>>> but just some thoughts on the top of my head and hopefully can be
>>>> helpful.
>>>>
>>>> And good idea on pre-rendering the template, believe template rendering
>>>> was
>>>> the biggest concern in the previous discussion. We've also chose the
>>>> pre-rendering+JSON approach in our smart sensor API
>>>> <
>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization
>>>> >
>>>> and
>>>> seems to be working fine--a supporting case for ur proposal ;) There's
>>>> a WIP
>>>> PR <https://github.com/apache/airflow/pull/5499> for it just in case
>>>> you
>>>> are interested--maybe we can even share some logics.
>>>>
>>>> Thumbs-up again for this and please don't heisitate to reach out if you
>>>> want to discuss further with us or need any help from us.
>>>>
>>>>
>>>> Cheers,
>>>> Kevin Y
>>>>
>>>> On Sat, Jul 27, 2019 at 12:54 AM Driesprong, Fokko <fokko@driesprong.frl
>>>> >
>>>> wrote:
>>>>
>>>> > Looks great Zhou,
>>>> >
>>>> > I have one thing that pops in my mind while reading the AIP; should
>>>> keep
>>>> > the caching on the webserver level. As the famous quote goes: *"There
>>>> are
>>>> > only two hard things in Computer Science: cache invalidation and
>>>> naming
>>>> > things." -- Phil Karlton*
>>>> >
>>>> > Right now, the fundamental change that is being proposed in the AIP is
>>>> > fetching the DAGs from the database in a serialized format, and not
>>>> parsing
>>>> > the Python files all the time. This will give already a great
>>>> performance
>>>> > improvement on the webserver side because it removes a lot of the
>>>> > processing. However, since we're still fetching the DAGs from the
>>>> database
>>>> > in a regular interval, cache it in the local process, so we still
>>>> have the
>>>> > two issues that Airflow is suffering from right now:
>>>> >
>>>> > 1. No snappy UI because it is still polling the database in a
>>>> regular
>>>> > interval.
>>>> > 2. Inconsistency between webservers because they might poll in a
>>>> > different interval, I think we've all seen this:
>>>> > https://www.youtube.com/watch?v=sNrBruPS3r4
>>>> >
>>>> > As I also mentioned in the Slack channel, I strongly feel that we
>>>> should be
>>>> > able to render most views from the tables in the database, so without
>>>> > touching the blob. For specific views, we could just pull the blob
>>>> from the
>>>> > database. In this case we always have the latest version, and we
>>>> tackle the
>>>> > second point above.
>>>> >
>>>> > To tackle the first one, I also have an idea. We should change the DAG
>>>> > parser from a loop to something that uses inotify
>>>> > https://pypi.org/project/inotify_simple/. This will change it from
>>>> polling
>>>> > to an event-driven design, which is much more performant and less
>>>> resource
>>>> > hungry. But this would be an AIP on its own.
>>>> >
>>>> > Again, great design and a comprehensive AIP, but I would include the
>>>> > caching on the webserver to greatly improve the user experience in
>>>> the UI.
>>>> > Looking forward to the opinion of others on this.
>>>> >
>>>> > Cheers, Fokko
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > Op za 27 jul. 2019 om 01:44 schreef Zhou Fang
>>>> <zhoufang@google.com.invalid
>>>> > >:
>>>> >
>>>> > > Hi Kaxi,
>>>> > >
>>>> > > Just sent out the AIP:
>>>> > >
>>>> > >
>>>> >
>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistence+in+DB+using+JSON+for+Airflow+Webserver+and+%28optional%29+Scheduler
>>>> > >
>>>> > > Thanks!
>>>> > > Zhou
>>>> > >
>>>> > >
>>>> > > On Fri, Jul 26, 2019 at 1:33 PM Zhou Fang <zh...@google.com>
>>>> wrote:
>>>> > >
>>>> > > > Hi Kaxil,
>>>> > > >
>>>> > > > We are also working on persisting DAGs into DB using JSON for
>>>> Airflow
>>>> > > > webserver in Google Composer. We target at minimizing the change
>>>> to the
>>>> > > > current Airflow code. Happy to get synced on this!
>>>> > > >
>>>> > > > Here is our progress:
>>>> > > > (1) Serializing DAGs using Pickle to be used in webserver
>>>> > > > It has been launched in Composer. I am working on the PR to
>>>> upstream
>>>> > it:
>>>> > > > https://github.com/apache/airflow/pull/5594
>>>> > > > Currently it does not support non-Airflow operators and we are
>>>> working
>>>> > on
>>>> > > > a fix.
>>>> > > >
>>>> > > > (2) Caching Pickled DAGs in DB to be used by webserver
>>>> > > > We have a proof-of-concept implementation, working on an AIP now.
>>>> > > >
>>>> > > > (3) Using JSON instead of Pickle in (1) and (2)
>>>> > > > Decided to use JSON because Pickle is not secure and human
>>>> readable.
>>>> > The
>>>> > > > serialization approach is very similar to (1).
>>>> > > >
>>>> > > > I will update the RP (https://github.com/apache/airflow/pull/5594)
>>>> to
>>>> > > > replace Pickle by JSON, and send our design of (2) as an AIP next
>>>> week.
>>>> > > > Glad to check together whether our implementation makes sense and
>>>> do
>>>> > > > improvements on that.
>>>> > > >
>>>> > > > Thanks!
>>>> > > > Zhou
>>>> > > >
>>>> > > >
>>>> > > > On Fri, Jul 26, 2019 at 7:37 AM Kaxil Naik <ka...@gmail.com>
>>>> > wrote:
>>>> > > >
>>>> > > >> Hi all,
>>>> > > >>
>>>> > > >> We, at Astronomer, are going to spend time working on DAG
>>>> > Serialisation.
>>>> > > >> There are 2 AIPs that are somewhat related to what we plan to
>>>> work on:
>>>> > > >>
>>>> > > >> - AIP-18 Persist all information from DAG file in DB
>>>> > > >> <
>>>> > > >>
>>>> > >
>>>> >
>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB
>>>> > > >> >
>>>> > > >> - AIP-19 Making the webserver stateless
>>>> > > >> <
>>>> > > >>
>>>> > >
>>>> >
>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless
>>>> > > >> >
>>>> > > >>
>>>> > > >> We plan to use JSON as the Serialisation format and store it as
>>>> a blob
>>>> > > in
>>>> > > >> metadata DB.
>>>> > > >>
>>>> > > >> *Goals:*
>>>> > > >>
>>>> > > >> - Make Webserver Stateless
>>>> > > >> - Use the same version of the DAG across Webserver & Scheduler
>>>> > > >> - Keep backward compatibility and have a flag (globally & at
>>>> DAG
>>>> > > level)
>>>> > > >> to turn this feature on/off
>>>> > > >> - Enable DAG Versioning (extended Goal)
>>>> > > >>
>>>> > > >>
>>>> > > >> We will be preparing a proposal (AIP) after some research and
>>>> some
>>>> > > initial
>>>> > > >> work and open it for the suggestions of the community.
>>>> > > >>
>>>> > > >> We already had some good brain-storming sessions with Twitter
>>>> folks
>>>> > > (DanD
>>>> > > >> &
>>>> > > >> Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from Uber)
>>>> which
>>>> > > >> will
>>>> > > >> be a good starting point for us.
>>>> > > >>
>>>> > > >> If anyone in the community is interested in it or has some
>>>> experience
>>>> > > >> about
>>>> > > >> the same and want to collaborate please let me know and join
>>>> > > >> #dag-serialisation channel on Airflow Slack.
>>>> > > >>
>>>> > > >> Regards,
>>>> > > >> Kaxil
>>>> > > >>
>>>> > > >
>>>> > >
>>>> >
>>>>
>>>
Re: Airflow DAG Serialisation
Posted by Zhou Fang <zh...@google.com.INVALID>.
Hi Kevin,
Yes. DAG persistence in DB is definitely the way to go. I referred to the
aysnc dag loader because it may alleviate your current problem (since it is
code ready).
It actually reduces the time to 15min, because DAGs are refreshed by the
background process in a streaming way and you don't need to restart
webserver per 20min.
Thanks,
Zhou
On Mon, Jul 29, 2019 at 3:14 PM Kevin Yang <yr...@gmail.com> wrote:
> Hi Zhou,
>
> Thank you for the pointer. This solves the issue gunicorn restart rate
> throttles webserver refresh rate but not the long DAG parsing time issue,
> right? Worst case scenario we still wait 30 mins for the change to show up,
> comparing to the previous 35 mins( I was wrong on the number, it should be
> 35 mins instead of 55 mins as the clock starts whenever the webserver
> restarts). I believe in the previous discussion, we firstly proposed this
> local webserver DAG parsing optimization to use the same DAG parsing logic
> in scheduler to speed up the parsing. Then the stateless webserver proposal
> came up and we were brought in that it is a better idea to persist DAGs
> into the DB and read directly from the DB, better DAG def consistency and
> webserver cluster consistency. I'm all supportive on the proposed structure
> in AIP-24 but -1 on just feed webserver with a single subprocess parsing
> the DAGs. I would image there won't be too many additional work to fetch
> from DB instead of a subprocess, would there?( haven't look into the
> serialization format part but assuming they are the same/similar)
>
> Cheers,
> Kevin Y
>
> On Mon, Jul 29, 2019 at 2:18 PM Zhou Fang <zh...@google.com> wrote:
>
>> Hi Kevin,
>>
>> The problem that DAG parsing takes a long time can be solved by
>> Asynchronous DAG loading: https://github.com/apache/airflow/pull/5594
>>
>> The idea is the a background process parses DAG files, and sends DAGs to
>> webserver process every [webserver] dagbag_sync_interval = 10s.
>>
>> We have launched it in Composer, so our users can set webserver worker
>> restart interval to 1 hour (or longer). The background DAG parsing
>> processing refresh all DAGs per [webserver] = collect_dags_interval = 30s
>> .
>>
>> If parsing all DAGs take 15min, you can see DAGs being gradually freshed
>> with this feature.
>>
>> Thanks,
>> Zhou
>>
>>
>> On Sat, Jul 27, 2019 at 2:43 AM Kevin Yang <yr...@gmail.com> wrote:
>>
>>> Nice job Zhou!
>>>
>>> Really excited, exactly what we wanted for the webserver scaling issue.
>>> Want to add another big drive for Airbnb to start think about this
>>> previously to support the effort: it can not only bring consistency
>>> between
>>> webservers but also bring consistency between webserver and
>>> scheduler/workers. It may be less of a problem if total DAG parsing time
>>> is
>>> small, but for us the total DAG parsing time is 15+ mins and we had to
>>> set
>>> the webserver( gunicorn subprocesses) restart interval to 20 mins, which
>>> leads to a worst case 15+20+15=50 mins delay between scheduler start to
>>> schedule things and users can see their deployed DAGs/changes...
>>>
>>> I'm not so sure about the scheduler performance improvement: currently we
>>> already feed the main scheduler process with SimpleDag through
>>> DagFileProcessorManager running in a subprocess--in the future we feed it
>>> with data from DB, which is likely slower( tho the diff should have
>>> negligible impact to the scheduler performance). In fact if we'd keep the
>>> existing behavior, try schedule only fresh parsed DAGs, then we may need
>>> to
>>> deal with some consistency issue--dag processor and the scheduler race
>>> for
>>> updating the flag indicating if the DAG is newly parsed. No big deal
>>> there
>>> but just some thoughts on the top of my head and hopefully can be
>>> helpful.
>>>
>>> And good idea on pre-rendering the template, believe template rendering
>>> was
>>> the biggest concern in the previous discussion. We've also chose the
>>> pre-rendering+JSON approach in our smart sensor API
>>> <
>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization
>>> >
>>> and
>>> seems to be working fine--a supporting case for ur proposal ;) There's a
>>> WIP
>>> PR <https://github.com/apache/airflow/pull/5499> for it just in case you
>>> are interested--maybe we can even share some logics.
>>>
>>> Thumbs-up again for this and please don't heisitate to reach out if you
>>> want to discuss further with us or need any help from us.
>>>
>>>
>>> Cheers,
>>> Kevin Y
>>>
>>> On Sat, Jul 27, 2019 at 12:54 AM Driesprong, Fokko <fokko@driesprong.frl
>>> >
>>> wrote:
>>>
>>> > Looks great Zhou,
>>> >
>>> > I have one thing that pops in my mind while reading the AIP; should
>>> keep
>>> > the caching on the webserver level. As the famous quote goes: *"There
>>> are
>>> > only two hard things in Computer Science: cache invalidation and naming
>>> > things." -- Phil Karlton*
>>> >
>>> > Right now, the fundamental change that is being proposed in the AIP is
>>> > fetching the DAGs from the database in a serialized format, and not
>>> parsing
>>> > the Python files all the time. This will give already a great
>>> performance
>>> > improvement on the webserver side because it removes a lot of the
>>> > processing. However, since we're still fetching the DAGs from the
>>> database
>>> > in a regular interval, cache it in the local process, so we still have
>>> the
>>> > two issues that Airflow is suffering from right now:
>>> >
>>> > 1. No snappy UI because it is still polling the database in a
>>> regular
>>> > interval.
>>> > 2. Inconsistency between webservers because they might poll in a
>>> > different interval, I think we've all seen this:
>>> > https://www.youtube.com/watch?v=sNrBruPS3r4
>>> >
>>> > As I also mentioned in the Slack channel, I strongly feel that we
>>> should be
>>> > able to render most views from the tables in the database, so without
>>> > touching the blob. For specific views, we could just pull the blob
>>> from the
>>> > database. In this case we always have the latest version, and we
>>> tackle the
>>> > second point above.
>>> >
>>> > To tackle the first one, I also have an idea. We should change the DAG
>>> > parser from a loop to something that uses inotify
>>> > https://pypi.org/project/inotify_simple/. This will change it from
>>> polling
>>> > to an event-driven design, which is much more performant and less
>>> resource
>>> > hungry. But this would be an AIP on its own.
>>> >
>>> > Again, great design and a comprehensive AIP, but I would include the
>>> > caching on the webserver to greatly improve the user experience in the
>>> UI.
>>> > Looking forward to the opinion of others on this.
>>> >
>>> > Cheers, Fokko
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > Op za 27 jul. 2019 om 01:44 schreef Zhou Fang
>>> <zhoufang@google.com.invalid
>>> > >:
>>> >
>>> > > Hi Kaxi,
>>> > >
>>> > > Just sent out the AIP:
>>> > >
>>> > >
>>> >
>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistence+in+DB+using+JSON+for+Airflow+Webserver+and+%28optional%29+Scheduler
>>> > >
>>> > > Thanks!
>>> > > Zhou
>>> > >
>>> > >
>>> > > On Fri, Jul 26, 2019 at 1:33 PM Zhou Fang <zh...@google.com>
>>> wrote:
>>> > >
>>> > > > Hi Kaxil,
>>> > > >
>>> > > > We are also working on persisting DAGs into DB using JSON for
>>> Airflow
>>> > > > webserver in Google Composer. We target at minimizing the change
>>> to the
>>> > > > current Airflow code. Happy to get synced on this!
>>> > > >
>>> > > > Here is our progress:
>>> > > > (1) Serializing DAGs using Pickle to be used in webserver
>>> > > > It has been launched in Composer. I am working on the PR to
>>> upstream
>>> > it:
>>> > > > https://github.com/apache/airflow/pull/5594
>>> > > > Currently it does not support non-Airflow operators and we are
>>> working
>>> > on
>>> > > > a fix.
>>> > > >
>>> > > > (2) Caching Pickled DAGs in DB to be used by webserver
>>> > > > We have a proof-of-concept implementation, working on an AIP now.
>>> > > >
>>> > > > (3) Using JSON instead of Pickle in (1) and (2)
>>> > > > Decided to use JSON because Pickle is not secure and human
>>> readable.
>>> > The
>>> > > > serialization approach is very similar to (1).
>>> > > >
>>> > > > I will update the RP (https://github.com/apache/airflow/pull/5594)
>>> to
>>> > > > replace Pickle by JSON, and send our design of (2) as an AIP next
>>> week.
>>> > > > Glad to check together whether our implementation makes sense and
>>> do
>>> > > > improvements on that.
>>> > > >
>>> > > > Thanks!
>>> > > > Zhou
>>> > > >
>>> > > >
>>> > > > On Fri, Jul 26, 2019 at 7:37 AM Kaxil Naik <ka...@gmail.com>
>>> > wrote:
>>> > > >
>>> > > >> Hi all,
>>> > > >>
>>> > > >> We, at Astronomer, are going to spend time working on DAG
>>> > Serialisation.
>>> > > >> There are 2 AIPs that are somewhat related to what we plan to
>>> work on:
>>> > > >>
>>> > > >> - AIP-18 Persist all information from DAG file in DB
>>> > > >> <
>>> > > >>
>>> > >
>>> >
>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB
>>> > > >> >
>>> > > >> - AIP-19 Making the webserver stateless
>>> > > >> <
>>> > > >>
>>> > >
>>> >
>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless
>>> > > >> >
>>> > > >>
>>> > > >> We plan to use JSON as the Serialisation format and store it as a
>>> blob
>>> > > in
>>> > > >> metadata DB.
>>> > > >>
>>> > > >> *Goals:*
>>> > > >>
>>> > > >> - Make Webserver Stateless
>>> > > >> - Use the same version of the DAG across Webserver & Scheduler
>>> > > >> - Keep backward compatibility and have a flag (globally & at
>>> DAG
>>> > > level)
>>> > > >> to turn this feature on/off
>>> > > >> - Enable DAG Versioning (extended Goal)
>>> > > >>
>>> > > >>
>>> > > >> We will be preparing a proposal (AIP) after some research and some
>>> > > initial
>>> > > >> work and open it for the suggestions of the community.
>>> > > >>
>>> > > >> We already had some good brain-storming sessions with Twitter
>>> folks
>>> > > (DanD
>>> > > >> &
>>> > > >> Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from Uber)
>>> which
>>> > > >> will
>>> > > >> be a good starting point for us.
>>> > > >>
>>> > > >> If anyone in the community is interested in it or has some
>>> experience
>>> > > >> about
>>> > > >> the same and want to collaborate please let me know and join
>>> > > >> #dag-serialisation channel on Airflow Slack.
>>> > > >>
>>> > > >> Regards,
>>> > > >> Kaxil
>>> > > >>
>>> > > >
>>> > >
>>> >
>>>
>>
Re: Airflow DAG Serialisation
Posted by Kevin Yang <yr...@gmail.com>.
Hi Zhou,
Thank you for the pointer. This solves the issue gunicorn restart rate
throttles webserver refresh rate but not the long DAG parsing time issue,
right? Worst case scenario we still wait 30 mins for the change to show up,
comparing to the previous 35 mins( I was wrong on the number, it should be
35 mins instead of 55 mins as the clock starts whenever the webserver
restarts). I believe in the previous discussion, we firstly proposed this
local webserver DAG parsing optimization to use the same DAG parsing logic
in scheduler to speed up the parsing. Then the stateless webserver proposal
came up and we were brought in that it is a better idea to persist DAGs
into the DB and read directly from the DB, better DAG def consistency and
webserver cluster consistency. I'm all supportive on the proposed structure
in AIP-24 but -1 on just feed webserver with a single subprocess parsing
the DAGs. I would image there won't be too many additional work to fetch
from DB instead of a subprocess, would there?( haven't look into the
serialization format part but assuming they are the same/similar)
Cheers,
Kevin Y
On Mon, Jul 29, 2019 at 2:18 PM Zhou Fang <zh...@google.com> wrote:
> Hi Kevin,
>
> The problem that DAG parsing takes a long time can be solved by
> Asynchronous DAG loading: https://github.com/apache/airflow/pull/5594
>
> The idea is the a background process parses DAG files, and sends DAGs to
> webserver process every [webserver] dagbag_sync_interval = 10s.
>
> We have launched it in Composer, so our users can set webserver worker
> restart interval to 1 hour (or longer). The background DAG parsing
> processing refresh all DAGs per [webserver] = collect_dags_interval = 30s.
>
> If parsing all DAGs take 15min, you can see DAGs being gradually freshed
> with this feature.
>
> Thanks,
> Zhou
>
>
> On Sat, Jul 27, 2019 at 2:43 AM Kevin Yang <yr...@gmail.com> wrote:
>
>> Nice job Zhou!
>>
>> Really excited, exactly what we wanted for the webserver scaling issue.
>> Want to add another big drive for Airbnb to start think about this
>> previously to support the effort: it can not only bring consistency
>> between
>> webservers but also bring consistency between webserver and
>> scheduler/workers. It may be less of a problem if total DAG parsing time
>> is
>> small, but for us the total DAG parsing time is 15+ mins and we had to set
>> the webserver( gunicorn subprocesses) restart interval to 20 mins, which
>> leads to a worst case 15+20+15=50 mins delay between scheduler start to
>> schedule things and users can see their deployed DAGs/changes...
>>
>> I'm not so sure about the scheduler performance improvement: currently we
>> already feed the main scheduler process with SimpleDag through
>> DagFileProcessorManager running in a subprocess--in the future we feed it
>> with data from DB, which is likely slower( tho the diff should have
>> negligible impact to the scheduler performance). In fact if we'd keep the
>> existing behavior, try schedule only fresh parsed DAGs, then we may need
>> to
>> deal with some consistency issue--dag processor and the scheduler race for
>> updating the flag indicating if the DAG is newly parsed. No big deal there
>> but just some thoughts on the top of my head and hopefully can be helpful.
>>
>> And good idea on pre-rendering the template, believe template rendering
>> was
>> the biggest concern in the previous discussion. We've also chose the
>> pre-rendering+JSON approach in our smart sensor API
>> <
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization
>> >
>> and
>> seems to be working fine--a supporting case for ur proposal ;) There's a
>> WIP
>> PR <https://github.com/apache/airflow/pull/5499> for it just in case you
>> are interested--maybe we can even share some logics.
>>
>> Thumbs-up again for this and please don't heisitate to reach out if you
>> want to discuss further with us or need any help from us.
>>
>>
>> Cheers,
>> Kevin Y
>>
>> On Sat, Jul 27, 2019 at 12:54 AM Driesprong, Fokko <fo...@driesprong.frl>
>> wrote:
>>
>> > Looks great Zhou,
>> >
>> > I have one thing that pops in my mind while reading the AIP; should keep
>> > the caching on the webserver level. As the famous quote goes: *"There
>> are
>> > only two hard things in Computer Science: cache invalidation and naming
>> > things." -- Phil Karlton*
>> >
>> > Right now, the fundamental change that is being proposed in the AIP is
>> > fetching the DAGs from the database in a serialized format, and not
>> parsing
>> > the Python files all the time. This will give already a great
>> performance
>> > improvement on the webserver side because it removes a lot of the
>> > processing. However, since we're still fetching the DAGs from the
>> database
>> > in a regular interval, cache it in the local process, so we still have
>> the
>> > two issues that Airflow is suffering from right now:
>> >
>> > 1. No snappy UI because it is still polling the database in a regular
>> > interval.
>> > 2. Inconsistency between webservers because they might poll in a
>> > different interval, I think we've all seen this:
>> > https://www.youtube.com/watch?v=sNrBruPS3r4
>> >
>> > As I also mentioned in the Slack channel, I strongly feel that we
>> should be
>> > able to render most views from the tables in the database, so without
>> > touching the blob. For specific views, we could just pull the blob from
>> the
>> > database. In this case we always have the latest version, and we tackle
>> the
>> > second point above.
>> >
>> > To tackle the first one, I also have an idea. We should change the DAG
>> > parser from a loop to something that uses inotify
>> > https://pypi.org/project/inotify_simple/. This will change it from
>> polling
>> > to an event-driven design, which is much more performant and less
>> resource
>> > hungry. But this would be an AIP on its own.
>> >
>> > Again, great design and a comprehensive AIP, but I would include the
>> > caching on the webserver to greatly improve the user experience in the
>> UI.
>> > Looking forward to the opinion of others on this.
>> >
>> > Cheers, Fokko
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > Op za 27 jul. 2019 om 01:44 schreef Zhou Fang
>> <zhoufang@google.com.invalid
>> > >:
>> >
>> > > Hi Kaxi,
>> > >
>> > > Just sent out the AIP:
>> > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistence+in+DB+using+JSON+for+Airflow+Webserver+and+%28optional%29+Scheduler
>> > >
>> > > Thanks!
>> > > Zhou
>> > >
>> > >
>> > > On Fri, Jul 26, 2019 at 1:33 PM Zhou Fang <zh...@google.com>
>> wrote:
>> > >
>> > > > Hi Kaxil,
>> > > >
>> > > > We are also working on persisting DAGs into DB using JSON for
>> Airflow
>> > > > webserver in Google Composer. We target at minimizing the change to
>> the
>> > > > current Airflow code. Happy to get synced on this!
>> > > >
>> > > > Here is our progress:
>> > > > (1) Serializing DAGs using Pickle to be used in webserver
>> > > > It has been launched in Composer. I am working on the PR to upstream
>> > it:
>> > > > https://github.com/apache/airflow/pull/5594
>> > > > Currently it does not support non-Airflow operators and we are
>> working
>> > on
>> > > > a fix.
>> > > >
>> > > > (2) Caching Pickled DAGs in DB to be used by webserver
>> > > > We have a proof-of-concept implementation, working on an AIP now.
>> > > >
>> > > > (3) Using JSON instead of Pickle in (1) and (2)
>> > > > Decided to use JSON because Pickle is not secure and human readable.
>> > The
>> > > > serialization approach is very similar to (1).
>> > > >
>> > > > I will update the RP (https://github.com/apache/airflow/pull/5594)
>> to
>> > > > replace Pickle by JSON, and send our design of (2) as an AIP next
>> week.
>> > > > Glad to check together whether our implementation makes sense and do
>> > > > improvements on that.
>> > > >
>> > > > Thanks!
>> > > > Zhou
>> > > >
>> > > >
>> > > > On Fri, Jul 26, 2019 at 7:37 AM Kaxil Naik <ka...@gmail.com>
>> > wrote:
>> > > >
>> > > >> Hi all,
>> > > >>
>> > > >> We, at Astronomer, are going to spend time working on DAG
>> > Serialisation.
>> > > >> There are 2 AIPs that are somewhat related to what we plan to work
>> on:
>> > > >>
>> > > >> - AIP-18 Persist all information from DAG file in DB
>> > > >> <
>> > > >>
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB
>> > > >> >
>> > > >> - AIP-19 Making the webserver stateless
>> > > >> <
>> > > >>
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless
>> > > >> >
>> > > >>
>> > > >> We plan to use JSON as the Serialisation format and store it as a
>> blob
>> > > in
>> > > >> metadata DB.
>> > > >>
>> > > >> *Goals:*
>> > > >>
>> > > >> - Make Webserver Stateless
>> > > >> - Use the same version of the DAG across Webserver & Scheduler
>> > > >> - Keep backward compatibility and have a flag (globally & at DAG
>> > > level)
>> > > >> to turn this feature on/off
>> > > >> - Enable DAG Versioning (extended Goal)
>> > > >>
>> > > >>
>> > > >> We will be preparing a proposal (AIP) after some research and some
>> > > initial
>> > > >> work and open it for the suggestions of the community.
>> > > >>
>> > > >> We already had some good brain-storming sessions with Twitter folks
>> > > (DanD
>> > > >> &
>> > > >> Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from Uber)
>> which
>> > > >> will
>> > > >> be a good starting point for us.
>> > > >>
>> > > >> If anyone in the community is interested in it or has some
>> experience
>> > > >> about
>> > > >> the same and want to collaborate please let me know and join
>> > > >> #dag-serialisation channel on Airflow Slack.
>> > > >>
>> > > >> Regards,
>> > > >> Kaxil
>> > > >>
>> > > >
>> > >
>> >
>>
>
Re: Airflow DAG Serialisation
Posted by Zhou Fang <zh...@google.com.INVALID>.
Hi Kevin,
The problem that DAG parsing takes a long time can be solved by
Asynchronous DAG loading: https://github.com/apache/airflow/pull/5594
The idea is the a background process parses DAG files, and sends DAGs to
webserver process every [webserver] dagbag_sync_interval = 10s.
We have launched it in Composer, so our users can set webserver worker
restart interval to 1 hour (or longer). The background DAG parsing
processing refresh all DAGs per [webserver] = collect_dags_interval = 30s.
If parsing all DAGs take 15min, you can see DAGs being gradually freshed
with this feature.
Thanks,
Zhou
On Sat, Jul 27, 2019 at 2:43 AM Kevin Yang <yr...@gmail.com> wrote:
> Nice job Zhou!
>
> Really excited, exactly what we wanted for the webserver scaling issue.
> Want to add another big drive for Airbnb to start think about this
> previously to support the effort: it can not only bring consistency between
> webservers but also bring consistency between webserver and
> scheduler/workers. It may be less of a problem if total DAG parsing time is
> small, but for us the total DAG parsing time is 15+ mins and we had to set
> the webserver( gunicorn subprocesses) restart interval to 20 mins, which
> leads to a worst case 15+20+15=50 mins delay between scheduler start to
> schedule things and users can see their deployed DAGs/changes...
>
> I'm not so sure about the scheduler performance improvement: currently we
> already feed the main scheduler process with SimpleDag through
> DagFileProcessorManager running in a subprocess--in the future we feed it
> with data from DB, which is likely slower( tho the diff should have
> negligible impact to the scheduler performance). In fact if we'd keep the
> existing behavior, try schedule only fresh parsed DAGs, then we may need to
> deal with some consistency issue--dag processor and the scheduler race for
> updating the flag indicating if the DAG is newly parsed. No big deal there
> but just some thoughts on the top of my head and hopefully can be helpful.
>
> And good idea on pre-rendering the template, believe template rendering was
> the biggest concern in the previous discussion. We've also chose the
> pre-rendering+JSON approach in our smart sensor API
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization
> >
> and
> seems to be working fine--a supporting case for ur proposal ;) There's a
> WIP
> PR <https://github.com/apache/airflow/pull/5499> for it just in case you
> are interested--maybe we can even share some logics.
>
> Thumbs-up again for this and please don't heisitate to reach out if you
> want to discuss further with us or need any help from us.
>
>
> Cheers,
> Kevin Y
>
> On Sat, Jul 27, 2019 at 12:54 AM Driesprong, Fokko <fo...@driesprong.frl>
> wrote:
>
> > Looks great Zhou,
> >
> > I have one thing that pops in my mind while reading the AIP; should keep
> > the caching on the webserver level. As the famous quote goes: *"There are
> > only two hard things in Computer Science: cache invalidation and naming
> > things." -- Phil Karlton*
> >
> > Right now, the fundamental change that is being proposed in the AIP is
> > fetching the DAGs from the database in a serialized format, and not
> parsing
> > the Python files all the time. This will give already a great performance
> > improvement on the webserver side because it removes a lot of the
> > processing. However, since we're still fetching the DAGs from the
> database
> > in a regular interval, cache it in the local process, so we still have
> the
> > two issues that Airflow is suffering from right now:
> >
> > 1. No snappy UI because it is still polling the database in a regular
> > interval.
> > 2. Inconsistency between webservers because they might poll in a
> > different interval, I think we've all seen this:
> > https://www.youtube.com/watch?v=sNrBruPS3r4
> >
> > As I also mentioned in the Slack channel, I strongly feel that we should
> be
> > able to render most views from the tables in the database, so without
> > touching the blob. For specific views, we could just pull the blob from
> the
> > database. In this case we always have the latest version, and we tackle
> the
> > second point above.
> >
> > To tackle the first one, I also have an idea. We should change the DAG
> > parser from a loop to something that uses inotify
> > https://pypi.org/project/inotify_simple/. This will change it from
> polling
> > to an event-driven design, which is much more performant and less
> resource
> > hungry. But this would be an AIP on its own.
> >
> > Again, great design and a comprehensive AIP, but I would include the
> > caching on the webserver to greatly improve the user experience in the
> UI.
> > Looking forward to the opinion of others on this.
> >
> > Cheers, Fokko
> >
> >
> >
> >
> >
> >
> >
> >
> > Op za 27 jul. 2019 om 01:44 schreef Zhou Fang
> <zhoufang@google.com.invalid
> > >:
> >
> > > Hi Kaxi,
> > >
> > > Just sent out the AIP:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistence+in+DB+using+JSON+for+Airflow+Webserver+and+%28optional%29+Scheduler
> > >
> > > Thanks!
> > > Zhou
> > >
> > >
> > > On Fri, Jul 26, 2019 at 1:33 PM Zhou Fang <zh...@google.com> wrote:
> > >
> > > > Hi Kaxil,
> > > >
> > > > We are also working on persisting DAGs into DB using JSON for Airflow
> > > > webserver in Google Composer. We target at minimizing the change to
> the
> > > > current Airflow code. Happy to get synced on this!
> > > >
> > > > Here is our progress:
> > > > (1) Serializing DAGs using Pickle to be used in webserver
> > > > It has been launched in Composer. I am working on the PR to upstream
> > it:
> > > > https://github.com/apache/airflow/pull/5594
> > > > Currently it does not support non-Airflow operators and we are
> working
> > on
> > > > a fix.
> > > >
> > > > (2) Caching Pickled DAGs in DB to be used by webserver
> > > > We have a proof-of-concept implementation, working on an AIP now.
> > > >
> > > > (3) Using JSON instead of Pickle in (1) and (2)
> > > > Decided to use JSON because Pickle is not secure and human readable.
> > The
> > > > serialization approach is very similar to (1).
> > > >
> > > > I will update the RP (https://github.com/apache/airflow/pull/5594)
> to
> > > > replace Pickle by JSON, and send our design of (2) as an AIP next
> week.
> > > > Glad to check together whether our implementation makes sense and do
> > > > improvements on that.
> > > >
> > > > Thanks!
> > > > Zhou
> > > >
> > > >
> > > > On Fri, Jul 26, 2019 at 7:37 AM Kaxil Naik <ka...@gmail.com>
> > wrote:
> > > >
> > > >> Hi all,
> > > >>
> > > >> We, at Astronomer, are going to spend time working on DAG
> > Serialisation.
> > > >> There are 2 AIPs that are somewhat related to what we plan to work
> on:
> > > >>
> > > >> - AIP-18 Persist all information from DAG file in DB
> > > >> <
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB
> > > >> >
> > > >> - AIP-19 Making the webserver stateless
> > > >> <
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless
> > > >> >
> > > >>
> > > >> We plan to use JSON as the Serialisation format and store it as a
> blob
> > > in
> > > >> metadata DB.
> > > >>
> > > >> *Goals:*
> > > >>
> > > >> - Make Webserver Stateless
> > > >> - Use the same version of the DAG across Webserver & Scheduler
> > > >> - Keep backward compatibility and have a flag (globally & at DAG
> > > level)
> > > >> to turn this feature on/off
> > > >> - Enable DAG Versioning (extended Goal)
> > > >>
> > > >>
> > > >> We will be preparing a proposal (AIP) after some research and some
> > > initial
> > > >> work and open it for the suggestions of the community.
> > > >>
> > > >> We already had some good brain-storming sessions with Twitter folks
> > > (DanD
> > > >> &
> > > >> Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from Uber)
> which
> > > >> will
> > > >> be a good starting point for us.
> > > >>
> > > >> If anyone in the community is interested in it or has some
> experience
> > > >> about
> > > >> the same and want to collaborate please let me know and join
> > > >> #dag-serialisation channel on Airflow Slack.
> > > >>
> > > >> Regards,
> > > >> Kaxil
> > > >>
> > > >
> > >
> >
>
Re: Airflow DAG Serialisation
Posted by Kevin Yang <yr...@gmail.com>.
Nice job Zhou!
Really excited, exactly what we wanted for the webserver scaling issue.
Want to add another big drive for Airbnb to start think about this
previously to support the effort: it can not only bring consistency between
webservers but also bring consistency between webserver and
scheduler/workers. It may be less of a problem if total DAG parsing time is
small, but for us the total DAG parsing time is 15+ mins and we had to set
the webserver( gunicorn subprocesses) restart interval to 20 mins, which
leads to a worst case 15+20+15=50 mins delay between scheduler start to
schedule things and users can see their deployed DAGs/changes...
I'm not so sure about the scheduler performance improvement: currently we
already feed the main scheduler process with SimpleDag through
DagFileProcessorManager running in a subprocess--in the future we feed it
with data from DB, which is likely slower( tho the diff should have
negligible impact to the scheduler performance). In fact if we'd keep the
existing behavior, try schedule only fresh parsed DAGs, then we may need to
deal with some consistency issue--dag processor and the scheduler race for
updating the flag indicating if the DAG is newly parsed. No big deal there
but just some thoughts on the top of my head and hopefully can be helpful.
And good idea on pre-rendering the template, believe template rendering was
the biggest concern in the previous discussion. We've also chose the
pre-rendering+JSON approach in our smart sensor API
<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization>
and
seems to be working fine--a supporting case for ur proposal ;) There's a WIP
PR <https://github.com/apache/airflow/pull/5499> for it just in case you
are interested--maybe we can even share some logics.
Thumbs-up again for this and please don't heisitate to reach out if you
want to discuss further with us or need any help from us.
Cheers,
Kevin Y
On Sat, Jul 27, 2019 at 12:54 AM Driesprong, Fokko <fo...@driesprong.frl>
wrote:
> Looks great Zhou,
>
> I have one thing that pops in my mind while reading the AIP; should keep
> the caching on the webserver level. As the famous quote goes: *"There are
> only two hard things in Computer Science: cache invalidation and naming
> things." -- Phil Karlton*
>
> Right now, the fundamental change that is being proposed in the AIP is
> fetching the DAGs from the database in a serialized format, and not parsing
> the Python files all the time. This will give already a great performance
> improvement on the webserver side because it removes a lot of the
> processing. However, since we're still fetching the DAGs from the database
> in a regular interval, cache it in the local process, so we still have the
> two issues that Airflow is suffering from right now:
>
> 1. No snappy UI because it is still polling the database in a regular
> interval.
> 2. Inconsistency between webservers because they might poll in a
> different interval, I think we've all seen this:
> https://www.youtube.com/watch?v=sNrBruPS3r4
>
> As I also mentioned in the Slack channel, I strongly feel that we should be
> able to render most views from the tables in the database, so without
> touching the blob. For specific views, we could just pull the blob from the
> database. In this case we always have the latest version, and we tackle the
> second point above.
>
> To tackle the first one, I also have an idea. We should change the DAG
> parser from a loop to something that uses inotify
> https://pypi.org/project/inotify_simple/. This will change it from polling
> to an event-driven design, which is much more performant and less resource
> hungry. But this would be an AIP on its own.
>
> Again, great design and a comprehensive AIP, but I would include the
> caching on the webserver to greatly improve the user experience in the UI.
> Looking forward to the opinion of others on this.
>
> Cheers, Fokko
>
>
>
>
>
>
>
>
> Op za 27 jul. 2019 om 01:44 schreef Zhou Fang <zhoufang@google.com.invalid
> >:
>
> > Hi Kaxi,
> >
> > Just sent out the AIP:
> >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistence+in+DB+using+JSON+for+Airflow+Webserver+and+%28optional%29+Scheduler
> >
> > Thanks!
> > Zhou
> >
> >
> > On Fri, Jul 26, 2019 at 1:33 PM Zhou Fang <zh...@google.com> wrote:
> >
> > > Hi Kaxil,
> > >
> > > We are also working on persisting DAGs into DB using JSON for Airflow
> > > webserver in Google Composer. We target at minimizing the change to the
> > > current Airflow code. Happy to get synced on this!
> > >
> > > Here is our progress:
> > > (1) Serializing DAGs using Pickle to be used in webserver
> > > It has been launched in Composer. I am working on the PR to upstream
> it:
> > > https://github.com/apache/airflow/pull/5594
> > > Currently it does not support non-Airflow operators and we are working
> on
> > > a fix.
> > >
> > > (2) Caching Pickled DAGs in DB to be used by webserver
> > > We have a proof-of-concept implementation, working on an AIP now.
> > >
> > > (3) Using JSON instead of Pickle in (1) and (2)
> > > Decided to use JSON because Pickle is not secure and human readable.
> The
> > > serialization approach is very similar to (1).
> > >
> > > I will update the RP (https://github.com/apache/airflow/pull/5594) to
> > > replace Pickle by JSON, and send our design of (2) as an AIP next week.
> > > Glad to check together whether our implementation makes sense and do
> > > improvements on that.
> > >
> > > Thanks!
> > > Zhou
> > >
> > >
> > > On Fri, Jul 26, 2019 at 7:37 AM Kaxil Naik <ka...@gmail.com>
> wrote:
> > >
> > >> Hi all,
> > >>
> > >> We, at Astronomer, are going to spend time working on DAG
> Serialisation.
> > >> There are 2 AIPs that are somewhat related to what we plan to work on:
> > >>
> > >> - AIP-18 Persist all information from DAG file in DB
> > >> <
> > >>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB
> > >> >
> > >> - AIP-19 Making the webserver stateless
> > >> <
> > >>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless
> > >> >
> > >>
> > >> We plan to use JSON as the Serialisation format and store it as a blob
> > in
> > >> metadata DB.
> > >>
> > >> *Goals:*
> > >>
> > >> - Make Webserver Stateless
> > >> - Use the same version of the DAG across Webserver & Scheduler
> > >> - Keep backward compatibility and have a flag (globally & at DAG
> > level)
> > >> to turn this feature on/off
> > >> - Enable DAG Versioning (extended Goal)
> > >>
> > >>
> > >> We will be preparing a proposal (AIP) after some research and some
> > initial
> > >> work and open it for the suggestions of the community.
> > >>
> > >> We already had some good brain-storming sessions with Twitter folks
> > (DanD
> > >> &
> > >> Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from Uber) which
> > >> will
> > >> be a good starting point for us.
> > >>
> > >> If anyone in the community is interested in it or has some experience
> > >> about
> > >> the same and want to collaborate please let me know and join
> > >> #dag-serialisation channel on Airflow Slack.
> > >>
> > >> Regards,
> > >> Kaxil
> > >>
> > >
> >
>
Re: Airflow DAG Serialisation
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Looks great Zhou,
I have one thing that pops in my mind while reading the AIP; should keep
the caching on the webserver level. As the famous quote goes: *"There are
only two hard things in Computer Science: cache invalidation and naming
things." -- Phil Karlton*
Right now, the fundamental change that is being proposed in the AIP is
fetching the DAGs from the database in a serialized format, and not parsing
the Python files all the time. This will give already a great performance
improvement on the webserver side because it removes a lot of the
processing. However, since we're still fetching the DAGs from the database
in a regular interval, cache it in the local process, so we still have the
two issues that Airflow is suffering from right now:
1. No snappy UI because it is still polling the database in a regular
interval.
2. Inconsistency between webservers because they might poll in a
different interval, I think we've all seen this:
https://www.youtube.com/watch?v=sNrBruPS3r4
As I also mentioned in the Slack channel, I strongly feel that we should be
able to render most views from the tables in the database, so without
touching the blob. For specific views, we could just pull the blob from the
database. In this case we always have the latest version, and we tackle the
second point above.
To tackle the first one, I also have an idea. We should change the DAG
parser from a loop to something that uses inotify
https://pypi.org/project/inotify_simple/. This will change it from polling
to an event-driven design, which is much more performant and less resource
hungry. But this would be an AIP on its own.
Again, great design and a comprehensive AIP, but I would include the
caching on the webserver to greatly improve the user experience in the UI.
Looking forward to the opinion of others on this.
Cheers, Fokko
Op za 27 jul. 2019 om 01:44 schreef Zhou Fang <zh...@google.com.invalid>:
> Hi Kaxi,
>
> Just sent out the AIP:
>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistence+in+DB+using+JSON+for+Airflow+Webserver+and+%28optional%29+Scheduler
>
> Thanks!
> Zhou
>
>
> On Fri, Jul 26, 2019 at 1:33 PM Zhou Fang <zh...@google.com> wrote:
>
> > Hi Kaxil,
> >
> > We are also working on persisting DAGs into DB using JSON for Airflow
> > webserver in Google Composer. We target at minimizing the change to the
> > current Airflow code. Happy to get synced on this!
> >
> > Here is our progress:
> > (1) Serializing DAGs using Pickle to be used in webserver
> > It has been launched in Composer. I am working on the PR to upstream it:
> > https://github.com/apache/airflow/pull/5594
> > Currently it does not support non-Airflow operators and we are working on
> > a fix.
> >
> > (2) Caching Pickled DAGs in DB to be used by webserver
> > We have a proof-of-concept implementation, working on an AIP now.
> >
> > (3) Using JSON instead of Pickle in (1) and (2)
> > Decided to use JSON because Pickle is not secure and human readable. The
> > serialization approach is very similar to (1).
> >
> > I will update the RP (https://github.com/apache/airflow/pull/5594) to
> > replace Pickle by JSON, and send our design of (2) as an AIP next week.
> > Glad to check together whether our implementation makes sense and do
> > improvements on that.
> >
> > Thanks!
> > Zhou
> >
> >
> > On Fri, Jul 26, 2019 at 7:37 AM Kaxil Naik <ka...@gmail.com> wrote:
> >
> >> Hi all,
> >>
> >> We, at Astronomer, are going to spend time working on DAG Serialisation.
> >> There are 2 AIPs that are somewhat related to what we plan to work on:
> >>
> >> - AIP-18 Persist all information from DAG file in DB
> >> <
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB
> >> >
> >> - AIP-19 Making the webserver stateless
> >> <
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless
> >> >
> >>
> >> We plan to use JSON as the Serialisation format and store it as a blob
> in
> >> metadata DB.
> >>
> >> *Goals:*
> >>
> >> - Make Webserver Stateless
> >> - Use the same version of the DAG across Webserver & Scheduler
> >> - Keep backward compatibility and have a flag (globally & at DAG
> level)
> >> to turn this feature on/off
> >> - Enable DAG Versioning (extended Goal)
> >>
> >>
> >> We will be preparing a proposal (AIP) after some research and some
> initial
> >> work and open it for the suggestions of the community.
> >>
> >> We already had some good brain-storming sessions with Twitter folks
> (DanD
> >> &
> >> Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from Uber) which
> >> will
> >> be a good starting point for us.
> >>
> >> If anyone in the community is interested in it or has some experience
> >> about
> >> the same and want to collaborate please let me know and join
> >> #dag-serialisation channel on Airflow Slack.
> >>
> >> Regards,
> >> Kaxil
> >>
> >
>
Re: Airflow DAG Serialisation
Posted by Zhou Fang <zh...@google.com.INVALID>.
Hi Kaxi,
Just sent out the AIP:
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistence+in+DB+using+JSON+for+Airflow+Webserver+and+%28optional%29+Scheduler
Thanks!
Zhou
On Fri, Jul 26, 2019 at 1:33 PM Zhou Fang <zh...@google.com> wrote:
> Hi Kaxil,
>
> We are also working on persisting DAGs into DB using JSON for Airflow
> webserver in Google Composer. We target at minimizing the change to the
> current Airflow code. Happy to get synced on this!
>
> Here is our progress:
> (1) Serializing DAGs using Pickle to be used in webserver
> It has been launched in Composer. I am working on the PR to upstream it:
> https://github.com/apache/airflow/pull/5594
> Currently it does not support non-Airflow operators and we are working on
> a fix.
>
> (2) Caching Pickled DAGs in DB to be used by webserver
> We have a proof-of-concept implementation, working on an AIP now.
>
> (3) Using JSON instead of Pickle in (1) and (2)
> Decided to use JSON because Pickle is not secure and human readable. The
> serialization approach is very similar to (1).
>
> I will update the RP (https://github.com/apache/airflow/pull/5594) to
> replace Pickle by JSON, and send our design of (2) as an AIP next week.
> Glad to check together whether our implementation makes sense and do
> improvements on that.
>
> Thanks!
> Zhou
>
>
> On Fri, Jul 26, 2019 at 7:37 AM Kaxil Naik <ka...@gmail.com> wrote:
>
>> Hi all,
>>
>> We, at Astronomer, are going to spend time working on DAG Serialisation.
>> There are 2 AIPs that are somewhat related to what we plan to work on:
>>
>> - AIP-18 Persist all information from DAG file in DB
>> <
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB
>> >
>> - AIP-19 Making the webserver stateless
>> <
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless
>> >
>>
>> We plan to use JSON as the Serialisation format and store it as a blob in
>> metadata DB.
>>
>> *Goals:*
>>
>> - Make Webserver Stateless
>> - Use the same version of the DAG across Webserver & Scheduler
>> - Keep backward compatibility and have a flag (globally & at DAG level)
>> to turn this feature on/off
>> - Enable DAG Versioning (extended Goal)
>>
>>
>> We will be preparing a proposal (AIP) after some research and some initial
>> work and open it for the suggestions of the community.
>>
>> We already had some good brain-storming sessions with Twitter folks (DanD
>> &
>> Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from Uber) which
>> will
>> be a good starting point for us.
>>
>> If anyone in the community is interested in it or has some experience
>> about
>> the same and want to collaborate please let me know and join
>> #dag-serialisation channel on Airflow Slack.
>>
>> Regards,
>> Kaxil
>>
>
Re: Airflow DAG Serialisation
Posted by Zhou Fang <zh...@google.com.INVALID>.
Hi Kaxil,
We are also working on persisting DAGs into DB using JSON for Airflow
webserver in Google Composer. We target at minimizing the change to the
current Airflow code. Happy to get synced on this!
Here is our progress:
(1) Serializing DAGs using Pickle to be used in webserver
It has been launched in Composer. I am working on the PR to upstream it:
https://github.com/apache/airflow/pull/5594
Currently it does not support non-Airflow operators and we are working on a
fix.
(2) Caching Pickled DAGs in DB to be used by webserver
We have a proof-of-concept implementation, working on an AIP now.
(3) Using JSON instead of Pickle in (1) and (2)
Decided to use JSON because Pickle is not secure and human readable. The
serialization approach is very similar to (1).
I will update the RP (https://github.com/apache/airflow/pull/5594) to
replace Pickle by JSON, and send our design of (2) as an AIP next week.
Glad to check together whether our implementation makes sense and do
improvements on that.
Thanks!
Zhou
On Fri, Jul 26, 2019 at 7:37 AM Kaxil Naik <ka...@gmail.com> wrote:
> Hi all,
>
> We, at Astronomer, are going to spend time working on DAG Serialisation.
> There are 2 AIPs that are somewhat related to what we plan to work on:
>
> - AIP-18 Persist all information from DAG file in DB
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB
> >
> - AIP-19 Making the webserver stateless
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless
> >
>
> We plan to use JSON as the Serialisation format and store it as a blob in
> metadata DB.
>
> *Goals:*
>
> - Make Webserver Stateless
> - Use the same version of the DAG across Webserver & Scheduler
> - Keep backward compatibility and have a flag (globally & at DAG level)
> to turn this feature on/off
> - Enable DAG Versioning (extended Goal)
>
>
> We will be preparing a proposal (AIP) after some research and some initial
> work and open it for the suggestions of the community.
>
> We already had some good brain-storming sessions with Twitter folks (DanD &
> Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from Uber) which will
> be a good starting point for us.
>
> If anyone in the community is interested in it or has some experience about
> the same and want to collaborate please let me know and join
> #dag-serialisation channel on Airflow Slack.
>
> Regards,
> Kaxil
>
Re: Airflow DAG Serialisation
Posted by Maxime Beauchemin <ma...@gmail.com>.
Great to see this happening!
On Fri, Jul 26, 2019 at 8:54 AM Jarek Potiuk <Ja...@polidea.com>
wrote:
> Great! That's definitely one of the most painful aspects of Airflow. Happy
> to help/comment/take part in the discussions and later in the
> implementation.
>
> On Fri, Jul 26, 2019 at 4:48 PM Deng Xiaodong <xd...@gmail.com> wrote:
>
> > Yes, it’s something really worth exploring.
> >
> > Recently I just finished a POC on DAG serialisation as well, while the
> > main objective was to allow “rewriting” some components, like worker (for
> > specific operators/sensors), in languages other than Python. We’re also
> > choosing JSON, so that most languages can easily process the information
> > from DAGs.
> >
> > Happy to know how this topic will be processed.
> >
> >
> > XD
> >
> > > On 26 Jul 2019, at 10:36 PM, Kaxil Naik <ka...@gmail.com> wrote:
> > >
> > > Hi all,
> > >
> > > We, at Astronomer, are going to spend time working on DAG
> Serialisation.
> > > There are 2 AIPs that are somewhat related to what we plan to work on:
> > >
> > > - AIP-18 Persist all information from DAG file in DB
> > > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB
> > >
> > > - AIP-19 Making the webserver stateless
> > > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless
> > >
> > >
> > > We plan to use JSON as the Serialisation format and store it as a blob
> in
> > > metadata DB.
> > >
> > > *Goals:*
> > >
> > > - Make Webserver Stateless
> > > - Use the same version of the DAG across Webserver & Scheduler
> > > - Keep backward compatibility and have a flag (globally & at DAG
> level)
> > > to turn this feature on/off
> > > - Enable DAG Versioning (extended Goal)
> > >
> > >
> > > We will be preparing a proposal (AIP) after some research and some
> > initial
> > > work and open it for the suggestions of the community.
> > >
> > > We already had some good brain-storming sessions with Twitter folks
> > (DanD &
> > > Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from Uber) which
> > will
> > > be a good starting point for us.
> > >
> > > If anyone in the community is interested in it or has some experience
> > about
> > > the same and want to collaborate please let me know and join
> > > #dag-serialisation channel on Airflow Slack.
> > >
> > > Regards,
> > > Kaxil
> >
> >
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
Re: Airflow DAG Serialisation
Posted by Jarek Potiuk <Ja...@polidea.com>.
Great! That's definitely one of the most painful aspects of Airflow. Happy
to help/comment/take part in the discussions and later in the
implementation.
On Fri, Jul 26, 2019 at 4:48 PM Deng Xiaodong <xd...@gmail.com> wrote:
> Yes, it’s something really worth exploring.
>
> Recently I just finished a POC on DAG serialisation as well, while the
> main objective was to allow “rewriting” some components, like worker (for
> specific operators/sensors), in languages other than Python. We’re also
> choosing JSON, so that most languages can easily process the information
> from DAGs.
>
> Happy to know how this topic will be processed.
>
>
> XD
>
> > On 26 Jul 2019, at 10:36 PM, Kaxil Naik <ka...@gmail.com> wrote:
> >
> > Hi all,
> >
> > We, at Astronomer, are going to spend time working on DAG Serialisation.
> > There are 2 AIPs that are somewhat related to what we plan to work on:
> >
> > - AIP-18 Persist all information from DAG file in DB
> > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB
> >
> > - AIP-19 Making the webserver stateless
> > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless
> >
> >
> > We plan to use JSON as the Serialisation format and store it as a blob in
> > metadata DB.
> >
> > *Goals:*
> >
> > - Make Webserver Stateless
> > - Use the same version of the DAG across Webserver & Scheduler
> > - Keep backward compatibility and have a flag (globally & at DAG level)
> > to turn this feature on/off
> > - Enable DAG Versioning (extended Goal)
> >
> >
> > We will be preparing a proposal (AIP) after some research and some
> initial
> > work and open it for the suggestions of the community.
> >
> > We already had some good brain-storming sessions with Twitter folks
> (DanD &
> > Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from Uber) which
> will
> > be a good starting point for us.
> >
> > If anyone in the community is interested in it or has some experience
> about
> > the same and want to collaborate please let me know and join
> > #dag-serialisation channel on Airflow Slack.
> >
> > Regards,
> > Kaxil
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Airflow DAG Serialisation
Posted by Deng Xiaodong <xd...@gmail.com>.
Yes, it’s something really worth exploring.
Recently I just finished a POC on DAG serialisation as well, while the main objective was to allow “rewriting” some components, like worker (for specific operators/sensors), in languages other than Python. We’re also choosing JSON, so that most languages can easily process the information from DAGs.
Happy to know how this topic will be processed.
XD
> On 26 Jul 2019, at 10:36 PM, Kaxil Naik <ka...@gmail.com> wrote:
>
> Hi all,
>
> We, at Astronomer, are going to spend time working on DAG Serialisation.
> There are 2 AIPs that are somewhat related to what we plan to work on:
>
> - AIP-18 Persist all information from DAG file in DB
> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB>
> - AIP-19 Making the webserver stateless
> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless>
>
> We plan to use JSON as the Serialisation format and store it as a blob in
> metadata DB.
>
> *Goals:*
>
> - Make Webserver Stateless
> - Use the same version of the DAG across Webserver & Scheduler
> - Keep backward compatibility and have a flag (globally & at DAG level)
> to turn this feature on/off
> - Enable DAG Versioning (extended Goal)
>
>
> We will be preparing a proposal (AIP) after some research and some initial
> work and open it for the suggestions of the community.
>
> We already had some good brain-storming sessions with Twitter folks (DanD &
> Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from Uber) which will
> be a good starting point for us.
>
> If anyone in the community is interested in it or has some experience about
> the same and want to collaborate please let me know and join
> #dag-serialisation channel on Airflow Slack.
>
> Regards,
> Kaxil
Re: Airflow DAG Serialisation
Posted by Dan Davydov <dd...@twitter.com.INVALID>.
This is awesome, and will bring tons of value to all Airflow users, thank
you for driving this!
On Fri, Jul 26, 2019 at 5:37 PM Kaxil Naik <ka...@gmail.com> wrote:
> Hi all,
>
> We, at Astronomer, are going to spend time working on DAG Serialisation.
> There are 2 AIPs that are somewhat related to what we plan to work on:
>
> - AIP-18 Persist all information from DAG file in DB
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB
> >
> - AIP-19 Making the webserver stateless
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless
> >
>
> We plan to use JSON as the Serialisation format and store it as a blob in
> metadata DB.
>
> *Goals:*
>
> - Make Webserver Stateless
> - Use the same version of the DAG across Webserver & Scheduler
> - Keep backward compatibility and have a flag (globally & at DAG level)
> to turn this feature on/off
> - Enable DAG Versioning (extended Goal)
>
>
> We will be preparing a proposal (AIP) after some research and some initial
> work and open it for the suggestions of the community.
>
> We already had some good brain-storming sessions with Twitter folks (DanD &
> Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from Uber) which will
> be a good starting point for us.
>
> If anyone in the community is interested in it or has some experience about
> the same and want to collaborate please let me know and join
> #dag-serialisation channel on Airflow Slack.
>
> Regards,
> Kaxil
>