You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Alex Van Boxel <al...@vanboxel.be> on 2016/11/04 16:31:23 UTC
Crazy Airflow DAG ideas
I think that I just made quite a crazy DAG. I'm wondering if people can top
this one. But, it's not about complexity is about the strangeness.
*So, who can top this? (I'm quite interested what sort of crazy DAG's are
out in the wild).*
First context: we have an on premises SQL Server (and I want to move
everything to the cloud). So what better to do to have a daily snapshot in
the cloud).
This dag does:
- Wait for the full backup to appear in Google Cloud Storage (the new
sensor)
- It spins up SQL Server on Google Cloud through Deployment manager and
does the following (though a PowerShell script):
* Import the backup from GCS
* Import the file in SQL Server
* Detach the database
* Detach the disk
* Take a Snapshot of the disk (db-YYYYMMDD-full)
* Mount again
* Minify
* Detach, Snapshot again (db-YYYYMMDD-mini), Attach
Downstream we have 3 sensors (the deployment operator doesn't wait till the
script is finished, just that the machines are ready):
* (A) Waiting for snapshot: db-YYYYMMDD-full
* (B) Waiting for snapshot: db-YYYYMMDD-mini
* (C) Looking at the serial port of the machine to look when it's finished
So in each stream:
(A) full_snapshot_sensor
- Creates a new SQL Server from the full snapshot, this will be a
reporting db (takes 3 minutes from nothing, quite cool)
- Change DNS entry
- Delete old deprecated versions of reporting
(C) if script is finished
- Delete the original SQL server for importing the database
[image: Screen Shot 2016-11-04 at 09.47.36.png]