Initial load best practices

[Rafael Barbosa <rr...@gmail.com>]

I am creating a number of small ETL tasks to pull data into a Redshift
instance. For every task, I want a one-time bootstrapping task to create
the target table (and maybe even the schema) and do the initial load.

Preferably, I would like the DAG to perform this initial load, instead
doing it manually. This would be particularly useful when testing the DAG
in different environments (e.g., local vs production).

I see two ways of doing this:

1. Have a bootstrap operator in the DAG that `create if not exists` the
table, and rely on the DAG start date to backfill the data in the initial
load. E.g. `bootstrap` -> `fetch` -> `load`.
2. Implement an operator that does the initial load if the table does not
exist, or a incremental load if it does. The start date for the initial
upload is a parameter for the operator. Similar to what is described in
this post
<https://medium.com/@rchang/a-beginners-guide-to-data-engineering-part-ii-47c4e7cbda71#a102>
in
"parameterize workflow".

How do you typically handle this situation? What are the best practices for
an initial load?

Best,
Rafael