Late arriving data from external apis

[Teresa Martyny <te...@omadahealth.com>]

Hey there,

We are moving our ETL over into airflow and re-writing our scripts in
python. Due to client-side queueing, offline iOS and Android data may take
up to 5 days to enter the raw data store for Mixpanel. Currently what we
have found is fastest/easiest is to just drop and replace 5 days of
Mixpanel data every day to handle that.

Moving into the airflow world, I thought it'd be nice to be able to use
airflow's features to be able to do some of this for us.

How are other folks handling late arriving data from external apis?

Thanks!

*Teresa Martyny*
pronouns: she, her, hers
Software Engineer | Data Team | Omada Health <https://www.omadahealth.com/>
500 Sansome St #200, SF, CA 94111

*What is Omada?* <https://vimeo.com/203386025>

-- 
This email may contain material that is confidential and/or privileged for 
the sole use of the intended recipient. Any review, reliance, or 
distribution by others or forwarding without express permission is strictly 
prohibited. If you are not the intended recipient, please contact the 
sender and delete all copies. Also note that email is not an appropriate 
way to send protected health information to Omada Health employees. Please 
use your discretion when responding to this email.