You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2019/10/03 12:58:36 UTC
[GitHub] [airflow] kaxil edited a comment on issue #5743:
[AIRFLOW-5088][AIP-24] Persisting serialized DAG in DB for webserver
scalability
kaxil edited a comment on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-530789821
Few things to do:
- [ ] Add https://github.com/astronomer/airflow/commit/baf12f626e6d56dfde735faaed71b2c30cb4befb and add tests for it
- [x] Reduce the info we store in Serialized DAGs by removing all the default arguments that are not overridden by users. Eg `owner` in DAG & Task etc. This will help reduce blob size as well as reduce the time spent in `_deserialise` method.
- [ ] Agree / dis-agree on using https://pypi.org/project/SQLAlchemy-JSONField/ instead of our code . It also has a nice option of specifying json library as compared to providing that info in the `create_engine.json_serializer` and `create_engine.json_deserializer` parameters in https://docs.sqlalchemy.org/en/13/core/type_basics.html#sqlalchemy.types.JSON
- [ ] Test serialisation code with zipped DAG files
cc @coufon @ashb
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services