You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/02 09:56:48 UTC

[GitHub] [airflow] kaxil opened a new issue #10116: Use SHA to compare Serialized DAGs

kaxil opened a new issue #10116:
URL: https://github.com/apache/airflow/issues/10116


   <!--
   
   Welcome to Apache Airflow!  For a smooth issue process, try to answer the following questions.
   Don't worry if they're not all applicable; just try to include what you can :-)
   
   If you need to include code snippets or logs, please put them in fenced code
   blocks.  If they're super-long, please use the details tag like
   <details><summary>super-long log</summary> lots of stuff </details>
   
   Please delete these comment blocks before submitting the issue.
   
   -->
   
   **Description**
   
   <!-- A short description of your feature -->
   Currently, we compare entire Serialized blobs to determine if the DAG changed or not before we write the Serialized DAGs to the Database which can be expensive.
   Instead, we will store the HASH / SHA of the Serialized JSON and store that in a separate column in the serialized_dag table.
   And before writing the Serialized Blob, we will compare the SHA to decide if DAG changed.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kaxil commented on issue #10116: Use Hash / SHA to compare Serialized DAGs

Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #10116:
URL: https://github.com/apache/airflow/issues/10116#issuecomment-667655282


   > What are the plans for 1.10.12?
   
   Mostly to fix all the Kubernetes related issues that came of the Kubernetes Refactors in Airflow 1.10.11 -- https://github.com/apache/airflow/issues?q=is%3Aopen+is%3Aissue+label%3Ak8s+milestone%3A%22Airflow+1.10.12%22
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kaxil edited a comment on issue #10116: Use Hash / SHA to compare Serialized DAGs

Posted by GitBox <gi...@apache.org>.
kaxil edited a comment on issue #10116:
URL: https://github.com/apache/airflow/issues/10116#issuecomment-667655282


   > What are the plans for 1.10.12?
   
   Mostly to fix all the Kubernetes related issues that came of the Kubernetes Refactors in Airflow 1.10.11 -- https://github.com/apache/airflow/issues?q=is%3Aopen+is%3Aissue+label%3Ak8s+milestone%3A%22Airflow+1.10.12%22
   
   We can hopefully get those fixed (PRs are directly to v1-10-* branches) so hopefully, we can get RCs at the end of next week
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kaxil commented on issue #10116: Use Hash / SHA to compare Serialized DAGs

Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #10116:
URL: https://github.com/apache/airflow/issues/10116#issuecomment-667654063


   Planning to get to this by tomorrow so we can release it with Airflow 1.10.12


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kaxil closed issue #10116: Use Hash / SHA to compare Serialized DAGs

Posted by GitBox <gi...@apache.org>.
kaxil closed issue #10116:
URL: https://github.com/apache/airflow/issues/10116


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #10116: Use Hash / SHA to compare Serialized DAGs

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #10116:
URL: https://github.com/apache/airflow/issues/10116#issuecomment-667654697


   What are the plans for 1.10.12? 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #10116: Use Hash / SHA to compare Serialized DAGs

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #10116:
URL: https://github.com/apache/airflow/issues/10116#issuecomment-667653976


   Oh yeah!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org