You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/27 14:56:40 UTC

[GitHub] [airflow] jpays opened a new issue #10603: Let users add custom metadata to the dag runs

jpays opened a new issue #10603:
URL: https://github.com/apache/airflow/issues/10603


   <!--
   
   Welcome to Apache Airflow!  For a smooth issue process, try to answer the following questions.
   Don't worry if they're not all applicable; just try to include what you can :-)
   
   If you need to include code snippets or logs, please put them in fenced code
   blocks.  If they're super-long, please use the details tag like
   <details><summary>super-long log</summary> lots of stuff </details>
   
   Please delete these comment blocks before submitting the issue.
   
   -->
   
   **Description**
   
   Currently there is no way to add custom metadata to a dag run, this would be an interesting enhancement.
   
   **Use case / motivation**
   
   This could have multiple use cases, here are some examples :
   
   * Let's say you have a dag that ends up loading data in a table, the number of lines loaded could be added as a custom metadata on the dag run
   * You have a dag that run multiple times every day, and that can process serveral perimeters of data depending on external factors, the perimeters processed could be added as a custom metadata on the dag run
   * 
   
   The main goal is to provide some easy-to-use feedback about what a dagrun has done. This could afterwards be displayed on the dagrun UI, but I think it is mainly relevant to use in charts, SQL queries or various dataviz products such has Tableau that could be plugged on the airflow DB.
   
   Ultimately, this data could be made available in the new API.
   
   I don't know if there is a way to achieve this differently.
   I can help adding this feature if necessary.
   
   **Related Issues**
   
   Could not find.
   
   
   
   Thanks &  Regards.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] jpays commented on issue #10603: Let users add custom metadata to the dag runs

Posted by GitBox <gi...@apache.org>.
jpays commented on issue #10603:
URL: https://github.com/apache/airflow/issues/10603#issuecomment-683675894


   You mean using the "conf" attribute of the DagRun ?
   
   That would indeed be an option I suppose, but :
   
   * `conf` is stored as `PickleType`, which is good if you need to pass any kind of Python object to the DagRun, but not suitable at all for external use such as SQL Query
   * from my understanding,  `conf` should be considered as the input values for the DagRun, and as such should not be tampered with during dag execution. Of course you can do it but input values should probably be immutable and that kind of metadata would be better off stored another place.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #10603: Let users add custom metadata to the dag runs

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #10603:
URL: https://github.com/apache/airflow/issues/10603#issuecomment-682002223


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] jpays edited a comment on issue #10603: Let users add custom metadata to the dag runs

Posted by GitBox <gi...@apache.org>.
jpays edited a comment on issue #10603:
URL: https://github.com/apache/airflow/issues/10603#issuecomment-683675894


   You mean using the "conf" attribute of the DagRun ?
   
   That would indeed be an option I suppose, but :
   
   * `conf` is stored as `PickleType`, which is good if you need to pass any kind of Python object to the DagRun, but not suitable at all for external use such as SQL Query
   * from my understanding,  `conf` should be considered as the input values for the DagRun, and as such should not be tampered with during dag execution. Of course you can do it, but input values should probably be immutable and that kind of metadata would be better off stored another place.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] sreenath-kamath commented on issue #10603: Let users add custom metadata to the dag runs

Posted by GitBox <gi...@apache.org>.
sreenath-kamath commented on issue #10603:
URL: https://github.com/apache/airflow/issues/10603#issuecomment-683334044


   Can't this be achieved using the configuration passed during the dag_runs


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org