You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/07/07 14:32:36 UTC

[GitHub] [airflow] kollurukailash opened a new issue, #24899: DAG store

kollurukailash opened a new issue, #24899:
URL: https://github.com/apache/airflow/issues/24899

   ### Description
   
   **Situation:** 
   Currently, Airflow limits the user to put DAG files in /dags folder or get it from the GitHub repository.
   
   **Proposal:** 
   Introducing DAG STORE, where all DAG files will reside on Cloud-Storage places like AWS S3, Azure Blob, and Google storage. With this in place, we drastically make DAGs available to the Airflow in an easy manner without copying and pasting them in the dag folder or restarting the airflow image, or no volume mounts.
   
   
   Like the feature? Give a like to the comment.
   
   My Thoughts: But Archieturly, This will can be network intensive as per the present DAG file processing. To address this, we  need On fly DAG file processing.
   
   
   ### Use case/motivation
   
   _No response_
   
   ### Related issues
   
   -
   
   ### Are you willing to submit a PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #24899: DAG store

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24899:
URL: https://github.com/apache/airflow/issues/24899#issuecomment-1177841255

   This is not a feature. This is a huge Architectural change on how Airlfow works.
   
   There were a few discussions about this in the past. This resulted of a number of AIPs (Airflow Improvement Proposals) - for example:
    
   * https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-20+DAG+manifest 
   * https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher
   
   None of those are complete and approved, they were discussed but the original authors have not followed up with them.
   If you would like to lead the effort, feel free to pic-up where they left (you will find links to dicsussion there) and re-start discussion on the devlist. There are a number of architectural changes it implies and it needs someone who will take a lead, discusses it, writes proposals, get the community feedback, organize voting and get it implemented. 
   
   If you wold like to take the lead on that @kollurukailash - feel free to take that Pathe - but it is the level that requires AIP, Design, discussion and voting. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #24899: DAG store

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #24899:
URL: https://github.com/apache/airflow/issues/24899#issuecomment-1177710884

   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk closed issue #24899: DAG store

Posted by GitBox <gi...@apache.org>.
potiuk closed issue #24899: DAG store
URL: https://github.com/apache/airflow/issues/24899


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org