You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/21 01:51:46 UTC

[GitHub] [airflow] mik-laj opened a new issue #11708: Git sync for plugins

mik-laj opened a new issue #11708:
URL: https://github.com/apache/airflow/issues/11708


   Hello,
   
   Helm Chart currently supports DAG sync, but some users have the need to sync plugins in a similar way as well.
   
   It would be fantastic if Helm Chart would make it easy to set up sync `~/airflow/plugins` from a separate repository as well.
   
   Best regards,
   Kamil BreguĊ‚a


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] DevanshuKoyalkar commented on issue #11708: Git sync for plugins in the Helm Chart

Posted by GitBox <gi...@apache.org>.
DevanshuKoyalkar commented on issue #11708:
URL: https://github.com/apache/airflow/issues/11708#issuecomment-717550037


   Hi mik-laj, I'll take this up


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] DevanshuKoyalkar commented on issue #11708: Git sync for plugins in the Helm Chart

Posted by GitBox <gi...@apache.org>.
DevanshuKoyalkar commented on issue #11708:
URL: https://github.com/apache/airflow/issues/11708#issuecomment-718138538


   I am getting a sense of the problem statement. I should be adding two new options like `dags.persistence.enabled`_ and `dags.gitSync.enabled ` for plugins which do the similar job right?
   Is there an equivalent of airflow_dags_mount_path for plugins?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #11708: Git sync for plugins in the Helm Chart

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #11708:
URL: https://github.com/apache/airflow/issues/11708#issuecomment-925725607


   This is a good idea to have GitSync for plugins, but I am afraid the solution where you keep your plugins together with DAGs is a very bad idea for Airflow 2, one that should be heavily discouraged.
   
   There are two reasons (actually the root cause is isolation and security):
   
   1) Plugins should not be "modifiable" by DAG writers. Plugins give you more "powers" over Airlfow that DAGs. The Plugins can modify UI of airflow (similarly as providers - with connections) and they are executed in the context of scheduler (not in the context of FileProcessor as all DAGs are) and you should not have the same people being able to write DAGs and modify plugins.  This is something that wil be only strenghtened in the future when we increase isolation of Ailrflow components and it's a prerequisite to future Airflow multi-tenancy. 
   
   But more importantly:
   
   2) In Airflow 2 DAGs are not available as mounted volume for Webserver in our Helm Chart (and they should NOT be). See the Official Helm Chart. This is precisely because of the increased isolation and the fact that webserver does not need them any more (it uses Serialized form of the DAGs from the DB). So if your Plugins modify the UI of Airflow, this solution will not work (unless you also additionally mount the DAG folder to webserver - but this violates the isolation that was improved in Airflow 2 - see point 1). 
   
   Generally speaking - you SHOULD separate DAGs and Plugins. Access to modigy those two in many cases should be controlled differently for security reasons.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] EliMor commented on issue #11708: Git sync for plugins in the Helm Chart

Posted by GitBox <gi...@apache.org>.
EliMor commented on issue #11708:
URL: https://github.com/apache/airflow/issues/11708#issuecomment-902120486


   FYI; https://github.com/bitnami/charts/blob/master/bitnami/airflow/values.yaml#L781


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] magnusfurugard commented on issue #11708: Git sync for plugins in the Helm Chart

Posted by GitBox <gi...@apache.org>.
magnusfurugard commented on issue #11708:
URL: https://github.com/apache/airflow/issues/11708#issuecomment-925636717


   It is possible to mount the plugins via git sync today, if you keep your plugins with your DAGs, e.g.; 
   ```
   # Your dag repo
   dags/...
   plugins/...
   ```
   
   You can then just set an `extraEnv` with the direct path. 
   ```
   extraEnv:
       - name: "AIRFLOW__CORE__PLUGINS_FOLDER"
         value: "/opt/airflow/dags/repo/plugins"
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] magnusfurugard edited a comment on issue #11708: Git sync for plugins in the Helm Chart

Posted by GitBox <gi...@apache.org>.
magnusfurugard edited a comment on issue #11708:
URL: https://github.com/apache/airflow/issues/11708#issuecomment-925636717


   It is possible to mount the plugins via git sync today, if you keep your plugins with your DAGs, e.g.; 
   ```
   # Your dag repo
   dags/...
   plugins/...
   ```
   
   You can then just set an `extraEnv` with the direct path. 
   ```
   extraEnv:
       - name: "AIRFLOW__CORE__PLUGINS_FOLDER"
         value: "/opt/airflow/dags/repo/plugins"
   ```
   
   Edit: You may not want to do this. See the excellent explanation from `potiuk` below :arrow_down: 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] mik-laj commented on issue #11708: Git sync for plugins in the Helm Chart

Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #11708:
URL: https://github.com/apache/airflow/issues/11708#issuecomment-717552472


   Assigned. 
   
   We are migrating to [the new unit test framework](https://github.com/apache/airflow/pull/11827), so you will have to wait a while before you can write tests ... or if you have the patience you can also write in the old framework, but this is very limiting and frustrating.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] mik-laj commented on issue #11708: Git sync for plugins in the Helm Chart

Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #11708:
URL: https://github.com/apache/airflow/issues/11708#issuecomment-718774274


   >  I should be adding two new options like dags.persistence.enabled_ and dags.gitSync.enabled for plugins which do the similar job right?
   
   Yes. It would be good to support the other configuration options for gitSync as well.
   
   > Is there an equivalent of airflow_dags_mount_path for plugins?
   
   By default, ``$AIRFLOW_HOME/plugins``. 
   See also:  
   https://airflow.readthedocs.io/en/latest/plugins.html
   https://github.com/apache/airflow/blob/9320fe6c149fff995d4d85af593a327c56731549/chart/values.yaml#L28
   https://airflow.readthedocs.io/en/latest/configurations-ref.html#plugins-folder


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] magnusfurugard edited a comment on issue #11708: Git sync for plugins in the Helm Chart

Posted by GitBox <gi...@apache.org>.
magnusfurugard edited a comment on issue #11708:
URL: https://github.com/apache/airflow/issues/11708#issuecomment-925636717


   It is possible to mount the plugins via git sync today, if you keep your plugins with your DAGs, e.g.; 
   ```
   # Your dag repo
   dags/...
   plugins/...
   ```
   
   You can then just set an `extraEnv` with the direct path. 
   ```
   extraEnv:
       - name: "AIRFLOW__CORE__PLUGINS_FOLDER"
         value: "/opt/airflow/dags/repo/plugins"
   ```
   
   :exclamation: Edit: You probably don't want to do this. See the excellent explanation from `potiuk` below :arrow_down: 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] thesuperzapper commented on issue #11708: Git sync for plugins in the Helm Chart

Posted by GitBox <gi...@apache.org>.
thesuperzapper commented on issue #11708:
URL: https://github.com/apache/airflow/issues/11708#issuecomment-901530913


   The [user-community chart](https://github.com/airflow-helm/charts/tree/main/charts/airflow) (not the official) has implemented support for using plugins from within your DAGs git-repo. (See PR https://github.com/airflow-helm/charts/pull/390)
   
   You only need to correctly set your `AIRFLOW__CORE__PLUGINS_FOLDER` to be a subfolder of your dag folder, which will probably be something like `/opt/airflow/dags/repo/MY_PLUGINS_FOLDER_IN_REPO`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] dollschasingmen commented on issue #11708: Git sync for plugins in the Helm Chart

Posted by GitBox <gi...@apache.org>.
dollschasingmen commented on issue #11708:
URL: https://github.com/apache/airflow/issues/11708#issuecomment-842646573


   any progress on this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] magnusfurugard edited a comment on issue #11708: Git sync for plugins in the Helm Chart

Posted by GitBox <gi...@apache.org>.
magnusfurugard edited a comment on issue #11708:
URL: https://github.com/apache/airflow/issues/11708#issuecomment-925636717


   It is possible to mount the plugins via git sync today, if you keep your plugins with your DAGs, e.g.; 
   ```
   # Your dag repo
   dags/...
   plugins/...
   ```
   
   You can then just set an `extraEnv` with the direct path. 
   ```
   extraEnv:
       - name: "AIRFLOW__CORE__PLUGINS_FOLDER"
         value: "/opt/airflow/dags/repo/plugins"
   ```
   
   Edit: You ~may not~ probably don't want to do this. See the excellent explanation from `potiuk` below :arrow_down: 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] magnusfurugard edited a comment on issue #11708: Git sync for plugins in the Helm Chart

Posted by GitBox <gi...@apache.org>.
magnusfurugard edited a comment on issue #11708:
URL: https://github.com/apache/airflow/issues/11708#issuecomment-925636717


   It is possible to mount the plugins via git sync today, if you keep your plugins with your DAGs, e.g.; 
   ```
   # Your dag repo
   dags/...
   plugins/...
   ```
   
   You can then just set an `extraEnv` with the direct path. 
   ```
   extraEnv:
       - name: "AIRFLOW__CORE__PLUGINS_FOLDER"
         value: "/opt/airflow/dags/repo/plugins"
   ```
   
   Edit: You may not want to do this. See the excellent explanation from `poutik` below :arrow_down: 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org