You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/03/11 06:15:12 UTC
[GitHub] [airflow] rbakhtaraev opened a new issue #14714: Problem with custom plugins when using git-sync container
rbakhtaraev opened a new issue #14714:
URL: https://github.com/apache/airflow/issues/14714
**Apache Airflow version**: 1.10.14/2.0.0 (it isnt important)
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): 1.20.1
**Environment**:
- **Cloud provider or hardware configuration**:
- **OS** (e.g. from /etc/os-release):
- **Kernel** (e.g. `uname -a`):
- **Install tools**:
- **Others**:
**What happened**:
Custom plugins don't work, when using git-sync container for fetch him from git.
I think that is due to the fact that plugins and dags are fetching later than worker is starting.
If i add some pause before starting worker/webserver through `sleep 30`, worker/webserver is starting postponed and plugins are loading.
Is there a better solution for my case?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #14714: Problem with custom plugins when using git-sync container
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #14714:
URL: https://github.com/apache/airflow/issues/14714#issuecomment-870094536
>I also found that this works even though I don't have the plugins folder and all the imports are within the dags folder. Which didn't workout when setting it to false. @kaxil Am I configuring it correctly? I thought this reload was intended for the plugins folder.
That does not feel right. The operator is unrelated to any sort of plugin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] KIRY4 edited a comment on issue #14714: Problem with custom plugins when using git-sync container
Posted by GitBox <gi...@apache.org>.
KIRY4 edited a comment on issue #14714:
URL: https://github.com/apache/airflow/issues/14714#issuecomment-879794991
Hello @kaxil ! We have a similar problem. But there are some differences. We are using Azure Kubernetes Service and for dags logs plugins and our custom_operators we are using Kubernetes PVC's with Azure file shares under the hood. Also we are using community Helm chart: https://github.com/airflow-helm/charts/tree/main/charts/airflow
After uploading custom operator on mounted as PV/PVC Azure file shares I expect to have it accessible from Airflow DAGs.
I have 2 folders: /opt/airflow/plugins and /opt/airflow/custom_operator.
Examples of imports (for test I jumped into Web POD):
From plugins/ I can call custom opertor like - from table_migration_operator import TableMigrationOperator
From custom_operator/ - from custom_operator.save_result_operator import SaveResultOperator
<img width="1484" alt="image" src="https://user-images.githubusercontent.com/42247943/125605632-82e702f0-92c7-42c5-8752-cd66b9d22f88.png">
Inside POD from Python everything works fine.
What I'm getting in UI:
<img width="642" alt="image" src="https://user-images.githubusercontent.com/42247943/125605859-a6c170c9-c61d-4d29-9514-e32a9b2c9d42.png">
Both ways are not working in UI until webserver will not be restarted (restart pod with webserver). After the restart everything is fine.
Airflow versions: 2.1.0 and 2.1.1
PATH=/home/airflow/.local/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PYTHONPATH=/opt/airflow:/opt/airflow/dags:/opt/airflow/logs:/opt/airflow/plugins:/opt/airflow/custom_operator:/opt/airflow/jars
AIRFLOW__WEBSERVER__RELOAD_ON_PLUGIN_CHANGE=True
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #14714: Problem with custom plugins when using git-sync container
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #14714:
URL: https://github.com/apache/airflow/issues/14714#issuecomment-796732186
@rbakhtaraev You can add an init container that will fetch the repository initially.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #14714: Problem with custom plugins when using git-sync container
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #14714:
URL: https://github.com/apache/airflow/issues/14714#issuecomment-879862588
Just to clarify (we recently agreed on naming) - this is the "User-community" chart :) - I hope my PR https://github.com/airflow-helm/charts/pull/294 will be merged soon and we will get rid of the ambiguity.
If you want to continue with Matthew's (User-community) chart you should raise the issue there https://github.com/airflow-helm/charts/ - we do not directly support other helm charts here, but we do offer help if you want to migrate to the official one - you can open a discussion or start discussing in slack (we just created `#helm-chart-official` channel there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed issue #14714: Problem with custom plugins when using git-sync container
Posted by GitBox <gi...@apache.org>.
kaxil closed issue #14714:
URL: https://github.com/apache/airflow/issues/14714
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #14714: Problem with custom plugins when using git-sync container
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #14714:
URL: https://github.com/apache/airflow/issues/14714#issuecomment-879862588
Just to clarify (we recently agreed on naming) - this is the "User-community" chart :) - I hope my PR https://github.com/airflow-helm/charts/pull/294 will be merged soon and we will get rid of the ambiguity.
If you want to continue with Matthew's (User-community) chart you should raise the issue there https://github.com/airflow-helm/charts/ - we do not directly support the non-official chart here, but we do offer help if you want to migrate to the official one - you can open a discussion or start discussing in slack (we just created `#helm-chart-official` channel there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] KIRY4 edited a comment on issue #14714: Problem with custom plugins when using git-sync container
Posted by GitBox <gi...@apache.org>.
KIRY4 edited a comment on issue #14714:
URL: https://github.com/apache/airflow/issues/14714#issuecomment-879794991
Hello @kaxil ! We have a similar problem. But there are some differences. We are using Azure Kubernetes Service and for dags logs plugins and our custom_operators we are using Kubernetes PVC's with Azure file shares under the hood. Also we are using community Helm chart: https://github.com/airflow-helm/charts/tree/main/charts/airflow
After uploading custom operator on mounted as PV/PVC Azure file shares I expect to have it accessible from Airflow DAGs.
I have 2 folders: /opt/airflow/plugins and /opt/airflow/custom_operator.
Examples of imports (for test I jumped into Web POD):
From plugins/ I can call custom opertor like - from table_migration_operator import TableMigrationOperator
From custom_operator/ - from custom_operator.save_result_operator import SaveResultOperator
<img width="1484" alt="image" src="https://user-images.githubusercontent.com/42247943/125605632-82e702f0-92c7-42c5-8752-cd66b9d22f88.png">
Inside POD from Python everything works fine.
What I'm getting in UI:
<img width="642" alt="image" src="https://user-images.githubusercontent.com/42247943/125605859-a6c170c9-c61d-4d29-9514-e32a9b2c9d42.png">
Both are not working in UI until webserver will not be restarted (restart pod with webserver). After the restart everything is fine.
Airflow versions: 2.1.0 and 2.1.1
PATH=/home/airflow/.local/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PYTHONPATH=/opt/airflow:/opt/airflow/dags:/opt/airflow/logs:/opt/airflow/plugins:/opt/airflow/custom_operator:/opt/airflow/jars
AIRFLOW__WEBSERVER__RELOAD_ON_PLUGIN_CHANGE=True
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] KIRY4 commented on issue #14714: Problem with custom plugins when using git-sync container
Posted by GitBox <gi...@apache.org>.
KIRY4 commented on issue #14714:
URL: https://github.com/apache/airflow/issues/14714#issuecomment-879794991
Hello @kaxil ! We have a similar problem. But there are some differences. We are using Azure Kubernetes Service and for dags logs plugins and our custom_operators we are using Kubernetes PVC's with Azure file shares under the hood. Also we are using community Helm chart: https://github.com/airflow-helm/charts/tree/main/charts/airflow
After uploading custom operator on mounted as PV/PVC Azure file shares I expect to have it accessible from Airflow DAGs.
I have 2 folders: /opt/airflow/plugins and /opt/airflow/custom_operator.
Examples of imports (for test I jumped into Web POD):
From plugins/ I can call custom opertor like - from table_migration_operator import TableMigrationOperator
From custom_operator/ - from custom_operator.save_result_operator import SaveResultOperator
<img width="1484" alt="image" src="https://user-images.githubusercontent.com/42247943/125605632-82e702f0-92c7-42c5-8752-cd66b9d22f88.png">
Inside POD from Python everything works fine.
What I'm getting in UI:
<img width="642" alt="image" src="https://user-images.githubusercontent.com/42247943/125605859-a6c170c9-c61d-4d29-9514-e32a9b2c9d42.png">
Both ways are not working in UI until webserver will not be restarted (restart pod with webserver). After the restart everything is fine.
Airflow versions: 2.1.0 and 2.1.1
PATH=/home/airflow/.local/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PYTHONPATH=/opt/airflow:/opt/airflow/dags:/opt/airflow/logs:/opt/airflow/plugins:/opt/airflow/custom_operator:/opt/airflow/jars
AIRFLOW__WEBSERVER__RELOAD_ON_PLUGIN_CHANGE=True
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #14714: Problem with custom plugins when using git-sync container
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #14714:
URL: https://github.com/apache/airflow/issues/14714#issuecomment-870094536
>I also found that this works even though I don't have the plugins folder and all the imports are within the dags folder. Which didn't workout when setting it to false. @kaxil Am I configuring it correctly? I thought this reload was intended for the plugins folder.
That does not feel right. The operator is unrelated to any sort of plugin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #14714: Problem with custom plugins when using git-sync container
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #14714:
URL: https://github.com/apache/airflow/issues/14714#issuecomment-809757473
Can you give an example code, please use normal python modules or package, check https://airflow.apache.org/docs/apache-airflow/stable/modules_management.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #14714: Problem with custom plugins when using git-sync container
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #14714:
URL: https://github.com/apache/airflow/issues/14714#issuecomment-797290282
> Can we add it into chart? I'm ready to send PR with edits.
Yes. It will be helpful. Happy to help with review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rbakhtaraev commented on issue #14714: Problem with custom plugins when using git-sync container
Posted by GitBox <gi...@apache.org>.
rbakhtaraev commented on issue #14714:
URL: https://github.com/apache/airflow/issues/14714#issuecomment-797284326
> @rbakhtaraev You can add an init container that will fetch the repository initially.
It's requires a support of chart locally (e.g. custom deployment.yaml), but that isn't convenient.
Can we add it into chart? I'm ready to send PR with edits.
How i see that:
We have an init-container with shared dir, which fetch a repo, after that shared dir passing into airflow and git-sync container and update continuously
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yesemsanthoshkumar commented on issue #14714: Problem with custom plugins when using git-sync container
Posted by GitBox <gi...@apache.org>.
yesemsanthoshkumar commented on issue #14714:
URL: https://github.com/apache/airflow/issues/14714#issuecomment-865639159
I faced the similar issue as well. The custom operators wouldn't load even if they are inside the dags folder. Tried loading it from the plugins dir as well.
What worked for me is setting `reload_on_plugin_change` to `True` in the values.yaml. Operators got picked up from the dags folder itself.
````
from datetime import datetime
from airflow.models import DAG
from airflow.operators.dummy import DummyOperator
from custom_operator import MyDummyOperator
default_args = {
'owner': 'airflow,
'start_date': datetime(2021, 6, 1),
'depends_on_past': False,
}
dag = DAG(
dag_id='import_test',
schedule_interval=None,
default_args=default_args,
catchup=False
)
op = DummyOperator(
task_id='dummy',
dag=dag
)
my_op = MyDummyOperator(
task_id='dummy_mine',
dag=dag
)
````
and my custom operator was
```
from airflow.operators.dummy import DummyOperator
class MyDummyOperator(DummyOperator):
pass
```
I also found that this works even though I don't have the plugins folder and all the imports are within the dags folder. Which didn't workout when setting it to false. @kaxil Am I on the right path?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #14714: Problem with custom plugins when using git-sync container
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #14714:
URL: https://github.com/apache/airflow/issues/14714#issuecomment-796488636
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #14714: Problem with custom plugins when using git-sync container
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #14714:
URL: https://github.com/apache/airflow/issues/14714#issuecomment-804403298
Hi @rbakhtaraev -- what were the things you were importing via plugins.
Just wanted to check since you can import Operators, sensors etc without plugins
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yesemsanthoshkumar edited a comment on issue #14714: Problem with custom plugins when using git-sync container
Posted by GitBox <gi...@apache.org>.
yesemsanthoshkumar edited a comment on issue #14714:
URL: https://github.com/apache/airflow/issues/14714#issuecomment-865639159
Airflow version: 2.0.2
I faced the similar issue as well. The custom operators wouldn't load even if they are inside the dags folder. Tried loading it from the plugins dir as well.
What worked for me is setting `reload_on_plugin_change` to `True` in the values.yaml. Operators got picked up from the dags folder itself.
````
from datetime import datetime
from airflow.models import DAG
from airflow.operators.dummy import DummyOperator
from custom_operator import MyDummyOperator
default_args = {
'owner': 'airflow,
'start_date': datetime(2021, 6, 1),
'depends_on_past': False,
}
dag = DAG(
dag_id='import_test',
schedule_interval=None,
default_args=default_args,
catchup=False
)
op = DummyOperator(
task_id='dummy',
dag=dag
)
my_op = MyDummyOperator(
task_id='dummy_mine',
dag=dag
)
````
and my custom operator was
```
from airflow.operators.dummy import DummyOperator
class MyDummyOperator(DummyOperator):
pass
```
I also found that this works even though I don't have the plugins folder and all the imports are within the dags folder. Which didn't workout when setting it to false. @kaxil Am I configuring it correctly? I thought this reload was intended for the plugins folder.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rbakhtaraev commented on issue #14714: Problem with custom plugins when using git-sync container
Posted by GitBox <gi...@apache.org>.
rbakhtaraev commented on issue #14714:
URL: https://github.com/apache/airflow/issues/14714#issuecomment-804604092
> Hi @rbakhtaraev -- what were the things you were importing via plugins.
>
> Just wanted to check since you can import Operators, sensors etc without plugins
Hi @kaxil , im using plugins to some custom clients for internal tools. For example, i have 5 dags, which imported two custom plugins.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yesemsanthoshkumar commented on issue #14714: Problem with custom plugins when using git-sync container
Posted by GitBox <gi...@apache.org>.
yesemsanthoshkumar commented on issue #14714:
URL: https://github.com/apache/airflow/issues/14714#issuecomment-865639159
I faced the similar issue as well. The custom operators wouldn't load even if they are inside the dags folder. Tried loading it from the plugins dir as well.
What worked for me is setting `reload_on_plugin_change` to `True` in the values.yaml. Operators got picked up from the dags folder itself.
````
from datetime import datetime
from airflow.models import DAG
from airflow.operators.dummy import DummyOperator
from custom_operator import MyDummyOperator
default_args = {
'owner': 'airflow,
'start_date': datetime(2021, 6, 1),
'depends_on_past': False,
}
dag = DAG(
dag_id='import_test',
schedule_interval=None,
default_args=default_args,
catchup=False
)
op = DummyOperator(
task_id='dummy',
dag=dag
)
my_op = MyDummyOperator(
task_id='dummy_mine',
dag=dag
)
````
and my custom operator was
```
from airflow.operators.dummy import DummyOperator
class MyDummyOperator(DummyOperator):
pass
```
I also found that this works even though I don't have the plugins folder and all the imports are within the dags folder. Which didn't workout when setting it to false. @kaxil Am I on the right path?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org