You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/04/28 12:52:10 UTC
[GitHub] [airflow] zorzigio opened a new issue #15572: import error
zorzigio opened a new issue #15572:
URL: https://github.com/apache/airflow/issues/15572
**Apache Airflow version**: 2.0.2
**Kubernetes version**:
- Client Version: v1.17.4
- Server Version: v1.18.14
**Environment**:
- **Cloud provider or hardware configuration**: Azure
- **OS**: "Debian GNU/Linux 10 (buster)"
- **Kernel**: Linux airflow2-webserver-96bfc89f7-qwk26 5.4.0-1043-azure #45~18.04.1-Ubuntu SMP Sat Mar 20 16:16:05 UTC 2021 x86_64 GNU/Linux
**What happened**:
My folder structure is the following
```bash
/opt/airflow
├── dags
│ ├── __init__.py
│ ├── example_k8s
│ │ └── test.py
│ └── utils
│ ├── defaults.py
│ ├── __init__.py
└── ...
```
In `test.py` I am importing from `defaults.py` as follows
```python
from utils.defaults import DEFAULT_ARGS, DEFAULT_NAMESPACE
```
This raises an error in the UI
```bash
Broken DAG: [/opt/airflow/dags/example_k8s/test.py] Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/opt/airflow/dags/example_k8s/test.py", line 200, in <module>
from utils.defaults import DEFAULT_ARGS, DEFAULT_NAMESPACE
ModuleNotFoundError: No module named 'utils'
```
Not sure why as the folder `/opt/airflow/dags` should be already included in the path (at least in the worker pod) as when I print `sys.path` from a `PythonOperator` I get
```bash
['/home/airflow/.local/bin', '/usr/local/lib/python36.zip', '/usr/local/lib/python3.6', '/usr/local/lib/python3.6/lib-dynload', '/home/airflow/.local/lib/python3.6/site-packages', '/usr/local/lib/python3.6/site-packages', '/opt/airflow/dags', '/opt/airflow/config', '/opt/airflow/plugins']
```
and executing the command
```bash
python /opt/airflow/dags/example_k8s/test.py
```
in the webserver does not raise any error.
If I explicitly add `/opt/airflow/dags` to the system path in `test.py`
```python
sys.path.insert(0, '/opt/airflow/dags/')
```
the error in the UI is no longer reported and the DAG appears in the UI.
**What you expected to happen**:
The import to work even without explicitly adding the path to the system
**How to reproduce it**:
See above
**Anything else we need to know**:
Airflow has been installed using Helm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj closed issue #15572: import error
Posted by GitBox <gi...@apache.org>.
mik-laj closed issue #15572:
URL: https://github.com/apache/airflow/issues/15572
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sawaca96 commented on issue #15572: import error
Posted by GitBox <gi...@apache.org>.
sawaca96 commented on issue #15572:
URL: https://github.com/apache/airflow/issues/15572#issuecomment-861061924
@zorzigio
>@sawaca96, I expected it to work, however when trying to install the Helm chart, it raises the following error
I want to get more information. then I can try to find cause.
What helm chart are you using ? I'm use [airflow-stable](https://github.com/airflow-helm/charts/tree/main/charts/airflow)
> As from the Airflow documentation, Airflow should include the dags path by default?
I don't think so. but if you want to use module in dags folder then you should include dags path.
>A workaraound is setting PYTHONPATH in values.yaml directly as env variable
I use this [offical values.yaml](https://github.com/airflow-helm/charts/blob/main/charts/airflow/values.yaml), I can't find env section
If you use kubernetes secrets to env then you use `extraEnv` like this
```
kubectl create secret generic some-secrets --from-literal='SOME_SECRETS'='SOME_SECRETS'
```
```
extraEnv:
- name: SOME_SECRETS
valueFrom:
secretKeyRef:
name: some-secrets
key: SOME_SECRETS
```
## Important
Note: **change** `PYTHONPATH: /opt/airflow/dags/repo` to `PYTHONPATH: $PYTHONPATH:/opt/airflow/dags/repo`
The former overwrites(existing path lost) the existing path and the latter is added to the existing path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT commented on issue #15572: import error
Posted by GitBox <gi...@apache.org>.
JavierLopezT commented on issue #15572:
URL: https://github.com/apache/airflow/issues/15572#issuecomment-831788775
I have the exact same problem but I am using docker-compose instead of Kubernetes. In Airflow 1.10.14 I did not have this problem. Importing airflow does not solve the problem, as neither does it using `sys.append('/opt/airflow/dags/utils')` nor `export PYTHONPATH=$PYTHONPATH:/opt/airflow/dags/utilities`
@zorzigio were you able to solve the issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kbumsik-angelswing commented on issue #15572: import error
Posted by GitBox <gi...@apache.org>.
kbumsik-angelswing commented on issue #15572:
URL: https://github.com/apache/airflow/issues/15572#issuecomment-861868794
@potiuk Actually, this problem also occurs in the official chart v1.0. The OP zorzigio is using the official Helm and sawaca96 uses unofficial one.
I had the same issue with Helm chart v1.0 and I resolved the issue by setting `PYTHONPATH` as zorzigio mentioned AND manually delete the existing Pod to force restarting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zorzigio commented on issue #15572: import error
Posted by GitBox <gi...@apache.org>.
zorzigio commented on issue #15572:
URL: https://github.com/apache/airflow/issues/15572#issuecomment-861322038
@sawaca96
> I want to get more information. then I can try to find cause.
>What helm chart are you using ? I'm use airflow-stable
I am using the official Helm Chart ([apache-airflow/airflow](https://github.com/apache/airflow/tree/main/chart))
>I don't think so. but if you want to use module in dags folder then you should include dags path.
In the [Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/modules_management.html#additional-modules-in-airflow) it is mentioned:
> Airflow adds three additional directories to the sys.path
> - DAGS folder: It is configured with option dags_folder in section [core].
> - Config folder: It is configured by setting AIRFLOW_HOME variable ({AIRFLOW_HOME}/config) by default.
> - Plugins Folder: It is configured with option plugins_folder in section [core].
However when running `print(sys.path)`, these folder do not appear (see [post](https://github.com/apache/airflow/issues/15572#issue-869913582))
>I use this offical values.yaml, I can't find env section
Here it is ([env section](https://github.com/apache/airflow/blob/1fba5402bb14b3ffa6429fdc683121935f88472f/chart/values.yaml#L173))
>In addition, explain extraEnv
Here is that section too ([extraEnv](https://github.com/apache/airflow/blob/1fba5402bb14b3ffa6429fdc683121935f88472f/chart/values.yaml#L219))
> Note
> change PYTHONPATH: /opt/airflow/dags/repo to PYTHONPATH: $PYTHONPATH:/opt/airflow/dags/repo
> The former overwrites(existing path lost) the existing path and the latter is added to the existing path.
The `PYTHONPATH` was not defined in the pods, so this makes no difference
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15572: import error
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15572:
URL: https://github.com/apache/airflow/issues/15572#issuecomment-861882945
sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sawaca96 edited a comment on issue #15572: import error
Posted by GitBox <gi...@apache.org>.
sawaca96 edited a comment on issue #15572:
URL: https://github.com/apache/airflow/issues/15572#issuecomment-845695724
@zorzigio
I'm use airflow helm chart and image info is below
```
image:
repository: apache/airflow
tag: 2.0.2-python3.8
```
I add just `PYTHONPATH` to config env
```
#values.yaml
airflow:
config:
PYTHONPATH: /opt/airflow/dags/repo # repo is created because of gitsync
```
then import error is gone 👍👍
my dags folder tree
```
dags
ㄴ __init__.py
ㄴ example.py
ㄴ utils
ㄴ decorator.py
ㄴ category_1
ㄴ file_1.py
ㄴ file_2.py
```
now I can `from dags import some_function` in file_1 and file_2
Also I can `from dags.utils.decorator import some` in example.py
😁
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sglickman commented on issue #15572: import error
Posted by GitBox <gi...@apache.org>.
sglickman commented on issue #15572:
URL: https://github.com/apache/airflow/issues/15572#issuecomment-851594955
Not sure if this is exactly your issue, but I was seeing something like this on Kubernetes, where I got UI errors about modules not found which I knew were present and in $PYTHONPATH (I confirmed this by running `kubectl exec $POD_NAME -it sh` and opening a Python interactive session on the scheduler container).
What ended up doing the trick for me was ensuring that the gitsync completed before the scheduler was brought up - that is, creating an git-sync initcontainer with the `GIT_SYNC_ONE_TIME` flag. hope this helps!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sawaca96 commented on issue #15572: import error
Posted by GitBox <gi...@apache.org>.
sawaca96 commented on issue #15572:
URL: https://github.com/apache/airflow/issues/15572#issuecomment-851786756
@JavierLopezT
I'm not sure this is working
```
version: "3"
services:
airflow:
image: apache/airflow
environment:
- PYTHONPATH: /opt/airflow/dags/repo
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15572: import error
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15572:
URL: https://github.com/apache/airflow/issues/15572#issuecomment-861713188
I think you should open the issue in the helm chart repo you are using or switch to the official chart supported by the Airlfow community. Airflow has an official chart as of recenlty https://airflow.apache.org/docs/helm-chart/stable/index.html and other chart has been long around but it's not really supported here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sawaca96 edited a comment on issue #15572: import error
Posted by GitBox <gi...@apache.org>.
sawaca96 edited a comment on issue #15572:
URL: https://github.com/apache/airflow/issues/15572#issuecomment-861061924
@zorzigio
>@sawaca96, I expected it to work, however when trying to install the Helm chart, it raises the following error
I want to get more information. then I can try to find cause.
What helm chart are you using ? I'm use [airflow-stable](https://github.com/airflow-helm/charts/tree/main/charts/airflow)
> As from the Airflow documentation, Airflow should include the dags path by default?
I don't think so. but if you want to use module in dags folder then you should include dags path.
>A workaraound is setting PYTHONPATH in values.yaml directly as env variable
I use this [offical values.yaml](https://github.com/airflow-helm/charts/blob/main/charts/airflow/values.yaml), I can't find env section
In addition, explain `extraEnv`
If you use kubernetes secrets to airflow env then you use `extraEnv` like this
```
kubectl create secret generic some-secrets --from-literal='SOME_SECRETS'='SOME_SECRETS'
```
```
extraEnv:
- name: SOME_SECRETS
valueFrom:
secretKeyRef:
name: some-secrets
key: SOME_SECRETS
```
## Important
Note: **change** `PYTHONPATH: /opt/airflow/dags/repo` to `PYTHONPATH: $PYTHONPATH:/opt/airflow/dags/repo`
The former overwrites(existing path lost) the existing path and the latter is added to the existing path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT edited a comment on issue #15572: import error
Posted by GitBox <gi...@apache.org>.
JavierLopezT edited a comment on issue #15572:
URL: https://github.com/apache/airflow/issues/15572#issuecomment-849542122
> @zorzigio
> I'm use airflow helm chart and image info is below
>
> ```
> image:
> repository: apache/airflow
> tag: 2.0.2-python3.8
> ```
>
> I add just `PYTHONPATH` to config env
>
> ```
> #values.yaml
> airflow:
> config:
> PYTHONPATH: /opt/airflow/dags/repo # repo is created because of gitsync
> ```
>
> then import error is gone 👍👍
> my dags folder tree
>
> ```
> dags
> ㄴ __init__.py
> ㄴ example.py
> ㄴ utils
> ㄴ decorator.py
> ㄴ category_1
> ㄴ file_1.py
> ㄴ file_2.py
> ```
>
> now I can `from dags import some_function` in file_1 and file_2
> Also I can `from dags.utils.decorator import some` in example.py
> 😁
Do you know how could I "translate" your solution to a docker-compose? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zorzigio edited a comment on issue #15572: import error
Posted by GitBox <gi...@apache.org>.
zorzigio edited a comment on issue #15572:
URL: https://github.com/apache/airflow/issues/15572#issuecomment-861871587
@potiuk I confirm that I am using the official Helm chart.
Could you reopen this issue please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kbumsik-angelswing edited a comment on issue #15572: import error
Posted by GitBox <gi...@apache.org>.
kbumsik-angelswing edited a comment on issue #15572:
URL: https://github.com/apache/airflow/issues/15572#issuecomment-861868794
@potiuk Actually, this problem also occurs in the official chart v1.0. The OP zorzigio is using the official Helm and sawaca96 uses the unofficial one.
I had the same issue with Helm chart v1.0 and I resolved the issue by setting `PYTHONPATH` as zorzigio mentioned AND manually delete the existing Pod to force restarting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zorzigio commented on issue #15572: import error
Posted by GitBox <gi...@apache.org>.
zorzigio commented on issue #15572:
URL: https://github.com/apache/airflow/issues/15572#issuecomment-859931031
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sawaca96 edited a comment on issue #15572: import error
Posted by GitBox <gi...@apache.org>.
sawaca96 edited a comment on issue #15572:
URL: https://github.com/apache/airflow/issues/15572#issuecomment-861061924
@zorzigio
>@sawaca96, I expected it to work, however when trying to install the Helm chart, it raises the following error
I want to get more information. then I can try to find cause.
What helm chart are you using ? I'm use [airflow-stable](https://github.com/airflow-helm/charts/tree/main/charts/airflow)
> As from the Airflow documentation, Airflow should include the dags path by default?
I don't think so. but if you want to use module in dags folder then you should include dags path.
>A workaraound is setting PYTHONPATH in values.yaml directly as env variable
I use this [offical values.yaml](https://github.com/airflow-helm/charts/blob/main/charts/airflow/values.yaml), I can't find env section
In addition, explain `extraEnv`
If you want to use kubernetes secrets in airflow env, then you use `extraEnv` like this
```
kubectl create secret generic some-secrets --from-literal='SOME_SECRETS'='SOME_SECRETS'
```
```
extraEnv:
- name: SOME_SECRETS
valueFrom:
secretKeyRef:
name: some-secrets
key: SOME_SECRETS
```
## Important
Note: **change** `PYTHONPATH: /opt/airflow/dags/repo` to `PYTHONPATH: $PYTHONPATH:/opt/airflow/dags/repo`
The former overwrites(existing path lost) the existing path and the latter is added to the existing path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #15572: import error
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #15572:
URL: https://github.com/apache/airflow/issues/15572
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zorzigio commented on issue #15572: import error
Posted by GitBox <gi...@apache.org>.
zorzigio commented on issue #15572:
URL: https://github.com/apache/airflow/issues/15572#issuecomment-832313455
@JavierLopezT, not a clean (or desirable) solution, however for me
```python
sys.path.insert(0, '/opt/airflow/dags')
```
at the top of the file removes the error from the UI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sawaca96 edited a comment on issue #15572: import error
Posted by GitBox <gi...@apache.org>.
sawaca96 edited a comment on issue #15572:
URL: https://github.com/apache/airflow/issues/15572#issuecomment-861061924
@zorzigio
>@sawaca96, I expected it to work, however when trying to install the Helm chart, it raises the following error
I want to get more information. then I can try to find cause.
What helm chart are you using ? I'm use [airflow-stable](https://github.com/airflow-helm/charts/tree/main/charts/airflow)
> As from the Airflow documentation, Airflow should include the dags path by default?
I don't think so. but if you want to use module in dags folder then you should include dags path.
>A workaraound is setting PYTHONPATH in values.yaml directly as env variable
I use this [offical values.yaml](https://github.com/airflow-helm/charts/blob/main/charts/airflow/values.yaml), I can't find env section
In addition, explain `extraEnv`
If you want to use kubernetes secrets in airflow env, then you use `extraEnv` like this
```
kubectl create secret generic some-secrets --from-literal='SOME_SECRETS'='SOME_SECRETS'
```
```
extraEnv:
- name: SOME_SECRETS
valueFrom:
secretKeyRef:
name: some-secrets
key: SOME_SECRETS
```
## Note
**change `PYTHONPATH: /opt/airflow/dags/repo` to `PYTHONPATH: $PYTHONPATH:/opt/airflow/dags/repo`**
The former overwrites(existing path lost) the existing path and the latter is added to the existing path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sawaca96 commented on issue #15572: import error
Posted by GitBox <gi...@apache.org>.
sawaca96 commented on issue #15572:
URL: https://github.com/apache/airflow/issues/15572#issuecomment-861890869
@zorzigio
> In the [Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/modules_management.html#additional-modules-in-airflow) it is mentioned:
I'm sorry I didn't understand correctly. now I understand. You mean your `PYTHONPATH` does not have a below path ?
```
[/home/rootcss/venvs/airflow/bin:/usr/lib/python38.zip:/usr/lib/python3.8:/usr/lib/python3.8/libdynload:/home/rootcss/venvs/airflow/lib/python3.8/sitepackages:/home/rootcss/airflow/dags:/home/rootcss/airflow/config:/home/rootcss/airflow/plugins]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sawaca96 commented on issue #15572: import error
Posted by GitBox <gi...@apache.org>.
sawaca96 commented on issue #15572:
URL: https://github.com/apache/airflow/issues/15572#issuecomment-845695724
@zorzigio
I'm use airflow helm chart and image info is below
```
image:
repository: apache/airflow
tag: 2.0.2-python3.8
```
I add just `PYTHONPATH` to config env
```
#values.yaml
airflow:
config:
PYTHONPATH: /opt/airflow/dags/repo # repo is created because of gitsync
```
then import error is gone 👍👍
my dags folder tree
```
dags
ㄴ __init__.py
ㄴ example.py
ㄴ utils
ㄴ decorator.py
ㄴ category_1
ㄴ file_1.py
ㄴ file_2.py
```
now I can `from dags import some_function` in file_1 and file_2
Also I can `from dags.utils.decorator import some` in example.py
good ! 😁
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zorzigio commented on issue #15572: import error
Posted by GitBox <gi...@apache.org>.
zorzigio commented on issue #15572:
URL: https://github.com/apache/airflow/issues/15572#issuecomment-828607978
@mik-laj thanks for looking into this.
The path `/opt/airflow/dags` is already included in the path as confirmed when running the command `print(sys.path)` from a task, however the UI is still giving error.
I also confirm that explicitly importing airflow (`import airflow`) does not solve the issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zorzigio commented on issue #15572: import error
Posted by GitBox <gi...@apache.org>.
zorzigio commented on issue #15572:
URL: https://github.com/apache/airflow/issues/15572#issuecomment-861871587
@potiuk I confirm that I am using the official Helm chart
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] duynguyenhoang commented on issue #15572: import error
Posted by GitBox <gi...@apache.org>.
duynguyenhoang commented on issue #15572:
URL: https://github.com/apache/airflow/issues/15572#issuecomment-887214103
Our team had same issue with import. Finally we decided to move the `utils` stuff to `plugins` out of `dags` folder and it works like a charm
You will see the structure is similar to:
```
/opt/airflow
├── dags
│ ├── __init__.py
│ ├── example_k8s
│ │ └── test.py
├── plugins
│ └── utils
│ ├── defaults.py
│ ├── __init__.py
└──
```
For everyone who deploy Airflow k8s cluster and using gitsync, there are some extra steps requires (We haven't tested it yet, but we are going to do that and update the result here.)
* Change `AIRFLOW__CORE__PLUGINS_FOLDER` to gitsync repo plugins folder, example `/opt/airflow/repo/plugin`
* Set this https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#reload-on-plugin-change to `True`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT commented on issue #15572: import error
Posted by GitBox <gi...@apache.org>.
JavierLopezT commented on issue #15572:
URL: https://github.com/apache/airflow/issues/15572#issuecomment-849542122
> @zorzigio
> I'm use airflow helm chart and image info is below
>
> ```
> image:
> repository: apache/airflow
> tag: 2.0.2-python3.8
> ```
>
> I add just `PYTHONPATH` to config env
>
> ```
> #values.yaml
> airflow:
> config:
> PYTHONPATH: /opt/airflow/dags/repo # repo is created because of gitsync
> ```
>
> then import error is gone 👍👍
> my dags folder tree
>
> ```
> dags
> ㄴ __init__.py
> ㄴ example.py
> ㄴ utils
> ㄴ decorator.py
> ㄴ category_1
> ㄴ file_1.py
> ㄴ file_2.py
> ```
>
> now I can `from dags import some_function` in file_1 and file_2
> Also I can `from dags.utils.decorator import some` in example.py
> 😁
Do you know how could I "translate" your solution to a docker-compose? Thanks!
Also, the error for me is happening just with some dags, so it seems to be randomish or something.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zorzigio commented on issue #15572: import error
Posted by GitBox <gi...@apache.org>.
zorzigio commented on issue #15572:
URL: https://github.com/apache/airflow/issues/15572#issuecomment-855713118
> @zorzigio
> I'm use airflow helm chart and image info is below
>
> ```
> image:
> repository: apache/airflow
> tag: 2.0.2-python3.8
> ```
>
> I add just `PYTHONPATH` to config env
>
> ```
> #values.yaml
> airflow:
> config:
> PYTHONPATH: /opt/airflow/dags/repo # repo is created because of gitsync
> ```
>
> then import error is gone
> my dags folder tree
>
> ```
> dags
> ㄴ __init__.py
> ㄴ example.py
> ㄴ utils
> ㄴ decorator.py
> ㄴ category_1
> ㄴ file_1.py
> ㄴ file_2.py
> ```
>
> now I can `from dags import some_function` in file_1 and file_2
> Also I can `from dags.utils.decorator import some` in example.py
Thanks @sawaca96, I didn't try this yet but I guess it will work.
As from the [Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/modules_management.html#additional-modules-in-airflow), Airflow should include the dags path by default?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #15572: import error
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #15572:
URL: https://github.com/apache/airflow/issues/15572#issuecomment-828586735
Airflow adds additional modules to sys.path, so if you want to test these modules then you have to add these paths as well. The easiest solution is to load Airflow: ``import airflow``. For more information, see:
http://airflow.apache.org/docs/apache-airflow/stable/modules_management.html#additional-modules-in-airflow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org