You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/02/24 07:13:49 UTC

[GitHub] [airflow] prithvisathiya opened a new issue #14412: STORE_DAG_CODE not displaying correct code for zipped dag files

prithvisathiya opened a new issue #14412:
URL: https://github.com/apache/airflow/issues/14412

<!--

Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)

If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>

Please delete these comment blocks before submitting the issue.

-->

<!--

IMPORTANT!!!

PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE
NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!

PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!

Please complete the next sections or the issue will be closed.
These questions are the first thing we need to know to understand the context.

-->

**Apache Airflow version**:
1.10.14

**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
1.19.4

**Environment**:

- **Cloud provider or hardware configuration**:
- **OS** (e.g. from /etc/os-release):
- **Kernel** (e.g. `uname -a`):
- **Install tools**: production docker image
- **Others**: python3.7

**What happened**:
I recently upgraded from 1.10.9 -> 1.10.14, and enabled both `STORE_SERIALIZED_DAGS` and `STORE_DAG_CODE` to True. Everything seems to work as expected, and the scheduler is able to execute the DAG and the tasks within. However, the "Code View" in the UI displays a very long hex string instead of the actual code.

```
\x66726f6d206461746574696...
```

I have narrowed it down to be an issue, only when dag file is stored within zip files. For example, `my_dag.py` and `my_other_dag.py` would show in the UI incorrectly in the following case:
```bash
├── my_dag.zip
│ ├── my_dag.py
│ └── utils.py
├── my_other_dag.zip
│ ├── my_other_dag.py
```
but this is fine:
```bash
├── my_dag.py
├── my_other_dag.py
├── utils.py
```

A quick look at the database also reflects the incorrectly parsed DAG code within the `source_code` column of the `public.dag_code` table. But the fact that the scheduler can execute the dag, and that the `data` column of the `public.serialized_dag` table contains the correct code, suggests that this issue is isolated only when storing within the `dag_code` table.

**What you expected to happen**:
I should expect the actual python code to show in UI
```python
import os
...
```

**How to reproduce it**:
Create 2 similar dags (1 within zip and 1 unzipped) and place into root dags folder. The UI will display the `dag2.py` incorrectly, but work fine for `dag1.py`
```bash
├── opt/airflow/dags
│ ├── dag1.py
│ ├── dag2.zip
│ │ └── dag2.py
```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org