You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "bernvaughn (via GitHub)" <gi...@apache.org> on 2026/04/16 16:24:18 UTC
[I] Airflow 3.2 unable to handle \0 in dag files [airflow]
bernvaughn opened a new issue, #65379:
URL: https://github.com/apache/airflow/issues/65379
### Under which category would you file this issue?
Airflow Core
### Apache Airflow version
3.2.0
### What happened and how to reproduce it?
Have a dag file in airflow 3.1.8 with a \0 literal character, then try to upgrade to airflow 3.2.0. Migrations will fail because \0 cannot be serialized into JSONB
### What you think should happen instead?
We have dag(s) that process CSV files that use \0 as a delimiter, and we have the delimiter defined in the dag file.
### Operating System
rhel 8.6
### Deployment
Docker-Compose
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
apache-airflow==3.2.0
apache-airflow-client==3.1.3
apache-airflow-providers-amazon==9.22.0
apache-airflow-providers-celery==3.15.0
apache-airflow-providers-common-compat==1.14.3
apache-airflow-providers-common-io==1.7.2
apache-airflow-providers-common-sql==1.34.0
apache-airflow-providers-fab==3.6.1
apache-airflow-providers-ftp==3.14.0
apache-airflow-providers-http==6.0.0
apache-airflow-providers-imap==3.10.0
apache-airflow-providers-postgres==6.5.0
apache-airflow-providers-sftp==5.5.0
apache-airflow-providers-smtp==2.4.5
apache-airflow-providers-snowflake==6.7.0
apache-airflow-providers-sqlite==4.2.0
apache-airflow-providers-ssh==4.2.0
apache-airflow-providers-standard==1.12.3
### Official Helm Chart version
Not Applicable
### Kubernetes Version
Not Applicable
### Helm Chart configuration
_No response_
### Docker Image customizations
FROM docker.io/apache/airflow:3.2.0-python3.12
# no-dd-sa:docker-best-practices/no-root-user
USER root
RUN apt-get update && \
apt-get install -y \
bash \
git \
build-essential \
gcc \
libffi-dev \
musl-dev \
libpq-dev \
xmlsec1 \
postgresql
# disable strict host key checking for ssh to support saspy
RUN echo "StrictHostKeyChecking no" >> /etc/ssh/ssh_config
USER airflow
# set dontwritebytecode to prevent __pycache__ directories
# we don't need them in the container, and it is making things bloat
ENV PYTHONDONTWRITEBYTECODE=1
COPY requirements/production.txt .
# use --no-compile to prevent .pyc files from being created
# use --no-cache-dir to prevent pip from caching the downloaded packages
# both of these is to save space / prevent bloat in the container
RUN pip install -r production.txt --no-compile --no-cache-dir
COPY plugins /opt/airflow/plugins
COPY dags /opt/airflow/dags
### Anything else?
We are upgrading from 3.1.8 to 3.2.0 to resolve security vulnerabilities, but the airflow migration step is failing in our deployment pipeline.
Specifically, the migration 0089 is where it is failing. This migration changes serialized_dag column to JSONB from JSON, which cannot handle the null character (literal \0) being stored in the database. We have dags that use this character to define a delimiter in some csv files we consume.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow 3.2 unable to handle \0 in dag files [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #65379:
URL: https://github.com/apache/airflow/issues/65379#issuecomment-4261668693
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow 3.2 unable to handle \0 in dag files [airflow]
Posted by "Pedrinhonitz (via GitHub)" <gi...@apache.org>.
Pedrinhonitz commented on issue #65379:
URL: https://github.com/apache/airflow/issues/65379#issuecomment-4537497549
Hello, I'm curious about the problem and would like to understand it better.
Thinking about it, perhaps it's possible to "fix" this without modifying Airflow. I might be wrong, but if you pass the delimiter through the task instead of the DAG body or `default_args`, I think you won't have this problem.
I did a local test and it worked; maybe this will help.
I created this code, where I assign the delimiter to variables, as you suggested, in the DAG body:
```python
from datetime import datetime
from airflow import DAG
from airflow.providers.standard.operators.python import PythonOperator
field_del = '@@\0@@'
record_del = '^^\0^^'
def process_csv(**context):
print(f"field_del repr: {field_del!r}")
print(f"record_del repr: {record_del!r}")
return {"ok": True}
with DAG(
dag_id="old_csv_delimiter_dag",
start_date=datetime(2024, 1, 1),
schedule=None,
catchup=False,
tags=["testing", "issue-65379"]
) as dag:
PythonOperator(
task_id="process_csv",
python_callable=process_csv,
op_kwargs={"field_del": field_del, "record_del": record_del}
)
```
And I created this other one as follows (this one worked locally):
```python
from datetime import datetime
from airflow import DAG
from airflow.providers.standard.operators.python import PythonOperator
def _delimiters():
nul = chr(0)
return f"@@{nul}@@", f"^^{nul}^^{nul}"
def process_csv(**context):
field_del, record_del = _delimiters()
print(f"field_del repr: {field_del!r}")
print(f"record_del repr: {record_del!r}")
assert '\x00' in field_del
assert '\x00' in record_del
return {"ok": True}
with DAG(
dag_id="fix_csv_delimiter_dag",
start_date=datetime(2024, 1, 1),
schedule=None,
catchup=False,
tags=["testing", "issue-65379"]
) as dag:
PythonOperator(
task_id="process_csv",
python_callable=process_csv
)
```
I might be wrong, but this prevents the DAG from changing the delimiter parameters and causes it to be assembled inside the task, avoiding null bytes in the string; at least it worked here. If it doesn't work, please provide more details about the error.
**This may not be the best solution, but it might help migrate the version faster than if someone else solved the problem.**
**_Cursor AI was used to assist in reviewing your issue with the claude-4.6-sonnet-medium model; there may be defects._**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org