You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/05/09 12:15:05 UTC
[GitHub] [airflow] lozuwa opened a new issue, #23584: Airflow 2.2.4 doesn't add new DAGs to the serialized_dag table, thus webserver doesn't show the new DAGs
lozuwa opened a new issue, #23584:
URL: https://github.com/apache/airflow/issues/23584
### Apache Airflow version
2.2.4
### What happened
We have upgraded from Airflow 1.10.5 to Airflow 2.2.4. Upgrade path was 1.10.5 -> 1.10.15 -> 2.1.0 -> 2.2.4
Everything seems to work with the exception that new DAGs cannot be added to Airflow. They are not displayed in either the serialized_dag table or the webserver UI.
### What you think should happen instead
If a new DAG is added to the dags folder path, Airflow should add it to the serialized_dag table and webserver should display it in its UI.
### How to reproduce
Steps:
1. Deploy Airflow 2.2.4 with gitSync enabled and persistance disabled.
2. Add a new DAG to the repository in the correct path.
3. New dag can be listed using command `airflow dags list` but is not displayed either in the database or the UI.
### Operating System
Kubernetes v1.20.15
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==3.3.0
apache-airflow-providers-google==6.8.0
apache-airflow-providers-slack==4.2.3
apache-airflow-providers-postgres==4.1.0
apache-airflow-providers-cncf-kubernetes==3.0.0
apache-airflow-providers-ssh==2.4.3
apache-airflow-providers-mysql
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
Standard deployment with the following parameters
```yaml
config:
core:
dags_folder: '{{ include "airflow_dags" . }}'
store_serialized_dags: "True"
store_dag_code: "True"
plugins_folder: "/opt/airflow/airflow-plugins/"
load_examples: "False"
executor: '{{ .Values.executor }}'
donot_pickle: "False"
secure_mode: "True"
max_active_runs_per_dag: "16"
dagbag_import_timeout: "120"
sql_engine_encoding: "utf-8"
sql_alchemy_pool_enabled: "True"
sql_alchemy_pool_size: "10"
sql_alchemy_max_overflow: "10"
sql_alchemy_pool_recycle: "60"
sql_alchemy_pool_pre_ping: "True"
logging:
logging_level: "INFO"
colored_console_log: "True"
# Remote logging
remote_logging: "True"
logging_config_class: "log_config.LOGGING_CONFIG"
remote_log_folder: "xxx"
remote_base_log_folder: "xxx"
task_log_reader: "s3.task"
#remote_log_conn_id: "s3liveintent"
remote_log_conn_id: "aws_default"
encrypt_s3_logs: "False"
metrics:
statsd_on: '{{ ternary "True" "False" .Values.statsd.enabled }}'
statsd_port: 9125
statsd_prefix: airflow
statsd_host: '{{ printf "%s-statsd" .Release.Name }}'
webserver:
enable_proxy_fix: "True"
expose_config: "True"
rbac: "True"
base_url: "http://xxx"
web_server_host: "0.0.0.0"
web_server_port: "8080"
web_server_worker_timeout: "300"
email:
email_backend: "airflow.utils.email.send_email_smtp"
smtp:
smtp_host: "xxx"
smtp_starttls: "True"
smtp_ssl: "False"
smtp_user: "xxx"
smtp_password: "xxx"
smtp_port: "587"
smtp_mail_from: "xxx"
celery:
worker_concurrency: 32
scheduler:
# statsd params included for Airflow 1.10 backward compatibility; moved to [metrics] in 2.0
statsd_on: '{{ ternary "True" "False" .Values.statsd.enabled }}'
statsd_port: 9125
statsd_prefix: airflow
statsd_host: '{{ printf "%s-statsd" .Release.Name }}'
# `run_duration` included for Airflow 1.10 backward compatibility; removed in 2.0.
run_duration: 41460
parsing_processes: "8"
min_file_process_interval: "60"
dag_dir_list_interval: "120"
catchup_by_default: "False"
job_heartbeat_sec: "10"
scheduler_heartbeat_sec: "10"
elasticsearch:
json_format: 'True'
log_id_template: "{dag_id}_{task_id}_{execution_date}_{try_number}"
elasticsearch_configs:
max_retries: 3
timeout: 30
retry_timeout: 'True'
kerberos:
keytab: '{{ .Values.kerberos.keytabPath }}'
reinit_frequency: '{{ .Values.kerberos.reinitFrequency }}'
principal: '{{ .Values.kerberos.principal }}'
ccache: '{{ .Values.kerberos.ccacheMountPath }}/{{ .Values.kerberos.ccacheFileName }}'
celery_kubernetes_executor:
kubernetes_queue: 'kubernetes'
kubernetes:
namespace: '{{ .Release.Namespace }}'
airflow_configmap: '{{ include "airflow_config" . }}'
airflow_local_settings_configmap: '{{ include "airflow_config" . }}'
pod_template_file: '{{ include "airflow_pod_template_file" . }}/pod_template_file.yaml'
worker_container_repository: '{{ .Values.images.airflow.repository | default .Values.defaultAirflowRepository }}'
worker_container_tag: '{{ .Values.images.airflow.tag | default .Values.defaultAirflowTag }}'
multi_namespace_mode: '{{ if .Values.multiNamespaceMode }}True{{ else }}False{{ end }}'
# yamllint enable rule:line-length
```
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #23584: Airflow 2.2.4 doesn't add new DAGs to the serialized_dag table, thus webserver doesn't show the new DAGs
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #23584:
URL: https://github.com/apache/airflow/issues/23584#issuecomment-1121022127
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org