You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/04/30 01:39:30 UTC
[GitHub] [airflow] trollhe opened a new issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
trollhe opened a new issue #15607:
URL: https://github.com/apache/airflow/issues/15607
hi,team:
i deployment airflow 2.0.2 in docker by docker-compose,airflow service start just 1 minute have many dags,bug 1 minute after dag is none,and throws bellow error:
already share dag folder to webservice/scheduler/flower container
```
Python version: 3.8.5
Airflow version: 2.0.2
Node: airflow-webserver
-------------------------------------------------------------------------------
Traceback (most recent call last):
File "/root/.local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/root/.local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/root/.local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/root/.local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/root/.local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/root/.local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.8/site-packages/airflow/www/auth.py", line 34, in decorated
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/airflow/www/decorators.py", line 97, in view_func
return f(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/airflow/www/decorators.py", line 60, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/airflow/www/views.py", line 1876, in tree
dag = current_app.dag_bag.get_dag(dag_id)
File "/usr/local/lib/python3.8/site-packages/airflow/utils/session.py", line 70, in wrapper
return func(*args, session=session, **kwargs)
File "/usr/local/lib/python3.8/site-packages/airflow/models/dagbag.py", line 181, in get_dag
self._add_dag_from_db(dag_id=dag_id, session=session)
File "/usr/local/lib/python3.8/site-packages/airflow/models/dagbag.py", line 237, in _add_dag_from_db
raise SerializedDagNotFound(f"DAG '{dag_id}' not found in serialized_dag table")
airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15607:
URL: https://github.com/apache/airflow/issues/15607#issuecomment-875576332
Closing it then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] neilharia7 edited a comment on issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
Posted by GitBox <gi...@apache.org>.
neilharia7 edited a comment on issue #15607:
URL: https://github.com/apache/airflow/issues/15607#issuecomment-889261261
I'm getting the same error with dynamic dags while running with airflow's docker-compose with `apache/airflow:2.1.2-python3.9`
```
Traceback (most recent call last):
airflow-scheduler_1 | File "/home/airflow/.local/lib/python3.9/site-packages/airflow/jobs/scheduler_job.py", line 1531, in _do_scheduling
airflow-scheduler_1 | self._schedule_dag_run(dag_run, active_runs_by_dag_id.get(dag_run.dag_id, set()), session)
airflow-scheduler_1 | File "/home/airflow/.local/lib/python3.9/site-packages/airflow/jobs/scheduler_job.py", line 1702, in _schedule_dag_run
airflow-scheduler_1 | dag = dag_run.dag = self.dagbag.get_dag(dag_run.dag_id, session=session)
airflow-scheduler_1 | File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/session.py", line 67, in wrapper
airflow-scheduler_1 | return func(*args, **kwargs)
airflow-scheduler_1 | File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dagbag.py", line 186, in get_dag
airflow-scheduler_1 | self._add_dag_from_db(dag_id=dag_id, session=session)
airflow-scheduler_1 | File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dagbag.py", line 258, in _add_dag_from_db
airflow-scheduler_1 | raise SerializedDagNotFound(f"DAG '{dag_id}' not found in serialized_dag table")
airflow-scheduler_1 | airflow.exceptions.SerializedDagNotFound: DAG 'cross-sell-v3-new' not found in serialized_dag table
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #15607:
URL: https://github.com/apache/airflow/issues/15607#issuecomment-866815170
@carlinix @hsnprsd Which Airflow version? Can you give us an example of how you generate DAGs for us to reproduce
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] warrenstephens commented on issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
Posted by GitBox <gi...@apache.org>.
warrenstephens commented on issue #15607:
URL: https://github.com/apache/airflow/issues/15607#issuecomment-875578356
Airflow webserver should probably handle the request without generating an Exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] warrenstephens commented on issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
Posted by GitBox <gi...@apache.org>.
warrenstephens commented on issue #15607:
URL: https://github.com/apache/airflow/issues/15607#issuecomment-875020003
Didn't find "tutorial" in the serialized_dag table either.
Here is what shows up in the docker-compose window:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #15607:
URL: https://github.com/apache/airflow/issues/15607
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] warrenstephens edited a comment on issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
Posted by GitBox <gi...@apache.org>.
warrenstephens edited a comment on issue #15607:
URL: https://github.com/apache/airflow/issues/15607#issuecomment-875567764
@kaxil
**I found the issue!**
It turns out that I had a minimized window open with the `http://localhost:8080/graph?dag_id=tutorial` in it! This was from doing the tutorial walkthru days prior with some other folks.
Now that I have closed that window the Airflow webserver exception no longer occurs (as shown in the screenshot above).
I guess that the minimized window was reconnecting to airflow and attempting to get info on the tutorial dag that it assumed was still serialized -- causing the airflow webserver to crash with the SerializedDagNotFound message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] warrenstephens edited a comment on issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
Posted by GitBox <gi...@apache.org>.
warrenstephens edited a comment on issue #15607:
URL: https://github.com/apache/airflow/issues/15607#issuecomment-875567764
@kaxil **I found the issue!**
It turns out that I had a minimized window open with the `http://localhost:8080/graph?dag_id=tutorial` in it! This was from doing the tutorial walkthru days prior with some other folks.
Now that I have closed that window the Airflow webserver exception no longer occurs (as shown in the screenshot above).
I guess that the minimized window was reconnecting to airflow and attempting to get info on the tutorial dag that it assumed was still serialized -- causing the airflow webserver to crash with the SerializedDagNotFound message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] warrenstephens commented on issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
Posted by GitBox <gi...@apache.org>.
warrenstephens commented on issue #15607:
URL: https://github.com/apache/airflow/issues/15607#issuecomment-875097100
Fixed the bizarre import error, and the DAG error still occurs -- which starts only after I login to the localhost:8080 interface.
```
airflow.exceptions.SerializedDagNotFound: DAG 'tutorial' not found in serialized_dag table
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] neilharia7 commented on issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
Posted by GitBox <gi...@apache.org>.
neilharia7 commented on issue #15607:
URL: https://github.com/apache/airflow/issues/15607#issuecomment-889261261
I'm getting the same error with dynamic dags while running with airflow's docker-compose with `apache/airflow:2.1.2-python3.9`
` Traceback (most recent call last):
airflow-scheduler_1 | File "/home/airflow/.local/lib/python3.9/site-packages/airflow/jobs/scheduler_job.py", line 1531, in _do_scheduling
airflow-scheduler_1 | self._schedule_dag_run(dag_run, active_runs_by_dag_id.get(dag_run.dag_id, set()), session)
airflow-scheduler_1 | File "/home/airflow/.local/lib/python3.9/site-packages/airflow/jobs/scheduler_job.py", line 1702, in _schedule_dag_run
airflow-scheduler_1 | dag = dag_run.dag = self.dagbag.get_dag(dag_run.dag_id, session=session)
airflow-scheduler_1 | File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/session.py", line 67, in wrapper
airflow-scheduler_1 | return func(*args, **kwargs)
airflow-scheduler_1 | File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dagbag.py", line 186, in get_dag
airflow-scheduler_1 | self._add_dag_from_db(dag_id=dag_id, session=session)
airflow-scheduler_1 | File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dagbag.py", line 258, in _add_dag_from_db
airflow-scheduler_1 | raise SerializedDagNotFound(f"DAG '{dag_id}' not found in serialized_dag table")
airflow-scheduler_1 | airflow.exceptions.SerializedDagNotFound: DAG 'cross-sell-v3-new' not found in serialized_dag table
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] carlinix commented on issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
Posted by GitBox <gi...@apache.org>.
carlinix commented on issue #15607:
URL: https://github.com/apache/airflow/issues/15607#issuecomment-866768557
I'm getting the same error. My dags are dynamic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15607:
URL: https://github.com/apache/airflow/issues/15607#issuecomment-875647978
> Airflow webserver should probably handle the request without generating an Exception.
Why do you think so ? Could you please elaborate?
I think showing an exception in logs is a good idea if someone sends a wrong request that triggers it. This is the default behaviour is some bad formed. wrong parameter, etc is sent, allowing to debug the problem when it happens. I do not honestly imagine better way of handling it. It gives a nice stack trace showing where it came from (and that it was generated by the UI). I think it's fair to expect that when a window from earlier version of the application runs in the background might have some "invalid" data cached (as it was in this case) that might trigger some exception - and it was totally unexpected (thus exception seems appropriate rather than handling it as a "normal" behaviour).
Do you have some concrete proposal how to improve such cases?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #15607:
URL: https://github.com/apache/airflow/issues/15607#issuecomment-889541719
@neilharia7 The problem for you seems like `cross-sell-v3-new` DAG does not exist when Airflow parses it sometimes, which is why Airflow removes it from the `serialized_dag` table. Increase `[processor_poll_interval](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#processor-poll-interval)` if you want to increase the interval on how frequently Airflow should parse your DAG
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] neilharia7 commented on issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
Posted by GitBox <gi...@apache.org>.
neilharia7 commented on issue #15607:
URL: https://github.com/apache/airflow/issues/15607#issuecomment-889282522
@kaxil
```
import datetime as dt
import json
import boto3
from airflow import DAG
from airflow.operators.dummy_operator import DummyOperator
from airflow.operators.python_operator import PythonOperator, BranchPythonOperator
from tasks_functions.custom_functions import customized_function
from zeus.config import *
from zeus.utils import *
default_args = {
'owner': 'neilharia7',
'start_date': dt.datetime(2020, 6, 26),
'retries': 1,
'retry_delay': 30
}
# fetch Dags from s3 bucket
s3_client = boto3.client('s3', region_name='ap-south-1')
flag, dag_information = False, dict()
try:
dag_information = json.loads(
s3_client.get_object(Bucket=Config.AWS.S3.bucket_name, Key=Config.AWS.S3.key_path)['Body'].read())
if dag_information:
flag = True
except Exception as e:
print(f'No dags registered {e}')
def number_of_keys(obj):
"""
# check if the number of keys are greater than 1
:param obj:
:return:
"""
if isinstance(obj, dict):
return True if len([k for k, v in obj.items()]) > 1 else False
if isinstance(obj, list):
return True if len(obj) > 1 else False
if flag:
for dag_data in dag_information.get('dag_structure', list()):
dag_registry = {
'owner': default_args.get('owner'),
'start_date': dt.datetime(2020, 6, 26),
'retries': dag_data.get('retries', default_args.get('retries')),
'retry_delay': dt.timedelta(seconds=dag_data.get('retry_delay', default_args.get('retry_delay'))),
'max_retry_delay': dt.timedelta(seconds=dag_data.get('max_retry_delay', 3600)),
'retry_exponential_backoff': dag_data.get('exponential_retry', True)
}
with DAG(
dag_id=dag_data.get('name', dag_data.get('dag_id')),
default_args=dag_registry,
schedule_interval=dag_data.get('scheduler', None)
) as dag:
# reverse mapping
data_list = dag_data.get('data', list())[::-1]
task_register = [dynamic_task_composer(task_data, dag) for task_data in data_list]
reverse_dict = {"data": data_list}
task_len = len(task_register)
# dynamic mapping
for child_idx, child_info in enumerate(reverse_dict['data']):
if child_info.get('parent_task'): # check if there are any parents of this task
for parent_idx, parent_info in enumerate(dag_data.get('data')):
if parent_info.get('task_name') in child_info.get('parent_task'):
task_register[child_idx] << task_register[task_len - parent_idx - 1]
# dynamic dag registration
globals()[dag_data.get('dag_id')] = dag
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil edited a comment on issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
Posted by GitBox <gi...@apache.org>.
kaxil edited a comment on issue #15607:
URL: https://github.com/apache/airflow/issues/15607#issuecomment-889541719
@neilharia7 The problem for you seems like `cross-sell-v3-new` DAG does not exist when Airflow parses it sometimes, which is why Airflow removes it from the `serialized_dag` table. Increase [`processor_poll_interval`](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#processor-poll-interval) if you want to increase the interval on how frequently Airflow should parse your DAG
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hsnprsd commented on issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
Posted by GitBox <gi...@apache.org>.
hsnprsd commented on issue #15607:
URL: https://github.com/apache/airflow/issues/15607#issuecomment-841803423
I'm getting the same error right now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] warrenstephens commented on issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
Posted by GitBox <gi...@apache.org>.
warrenstephens commented on issue #15607:
URL: https://github.com/apache/airflow/issues/15607#issuecomment-875567764
@kaxil **I found the issue!**
It turns out that I had a minimized window open with the `http://localhost:8080/graph?dag_id=tutorial` in it! This was from doing the tutorial walkthru days prior with some other folks.
Now that I have closed that window the Airflow webserver exception no longer occurs (as shown in the screenshot above).
I guess that the minimized window was reconnecting to airflow and attempting to get info on the tutorial dag that it assumed was still serialized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #15607:
URL: https://github.com/apache/airflow/issues/15607#issuecomment-874966311
@warrenstephens Check for the `tutorial` dag entry in `dag` table too. These tables are in Metadata DB (Postgres or MySQL) depending on what you are using in your docker-compose
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil edited a comment on issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
Posted by GitBox <gi...@apache.org>.
kaxil edited a comment on issue #15607:
URL: https://github.com/apache/airflow/issues/15607#issuecomment-889267511
Please post your DAG file, it won't be possible for us in any way to help without it @neilharia7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] warrenstephens commented on issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
Posted by GitBox <gi...@apache.org>.
warrenstephens commented on issue #15607:
URL: https://github.com/apache/airflow/issues/15607#issuecomment-875017440
Alrighty!
```
from airflow.models import DAG
from airflow.operators.bash_operator import BashOperator
from airflow.operators.dummy_operator import DummyOperator
from dug_helpers.dug_utils import DugUtil, get_topmed_files, get_dbgap_files
from roger.dag_util import default_args, create_python_task
DAG_ID = 'annotate_dug'
""" Build the workflow's tasks and DAG. """
with DAG(
dag_id=DAG_ID,
default_args=default_args,
schedule_interval=None
) as dag:
"""Build workflow tasks."""
intro = BashOperator(task_id='Intro',
bash_command='echo running tranql translator && exit 0',
dag=dag)
# make_kg_tagged = create_python_task(dag, "create_kgx_files", DugUtil.make_kg_tagged)
# Unzip and get files, avoid this because
# 1. it takes a bit of time making the dag itself, webserver hangs
# 2. Every task in this dag would still need to execute this part
# making it redundant
# 3. tasks like intro would fail because they don't have the data dir mounted.
get_topmed_files = create_python_task(dag, "get_topmed_data", get_topmed_files)
extract_db_gap_files = create_python_task(dag, "get_dbgab_data", get_dbgap_files)
annotate_topmed_files = create_python_task(dag, "annotate_topmed_files", DugUtil.annotate_topmed_files)
annotate_db_gap_files = create_python_task(dag, "annotate_db_gap_files", DugUtil.annotate_db_gap_files)
make_kg_tagged = create_python_task(dag, "make_tagged_kgx", DugUtil.make_kg_tagged)
dummy_stepover = DummyOperator(
task_id="continue",
)
intro >> [get_topmed_files, extract_db_gap_files] >> dummy_stepover >>\
[annotate_topmed_files, annotate_db_gap_files] >> make_kg_tagged
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hsnprsd edited a comment on issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
Posted by GitBox <gi...@apache.org>.
hsnprsd edited a comment on issue #15607:
URL: https://github.com/apache/airflow/issues/15607#issuecomment-841803423
I'm getting the same error right now. I was generating the dag dynamically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hsnprsd edited a comment on issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
Posted by GitBox <gi...@apache.org>.
hsnprsd edited a comment on issue #15607:
URL: https://github.com/apache/airflow/issues/15607#issuecomment-841803423
I'm getting the same error right now. I was generating the dag dynamically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15607:
URL: https://github.com/apache/airflow/issues/15607#issuecomment-878486738
> In a broader sense, it seems to me that when an error of "X not found" happens in most applications then no exception is generated, merely an error message stating that "X was not found".
Absolutely agree, it this is "expected" error. I.e. if there is an action from a user or known reason/flow that leads to it, you can - and should - provide a description and remediation,
Conversely, if something is the result of some totally unexpected condition which is likely a bug in the software or bug in migration process that was not possible to handle - it is virtually impossible to predict that it can happen. The statement that "all errors "SHOULD" be handled without generating an exception" takes the simplistic point of view that every single flow in the program is checked an handled and software is bug free. Which is not only not reasonable expectation it is also virtually impossible to prove from the software engineering point of view.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] warrenstephens commented on issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
Posted by GitBox <gi...@apache.org>.
warrenstephens commented on issue #15607:
URL: https://github.com/apache/airflow/issues/15607#issuecomment-875024881
This is from the annotate.py.log. Looks like some sort of bizarre import error to me.
```
[2021-07-06 16:23:52,806] {scheduler_job.py:182} INFO - Started process (PID=40) to work on /opt/airflow/dags/annotate.py
[2021-07-06 16:23:52,810] {scheduler_job.py:629} INFO - Processing file /opt/airflow/dags/annotate.py for tasks to queue
[2021-07-06 16:23:52,828] {logging_mixin.py:104} INFO - [2021-07-06 16:23:52,828] {dagbag.py:448} INFO - Filling up the DagBag from /opt/airflow/dags/annotate.py
[2021-07-06 16:24:22,863] {logging_mixin.py:104} INFO - [2021-07-06 16:24:22,856] {timeout.py:36} ERROR - Process timed out, PID: 40
[2021-07-06 16:24:22,916] {logging_mixin.py:104} INFO - [2021-07-06 16:24:22,873] {dagbag.py:305} ERROR - Failed to import: /opt/airflow/dags/annotate.py
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/dagbag.py", line 302, in _load_modules_from_file
loader.exec_module(new_module)
File "<frozen importlib._bootstrap_external>", line 783, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/opt/airflow/dags/annotate.py", line 5, in <module>
from dug_helpers.dug_utils import DugUtil, get_topmed_files, get_dbgap_files
File "/opt/airflow/dags/dug_helpers/dug_utils.py", line 21, in <module>
from roger.core import Util
File "/opt/airflow/dags/roger/core.py", line 14, in <module>
from bmt import Toolkit
File "/home/airflow/.local/lib/python3.8/site-packages/bmt/__init__.py", line 1, in <module>
from bmt.toolkit import Toolkit
File "/home/airflow/.local/lib/python3.8/site-packages/bmt/toolkit.py", line 5, in <module>
from biolinkml.meta import SchemaDefinition, Element, Definition, ClassDefinition, SlotDefinition
File "/home/airflow/.local/lib/python3.8/site-packages/biolinkml/meta.py", line 16, in <module>
from biolinkml.utils.metamodelcore import empty_list, empty_dict, bnode
File "/home/airflow/.local/lib/python3.8/site-packages/biolinkml/utils/metamodelcore.py", line 9, in <module>
from ShExJSG.ShExJ import IRIREF, PN_PREFIX
File "/home/airflow/.local/lib/python3.8/site-packages/ShExJSG/__init__.py", line 1, in <module>
from ShExJSG.SchemaWithContext import Schema
File "/home/airflow/.local/lib/python3.8/site-packages/ShExJSG/SchemaWithContext.py", line 3, in <module>
from ShExJSG import ShExJ
File "/home/airflow/.local/lib/python3.8/site-packages/ShExJSG/ShExJ.py", line 55, in <module>
class PN_PREFIX(jsg.JSGString):
File "/home/airflow/.local/lib/python3.8/site-packages/ShExJSG/ShExJ.py", line 56, in PN_PREFIX
pattern = jsg.JSGPattern(r'({PN_CHARS_BASE})((({PN_CHARS})|\.)*({PN_CHARS}))?'.format(PN_CHARS=PN_CHARS.pattern, PN_CHARS_BASE=PN_CHARS_BASE.pattern))
File "/home/airflow/.local/lib/python3.8/site-packages/pyjsg/jsglib/jsg_strings.py", line 18, in __init__
self.pattern_re = re.compile(pattern, flags=re.DOTALL)
File "/usr/local/lib/python3.8/re.py", line 252, in compile
return _compile(pattern, flags)
File "/usr/local/lib/python3.8/re.py", line 304, in _compile
p = sre_compile.compile(pattern, flags)
File "/usr/local/lib/python3.8/sre_compile.py", line 768, in compile
code = _code(p, flags)
File "/usr/local/lib/python3.8/sre_compile.py", line 604, in _code
_compile_info(code, p, flags)
File "/usr/local/lib/python3.8/sre_compile.py", line 590, in _compile_info
charset, hascased = _optimize_charset(charset)
File "/usr/local/lib/python3.8/sre_compile.py", line 312, in _optimize_charset
charmap[i] = 1
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/timeout.py", line 37, in handle_timeout
raise AirflowTaskTimeout(self.error_message)
airflow.exceptions.AirflowTaskTimeout: DagBag import timeout for /opt/airflow/dags/annotate.py after 30.0s, PID: 40
[2021-07-06 16:24:22,925] {scheduler_job.py:641} WARNING - No viable dags retrieved from /opt/airflow/dags/annotate.py
[2021-07-06 16:24:23,127] {scheduler_job.py:190} INFO - Processing /opt/airflow/dags/annotate.py took 30.356 seconds
[2021-07-06 16:24:54,072] {scheduler_job.py:182} INFO - Started process (PID=50) to work on /opt/airflow/dags/annotate.py
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #15607:
URL: https://github.com/apache/airflow/issues/15607#issuecomment-889267511
> I'm getting the same error with dynamic dags while running with airflow's docker-compose with `apache/airflow:2.1.2-python3.9`
>
> ```
> Traceback (most recent call last):
> airflow-scheduler_1 | File "/home/airflow/.local/lib/python3.9/site-packages/airflow/jobs/scheduler_job.py", line 1531, in _do_scheduling
> airflow-scheduler_1 | self._schedule_dag_run(dag_run, active_runs_by_dag_id.get(dag_run.dag_id, set()), session)
> airflow-scheduler_1 | File "/home/airflow/.local/lib/python3.9/site-packages/airflow/jobs/scheduler_job.py", line 1702, in _schedule_dag_run
> airflow-scheduler_1 | dag = dag_run.dag = self.dagbag.get_dag(dag_run.dag_id, session=session)
> airflow-scheduler_1 | File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/session.py", line 67, in wrapper
> airflow-scheduler_1 | return func(*args, **kwargs)
> airflow-scheduler_1 | File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dagbag.py", line 186, in get_dag
> airflow-scheduler_1 | self._add_dag_from_db(dag_id=dag_id, session=session)
> airflow-scheduler_1 | File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dagbag.py", line 258, in _add_dag_from_db
> airflow-scheduler_1 | raise SerializedDagNotFound(f"DAG '{dag_id}' not found in serialized_dag table")
> airflow-scheduler_1 | airflow.exceptions.SerializedDagNotFound: DAG 'cross-sell-v3-new' not found in serialized_dag table
> ```
Please post your DAG file, it won't be possible for us in any way to help without it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] warrenstephens commented on issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
Posted by GitBox <gi...@apache.org>.
warrenstephens commented on issue #15607:
URL: https://github.com/apache/airflow/issues/15607#issuecomment-874943564
Using airflow 2.1.1 with docker-compose as well.
I would love to be able to clean out this DAG 'tutorial' stuff, but I have no idea where the "serialized dag table" is! How do I get to it?
```
airflow-webserver_1 | 172.20.0.1 - - [06/Jul/2021:17:15:43 +0000] "GET /object/task_instances?dag_id=annotate_dug&execution_date=2021-07-02T21%3A19%3A24.504747%2B00%3A00 HTTP/1.1" 200 4564 "http://localhost:8080/graph?dag_id=annotate_dug" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36"
airflow-webserver_1 | [2021-07-06 17:15:45,148] {app.py:1891} ERROR - Exception on /object/task_instances [GET]
airflow-webserver_1 | Traceback (most recent call last):
airflow-webserver_1 | File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
airflow-webserver_1 | response = self.full_dispatch_request()
airflow-webserver_1 | File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
airflow-webserver_1 | rv = self.handle_user_exception(e)
airflow-webserver_1 | File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
airflow-webserver_1 | reraise(exc_type, exc_value, tb)
airflow-webserver_1 | File "/home/airflow/.local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
airflow-webserver_1 | raise value
airflow-webserver_1 | File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
airflow-webserver_1 | rv = self.dispatch_request()
airflow-webserver_1 | File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
airflow-webserver_1 | return self.view_functions[rule.endpoint](**req.view_args)
airflow-webserver_1 | File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/auth.py", line 34, in decorated
airflow-webserver_1 | return func(*args, **kwargs)
airflow-webserver_1 | File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/decorators.py", line 60, in wrapper
airflow-webserver_1 | return f(*args, **kwargs)
airflow-webserver_1 | File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/views.py", line 2611, in task_instances
airflow-webserver_1 | dag = current_app.dag_bag.get_dag(dag_id)
airflow-webserver_1 | File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/session.py", line 65, in wrapper
airflow-webserver_1 | return func(*args, session=session, **kwargs)
airflow-webserver_1 | File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/dagbag.py", line 178, in get_dag
airflow-webserver_1 | self._add_dag_from_db(dag_id=dag_id, session=session)
airflow-webserver_1 | File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/dagbag.py", line 234, in _add_dag_from_db
airflow-webserver_1 | raise SerializedDagNotFound(f"DAG '{dag_id}' not found in serialized_dag table")
airflow-webserver_1 | airflow.exceptions.SerializedDagNotFound: DAG 'tutorial' not found in serialized_dag table
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #15607:
URL: https://github.com/apache/airflow/issues/15607#issuecomment-875013182
Can you post the code of your DAG with dag_id `annotate_dug`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hsnprsd commented on issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
Posted by GitBox <gi...@apache.org>.
hsnprsd commented on issue #15607:
URL: https://github.com/apache/airflow/issues/15607#issuecomment-841803423
I'm getting the same error right now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] christine-le commented on issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
Posted by GitBox <gi...@apache.org>.
christine-le commented on issue #15607:
URL: https://github.com/apache/airflow/issues/15607#issuecomment-911984237
Just wanted to chime in to say we ran into the exact same error under pretty similar set up as @neilharia7 with a set of dynamic dags. Setting processor_poll_interval to 30 seconds (up from the default of 1 second) resolved the error for us.
We may play around with this number a bit and see if we can reduce this number a bit. We have a couple thousand dags, many of which are dynamically generated, so may just need more time for parsing.
Details of our set up:
Python 3.6
Airflow 2.0.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #15607:
URL: https://github.com/apache/airflow/issues/15607#issuecomment-830030384
Are you generating the DAG Dynamically? Can you post your DAG here please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] trollhe commented on issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
Posted by GitBox <gi...@apache.org>.
trollhe commented on issue #15607:
URL: https://github.com/apache/airflow/issues/15607#issuecomment-829738425
only webserver and schedule service in one container,else have the error.but need modify dag file,i don`t know how can serialized_dag after dag file modify.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] warrenstephens commented on issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
Posted by GitBox <gi...@apache.org>.
warrenstephens commented on issue #15607:
URL: https://github.com/apache/airflow/issues/15607#issuecomment-875006056
I fought my way into the running docker container as the "airflow" user, and listed the tables, one of which was "dag", selected * from dag, which showed 3 dag_id values -- none of which were "tutorial" -- they were the dags that I expected.
So where is the "tutorial" dag?
For those playing along at home, here is how I got into the running postgres container:
```
docker exec -it roger_postgres_1 /bin/bash
```
where I found "roger_postgres_1" by running "docker ps" at the command line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] warrenstephens commented on issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table
Posted by GitBox <gi...@apache.org>.
warrenstephens commented on issue #15607:
URL: https://github.com/apache/airflow/issues/15607#issuecomment-878254018
I guess it is my misunderstanding -- as I am new to airflow in the first place. The error looked to me as if the airflow webserver itself was crashing, rather than something on the user side. To me it is not at all clear that the error came from the UI. It appears that other people are similarly confused.
In a broader sense, it seems to me that when an error of "X not found" happens in most applications then no exception is generated, merely an error message stating that "X was not found".
I do not see in the exception that was generated what would lead me to understand what was actually happening.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org