You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Ben Laird <br...@gmail.com> on 2018/08/09 19:35:32 UTC
Broken DAG message won't go away in webserver
Hello -
I've noticed this several times and not sure what the solution is. If I
have a DAG error at some point, I'll see message in the webserver that says
"Broken DAG: [Error]". However, after fixing the code, restarting the
webserver, etc, the error persists. After closing it out, it will just pop
up again after reloading.
The only way I was able to delete was by doing a `airflow resetdb`. I'd
like to avoid manually deleting records from the DB, as now in prod we
cannot just kill the DB state.
Any suggestions?
Thanks,
Ben Laird
Re: Broken DAG message won't go away in webserver
Posted by Dan Davydov <dd...@twitter.com.INVALID>.
The scheduler should clear import errors for DAGfiles that no longer exist
or DAGs that have fixed the import errors. Take a look at
https://github.com/apache/incubator-airflow/blob/master/airflow/jobs.py#L761
and where it is called.
If it's not working there might be a bug.
On Fri, Aug 10, 2018 at 10:19 AM Ben Laird <br...@gmail.com> wrote:
> What is the desired behavior then? If I fix an error in a DAG, there should
> be a way to clear the state in the error table. Since these errors pop up
> in the webserver logs, and restarting the webserver with no error signals
> that the DAGs are healthy, should restarting the webserver clear the error
> table? Or have a CLI command as a fallback?
>
>
> On Thu, Aug 9, 2018 at 5:35 PM, Alex Guziel <alex.guziel@airbnb.com
> .invalid>
> wrote:
>
> > IIRC the scheduler sets these messages in the error table in the db.
> >
> > On Thu, Aug 9, 2018 at 2:13 PM, Ben Laird <br...@gmail.com> wrote:
> >
> > > The messages persist even after restarting the webserver. I've verified
> > > with other airflow users in the office that they'd have to manually
> > delete
> > > records from the 'import_error' table.
> > >
> > > When you say 'sync your DAGs', what do you mean exactly? When we fix a
> > DAG,
> > > we'd normally kill the webserver process, push a zip containing our dag
> > > directory (with the fixed code), unzip and restart the webserver.
> > >
> > > Thanks
> > >
> > > On Thu, Aug 9, 2018 at 4:43 PM, Taylor Edmiston <te...@gmail.com>
> > > wrote:
> > >
> > > > Yeah, you definitely shouldn't need to do a resetdb for that.
> > > >
> > > > Did you try restarting the webserver?
> > > >
> > > > How do you sync your DAGs to the webserver? Is it possible the fixed
> > DAG
> > > > didn't get synced there?
> > > >
> > > > For me, IIRC, the error stops persisting once the DAG is fixed and
> > > synced.
> > > >
> > > > *Taylor Edmiston*
> > > > Blog <https://blog.tedmiston.com/> | CV
> > > > <https://stackoverflow.com/cv/taylor> | LinkedIn
> > > > <https://www.linkedin.com/in/tedmiston/> | AngelList
> > > > <https://angel.co/taylor> | Stack Overflow
> > > > <https://stackoverflow.com/users/149428/taylor-edmiston>
> > > >
> > > >
> > > > On Thu, Aug 9, 2018 at 3:35 PM, Ben Laird <br...@gmail.com>
> wrote:
> > > >
> > > > > Hello -
> > > > >
> > > > > I've noticed this several times and not sure what the solution is.
> > If I
> > > > > have a DAG error at some point, I'll see message in the webserver
> > that
> > > > says
> > > > > "Broken DAG: [Error]". However, after fixing the code, restarting
> the
> > > > > webserver, etc, the error persists. After closing it out, it will
> > just
> > > > pop
> > > > > up again after reloading.
> > > > >
> > > > > The only way I was able to delete was by doing a `airflow resetdb`.
> > I'd
> > > > > like to avoid manually deleting records from the DB, as now in prod
> > we
> > > > > cannot just kill the DB state.
> > > > >
> > > > > Any suggestions?
> > > > >
> > > > > Thanks,
> > > > > Ben Laird
> > > > >
> > > >
> > >
> >
>
Re: Broken DAG message won't go away in webserver
Posted by Ben Laird <br...@gmail.com>.
What is the desired behavior then? If I fix an error in a DAG, there should
be a way to clear the state in the error table. Since these errors pop up
in the webserver logs, and restarting the webserver with no error signals
that the DAGs are healthy, should restarting the webserver clear the error
table? Or have a CLI command as a fallback?
On Thu, Aug 9, 2018 at 5:35 PM, Alex Guziel <al...@airbnb.com.invalid>
wrote:
> IIRC the scheduler sets these messages in the error table in the db.
>
> On Thu, Aug 9, 2018 at 2:13 PM, Ben Laird <br...@gmail.com> wrote:
>
> > The messages persist even after restarting the webserver. I've verified
> > with other airflow users in the office that they'd have to manually
> delete
> > records from the 'import_error' table.
> >
> > When you say 'sync your DAGs', what do you mean exactly? When we fix a
> DAG,
> > we'd normally kill the webserver process, push a zip containing our dag
> > directory (with the fixed code), unzip and restart the webserver.
> >
> > Thanks
> >
> > On Thu, Aug 9, 2018 at 4:43 PM, Taylor Edmiston <te...@gmail.com>
> > wrote:
> >
> > > Yeah, you definitely shouldn't need to do a resetdb for that.
> > >
> > > Did you try restarting the webserver?
> > >
> > > How do you sync your DAGs to the webserver? Is it possible the fixed
> DAG
> > > didn't get synced there?
> > >
> > > For me, IIRC, the error stops persisting once the DAG is fixed and
> > synced.
> > >
> > > *Taylor Edmiston*
> > > Blog <https://blog.tedmiston.com/> | CV
> > > <https://stackoverflow.com/cv/taylor> | LinkedIn
> > > <https://www.linkedin.com/in/tedmiston/> | AngelList
> > > <https://angel.co/taylor> | Stack Overflow
> > > <https://stackoverflow.com/users/149428/taylor-edmiston>
> > >
> > >
> > > On Thu, Aug 9, 2018 at 3:35 PM, Ben Laird <br...@gmail.com> wrote:
> > >
> > > > Hello -
> > > >
> > > > I've noticed this several times and not sure what the solution is.
> If I
> > > > have a DAG error at some point, I'll see message in the webserver
> that
> > > says
> > > > "Broken DAG: [Error]". However, after fixing the code, restarting the
> > > > webserver, etc, the error persists. After closing it out, it will
> just
> > > pop
> > > > up again after reloading.
> > > >
> > > > The only way I was able to delete was by doing a `airflow resetdb`.
> I'd
> > > > like to avoid manually deleting records from the DB, as now in prod
> we
> > > > cannot just kill the DB state.
> > > >
> > > > Any suggestions?
> > > >
> > > > Thanks,
> > > > Ben Laird
> > > >
> > >
> >
>
Re: Broken DAG message won't go away in webserver
Posted by Alex Guziel <al...@airbnb.com.INVALID>.
IIRC the scheduler sets these messages in the error table in the db.
On Thu, Aug 9, 2018 at 2:13 PM, Ben Laird <br...@gmail.com> wrote:
> The messages persist even after restarting the webserver. I've verified
> with other airflow users in the office that they'd have to manually delete
> records from the 'import_error' table.
>
> When you say 'sync your DAGs', what do you mean exactly? When we fix a DAG,
> we'd normally kill the webserver process, push a zip containing our dag
> directory (with the fixed code), unzip and restart the webserver.
>
> Thanks
>
> On Thu, Aug 9, 2018 at 4:43 PM, Taylor Edmiston <te...@gmail.com>
> wrote:
>
> > Yeah, you definitely shouldn't need to do a resetdb for that.
> >
> > Did you try restarting the webserver?
> >
> > How do you sync your DAGs to the webserver? Is it possible the fixed DAG
> > didn't get synced there?
> >
> > For me, IIRC, the error stops persisting once the DAG is fixed and
> synced.
> >
> > *Taylor Edmiston*
> > Blog <https://blog.tedmiston.com/> | CV
> > <https://stackoverflow.com/cv/taylor> | LinkedIn
> > <https://www.linkedin.com/in/tedmiston/> | AngelList
> > <https://angel.co/taylor> | Stack Overflow
> > <https://stackoverflow.com/users/149428/taylor-edmiston>
> >
> >
> > On Thu, Aug 9, 2018 at 3:35 PM, Ben Laird <br...@gmail.com> wrote:
> >
> > > Hello -
> > >
> > > I've noticed this several times and not sure what the solution is. If I
> > > have a DAG error at some point, I'll see message in the webserver that
> > says
> > > "Broken DAG: [Error]". However, after fixing the code, restarting the
> > > webserver, etc, the error persists. After closing it out, it will just
> > pop
> > > up again after reloading.
> > >
> > > The only way I was able to delete was by doing a `airflow resetdb`. I'd
> > > like to avoid manually deleting records from the DB, as now in prod we
> > > cannot just kill the DB state.
> > >
> > > Any suggestions?
> > >
> > > Thanks,
> > > Ben Laird
> > >
> >
>
Re: Broken DAG message won't go away in webserver
Posted by Ben Laird <br...@gmail.com>.
The messages persist even after restarting the webserver. I've verified
with other airflow users in the office that they'd have to manually delete
records from the 'import_error' table.
When you say 'sync your DAGs', what do you mean exactly? When we fix a DAG,
we'd normally kill the webserver process, push a zip containing our dag
directory (with the fixed code), unzip and restart the webserver.
Thanks
On Thu, Aug 9, 2018 at 4:43 PM, Taylor Edmiston <te...@gmail.com> wrote:
> Yeah, you definitely shouldn't need to do a resetdb for that.
>
> Did you try restarting the webserver?
>
> How do you sync your DAGs to the webserver? Is it possible the fixed DAG
> didn't get synced there?
>
> For me, IIRC, the error stops persisting once the DAG is fixed and synced.
>
> *Taylor Edmiston*
> Blog <https://blog.tedmiston.com/> | CV
> <https://stackoverflow.com/cv/taylor> | LinkedIn
> <https://www.linkedin.com/in/tedmiston/> | AngelList
> <https://angel.co/taylor> | Stack Overflow
> <https://stackoverflow.com/users/149428/taylor-edmiston>
>
>
> On Thu, Aug 9, 2018 at 3:35 PM, Ben Laird <br...@gmail.com> wrote:
>
> > Hello -
> >
> > I've noticed this several times and not sure what the solution is. If I
> > have a DAG error at some point, I'll see message in the webserver that
> says
> > "Broken DAG: [Error]". However, after fixing the code, restarting the
> > webserver, etc, the error persists. After closing it out, it will just
> pop
> > up again after reloading.
> >
> > The only way I was able to delete was by doing a `airflow resetdb`. I'd
> > like to avoid manually deleting records from the DB, as now in prod we
> > cannot just kill the DB state.
> >
> > Any suggestions?
> >
> > Thanks,
> > Ben Laird
> >
>
Re: Broken DAG message won't go away in webserver
Posted by Taylor Edmiston <te...@gmail.com>.
Yeah, you definitely shouldn't need to do a resetdb for that.
Did you try restarting the webserver?
How do you sync your DAGs to the webserver? Is it possible the fixed DAG
didn't get synced there?
For me, IIRC, the error stops persisting once the DAG is fixed and synced.
*Taylor Edmiston*
Blog <https://blog.tedmiston.com/> | CV
<https://stackoverflow.com/cv/taylor> | LinkedIn
<https://www.linkedin.com/in/tedmiston/> | AngelList
<https://angel.co/taylor> | Stack Overflow
<https://stackoverflow.com/users/149428/taylor-edmiston>
On Thu, Aug 9, 2018 at 3:35 PM, Ben Laird <br...@gmail.com> wrote:
> Hello -
>
> I've noticed this several times and not sure what the solution is. If I
> have a DAG error at some point, I'll see message in the webserver that says
> "Broken DAG: [Error]". However, after fixing the code, restarting the
> webserver, etc, the error persists. After closing it out, it will just pop
> up again after reloading.
>
> The only way I was able to delete was by doing a `airflow resetdb`. I'd
> like to avoid manually deleting records from the DB, as now in prod we
> cannot just kill the DB state.
>
> Any suggestions?
>
> Thanks,
> Ben Laird
>