You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@airflow.apache.org by Chris Palmer <ch...@crpalmer.com> on 2020/05/01 15:08:44 UTC

Testing DAG parsing/Airflow CLI exit codes

Hi all,

I'm wondering if others have experience with testing of DAG parsing. We use
CircleCI to automate deployment of DAGs to our environments, and I'd like
to add some simple test steps to check things like can Airflow parse all
the DAGs without errors, i.e. running `airflow list_dags` without errors.

I'm working with Airflow 1.10.6 at the moment. I was hoping that the CLI
would exit with useful return codes but that doesn't appear to be the case.

Does that mean I'm stuck capturing the output of the command and looking
for strings like 'failed to import'?

Anyone have experience with this?

Thanks
Chris

Re: Testing DAG parsing/Airflow CLI exit codes

Posted by Chris Palmer <ch...@crpalmer.com>.

Thanks Kamil,

That makes a lot of sense. Looks like that documentation section doesn't
exist in 1.10.6 (or earlier) so I hadn't seen it.

Chris

On Fri, May 1, 2020 at 11:27 AM Kamil Breguła <ka...@polidea.com>
wrote:

> Hello,
>
> It is best to write unit tests that will check that the files are correct.
>
> https://airflow.readthedocs.io/en/latest/best-practices.html?highlight=testing#unit-tests
>
> https://github.com/apache/airflow/blob/4421f011eeec2d1022a39933e27f530fb9f9c1b1/tests/test_example_dags.py#L34
>
> Best regards,
> Kamil
>
> On Fri, May 1, 2020 at 5:09 PM Chris Palmer <ch...@crpalmer.com> wrote:
> >
> > Hi all,
> >
> > I'm wondering if others have experience with testing of DAG parsing. We
> use CircleCI to automate deployment of DAGs to our environments, and I'd
> like to add some simple test steps to check things like can Airflow parse
> all the DAGs without errors, i.e. running `airflow list_dags` without
> errors.
> >
> > I'm working with Airflow 1.10.6 at the moment. I was hoping that the CLI
> would exit with useful return codes but that doesn't appear to be the case.
> >
> > Does that mean I'm stuck capturing the output of the command and looking
> for strings like 'failed to import'?
> >
> > Anyone have experience with this?
> >
> > Thanks
> > Chris
> >
> >
>

Re: Testing DAG parsing/Airflow CLI exit codes

Posted by Kamil Breguła <ka...@polidea.com>.

Hello,

It is best to write unit tests that will check that the files are correct.
https://airflow.readthedocs.io/en/latest/best-practices.html?highlight=testing#unit-tests
https://github.com/apache/airflow/blob/4421f011eeec2d1022a39933e27f530fb9f9c1b1/tests/test_example_dags.py#L34

Best regards,
Kamil

On Fri, May 1, 2020 at 5:09 PM Chris Palmer <ch...@crpalmer.com> wrote:
>
> Hi all,
>
> I'm wondering if others have experience with testing of DAG parsing. We use CircleCI to automate deployment of DAGs to our environments, and I'd like to add some simple test steps to check things like can Airflow parse all the DAGs without errors, i.e. running `airflow list_dags` without errors.
>
> I'm working with Airflow 1.10.6 at the moment. I was hoping that the CLI would exit with useful return codes but that doesn't appear to be the case.
>
> Does that mean I'm stuck capturing the output of the command and looking for strings like 'failed to import'?
>
> Anyone have experience with this?
>
> Thanks
> Chris
>
>