You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Dennis O'Brien (JIRA)" <ji...@apache.org> on 2016/12/01 21:42:58 UTC
[jira] [Updated] (AIRFLOW-664) airflow built from master unstable
since change to parse DAG files in child process
[ https://issues.apache.org/jira/browse/AIRFLOW-664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dennis O'Brien updated AIRFLOW-664:
-----------------------------------
Description:
Prior to commit fdb7e949140b735b8554ae5b22ad752e86f6ebaf I was able to pip build, install, and run airflow from the github commit. E.g.,
{code}
pip install -e git://github.com/apache/incubator-airflow.git@835bcb6234956445d819ff39a5790f635c201c5d#egg=airflow[celery,crypto,hive,jdbc,ldap,password,postgres,s3,vertica]
{code}
Since this commit, airflow is not functional.
* The on/off button to enable a DAG does not appear.
* Next to each DAG an info icon appears with the hover text "This DAG seems to be existing only locally. The master scheduler doesn't seem to be aware of its existence."
* DAGs cannot be run from the command line.
I am able via the command line to list the dags and list the tasks for a dag, but trigger_dag does not actually cause any tasks to run (but no error messages either).
{code}
airflow@41e0e6920223:~$ airflow trigger_dag example_bash_operator
[2016-12-01 19:24:21,928] {__init__.py:50} INFO - Using executor CeleryExecutor
[2016-12-01 19:24:22,218] {models.py:162} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-01 19:24:22,341] {cli.py:180} INFO - Created <DagRun example_bash_operator @ 2016-12-01 19:24:22: manual__2016-12-01T19:24:22.295709, externally triggered: True>
{code}
This commit was #1636 "[AIRFLOW-160] Parse DAG files through child processes". I haven't identified what part of this commit is to blame. Or, what part of my configuration/setup is incompatible with this change.
My setup:
* Docker and docker-compose with...
* Ubuntu 16.04
* Python 3.5.2 (via Miniconda distribution)
* Separate container instances for scheduler, webserver, flower, and worker.
* Docker container running mysql 5.7
* Docker container running redis 3.2
* Using CeleryExecutor with redis broker
* Building and installing from master using `pip install -e git://...`
I have a github project that uses Docker and docker-compose to build and test these components. [https://github.com/dennisobrien/airflow-docker-compose-example] .
* pip install line in [Dockerfile|https://github.com/dennisobrien/airflow-docker-compose-example/blob/master/Dockerfile#L58]
* [airflow.cfg template|https://github.com/dennisobrien/airflow-docker-compose-example/blob/master/config/airflow.cfg.tpl]
* [docker-entrypoint.sh|https://github.com/dennisobrien/airflow-docker-compose-example/blob/master/bin/docker-entrypoint.sh] defines how each service is run.
* [docker-compose.yml|https://github.com/dennisobrien/airflow-docker-compose-example/blob/master/docker-compose.yml] orchestrates the components.
Let me know if there are additional tests I can do to narrow this down. I suspect since others have not complained about this that it is probably something related to my setup.
was:
Prior to commit fdb7e949140b735b8554ae5b22ad752e86f6ebaf I was able to pip build, install, and run airflow from the github commit. E.g.,
{code}
pip install -e git://github.com/apache/incubator-airflow.git@835bcb6234956445d819ff39a5790f635c201c5d#egg=airflow[celery,crypto,hive,jdbc,ldap,password,postgres,s3,vertica]
{code}
Since this commit, airflow is not functional.
* The on/off button to enable a button does not appear.
* Next to each DAG an info icon appears with the hover text "This DAG seems to be existing only locally. The master scheduler doesn't seem to be aware of its existence."
* DAGs cannot be run from the command line.
I am able via the command line to list the dags and list the tasks for a dag, but trigger_dag does not actually cause any tasks to run (but no error messages either).
{code}
airflow@41e0e6920223:~$ airflow trigger_dag example_bash_operator
[2016-12-01 19:24:21,928] {__init__.py:50} INFO - Using executor CeleryExecutor
[2016-12-01 19:24:22,218] {models.py:162} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-01 19:24:22,341] {cli.py:180} INFO - Created <DagRun example_bash_operator @ 2016-12-01 19:24:22: manual__2016-12-01T19:24:22.295709, externally triggered: True>
{code}
This commit was #1636 "[AIRFLOW-160] Parse DAG files through child processes". I haven't identified what part of this commit is to blame. Or, what part of my configuration/setup is incompatible with this change.
My setup:
* Docker and docker-compose with...
* Ubuntu 16.04
* Python 3.5.2 (via Miniconda distribution)
* Separate container instances for scheduler, webserver, flower, and worker.
* Docker container running mysql 5.7
* Docker container running redis 3.2
* Using CeleryExecutor with redis broker
* Building and installing from master using `pip install -e git://...`
I have a github project that uses Docker and docker-compose to build and test these components. [https://github.com/dennisobrien/airflow-docker-compose-example] .
* pip install line in [Dockerfile|https://github.com/dennisobrien/airflow-docker-compose-example/blob/master/Dockerfile#L58]
* [airflow.cfg template|https://github.com/dennisobrien/airflow-docker-compose-example/blob/master/config/airflow.cfg.tpl]
* [docker-entrypoint.sh|https://github.com/dennisobrien/airflow-docker-compose-example/blob/master/bin/docker-entrypoint.sh] defines how each service is run.
* [docker-compose.yml|https://github.com/dennisobrien/airflow-docker-compose-example/blob/master/docker-compose.yml] orchestrates the components.
Let me know if there are additional tests I can do to narrow this down. I suspect since others have not complained about this that it is probably something related to my setup.
> airflow built from master unstable since change to parse DAG files in child process
> -----------------------------------------------------------------------------------
>
> Key: AIRFLOW-664
> URL: https://issues.apache.org/jira/browse/AIRFLOW-664
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
> Reporter: Dennis O'Brien
>
> Prior to commit fdb7e949140b735b8554ae5b22ad752e86f6ebaf I was able to pip build, install, and run airflow from the github commit. E.g.,
> {code}
> pip install -e git://github.com/apache/incubator-airflow.git@835bcb6234956445d819ff39a5790f635c201c5d#egg=airflow[celery,crypto,hive,jdbc,ldap,password,postgres,s3,vertica]
> {code}
> Since this commit, airflow is not functional.
> * The on/off button to enable a DAG does not appear.
> * Next to each DAG an info icon appears with the hover text "This DAG seems to be existing only locally. The master scheduler doesn't seem to be aware of its existence."
> * DAGs cannot be run from the command line.
> I am able via the command line to list the dags and list the tasks for a dag, but trigger_dag does not actually cause any tasks to run (but no error messages either).
> {code}
> airflow@41e0e6920223:~$ airflow trigger_dag example_bash_operator
> [2016-12-01 19:24:21,928] {__init__.py:50} INFO - Using executor CeleryExecutor
> [2016-12-01 19:24:22,218] {models.py:162} INFO - Filling up the DagBag from /usr/local/airflow/dags
> [2016-12-01 19:24:22,341] {cli.py:180} INFO - Created <DagRun example_bash_operator @ 2016-12-01 19:24:22: manual__2016-12-01T19:24:22.295709, externally triggered: True>
> {code}
> This commit was #1636 "[AIRFLOW-160] Parse DAG files through child processes". I haven't identified what part of this commit is to blame. Or, what part of my configuration/setup is incompatible with this change.
> My setup:
> * Docker and docker-compose with...
> * Ubuntu 16.04
> * Python 3.5.2 (via Miniconda distribution)
> * Separate container instances for scheduler, webserver, flower, and worker.
> * Docker container running mysql 5.7
> * Docker container running redis 3.2
> * Using CeleryExecutor with redis broker
> * Building and installing from master using `pip install -e git://...`
> I have a github project that uses Docker and docker-compose to build and test these components. [https://github.com/dennisobrien/airflow-docker-compose-example] .
> * pip install line in [Dockerfile|https://github.com/dennisobrien/airflow-docker-compose-example/blob/master/Dockerfile#L58]
> * [airflow.cfg template|https://github.com/dennisobrien/airflow-docker-compose-example/blob/master/config/airflow.cfg.tpl]
> * [docker-entrypoint.sh|https://github.com/dennisobrien/airflow-docker-compose-example/blob/master/bin/docker-entrypoint.sh] defines how each service is run.
> * [docker-compose.yml|https://github.com/dennisobrien/airflow-docker-compose-example/blob/master/docker-compose.yml] orchestrates the components.
> Let me know if there are additional tests I can do to narrow this down. I suspect since others have not complained about this that it is probably something related to my setup.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)