You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Jon Davies (JIRA)" <ji...@apache.org> on 2018/07/10 13:11:00 UTC
[jira] [Updated] (AIRFLOW-2741) Example Kubernetes pod operator
doesn't work
[ https://issues.apache.org/jira/browse/AIRFLOW-2741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jon Davies updated AIRFLOW-2741:
--------------------------------
Description:
I'm trying to make the DAG from this blog post work:
https://www.techatbloomberg.com/blog/airflow-on-kubernetes/
But it fails with:
{code}
airflow-test/airflow-5cc9cf9f99-v6rtd[scheduler]: [2018-07-10 13:00:41,375] {kubernetes_executor.py:579} INFO - self.running: {(u'kubernetes_sample', u'run_this_first', datetime.datetime(2018, 7, 10, 13, 0, 17, 332338, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=0, name=None))): u'airflow run kubernetes_sample run_this_first 2018-07-10T13:00:17.332338+00:00 --local -sd /root/airflow/dags'}
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: + airflow run kubernetes_sample run_this_first 2018-07-10T13:00:17.332338+00:00 --local -sd /root/airflow/dags
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: [2018-07-10 13:00:57,667] {settings.py:174} INFO - setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: [2018-07-10 13:00:57,919] {__init__.py:51} INFO - Using executor LocalExecutor
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: [2018-07-10 13:00:58,023] {models.py:257} INFO - Filling up the DagBag from /root/airflow/dags
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: [2018-07-10 13:00:58,029] {models.py:348} INFO - File /usr/local/lib/python2.7/dist-packages/airflow/example_dags/__init__.py assumed to contain no DAGs. Skipping.
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: Traceback (most recent call last):
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: File "/usr/local/bin/airflow", line 32, in <module>
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: args.func(args)
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: File "/usr/local/lib/python2.7/dist-packages/airflow/utils/cli.py", line 74, in wrapper
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: return f(*args, **kwargs)
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 475, in run
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: dag = get_dag(args)
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 146, in get_dag
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: 'parse.'.format(args.dag_id))
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: airflow.exceptions.AirflowException: dag_id could not be found: kubernetes_sample. Either the dag did not exist or it failed to parse.
{code}
...note that the DAG isn't in my Docker image, I just cp'ed it to the webserver/scheduler as the blog post did in "Step 5: Upload a test document".
was:
I'm trying to make the DAG from this blog post work:
https://www.techatbloomberg.com/blog/airflow-on-kubernetes/
But it fails with:
{code}
airflow-test/airflow-5cc9cf9f99-v6rtd[scheduler]: [2018-07-10 13:00:41,375] {kubernetes_executor.py:579} INFO - self.running: {(u'kubernetes_sample', u'run_this_first', datetime.datetime(2018, 7, 10, 13, 0, 17, 332338, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=0, name=None))): u'airflow run kubernetes_sample run_this_first 2018-07-10T13:00:17.332338+00:00 --local -sd /root/airflow/dags'}
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: + airflow run kubernetes_sample run_this_first 2018-07-10T13:00:17.332338+00:00 --local -sd /root/airflow/dags
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: [2018-07-10 13:00:57,667] {settings.py:174} INFO - setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: [2018-07-10 13:00:57,919] {__init__.py:51} INFO - Using executor LocalExecutor
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: [2018-07-10 13:00:58,023] {models.py:257} INFO - Filling up the DagBag from /root/airflow/dags
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: [2018-07-10 13:00:58,029] {models.py:348} INFO - File /usr/local/lib/python2.7/dist-packages/airflow/example_dags/__init__.py assumed to contain no DAGs. Skipping.
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: Traceback (most recent call last):
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: File "/usr/local/bin/airflow", line 32, in <module>
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: args.func(args)
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: File "/usr/local/lib/python2.7/dist-packages/airflow/utils/cli.py", line 74, in wrapper
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: return f(*args, **kwargs)
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 475, in run
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: dag = get_dag(args)
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 146, in get_dag
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: 'parse.'.format(args.dag_id))
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: airflow.exceptions.AirflowException: dag_id could not be found: kubernetes_sample. Either the dag did not exist or it failed to parse.
{code}
...note that the DAG isn't in my Docker image, I just cp'ed it to the webserver/scheduler as the blog post did.
> Example Kubernetes pod operator doesn't work
> --------------------------------------------
>
> Key: AIRFLOW-2741
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2741
> Project: Apache Airflow
> Issue Type: Bug
> Reporter: Jon Davies
> Priority: Major
>
> I'm trying to make the DAG from this blog post work:
>
> https://www.techatbloomberg.com/blog/airflow-on-kubernetes/
> But it fails with:
> {code}
> airflow-test/airflow-5cc9cf9f99-v6rtd[scheduler]: [2018-07-10 13:00:41,375] {kubernetes_executor.py:579} INFO - self.running: {(u'kubernetes_sample', u'run_this_first', datetime.datetime(2018, 7, 10, 13, 0, 17, 332338, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=0, name=None))): u'airflow run kubernetes_sample run_this_first 2018-07-10T13:00:17.332338+00:00 --local -sd /root/airflow/dags'}
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: + airflow run kubernetes_sample run_this_first 2018-07-10T13:00:17.332338+00:00 --local -sd /root/airflow/dags
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: [2018-07-10 13:00:57,667] {settings.py:174} INFO - setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: [2018-07-10 13:00:57,919] {__init__.py:51} INFO - Using executor LocalExecutor
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: [2018-07-10 13:00:58,023] {models.py:257} INFO - Filling up the DagBag from /root/airflow/dags
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: [2018-07-10 13:00:58,029] {models.py:348} INFO - File /usr/local/lib/python2.7/dist-packages/airflow/example_dags/__init__.py assumed to contain no DAGs. Skipping.
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: Traceback (most recent call last):
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: File "/usr/local/bin/airflow", line 32, in <module>
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: args.func(args)
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: File "/usr/local/lib/python2.7/dist-packages/airflow/utils/cli.py", line 74, in wrapper
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: return f(*args, **kwargs)
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 475, in run
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: dag = get_dag(args)
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 146, in get_dag
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: 'parse.'.format(args.dag_id))
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: airflow.exceptions.AirflowException: dag_id could not be found: kubernetes_sample. Either the dag did not exist or it failed to parse.
> {code}
> ...note that the DAG isn't in my Docker image, I just cp'ed it to the webserver/scheduler as the blog post did in "Step 5: Upload a test document".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)