You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/03/11 03:01:31 UTC

[GitHub] [airflow] super017 opened a new issue #14713: how to run a DAG in distrbuted airflow

super017 opened a new issue #14713:
URL: https://github.com/apache/airflow/issues/14713


   The version of airflow I deployed is 1.10.14.A master and three workers.I want to complete several Workers to complete a DAG.
   my code :
   from datetime import timedelta,datetime
   from airflow import DAG
   from airflow.operators.python_operator import PythonOperator
   from airflow.utils.dates import days_ago
   from airflow.operators.bash_operator import BashOperator
   import time
   import math
   
   
   default_args = {
       'owner':'airflow',
       'retries':3,
       'depends_on_past':False,
       'start_date': days_ago(2),
       'provide_context':True,
   }
   dag = DAG(
       'xhcyeh',
       default_args=default_args,
       schedule_interval=timedelta(minutes=5)
   )
   
   def function1(**kwargs):
       sum =2
       #x = kwargs['dag_run'].conf['x']
       #x = kwargs.get("dag_run").conf.get("x")
       x = 5
       for i in range(x):
           sum +=i
       print(sum)
       f = open('/home/lis/airflow/dags/x.txt','w')
       f.write(str(sum))
       f.close()
       kwargs['task_instance'].xcom_push(key='sea1',value=str(sum))
       return sum
   
   def function2(**kwargs):
       sum = kwargs['task_instance'].xcom_pull(key='sea1',task_ids='function1')
       f = open('/home/ls/airflow/dags/y.txt','w')
       f.write(str(math.pow(int(sum),2)))
       f.close()
       return math.pow(int(sum),2)
   
   run_this = PythonOperator(
       task_id='function1',
       python_callable=function1,
       queue='cdh6-2-node93',
       dag=dag,
   )
   
   t2 = PythonOperator(
       task_id='function2',
       python_callable= function2,
       queue='cdh6-2-node94',
       dag=dag,
   )
   run_this >> t2
   I want to test two workers working together to complete a simple calculation.What happens is that the first function is finished and the second function stays in the scheduled state.What happens is that the first function is finished and the second function stays in the scheduled state.The flower log shows celery failed. There is also no record of parameter passing in the XCOM table of the background database.Is there something wrong with my code?What changes should I make


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] mik-laj commented on issue #14713: how to run a DAG in distrbuted airflow

Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #14713:
URL: https://github.com/apache/airflow/issues/14713#issuecomment-796733188


   Github Issues is not intended to provide support. Please inquire on the "troubleshooting" channel on the Slack workspace of this project.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] mik-laj closed issue #14713: how to run a DAG in distrbuted airflow

Posted by GitBox <gi...@apache.org>.
mik-laj closed issue #14713:
URL: https://github.com/apache/airflow/issues/14713


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #14713: how to run a DAG in distrbuted airflow

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #14713:
URL: https://github.com/apache/airflow/issues/14713#issuecomment-796400038


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org