[jira] [Work started] (AIRFLOW-7043) CLONE - The mirrored data folder for BigQuery_operators can't be accessed on manual runs

["Valerio Dacanal (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-7043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Work on AIRFLOW-7043 started by Valerio Dacanal.
------------------------------------------------
> CLONE - The mirrored data folder for BigQuery_operators can't be accessed on manual runs
> ----------------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-7043
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-7043
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: contrib, DAG
>    Affects Versions: 1.10.9
>         Environment: Windows 10 Pro, i7-4790S Processor, 16MB RAM
>            Reporter: Valerio Dacanal
>            Assignee: Valerio Dacanal
>            Priority: Blocker
>
> Using Airflow through the Google Cloud Composer, I've placed a series of text files in the /data folder of the bucket as suggested in the documentation for where to store external data files and have written a BigQueryOperator of the following format: 
> {{ 
> with open('/home/airflow/gcs/data/{0}.txt'.format(models.Variable.get('tmpcreatives')),'r') as tmp_file: tmp_transfer = tmp_file.read() bq_sql_tmptransfer = bigquery_operator.BigQueryOperator( task_id = 'task1', sql = """ {0} """.format(tmp_transfer.format(tradata = dag.params["ClientDatabase"]+dag.params["bq_param1"],rawdata = dag.params["ClientDatabase"]+dag.params["bq_param2"])), use_legacy_sql = False ) 
> }} 
> On scheduled runs, the DAG run's fine and completes the task, however if I try to manually trigger the DAG or look at the run logs it comes up with the message 'DAG "DataCreation_DAG_" seems to be missing' This is only an issue when I use the open() function, if I replace that section with a hardcoded string then the DAG works fine even on manual runs, I think it's a bug with mounting the /data file from the cloud shell bucket but not entirely sure



--
This message was sent by Atlassian Jira
(v8.3.4#803005)