You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Trevor Edwards (JIRA)" <ji...@apache.org> on 2017/08/11 15:50:00 UTC
[jira] [Created] (AIRFLOW-1504) Log Cluster Name on Dataproc
Operator When Executed
Trevor Edwards created AIRFLOW-1504:
---------------------------------------
Summary: Log Cluster Name on Dataproc Operator When Executed
Key: AIRFLOW-1504
URL: https://issues.apache.org/jira/browse/AIRFLOW-1504
Project: Apache Airflow
Issue Type: Improvement
Reporter: Trevor Edwards
Priority: Minor
Attachments: dataproc_cluster_logging.py
Since Jinja template substitution happens only just before execution of a task instance, it is not possible to inspect the cluster name populated for a Dataproc operator. This makes it hard to debug issues such as using ts_nodash as part of the clustername, as printing when the DAG is read would give the output:
Cluster name: smoke-cluster-{{ ts_nodash }}
then the operator would execute and the user receives the error:
<HttpError 400 when requesting ... returned "clusterName must be a match of regex '(?:[a-z](?:[-a-z0-9]{0,49}[a-z0-9])?).'">
but the user has no idea what cluster name was provided to the API.
See attached example pipeline.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)