[GitHub] [airflow] nicolas-settembrini commented on issue #16911: Error in passing metadata to DataprocClusterCreateOperator

[GitBox <gi...@apache.org>]

nicolas-settembrini commented on issue #16911:
URL: https://github.com/apache/airflow/issues/16911#issuecomment-992769308


   Hi, sorry to write here but i didn't find another place talking about this. 
   
   I am using Version: 2.1.4+composer and I have a DAG where i defined the DataprocClusterCreateOperator like this:
   
       create_dataproc =  dataproc_operator.DataprocClusterCreateOperator(
         task_id='create_dataproc',
         cluster_name='dataproc-cluster-demo-{{ ds_nodash }}',
         num_workers=2,
         region='us-east4',
         zone='us-east4-a',
         subnetwork_uri='projects/example',
         internal_ip_only=True,
         tags=['allow-iap-ssh'],
         init_actions_uris=['gs://goog-dataproc-initialization-actions-us-east4/connectors/connectors.sh'],
         metadata=[('spark-bigquery-connector-url','gs://spark-lib/bigquery/spark-2.4-bigquery-0.23.1-preview.jar')],
         labels=dict(equipo='dm',ambiente='dev',etapa='datapreparation',producto='x',modelo='x'),
         master_machine_type='n1-standard-1',
         worker_machine_type='n1-standard-1',
         image_version='1.5-debian10'
         )
         
   I passed the metadata as a sequence of tuples as i read here, using the dict is not working.
   
   Also, the metadata is not being rendered in the cluster_config. 
   
   @pateash could you please explain a more detailed way to use your workaround? In what part of the dag could i use the workaround?
   
   Thanks in advance
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org