[GitHub] [airflow] iostreamdoth edited a comment on issue #16243: When triggering a job via airfow UI, job fails abruptly with Negsignal.SIGSEGV

[GitBox <gi...@apache.org>]

iostreamdoth edited a comment on issue #16243:
URL: https://github.com/apache/airflow/issues/16243#issuecomment-881909400


   I have encountered similar error in airflow 1.10.15 (on google cloud composer) with DataprocInstantiateWorkflowTemplateOperator. I think it has nothing to do with airflow as is but may be with google api's or some sort of library version mismatch.  I have 17 tasks running  in parallel for the above operator. and 15/17 fail with `Task exited with return code Negsignal.SIGSEGV `
   
   I m not able to get in more information but Here is what I did to get fewer errors. 
   
   I override execute function of the above operator.
   ```
   class InstantiateDataProcWorkflowTemplate(dataproc.DataprocInstantiateWorkflowTemplateOperator):
   
       def execute(self, context):
           def callback(operation_future):
               # Handle result.
               result = operation_future.result()
   
           hook = DataprocHook(gcp_conn_id=self.gcp_conn_id, impersonation_chain=self.impersonation_chain)
           self.log.info('Instantiating template %s', self.template_id)
           operation = hook.instantiate_workflow_template(
               project_id=self.project_id,
               location=self.region,
               template_name=self.template_id,
               version=self.version,
               request_id=self.request_id,
               parameters=self.parameters,
               retry=self.retry,
               timeout=self.timeout,
               metadata=self.metadata,
           )
           self.log.info('Template Started.')
           while not operation.done():
               self.log.info('Workflow template still running.')
               time.sleep(600)
           self.log.info('Template Finished.')
   ```
   Doing the above did result in fewer task failing around 3/17 failed but the failure exists for no substantial reason.  
   
   If I remove the while block `while not operation.done():` or operation.result() from the original [here](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/_modules/airflow/providers/google/cloud/operators/dataproc.html#DataprocInstantiateWorkflowTemplateOperator) . The task runs successfully but no wait for completion.  This lead me to believe that there is some bug with asyn operation call on google.api_core. are majority of people facing this problem using google provider operators?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org